At-least-once #

In modern distributed systems, data loss is one of the worst failure scenarios software engineers must avoid. To answer this reliability challenge, RabbitMQ is by default designed to support and operate under the At-Least-Once delivery guarantee spectrum.

The basic principle of the At-Least-Once guarantee is absolute priority on data integrity: the broker guarantees that every published message will be delivered and processed by consumers at least once. However, to provide this loss-free guarantee, distributed systems must pay a price in the form of the possibility of duplicate messages (duplication). In production environments, momentary network failures or consumer application crashes at the wrong time can cause the same message to be delivered repeatedly. This article deeply discusses how RabbitMQ achieves the At-Least-Once guarantee, dissects the scenarios causing message duplication, formulates idempotency handling strategies on the consumer side, and presents robust Go code implementations.

The Position of At-Least-Once in the Message Lifecycle #

The At-Least-Once guarantee is achieved by applying a strict two-way confirmation cycle on the publishing side (Publisher Confirms) and the consumption side (Consumer Acknowledgements).

flowchart TD
    Producer["Producer (Publisher)"] -->|"1. Publish + Confirm Mode"| Exchange["Exchange"]
    Exchange -->|"2. Route Message"| Queue["Queue (Durable)"]
    Queue -->|"3. Consensus Replication (Quorum)"| Replicas["Follower Node"]
    
    Queue -->|"4. Push (basic.deliver)"| Consumer["Consumer Application"]
    Consumer -->|"5. Business Process Success"| DB["Database (Atomic Commit)"]
    DB -->|"6. Send Manual ACK"| Queue
    Queue -->|"7. Delete Message"| Delete["Done"]

    style Producer stroke:#0288d1,stroke-width:2px
    style Queue stroke:#388e3c,stroke-width:2px
    style Consumer stroke:#e65100,stroke-width:2px

In this model, messages are never deleted from broker storage before the consumer sends an explicit basic.ack signal. This signal may only be sent after all business logic and database storage on the consumer side are successfully completed.

How Does RabbitMQ Achieve the At-Least-Once Guarantee? #

The success of the At-Least-Once guarantee doesn’t rest on a single feature; it’s the result of close collaboration between the producer, the RabbitMQ broker, and the consumer. There are three main pillars composing this guarantee:

1. The Publishing Side (Producer): Publisher Confirms #

Without broker confirmation, producers have no way to know whether the messages they sent truly reached the broker. Producers enable Publisher Confirms mode on the communication channel. The broker sends an ACK back to the producer only after the message is successfully received, routed to the queue, and (if persistent) secured to disk. If a network issue occurs before the broker receives the message, the producer detects the lost confirmation and performs a retry.

2. The Storage Side (Broker): Durable & Persistent #

To ensure messages survive broker restarts or hardware failures, we must configure permanent storage:

  • Declaring queues with the durable = true flag so queue structures aren’t deleted when the broker restarts.
  • Sending messages with the delivery_mode = 2 (persistent) property so the message body payload is physically written to disk segment files (msg_store_persistent) using fsync system calls.
  • If using Quorum Queues, messages are consistently replicated to a majority of cluster nodes using Raft logs before the broker gives a success confirmation to the producer.

3. The Consumption Side (Consumer): Manual Acknowledgements #

Consumers must turn off automatic confirmation (autoAck = false) and explicitly send the basic.ack command manually after all data processing completes. Until the manual ACK is received by the broker, the broker marks the message as Unacknowledged in RAM memory and keeps its physical copy.

Main Scenarios Where Duplicate Messages Occur #

Because distributed systems always face network uncertainty, message duplication is an unavoidable logical consequence of preventing message loss.

Here are three main scenarios where message duplication occurs in production:

Scenario 1: Consumer Crash After Business Processing, Before Sending the ACK #

This is the most common failure scenario in microservices applications:

  1. A consumer receives an order message from RabbitMQ.
  2. The consumer successfully processes the order and deducts stock in the main database.
  3. Right before the consumer can call the d.Ack(false) function to send it over the TCP socket to the broker, the consumer server crashes or loses power.
  4. RabbitMQ detects that the TCP connection with that consumer dropped suddenly.
  5. Because the message status is still Unacknowledged, RabbitMQ requeues the message to the queue head.
  6. When a new consumer starts up, it receives the same order message for the second time, even though the stock in the database was actually already deducted on the first attempt.

Scenario 2: Network Failure (Network Partition) #

This scenario occurs when both the consumer and broker are healthy, but the communication path between them is disrupted:

  1. The consumer successfully processes the message and calls d.Ack(false).
  2. The ACK signal is sent to the network, but a momentary network glitch causes that ACK packet to be lost mid-journey and never reach the broker.
  3. The broker considers the consumer unresponsive because the TCP connection experiences a heartbeat timeout.
  4. The broker returns the message to the queue and pushes it to another consumer. The other consumer now processes a message that was actually already successfully completed by the first consumer.

Scenario 3: Channel Closure from a Protocol Exception #

If a consumer makes an AMQP protocol operation mistake (like trying to ACK a message with the wrong delivery_tag, or calling ACK on a different channel):

  • The broker force-closes (channel exception) that entire channel.
  • The channel closure automatically cancels all in-flight message statuses being processed on that channel, forcing the broker to mass-requeue all those messages, triggering double processing of messages that were actually running successfully in consumer memory.

Idempotency Handling Strategies on the Consumer Side #

Because message duplication is an unavoidable architectural certainty under the At-Least-Once model, consumer applications must be designed to be Idempotent. Idempotent means our application can receive and process the same message repeatedly, but only produces the business side effect exactly once.

There are three main strategies for building idempotent consumers in production:

1. Unique Event ID & Deduplication Store #

Every message published by the producer must be tagged with a unique UUID identifier in the message_id property or inside the body payload.

  • How It Works: Consumers maintain a centralized fast storage database (like Redis with an expiration/TTL, or a dedicated relational database table). Before processing a message, the consumer checks whether that message ID already exists in the deduplication store. If it does, the consumer immediately ignores the message and sends an ACK to the broker without reprocessing the business logic.

2. Database Constraints (Unique Indexes) #

Leveraging relational database integrity features as the last line of defense.

  • How It Works: If a message represents a data creation process (e.g., new user registration), we can create a Unique Constraint on the email address column in the database. If a duplicate message occurs, the second write attempt is atomically rejected by the database with a duplicate key error. The consumer catches this error, considers the processing successful, and sends an ACK to the broker without triggering a system failure.

3. State Machine-Based Business Logic #

Avoiding relative accumulating math operations and replacing them with explicit logical state transition operations.

  • Anti-Pattern Example: Sending the message command Amount = Amount - 10000 to deduct a balance. If duplication occurs, the user’s balance is deducted twice (-20000).
  • Idempotent Solution Example: Sending a message containing an order status transition instruction from PENDING to PAID on a specific payment transaction ID row. Before changing the balance, check whether that transaction ID row already has the PAID status in the database. If the status is already PAID, safely ignore the duplicate message.

Go Code Implementation: Robust Consumer with Redis Deduplication #

Here is a complete Go language implementation example for an At-Least-Once consumer. This code uses a Redis connection to atomically check and store message deduplication keys before executing the main business logic, guaranteeing safe processing against duplication dangers.

package main

import (
	"context"
	"fmt"
	"log"
	"os"
	"os/signal"
	"syscall"
	"time"

	"github.com/redis/go-redis/v9"
	amqp "github.com/rabbitmq/amqp091-go"
)

var (
	ctx         = context.Background()
	rdb         *redis.Client
	amqpURL     = "amqp://guest:***@localhost:5672/"
	queueName   = "durable-order-payments"
	redisPrefix = "processed_message:"
)

func init() {
	// Initialize the Redis Cache Connection for the Deduplication Store
	rdb = redis.NewClient(&redis.Options{
		Addr: "localhost:6379",
	})
}

func main() {
	// 1. Open a TCP Connection to RabbitMQ
	conn, err := amqp.Dial(amqpURL)
	if err != nil {
		log.Fatalf("Gagal terhubung ke RabbitMQ: %s", err)
	}
	defer conn.Close()

	ch, err := conn.Channel()
	if err != nil {
		log.Fatalf("Gagal membuka channel: %s", err)
	}
	defer ch.Close()

	// 2. Declare a Durable Queue
	_, err = ch.QueueDeclare(
		queueName,
		true,  // durable: true (Queue survives restarts)
		false, // auto-delete
		false, // exclusive
		false, // no-wait
		nil,   // arguments
	)
	if err != nil {
		log.Fatalf("Gagal deklarasi antrean: %s", err)
	}

	// 3. Configure Prefetch QoS for Fair Dispatch
	err = ch.Qos(10, 0, false)
	if err != nil {
		log.Fatalf("Gagal menyetel QoS: %s", err)
	}

	// 4. Register the Consumer with manual ACK (autoAck = false)
	msgs, err := ch.Consume(
		queueName,
		"",    // consumer tag
		false, // autoAck: false (THE MAIN KEY TO AT-LEAST-ONCE ON THE CONSUMER!)
		false, // exclusive
		false, // no-local
		false, // no-wait
		nil,   // arguments
	)
	if err != nil {
		log.Fatalf("Gagal mendaftarkan konsumen: %s", err)
	}

	sigChan := make(chan os.Signal, 1)
	signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)

	log.Println("[*] Konsumen At-Least-Once aktif. Tekan Ctrl+C untuk keluar.")

	go func() {
		for d := range msgs {
			// Check the uniqueness of the Message ID (Deduplication)
			messageID := d.MessageId
			if messageID == "" {
				log.Printf("[Peringatan] Pesan dengan Tag %d tidak memiliki MessageId. Mengabaikan pengecekan deduplikasi...", d.DeliveryTag)
				// DON'T ignore messages in production; here we only give a warning
			}

			// Use an atomic Redis SETNX (Set if Not Exists) operation
			redisKey := fmt.Sprintf("%s%s", redisPrefix, messageID)
			// We set the deduplication key with an expiration time (TTL) of 24 hours
			isUnique, err := rdb.SetNX(ctx, redisKey, "processed", 24*time.Hour).Result()
			
			if err != nil {
				log.Printf("[ERROR] Gagal berkomunikasi dengan Redis: %s. Melakukan requeue pesan...", err)
				// If Redis is down, return the message to the queue to retry
				d.Nack(false, true)
				continue
			}

			if !isUnique {
				// DUPLICATE MESSAGE DETECTED: Skip the business process, send ACK directly
				log.Printf("[INFO] Pesan duplikat terdeteksi (ID: %s). Langsung mengirim ACK...", messageID)
				d.Ack(false)
				continue
			}

			// 5. Execute the Main Business Logic
			log.Printf("Memproses pembayaran transaksi untuk ID Pesan: %s", messageID)
			processSuccess := executePaymentLogic(d.Body)

			if !processSuccess {
				// PROCESS FAILED: Delete the Redis deduplication key and requeue
				log.Printf("[INFO] Gagal memproses transaksi bisnis. Menolak pesan...")
				rdb.Del(ctx, redisKey)
				d.Nack(false, true) // requeue = true
				continue
			}

			// 6. Send the Manual ACK after Successful Processing & Database Commit
			err = d.Ack(false)
			if err != nil {
				log.Printf("[ERROR] Gagal mengirim ACK ke broker: %s", err)
				// If the ACK fails to send because the network drops now,
				// the message is requeued by the broker and redelivered.
				// However, on the second attempt, the Redis SETNX detects the "processed" key
				// and prevents the business transaction from being processed twice.
			} else {
				log.Printf("✓ Sukses memproses dan meng-ACK pesan: %s", messageID)
			}
		}
	}()

	<-sigChan
	log.Println("[*] Shutdown aman diselesaikan.")
}

func executePaymentLogic(payload []byte) bool {
	// Simulate relational database balance deduction logic
	return true
}

Anti-Patterns vs Practical Solutions in Production #

Avoid the following fatal mistakes when designing systems with the At-Least-Once guarantee:

Anti-Pattern: Using Auto-Ack and Moving Requeue to the Application Side #

A developer disables manual ACK in RabbitMQ (autoAck = true) for fast throughput, but writes internal error-handling code in the consumer application tasked with manually re-publishing failed messages to the same queue.

Why is this wrong? #

This pattern is very vulnerable to data loss. If the consumer application crashes mid-processing (before republishing the failed message), the message is lost forever because the broker already deleted it from the start. Additionally, manually republishing messages changes the original message metadata (like the original send date, original message ID, and routing headers), and makes error troubleshooting analysis harder.

  • Solution: Let RabbitMQ handle requeueing natively. Use manual confirmation (autoAck = false). If a temporary processing failure occurs, just call d.Nack(false, true). The broker safely returns the message to the queue without changing the original message metadata.

Summary #

  • At-Least-Once Guarantee — The guarantee that every message is delivered and processed by consumers at least once, eliminating the risk of silent data loss.
  • Message Duplication Risk — A logical consequence of this model. Duplication occurs from consumer crashes after successful business processing but before the ACK is sent, or from network disruptions.
  • Achievement Pillars — Requires Publisher Confirms on the producer, durable queues and persistent messages in the broker (or quorum queues), and manual ACKs on the consumer.
  • Must Be Idempotent — Consumers must be designed idempotent using unique deduplication key checks (e.g., via Redis), unique database index constraints, or state transition logic.
  • Correct Execution Order — Always complete business logic and commit database transactions first, then call the d.Ack instruction to the broker.
  • The Role of Quorum Queues — Quorum queues strengthen At-Least-Once by ensuring data is consistently replicated to a majority of cluster nodes before producers receive confirmation.

← Previous: At-most-once   Next: Exactly-once →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact