Exactly-once #

In discussions of message queuing system architecture, the Exactly-Once delivery guarantee is the most desired guarantee among developers. This guarantee promises that every message will never be lost, and at the same time, never processed more than once by consumer applications.

However, in the real world of distributed systems, claims about native exactly-once guarantees are often misleading. Theoretically and practically, guaranteeing that a physical message flows over network cables exactly once is impossible when faced with hardware and network failures. Nevertheless, we can design our system architecture so the business effect of message processing occurs exactly once (exactly-once processing/effect). This article dissects the theoretical limits of distributed systems through the Two Generals Theorem, separates the fundamental difference between physical delivery and logical processing, formulates strategies for achieving the exactly-once effect, and presents transactional consumer code implementations in Go.

The Theoretical Limits of Distributed Systems: The Two Generals Theorem #

To understand why pure exactly-once delivery is a physical impossibility, we must study a classic computer science problem: the Two Generals Problem.

The Two Generals Problem Case Study #

Imagine two army generals (General A and General B) who want to attack the same enemy city. They can only defeat the enemy if they attack simultaneously. If only one general attacks, that general’s army is destroyed. The only communication medium between the two generals is couriers who must pass through enemy territory, meaning the couriers risk being captured (unreliable communication).

flowchart LR
    A["General A<br>'Attack tomorrow at 8!'"] -- "Courier (Unreliable Network)" --> B["General B"]
  1. General A sends a courier to General B with the message: “Attack tomorrow at 8 AM”.
  2. The courier arrives successfully. However, General B knows General A doesn’t know whether the courier arrived or was captured. General B must send a confirmation courier (ACK) back to General A.
  3. General B sends a courier back: “Message received, I agree to attack at 8”.
  4. The confirmation courier arrives successfully at General A. However, now General A realizes General B doesn’t know whether the confirmation courier survived or was captured by the enemy. If General B thinks its confirmation didn’t arrive, General B won’t dare attack for fear General A cancels the attack.
  5. General A must send a confirmation-of-confirmation courier (ACK-of-ACK) to General B.
  6. This process repeats endlessly.

In distributed systems mathematics, it has been proven that there is no finite number of confirmations that can guarantee both parties are 100% agreed on communication status over an unreliable medium.

In RabbitMQ architecture, the broker acts as General A and the consumer acts as General B. When a momentary network failure occurs exactly when the consumer sends the ACK confirmation to the broker, the broker cannot distinguish whether the consumer crashed before processing the data, or the process succeeded but the ACK was lost on the way. For data safety, the broker must choose the safest option: resend the message (requeue), which automatically cancels the exactly-once delivery guarantee.

Exactly-Once Delivery vs Exactly-Once Processing (Effect) #

Even though we can’t prevent messages from being delivered more than once by the network (At-Least-Once), we can guarantee that our application only executes the side effects of a message exactly once. We must strictly separate these two concepts:

1. Exactly-Once Delivery (Physical) #

The broker guarantees the physical message only flows over network cables exactly once from the broker socket to the consumer socket. This is a theoretical impossibility on unreliable networks because if a connection drop occurs mid-delivery of a TCP packet, the broker must redeliver to prevent data loss.

2. Exactly-Once Processing / Effect (Logical) #

The broker performs redeliveries (message duplication on the network is allowed), but our consumer application has a mechanism to detect that duplication and ensure business logic (like balance deductions, inventory updates) is only executed once. This is the realistic target we must implement at the application code level.

Architecture Strategies for Achieving the Exactly-Once Effect #

To realize the exactly-once effect in our system, we must apply strict transactional design patterns on the producer and consumer sides:

1. The Producer Side: Transactional Outbox Pattern #

To prevent producers from publishing dangling messages (e.g., an order is stored in the database, but the message-sending process to RabbitMQ fails), we use the Transactional Outbox Pattern:

  1. The producer application writes the main business data and the message-to-send data (in the outbox table) in the same local database transaction atomically.
  2. A background worker periodically reads the outbox table, publishing messages to RabbitMQ with Publisher Confirms enabled.
  3. After receiving a success confirmation from RabbitMQ, the worker marks the message status in the outbox table as sent or deletes it.

2. The Consumer Side: Atomic Commit Pattern & Manual ACK after Commit #

On the receiving side, we must tie the message uniqueness record to the main database transaction atomically:

  1. Start a Database Transaction: When receiving a message, open a new relational database transaction (BEGIN TRANSACTION).
  2. Verify & Write Deduplication: Query whether the unique message ID already exists in the deduplication log table (e.g., the processed_messages table). If it exists, immediately roll back the transaction and send an ACK to the broker to discard the duplicate message. If not, insert the message ID into the deduplication log table.
  3. Execute Business Logic: Run business data updates (e.g., deduct account balance) in the same transaction.
  4. Commit the Transaction: Complete the transaction atomically (COMMIT). At this point, the message uniqueness record and business effect are locked together. If the database goes down mid-way, all operations are automatically rolled back without partial side effects.
  5. Send the ACK Signal: After the commit succeeds, call d.Ack(false) to delete the message from RabbitMQ.
flowchart TD
    A["Receive Message (Delivery)"] --> B["Open Database Transaction"]
    B --> C["Message ID Record & Business Logic"]
    C --> D{"Database Commit Success?"}
    D -->|No| E["DB Rollback & Requeue Message"]
    D -->|Yes| F["Send Manual ACK to Broker"]

If the consumer application crashes right after step 4 (Commit successful) but before step 5 (ACK sent):

  • RabbitMQ requeues the message and sends it back.
  • On the second attempt, when the consumer tries step 2, the message ID write attempt to the deduplication table is atomically cancelled because it collides with the Unique Constraint of the primary key already written on the first attempt.
  • The consumer detects this collision, skips the business logic, and calls the ACK to cleanly clear the queue without double processing.

Go Code Implementation: Atomic Transactional Consumer #

Here is an atomic transactional consumer implementation example in Go using a relational SQL database (using the standard driver) to demonstrate the atomic commit pattern for achieving the exactly-once effect.

package main

import (
	"context"
	"database/sql"
	"errors"
	"log"
	"os"
	"os/signal"
	"syscall"
	"time"

	_ "github.com/lib/pq" // PostgreSQL Driver
	amqp "github.com/rabbitmq/amqp091-go"
)

const (
	amqpURL   = "amqp://guest:***@localhost:5672/"
	queueName = "transactional-orders"
	dbDSN     = "postgres://postgres:***@localhost:5432/shop?sslmode=disable"
)

type OrderMsg struct {
	OrderID   string  `json:"order_id"`
	UserID    string  `json:"user_id"`
	Amount    float64 `json:"amount"`
}

func main() {
	// Initialize the PostgreSQL Connection
	db, err := sql.Open("postgres", dbDSN)
	if err != nil {
		log.Fatalf("Gagal terhubung ke database: %s", err)
	}
	defer db.Close()

	// Open a TCP Connection to RabbitMQ
	conn, err := amqp.Dial(amqpURL)
	if err != nil {
		log.Fatalf("Gagal terhubung ke RabbitMQ: %s", err)
	}
	defer conn.Close()

	ch, err := conn.Channel()
	if err != nil {
		log.Fatalf("Gagal membuka channel: %s", err)
	}
	defer ch.Close()

	// Declare a Durable Queue
	_, err = ch.QueueDeclare(
		queueName,
		true,  // durable
		false,
		false,
		false,
		nil,
	)
	if err != nil {
		log.Fatalf("Gagal deklarasi antrean: %s", err)
	}

	// Limit the QoS prefetch
	err = ch.Qos(5, 0, false)
	if err != nil {
		log.Fatalf("Gagal menyetel QoS: %s", err)
	}

	msgs, err := ch.Consume(
		queueName,
		"",
		false, // autoAck: false (manual ACK for transactional control)
		false,
		false,
		false,
		nil,
	)
	if err != nil {
		log.Fatalf("Gagal mendaftarkan konsumen: %s", err)
	}

	sigChan := make(chan os.Signal, 1)
	signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	go func() {
		for d := range msgs {
			messageID := d.MessageId
			if messageID == "" {
				log.Printf("[ERROR] Pesan tidak memiliki MessageId. Mengabaikan pemrosesan demi keamanan data.")
				d.Reject(false) // Discard the message without requeue
				continue
			}

			// Start the Atomic Database Transaction Process
			processErr := executeTransactionalProcessing(ctx, db, messageID, d.Body)

			if processErr != nil {
				if errors.Is(processErr, errDuplicateMessage) {
					// DUPLICATE MESSAGE DETECTED: Send ACK directly to delete it from the queue
					log.Printf("[INFO] Duplikasi terdeteksi untuk ID: %s. Melompati logika bisnis dan meng-ACK pesan.", messageID)
					d.Ack(false)
				} else {
					// TEMPORARY INFRASTRUCTURE ERROR: Requeue to retry
					log.Printf("[ERROR] Kegagalan database: %s. Melakukan requeue pesan...", processErr)
					d.Nack(false, true) // requeue = true
				}
				continue
			}

			// 5. Send the ACK only after the database COMMIT is 100% SUCCESSFUL
			err = d.Ack(false)
			if err != nil {
				log.Printf("[ERROR] Transaksi DB sukses commit, namun gagal mengirim ACK: %s", err)
				// The DB data is safe; RabbitMQ resends the message because the ACK didn't arrive.
				// However, on the second delivery, the insert into the processed_messages table
				// detects the duplicate key and immediately ACKs the message without double side effects.
			} else {
				log.Printf("✓ Transaksi selesai secara atomik untuk ID: %s", messageID)
			}
		}
	}()

	<-sigChan
	log.Println("[*] Memulai graceful shutdown...")
	cancel()
	time.Sleep(1 * time.Second)
}

var errDuplicateMessage = errors.New("pesan duplikat terdeteksi")

// Transaction processing function with the atomic commit pattern
func executeTransactionalProcessing(ctx context.Context, db *sql.DB, messageID string, body []byte) error {
	// 1. Start the SQL Transaction
	tx, err := db.BeginTx(ctx, nil)
	if err != nil {
		return err
	}
	// Make sure we always roll back if a panic or error occurs before commit
	defer tx.Rollback()

	// 2. Try to Record the Message ID in the Deduplication Table
	// The processed_messages table must have a primary key or unique constraint on the message_id column
	queryDeduplicate := `INSERT INTO processed_messages (message_id, processed_at) VALUES ($1, NOW())`
	_, err = tx.ExecContext(ctx, queryDeduplicate, messageID)
	if err != nil {
		// Check whether the error is caused by a unique constraint collision (duplicate key)
		if isUniqueConstraintViolation(err) {
			return errDuplicateMessage
		}
		return err
	}

	// 3. Execute the Main Business Logic in the Same Transaction
	// Example: Deduct the user's balance
	queryBalance := `UPDATE users SET balance = balance - 150000 WHERE id = $1 AND balance >= 150000`
	res, err := tx.ExecContext(ctx, queryBalance, "user_id_dummy")
	if err != nil {
		return err
	}
	rows, err := res.RowsAffected()
	if err != nil || rows == 0 {
		return fmt.Errorf("gagal memotong saldo, dana tidak cukup atau user tidak ditemukan")
	}

	// 4. Commit the Transaction Atomically
	err = tx.Commit()
	if err != nil {
		return err
	}

	return nil
}

func isUniqueConstraintViolation(err error) bool {
	// In production systems, parse the PostgreSQL error code (e.g., code '23505' for unique_violation)
	return true // Simplified for the illustration example
}

Anti-Patterns vs Practical Solutions in Production #

Avoid the following fatal assumption mistakes when dealing with the exactly-once concept:

Anti-Pattern: Using Non-Transactional Redis as a Deduplicator Separate from the Business Database #

Receiving a message, verifying and writing the deduplication key to a Redis Cache, then writing the main business logic to a PostgreSQL database separately without unified atomic transaction supervision.

Why is this wrong? #

Redis and PostgreSQL are two independent storage systems that don’t know each other’s transaction status. If the Redis key write succeeds, but then the business transaction data write to PostgreSQL fails (e.g., from a data type collision or PostgreSQL suddenly restarting), the PostgreSQL database rolls back but the Redis deduplication key is already stored as processed. When RabbitMQ resends the message after detecting the failure, the second attempt is immediately blocked by Redis because the key already exists, so our business transaction data will never be written. This triggers severe data inconsistency.

  • Solution: Always store the message deduplication log key in the same database engine as the main business logic database. Use relational database ACID transactions to secure the message deduplication log write and main business data together in a single atomic commit block.

Summary #

  • Physical Exactly-Once Delivery Is Impossible — Network transmission uncertainty (the Two Generals Theorem) prevents guaranteeing that a physical message only flows once over network cables without data loss risk.
  • Exactly-Once Processing Is the Realistic Target — We allow physical delivery duplication (At-Least-Once), but design application code so business side effects are only executed exactly once.
  • Consumer Atomic Commit Pattern — Open a new relational database transaction, insert the unique message ID into a deduplication table (with a unique constraint), execute business data, commit, then call the manual ACK.
  • Automatic Rollback Impact — If the database transaction process fails mid-way, all writes are rolled back and the message returns to the queue without partial side effects that damage data.
  • Outbox Pattern on the Producer — The publishing side must also use an outbox pattern integrated with local database transactions to guarantee messages aren’t sent dangling without the main data commit.
  • Not a Configuration Feature — The exactly-once guarantee is the result of comprehensive architectural design at the application level, not an instant feature activated just by ticking a broker configuration box.

← Previous: At-least-once   Next: Publisher confirms →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact