Retry Pattern #

In developing message-based systems in production, we can’t only think about the happy path. How our system responds to failures is often the main differentiator between hobby-scale applications and resilient enterprise-class architectures. Throughout this section, we’ve studied various error-handling elements separately—from the requeue vs drop dilemma, Dead Letter Exchange (DLX) functionality, Message TTL lifetimes, to the Exponential Backoff algorithm for dampening thundering herds. However, in the real world, these elements shouldn’t stand alone partially.

To build a system with high fault tolerance, we must combine all those concepts into a unified architecture design pattern called the Three-Tier Retry Pattern. This pattern acts as an industry-standard architecture blueprint separating failed message traffic, managing wait times asynchronously, and automatically isolating corrupted messages without disrupting the smooth processing of other healthy messages in our main queue.

Three-Tier Retry Topology #

The Three-Tier Retry Pattern architecture divides the queue topology on our RabbitMQ broker into three independent paths, each holding very specific functional responsibilities. By separating these paths, we avoid contaminating the main queue with problematic messages.

The three main queues forming this pattern include:

1. Main Queue #

This queue is the first entry gate for all normal data traffic published by producers. Our main consumers actively listen to this queue under daily operational conditions.

• Characteristics: High throughput, optimized for low latency. This queue is configured with a Dead Letter Exchange (DLX) pointing to the retry exchange, so if processing failures occur, messages can be immediately removed from this main queue as quickly as possible to free up prefetch quotas.

2. Retry Delay Queue #

This queue acts as a temporary holding area for all messages that fail to process due to transient (temporary) errors.

• Characteristics: No consumer applications are connected to this queue. This queue leverages the Queue-level TTL property (x-message-ttl) as an asynchronous timer countdown.
• This queue is also configured with a Dead Letter Exchange pointing back to the Main Exchange. Once the TTL wait period elapses, the broker automatically flows the message back to the main queue to be retried.

3. Error DLQ #

This terminal queue is the final resting place for messages experiencing permanent failures from the start, or transient messages that have exceeded the Max Retry Limit we set.

• Characteristics: This queue is only accessed passively. Messages inside are safely stored without expiration time limits for manual audit needs, code bug fixes, or controlled transaction data reconciliation by operations teams.

flowchart TD
    Producer["Producer"] --> MainEx(("Main Exchange"))
    MainEx --> MainQueue["Main Queue"]
    Consumer["Main Consumer"] -. "Consume" .-> MainQueue
    
    MainQueue -->|"Nack (requeue=false)"| RetryEx(("Retry Exchange"))
    RetryEx --> DelayQueue["Delay Queue (TTL Delay)"]
    DelayQueue -. "Expired" .-> MainEx
    
    Consumer -->|"If Retry > Max"| ErrorEx(("Error Exchange"))
    ErrorEx --> ErrorDLQ["Error DLQ (Manual Audit)"]

By applying this three-tier physical separation, our main queue keeps flowing millions of other successful transaction messages at maximum speed, while failed messages are held asynchronously in the delay queue or safely isolated in the Error DLQ.

Step-by-Step Message Lifecycle #

To understand how data flows inside this architecture, let’s trace a message’s lifecycle from start to finish of processing in several scenarios:

Scenario A: Normal Processing (Happy Path) #

1. A producer publishes a message to main.exchange with the order.create routing key.
2. The broker routes the message to the main.queue.
3. The main consumer takes the message, successfully processes the business data into the database, and sends a manual Ack.
4. The broker permanently deletes the message from main.queue.

Scenario B: Experiencing a Transient Failure (Retry Flow) #

1. A producer publishes a message to main.queue.
2. The main consumer takes the message, but the database connection times out during query execution.
3. The consumer detects this error as transient, reads the x-retry-count counter header on the message (the current value is 0).
4. The consumer copies the message payload, adds a new x-retry-count = 1 header, then publishes that new message to retry.exchange with the order.retry routing key.
5. The consumer sends an Ack on the original message in main.queue to delete it from the main queue.
6. The broker places that new message in the retry.delay.10s.queue with a 10-second TTL.
7. For 10 seconds, the message stays in the delay queue without disturbing the main consumer.
8. After 10 seconds, the message expires. The broker routes that message through the DLX back to main.exchange, which then routes it back to main.queue.
9. The main consumer receives that message again to try processing it from scratch.

Scenario C: Retry Limit Exhausted (Isolation to the Error DLQ) #

1. After being tried 3 times (counter x-retry-count = 3), the database is still down or the message payload turns out to contain a permanent logic bug.
2. The consumer takes the message for the 4th attempt; the process fails again.
3. The consumer reads the x-retry-count header with value 3. Because our maximum retry limit is set to 3, the consumer declares this message a Poison Message.
4. The consumer publishes that message to the isolation exchange error.exchange with the order.error routing key, then sends an Ack to the main queue to clean main.queue.
5. The broker places that message into the terminal queue order.error.dlq.
6. The retry flow stops. The message rests in the final DLQ and the system triggers monitoring alarms so developer teams intervene for investigation.

Open Circuit vs Closed Circuit Design (Circular Retry Mitigations) #

One of the most common architecture design errors when implementing Retry Patterns is unintentionally creating a Circular Retry Loop.

Circular retry occurs when we design a flow where failed messages go to the delay queue, then after the TTL expires return to the main queue, and if they fail again are immediately sent back to the delay queue without any failure counter limit evaluation. The problematic message keeps spinning forever like a vicious cycle between the main queue and delay queue. This causes never-ending data pile-ups, constantly high broker CPU consumption, and broker RAM memory leaks.

To mitigate this danger, we must design the system with an Open Circuit at the consumer application level:

• Mandatory Counter Checks: Every time a consumer catches a processing failure, the consumer must check the retry counter value (x-retry-count or the x-death history) first.
• Strict Termination Criteria: If the counter has exceeded the maximum threshold (e.g., a maximum of 3 or 5 times), the consumer is strictly forbidden from sending the message back to the retry exchange. The message must be immediately evacuated to the destination Error Exchange to break the circular flow.
• Routing Key Separation: Differentiate routing keys for the normal flow (order.create), retry flow (order.retry), and permanent error flow (order.error). Never mix all three using overly loose * or # wildcards on a single topic exchange to avoid message misrouting.

Go Code Implementation (Golang) #

Here is a complete Go consumer program implementing the Three-Tier Retry Pattern architecture fully and safely from circular retry loop dangers.

package main

import (
	"context"
	"encoding/json"
	"log"
	"os"
	"os/signal"
	"syscall"
	"time"

	amqp "github.com/rabbitmq/amqp091-go"
)

// MessagePayload represents our main transaction message structure
type MessagePayload struct {
	ID        string    `json:"id"`
	AccountNo string    `json:"account_no"`
	Amount    float64   `json:"amount"`
	Timestamp time.Time `json:"timestamp"`
}

func main() {
	// 1. Open the physical connection to the RabbitMQ broker
	conn, err := amqp.Dial("amqp://guest:***@localhost:5672/")
	if err != nil {
		log.Fatalf("Gagal membuka koneksi ke RabbitMQ: %v", err)
	}
	defer conn.Close()

	// 2. Create an AMQP channel
	ch, err := conn.Channel()
	if err != nil {
		log.Fatalf("Gagal membuka channel AMQP: %v", err)
	}
	defer ch.Close()

	// -------------------------------------------------------------
	// THREE-TIER RETRY PATTERN TOPOLOGY DECLARATION
	// -------------------------------------------------------------

	// TIER 3 GROUP: ERROR ISOLATION (ERROR DLQ)
	err = ch.ExchangeDeclare("error.exchange", "direct", true, false, false, false, nil)
	if err != nil {
		log.Fatalf("Gagal deklarasi error.exchange: %v", err)
	}
	errorDlq, err := ch.QueueDeclare(
		"payment.error.dlq",
		true,
		false,
		false,
		false,
		amqp.Table{"x-queue-type": "quorum"},
	)
	if err != nil {
		log.Fatalf("Gagal deklarasi payment.error.dlq: %v", err)
	}
	err = ch.QueueBind(errorDlq.Name, "payment.error", "error.exchange", false, nil)
	if err != nil {
		log.Fatalf("Gagal binding error queue: %v", err)
	}

	// TIER 2 GROUP: RETRY DELAY (RETRY QUEUE WITH TTL)
	err = ch.ExchangeDeclare("retry.exchange", "direct", true, false, false, false, nil)
	if err != nil {
		log.Fatalf("Gagal deklarasi retry.exchange: %v", err)
	}
	
	// The delay queue is configured with a 15-second TTL and a DLX back to main.exchange
	delayQueueArgs := amqp.Table{
		"x-queue-type":               "quorum",
		"x-message-ttl":             int32(15000),       // Delay time: 15 seconds
		"x-dead-letter-exchange":     "main.exchange",    // Return to the main exchange after expiry
		"x-dead-letter-routing-key": "payment.execute",   // Use the main routing key when returning
	}
	delayQueue, err := ch.QueueDeclare(
		"payment.retry.15s.delay",
		true,
		false,
		false,
		false,
		delayQueueArgs,
	)
	if err != nil {
		log.Fatalf("Gagal deklarasi payment.retry.15s.delay: %v", err)
	}
	err = ch.QueueBind(delayQueue.Name, "payment.retry", "retry.exchange", false, nil)
	if err != nil {
		log.Fatalf("Gagal binding delay queue: %v", err)
	}

	// TIER 1 GROUP: MAIN (MAIN QUEUE)
	err = ch.ExchangeDeclare("main.exchange", "direct", true, false, false, false, nil)
	if err != nil {
		log.Fatalf("Gagal deklarasi main.exchange: %v", err)
	}
	mainQueue, err := ch.QueueDeclare(
		"payment.main.queue",
		true,
		false,
		false,
		false,
		amqp.Table{"x-queue-type": "quorum"},
	)
	if err != nil {
		log.Fatalf("Gagal deklarasi payment.main.queue: %v", err)
	}
	err = ch.QueueBind(mainQueue.Name, "payment.execute", "main.exchange", false, nil)
	if err != nil {
		log.Fatalf("Gagal binding main queue: %v", err)
	}

	// 3. Set the Prefetch QoS
	err = ch.Qos(10, 0, false)
	if err != nil {
		log.Fatalf("Gagal menyetel Prefetch Qos: %v", err)
	}

	log.Println("[INFO] Topologi Three-Tier Retry sukses diaktifkan. Memulai konsumen...")

	// -------------------------------------------------------------
	// MAIN CONSUMER REGISTRATION
	// -------------------------------------------------------------
	msgs, err := ch.Consume(
		mainQueue.Name,
		"payment-main-worker",
		false, // manual ACK required
		false,
		false,
		false,
		nil,
	)
	if err != nil {
		log.Fatalf("Gagal registrasi konsumen: %v", err)
	}

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	go func() {
		for d := range msgs {
			log.Printf("[RECEIVED] Memproses transaksi ID: %s", d.MessageId)

			// Process the main business data
			err := processTransaction(d.Body)
			if err != nil {
				// A message processing failure occurred

				// 1. Extract the current retry counter value from the custom header
				var retryCount int32 = 0
				if rawVal, ok := d.Headers["x-retry-count"]; ok {
					if val, assertOk := rawVal.(int32); assertOk {
						retryCount = val
					}
				}

				maxRetryLimit := int32(3) // Limit to a maximum of only 3 retries
				if retryCount >= maxRetryLimit {
					// CASE A: The retry limit is exhausted (switch to TIER 3 - Error DLQ)
					log.Printf("[POISON] Transaksi %s gagal setelah %d kali retry. Memindahkan ke Error DLQ...", d.MessageId, retryCount)
					
					errPublish := ch.PublishWithContext(ctx,
						"error.exchange", // Send to the permanent error exchange
						"payment.error",
						false,
						false,
						amqp.Publishing{
							ContentType: d.ContentType,
							MessageId:   d.MessageId,
							Headers:     d.Headers, // Keep the history headers
							Body:        d.Body,
						},
					)
					if errPublish != nil {
						log.Printf("[ERROR] Gagal mempublikasikan ke error exchange: %v", errPublish)
						d.Nack(false, true) // Fallback to the main queue if the broker errors
						continue
					}
				} else {
					// CASE B: Still under the limit (switch to TIER 2 - Retry Delay)
					nextRetryCount := retryCount + 1
					log.Printf("[RETRY] Transaksi %s gagal. Mengirimkan ke antrean delay untuk retry ke-%d...", d.MessageId, nextRetryCount)

					nextHeaders := d.Headers
					if nextHeaders == nil {
						nextHeaders = amqp.Table{}
					}
					nextHeaders["x-retry-count"] = nextRetryCount

					errPublish := ch.PublishWithContext(ctx,
						"retry.exchange", // Send to the retry exchange
						"payment.retry",
						false,
						false,
						amqp.Publishing{
							ContentType: d.ContentType,
							MessageId:   d.MessageId,
							Headers:     nextHeaders, // Write the new counter
							Body:        d.Body,
						},
					)
					if errPublish != nil {
						log.Printf("[ERROR] Gagal mempublikasikan ke retry exchange: %v", errPublish)
						d.Nack(false, true)
						continue
					}
				}

				// Send an ACK to delete the original failed message from main.queue.
				// The new message is now safely flowed to the delay queue or error DLQ.
				d.Ack(false)
				continue
			}

			// Successful processing
			log.Printf("[SUCCESS] Transaksi %s sukses terproses. Mengirimkan ACK...", d.MessageId)
			d.Ack(false)
		}
	}()

	// Wait for the OS shutdown signal
	sigChan := make(chan os.Signal, 1)
	signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
	<-sigChan

	log.Println("[INFO] Menghentikan konsumen utama secara aman...")
}

// processTransaction is a simulated database business logic function
func processTransaction(body []byte) error {
	var tx MessagePayload
	err := json.Unmarshal(body, &tx)
	if err != nil {
		return err // Returns a json unmarshal error
	}
	return nil
}

Anti-Patterns vs Practical Solutions #

To make our Three-Tier Retry Pattern architecture operate optimally, avoid the following anti-patterns:

Anti-Pattern: Combining the Delay Queue and Final Error DLQ into the Same Exchange #

Designing a topology where delayed (time-waiting) messages and final error messages are sent to the same exchange, only differentiated by thin routing keys, without physically separating their holding queues.

Why is this wrong? #

This pattern is highly vulnerable to misrouting if other developer teams accidentally register new queue bindings using loose wildcard characters (like payment.#).

Messages that should be held in the delay queue for 15 minutes can accidentally flow directly to the Error DLQ, or poison messages that should be isolated in the Error DLQ loop back into the main queue. Additionally, merging these paths makes metric monitoring harder on monitoring tools (like Prometheus), because we can’t distinguish temporary latency metrics from fatal error metrics.

Practical Solution #

Always apply strict physical separation. Use clearly separate exchanges: retry.exchange for asynchronous delay flows, and error.exchange for permanent error holding. Ensure no delay queue binding path overlaps with the final DLQ routing keys.

Summary #

• The Three-Tier Retry Concept — Combining error-handling elements into a three-tier architecture dividing message traffic into a main path (Main Queue), delay path (Retry Delay Queue), and isolation path (Error DLQ).
• Main Queue Function — Receives new normal data from producers and releases failed messages as quickly as possible through the DLX to keep throughput high.
• Delay Queue Function — Uses Queue-level TTL (x-message-ttl) as an asynchronous countdown timer and DLX to return messages to the main queue, without any active consumers connected.
• Error DLQ Function — Stores problematic messages or messages that exhausted their retry allocation for manual investigation by operations teams without expiration time limits.
• Circular Retry Mitigation — Must implement an open circuit by limiting the maximum retry count (counter checks) at the consumer level to break circular flows that can trigger broker crash loops.
• Strict Physical Separation — Declare clearly different exchanges and queues (retry.exchange vs error.exchange) to avoid misrouting from wildcard bindings and make monitoring metric tracking easier.

← Previous: Poison Message Handling   Next: Queue Comparison →