Message TTL #

In asynchronous processing architectures, not all information flowed by producers has unlimited relevance. Much business data is naturally only valuable within a certain time frame. For example, an OTP (One-Time Password) verification code should only be usable within 2 or 3 minutes after being sent, or real-time stock price data updated every few seconds becomes stale and useless if its processing is delayed by a piling-up queue. Letting these stale messages keep flowing in the main queue until processed by consumers not only wastes application computing power, but can also damage our system’s business logic consistency.

To manage the temporal relevance of these messages, RabbitMQ provides a native feature called Message TTL (Time-To-Live). TTL lets us set message validity inside the queue system precisely in milliseconds. Once a message’s lifetime runs out before being processed by a consumer, the broker automatically removes that message from the main queue. This feature is crucial both for automatic stale data cleanup and for designing delay retry architectures without requiring an external scheduler system.

Two TTL Configuration Levels #

RabbitMQ provides high flexibility by allowing us to configure expiration time limits (TTL) at two different operational levels: at the queue declaration level (Queue-level) and at the individual message delivery level (Message-level).

1. Queue-Level TTL (x-message-ttl) #

Queue-level TTL is applied uniformly to all messages entering a specific queue. This configuration is defined when we first declare the queue using the x-message-ttl argument with a positive integer value in milliseconds.

// Setting a Queue-level TTL of 60 seconds (60000 ms)
args := amqp.Table{
    "x-message-ttl": int32(60000), 
}

When this configuration is active, the RabbitMQ broker acts as an automatic timer for that queue. Every message landing in the queue has exactly the same lifetime (in the example above, 60 seconds). Once a message stays in the queue beyond that time limit, the broker immediately declares it expired. This pattern is ideal for queues handling homogeneous message types, like mobile push notification queues or temporary log cleanup queues.

2. Message-Level TTL (expiration) #

If we need more dynamic control, where each message has a different expiration depending on its transaction context, we can use Message-level TTL. This configuration isn’t set on the queue, but inserted by the producer directly into the message metadata properties when publishing. This property is named expiration and is written as a numeric string in milliseconds.

// Setting a dynamic Message-level TTL of 15 seconds (15000 ms)
err := ch.PublishWithContext(ctx,
    "main.exchange",
    "payment.verify",
    false,
    false,
    amqp.Publishing{
        ContentType: "application/json",
        Expiration:  "15000", // Milliseconds string
        Body:        payload,
    },
)

Using this method, the same queue can hold messages with varying expiration times—for example, the first message expires in 10 seconds, while the second message behind it only expires after 5 minutes.

Conflict Resolution Rules #

What if we declare a queue with an x-message-ttl of 30 seconds, but the producer sends an individual message to that queue with an expiration property of 10 seconds or 50 seconds?

RabbitMQ handles this conflict with a very fair rule: The smallest TTL value wins.

  • If the individual message property sets a lifetime of 10 seconds (smaller than the 30-second queue limit), that message expires in 10 seconds.
  • If the individual message property sets a lifetime of 50 seconds (larger than the 30-second queue limit), the broker rejects that value and limits the message lifetime to a maximum of 30 seconds per the queue limit.

This rule ensures our main queue still has a hard upper expiration limit to protect broker memory consumption, without disabling producer flexibility to set stricter expiration limits.

How the Broker Evaluates TTL (The Head-of-Line Blocking Danger) #

Although RabbitMQ supports Queue-level TTL and Message-level TTL simultaneously, there’s a fundamental difference in how the broker engine evaluates and cleans expired messages for both methods. Understanding this internal working difference is crucial to prevent memory failures on our broker.

Active Evaluation on Queue-Level TTL #

When we set TTL at the queue level (x-message-ttl), the RabbitMQ broker internally uses an efficient sequential data structure and actively monitors message expiration times. Because all messages have uniform expiration durations, the earliest incoming message certainly expires earliest too.

The broker can easily track the queue head pointer and instantly delete or divert expired messages, without having to scan the queue contents. Memory cleanup runs very fast and efficiently.

Passive Evaluation on Message-Level TTL (Head-of-Line Blocking) #

This behavior is very different when we use Message-level TTL (expiration). Because each message’s lifetime differs, the RabbitMQ broker can’t easily predict which message will expire first. Scanning an entire queue of millions of messages every millisecond to constantly search for expired messages would consume enormously large broker CPU resources.

To avoid that performance degradation, RabbitMQ chooses a shortcut: The broker only evaluates individual message expiration when the message has reached the front of the queue (head of the queue) and is ready to be delivered to a consumer.

This passive approach gives birth to a classic distributed systems problem called Head-of-Line (HoL) Blocking at the message level. Let’s study the following scenario:

flowchart LR
    Tail["Tail (Back)"] --> MessageC["Message C (TTL: 5 Seconds)"]
    MessageC --> MessageB["Message B (TTL: 10 Seconds)"]
    MessageB --> MessageA["Message A (TTL: 10 Minutes)"]
    MessageA --> Head["Head (Front)"]
  1. A producer sends Message A with a 10-minute TTL, followed by Message B with a 10-second TTL, and Message C with a 5-second TTL.
  2. All messages enter the main queue, which currently has no active consumers.
  3. After 5 seconds pass, theoretically Message C has expired. After 10 seconds, Message B also expires.
  4. However, because Message A (which has a 10-minute TTL) is at the front of the queue (head), the RabbitMQ broker doesn’t check the expiration status of Message B and Message C.
  5. As a result, the expired Message B and Message C stay stored in broker memory and disk, occupying storage capacity, until 10 minutes pass or until Message A is consumed by the application.

The latent danger of this HoL Blocking is unexpected stale message pile-ups in broker memory, which can trigger disk space limit alarms if producers randomly send very long TTL messages in front of millions of short TTL messages.

The Delay Queue Pattern (Retry Delay) Using TTL and DLX #

One of the most popular and elegant architecture design patterns in RabbitMQ is leveraging the TTL expiration effect combined with a Dead Letter Exchange (DLX) to build a Delay Queue / Retry Delay mechanism. This pattern lets us postpone reprocessing failed messages without holding consumer threads with time.Sleep and without needing an external database scheduler.

Here is the Delay Queue pattern workflow diagram:

flowchart TD
    MQ[Main Queue: main.queue] -->|1. Process Fails| C{Consumer}
    C -->|2. nack with requeue=false| DLX_R(Retry Exchange: retry.exchange)
    DLX_R -->|3. Send to| DQ[Delay Queue: retry.delay.queue]
    
    subgraph Queue Without Consumers
        DQ
    end
    
    DQ -->|4. TTL Expired| DLX_M(Main Exchange: main.exchange)
    DLX_M -->|5. Re-enters| MQ
  1. Step 1: A consumer reads a message from the Main Queue (main.queue).
  2. Step 2: A temporary processing failure occurs (e.g., database timeout). The consumer rejects that message by sending basic.nack(requeue=false).
  3. Step 3: Because main.queue is configured with a DLX pointing to retry.exchange, the RabbitMQ broker automatically diverts the failed message to the Retry Exchange. From that exchange, the message enters the Delay Queue (retry.delay.queue).
  4. Step 4: retry.delay.queue is specially designed without any active consumers listening to it. This queue is configured with two vital parameters:
    • x-message-ttl set to the wait time value we want (e.g., 10000 ms or 10 seconds).
    • x-dead-letter-exchange set to point back to the Main Exchange (main.exchange).
  5. Step 5: The message is held in retry.delay.queue for 10 seconds. Once the 10 seconds run out, the broker considers the message expired.
  6. Step 6: Because the delay queue has a DLX configuration pointing to the Main Exchange, the RabbitMQ broker automatically sends the expired message back to the Main Exchange, which then routes it back to the Main Queue (main.queue) to be reprocessed by the consumer.

This pattern runs entirely asynchronously and distributed on the RabbitMQ broker side. Our consumers are free to process other healthy messages in the main queue without being disturbed by the failed message waiting process.

Real-World Message TTL Use Cases #

Setting message lifetime limits is very important for the following industry scenarios:

  • One-Time Passwords (OTP) & Verification Codes: SMS or Email OTPs have very strict expiration times (usually 2-5 minutes). If our SMS gateway was down for 1 hour, hoarding millions of OTPs and sending them after the gateway recovers is pointless because those codes are certainly no longer valid for users.
  • High-Frequency IoT Telemetry Sensors: Machine temperature sensors send data every 2 seconds. If network obstructions occur, temperature data sent 1 minute ago is no longer relevant because we need the most current temperature readings. We can set a 6-second message TTL so stale data is automatically discarded.
  • Real-time Pricing & Market Feed: Stock price or foreign exchange rate providers update data constantly. Price messages stuck in a queue for more than 10 seconds must be considered expired so user applications don’t display misleading stale prices.
  • Idempotent Cache Invalidation: Messages to clear local application memory caches after database updates. If update events run very fast, we only need to process the latest invalidation event and discard stale old events.

TTL Behavior on Quorum Queues #

As a modern queue type oriented toward Raft consensus replication, the Quorum Queue fully supports the Message TTL feature with very strict consistency guarantees.

On Classic Queues, expired message cleanup is performed independently by the node where the queue is active. If a network split occurs, the isolated node may have a different time perception and discard messages unilaterally.

However, on Quorum Queues, the message expiration evaluation and execution process is managed with distributed discipline:

  1. Only the Quorum Queue Leader node is authorized to evaluate whether a message’s lifetime has run out based on its local system time.
  2. When the Leader detects an expired message, it doesn’t immediately delete the message from its local memory.
  3. The Leader writes the dead message deletion command to the Raft log.
  4. That log is replicated to Follower nodes.
  5. Once quorum approves, the message is consistently deleted across all cluster replicas, or safely diverted to the DLX.

This mechanism ensures message expiration status stays synchronized and consistent across the entire cluster, preventing duplicate stale message deliveries from sudden leader node crash failovers.

Go Code Implementation (Golang) #

Here is a complete Go program example declaring a Delay Queue architecture using Queue-level TTL and Dead Letter Exchange (DLX) configurations with the amqp091-go library.

package main

import (
	"context"
	"log"
	"os"
	"os/signal"
	"syscall"
	"time"

	amqp "github.com/rabbitmq/amqp091-go"
)

func main() {
	// 1. Open the physical connection to the RabbitMQ broker
	conn, err := amqp.Dial("amqp://guest:***@localhost:5672/")
	if err != nil {
		log.Fatalf("Gagal membuka koneksi ke RabbitMQ: %v", err)
	}
	defer conn.Close()

	// 2. Create an AMQP channel
	ch, err := conn.Channel()
	if err != nil {
		log.Fatalf("Gagal membuka channel AMQP: %v", err)
	}
	defer ch.Close()

	// -------------------------------------------------------------
	// RETRY DELAY QUEUE TOPOLOGY DECLARATION (WITH TTL)
	// -------------------------------------------------------------

	// A. Declare the Retry Exchange
	err = ch.ExchangeDeclare(
		"retry.exchange",
		"direct",
		true,
		false,
		false,
		false,
		nil,
	)
	if err != nil {
		log.Fatalf("Gagal mendeklarasikan retry.exchange: %v", err)
	}

	// B. Declare the Delay Queue with a 10-Second TTL
	// This queue functions to hold messages, without any connected consumers.
	// Once the 10 seconds run out, dead messages are sent back to the main exchange.
	delayQueueArgs := amqp.Table{
		"x-queue-type":               "quorum",
		"x-message-ttl":             int32(10000),       // Retry delay period: 10,000 ms
		"x-dead-letter-exchange":     "main.exchange",    // Return to the main exchange after expiry
		"x-dead-letter-routing-key": "payment.execute",   // Use this routing key when returning
	}
	delayQueue, err := ch.QueueDeclare(
		"retry.delay.10s.queue", // delay queue name
		true,                    // durable
		false,                   // auto-delete
		false,                   // exclusive
		false,                   // no-wait
		delayQueueArgs,
	)
	if err != nil {
		log.Fatalf("Gagal mendeklarasikan retry.delay.queue: %v", err)
	}

	// Bind the delay queue to the retry exchange
	err = ch.QueueBind(
		delayQueue.Name,
		"payment.retry",
		"retry.exchange",
		false,
		nil,
	)
	if err != nil {
		log.Fatalf("Gagal binding delay queue: %v", err)
	}

	// -------------------------------------------------------------
	// MAIN QUEUE TOPOLOGY DECLARATION
	// -------------------------------------------------------------

	// C. Declare the Main Exchange
	err = ch.ExchangeDeclare(
		"main.exchange",
		"direct",
		true,
		false,
		false,
		false,
		nil,
	)
	if err != nil {
		log.Fatalf("Gagal mendeklarasikan main.exchange: %v", err)
	}

	// D. Declare the Main Queue
	// If a message fails on this queue, it is diverted to the retry exchange
	mainQueueArgs := amqp.Table{
		"x-queue-type":                 "quorum",
		"x-dead-letter-exchange":     "retry.exchange", // Send to the retry exchange if it fails
		"x-dead-letter-routing-key": "payment.retry",    // Use the failure routing key
	}
	mainQueue, err := ch.QueueDeclare(
		"main.queue",
		true,
		false,
		false,
		false,
		mainQueueArgs,
	)
	if err != nil {
		log.Fatalf("Gagal mendeklarasikan main.queue: %v", err)
	}

	err = ch.QueueBind(
		mainQueue.Name,
		"payment.execute",
		"main.exchange",
		false,
		nil,
	)
	if err != nil {
		log.Fatalf("Gagal binding main queue: %v", err)
	}

	log.Println("[INFO] Topologi Delay Queue sukses diinisialisasi. Menjalankan konsumen...")

	// -------------------------------------------------------------
	// SIMULATED REQUEUE CONTROL CONSUMER WITH DELAY
	// -------------------------------------------------------------

	msgs, err := ch.Consume(
		mainQueue.Name,
		"payment-worker",
		false, // manual ACK required
		false,
		false,
		false,
		nil,
	)
	if err != nil {
		log.Fatalf("Gagal memulai konsumen: %v", err)
	}

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	go func() {
		for d := range msgs {
			log.Printf("[RECEIVED] Memproses transaksi: %s", d.MessageId)

			// Simulate a temporary database connection failure (transient error)
			// We reject the message with requeue = false so it is diverted to the DLX (retry.exchange)
			log.Printf("[WARN] Terjadi gangguan transient database. Mengirimkan pesan ke delay queue...")
			err := d.Nack(
				false, // multiple
				false, // requeue = false (the message enters retry.exchange -> retry.delay.10s.queue)
			)
			if err != nil {
				log.Printf("Gagal mengirimkan NACK: %v", err)
			}
		}
	}()

	// Wait for the OS interrupt signal
	sigChan := make(chan os.Signal, 1)
	signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
	<-sigChan

	log.Println("[INFO] Mematikan service secara aman...")
}

Anti-Patterns vs Practical Solutions #

Incorrect TTL usage can cause severe broker performance degradation. Here are several common mistakes (anti-patterns) that must be avoided:

Anti-Pattern 1: Using Dynamic Message-Level TTL for Retry Delay Patterns on a Single Queue #

Producers send failed messages with different dynamic Expiration properties (e.g., message A given 10 minutes, message B given 10 seconds) into the same delay queue to create dynamic retry delays.

Why is this wrong? #

Because RabbitMQ only evaluates message-level expiration when the message reaches the queue head (Head-of-Line Blocking), message B with a 10-second TTL behind message A with a 10-minute TTL will never expire on time. Message B is forced to wait 10 minutes following the lifetime of message A in front of it. This makes retry delay times chaotic, inconsistent, and triggers RAM memory pile-ups on the broker.

Practical Solution #

  1. Multiple Dedicated Delay Queues: Declare several delay queues with different static TTLs using Queue-level TTL, e.g., retry.delay.5s.queue, retry.delay.1m.queue, and retry.delay.10m.queue. Direct failed messages to the delay queue matching their retry level. Because Queue-level TTL is used, expiration runs actively and is free from HoL blocking problems.
  2. Delayed Message Plugin: Install the official RabbitMQ x-delayed-message plugin. This plugin lets us set dynamic per-message delay times safely using the broker’s internal Mnesia database to hold messages on the exchange side, avoiding physical queues before the time arrives.

Anti-Pattern 2: Setting TTL Without Configuring a Dead Letter Exchange #

Enabling the TTL feature on main queues or individual messages to limit stale backlogs, but letting expired messages be permanently deleted without being bound to a DLX.

Why is this wrong? #

This pattern is vulnerable to important transaction data loss without a trace (silent data loss). If severe backlog pile-ups occur from consumers being down for several hours, millions of valid transaction messages expire and are deleted forever by the broker. We’ll never know which transactions were discarded and have no way to perform data reconciliation.

Practical Solution #

Always combine TTL usage with a Dead Letter Exchange (DLX). Expired messages must be diverted to a dedicated Dead Letter Queue (DLQ) so operations teams can monitor message expiration rates and manually audit expired transactions.

Summary #

  • Understanding Message TTL — Time-To-Live determines the maximum message lifetime limit inside a queue in milliseconds before being considered expired by the broker.
  • Queue-Level vs Message-Level — Queue-Level TTL (x-message-ttl) limits all messages uniformly and is actively evaluated. Message-Level TTL (expiration) is dynamically set per message by the producer and passively evaluated.
  • The Head-of-Line Blocking Problem — Passive evaluation on Message-Level TTL causes expired messages in the back rows to stay held in memory and unable to exit while the queue head message hasn’t expired or been consumed.
  • Delay Queue Mechanism — Retry delay patterns can be built without an external scheduler by flowing failed messages into a consumer-less queue configured with TTL (as a timer) and DLX (as the route back to the main queue).
  • Quorum Queue Consistency — Expiration evaluation on Quorum Queues is consistently managed by the Leader node through Raft consensus, avoiding status desynchronization between cluster replicas.
  • Best Design Solutions — Avoid setting dynamic TTLs on a single queue; use a multiple delay queue pattern with uniform static TTLs or install the x-delayed-message plugin.

← Previous: Dead Letter Exchange (DLX)   Next: Exponential Backoff →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact