Queueing #

In a message’s journey through the RabbitMQ broker (message lifecycle), after the message is published by the producer (Publishing) and successfully routed by the Exchange (Routing & Binding), the message enters a phase often considered passive but actually crucial for system stability and performance: the Queueing phase.

The Queueing phase is the stage where messages are safely stored in the destination queue, in a ready-to-consume status (Ready state), waiting to be delivered to consumer applications (delivery). It is in this phase that all system workload (backlog) accumulates, RAM memory and disk storage management are tested extremely, and cluster consensus coordination runs. Understanding internal queueing dynamics isn’t just knowing how ordinary FIFO queues work; it’s understanding how RabbitMQ manages Erlang memory, splits storage states, sets expiration limits, and applies flow control protections when the system is under high throughput pressure.

The Position of Queueing in the Message Lifecycle #

The Queueing phase occurs right after the broker decides the destination queue at the routing stage, and (if the message is marked persistent) after the message body is successfully secured to physical storage.

flowchart TD
    Routing["Routing Stage Complete"] --> Enqueue{"Enqueue Process"}
    
    subgraph StorageEngine["Storage Engine (Queue)"]
        Enqueue --> Ready["Status: Ready"]
        Ready --> MemoryStore["RAM (q1, q2, q3, q4)"]
        Ready --> DiskStore["Disk (Spill / Lazy)"]
    end
    
    Ready --> TTLCheck{"TTL / Expiration Evaluation"}
    TTLCheck -->|Expired| DLX["Discard or Route to DLX"]
    TTLCheck -->|Valid| Delivery["Delivered to Consumer (basic.deliver)"]
    
    Delivery --> Unacked["Status: Unacknowledged"]

When a message is in the Queueing phase, its status is explicitly marked as Ready. The message stays in this condition until the broker selects it for delivery to a qualifying consumer, which then changes the message status to Unacknowledged.

Internal Queue Storage Anatomy: Classic Queue v1 vs v2 #

To manage message queues flexibly yet efficiently, the RabbitMQ Classic Queue splits its storage system into several memory and disk layers. In Classic Queue version 1 (CQv1), this is managed by an Erlang component named backing_queue (specifically the rabbit_variable_queue module).

1. The CQv1 Queue Structure: Layers q1 to q4 #

The rabbit_variable_queue module splits the queue in RAM memory and disk into 5 internal data structures shaped like small queues to maintain FIFO order while minimizing I/O access latency:

  • q1: A RAM memory queue holding messages that just entered the queue.
  • q2: A RAM memory queue containing messages that were written to disk but whose copies are still kept in RAM. This layer acts as a buffer before messages are fully paged out of RAM.
  • delta: A message queue that is disk-only. Message payloads in delta no longer consume any RAM space at all, leaving only reference index data.
  • q3: A RAM memory queue containing messages read back from disk (delta) before being delivered to consumers.
  • q4: A RAM memory queue containing messages ready to be delivered directly to consumers from RAM.
flowchart LR
    A["Incoming Message"] --> B["q1 (RAM)"] --> C["q2 (RAM)"] --> D["delta (DISK ONLY)"] --> E["q3 (RAM)"] --> F["q4 (RAM)"] --> G["Consumer"]

The broker moves messages between these layers dynamically based on RAM memory load. If RAM is abundant, messages flow quickly from q1 straight to q4. However, if RAM starts filling up, messages are moved (paged out) from q2 into the delta disk file, and then read back (paged in) into q3 when consumers are ready to process them.

2. Improvements in Classic Queue v2 (CQv2) #

Since RabbitMQ version 3.10, the Classic Queue v2 (CQv2) architecture was introduced. CQv2 simplifies the CQv1 five-layer mechanism by implementing a new unified log storage engine.

In CQv2, messages are no longer moved gradually through complicated RAM queues. Instead, messages are written directly to an append-only log file on disk in a unified manner. CQv2 dramatically reduces the baseline memory footprint per queue and delivers far more stable throughput performance, avoiding the latency spikes that often occur on CQv1 when massive RAM-to-disk transition processes happen.

Storage Comparison: Classic, Lazy, and Quorum Queues #

Each RabbitMQ queue type has a different way of managing this Queueing phase. The queue type choice largely determines how data is stored and how cluster consensus is achieved.

1. Classic Queue (Default) #

The Classic Queue is designed with the RAM-first philosophy. The broker tries to hold messages in RAM memory as much as possible to chase the lowest latency performance.

  • Characteristics: Transient messages are only stored in RAM (except under memory pressure). Persistent messages are written to disk asynchronously while keeping copies in RAM for fast delivery readiness.
  • Weakness: If a large message pile-up (backlog) occurs, RAM consumption balloons, forcing the broker to activate intensive disk paging processes and drastically degrading performance.

2. Lazy Queue #

The Lazy Queue is designed with the Disk-first philosophy. This queue proactively moves messages directly to disk as soon as messages are received by the broker.

  • Characteristics: Messages are only loaded into RAM memory when truly needed by consumers (when delivered). RAM usage for Lazy Queues is very small and stable, regardless of how long the queue backlog is.
  • Best Scenario: Ideal for queues with large message backlogs (e.g., millions of messages), or for batch processing systems that collect messages over a long period before processing them all at once.

3. Quorum Queue #

The Quorum Queue is a distributed queue type based on Raft Consensus that prioritizes extreme data safety above everything.

  • Characteristics: Every message entering a Quorum Queue is written to a local disk Raft log file first, then replicated to follower nodes in the cluster.
  • Enqueue Mechanism: The enqueue process is only considered complete and valid (committed) after a majority of cluster replica nodes (quorum) successfully write that log to their respective physical disks. Message ordering is strictly guaranteed by the global Raft log index.

Queue State Dynamics: Ready vs Unacknowledged #

During the Queueing phase, RabbitMQ distinguishes message status inside queues into two main categories that are very important to monitor through dashboards or Prometheus metrics.

1. Ready State #

A message in the Ready status is a message currently in the storage queue, not yet delivered to any consumer, and ready to be routed as soon as a consumer connects with receive capacity (prefetch quota).

  • Storage: Ready messages can be in RAM (q1, q2, q4) or on disk (delta / Lazy storage).
  • Indicator: A continuously increasing Ready message count is the main indicator of system backlog or consumer application failures.

2. Unacknowledged (Unacked) State #

A message changes status to Unacknowledged when the broker has delivered it to a consumer through the AMQP protocol (basic.deliver), but the broker is still waiting for the confirmation signal back (basic.ack) from that consumer.

  • Storage: The payload of Unacked messages is still retained by the broker in RAM memory or disk. The broker must not delete these messages from physical storage because if the consumer connection drops before sending the ACK, the broker must return those messages to the Ready status (requeue) for redelivery.
  • Limit: The maximum number of Unacked messages a consumer can hold per channel is set by the QoS Prefetch Limit setting.

Message Ordering Guarantees and Their Limits #

By default, RabbitMQ guarantees that incoming message order exiting a queue follows the strict FIFO (First-In, First-Out) principle. If a producer publishes message A then message B to the same queue, consumers receive message A before message B.

However, in real distributed systems in production, there are several scenarios that can break or change this FIFO ordering guarantee:

1. Message Requeueing #

If a consumer rejects a message (using basic.nack or basic.reject) with the requeue = true parameter, the message is returned to the queue.

  • Impact: RabbitMQ tries to place the rejected message back at the front of the queue (head of the queue). If new messages enter at that time, the requeued message is reprocessed first. However, if there are several parallel consumers, logical processing order in the application can become irregular because other consumers may be processing messages that arrived later.

2. Using Priority Queues (x-max-priority) #

When a queue is configured as a Priority Queue, the broker no longer uses pure FIFO queues.

  • Impact: The broker checks the priority property (priority in the AMQP header) of every incoming message. Messages with higher priority are inserted in front of lower-priority messages, jumping the existing queue.

3. Using Multiple Consumers #

Even though the broker sends messages over the network in strict FIFO order, if there is more than one consumer instance reading from the same queue in parallel:

  • Impact: Processing speed on each consumer instance differs (e.g., due to network latency or CPU load). A consumer receiving message B may finish processing faster than the consumer processing message A, so at the final database level, execution order appears reversed.

Message Lifetime Mechanisms: TTL (Time-To-Live) #

RabbitMQ lets us limit a message’s lifetime inside a queue using the TTL mechanism. The TTL timer starts actively counting exactly when the message enters the Queueing phase.

There are two main ways to apply TTL in RabbitMQ:

1. Per-Queue TTL (x-message-ttl) #

The expiration limit is applied globally at the queue configuration level. All messages entering that queue have a uniform lifetime.

  • Internal Optimization: Because all messages have the same TTL, the message expiration order always aligns with the queue’s FIFO order. The broker can very efficiently delete expired messages by only checking the message at the front of the queue (head of the queue). Once the front message expires, the broker deletes it, then checks the next message.

2. Per-Message TTL (The expiration Property) #

The expiration limit is determined individually by the producer for each message when published.

  • Performance Constraint: Because messages in the middle of the queue can have shorter TTLs than messages in front of them, the Classic Queue does not continuously scan the entire queue for expired messages (because that memory scanning operation is too expensive). The broker only detects and discards expired messages when those messages naturally flow to the head of the queue and are about to be delivered to consumers.
  • Impact: Already-expired messages in the middle of the queue still consume RAM/disk capacity until the messages in front of them are consumed.

Max Length Limits and Overflow Policies #

To prevent queues from growing without limits, which can cause disk or RAM memory capacity exhaustion, we can configure physical limits on queues using the arguments:

  • x-max-length: Limits the maximum number of messages allowed in the queue.
  • x-max-length-bytes: Limits the total accumulated payload size (in bytes) of all messages in the queue.

When these limits are exceeded during the enqueue process, the broker executes the Overflow Policy we specify:

flowchart TD
    A["Queue Full"] -->|"Evaluate x-overflow argument"| B{"x-overflow?"}
    B -->|drop-head| C["Discard the oldest message at the queue head"]
    B -->|reject-publish| D["Send NACK to the producer and discard the new message"]
    B -->|reject-publish-dlx| E["Send the new message to the DLX and send NACK to the producer"]

1. drop-head (Default) #

The broker instantly deletes the oldest message at the head of the queue to make room for the new message entering at the queue tail. If configured with a Dead Letter Exchange (DLX), the discarded oldest message is sent to the DLX.

2. reject-publish #

The broker rejects new messages trying to enter the full queue. The broker discards that new message and sends a NACK (Negative Confirmation) signal back to the producer through the Publisher Confirms mechanism. This tells the producer the message failed to be queued due to full capacity.

3. reject-publish-dlx #

Similar to reject-publish, but the new message rejected by the full queue is immediately diverted to the configured Dead Letter Exchange (DLX), while the producer still receives the rejection confirmation.

Backlog and Backpressure Triggers (Flow Control) #

The Queueing phase is the main barometer of RabbitMQ system health. When producer message publication speed far exceeds consumer message consumption speed, a backlog accumulates.

A large message backlog triggers a series of internal protections inside the broker:

1. Intensive Erlang Garbage Collection (GC) #

Every queue runs as one Erlang process. When process memory balloons from holding thousands of messages, the Erlang VM periodically runs Garbage Collection cycles to free RAM. This GC process consumes enormous CPU resources and can temporarily block queue data read execution.

2. Credit Flow Control #

To prevent the broker from running completely out of memory during severe backlogs, RabbitMQ uses the Credit Flow Control algorithm:

  • The queue process (rabbit_amqqueue_process) limits granting “credit” to channel processes (rabbit_channel).
  • Channel processes that run out of credit stop reading data from the TCP connection reader process (rabbit_reader).
  • As a result, the producer connection’s TCP socket stops reading data (TCP window size shrinks to zero). Producers feel their connection is blocked or delayed (blocked connection).

This backpressure mechanism ensures the broker never receives data beyond its ability to secure that data to disk or RAM.

Go Code Implementation: Configuring Complete Queue Attributes #

Here is a complete Go language implementation to declare a Classic Queue by configuring TTL limits, maximum length limits, the reject-publish overflow policy, and setting the queue to Lazy mode.

package main

import (
	"context"
	"log"
	"time"

	amqp "github.com/rabbitmq/amqp091-go"
)

func main() {
	// 1. Open a Connection to the Broker
	conn, err := amqp.Dial("amqp://guest:***@localhost:5672/")
	if err != nil {
		log.Fatalf("Gagal terhubung ke RabbitMQ: %s", err)
	}
	defer conn.Close()

	ch, err := conn.Channel()
	if err != nil {
		log.Fatalf("Gagal membuka channel: %s", err)
	}
	defer ch.Close()

	// 2. Enable Publisher Confirms
	// Must be enabled because we use the 'reject-publish' overflow policy
	err = ch.Confirm(false)
	if err != nil {
		log.Fatalf("Gagal mengaktifkan Publisher Confirms: %s", err)
	}
	confirmChan := ch.NotifyPublish(make(chan amqp.Confirmation, 1))

	// 3. Define the Queue Configuration Arguments (Queueing Phase)
	args := amqp.Table{
		// Set the message TTL in the queue to 30 seconds (30000 ms)
		"x-message-ttl": int32(30000),
		
		// Limit the maximum queue length to 1000 messages
		"x-max-length": int32(1000),
		
		// Define the overflow policy if the queue is full (reject new messages)
		"x-overflow": "reject-publish",
		
		// Configure the queue to operate in Lazy mode (disk-first)
		"x-queue-mode": "lazy",
	}

	queueName := "highly-controlled-queue"

	// 4. Declare the Queue with Custom Arguments
	_, err = ch.QueueDeclare(
		queueName,
		true,  // durable
		false, // auto-delete
		false, // exclusive
		false, // no-wait
		args,  // Injecting our queueing configuration!
	)
	if err != nil {
		log.Fatalf("Gagal mendeklarasikan antrean: %s", err)
	}

	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
	defer cancel()

	// 5. Send a Test Message
	payload := []byte(`{"event":"user_signup","timestamp":1781290382}`)
	err = ch.PublishWithContext(ctx,
		"", // Default Exchange
		queueName,
		false, // mandatory
		false, // immediate
		amqp.Publishing{
			DeliveryMode: amqp.Persistent,
			ContentType:  "application/json",
			Body:         payload,
		},
	)
	if err != nil {
		log.Fatalf("Gagal mempublikasikan pesan: %s", err)
	}

	// 6. Wait for Confirmation from the Broker
	confirm := <-confirmChan
	if confirm.Ack {
		log.Println("✓ Pesan sukses diterima dan masuk ke antrean!")
	} else {
		// If the queue is full and x-overflow is set to reject-publish, the broker sends NACK
		log.Println("✗ Pesan ditolak oleh antrean (NACK) — kemungkinan batas max-length terlampaui!")
	}
}

Anti-Patterns vs Practical Solutions in Production #

Avoid the following queueing phase design mistakes that often damage cluster performance:

1. Using Random Per-Message TTL on Long Classic Queues #

Sending messages with very randomly varied expiration property values into one Classic Queue with a backlog of millions of messages.

Why is this wrong? #

Because the RabbitMQ Classic Queue only discards expired messages when those messages reach the head of the queue, a message with a 1-second TTL sitting behind a message with a 1-hour TTL will never be deleted before the 1-hour message is consumed or expires. This causes binary garbage pile-ups in RAM and disk, and triggers hidden memory leaks.

  • Solution: If you need dynamic, random message lifetime handling, use Quorum queues combined with a Dead Letter Exchange (DLX), or create dedicated queues per TTL category (e.g., a dedicated 10-second TTL queue, a dedicated 5-minute TTL queue) to maintain uniform deletion times.

2. Misusing Priority Queues for All Data Flows #

Enabling the x-max-priority argument with very high priority values (e.g., x-max-priority: 255) on all queues in the broker system.

Why is this wrong? #

Every priority level forces the queue process to maintain additional memory index structures in Erlang. Using priority levels above 10 consumes enormous RAM and CPU memory for reordering every time a new message enters. This drops throughput by more than 60%.

  • Solution: Limit the maximum queue priority value between 1 and 10 only (e.g., x-max-priority: 5). This is already more than enough to distinguish critical VIP messages from ordinary telemetry messages without sacrificing Erlang VM stability.

Summary #

  • Classic backing_queue Layers — Classic Queue v1 flows messages through 5 internal layers (q1 to q4 and the disk delta) to balance RAM consumption and disk write performance. Classic Queue v2 simplifies this with a single unified log engine.
  • Ready vs Unacknowledged — Ready messages wait to be delivered to consumers. Unacked messages are those already delivered to consumers but not yet receiving ACK confirmation back.
  • FIFO Integrity & Requeue — RabbitMQ guarantees FIFO order by default, but this order can be disrupted by requeued message rejections, priority usage, or parallel consumption by many consumers.
  • Per-Queue vs Per-Message TTL — Per-Queue TTL is processed instantly at the queue head, while Per-Message TTL is only evaluated when the message in question naturally reaches the queue head.
  • Overflow Policies — The x-overflow argument determines broker actions when the queue is full: discarding the oldest message (drop-head) or rejecting new incoming messages (reject-publish).
  • Credit Flow Backpressure — Severe backlogs trigger internal flow control mechanisms that suspend producer TCP socket reads to secure the broker from RAM exhaustion.

← Previous: Persistent   Next: Delivery →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact