Lazy vs Classic Queue #

In large-scale message distribution systems, one of the biggest challenges for system administrators and software architects is managing accumulated message backlogs (message backlog). Backlogs can happen at any time when consumer applications break down, die during maintenance, or lose the race in processing data traffic spikes compared to producers.

Inside RabbitMQ, how the broker manages RAM memory and disk under backlog pressure is determined by the queue type and storage configuration. Historically, RabbitMQ provides two storage behavior modes for Classic Queues: Default (Memory-First) and Lazy (Disk-First). Understanding the difference between these two modes, how the Erlang BEAM VM runtime allocates heap memory, and how modern RabbitMQ architecture evolution (4.0+) overhauled these storage engines is essential for keeping the broker stable and avoiding total outages from memory exhaustion.

The Classic Storage Model (Memory-First) #

The traditional Classic Queue (run in default mode) uses a RAM-priority storage strategy (memory-first behavior). The main goal of this design is maximizing throughput and minimizing message delivery latency by avoiding disk I/O operations as much as possible.

Message Ingress Workflow #

When a producer publishes a message to a default Classic Queue:

1. RAM Storage: The message is immediately received and stored in the broker’s main RAM memory, specifically in the Erlang queue process’s heap memory.
2. Fast Delivery: If there’s an active consumer ready to receive, the message is delivered directly from RAM to the consumer without ever touching physical disk. This provides sub-millisecond latency.
3. Watermark Evaluation: If the message isn’t immediately consumed and starts piling up, RabbitMQ keeps holding messages in RAM until broker memory usage approaches the Memory High Watermark warning threshold.

The RAM-to-Disk Paging Mechanism #

When broker memory usage exceeds the watermark (by default configured at 40% of the server’s total physical RAM), RabbitMQ enters an emergency condition and activates the Paging mechanism.

flowchart LR
    Producer["Producer"] -->|"Publish"| RAM["Erlang Process RAM Heap"]
    RAM -->|"Watermark Exceeded"| Paging["Batch Paging"]
    Paging --> Disk["Disk"]
    Disk -. "Flow Control Blocked" .-> Producer

This paging process runs as follows:

1. Batch Serialization: The broker temporarily stops accepting new messages and starts collecting the oldest messages in RAM, performing binary serialization, and writing them as batches to disk (page files).
2. Garbage Collection: After messages are written to disk, the message data references in RAM are deleted, triggering the Erlang Garbage Collector (GC) process to free heap memory space.
3. Flow Control Throttling: During the paging process, the broker activates Flow Control on producer channels. Producers are temporarily blocked from sending new messages (blocking state) to give the broker time to lower RAM usage. This process triggers very disruptive latency spikes for producer application performance.

Internal Lazy Queue Logic (Disk-First) #

To overcome the broker instability from the emergency paging process above, RabbitMQ introduced the Lazy Queue configuration (also known as "lazy" mode). The Lazy Queue takes the opposite approach: physical disk priority (disk-first behavior).

Lazy Storage Workflow #

When a queue is declared as lazy (x-queue-mode: lazy):

1. Direct-to-Disk Writing: As soon as a message is received from the exchange, it is written sequentially to storage files on the physical disk.
2. Minimal RAM: The broker doesn’t hold message payload contents in the Erlang process RAM heap. RAM memory is only used to store very small message index pointers to track message order.
3. On-Demand Loading: When a consumer requests a message, RabbitMQ reads the message from disk into RAM on-demand (only when needed), delivers it to the consumer, and immediately frees the RAM again.

flowchart TD
    subgraph ClassicQueue["Classic Queue (Memory-First)"]
        direction TB
        C_In["Publish Message"] --> C_RAM{"Enough RAM?"}
        C_RAM -->|Yes| C_Mem["Store in RAM Heap (Fast)"]
        C_RAM -->|No| C_Page["Batch Paging to Disk (CPU & Flow Control Overhead)"]
    end

    subgraph LazyQueue["Lazy Queue (Disk-First)"]
        direction TB
        L_In["Publish Message"] --> L_Disk["Write Directly to Disk"]
        L_Disk --> L_RAM["Store Small Index in RAM"]
        L_RAM --> L_Cons["Load to RAM Only When Consumed"]
    end

Eliminating Erlang Garbage Collection (GC) Overhead #

In the Erlang BEAM runtime, every queue runs as an isolated lightweight process with its own private heap memory. When a default Classic Queue holds 5 million messages, that process’s heap size balloons to gigabyte scale.

• GC Problem: Every time the Erlang Garbage Collector runs to clean that process’s memory, it must linearly scan the entire large heap memory graph. This process consumes CPU cycles intensively and triggers micro-stuttering on the broker.
• Lazy Solution: Because the Lazy Queue moves messages from the heap memory to disk directly, the queue’s Erlang process heap size stays at a minimal level (just a few kilobytes). This completely removes the Garbage Collector load, keeping broker CPU utilization stable and constant even when the queue holds tens of millions of backlog messages.

Modern Evolution: Storage Engine v2 in RabbitMQ 4.0+ #

RabbitMQ’s storage architecture has undergone a major overhaul since version 3.10, reaching its peak in RabbitMQ 4.0+ with the introduction of Classic Queue v2 (CQv2).

What is CQv2? #

Previously, small messages were stored in the message index (queue_index) and large messages were stored in the message store (msg_store). CQv2 combines both components into one unified, far more efficient segment-based binary storage engine.

This new CQ version reduces the functional differences between Classic default and Lazy:

• Smart Auto-Paging: CQv2 proactively writes messages to disk much earlier before RAM memory hits the high watermark, without triggering producer blocking (flow control).
• Read-Ahead Cache: CQv2 includes an internal read cache mechanism. If consumers are actively consuming messages, messages are taken directly from the RAM cache. If a backlog occurs, the CQv2 system automatically switches behavior to resemble a Lazy Queue transparently.

Even though CQv2 in RabbitMQ 4.0+ greatly reduces the need to manually set "lazy" mode, Lazy Queue configuration remains an important option when we use hardware with limited RAM but must handle giant queue pile-ups.

Performance Trade-offs: Latency vs Memory Capacity #

Choosing between Classic default and Lazy Queues is an architectural decision demanding a balance between speed performance and capacity tolerance:

Erlang Paging Details & Physical Disk Representation #

To appreciate Lazy Queue reliability in production, we must see how the Erlang runtime manages internal memory allocation and how messages are physically stored on the server filesystem.

The Paging Process at the Erlang Runtime Level #

Inside the broker, global memory status is watched by the vm_memory_monitor memory monitor. If broker RAM usage is detected exceeding the vm_memory_high_watermark_paging_ratio threshold (by default set at 50% of the Memory High Watermark, or 20% of total physical RAM), RabbitMQ forces Classic queue processes into a memory evacuation cycle.

At the Erlang code level, the rabbit_amqqueue_process calls the backing_queue module to trigger the ram_duration_changed/2 function. This process:

1. Traversing the Heap: Scans all message binary data held in the queue process’s heap memory from oldest to newest.
2. Binary Serialization: Converts Erlang term message structures into raw binary representations.
3. Disk Append: Sends that binary data to the Erlang file port driver to be written to storage files.
4. Heap Reclamation: Forcibly calls the Garbage Collector on that queue process to free heap memory space.

This heap traversal and aggressive Garbage Collector invocation process consumes enormous CPU cycles. If the broker experiences a sustained backlog, the CPU gets stuck in continuous memory traversal cycles, dramatically reducing broker throughput.

Physical Folder Structure Representation on Disk #

Messages diverted to disk (either by Lazy Queues directly or by Classic Queues through the paging process) are physically stored under the RabbitMQ node data directory:

/var/lib/rabbitmq/mnesia/rabbit@hostname/msg_stores/vhosts/<vhost_uuid>/queues/<queue_uuid>/

Inside that queue-specific directory, we find several important files:

• .idx (Index Files): Index files recording message order, acknowledgement status, and physical payload positions on disk.
• .rdq (Raw Data Queues): Segmented raw message data binary files. Messages are written sequentially (append-only) to optimize disk write speed.

The Lazy Queue minimizes overhead by writing messages directly to these .rdq files the moment messages are received, avoiding temporary storage in the Erlang process RAM heap. However, because data is read from disk dynamically when consumed, our disk I/O speed (IOPS) and SSD seek speed become the main determinant of total message consumption latency.

Go Implementation: Declaring a Lazy Queue #

To configure a queue to run in lazy mode, we must include the special "x-queue-mode" argument with the value "lazy" when declaring the queue. Here is a complete code example using the Go language and the github.com/rabbitmq/amqp091-go library.

package main

import (
	"context"
	"log"
	"time"

	amqp "github.com/rabbitmq/amqp091-go"
)

func main() {
	// 1. Create a connection to RabbitMQ
	conn, err := amqp.Dial("amqp://guest:***@localhost:5672/")
	if err != nil {
		log.Fatalf("Gagal terhubung ke RabbitMQ: %s", err)
	}
	defer conn.Close()

	ch, err := conn.Channel()
	if err != nil {
		log.Fatalf("Gagal membuka channel: %s", err)
	}
	defer ch.Close()

	// 2. Define the Arguments for LAZY Mode
	// The "x-queue-mode" argument is set to "lazy"
	queueArgs := amqp.Table{
		"x-queue-mode": "lazy",
	}

	queueName := "large-backlog-lazy-queue"

	// 3. Declare the Queue with Lazy Arguments
	_, err = ch.QueueDeclare(
		queueName,
		true,  // durable: persistent queue
		false, // auto-delete
		false, // exclusive
		false, // no-wait
		queueArgs, // Registering the lazy table arguments!
	)
	if err != nil {
		log.Fatalf("Gagal mendeklarasikan lazy queue: %s", err)
	}
	log.Printf("✓ Lazy Queue %s berhasil dideklarasikan.", queueName)

	// 4. Publish a Message to the Lazy Queue
	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
	defer cancel()

	payload := []byte(`{"event":"background_job","data":"payload_besar_..."}`)

	err = ch.PublishWithContext(ctx,
		"", // Default Exchange
		queueName,
		false,
		false,
		amqp.Publishing{
			DeliveryMode: amqp.Persistent, // Writes the message safely to disk
			ContentType:  "application/json",
			Body:         payload,
		},
	)
	if err != nil {
		log.Fatalf("Gagal mengirim pesan: %s", err)
	}
	log.Println("✓ Pesan dikirim langsung ke media penyimpanan disk fisik.")
}

Anti-Patterns to Avoid #

When designing message storage infrastructure, avoid the following common mistakes:

1. Using Classic Default to Store Giant Backlogs #

Letting millions of messages pile up in a default Classic Queue without supervision, assuming server RAM will always be sufficient.

Why is this wrong? #

When the backlog grows past the RAM watermark, RabbitMQ activates the emergency paging process. All message publication activity stops due to Flow Control, and server CPU is consumed entirely by Erlang Garbage Collector scanning. The broker can become unresponsive (hanging) and get ejected from the cluster by other nodes.

• Solution: If a queue is designed to hold large batch data or consumers are often shut down for long periods, always declare that queue as a Lazy Queue from the start.

2. Enabling Lazy Mode on Slow Disk Storage #

Running a Lazy Queue on servers using mechanical hard disk drives (HDDs) or network storage with low IOPS capacity while demanding sub-millisecond application latency.

Why is this wrong? #

Because the Lazy Queue writes messages to disk on ingress and reads from disk on egress, message delivery speed is entirely limited by physical disk I/O speed. Using slow disks causes message consumption latency to spike drastically and limits our system throughput.

• Solution: Make sure the RabbitMQ server uses solid-state storage (local SSD or NVMe) with high IOPS when enabling lazy mode for production workloads.

Summary #

• Storage Strategy Difference — The default Classic Queue uses a memory-first strategy for maximum speed, while the Lazy Queue uses disk-first to secure broker RAM memory usage stability.
• RAM Paging Mechanism — If the Classic default exceeds the RAM high watermark, the broker is forced to do emergency paging to disk and block producer message delivery (Flow Control).
• GC Churn Free — The Lazy Queue keeps Erlang process heap sizes small because message payloads are moved to disk. This significantly removes the BEAM VM Garbage Collector load.
• RabbitMQ 4.0 CQv2 — Modern RabbitMQ versions implement Classic Queue v2, which proactively aligns RAM cache reads with early disk WAL writes automatically and efficiently.
• SSD Hardware Requirement — Always use high-IOPS SSD/NVMe storage if our system uses Lazy Queues to prevent disk I/O from becoming a latency bottleneck.
• Parameter Configuration — The Lazy Queue is enabled through the "x-queue-mode" argument with the value "lazy" during the queue declaration process by the client.

← Previous: Autodelete   Next: Quorum →