Persistent #

In a message’s journey (message lifecycle) inside the RabbitMQ broker, after the message successfully passes the Routing stage and enters the destination queue, the broker must make a crucial decision about the message’s survival: the Persistence phase.

The Persistence phase determines whether the message only lives temporarily in volatile RAM memory or must be immortalized into non-volatile physical disk storage to survive power outages, system crashes, or broker restarts. However, message persistence in RabbitMQ is not just writing a binary file to the hard drive. Behind the scenes, there are message size limit optimizations, distributed Write-Ahead Log (WAL) writing, and disk segment recycling processes (segment compaction). This article thoroughly discusses RabbitMQ’s internal message storage mechanisms, optimization differences for small and large messages, the Quorum Queue binary storage lifecycle, and Go code implementations to guarantee our transaction data safety.

The Position of Persistence in the Message Lifecycle #

The Persistence phase occurs right after the routing process (rabbit_router:route/2) finishes determining the destination queue, but before the message is declared ready for consumption (Ready state) or delivered to consumer applications.

flowchart TD
    Msg["Incoming Message (DeliveryMode = 2)"] --> Route{"Route Found"}
    Route --> SizeCheck{"Message Size Evaluation"}
    
    subgraph InlineStore["Index Storage (Inline)"]
        SizeCheck -->|< 4096 bytes| Inline["Write directly to Queue Index (.idx)"]
    end
    
    subgraph StoreMsgStore["msg_store Storage (Split)"]
        SizeCheck -->|> 4096 bytes| WriteIndex["Write reference to Queue Index (.idx)"]
        SizeCheck -->|> 4096 bytes| WriteStore["Write payload to msg_store_persistent (.rdq)"]
    end

    Inline --> Commit{"ACK Publisher Confirm"}
    WriteIndex --> Commit
    WriteStore --> Commit

When a persistent message (delivery_mode = 2) enters a Durable Queue:

1. Size Evaluation: The broker checks the message body size.
2. Inline Storage (Small Messages): If the message is small, the broker optimizes by writing the message directly to the queue index.
3. Separate Storage (Large Messages): If the message is large, the broker writes the message to the global msg_store repository and only puts its reference address in the queue index file.
4. Confirm Execution: Once the physical data is successfully fsynced to disk, the broker sends confirmation to the producer.

Erlang Process Architecture and Behind-the-Scenes Filesystem Structure #

To understand how persistence works at the operating system level, we must see how RabbitMQ manages Erlang actors and their data representation in the filesystem.

Inside RabbitMQ’s internal architecture, message storage is managed by two main Erlang processes running as singleton workers per virtual host (vhost):

• msg_store_persistent: Responsible for storing all persistent messages entering that vhost.
• msg_store_transient: Responsible for storing non-persistent (transient) messages forced to disk because the broker ran out of RAM memory (the paging out process).

Folder Structure on Physical Disk #

Persistence data is stored under the RabbitMQ node’s Mnesia data directory. By default, the path follows this pattern:

/var/lib/rabbitmq/mnesia/rabbit@<hostname>/msg_stores/vhosts/<vhost_hash>/

Inside this subfolder, we find the storage directory structure:

• msg_store_persistent/: A folder containing segment files with sequential names like 0.rdq, 1.rdq, 2.rdq, and so on. Each .rdq (RabbitMQ Data Queue) file has a default size limit of 16 megabytes (16,777,216 bytes).
• queues/: Every Classic Queue has its own subfolder named after that queue’s UUID hash (e.g., queues/6C8D8.../). Inside this queue folder, the .idx index files are stored.

The refc Binaries Mechanism in RAM Memory #

The Erlang VM (BEAM) uses the Reference-Counted Binary (refc binary) data type for large binary data (usually above 64 bytes). RabbitMQ message payloads are represented as refc binaries allocated outside the queue process heap (off-heap binary allocator).

When a producer publishes one message to a Fanout-type Exchange that routes it to 10 different queues, the message payload stays stored at one single physical memory location. Each queue only receives a small 24-byte reference pointer named ProcBin pointing to that message body.

This refc concept also applies when messages are written to disk through msg_store. The broker only writes the message payload once to a .rdq file in msg_store_persistent, while each of the 10 queues writes its own index reference entry to its respective .idx file. When all queues have deleted (ACKed) the message, the payload’s reference count drops to zero, and only then is the space in the .rdq segment marked as garbage for cleanup during Log Compaction.

Message State Transition Lifecycle: RAM vs Disk #

Throughout its life inside the broker, a message transitions between several states or memory conditions. RabbitMQ tracks this status dynamically to optimize RAM usage and disk I/O performance.

There are three main message storage states managed by the queue process (rabbit_amqqueue_process):

The Memory Paging Out Mechanism #

When the number of messages in queues piles up (backlog) or the system as a whole approaches the RAM alarm limit (high memory watermark), RabbitMQ activates the Memory Pager mechanism.

The pager’s main job is to rescue the broker from an out-of-memory crash (OOM - Out of Memory). The pager scans queues with the largest memory consumption, then orders the queue processes to page out:

1. Message bodies in RAM memory are freed (deleted from RAM).
2. If the message was never written to disk before (e.g., a transient message), its payload is written to the msg_store_transient folder.
3. In RAM, the queue only leaves very small reference index data.
4. If a consumer requests that message later, the broker performs an asynchronous disk read to fetch the payload from disk to RAM (page in), then delivers it to the consumer.

This mechanism ensures broker stability, but with the consequence of significant throughput performance degradation because we switch from nanosecond memory access speeds to millisecond disk I/O latency.

Inline Storage vs msg_store Repository Optimization #

RabbitMQ splits the message-to-disk writing mechanism into two strategies based on message size to minimize disk I/O load (disk write amplification). The built-in threshold for this split is set by the queue_index_embed_msgs_below parameter (by default 4096 bytes or 4 KB).

1. Small Messages (< 4 KB): Inline Storage #

If the message size (including body payload, properties, and headers) is smaller than 4 KB:

• Logic: RabbitMQ writes the entire message contents directly into the queue index file, i.e., the .idx file (like journal.idx on CQv2 or segment index files).
• Advantage: Avoids calling double disk write functions (double write). The broker doesn’t need to write to index files and payload files separately, saving up to 50% of IOPS operations for small messages.

2. Large Messages (> 4 KB): Separate Storage (Split Store) #

If the message size exceeds the 4 KB limit:

• Logic: The broker splits the write into two places. The binary message body payload is written to the global msg_store_persistent repository (binary segment files .rdq with a 16MB maximum size). Meanwhile, metadata information (binary offset position and byte length) is written to the queue’s .idx index file.
• Advantage: Enables efficient RAM cleanup. If a queue experiences a long backlog, the broker can easily delete the 10 MB payload from RAM and only hold small reference indexes in RAM without damaging queue order.

Segmentation & Log Compaction Mechanisms in msg_store #

Large messages are written sequentially (append-only) into 16MB .rdq segment files inside the msg_store_persistent subfolder. Because they’re written append-only, RabbitMQ never modifies or deletes data in the middle of an active segment file when consumers ACK specific messages.

How Is Disk Garbage Cleaned? #

When a consumer successfully processes a message and sends an ACK, that message is marked as “deleted” (garbage) in the queue index file, but its physical binary data remains in the 16MB .rdq segment file. This leaves binary empty gaps (binary gaps).

To prevent the disk from filling with garbage data, RabbitMQ runs a background process called Log Compaction (Segment Recompilation):

1. Ratio Evaluation: The broker tracks the valid data vs garbage data ratio on every disk segment.
2. Recompilation: If the valid data percentage in a segment falls below a threshold (by default < 50%), that segment is nominated for compression.
3. Merge & Delete: The compression process reads two or more fragmented segments, copies the still-valid messages into a new dense segment file, then deletes the old segment files from disk.

flowchart LR
    S1["\"Segment 1 (16MB): [ Valid] [ Garbage ] [ Valid ]"] --> SB["\"New Segment: [ Valid] [ Valid ]"]
    S2["\"Segment 2 (16MB): [ Garbage] [ Valid ] [ Garbage ]"] --> SB

[!WARNING] Compaction Impact in Production: The Log Compaction process consumes very intensive CPU and disk I/O (Read/Write) resources. Under high throughput pressure, if many segment compressions happen simultaneously, the broker can experience latency spikes and slow down producer confirm response times.

fsync Internals and Batch Writing Procedures #

When we send persistent messages with Publisher Confirms enabled, RabbitMQ doesn’t immediately call the fsync system command synchronously for every single message. Doing so would limit our system throughput to only a few hundred messages per second (because of physical disk latency limits).

The Batch Flush Mechanism #

To achieve tens of thousands of messages per second throughput while still guaranteeing data persistence, RabbitMQ implements asynchronous batch writes inside the rabbit_msg_store actor:

1. Write Buffer: Incoming messages are placed into an internal memory queue (flush queue).
2. Trigger Criteria: The broker performs disk writes and triggers fsync when one of the following conditions is met:
   • The number of messages in the buffer reaches a certain limit.
   • The wait time since the first message entered the buffer is exceeded (referring to the internal flush_after parameter or a certain millisecond wait time).
   • No new messages enter the broker for a few microseconds.
3. Confirm ACK: After the fsync operation is successfully completed by the operating system and disk controller, the broker sends ACK confirmations to the producer for that entire batch of messages.

This mechanism enables the best compromise between extreme data safety and high processing performance.

WAL Persistence and Snapshots on Quorum Queues #

The persistence mechanisms above apply to Classic Queues. For Quorum Queues, because they’re based on the Raft consensus algorithm, their data persistence path is fully managed by the Erlang ra library separately.

• Raft WAL (Write-Ahead Log): Every Quorum Queue replica node writes incoming messages to a local Raft WAL file on disk before evaluating commit status.
• Raft Snapshots (Log Compaction): Unlike msg_store, which does asynchronous segment compaction, the ra library writes a summary of the current queue state (snapshot) to disk after a number of log entries are committed (default 1024 entries). After the snapshot is safely written, all binary WAL transaction logs before that snapshot index are instantly deleted to save disk storage space.

Disaster Scenarios and Recovery #

What happens when a RabbitMQ node dies suddenly (e.g., from a power loss)? RabbitMQ is designed to recover its state consistently when it is restarted.

The Bootstrapping and Reconstruction Process #

When RabbitMQ boots back up after an unclean shutdown (dirty shutdown):

1. Mnesia Database Initialization: The broker loads cluster metadata and table schemas.
2. .idx Index File Scanning: The broker scans all queue index files in the queues/ directory. The broker verifies checksums and index file structures to detect data corruption.
3. Pointer Verification to .rdq: For every persistent message in the index, the broker ensures the binary offset in the .rdq file indeed points to a valid message body.
4. Queue Reconstruction: If a mismatch is found (e.g., a message in the index doesn’t exist in the .rdq file because of a pending operating system fsync operation), the broker deletes that corrupted index entry to prevent queue state corruption.
5. Mnesia Rebuild if Needed: If the Mnesia schema database is corrupted, the broker may require administrator intervention to restore data from backups or rebuild replication from neighboring nodes.

Go Code Implementation: Publishing Persistent Messages #

To guarantee messages are written permanently to disk, producers must set the DeliveryMode property value to amqp.Persistent (the number 2) and use Publisher Confirms to ensure physical disk synchronization completes.

package main

import (
	"context"
	"log"
	"time"

	amqp "github.com/rabbitmq/amqp091-go"
)

func main() {
	// 1. Open a TCP Connection
	conn, err := amqp.Dial("amqp://guest:***@localhost:5672/")
	if err != nil {
		log.Fatalf("Gagal terhubung ke RabbitMQ: %s", err)
	}
	defer conn.Close()

	ch, err := conn.Channel()
	if err != nil {
		log.Fatalf("Gagal membuka channel: %s", err)
	}
	defer ch.Close()

	// 2. Enable Publisher Confirms
	// The broker only sends ACK after a persistent message is truly written to disk (fsync)
	err = ch.Confirm(false)
	if err != nil {
		log.Fatalf("Gagal mengaktifkan confirms: %s", err)
	}
	confirmChan := ch.NotifyPublish(make(chan amqp.Confirmation, 1))

	// 3. Declare a Durable Queue
	queueName := "persistent-orders-queue"
	_, err = ch.QueueDeclare(
		queueName,
		true,  // durable: Must be true so the queue survives restarts!
		false, // auto-delete
		false, // exclusive
		false, // no-wait
		nil,
	)
	if err != nil {
		log.Fatalf("Gagal mendeklarasikan antrean: %s", err)
	}

	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
	defer cancel()

	payload := []byte(`{"order_id":"ORD-99001","amount":1500000}`)

	// 4. Publish the Message with DeliveryMode Persistent (2)
	err = ch.PublishWithContext(ctx,
		"", // Default Exchange
		queueName,
		false, // mandatory
		false, // immediate
		amqp.Publishing{
			DeliveryMode: amqp.Persistent, // THE MAIN KEY TO MESSAGE PERSISTENCE!
			ContentType:  "application/json",
			Body:         payload,
		},
	)
	if err != nil {
		log.Fatalf("Gagal menerbitkan pesan: %s", err)
	}

	// 5. Wait for the Physical Disk Write (fsync) Confirmation from the Broker
	confirm := <-confirmChan
	if confirm.Ack {
		log.Println("✓ Pesan sukses tertulis di disk fisik dan terkonfirmasi!")
	} else {
		log.Println("✗ Pesan gagal tertulis di disk (NACK)!")
	}
}

Anti-Patterns vs Data Safety Solutions #

Avoid the following persistence configuration mistakes in production:

1. Enabling Full Persistence on Slow Disk Media #

Configuring all queues and messages (including telemetry logs and debug metrics) with the amqp.Persistent property on servers with mechanical HDD disks or very low cloud IOPS capacity.

Why is this wrong? #

Every persistent message forces the broker to run fsync system call cycles to secure data to disk. On slow disks, disk write queues pile up. The broker is forced to block producer connections (Flow Control), dropping overall system throughput from tens of thousands to only hundreds of messages per second.

• Solution: Apply workload separation (separation of concerns). Use durable queues and persistent messages only for critical transactional data (payments, orders, ledgers). Non-critical data (logging, user clicks, telemetry) must be sent as transient messages (DeliveryMode = 1) to save disk IOPS capacity.

2. Ignoring Disk Fragmentation Monitoring #

Running a broker with nearly full disk storage capacity (e.g., above 80% capacity), assuming the log compaction process will always clean memory instantly.

Why is this wrong? #

The log compaction process needs extra free disk storage space temporarily to write new segment files before it can delete the old fragmented segment files. If our disk is too full, the compression process fails to run because it runs out of disk space, causing the broker to stop entirely from a triggered Disk Alarm.

• Solution: Always monitor broker disk capacity and set the disk alarm limit (disk_free_limit) safely (at minimum 1.5 times the server’s total RAM memory, or a static limit like 5 GB). Make sure there’s extra disk space to facilitate smooth log compaction processes.

Summary #

• Inline vs Split Store — Messages under 4 KB (queue_index_embed_msgs_below) are written directly inline to the queue’s .idx index file to save IOPS operations. Messages above 4 KB are written separately to the msg_store_persistent repository.
• Binary Log Compaction — RabbitMQ cleans ACKed message binaries from disk asynchronously by copying valid data from several 16MB .rdq segments to a new segment and deleting the old segments.
• Raft WAL & Snapshots — Quorum Queues manage persistence using the distributed Write-Ahead Log (WAL) mechanism and periodically create snapshots for Raft log compression.
• fsync & Confirm Costs — ACK confirmations are sent to producers only after the broker successfully ensures data is fsynced safely to physical disk or Quorum cluster majority is reached.
• Beware of I/O Throttling — Excessive persistent message usage on slow disks triggers Flow Control that blocks producers to secure disk I/O capacity.
• Disk Free Space Alarm — Always monitor remaining free disk capacity to facilitate the asynchronous log compaction process that needs temporary free space when writing new segments.

← Previous: Routing & Binding   Next: Queueing →