Quorum Queue #

In the modern RabbitMQ ecosystem, if we talk about designing reliable, fault-tolerant, High Availability production-scale message infrastructure, almost the entire conversation leads to one queue type: the Quorum Queue.

Previously, RabbitMQ relied on the Mirrored Queue feature (actively replicated classic queues). However, Mirrored Queues were often vulnerable to data inconsistency problems during network partitions (split-brain) and had slow, CPU-heavy synchronization mechanisms. The Quorum Queue arrives as the modern industry standard in RabbitMQ to replace the now-deprecated Mirrored Queue. Leveraging the distributed Raft consensus algorithm, the Quorum Queue provides strict data consistency guarantees and deterministic disaster recovery. This article dissects in depth the Quorum Queue’s internal architecture, the Erlang ra library controlling it, how log replication works, cluster failure scenario mitigation, and practical Go implementations.

What is a Quorum Queue & Raft Consensus? #

A Quorum Queue is a distributed queue type designed to maintain high data availability and consistency in multi-node RabbitMQ clusters. This queue is declared by setting the "x-queue-type": "quorum" parameter.

The Raft Consensus Algorithm #

The Quorum Queue is built on the Raft Consensus Algorithm, a protocol designed to manage state machine log replication in distributed systems. Inside a cluster:

• Leader: For every declared Quorum Queue, one node is appointed as the Leader. All message publish and consume operations must pass through this Leader node.
• Followers: The other nodes in the cluster appointed as replicas act as Followers. They don’t serve clients directly, but passively replicate the log sent by the Leader.
• Majority (Quorum): For a message to be considered successfully received and safe, it must be written to disk by a majority of the replica group member nodes. The majority quorum formula is: $$Q = \lfloor N/2 \rfloor + 1$$ Where $N$ is the total number of queue replicas. In a 3-node cluster, the required majority is 2 nodes. In a 5-node cluster, 3 nodes are needed.

flowchart TD
    Producer["Producer (Publish)"] --> Leader["Raft Leader (Node A)"]
    
    subgraph Cluster["3-Node RabbitMQ Cluster"]
        Leader -->|1. Append Log| LogA["Local WAL (Node A)"]
        Leader -->|2. Replicate Log| Follower1["Raft Follower (Node B)"]
        Leader -->|2. Replicate Log| Follower2["Raft Follower (Node C)"]
        
        Follower1 -->|3. Confirm ACK| LogB["Local WAL (Node B)"]
        Follower2 -. No Response Yet .-> LogC["Local WAL (Node C)"]
    end
    
    LogB -->|4. Quorum Reached: 2/3 Nodes Agree| Commit{"Commit State Machine"}
    Commit -->|5. Confirm ACK| Producer

Internal Erlang Architecture (The ra Library) #

RabbitMQ doesn’t write the Raft algorithm from scratch. At the Erlang BEAM runtime level, RabbitMQ implements Raft using a dedicated open-source library named ra developed by the RabbitMQ core team.

The ra Library and the State Machine Engine #

Every time a quorum queue is created, RabbitMQ spawns one Raft consensus group through the ra library. Each queue replica on different nodes runs as an independent Erlang process communicating through the Erlang distribution protocol.

• Write-Ahead Log (WAL): When a message is published to the Leader, it is not immediately inserted into the in-memory queue data structure. The ra library writes the message into a dedicated binary log file called the Write-Ahead Log (WAL) on the Leader node’s local disk sequentially (append-only).
• Log Replication: The Leader sends new WAL entries to all Follower nodes. Followers write that data to their local WAL files and send confirmation signals back to the Leader.
• State Machine Commit: After receiving confirmation from a majority of Follower nodes, the Leader declares the entry committed. The Leader then inserts the message contents into the local state machine (representing the active in-memory queue) and triggers the Publisher Confirm ACK signal to the producer client application.

Physical Quorum Storage on Disk #

Physically, Quorum Queue log data and snapshots are stored in the node data folder under the sub-directory:

/var/lib/rabbitmq/mnesia/rabbit@hostname/quorum/rabbit@hostname/

Inside this directory, the ra library manages WAL files and snapshot files. Snapshots are used for the log compaction process. If WAL files get too large, the ra library summarizes the current queue state into a single snapshot file and deletes old transaction logs to save disk space.

Cluster Failure Scenario Analysis #

The Quorum Queue’s main strength is its ability to face node failures without triggering data corruption. Let’s analyze the behavior of a 3-node cluster (Node A = Leader, Node B = Follower, Node C = Follower) in various crash scenarios:

Scenario 1: One Follower Node Dies (Node C) #

If Node C suddenly loses power or dies:

1. Operations Keep Running: Node A (Leader) and Node B (Follower) are still active. The active node count = 2 of 3 (meeting the majority quorum $\ge 2$).
2. Confirmations Still Sent: Producers can still send messages, and messages are still confirmed because Node A and Node B successfully wrote to disk.
3. Automatic Recovery: When Node C comes back alive, it requests the lagging log synchronization from Node A. Node A sends the remaining WAL logs asynchronously until Node C is balanced (up-to-date).

Scenario 2: The Leader Node Dies (Node A) #

If Node A experiences a total failure:

1. Leader Election: Failure detection is triggered by the loss of the Raft heartbeat. Node B and Node C, detecting the Leader’s death, start the Leader Election phase.
2. Democratic Selection: The node with the most up-to-date WAL log (e.g., Node B) is elected as the new Leader.
3. Fast Transition: Producer and consumer clients are automatically redirected to Node B by the client library. The queue returns to normal operation within seconds.

Scenario 3: Network Partition (Split-Brain Immunity) #

Imagine a network disruption splits the 3-node cluster into two isolated parts: the Left Side (Node A - Leader) and the Right Side (Node B and Node C).

flowchart TD
    subgraph Left["Left Side (Partition A)"]
        NodeA["Node A (current Leader)"]
        Fail["Quorum Failed (1/3)<br>Publishes REJECTED"]
        NodeA --> Fail
    end
    
    subgraph Right["Right Side (Partition B)"]
        NodeBC["Node B + Node C (Election Occurs)"]
        Success["Node B elected as Leader<br/>Quorum Reached (2/3)<br/>Publishes ACCEPTED"]
        NodeBC --> Success
    end
    
    Left -. "Network Disconnected" .-> Right

• Left Side (Node A): Node A detects it can no longer communicate with Node B and Node C. When a new message enters Node A, it tries to replicate but fails to reach a majority (only 1 of 3 nodes agrees). Node A rejects that message publication and withholds sending an ACK to the producer.
• Right Side (Node B & Node C): Both nodes detect the loss of Leader A. Because they together form a majority (2 of 3 nodes), they hold an election and appoint Node B as the new Leader. The right side can still safely receive and confirm new messages.
• Reconciliation (Healing): When the network recovers, Node A detects the legitimate presence of Leader B. Node A steps down to Follower, discards all local logs that never got committed to a majority, and synchronizes all its data following Node B’s log. Data inconsistency is completely prevented.

Quorum vs Classic vs Mirrored (Legacy) Comparison #

Here is a technical comparison table to understand the differences between the Quorum Queue and other RabbitMQ queue types:

FIFO & Memory Management on Quorum Queues #

As a distributed consensus-based system, there are two important operational aspects often questioned: how the Quorum Queue guarantees message order (FIFO) and how it manages RAM memory usage.

FIFO Ordering Guarantees in Raft Consensus #

The ra library in Erlang guarantees message order (First-In, First-Out) stays consistent even though messages are replicated to many Follower nodes in parallel.

• Sequential Log Indexing: Every log entry received by the Leader is given a monotonically sequential log index number (log index) and a leadership term marker (term).
• Ordered Replication: The Leader sends log replication to Followers in the exact same index order. Followers write those entries to their local disks in that index order.
• Committed Index Barrier: Messages are never delivered to consumer applications before their log index passes the committed_index (the index approved by a majority of nodes). Consumers only read data from the already-committed state machine, ensuring FIFO order is deterministically maintained across the cluster, even after Leader failures and new election transitions.

Memory Management Optimization (Soft & Hard Limits) #

Unlike Classic Queues, which can move message payloads entirely from RAM to disk (lazy mode), early Quorum Queue versions tended to hold committed messages in RAM as a cache to maintain fast consumption performance. However, if a large backlog occurs, this can cause the broker to run out of memory.

To address this, modern RabbitMQ versions introduce Quorum memory limit configurations:

1. quorum_commands_soft_limit: The soft limit on the number of Raft log entries allowed in RAM memory (default 256 entries). If the backlog exceeds this limit, the broker starts emptying message payloads from RAM, leaving only their pointer indexes.
2. quorum_commands_hard_limit: The hard limit (default 1024 entries). If the message pile-up exceeds this limit, the broker immediately forcibly pages Raft logs to disk aggressively and limits producer ingress throughput to secure RAM memory stability.

Go Code Implementation: Declaring a Quorum Queue #

To create a Quorum Queue, we must declare the queue with the "x-queue-type" argument set to "quorum". This queue must be declared as durable (durable = true).

Here is a complete Go implementation code example:

package main

import (
	"context"
	"log"
	"time"

	amqp "github.com/rabbitmq/amqp091-go"
)

func main() {
	// 1. Connect to the RabbitMQ cluster
	conn, err := amqp.Dial("amqp://guest:***@localhost:5672/")
	if err != nil {
		log.Fatalf("Gagal terhubung ke RabbitMQ: %s", err)
	}
	defer conn.Close()

	ch, err := conn.Channel()
	if err != nil {
		log.Fatalf("Gagal membuka channel: %s", err)
	}
	defer ch.Close()

	// 2. Enable Publisher Confirms
	// Very important for quorum queues so producers know when Raft consensus is reached
	err = ch.Confirm(false)
	if err != nil {
		log.Fatalf("Gagal mengaktifkan Publisher Confirms: %s", err)
	}
	confirmChan := ch.NotifyPublish(make(chan amqp.Confirmation, 1))

	// 3. Define the Quorum Queue Arguments
	queueArgs := amqp.Table{
		"x-queue-type": "quorum", // Setting the queue type to Quorum!
	}

	queueName := "critical-transactions-queue"

	// 4. Queue Declaration (Must be Durable = true)
	_, err = ch.QueueDeclare(
		queueName,
		true,      // durable: must be true for Quorum Queues!
		false,     // auto-delete
		false,     // exclusive
		false,     // no-wait
		queueArgs, // Registering the quorum type arguments
	)
	if err != nil {
		log.Fatalf("Gagal mendeklarasikan quorum queue: %s", err)
	}
	log.Printf("✓ Quorum Queue %s berhasil dideklarasikan.", queueName)

	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
	defer cancel()

	payload := []byte(`{"transaction_id":"TX-9090","amount":2500000}`)

	// 5. Send a persistent message
	err = ch.PublishWithContext(ctx,
		"", // Default Exchange
		queueName,
		false,
		false,
		amqp.Publishing{
			DeliveryMode: amqp.Persistent, // Writes the message to the physical disk WAL
			ContentType:  "application/json",
			Body:         payload,
		},
	)
	if err != nil {
		log.Fatalf("Gagal mengirim pesan ke quorum: %s", err)
	}

	// Wait for majority node consensus to be reached and ACKed
	confirm := <-confirmChan
	if confirm.Ack {
		log.Println("✓ Pesan berhasil direplikasi ke mayoritas node klaster dan terkonfirmasi!")
	} else {
		log.Println("✗ Pesan gagal mencapai konsensus mayoritas klaster (NACK)!")
	}
}

Anti-Patterns to Avoid #

Quorum distributed design demands compliance with the following distributed system rules to avoid performance degradation:

1. Using an Even Number of Replicas (2 or 4 Nodes) #

Declaring a Quorum Queue to be distributed across an even number of cluster nodes.

Why is this wrong? #

Systems with an even replica count are inefficient and waste disk/network resources without adding fault tolerance. Let’s calculate:

• If we use 2 nodes, the required quorum is 2 nodes ($2/2 + 1 = 2$). If 1 node dies, the system can’t reach quorum, so the queue becomes immediately unavailable. Fault tolerance = 0 nodes.
• If we use 3 nodes, the required quorum is 2 nodes ($3/2 + 1 = 2$). If 1 node dies, the system keeps running because there are 2 active nodes. Fault tolerance = 1 node.
• If we use 4 nodes, the required quorum is 3 nodes ($4/2 + 1 = 3$). If 1 node dies, 3 remain (can run). But if 2 nodes die, only 2 remain (can’t reach quorum). Fault tolerance = 1 node.

We can see that a 4-node cluster has exactly the same fault tolerance as a 3-node cluster, yet wastes disk and network resources on a useless fourth replica.

• Solution: Always use an odd replica count, namely 3 or 5 nodes, to optimize fault tolerance against infrastructure cost.

2. Using Quorum Queues for Temporary (Short-lived) Queues #

Declaring short-lived RPC response queues or dynamic monitoring queues as Quorum Queues.

Why is this wrong? #

Creating and deleting Raft consensus groups at the ra library level requires enormous cluster synchronization computational cost in the Mnesia database. If our application dynamically creates and deletes Quorum Queues many times per minute, cluster CPU performance collapses from database schema locks (Mnesia schema locks).

• Solution: Only use Quorum Queues for static, permanent business transactional data queues. Short-lived temporary queues must always use Exclusive Queues or Transient Queues.

Summary #

• Distributed Raft Consensus — The Quorum Queue guarantees high consistency by replicating messages to several cluster nodes using the Raft consensus algorithm managed by the Erlang ra library.
• Majority (Quorum) Requirement — Messages are only considered safe and trigger ACK confirmations to producers if successfully written to local physical disk (WAL) by a majority of replica group member nodes.
• Split-Brain Immunity — The Raft consensus design prevents dual-master (split-brain) scenarios during cluster network partitions by rejecting writes on the network minority side.
• Must Be Durable — Architecturally, Quorum Queues must be declared with the durable = true parameter and paired with persistent messages (DeliveryMode = 2).
• Odd Replica Count — Always configure the replica count in an odd number (3 or 5 nodes) for consensus efficiency and system failure resilience optimization.
• Avoid Dynamic Churn — Never use Quorum Queues for short-lived dynamic queues (like RPC responses) to avoid Mnesia cluster schema lock loads.

← Previous: Lazy vs Classic   Next: Publishing →