Use Quorum #

In designing production-level message-based system architectures, guaranteeing message reliability and high availability is the top priority. When using RabbitMQ queues, one of the most fundamental decisions we must make is determining the right queue type to store our transaction data loads. For years, the default choice fell on Classic Queues configured as durable with messages sent persistently. This approach has indeed proven fast and easy to operate. However, for critical modern business workloads—where losing even one message can cause financial losses or serious data inconsistencies—the modern RabbitMQ best practice is always use Quorum Queues.

Quorum Queues are specifically designed to overcome the inherent limitations of Classic Queues in distributed multi-node cluster environments. Many developer teams wrongly assume that ticking the durable and persistent options makes their data fully safe from all forms of failure. Durability and persistence alone actually only protect data from power outage incidents or ordinary broker restarts on a single node. Those two parameters provide no protection at all if the node hosting the queue experiences permanent hardware failure or totally loses cluster connection. To get true cluster resilience, we need consistent log replication, and that’s the main foundation offered by Quorum Queues.

Why Isn’t a Classic Durable Queue Enough? #

To understand why we must migrate from Classic Queues, we need to dissect the data storage anatomy of durable Classic Queues in RabbitMQ clusters. Even if we run a 3-node cluster, Classic Queues are local by default. The queue physically only resides on one specific node, called the home node. Other cluster nodes only store metadata pointers referencing that home node.

When producers publish messages to an exchange, the broker routes those messages through the cluster network to be written to disk on the queue’s home node. If that home node suddenly dies permanently from disk corruption or VM damage:

1. Availability Loss: The queue immediately goes Down and becomes completely inaccessible to producers and consumers. Other nodes have no message copies to continue processing.
2. Silent Data Loss: Even if the server successfully restarts hours later, if the disk suffered physical damage, important messages settled inside it are lost forever.
3. Mirrored Queue Problems (Classic Mirrored Queues Deprecation): RabbitMQ previously had a replication solution called Mirrored Queues (active using ha-mode policies). However, mirrored queues have synchronization problems highly vulnerable to blocking during network recovery, often triggering data loss when queue leaders die before synchronization completes, and their performance is very unpredictable. The Mirrored Queues feature is now fully deprecated and removed from modern RabbitMQ versions, completely replaced by Quorum Queues.

Quorum Queue Replication Mechanisms & Raft Consensus #

Quorum Queues are built on the Raft consensus algorithm implementation (using an Erlang library named ra). Raft is a distributed consensus protocol designed to be easy to understand and guarantees very strong data consistency across cluster nodes.

Under the Raft protocol, Quorum Queue data replication operates with a very structured workflow:

flowchart TD
    Producer["Producer Application"] -->|"1. Publish Message"| Leader["Raft Leader (Home Node)"]
    Leader -->|"2. Write to Local WAL Disk"| LeaderWAL[("Leader Disk WAL")]
    
    Leader -->|"3. Replicate Raft Log"| Follower1["Raft Follower Node A"]
    Leader -->|"3. Replicate Raft Log"| Follower2["Raft Follower Node B"]
    
    Follower1 -->|"4. Write to Disk WAL"| Follower1WAL[("Follower A Disk WAL")]
    Follower2 -->|"4. Write to Disk WAL"| Follower2WAL[("Follower B Disk WAL")]
    
    Follower1WAL -->|"5. Send Write ACK"| Leader
    Follower2WAL -->|"5. Send Write ACK"| Leader
    
    Leader -->|"6. Commit to State Machine & Send Confirm"| Producer

    style Leader stroke:#0288d1,stroke-width:2px
    style Follower1 stroke:#7b1fa2,stroke-width:2px
    style Follower2 stroke:#7b1fa2,stroke-width:2px

1. The Roles of Leaders and Followers #

Every Quorum Queue consists of one Leader instance (handling all write operations from producers and reads from consumers) plus several Follower instances evenly distributed across other cluster nodes.

2. Majority Rules (Write Quorum) #

When producers send messages, Leader nodes don’t immediately send success confirmations (publisher confirms) to producers. Leaders first write those messages to their local disk Write-Ahead Logs (WAL), then spread those replication logs to all Follower nodes in parallel.

A message is considered safely committed (committed state) if and only if a majority of the total cluster replicas have successfully written that log to their respective physical disks and sent approval signals back to the Leader. The quorum majority formula is defined as: $$\lfloor N/2 \rfloor + 1$$ Where $N$ is the configured queue replica count. For a 3-node cluster, the majority is 2 nodes (Leader + 1 Follower). Once the majority is reached, the Leader sends the Publisher Confirm signal to the producer and makes that message ready for consumer application consumption.

3. Seamless Failover (Automatic Leader Election) #

If the Leader node crashes, the remaining Follower nodes detect the absence of heartbeat timeout signals from the Leader. They immediately hold an automatic leader election vote to choose a new Leader from one of the Followers holding the most up-to-date log copy (synchronized replica). This process runs in milliseconds, without data loss, and without requiring manual intervention from operations teams.

Configuring the Right Cluster Node Sizes #

When designing RabbitMQ clusters using Quorum Queues, the node count we use is the determining factor for system availability. We must use odd node counts, with a minimum recommendation of 3 nodes or 5 nodes in production environments.

Why Odd Counts? #

Raft consensus mathematical rules require majority votes to make decisions. Let’s compare even vs odd cluster configurations during node failures:

• 2-Node Cluster Case (Even - Highly Not Recommended):
  • Quorum Formula: $\lfloor 2/2 \rfloor + 1 = 2$ nodes.
  • That means both nodes must be active for messages to commit. If 1 node dies, the remaining node (50% of the cluster) can’t form a majority vote (>50%). As a result, all Quorum Queue message write processes freeze completely. A 2-node cluster has a failure tolerance of zero nodes.
• 3-Node Cluster Case (Odd - Production Standard):
  • Quorum Formula: $\lfloor 3/2 \rfloor + 1 = 2$ nodes.
  • If 1 node dies, there are still 2 active nodes. 2 of 3 nodes is a majority (66%). The cluster keeps functioning fully and can elect a new Leader. The 3-node cluster failure tolerance is 1 node.
• 5-Node Cluster Case (Odd - Large Scale):
  • Quorum Formula: $\lfloor 5/2 \rfloor + 1 = 3$ nodes.
  • If 2 nodes die simultaneously, there are still 3 active nodes (60%). The cluster stays stable. The 5-node cluster failure tolerance is 2 nodes.

Performance Trade-off Analysis (The Cost of Reliability) #

Choosing Quorum Queues means prioritizing data consistency and reliability above other aspects. This choice brings trade-offs we must manage wisely:

1. Higher Publisher Confirm Latency #

Because every message must go through network serialization processes to follower nodes and wait for WAL disk write operations on the majority of nodes to finish, confirmation response times to producers increase compared to Classic Queues. Classic Queues only need one local disk write before replying with confirmations. We must optimize producers to use Asynchronous Confirms (avoiding one-by-one synchronization modes) so throughput stays high.

2. Disk Write Amplification #

Every Quorum Queue message is written to disk several times on different nodes. This means cluster disk I/O utilization increases significantly. Make sure production RabbitMQ nodes use high-speed storage media like NVMe SSDs with adequate IOPS capacity, and periodically monitor I/O latency metrics.

3. Larger RAM and CPU Memory Usage #

Running per-queue Raft consensus engines requires larger Erlang heap memory allocations to maintain machine states, detect Raft connection heartbeats, and manage in-memory log buffers. Don’t use Quorum Queues for thousands of short-lived dynamic queues; Quorum Queues are designed for long-lived static topologies with high message volumes.

Implementing Quorum Queues in Go Applications #

When implementing Quorum Queues in Go applications, we just insert the custom "x-queue-type": "quorum" argument during queue declarations. Additionally, it’s highly recommended to include the "x-delivery-limit" argument to limit the number of automatic retries by the broker if repeated message processing failures occur (poison message protection).

Here is complete Go code for safely declaring and processing messages using Quorum Queues:

package main

import (
	"context"
	"log"
	"os"
	"os/signal"
	"syscall"
	"time"

	amqp "github.com/rabbitmq/amqp091-go"
)

const (
	amqpURI      = "amqp://admin:***@rabbitmq-cluster:5672/"
	queueName    = "queue.order.process"
	exchangeName = "exchange.order.direct"
	routingKey   = "order.created"
)

type OrderConsumer struct {
	conn    *amqp.Connection
	channel *amqp.Channel
	close   chan *amqp.Error
}

// InitQueue builds connections and safely declares Quorum Queues
func (oc *OrderConsumer) InitQueue() error {
	var err error
	log.Println("Menghubungkan ke cluster RabbitMQ...")
	oc.conn, err = amqp.Dial(amqpURI)
	if err != nil {
		return err
	}

	oc.channel, err = oc.conn.Channel()
	if err != nil {
		oc.conn.Close()
		return err
	}

	// 1. Declare the Main Exchange
	log.Printf("Mendeklarasikan exchange '%s'...\n", exchangeName)
	err = oc.channel.ExchangeDeclare(
		exchangeName,
		"direct",
		true,  // durable
		false, // auto-deleted
		false, // internal
		false, // no-wait
		nil,   // arguments
	)
	if err != nil {
		oc.channel.Close()
		oc.conn.Close()
		return err
	}

	// 2. Declare the Quorum Queue with high-resilience parameter configurations
	log.Printf("Mendeklarasikan Quorum Queue '%s'...\n", queueName)
	queueArgs := amqp.Table{
		"x-queue-type":     "quorum", // Enables the Raft-based Quorum Queue type
		"x-delivery-limit": 5,        // Maximum consumption attempts before automatic discard/to DLQ
	}

	_, err = oc.channel.QueueDeclare(
		queueName,
		true,  // durable (must be set to true for quorum queues)
		false, // auto-delete (must be set to false)
		false, // exclusive (must be set to false)
		false, // no-wait
		queueArgs,
	)
	if err != nil {
		oc.channel.Close()
		oc.conn.Close()
		return err
	}

	// 3. Bind (Bind) the Quorum Queue to the Exchange
	log.Println("Mengikat queue ke exchange...")
	err = oc.channel.QueueBind(
		queueName,
		routingKey,
		exchangeName,
		false,
		nil,
	)
	if err != nil {
		oc.channel.Close()
		oc.conn.Close()
		return err
	}

	// Limit the QoS prefetch limit so consumers process messages in a controlled manner
	err = oc.channel.Qos(10, 0, false)
	if err != nil {
		oc.channel.Close()
		oc.conn.Close()
		return err
	}

	oc.close = make(chan *amqp.Error)
	oc.channel.NotifyClose(oc.close)

	return nil
}

func (oc *OrderConsumer) StartConsume(ctx context.Context) {
	deliveries, err := oc.channel.Consume(
		queueName,
		"order-worker-1",
		false, // auto-ack set to false so messages are safe if consumers crash
		false,
		false,
		false,
		nil,
	)
	if err != nil {
		log.Printf("Gagal memulai consume loop: %v\n", err)
		return
	}

	log.Println("Worker aktif. Menunggu transaksi masuk...")

	for {
		select {
		case <-ctx.Done():
			log.Println("Menghentikan worker secara graceful...")
			return
		case errClosed := <-oc.close:
			if errClosed != nil {
				log.Printf("Channel koneksi ditutup secara sepihak: %v. Reconnecting...\n", errClosed)
				oc.Reconnect(ctx)
				return
			}
		case msg, ok := <-deliveries:
			if !ok {
				return
			}
			oc.handleOrder(msg)
		}
	}
}

func (oc *OrderConsumer) Reconnect(ctx context.Context) {
	for {
		select {
		case <-ctx.Done():
			return
		default:
			log.Println("Mencoba menghubungkan kembali ke cluster dalam 5 detik...")
			time.Sleep(5 * time.Second)

			err := oc.InitQueue()
			if err == nil {
				log.Println("✓ Berhasil terhubung kembali.")
				go oc.StartConsume(ctx)
				return
			}
			log.Printf("Gagal menghubungkan kembali: %v\n", err)
		}
	}
}

func (oc *OrderConsumer) handleOrder(msg amqp.Delivery) {
	// Evaluate the x-delivery-count header to monitor message failure status
	deliveryCount, exists := msg.Headers["x-delivery-count"]
	if exists {
		log.Printf("[METRIK] Pesan ini sudah dicoba sebanyak %v kali\n", deliveryCount)
	}

	log.Printf("[TRANSAKSI] Memproses order: ID=%s, Payload=%s\n", msg.CorrelationId, string(msg.Body))
	
	// Simulate transaction database storage logic
	err := oc.saveToDatabase(msg.Body)
	if err != nil {
		log.Printf("[GAGAL] Gagal memproses order ID %s: %v. Mengirimkan NACK...\n", msg.CorrelationId, err)
		
		// Send a NACK with requeue = false so it's forwarded to the DLQ if the delivery limit is exceeded
		msg.Nack(false, false)
		return
	}

	// Send an ACK if the transaction process succeeds
	msg.Ack(false)
	log.Printf("[SUKSES] ACK dikirim untuk order ID %s\n", msg.CorrelationId)
}

func (oc *OrderConsumer) saveToDatabase(payload []byte) error {
	// Simulate database I/O latency
	time.Sleep(50 * time.Millisecond)
	return nil
}

func (oc *OrderConsumer) Close() {
	if oc.channel != nil {
		oc.channel.Close()
	}
	if oc.conn != nil {
		oc.conn.Close()
	}
}

func main() {
	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	oc := &OrderConsumer{}
	err := oc.InitQueue()
	if err != nil {
		log.Fatalf("Gagal inisialisasi awal ke cluster RabbitMQ: %v\n", err)
	}
	defer oc.Close()

	go oc.StartConsume(ctx)

	stop := make(chan os.Signal, 1)
	signal.Notify(stop, os.Interrupt, syscall.SIGTERM)

	<-stop
	log.Println("Menerima sinyal shutdown...")
	cancel()
	time.Sleep(1 * time.Second)
}

Feature Comparison: Classic Queue vs Quorum Queue #

Here is an in-depth comparison matrix table to help us distinguish durable Classic Queue capabilities from Quorum Queues:

Summary #

• Durability vs Replication — Durable and persistent only protect data from single server restarts, while Quorum Queues protect data from physical hardware failures or node crashes.
• Use Raft Consensus — Quorum Queues replicate message write logs to a majority of cluster nodes before replying ACKs to producers, ensuring data integrity stays consistent.
• Use Odd Node Counts — Always run production clusters with 3 or 5 nodes to allow stable majority vote formation during failovers.
• Understand Performance Trade-offs — Quorum Queue reliability demands performance costs in the form of higher confirm latency and more intensive disk I/O loads compared to classic queues.

← Previous: Design First   Next: Message Size →