Characteristic #

When we evaluate message broker middleware technology for production system architecture needs, understanding a list of marketing features is not enough. Real-world operational success depends heavily on how we understand the system’s characteristics and dynamic behavior under extreme workloads, when the network fluctuates, or when server memory capacity approaches its maximum threshold. RabbitMQ has very distinctive operational characteristics that fundamentally distinguish it from log-based message processing systems like Apache Kafka. This article dissects RabbitMQ’s core characteristics in depth so we can design distributed systems that are reliable, efficient, and resilient to failure.

Push-Based vs Pull-Based Delivery Models #

One of the most fundamental architectural differences between RabbitMQ and other message brokers (such as Apache Kafka) lies in the message delivery model to consumers. RabbitMQ adopts the Push-Based delivery model (Broker-Push), while Kafka adopts the Pull-Based model (Consumer-Pull).

flowchart TD
    subgraph PushModel["Push-Based Model (RabbitMQ)"]
        direction TB
        BrokerP["RabbitMQ Broker (Smart)"] -->|"Actively pushes messages"| SocketP["TCP Socket"]
        SocketP --> ConsumerP["Consumer (Dumb)"]
        ConsumerP -. ACK .-> BrokerP
    end

1. How the Push Model Works #

In the push-based model, the RabbitMQ server actively monitors message availability in queues. The moment a message lands in a queue and there is a connected consumer with capacity, RabbitMQ instantly launches the data through the TCP socket toward the consumer. The consumer doesn’t need to repeatedly call polling functions on the broker; it only needs to register a callback function (event listener) and wait for messages to be delivered by the broker.

  • Advantage: Message delivery latency is very low (in sub-milliseconds) because messages are delivered immediately when available, without polling cycle gaps.
  • Weakness: If the broker’s delivery speed far exceeds the consumer’s processing ability, the consumer can become overloaded, run out of memory, and eventually crash.

2. The Importance of QoS and Prefetch Count Settings #

To prevent consumers from being overwhelmed by the broker’s endless message pushes, we must use the QoS (Quality of Service) setting called Prefetch Count on our consumers.

  • prefetch_count sets the maximum limit of unacknowledged messages RabbitMQ may send to one consumer connection.
  • If we set prefetch_count = 10, RabbitMQ sends a maximum of the first 10 messages to the consumer. The broker then holds the remaining messages in its queue and will not send the 11th message before the consumer finishes processing and sends at least one success confirmation (Acknowledgement/ACK) back to the broker.
sequenceDiagram
    autonumber
    participant Broker as RabbitMQ Broker
    participant Consumer as Consumer (Prefetch = 1)

    Broker->>Consumer: Push Message 1
    Note over Consumer: Processing Message 1 (Takes 2 Seconds)
    Note over Broker: Message 2 Arrives in Queue (Held by Broker!)
    Consumer-->>Broker: ACK Message 1
    Note over Broker: Message 1 Deleted from Queue
    Broker->>Consumer: Push Message 2

Setting prefetch_count to 0 (unlimited) in a production environment is a dangerous action because it can trigger a crash on consumer instances during a sudden message surge (traffic spike).

At-Least-Once Delivery Guarantee and Idempotency Consequences #

In the world of distributed systems, network failures and server crashes are a certainty. Therefore, delivery guarantees are a crucial characteristic. By default, RabbitMQ guarantees message delivery at the At-Least-Once Delivery level.

1. How Does RabbitMQ Guarantee At-Least-Once? #

This guarantee is achieved through two coordination mechanisms:

  • Publisher Confirms: A mechanism where RabbitMQ sends a confirmation (ACK) back to the producer after the message is successfully received by the exchange and safely written to the queue. If the broker crashes before storing the message, the producer doesn’t receive an ACK and is responsible for republishing the message.
  • Consumer Acknowledgements: The consumer must send a confirmation (ACK) back to the broker after successfully processing a message. If the consumer dies suddenly mid-process, its TCP connection drops. RabbitMQ detects this disconnection, returns the message status to undelivered, and immediately sends it to another available consumer (redelivery).

2. The Logical Limit of Exactly-Once #

Many developers hope to get an Exactly-Once guarantee from a message broker. However, in distributed systems theory (as proven by the Two Generals’ Problem), a pure Exactly-Once guarantee at the network transport level is impossible without a very slow, throughput-destroying two-phase commit coordination.

For example, if a consumer finishes processing a message and its database update succeeds, but the network dies exactly one millisecond before the consumer sends the ACK packet to RabbitMQ, then:

  1. RabbitMQ assumes the consumer died without completing its task.
  2. RabbitMQ sends the same message to another consumer.
  3. The second consumer processes the same message a second time.

3. The Absolute Solution: Consumer Idempotency #

Because message duplication is an unavoidable side effect of the At-Least-Once guarantee, we must design our consumer logic to be Idempotent. Idempotency means processing the same message repeatedly produces the same end result as processing it once.

We can implement idempotency by storing a unique Message ID in the consumer’s database before performing the main data processing. If that message ID already exists in the database, we simply ignore the message and send a success ACK to RabbitMQ.

FIFO Queue Anatomy and Message Ordering Limitations #

The queue characteristic of RabbitMQ theoretically follows the FIFO (First-In, First-Out) principle. The first message entering the queue is also the first delivered to consumers. However, in production environments, this FIFO correctness has strict limitations that developers often forget.

When Does Message Ordering Fail? #

There are several common scenarios that can break message order in RabbitMQ:

A. Multi-Consumer (Parallel Consumers) #

If we have one queue read by several consumer instances simultaneously to speed up throughput, message processing order breaks instantly. Even though RabbitMQ sends messages in order (Message 1 to Consumer A, Message 2 to Consumer B), differences in CPU speed or local database latency can cause Consumer B to finish Message 2 before Consumer A finishes Message 1.

B. Message Requeue #

If a consumer returns a message to the queue using the basic.nack(requeue=true) or basic.reject(requeue=true) instruction due to a temporary processing failure, the message is placed back at the head of the queue. This breaks the original message order because the failed message gets reprocessed after the newer messages behind it have been sent.

C. Priority Queues #

If we configure a queue as a Priority Queue, RabbitMQ rearranges message order inside the queue based on the priority level attached to message properties, not based on message arrival time.

D. Message TTL (Time-To-Live) #

Messages with different expiration times (TTL) inside a queue can break FIFO because a message in the middle of the queue can be automatically deleted when it expires, changing the delivery arrangement for subsequent messages.

How to Maintain Strict Ordering #

If our application needs an absolute message ordering guarantee (e.g., bank account mutation history), we must constrain our queue topology:

  1. Use Single Active Consumer on that queue so only one consumer thread processes data at a time.
  2. Never perform requeue=true on failed messages. Send failed messages to a separate queue (DLQ - Dead Letter Queue) for manual processing or a separate remediation flow.

State Management Differences: Durability vs Message Persistence #

To avoid losing messages due to server failures, we must understand how RabbitMQ manages its state storage. Many developers equate “Durable” with “Persistent”. In fact, they are different configurations that must be combined correctly.

1. Durable Queue #

  • Definition: Refers to the metadata structure of the queue itself.
  • Behavior: If a queue is configured as Durable, the queue name, exchange configuration, and binding relationships are permanently recorded in the broker’s disk storage. If the RabbitMQ server crashes or is restarted, the empty queue is automatically recreated when the server starts.
  • Important: Configuring a queue as Durable does not automatically make the messages inside it survive a server restart.

2. Persistent Message #

  • Definition: Refers to the content and payload of the message itself.
  • Behavior: The producer must actively mark the message with the delivery_mode = 2 (persistent) delivery property when publishing. When the RabbitMQ broker receives a message with this marker, it immediately writes the message data to a storage log file on disk before sending a success confirmation (ACK) to the producer.
  • Successful Combination: To guarantee messages survive a server restart, we must send Persistent messages to a queue configured as Durable.
Durable Queue + Transient Message = Messages LOST on server restart
Transient Queue + Persistent Message = Queue LOST (and its messages) on server restart
Durable Queue + Persistent Message = Messages SURVIVE server restart (Production Guarantee)

3. RabbitMQ Is Not a Database #

It’s important to always remember that RabbitMQ is designed as a transient message store. Its internal characteristics are optimized for processing empty queues or short queues. As soon as a message is consumed and ACKed, RabbitMQ immediately deletes the message from disk for I/O efficiency. If we hoard millions of messages in a RabbitMQ queue without consuming them (turning it into a long-term database), the RabbitMQ server’s RAM and disk I/O performance will degrade drastically due to Erlang memory management overhead.

Advanced Flow Control and Memory Management #

As middleware running on the Erlang VM, RabbitMQ is very sensitive to server Memory Exhaustion. To keep the server stable so it doesn’t collapse under a flood of data from aggressive producers, RabbitMQ implements high-level safety mechanisms called Flow Control and Watermark Alarms.

1. Memory and Disk Watermark Alarms #

RabbitMQ monitors RAM usage and disk space in real time. By default:

  • Memory Watermark: If RabbitMQ’s RAM usage exceeds 40% of the server’s total physical RAM, RabbitMQ activates a memory alarm.
  • Disk Free Limit: If the free disk space on the server drops below the safe limit (default 50MB or configured as a memory equivalent), RabbitMQ activates a disk alarm.

Once these alarms are active, RabbitMQ drastically blocks reading data from all producer connection TCP sockets. Producers cannot publish new messages until server memory or disk usage drops back to a safe level.

2. Credit-Based Flow Control in the Erlang BEAM #

At a more micro level, the Erlang BEAM VM uses a Credit-Based Flow Control scheme to manage interactions between Erlang processes inside the broker. Every data-sending process (e.g., the TCP connection receiving process) has a limited credit quota to send data to the next process (e.g., the queue process). If the queue process is busy writing data to disk, it won’t grant new credit to the connection receiving process. Without credit, the connection receiving process stops reading data from the TCP network socket, which automatically triggers TCP Backpressure on the producer side.

Anti-Pattern vs Solution: Priority Queues Without Prefetch Limits or Unlimited Requeue #

Let’s study a fatal mistake that often occurs from misunderstanding RabbitMQ’s asynchronous message delivery characteristics.

Anti-Pattern Code: Unlimited Requeue Using basic.nack #

In the code below, when the consumer fails to process a message due to an internal database failure (e.g., a busy database), the consumer immediately returns the message to the original queue using requeue=true continuously. This triggers an infinite loop that consumes 100% of both the consumer’s and the broker’s CPU.

// ANTI-PATTERN: Unlimited requeue triggers a CPU starvation loop
func StartBadConsumer() {
    msgs, _ := GlobalChannel.Consume("payment-queue", "", false, false, false, false, nil)
    
    for d := range msgs {
        err := processPayment(d.Body)
        if err != nil {
            // ✗ AVOID: Requeueing endlessly without limits for system errors
            log.Printf("Gagal memproses, kembalikan ke antrean: %v", err)
            d.Nack(false, true) // requeue = true
            
            // Problem: The same message is immediately pushed back by RabbitMQ
            // to this consumer within sub-milliseconds. The consumer fails again,
            // nacks again, and triggers 100% CPU consumption instantly.
        } else {
            d.Ack(false)
        }
    }
}

Practical Solution: Retry Queue with DLQ and Backoff #

The best approach for temporary failures is limiting the retry count and using a combination of a Dead Letter Exchange (DLX) or publishing messages to a retry queue with a time delay.

// CORRECT: Using a Dead Letter Queue and retry limits
func StartGoodConsumer() {
    // Set prefetch_count so the consumer isn't flooded with data
    _ = GlobalChannel.Qos(1, 0, false)
    
    msgs, _ := GlobalChannel.Consume("payment-queue", "", false, false, false, false, nil)
    
    for d := range msgs {
        err := processPayment(d.Body)
        if err != nil {
            log.Printf("Gagal memproses pesan: %v", err)
            
            // Check whether this message has been redelivered many times before
            retryCount := getRetryCountFromHeaders(d.Headers)
            
            if retryCount >= 3 {
                // ✓ SOLUTION: Send to the Dead Letter Queue (DLQ) after 3 failures
                log.Println("Pesan mencapai batas maksimal percobaan. Pindahkan ke DLQ.")
                d.Nack(false, false) // requeue = false, the message automatically enters the DLX if configured
            } else {
                // ✓ SOLUTION: Publish to the retry queue with an incremented counter header
                incrementAndPublishToRetryQueue(d)
                d.Ack(false) // Remove from the main queue to avoid triggering a blocking loop
            }
        } else {
            d.Ack(false)
        }
    }
}

Summary #

  • Push-Based Model — A proactive broker-to-consumer delivery characteristic providing very low latency, but requiring the prefetch_count setting to prevent overload.
  • Idempotency Requirement — The logical consequence of distributed failures at the At-Least-Once delivery guarantee level, requiring consumers to recognize duplicate messages on their own.
  • FIFO Limitations — Message ordering rules that only apply to single-consumer scenarios without message requeue or priority queue scenarios.
  • Flow Control & Watermarks — Erlang and TCP internal safety characteristics that protect the RabbitMQ server from memory exhaustion collapse by dynamically blocking producers.

← Previous: Decoupling   Next: Broker & Node →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact