Erlang VM #

Many software engineering teams run RabbitMQ every day without ever realizing the most fundamental architectural fact: RabbitMQ runs on the Erlang BEAM VM virtual machine. The BEAM VM is not some small implementation detail that can be ignored; this runtime is what exclusively gives RabbitMQ its legendary stability characteristics, extreme fault tolerance, and the ability to handle hundreds of thousands of concurrent connections simultaneously. Without understanding how the Erlang VM works internally, we will often misdiagnose performance problems, misconfigure memory allocation, or misdesign reliable cluster architectures. This article dissects in depth the architectural implications of the Erlang VM for RabbitMQ operations in production environments.

The Lightweight Process Concurrency Model #

The main strength of the Erlang BEAM VM lies in its concurrency model, based on Lightweight Processes. Erlang processes are not the same as operating system processes (OS Processes) and are also very different from operating system threads (OS Threads).

Here is a comparison of concurrency characteristics at the operating system level versus the Erlang BEAM:

Inside the RabbitMQ runtime, this lightweight process model is mapped granularly to every component:

• Every connected TCP client connection is managed by 1 Erlang process.
• Every logical Channel inside that connection is managed by its own dedicated Erlang process.
• Every Queue is managed by 1 main Erlang process.

Because the cost of creating and maintaining Erlang processes is so cheap, a single RabbitMQ node can handle more than 100,000 active TCP connections concurrently without experiencing OS thread exhaustion. The Erlang VM’s internal scheduler (Erlang Scheduler) automatically maps these millions of lightweight processes to the server’s physical CPU cores dynamically and in parallel.

Memory Management: Per-Process Garbage Collection #

For developers used to runtimes like the Java Virtual Machine (JVM) or the Go runtime, latency spikes from Garbage Collection (GC) are a classic challenge. On the JVM, when memory allocation is full, GC runs a Stop-The-World phase that temporarily halts the entire program’s execution to sweep memory. This can trigger fatal latency spikes in real-time messaging systems.

The Erlang BEAM VM solves this problem with a revolutionary approach: Per-Process Garbage Collection.

flowchart TD
    subgraph BEAM_VM["Erlang BEAM VM Memory"]
        direction LR
        subgraph Process_A["Process A (Queue 1)"]
            HeapA["Local Heap (Independent GC)"]
        end
        subgraph Process_B["Process B (Channel 1)"]
            HeapB["Local Heap (Independent GC)"]
        end
        subgraph Process_C["Process C (Connection 1)"]
            HeapC["Local Heap (Independent GC)"]
        end
    end

Inside the BEAM VM:

1. No Global Heap: There is no large memory space shared by all processes. Every Erlang process has its own isolated local memory allocation (heap and stack).
2. Localized GC: Garbage Collection only runs at the individual process level. When a queue finishes its task and discards messages, GC only sweeps the memory local to that queue process.
3. No Stop-The-World: Because GC runs in isolation on each process, the RabbitMQ node as a whole never experiences a global pause (Stop-The-World pause). Message delivery latency stays stable at sub-millisecond levels even when the server is under heavy memory load.

Supervision Trees and the Let It Crash Philosophy #

One of the biggest contributions of Erlang’s OTP (Open Telecom Platform) framework is the introduction of the Supervision Tree pattern and the “Let It Crash” design philosophy.

In traditional programming languages, developers write lots of defensive error-handling code using try-catch blocks on every line to prevent the application from dying suddenly. However, in complex distributed systems, it’s very hard to predict every type of failure.

Erlang takes the opposite approach:

• Let It Crash: If a process hits an invalid state (e.g., parsing a corrupted message payload or a database timeout), the process is allowed to die instantly.
• Failure Isolation: Because every process is strictly isolated, the death of a particular channel or connection process never affects the stability of other client connections.
• Supervision Tree: Every worker process is monitored by a Supervisor process. If a worker dies, the supervisor detects it and performs recovery based on a defined strategy (e.g., restarting the worker process with clean state from the Mnesia database).

flowchart TD
    Main["Node Main Supervisor"] --> ConnSup["Conn Supervisor"]
    Main --> QueueSup["Queue Supervisor"]
    ConnSup --> ConnProc["Connection Proc<br/>'(Crash & Restart)'"]
    QueueSup --> QueueProc["Queue Proc (Billing)<br/>'(Stays Active & Stable)'"]

If a temporary network disruption corrupts a TCP connection state, that connection process dies, is cleaned up by the supervisor, and the consumer is disconnected cleanly. When the consumer tries to reconnect, a new process is created without any leftover stale state.

Erlang Cookie and Cross-Node Distribution Communication #

RabbitMQ’s ability to form multi-node clusters natively is based on the Erlang VM’s built-in distribution feature. Erlang nodes can communicate and send messages to each other across networks without needing additional libraries.

1. Erlang Distribution Protocol #

Cluster nodes communicate using Erlang’s binary protocol over an internal TCP port (default port 25672). This protocol is used to synchronize Mnesia data and replicate Quorum Queue messages.

2. Security with the Erlang Cookie #

To ensure only authorized nodes can join a cluster, Erlang uses a secret token-based authentication system called the Erlang Cookie.

• This cookie is a simple text string usually stored in the /var/lib/rabbitmq/.erlang.cookie file on Linux systems.
• Every node in the cluster must have an identical cookie file content, character for character. If one node has a different cookie, the Erlang VM rejects the connection handshake, and the cluster fails to form.

3. Network Ports That Must Be Opened #

When configuring firewalls or cluster network security systems, we must open several important ports used by the Erlang VM:

• 4369 (epmd - Erlang Port Mapper Daemon): Erlang’s internal DNS service for mapping node names to physical ports.
• 25672: The inter-node communication port for RabbitMQ.
• 15672: The HTTP API and Management dashboard port.
• 5672: The AMQP client port.

Single-Node Throughput Limits Due to Single-Thread Queue Limits #

Even though the Erlang VM is very efficient, we must be aware of one important architectural limitation inherent in how RabbitMQ designs its queues.

[!IMPORTANT] One classic queue or one Quorum Queue instance is internally run by a single Erlang process.

In the Erlang BEAM runtime, one Erlang process cannot be split across multiple CPU cores; it can only be executed by one physical CPU core at a time. The operational implications are:

• If we send messages at very high speed into one single queue on a high-end server with 64 CPU cores, that queue’s performance remains bottlenecked when the CPU core managing the queue process reaches 100% usage. The other 63 CPU cores sit idle.
• Design Solution: To use all server CPU cores optimally, we must design our system to use multiple queues, or leverage the Consistent Hash Exchange plugin to spread messages evenly across dozens of different queues behind the scenes for optimal horizontal scalability.

Scheduling Management (BEAM Scheduler) and Preemptive Scheduling #

To understand how the Erlang BEAM VM distributes workloads fairly across processors, we must study its internal scheduling system (BEAM Scheduler). On startup, the BEAM VM automatically detects the number of physical CPU cores on our server, then creates exactly one Scheduler Thread for each physical CPU core.

Unlike other runtimes that rely on cooperative scheduling — where a running process must voluntarily hand CPU control back to the system — Erlang uses Preemptive/Reduction-Based Scheduling.

This scheduling mechanism works as follows:

1. The Reductions Concept: Every small work operation in Erlang (such as a function call, socket write, or pattern match) is assigned a cost value called a Reduction. One reduction equals one small unit of CPU work.
2. Work Quota Limit: Every Erlang process selected to run on a scheduler is given a maximum quota of 2,000 Reductions.
3. Forced Preemption: Once the process exhausts its 2,000 reduction work units, the BEAM VM forcibly yields the process, puts it back in the ready run queue, and immediately runs the next queued process.

In RabbitMQ operational context, preemptive scheduling guarantees fairness in computing power distribution:

• Even if one queue is processing massive volumes of data (e.g., CPU-intensive binary data compression), that queue can never monopolize a CPU core.
• Other client connections responsible for sending Heartbeat signals or small messages still get periodic CPU execution time, preventing false disconnections caused by CPU starvation.

Erlang Process Mailbox Architecture and Mailbox Bloat Risk #

Every Erlang process in the RabbitMQ runtime acts like an independent actor that communicates purely through asynchronous message passing. Every process has its own internal memory queue called a Mailbox.

flowchart LR
    P1["Sender Process A"] -->|"Send Message"| MB["Mailbox (Local RAM)"]
    P2["Sender Process B"] -->|"Send Message"| MB
    subgraph TargetProcess["Receiving Erlang Process"]
        MB -->|"Pattern Matching"| Engine["Work Engine"]
    end

When a sender process (e.g., a Connection Process) wants to send data to a target process (e.g., a Queue Process):

1. The message is copied into the target process’s Mailbox local RAM.
2. The target process reads messages one by one from the Mailbox in order using Pattern Matching.
3. Successfully matched and processed messages are immediately deleted from Mailbox memory.

The Mailbox Bloat Threat in Production #

Even though this Mailbox architecture is very clean by design, there is one serious performance risk known as Mailbox Bloat.

• If messages enter a process’s Mailbox far faster than the process’s processing ability, or if the process is held back by slow disk I/O operations, the Mailbox keeps accumulating unprocessed messages.
• Because the Mailbox lives entirely in RAM memory, this bloat causes the RabbitMQ node’s RAM usage to increase exponentially in a short time.
• Unlike RabbitMQ queues, which have a feature to move messages to disk (paging to disk) when memory is full, Erlang Mailboxes have no paging-to-disk feature. If a Mailbox bloats past physical RAM limits, the operating system triggers the OOM Killer to forcibly kill the RabbitMQ process.
• Common Cause: Excessive Channel Multiplexing on a single TCP connection, where one Erlang Connection process must channel data traffic from thousands of concurrent channels simultaneously, overwhelming the process while reading data.

Anti-Pattern vs Solution: Channel Leakage from Unclosed Local Threads #

Let’s study one of the most common programming mistakes that triggers RabbitMQ server operational failures from Erlang VM memory exhaustion.

Anti-Pattern Case: Channel Process Leakage #

In the AMQP protocol, a Channel is used to split the main TCP connection into several logical lanes. However, some developers dynamically create a new channel every time they want to publish a message, but forget to close the channel after the message is sent.

// ANTI-PATTERN: Creating channels without closing them triggers Erlang process leaks
func PublishBad(conn *amqp.Connection, message []byte) {
    // ✗ AVOID: Creating new channels repeatedly without closing them
    ch, _ := conn.Channel()
    
    _ = ch.Publish("my-exchange", "routing-key", false, false, amqp.Publishing{
        ContentType: "text/plain",
        Body:        message,
    })
    
    // Problem: The Erlang process for this channel stays alive in the RabbitMQ server's memory.
    // If this function is called 10,000 times, 10,000 idle Erlang processes accumulate.
    // The RabbitMQ server slowly slows down and eventually crashes from OOM.
}

Practical Solution: Using Automatic Closing or Reusing Channels #

We must always ensure every channel is safely closed using a resource cleanup block (such as defer in Go, try-with-resources in Java, or a finally block in Node.js), or design the application to reuse the same channel for repeated sends.

// CORRECT: Guaranteeing the channel is closed after use
func PublishGood(conn *amqp.Connection, message []byte) {
    ch, err := conn.Channel()
    if err != nil {
        log.Printf("Gagal membuka channel: %v", err)
        return
    }
    // ✓ SOLUTION: Always use defer to guarantee channel closure
    defer ch.Close()
    
    _ = ch.Publish("my-exchange", "routing-key", false, false, amqp.Publishing{
        ContentType: "text/plain",
        Body:        message,
    })
}

Summary #

• Lightweight Processes — Erlang’s lightweight process design (2-4KB per process) lets RabbitMQ manage hundreds of thousands of concurrent connections without operating system thread exhaustion.
• Per-Process Garbage Collection — A local memory cleanup mechanism on each Erlang process that eliminates global pause latency (Stop-The-World GC).
• Let It Crash Philosophy — Erlang’s error-handling philosophy relying on automatic recovery supervision rather than writing complex defensive try-catch code.
• Single-Core Queue Limit — One RabbitMQ queue process is bound to one physical CPU core, requiring us to use many queues for optimal horizontal scalability.

← Previous: Single vs Cluster   Next: Metadata & State →