Throughput & Scalability #

In advanced microservices architecture design, messaging system performance can’t be assessed only from raw benchmark numbers scattered across the internet. Every messaging platform is designed with a series of deliberate architectural trade-offs chosen to optimize certain performance dimensions. Apache Kafka is deliberately optimized to process giant data volumes with extreme throughput, while RabbitMQ is designed to provide high routing flexibility with consistent low-latency guarantees for individual messages.

Understanding how both platforms manage horizontal scalability and throughput performance is essential to prevent infrastructure bottlenecks when data traffic loads spike sharply. This article deeply dissects the scalability model comparison and the internal mechanisms enabling Kafka to achieve millions of messages per second throughput, compared with RabbitMQ’s performance characteristics.

Horizontal Scalability Models #

Horizontal scalability defines how easily and efficiently a system increases data processing capacity by adding more server nodes (brokers) or consumer application instances to a cluster.

1. RabbitMQ’s Scaling Model #

RabbitMQ offers very dynamic and flexible consumer scaling. On the queue side, we can connect any number of parallel consumers at any time (Competing Consumers Pattern).

However, at the broker level, RabbitMQ’s horizontal scaling has an upper limit influenced by state tracking overhead. Because the RabbitMQ broker must record transactional ACK/NACK status for every individual message in RAM memory, adding more brokers to a cluster doesn’t linearly increase single-queue performance.

To scale RabbitMQ queues, we must:

• Logically divide queues into several separate queues (queue sharding).
• Use Quorum Queues for Raft consensus data replication, which unfortunately adds network and disk I/O overhead between leader and follower nodes.

2. Kafka’s Scaling Model #

Kafka implements a very structured and linear horizontal scaling model through topic partitions (Partition Scaling). Scalability in Kafka is strictly managed at the partition design level from the start.

On the broker side, adding new nodes to a Kafka cluster directly increases storage capacity and throughput. Kafka can evenly distribute partitions from one topic across all those new brokers.

On the consumer side, the maximum group concurrency limit is hard-bounded by the number of topic partitions. Adding consumers beyond the partition count on the same topic within one consumer group doesn’t help, because those extra consumers just sit idle without getting data partitions to read. Consumer scaling in Kafka demands mature topic partition planning at the initial design stage.

Extreme Throughput Mechanisms in Apache Kafka #

Apache Kafka is specifically designed to process petabyte-scale data with extraordinary efficiency. Kafka’s ability to process millions of events per second on standard hardware is supported by the following three internal technology pillars:

1. Sequential Disk I/O #

Many developers assume writing data to storage disks (hard drives/SSDs) is always slow compared to writing to RAM. This assumption isn’t entirely true. Writing data to disk feels very slow if we perform random disk access, because the hard drive’s physical head must move to find scattered disk sectors.

Kafka discards random access by writing event logs Sequentially (Append-Only). New data is always added at the end of the partition log file. Sequential data writing speeds on modern disks are actually very fast, almost equivalent to RAM memory write speeds, because there’s no disk head movement overhead.

2. OS Pagecache Optimization #

Kafka minimizes direct application interaction with physical disks. When producers send data, the Kafka broker doesn’t immediately call the fsync command that blocks threads to synchronously write data to disk platters.

The Kafka broker writes messages to the operating system’s page memory called the Pagecache. The Linux OS asynchronously performs flushing processes of data from pagecache to physical disks in background threads.

When consumers request data, if the data was just written by producers, that data is likely still warm in the RAM pagecache. The Kafka broker can serve that data directly from OS RAM to consumers without triggering any physical disk read operations (zero disk read overhead).

3. Zero-Copy Transfer Mechanisms (sendfile) #

In conventional web architectures (including RabbitMQ), when message data is read from disk to be sent to network socket buffers, data must pass through several memory copy cycles and CPU context switching:

flowchart LR
    Disk["Disk"] --> Pagecache["Kernel Pagecache"]
    Pagecache -->|"Copy"| UserSpace["User Space (JVM/Erlang Heap)"]
    UserSpace -->|"Copy"| Socket["Socket Buffer"]
    Socket --> NIC["NIC Buffer"]

This process is very CPU-wasteful because the same data is copied multiple times across kernel and user space memory regions.

Kafka eliminates this overhead using a Linux kernel system call named sendfile, which activates the Zero-Copy technology. With zero-copy, the Kafka broker instructs the operating system to directly copy data bytes from the kernel pagecache straight to the network card buffer (NIC Buffer) without loading that data into JVM application space memory at all:

flowchart LR
    Disk["Disk"] --> Pagecache["Kernel Pagecache"]
    Pagecache -->|"Direct Copy via DMA/sendfile"| NIC["NIC Buffer (Network Card)"]

This zero-copy mechanism reduces CPU context switching from 4 times to 2 times, and eliminates data copying in application memory, drastically saving broker CPU and RAM consumption when serving millions of data read requests.

RabbitMQ Performance and Latency Characteristics #

With all of Kafka’s I/O optimization sophistication above, why does RabbitMQ remain a popular choice? The answer lies in the Latency dimension.

RabbitMQ is designed to prioritize the smallest possible individual message delivery latency. When a message enters RabbitMQ, the broker acting as a Smart Broker immediately evaluates the Mnesia routing table in RAM and pushes that message as soon as possible to actively waiting consumer TCP socket connections.

This delivery process runs instantly for every individual message without waiting for data batch accumulation. As a result, RabbitMQ provides very consistently low queueing latency at the single millisecond level (sub-millisecond latency) for normal data traffic.

Conversely, Kafka relies on Batching (mass message grouping) techniques to achieve high throughput. Kafka producers hold messages in local memory for a few milliseconds (e.g., set linger.ms = 5) to collect 100 or 1,000 messages before sending them collectively to the broker. This sacrifices individual message latency for much larger total system throughput.

Latency & Throughput Comparison Table #

Here is a comparison table of performance and scalability characteristics between RabbitMQ and Kafka:

Go Code Implementation (Golang) #

Here is a simple benchmark program example in Go illustrating the tactical difference between individual low-latency message publishing on RabbitMQ vs high-throughput batching message delivery on Kafka.

1. Instant Message Publishing in RabbitMQ (Low Latency) #

In RabbitMQ, messages are sent instantly one by one to directly trigger processing without accumulation delays.

package main

import (
	"context"
	"log"
	"time"

	amqp "github.com/rabbitmq/amqp091-go"
)

func main() {
	conn, err := amqp.Dial("amqp://guest:***@localhost:5672/")
	if err != nil {
		log.Fatalf("Koneksi RabbitMQ gagal: %v", err)
	}
	defer conn.Close()

	ch, err := conn.Channel()
	if err != nil {
		log.Fatalf("Gagal membuka channel: %v", err)
	}
	defer ch.Close()

	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
	defer cancel()

	start := time.Now()
	totalMessages := 5000

	for i := 0; i < totalMessages; i++ {
		// Every message is published instantly one by one
		err = ch.PublishWithContext(ctx,
			"",                     // default exchange
			"payment.normal.queue", // routing key / queue name
			false,
			false,
			amqp.Publishing{
				ContentType: "text/plain",
				Body:        []byte("Instantly sent message payload"),
			},
		)
		if err != nil {
			log.Printf("Gagal publish ke RabbitMQ: %v", err)
		}
	}

	duration := time.Since(start)
	log.Printf("[RABBITMQ] Selesai mengirim %d pesan dalam %v (Rata-rata %.2f pesan/detik)",
		totalMessages, duration, float64(totalMessages)/duration.Seconds())
}

2. Batch Message Publishing in Kafka (High Throughput) #

In Kafka, producers automatically group messages into one batch request (BatchSize and BatchTimeout) before sending them collectively to the broker to save network I/O.

package main

import (
	"context"
	"log"
	"time"

	"github.com/segmentio/kafka-go"
)

func main() {
	// Configure the Kafka Writer with batching features enabled
	w := &kafka.Writer{
		Addr:         kafka.TCP("localhost:9092"),
		Topic:        "payment-telemetry",
		Balancer:     &kafka.LeastBytes{},
		BatchSize:    1000,            // Send after collecting 1,000 messages
		BatchTimeout: 5 * time.Millisecond, // Or send if 5 ms have passed
	}
	defer w.Close()

	ctx := context.Background()
	start := time.Now()
	totalMessages := 5000

	messages := make([]kafka.Message, totalMessages)
	for i := 0; i < totalMessages; i++ {
		messages[i] = kafka.Message{
			Value: []byte("Batch sent message payload telemetry data FX pricing info"),
		}
	}

	// Write all messages in a mass batch
	err := w.WriteMessages(ctx, messages...)
	if err != nil {
		log.Fatalf("Gagal menulis ke Kafka: %v", err)
	}

	duration := time.Since(start)
	log.Printf("[KAFKA] Selesai mengirim %d pesan (Batching) dalam %v (Rata-rata %.2f pesan/detik)",
		totalMessages, duration, float64(totalMessages)/duration.Seconds())
}

Anti-Patterns vs Practical Solutions #

Randomly designing partitions in Kafka is a common architecture mistake (anti-pattern) that can degrade system stability.

Anti-Pattern: Setting Excessively High Partition Counts (Over-Partitioning) Without a Plan #

Declaring Kafka topics with thousands of partitions (e.g., 5,000 partitions for one topic) on small clusters only assuming “the more partitions, the faster the scalability”.

Why is this wrong? #

Although partitions are Kafka’s scalability unit, excessively high partition counts on small clusters trigger several serious dangers:

1. Metadata File Overhead: Every partition maps to a physical folder directory in the OS containing index files and data log segments. Having thousands of partitions means the broker must open thousands of file handlers simultaneously, triggering operating system limit issues (open files limit errors).
2. Very High Failover Latency: Every partition is controlled by one broker acting as the Leader. If one Kafka broker in the cluster crashes, other brokers must negotiate to elect new Leaders for the hundreds of abandoned partitions. This metadata coordination process (via KRaft or ZooKeeper) takes a long time, freezes data consumption processes, and drastically degrades cluster performance for several minutes.

Practical Solution #

Plan partition counts measurably from the start using the following formula:

$$Partisi = \max\left(\frac{Target_Throughput}{Throughput_Producer}, \frac{Target_Throughput}{Throughput_Consumer}\right)$$

If one of our consumer applications can process 10 MB of data per second, and our system’s target throughput is 40 MB of data per second, then we only need to declare a topic with 4 partitions. This number is considered very optimal for serving consumer horizontal scalability without dirtying broker metadata.

Summary #

• Performance Trade-offs — RabbitMQ is optimized for consistently low latency on individual messages. Kafka is optimized for extreme throughput performance on giant data volumes.
• Kafka Performance Pillars — Kafka achieves millions of messages per second using sequential disk I/O, OS RAM pagecache utilization, and Zero-Copy memory transfers (sendfile).
• RabbitMQ RAM Overhead — The RabbitMQ broker tracks granular per-message ACK/NACK state, so broker RAM memory load is vulnerable to ballooning with large message backlog pile-ups.
• Batching vs Instant — Kafka uses producer/consumer batching to save network bandwidth with the consequence of adding a little latency. RabbitMQ sends messages instantly (push-based) without batching delays.
• Partition-Bound Scaling — Kafka consumer scalability is limited to a maximum of the topic partition count. RabbitMQ is free to add dynamic competing consumers without queue topology restrictions.
• Avoid Over-Partitioning — Don’t randomly declare thousands of topic partitions because it triggers OS open file overhead and slows broker leader failovers during crashes.

← Previous: Replay Capability   Next: RabbitMQ as Database →