Replay Capability #

In event-driven architectures, one need that often arises is the ability to reprocess messages that were delivered in the past. This scenario is generally triggered by various operational needs, such as massive disaster recovery failures on consumer application databases requiring us to recover data from a certain time, the need to retrain machine learning models using historical datasets, or when we want to add a new microservice that must build its own internal state from transaction history since the beginning of time.

This ability to rewind message reading time is known in the industry as Event Replay or Time-Travel Reading. This is where the very contrasting difference lies between RabbitMQ’s dynamic queue philosophy and Apache Kafka’s immutable log. While one platform is designed to forget the past for present efficiency, the other platform actually documents every event immutably as the main source of truth. This article deeply dissects the replay capability comparison between Kafka and RabbitMQ.

Replay Mechanisms in Apache Kafka (Native Log Retention) #

Apache Kafka provides Event Replay capabilities natively at its core architecture level. Because Kafka treats every topic partition as an Append-Only Immutable Log stored in sequential disks, data isn’t deleted after being read by consumers.

Every message in a partition is identified with a unique, sequential linear index number called an Offset. When consumer applications read data, consumers only need to move their read offset pointer.

[ Log Start ] ──────────────────────────────────────────────> [ Log End ]
 Offset 0     Offset 1     Offset 2     Offset 3     Offset 4
┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
│ Message A │ │ Message B │ │ Message C │ │ Message D │ │ Message E │
└──────────┘ └──────────┘ └──────────┘ └──────────┘ └──────────┘
                                           ▲
                                           │ Current Offset: 3 (Message D)
                                           │
                        [ REWIND OFFSET ] ─┘ Move Back to Offset 1 (Message B)

To replay data processing into the past, we don’t need to modify messages on the broker or ask producers to resend data. Consumers only need to send an instruction to the Kafka broker to move their read offset position backward (offset rewind).

Kafka supports several methods for determining replay read starting points:

• earliest: Forces consumers to rewind reads to offset 0 (the earliest message still stored in disk retention).
• latest: Ignores all historical data and only listens to new messages arriving after the consumer becomes active.
• Manual Offset Seek: Precisely moves consumer offsets to a specific index number (e.g., jumping directly to offset 15000 to reprocess data from that point).
• Timestamp-Based Seek: Asks the broker to find the smallest offset written at a specific time (e.g., “find the offset recorded exactly on June 9, 2026 at 09.00”). This is very practical for recovering database failures that occurred since a specific time.

Kafka Log Retention Policies #

Although Kafka logs are immutable, we can’t store all the world’s transaction data forever on one disk server without limits. Kafka manages disk storage capacity using Retention Policies flexibly configurable per topic:

1. Time-based Retention #

Uses the log.retention.hours or log.retention.ms parameters. This policy determines how long messages may settle on disk before being automatically deleted. Example: if set to 7 days (default), transaction data 8 days old is gradually deleted by the broker from the oldest log segments, regardless of whether all consumers have read that data or not.

2. Size-based Retention #

Uses the log.retention.bytes parameter. This policy limits the maximum partition log file size to a certain capacity (e.g., 50 GB). If the file size exceeds that limit due to dense producer data flows, Kafka deletes the oldest log segments to keep disk capacity safe.

3. Log Compaction #

Uses the log.cleanup.policy = compact policy. This is an advanced feature where Kafka scans log files and only retains messages with the latest value payload for each unique partition key, deleting older history versions.

This pattern is ideal for replicating database statuses (Change Data Capture / CDC), where we only care about the most current balance or user profile status, not the mutation history every second.

Replay Limitations in RabbitMQ #

RabbitMQ’s traditional queue model, which adheres to the destructive read principle, deletes messages from RAM and disk as soon as consumers send ACKs. This makes regular RabbitMQ unable to natively support Event Replay.

If we need failed data reprocessing in RabbitMQ, we must choose one of the following alternative solutions:

1. Republishing from External Databases (Manual Tactic) #

Because RabbitMQ doesn’t store message history, developer teams must design applications to write every message payload published by producers into a relational database (like PostgreSQL) or NoSQL (like MongoDB) first as an audit log.

If a database disaster occurs on consumers and we need to replay data from 3 hours ago, we must write special migration scripts to read log rows from that external database and manually re-publish them to the RabbitMQ Exchange. This tactic takes time, is error-prone, and adds I/O load to our log database.

2. Leveraging RabbitMQ Streams (Modern Native Tactic) #

To cover this fatal weakness and compete with Kafka’s advantages, RabbitMQ since version 3.9 introduced a new queue type called RabbitMQ Streams.

Streams is an implementation of an immutable append-only log running inside the RabbitMQ ecosystem. Unlike Classic or Quorum queues, messages in Streams-type queues aren’t deleted after consumption. Consumer ACK status is ignored by the broker for physical deletion. Messages are stored sequentially on disk and limited only by time-based or size-based retention.

flowchart LR
    Consumer["Consumer"] -->|"Read from Offset X / Timestamp Y"| Streams["RabbitMQ Streams (Append-Only Log)"]

Consumers connected to RabbitMQ Streams can use a special protocol (using port 5552 instead of the standard AMQP port 5672) to determine initial read offset positions, rewind offsets, or read from specific timestamps natively similar to Kafka. This provides a solution for teams needing replay features but wanting to keep a single broker technology (RabbitMQ) in their infrastructure.

Replay Capability Comparison Table #

Here is a comparative matrix table comparing replay capabilities in Kafka, regular RabbitMQ (Classic/Quorum), and RabbitMQ Streams:

Go Code Implementation (Golang) #

To show the technical difference in replay handling, here are consumer implementation examples in Go for Apache Kafka (using kafka-go with offset rewind settings to the log start) and RabbitMQ Streams consumers (using the official rabbitmq/stream-go-client driver).

1. Kafka Consumer (Replaying Data from the Beginning / Earliest Offset) #

In Kafka, we set StartOffset to kafka.FirstOffset to replay all historical data from the start of the stored partition log.

package main

import (
	"context"
	"log"
	"os"
	"os/signal"
	"syscall"

	"github.com/segmentio/kafka-go"
)

func main() {
	// Configure the Kafka reader with replay-from-start settings
	r := kafka.NewReader(kafka.ReaderConfig{
		Brokers:     []string{"localhost:9092"},
		Topic:       "audit-events",
		GroupID:     "audit-replay-group-v1", // New group name to trigger a reread
		StartOffset: kafka.FirstOffset,       // REWIND FROM THE EARLIEST EVENT (Offset 0)
	})
	defer r.Close()

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	log.Println("[KAFKA] Mulai memproses ulang data audit dari awal log...")

	go func() {
		for {
			msg, err := r.ReadMessage(ctx)
			if err != nil {
				log.Printf("Gagal membaca event: %v", err)
				break
			}
			// Reprocess historical data
			log.Printf("[REPLAY] Offset: %d | Payload: %s", msg.Offset, string(msg.Value))
		}
	}()

	sigChan := make(chan os.Signal, 1)
	signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
	<-sigChan
}

2. RabbitMQ Streams Consumer (Replaying Data from the Beginning via the Stream Protocol) #

In RabbitMQ Streams, we can set the initial read offset using the OffsetSpecification type to the First value to trigger reading from the earliest event natively.

package main

import (
	"log"
	"os"
	"os/signal"
	"syscall"

	"github.com/rabbitmq/stream-go-client/pkg/amqp"
	"github.com/rabbitmq/stream-go-client/pkg/stream"
)

func main() {
	// 1. Open a connection to the RabbitMQ Stream port (default port 5552)
	env, err := stream.NewEnvironment(
		stream.NewEnvironmentOptions().
			SetHost("localhost").
			SetPort(5552).
			SetUser("guest").
			SetPassword("guest"),
	)
	if err != nil {
		log.Fatalf("Gagal koneksi ke RabbitMQ Stream port: %v", err)
	}
	defer env.Close()

	// 2. Register a consumer on the stream-type queue
	// We set the OffsetSpecification to First() to trigger a data replay
	handle, err := env.NewConsumer(
		"audit.stream.queue",
		func(ctx stream.ConsumerContext, message *amqp.Message) {
			// Historical message reprocessing callback
			log.Printf("[STREAM REPLAY] Offset: %d | Payload: %s", 
				ctx.Consumer.GetLastConsumedOffset(), string(message.GetData()))
		},
		stream.NewConsumerOptions().
			SetOffset(stream.OffsetSpecification{}.First()), // REPLAY FROM THE START
	)
	if err != nil {
		log.Fatalf("Gagal mendaftarkan stream consumer: %v", err)
	}
	defer handle.Close()

	log.Println("[RABBITMQ STREAM] Mulai membaca ulang stream audit dari awal...")

	sigChan := make(chan os.Signal, 1)
	signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
	<-sigChan
}

Anti-Patterns vs Practical Solutions #

Trying to simulate replay capabilities on the wrong queue type can cause cluster paralysis. Here is the most commonly encountered anti-pattern:

Anti-Pattern: Piling Messages in Classic/Quorum RabbitMQ Queues for Future Replay Needs #

Designing systems where producers send historical event messages to Classic or Quorum RabbitMQ queues deliberately without active consumers, intending to hoard that data in broker memory so that someday (e.g., a few weeks later) it can be consumed by new services needing historical data.

Why is this wrong? #

This is a fatal misuse of RabbitMQ queue functions. Classic and Quorum queues are optimally designed to handle short-term queues that are quickly emptied.

Hoarding millions of messages in active queues without consumption will:

1. Exhaust Broker RAM: The Erlang broker must maintain index metadata, search status, and message payloads in RAM. When the memory watermark threshold is exceeded, the broker freezes (blocked state) and stops all producer publishing activity.
2. I/O Performance Degradation: When RAM memory is full, the RabbitMQ broker is forced to continuously page (write transient messages to disk and delete them from RAM). Later data reads run very slowly because the broker must call disk I/O randomly (random reads) to read data from fragmented segment files.

Practical Solution #

If our system has an absolute need to store raw event data for months and requires free offset rewind capabilities, use Apache Kafka as the main data pipeline platform. If our infrastructure is limited to only one RabbitMQ broker technology, use the special RabbitMQ Streams queue type natively optimized for low-latency immutable linear disk storage, not regular Classic or Quorum queues.

Summary #

• Replay Philosophy — Event Replay is the ability to rewind historical message reading time. Kafka natively supports this feature, while regular RabbitMQ deletes messages after ACKs.
• Kafka Offset Mechanisms — Kafka tracks read positions using linear Offset indexes. Consumers can replay by asking the broker to rewind their read offsets (offset rewind) to the start (earliest) or any specific point freely.
• Kafka Retention Policies — Kafka logs are gradually deleted based on data age (time) or storage capacity (bytes), not consumer ACK status.
• Log Compaction Features — Kafka can compact logs to retain only the latest status payload per unique partition key, very useful for state replication.
• RabbitMQ Streams Alternatives — Since version 3.9, RabbitMQ provides the Streams queue type adopting the immutable append-only log model and stream protocol (port 5552) to support asynchronous replay.
• Message Pile-Up Dangers — Never hoard millions of consumer-less messages in Classic/Quorum RabbitMQ queues for replay purposes because it triggers memory watermark alarms that freeze the broker. Use Kafka or RabbitMQ Streams.

← Previous: Ordering Behavior   Next: Throughput & Scalability →