Message Size #

When designing microservices architecture applications communicating with each other using RabbitMQ, one important aspect often overlooked is the size of the message payload being sent. On paper, RabbitMQ indeed has extraordinary flexibility to accept messages of varying sizes. The default limit set by the broker can even reach 128 MB per message. However, this technical capability is often misinterpreted as design approval. Many developer teams are tempted to use RabbitMQ as a large file transfer medium, like attaching PDF transaction document files, user profile images, binary log data, or exporting CSV reports tens of megabytes in size directly into queue payloads.

In the world of messaging systems, RabbitMQ is positioned exclusively as a fast transport layer or transient coordination pipe, not as a blob data storage medium. Sending messages of excessively large sizes is one of the most fatal anti-patterns that can disrupt cluster operational stability. Every extra byte in message payloads carries hidden performance consequences that multiply on the broker side, from ballooning Erlang VM RAM memory consumption, drastic cluster network throughput drops, to increased producer connection locking risks from reaching memory watermark alarms.

Why Do Message Sizes Greatly Impact Broker Performance? #

To understand why RabbitMQ brokers are so sensitive to message sizes, we must analyze how the Erlang BEAM Virtual Machine runtime manages memory allocations for every incoming message:

1. Erlang Memory Allocation Mechanisms (Heap vs Refc Binaries) #

Inside the Erlang BEAM VM, very small messages (<64 bytes) are stored directly in the heap memory of the Erlang process representing that queue. However, for larger messages, Erlang places them in an external memory area called Refc Binaries (Reference-Counted Binaries), which are off-heap. Queue Erlang processes only hold small pointers of a few bytes referencing the original physical memory.

Although this refc binaries mechanism is very efficient because it avoids copying large payloads between Erlang processes, problems appear when those messages pile up in queues (backlogs). The broker must keep maintaining those binary references in RAM. When Garbage Collection (GC) processes run on queue Erlang processes, tracking too many large binary references triggers intensive CPU overhead and slows down BEAM VM process scheduling performance (reduction scheduling latency).

2. Performance-Killing Disk Paging (Disk Paging Swap) #

RabbitMQ uses a dynamic memory management system. If total broker RAM usage exceeds the default threshold (usually 40% of the server’s total physical RAM, known as vm_memory_high_watermark), RabbitMQ panics and activates the Disk Paging mechanism.

In this condition, the broker force-moves active messages from RAM to disk storage media to free memory space. If queues are filled with millions of small messages, this paging process runs relatively stably. However, if queues hold large messages (e.g., 10 MB per message), disk I/O write processes immediately experience disk I/O saturation. During paging processes, the broker refuses to accept new messages from producers by blocking TCP connections (connection blocking), resulting in producer system paralysis.

3. Cluster Replication Costs (Quorum Queue Replication Overhead) #

If we use Quorum Queues, every message sent by producers must be replicated through the local cluster network to Follower nodes before being confirmed. If we send a 5 MB payload on a 3-node cluster, then:

• The producer sends 5 MB to the Leader.
• The Leader sends 5 MB to Follower A.
• The Leader sends 5 MB to Follower B.
• The Leader writes 5 MB to its local WAL disk.
• Followers A and B write 5 MB to their respective WAL disks.

The total internal network traffic created for that one message is 15 MB, and the total cluster disk write is 15 MB. If our system throughput is 100 messages per second, the internal cluster network bandwidth load immediately spikes to 1.5 GB/s, a number that easily cripples standard gigabit network cards and triggers Raft election timeouts.

Recommended Payload Sizes in Production Environments #

Based on large-scale performance testing in the industry, here are healthy message size references for maintaining RabbitMQ cluster stability:

• Optimal Size (Highly Recommended): Under 100 KB (ideal in the 1 KB to 10 KB range). At this size range, RabbitMQ can process messages entirely in RAM memory with sub-millisecond latency and throughput up to tens of thousands of messages per second.
• Maximum Safe Size: 1 MB. Messages above 1 MB must be considered design exceptions and require strict infrastructure capacity supervision.
• Danger Zone: Above 5 MB. Constantly sending messages above this size almost certainly triggers disk paging, confirm latency spikes, and cluster instability during workload spikes.

Design Solution: The Claim Check Pattern (Message Pointer Pattern) #

To handle business scenarios where we must process large data (like monthly CSV report processing, medical image processing, or PDF file digital signatures), we’re strictly forbidden from attaching that physical data into messages. The best architecture solution for this problem is applying the Claim Check Pattern (also known as the Reference Message Pattern).

The working concept of the Claim Check Pattern is very simple yet elegant:

1. Data Separation: Instead of sending large binary data to RabbitMQ, producer applications first upload that large data to cheap, efficient external storage media for blob data (like AWS S3, Google Cloud Storage, MinIO, or cold storage databases).
2. Send Pointers: After successful uploads, producers get a location reference in the form of a unique URI or data ID (pointer). Producers then create a lightweight JSON message containing only that pointer along with minimal metadata (e.g., user IDs or operation types) and publish it to RabbitMQ.
3. Claim Data: Consumer applications receive lightweight pointer messages from RabbitMQ queues, use the URI or ID inside message payloads to download the original large data from Object Storage, execute business logic, and after finishing, send ACKs to the broker.

flowchart TD
    Producer["Producer Application"] -->|"1. Upload Large Payload"| ObjectStorage[("S3 / MinIO Object Storage")]
    ObjectStorage -->|"2. Return Pointer URI (s3://bucket/file.pdf)"| Producer
    
    Producer -->|"3. Send Lightweight Pointer Message"| RabbitMQ(("RabbitMQ Broker"))
    RabbitMQ -->|"4. Deliver Pointer Message"| Consumer["Consumer Application"]
    
    Consumer -->|"5. Download Large Payload via Pointer"| ObjectStorage
    ObjectStorage -->|"6. Return Original File"| Consumer
    
    Consumer -->|"7. Execute Business & Send ACK"| RabbitMQ

    style ObjectStorage stroke:#0288d1,stroke-width:2px
    style RabbitMQ stroke:#7b1fa2,stroke-width:2px

With this pattern, RabbitMQ only processes small JSON messages a few hundred bytes in size. The broker stays super fast, RAM memory stable, cluster replication instant, and Object Storage handles large file transfer loads with the bandwidth capacity it was indeed designed for.

Claim Check Pattern Implementations in Go and S3 #

Here is a complete Claim Check Pattern implementation using the Go programming language. The scenario is processing large PDF reports. We use the AWS SDK library to simulate uploads to S3 Object Storage, and the Go AMQP library to flow lightweight pointer messages:

1. Producer Implementations (Publishers) #

Producers are tasked with uploading large data to storage, designing pointer payloads, and publishing them:

package main

import (
	"context"
	"encoding/json"
	"fmt"
	"log"
	"math/rand"
	"time"

	amqp "github.com/rabbitmq/amqp091-go"
)

const (
	amqpURI      = "amqp://admin:***@rabbitmq-cluster:5672/"
	exchangeName = "exchange.report.direct"
	routingKey   = "report.generate"
)

// ReportGeneratedEvent represents the lightweight pointer message (Claim Check)
type ReportGeneratedEvent struct {
	EventID     string    `json:"event_id"`
	GeneratedAt time.Time `json:"generated_at"`
	ReportID    string    `json:"report_id"`
	StorageURI  string    `json:"storage_uri"` // Pointer referencing Object Storage
	FileSize    int64     `json:"file_size"`
}

// SimulateS3Upload simulates the process of uploading a 15MB PDF report file to AWS S3
func SimulateS3Upload(reportID string) (string, error) {
	log.Printf("[S3] Mengunggah berkas PDF laporan bulanan (15 MB) untuk ID: %s...\n", reportID)
	// Simulate the network I/O latency of uploading files to cloud storage
	time.Sleep(250 * time.Millisecond)
	
	s3URI := fmt.Sprintf("s3://produksi-laporan-bucket/bulanan/pdf/%s.pdf", reportID)
	log.Printf("[S3] Unggahan sukses. Lokasi penyimpanan: %s\n", s3URI)
	return s3URI, nil
}

func main() {
	// Connect to the RabbitMQ broker
	conn, err := amqp.Dial(amqpURI)
	if err != nil {
		log.Fatalf("Gagal terhubung ke RabbitMQ: %v\n", err)
	}
	defer conn.Close()

	ch, err := conn.Channel()
	if err != nil {
		log.Fatalf("Gagal membuka channel: %v\n", err)
	}
	defer ch.Close()

	// Make sure the publisher confirms mode is active for reliability
	err = ch.Confirm(false)
	if err != nil {
		log.Fatalf("Gagal mengaktifkan publisher confirms: %v\n", err)
	}

	confirms := ch.NotifyPublish(make(chan amqp.Confirmation, 1))

	// 1. Simulate creating large data and uploading to S3
	reportID := fmt.Sprintf("REP-%d", rand.Intn(1000000))
	storageURI, err := SimulateS3Upload(reportID)
	if err != nil {
		log.Fatalf("Gagal mengunggah file ke S3: %v\n", err)
	}

	// 2. Create a lightweight event payload only containing the location pointer
	event := ReportGeneratedEvent{
		EventID:     fmt.Sprintf("EVT-%d", rand.Intn(1000000)),
		GeneratedAt: time.Now().UTC(),
		ReportID:    reportID,
		StorageURI:  storageURI, // Sending a pointer, not the original PDF data bytes
		FileSize:    15728640,  // 15 MB
	}

	body, err := json.Marshal(event)
	if err != nil {
		log.Fatalf("Gagal melakukan serialisasi JSON: %v\n", err)
	}

	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
	defer cancel()

	// 3. Publish the lightweight pointer message to RabbitMQ
	log.Printf("[PRODUSEN] Mempublikasikan pesan pointer ringan (Ukuran: %d bytes)...\n", len(body))
	err = ch.PublishWithContext(ctx,
		exchangeName,
		routingKey,
		true, // mandatory
		false,
		amqp.Publishing{
			ContentType:  "application/json",
			DeliveryMode: amqp.Persistent, // Make sure the message is safely written to disk
			Body:         body,
			CorrelationId: reportID,
		},
	)
	if err != nil {
		log.Fatalf("Gagal mempublikasikan pesan: %v\n", err)
	}

	// Wait for the receipt confirmation from the broker
	ack := <-confirms
	if ack.Ack {
		log.Println("[PRODUSEN] Berhasil menerima konfirmasi ACK dari broker.")
	} else {
		log.Println("[PRODUSEN] Menerima sinyal NACK. Pesan gagal masuk antrean.")
	}
}

2. Consumer Implementations #

Consumers receive lightweight pointer messages, download the original large files from S3, and process them:

package main

import (
	"context"
	"encoding/json"
	"log"
	"os"
	"os/signal"
	"syscall"
	"time"

	amqp "github.com/rabbitmq/amqp091-go"
)

const (
	amqpURI   = "amqp://admin:***@rabbitmq-cluster:5672/"
	queueName = "queue.report.process"
)

// ReportGeneratedEvent represents the received pointer message structure
type ReportGeneratedEvent struct {
	EventID     string    `json:"event_id"`
	GeneratedAt time.Time `json:"generated_at"`
	ReportID    string    `json:"report_id"`
	StorageURI  string    `json:"storage_uri"`
	FileSize    int64     `json:"file_size"`
}

// DownloadFromS3 simulates downloading the 15MB PDF report file based on the pointer URI
func DownloadFromS3(storageURI string) ([]byte, error) {
	log.Printf("[S3] Mendownload berkas asli dari lokasi: %s...\n", storageURI)
	// Simulate the I/O latency of downloading large files over the internet
	time.Sleep(300 * time.Millisecond)
	
	log.Println("[S3] Download selesai. Berkas siap diproses.")
	return []byte("%PDF-1.4 ... isi dokumen laporan palsu ..."), nil
}

func main() {
	conn, err := amqp.Dial(amqpURI)
	if err != nil {
		log.Fatalf("Koneksi gagal: %v\n", err)
	}
	defer conn.Close()

	ch, err := conn.Channel()
	if err != nil {
		log.Fatalf("Gagal membuka channel: %v\n", err)
	}
	defer ch.Close()

	// Limit the prefetch limit so workers aren't overloaded with excessive parallel consumption
	err = ch.Qos(5, 0, false)
	if err != nil {
		log.Fatalf("Gagal menetapkan QoS: %v\n", err)
	}

	deliveries, err := ch.Consume(
		queueName,
		"report-processor-worker",
		false, // manual ACK required
		false,
		false,
		false,
		nil,
	)
	if err != nil {
		log.Fatalf("Gagal mendaftarkan consumer: %v\n", err)
	}

	log.Println("Worker laporan aktif. Menunggu pesan masuk...")

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	stop := make(chan os.Signal, 1)
	signal.Notify(stop, os.Interrupt, syscall.SIGTERM)

	go func() {
		for msg := range deliveries {
			var event ReportGeneratedEvent
			err := json.Unmarshal(msg.Body, &event)
			if err != nil {
				log.Printf("Gagal parsing JSON payload: %v. Message rejected.\n", err)
				msg.Nack(false, false) // Reject without requeue so it enters the DLQ
				continue
			}

			log.Printf("[KONSUMEN] Menerima event pointer '%s' untuk Report ID '%s'\n", event.EventID, event.ReportID)

			// 1. Claim the original large data from S3 using the pointer URI
			fileBytes, err := DownloadFromS3(event.StorageURI)
			if err != nil {
				log.Printf("[GAGAL] Gagal mendownload data dari S3: %v. Requeuing...\n", err)
				msg.Nack(false, true) // Requeue the pointer message to retry
				continue
			}

			// 2. Execute business logic (e.g., conversion, rendering, or PDF OCR processes)
			processReport(event.ReportID, fileBytes)

			// 3. Send the success ACK to the broker
			msg.Ack(false)
			log.Printf("[SUKSES] Pesan dengan tag %d berhasil diselesaikan.\n", msg.DeliveryTag)
		}
	}()

	<-stop
	log.Println("Menerima sinyal shutdown. Menghentikan worker...")
	cancel()
	time.Sleep(1 * time.Second)
}

func processReport(reportID string, data []byte) {
	log.Printf("[PROSES] Memproses analisis dokumen '%s' (Ukuran: %d bytes)...\n", reportID, len(data))
	// Simulate CPU processing
	time.Sleep(100 * time.Millisecond)
}

Comparison: Large Payloads vs Lightweight Payloads (Claim Check) #

Here is an in-depth comparison matrix for understanding the operational impact differences on brokers when we directly publish large messages vs using pointer message patterns:

Summary #

• Transport Layer Roles — Treat RabbitMQ purely as a fast, lightweight asynchronous intermediary pipe. Avoid storing or transferring large binary data directly through brokers.
• Limit Message Sizes — Strive to keep production message payload sizes under 100 KB, and set a maximum tolerance limit of 1 MB.
• Apply the Claim Check Pattern — Upload large data (PDFs, images, CSV files) to Object Storage first, then flow asynchronous messages only containing those storage location pointer URIs.
• Optimize Storage Bandwidth — Let Object Storage infrastructure (like AWS S3 or Google Cloud Storage) carry large file transfer traffic loads, keeping RabbitMQ clean and stable.

← Previous: Use Quorum   Next: Monitoring →