One Queue for All Events #

In designing asynchronous architectures, simplicity is often considered the best achievement. To avoid the governance complexity of many queues, some developers take an extreme shortcut: flowing all messages and event types from various business domains into one single monolithic queue (e.g., a queue named queue.app.events). This approach looks very elegant and practical in early development. We only need to monitor one queue, manage one consumer program, and don’t need to worry about complex routing rules on the broker.

However, as systems grow and data traffic variations increase, this monolithic queue quickly transforms into a crippling architectural bottleneck. Combining dissimilar messages into one channel damages one of the queue’s main purposes: acting as a fault isolation boundary. This article deeply discusses the dangers behind the One Queue for All Events anti-pattern, analyzes Head-of-Line Blocking (HOLB) impacts, serialization coupling tight dependencies, and how we should hygienically design queue division based on functional domains and urgency levels (SLA).

Why Does This Pattern Occur and Look Attractive? #

The tendency to centralize all messages into one monolithic queue is usually driven by the following design motivations:

1. Infrastructure Simplification: Operations teams don’t want to be bothered managing many queues on the RabbitMQ Dashboard. They assume one large channel is easier to maintain than ten small queues.
2. Ease of Writing Consumers: Developers just write one single consumer application listening to that queue, using a giant switch-case branching code block to process messages by event type.
3. The Scalability Illusion: Developers assume that if load increases, they can just duplicate that consumer application (competing consumers) horizontally to process that single queue faster.
4. Lack of Workload Analysis: When designing systems, developer teams don’t group messages by technical characteristics (like processing duration, payload size, and delay tolerance).

Although this simplicity eases initial product launches (MVPs), it hides operational complexity that explodes once the system faces real production traffic.

Head-of-Line Blocking (HOLB) Impacts #

The most instant technical danger of using monolithic queues is Head-of-Line Blocking (HOLB). Because RabbitMQ queues naturally follow FIFO (First-In, First-Out) rules, messages at the front of the line must be processed and deleted first before messages behind them can be accessed.

Let’s imagine a system mixing the following two message types into one queue.app.events queue:

• event.user.otp_requested (OTP SMS delivery event — strict SLA: must be sent < 3 seconds).
• event.report.generate_pdf (Monthly financial report generation task — loose SLA: takes 30 to 60 seconds per document).

When a user requests printing a large monthly financial report, the broker floods the queue with PDF generation tasks. If moments later another user tries to log in and requests an OTP code:

• The OTP request message enters the queue’s back position, stuck behind dozens of slow PDF generation tasks.
• Even if we have several active consumers, the memory- and CPU-heavy PDF generation process monopolizes all worker resources.
• Users waiting for OTPs experience login code delivery delays up to minutes, instantly damaging user experience.

In distributed systems, we must not let latency-sensitive transactional messages be obstructed by computationally heavy background jobs.

Serialization Coupling and Code Dependency Problems #

Besides runtime performance issues like HOLB, monolithic queues also create Monolithic Coupling problems at the application code level.

flowchart TD
    Queue["queue.app.events"] --> Consumer["Consumer App<br/>'(Must import structs & libraries from<br/>Order, Payment, Inventory, User<br/>to deserialize payloads)'"]

If one queue holds events from the Order, Payment, Inventory, and User domains, the consumer application listening to that queue must have all the struct/DTO class definitions and serialization libraries from each of those domains.

This badly impacts software release cycles:

1. Microservices Boundary Violations: Every time the Ordering team changes their JSON data structures, consumer developer teams must update their code and re-deploy, even though they have no interest in that Ordering data.
2. Runtime Incompatibilities: A bad format data error on one event message from one domain (e.g., an Inventory payload data type error) can make the consumer application experience crash loops. Because consumers die, message processing for healthy Payment and User domains also stops entirely.
3. Migration Difficulties: We lose the ability to migrate or rewrite one specific consumer using another more efficient programming language (e.g., migrating PDF workers from Python to Go) because that worker is forced to process all other event types in the monolithic queue.

SLA and Priority Management Failures #

Every message in business systems has different Service Level Agreements (SLAs) and retry/dead-letter policy handling behaviors:

• Payment Messages: Need maximum durability, strict Publisher Confirms, and manual ACKs with Quorum Queue isolation. On failures, messages must be exponentially retried over long periods before giving up.
• IoT Telemetry Messages: Have very high volumes, aren’t sensitive to single data losses, and need non-durable Classic Queue performance with auto-acks for maximum throughput.
• Push Notification Messages: Need fast delivery, have short expiration periods (TTL), and if they fail, can be simply discarded without complex retries.

If we mix all three into one monolithic queue, we’re forced to equalize broker configurations. We can’t set small Prefetch Counts specifically for heavy tasks, can’t separate DLQs for isolated financial transaction failure analysis, and can’t scale worker counts independently. Our system loses operational precision.

The Correct Design Pattern: Domain/SLA-Based Segregation #

The correct approach is disciplined Queue Segregation. We must design queues based on microservices functional domain boundaries and data processing SLA characteristics.

Queue segregation design rules of thumb:

1. Queue per Consumer Group per Domain Use-Case: Every independent microservice acting as a consumer must have a dedicated queue specifically holding the data it needs.
2. Fast vs Slow Lane Separation: Long execution duration tasks (like media conversion, CSV exports, file encryption) must be allocated to separate queues from lightweight transactional event queues.
3. Isolated Retry and DLQs: Every main business queue must have its own retry queue and Dead Letter Queue (DLQ) pair so failure tracing (debugging) processes run specifically without polluting other domains.

Single Queue vs Segregated Queue Problems #

The diagram below illustrates HOLB congestion dangers on a single monolithic queue vs smooth data flows after segregation by domain and priority level (SLA).

flowchart TD
    subgraph Monolithic["Monolithic Queue Model (Clogged)"]
        direction TB
        P1[Multi-Domain Producer] -->|Publish| EX1(Exchange)
        EX1 -->|created & pdf| Q_Mono["queue.app.events (Monolithic FIFO)"]
        Q_Mono -->|Clogged by PDF Tasks| C_Mono[Consumer Worker]
        C_Mono -->|Stalled| H_OTP["OTP SMS (Delayed!)"]
        C_Mono -->|Processing| H_PDF["Generate PDF (Slow)"]
    end

    subgraph Segregated["Segregated Queue Model (Smooth & Safe)"]
        direction TB
        P2[Multi-Domain Producer] -->|Publish| EX2(Topic Exchange)
        EX2 -->|order.otp| Q_Fast["queue.otp.high-priority (Fast SLA)"]
        EX2 -->|order.report| Q_Slow["queue.reports.low-priority (Slow SLA)"]
        Q_Fast --> C_Fast[Fast Worker e.g. 5 replicas]
        Q_Slow --> C_Slow[Slow Worker e.g. 1 replica]
        C_Fast --> H_OTP_Fast[Instant SMS Delivery]
        C_Slow --> H_PDF_Slow[Background PDF Generation]
    end

    style Q_Mono stroke:#f44336,stroke-width:2px
    style Q_Fast stroke:#4caf50,stroke-width:2px
    style Q_Slow stroke:#00bcd4,stroke-width:2px

Go Code Implementation: Queue Segregation by SLA and Domain #

Here is a Go implementation example showing how we declare separate queues based on SLA (a high-priority queue for instant transactions and a low-priority queue for slow report processing), and how we distribute workers independently for each of those queues.

package main

import (
	"context"
	"encoding/json"
	"log"
	"time"

	amqp "github.com/rabbitmq/amqp091-go"
)

// TransactionPayload represents instant payment transactions (High SLA).
type TransactionPayload struct {
	TxID      string    `json:"tx_id"`
	Amount    float64   `json:"amount"`
	CreatedAt time.Time `json:"created_at"`
}

// ReportPayload represents financial report document generation tasks (Low SLA).
type ReportPayload struct {
	ReportID  string `json:"report_id"`
	UserID    string `json:"user_id"`
	ScopeYear int    `json:"scope_year"`
}

// WorkerPool manages workers for segregated queues.
type WorkerPool struct {
	conn *amqp.Connection
}

// StartHighPriorityConsumer consumes the transaction queue with many worker replica replications.
func (wp *WorkerPool) StartHighPriorityConsumer(queueName string) {
	ch, err := wp.conn.Channel()
	if err != nil {
		log.Fatalf("Gagal membuka channel transaksi: %v", err)
	}

	// Limit prefetch so work distribution to worker replicas is even
	err = ch.Qos(20, 0, false)
	if err != nil {
		log.Printf("Gagal set Qos transaksi: %v", err)
	}

	msgs, err := ch.Consume(queueName, "", false, false, false, false, nil)
	if err != nil {
		log.Fatalf("Gagal konsumsi antrean transaksi: %v", err)
	}

	// Run 5 parallel worker goroutines to handle transactions instantly
	for i := 1; i <= 5; i++ {
		go func(workerID int) {
			log.Printf("[Worker Transaksi-%d] Siap memproses transaksi...", workerID)
			for msg := range msgs {
				var tx TransactionPayload
				_ = json.Unmarshal(msg.Body, &tx)

				// Simulate fast processing (e.g., memory validation & DB updates)
				log.Printf("[Worker Transaksi-%d] MEMPROSES TxID: %s, Nominal: %.2f", workerID, tx.TxID, tx.Amount)
				time.Sleep(100 * time.Millisecond) // low latency

				_ = msg.Ack(false)
				log.Printf("[Worker Transaksi-%d] Sukses ACK TxID: %s", workerID, tx.TxID)
			}
		}(i)
	}
}

// StartLowPriorityConsumer consumes the report queue with limited workers so the database isn't overloaded.
func (wp *WorkerPool) StartLowPriorityConsumer(queueName string) {
	ch, err := wp.conn.Channel()
	if err != nil {
		log.Fatalf("Gagal membuka channel laporan: %v", err)
	}

	// Prefetch is set small because each message's processing duration is very long
	err = ch.Qos(2, 0, false)
	if err != nil {
		log.Printf("Gagal set Qos laporan: %v", err)
	}

	msgs, err := ch.Consume(queueName, "", false, false, false, false, nil)
	if err != nil {
		log.Fatalf("Gagal konsumsi antrean laporan: %v", err)
	}

	// Just run 1 worker goroutine to limit CPU/DB resource consumption
	go func() {
		log.Printf("[Worker Laporan] Siap memproses pembuatan dokumen PDF...")
		for msg := range msgs {
			var rpt ReportPayload
			_ = json.Unmarshal(msg.Body, &rpt)

			// Simulate heavy processing (e.g., slow DB aggregation queries & PDF file rendering)
			log.Printf("[Worker Laporan] MEMPROSES PEMBUATAN PDF Laporan ID: %s, Tahun: %d", rpt.ReportID, rpt.ScopeYear)
			time.Sleep(3 * time.Second) // the work takes a long time

			_ = msg.Ack(false)
			log.Printf("[Worker Laporan] Sukses ACK Laporan ID: %s", rpt.ReportID)
		}
	}()
}

func main() {
	conn, err := amqp.Dial("amqp://guest:***@localhost:5672/")
	if err != nil {
		log.Fatalf("Gagal koneksi ke RabbitMQ: %v", err)
	}
	defer conn.Close()

	ch, err := conn.Channel()
	if err != nil {
		log.Fatalf("Gagal membuka setup channel: %v", err)
	}
	defer ch.Close()

	// 1. Declare the main exchange
	exchangeName := "app.events"
	_ = ch.ExchangeDeclare(exchangeName, "topic", true, false, false, false, nil)

	// 2. Declare the High SLA Queue (Transactions)
	txQueue := "queue.payment.transactions"
	_, _ = ch.QueueDeclare(txQueue, true, false, false, false, nil)
	_ = ch.QueueBind(txQueue, "payment.transaction.execute", exchangeName, false, nil)

	// 3. Declare the Loose SLA Queue (Reports)
	reportQueue := "queue.reports.billing"
	_, _ = ch.QueueDeclare(reportQueue, true, false, false, false, nil)
	_ = ch.QueueBind(reportQueue, "billing.report.generate", exchangeName, false, nil)

	pool := &WorkerPool{conn: conn}

	// Start the separate worker pools
	pool.StartHighPriorityConsumer(txQueue)
	pool.StartLowPriorityConsumer(reportQueue)

	// Simulate message publication by producers
	ctx := context.Background()
	
	// Send the report task (Loose SLA) first
	rptData, _ := json.Marshal(ReportPayload{ReportID: "RPT-2026", UserID: "USR-99", ScopeYear: 2026})
	_ = ch.PublishWithContext(ctx, exchangeName, "billing.report.generate", false, false, amqp.Publishing{
		ContentType: "application/json",
		Body:        rptData,
	})
	log.Println("[Producer] Mempublikasikan tugas pembuatan laporan PDF (SLA Longgar)")

	// Send several instant transactions (High SLA)
	for i := 1; i <= 3; i++ {
		txData, _ := json.Marshal(TransactionPayload{TxID: "TX-100" + string(rune(i)), Amount: 50000.0, CreatedAt: time.Now()})
		_ = ch.PublishWithContext(ctx, exchangeName, "payment.transaction.execute", false, false, amqp.Publishing{
			ContentType: "application/json",
			Body:        txData,
		})
		log.Println("[Producer] Mempublikasikan transaksi pembayaran instan (SLA Tinggi)")
	}

	// Wait so processing is recorded in the console log before the program exits
	time.Sleep(10 * time.Second)
}

Monolithic Queue vs Segregated Queue Design Comparison Table #

The following table presents detailed architectural impact comparisons between a single monolithic queue and segregated queues.

Queue Segregation Audit Checklist #

Audit our system’s queue segregation using the checklist guide below:

WORKLOAD ISOLATION TESTING:
  □ Are messages requiring sub-1-second processing (e.g., OTP, instant notifications) in separate queues from heavy processing messages?
  □ Does every consumer microservice application consume a separate queue specifically configured for its own service function?
  □ Are payload deserialization failures or error handling in one domain queue (e.g., Inventory) safe from stopping consumption of other domain queues (e.g., Payment)?
  □ Can we increase worker pod replica counts for one specific task type without having to distribute workers for other task types?
  □ Is the DLQ configured specifically per main business queue, not one giant DLQ holding all system corrupted messages?

CORRECTIVE ACTIONS IF THE ANSWER IS NO:
  □ Identify business event types and group them into 3 main SLA categories: Instant (Real-time), Transactional (Medium), and Batch (Slow).
  □ Declare separate static queues for each SLA category and install specific routing key bindings.

Summary #

• Head-of-Line Blocking — FIFO on monolithic queues forces instant latency-sensitive messages (OTP SMS) to queue behind heavy messages taking very long processing times (PDF generation/DB aggregation).
• Monolithic Coupling — Combining all messages in one queue forces consumer applications to import all payload schema dependency libraries and deserialize all message types, destroying microservices isolation boundaries.
• Crash Loop Vulnerabilities — Parsing format failures on one event type in monolithic queues can trigger consumer crash loops, stopping all other domain event processing.
• Queue Segregation — Healthy architectures apply queue separation based on microservices functional domains and data processing SLA characteristics (fast vs slow lane separation).
• Precise Scaling — Queue segregation lets us apply competing consumers granularly and efficiently (adding transaction worker replicas, limiting workers for heavy tasks so databases aren’t overloaded).

Closing #

Uniting all system events into one queue indeed provides implementation ease in early projects. However, this false simplicity trades away our distributed system’s long-term stability. Once system traffic increases, monolithic queues become hidden congestion points that are difficult to diagnose.

Remember this design rule: Queues in RabbitMQ are failure isolation tools and data processing SLA determination instruments. Use queue boundaries precisely to protect critical messages from background process disruptions.

By applying domain/SLA-based queue segregation and distributing workers independently, we ensure every business event is processed according to its priority, our systems are free from Head-of-Line Blocking, and our microservices teams work independently without harmful code coupling.

← Previous: Over Queue   Next: Not Using DLQ →