Over-Queue (Queue Explosion) #

Declaring queues in RabbitMQ is very easy. Just by calling one command line in producer or consumer applications, new queues are created in milliseconds. This ease and flexibility often triggers an incorrect tendency in developer teams: creating new queues for every small need without a centralized topology governance plan. This phenomenon is known as Over-Queue or Queue Explosion.

Many developers consider queues a cheap component without meaningful performance costs. As a result, they design systems by creating dedicated queues per entity ID, per user, per transaction, or creating new queues for every small action variation of the business domain. Behind this ease, uncontrolled stacks of hundreds or thousands of queues burden RabbitMQ’s internal metadata database, slow cluster synchronization, complicate observability, and drastically degrade broker throughput. This article deeply dissects why excessive queue count accumulation is a latent danger in RabbitMQ, how it impacts internal broker performance, and how we should apply Multiplexing patterns to design lean, efficient topologies.

The Queue Explosion Phenomenon in Production Systems #

In complex production environments, Queue Explosion usually occurs organically through several common design error patterns:

1. Dynamic Entity ID-Based Queues (Ad-hoc Runtime Queues): Developers create new queues at runtime using dynamic parameters, e.g., queue.order.payment.12093 where the last number is a unique transaction ID. When a new transaction occurs, the producer application automatically declares a new queue for that transaction.
2. Queue Segregation Based on Granular Actions: Instead of channeling order activities to one shared queue, developers split them into many small queues for every order status, like:
   • queue.order.created
   • queue.order.validated
   • queue.order.pending.payment
   • queue.order.payment.success
   • queue.order.payment.failed
   • queue.order.shipped
   • queue.order.delivered
3. “One Team, One Queue” Designs Without Coordination: In large organizations with many microservices developer teams, the absence of centralized topology governance causes every team to declare their own queues ad-hoc without checking whether similar queues already exist.
4. Leaking Temporary Queues: Using request-reply (RPC) patterns that declare temporary reply queues without configuring them with Auto-delete or Exclusive attributes, so those queues settle forever on the broker even after the RPC call process finishes.

Why Do Many Queues Destroy Broker Performance? #

To understand why Over-Queue is a serious threat to RabbitMQ stability, we must look at how RabbitMQ manages objects under its hood.

1. Mnesia Metadata Database Overhead #

RabbitMQ uses an internal distributed transactional database written in the Erlang language called Mnesia. This database tracks the entire cluster topology schema, including exchange name data, queue configurations, binding rules, routing keys, and mappings to the physical nodes where those queues reside.

• Every time we declare a new queue or change bindings, RabbitMQ must perform a write transaction on Mnesia.
• This Mnesia metadata is stored in RAM memory on every cluster node to ensure instant routing speed.
• When the queue count reaches thousands, the RAM consumption used just to store Mnesia metadata balloons.
• If we run a multi-node cluster, every dynamic queue change triggers Mnesia synchronization processes to all nodes. This distributed synchronization consumes large network bandwidth and CPU cycles, obstructing the main message routing process.

2. Erlang Process Resource Allocation per Queue #

Inside the Erlang BEAM VM, every RabbitMQ queue is represented as one independent, isolated Erlang process.

• Every Erlang process has an initial memory allocation (stack and heap), its own message queue, and consumes around 10 KB to 20 KB of memory even when the queue is empty without messages.
• Although the Erlang VM is very efficient at handling millions of processes, having tens of thousands of active empty queues still triggers large static RAM consumption.
• More critically, the Erlang VM scheduler must continuously monitor the active/inactive status of thousands of these queue processes. This triggers high CPU context switching and degrades real message processing performance.

3. Connection Fragmentation and Message Memory Allocation #

When data flows into many small queues, those messages fragment across various separate Erlang processes.

1. The Erlang Garbage Collection (GC) mechanism must scan each queue process separately. This memory fragmentation degrades overall broker memory management efficiency.
2. Every consumer application must maintain separate connections and channels to each of those unique queues. This triggers TCP connection and open channel count spikes on the broker, accelerating OS file descriptor limit exhaustion.

Topology Design: Dynamic vs Static #

Before writing code, we must compare the operational impacts of the following two topology design approaches:

1. Dynamic Topologies (Anti-Pattern) #

Producer and consumer applications declare queues dynamically at runtime using their program code. Topologies change wildly based on user activity.

• Impact: Queue counts are unpredictable. It’s hard to create monitoring dashboards in Grafana because queue names change dynamically. Infrastructure operations teams have no visibility into broker capacity.

2. Static Topologies (Best Practice) #

The entire topology (Exchanges, Queues, Bindings) is determined and declared statically before applications run. Declarations are done through Infrastructure as Code (IaC) tools like Terraform, RabbitMQ JSON Definitions Files, or strict initial scripts at CI/CD pipeline stages.

• Impact: Queue counts stay pinned at consistent, predictable numbers. Metric monitoring becomes very easy. Topology changes must go through code review processes (Merge Requests), preventing unknown ad-hoc queues from appearing on the broker.

The Correct Design Pattern: Multiplexing with Routing Keys #

The solution to overcome Over-Queue is applying the Multiplexing pattern. Instead of creating one separate queue for every small action or unique ID, we create one static domain/SLA-based queue, channel all event types under that domain into the same queue, and use rich Routing Keys to distinguish event types on the consumer side.

For example, instead of creating 7 queues for every order status, we design the topology as follows:

1. One Exchange: We create one Topic type Exchange named order.events.
2. One Main Queue: We create one static queue named queue.order.processing.
3. Dynamic Binding: We bind that queue to the order.events Exchange using the order.* routing key pattern.
4. Specific Routing Keys: Producers publish messages with routing keys like order.created, order.payment.success, or order.shipped.
5. Multiplexing Consumption: Consumers pull all messages from queue.order.processing, sort event types by routing key at the application code level, and channel them to appropriate handler functions.

If another microservices team (e.g., the Logistics team) only cares about shipping events, they just create one static queue queue.logistic.order-updates and bind it to the same exchange with the order.shipped pattern.

Topology Design Comparison: Dynamic vs Multiplexing #

The diagram below illustrates the difference between the dynamic queue stacking model triggering Queue Explosion and the structured Multiplexing model using a static Topic Exchange.

flowchart TD
    subgraph Model Salah ["Over-Queue Model (Dynamic & Ad-hoc)"]
        direction TB
        P1[Order Producer] -->|Publish| EX1(Direct Exchange)
        EX1 -->|order.created| Q1["queue.order.created.123 (Dynamic)"]
        EX1 -->|order.shipped| Q2["queue.order.shipped.456 (Dynamic)"]
        EX1 -->|order.delivered| Q3["queue.order.delivered.789 (Dynamic)"]
        Q1 --> C1[Consumer 1]
        Q2 --> C2[Consumer 2]
        Q3 --> C3[Consumer 3]
    end

    subgraph Model Benar ["Multiplexing Model (Static & Lean)"]
        direction TB
        P2[Order Producer] -->|Publish| EX2(Topic Exchange: order.events)
        EX2 -->|"order.*"| Q_Main["queue.order.processing (Static)"]
        Q_Main --> C_Main[Consumer Group: Order Processor]
        C_Main -->|Routing Key check| H1[Handler Created]
        C_Main -->|Routing Key check| H2[Handler Shipped]
        C_Main -->|Routing Key check| H3[Handler Delivered]
    end

    style Q1 stroke:#f44336,stroke-width:2px
    style Q2 stroke:#f44336,stroke-width:2px
    style Q3 stroke:#f44336,stroke-width:2px
    style Q_Main stroke:#4caf50,stroke-width:2px

Go Code Implementation: Multiplexing Using Topic Exchanges #

Below is a Go implementation code example showing how we design a static topology with Topic Exchanges, flow various sub-events into one processing queue, and sort messages by routing key on the consumer side.

package main

import (
	"context"
	"encoding/json"
	"log"
	"time"

	amqp "github.com/rabbitmq/amqp091-go"
)

// EventPayload represents the common transaction event data schema.
type EventPayload struct {
	OrderID   string    `json:"order_id"`
	Amount    float64   `json:"amount"`
	Timestamp time.Time `json:"timestamp"`
}

// OrderEventHandler manages message processing centrally.
type OrderEventHandler struct{}

// Handle sorts and processes events based on the routing key.
func (h *OrderEventHandler) Handle(routingKey string, body []byte) {
	var payload EventPayload
	if err := json.Unmarshal(body, &payload); err != nil {
		log.Printf("[Handler] Error serialisasi payload: %v", err)
		return
	}

	// Select business logic based on the routing key (Multiplexing)
	switch routingKey {
	case "order.created":
		h.handleOrderCreated(payload)
	case "order.payment_success":
		h.handlePaymentSuccess(payload)
	case "order.shipped":
		h.handleOrderShipped(payload)
	default:
		log.Printf("[Handler] Menerima event tak dikenal: %s", routingKey)
	}
}

func (h *OrderEventHandler) handleOrderCreated(p EventPayload) {
	log.Printf("[Handler - Created] Memproses inisialisasi order ID: %s, senilai: %.2f", p.OrderID, p.Amount)
}

func (h *OrderEventHandler) handlePaymentSuccess(p EventPayload) {
	log.Printf("[Handler - Payment] Mencatat pembayaran sukses untuk order ID: %s", p.OrderID)
}

func (h *OrderEventHandler) handleOrderShipped(p EventPayload) {
	log.Printf("[Handler - Shipped] Memperbarui status pengiriman logistik untuk order ID: %s", p.OrderID)
}

func main() {
	// 1. Connect to the RabbitMQ broker
	conn, err := amqp.Dial("amqp://guest:***@localhost:5672/")
	if err != nil {
		log.Fatalf("Gagal terhubung ke RabbitMQ: %v", err)
	}
	defer conn.Close()

	ch, err := conn.Channel()
	if err != nil {
		log.Fatalf("Gagal membuka channel: %v", err)
	}
	defer ch.Close()

	// 2. Declare the static Exchange (Topic Exchange)
	exchangeName := "order.events"
	err = ch.ExchangeDeclare(
		exchangeName, // name
		"topic",      // type
		true,         // durable
		false,        // auto-deleted
		false,        // internal
		false,        // no-wait
		nil,          // arguments
	)
	if err != nil {
		log.Fatalf("Gagal deklarasi exchange: %v", err)
	}

	// 3. Declare a single static queue for order processing
	queueName := "queue.order.processing"
	_, err = ch.QueueDeclare(
		queueName, // name
		true,      // durable
		false,     // auto-deleted
		false,     // exclusive
		false,     // no-wait
		nil,       // arguments
	)
	if err != nil {
		log.Fatalf("Gagal deklarasi queue: %v", err)
	}

	// 4. Bind the static queue to the Topic Exchange with the "order.*" wild-card pattern
	// This captures order.created, order.payment_success, order.shipped, etc.
	bindingKey := "order.*"
	err = ch.QueueBind(
		queueName,
		bindingKey,
		exchangeName,
		false,
		nil,
	)
	if err != nil {
		log.Fatalf("Gagal binding queue ke exchange: %v", err)
	}

	log.Printf("[Init] Topologi statis dideklarasikan. Menunggu pesan di %s...", queueName)

	// Initialize the event processing handler
	handler := &OrderEventHandler{}

	// 5. Activate the consumer loop
	msgs, err := ch.Consume(
		queueName, // queue
		"",        // consumer tag (auto-generated if empty)
		false,     // auto-ack (we're advised to use manual ACKs)
		false,     // exclusive
		false,     // no-local
		false,     // no-wait
		nil,       // arguments
	)
	if err != nil {
		log.Fatalf("Gagal mendaftarkan consumer: %v", err)
	}

	go func() {
		for msg := range msgs {
			// Run multiplexing based on the Routing Key
			handler.Handle(msg.RoutingKey, msg.Body)

			// Send the manual ACK confirmation
			err := msg.Ack(false)
			if err != nil {
				log.Printf("Gagal mengirim ACK: %v", err)
			}
		}
	}()

	// Simulate event delivery by producers in the main thread
	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
	defer cancel()

	events := []struct {
		RoutingKey string
		Payload    EventPayload
	}{
		{
			RoutingKey: "order.created",
			Payload:    EventPayload{OrderID: "order-101", Amount: 25000.0, Timestamp: time.Now()},
		},
		{
			RoutingKey: "order.payment_success",
			Payload:    EventPayload{OrderID: "order-101", Amount: 25000.0, Timestamp: time.Now()},
		},
		{
			RoutingKey: "order.shipped",
			Payload:    EventPayload{OrderID: "order-101", Amount: 0.0, Timestamp: time.Now()},
		},
	}

	for _, ev := range events {
		body, _ := json.Marshal(ev.Payload)
		err = ch.PublishWithContext(ctx,
			exchangeName,
			ev.RoutingKey,
			false,
			false,
			amqp.Publishing{
				ContentType:  "application/json",
				DeliveryMode: amqp.Persistent,
				Body:         body,
			},
		)
		if err != nil {
			log.Printf("Gagal publish event %s: %v", ev.RoutingKey, err)
		} else {
			log.Printf("[Producer] Sukses mempublikasikan event dengan key: %s", ev.RoutingKey)
		}
		time.Sleep(500 * time.Millisecond) // illustrative time gap
	}

	// Block execution so the program doesn't exit immediately
	time.Sleep(3 * time.Second)
}

Queue Count Impact Comparison on Systems #

The table below compares operational efficiency between systems experiencing Queue Explosion and systems applying queue limitation using the Multiplexing pattern.

Queue Count Audit Checklist #

Use the following checklist when performing routine audits to prevent Queue Explosion dangers on our RabbitMQ cluster:

BROKER TOPOLOGY MONITORING:
  □ Is the total active queue count on the RabbitMQ cluster controlled (e.g., under 100 queues for large-scale systems)?
  □ Are all queue names static without dynamic variables like user IDs, timestamps, or transaction IDs?
  □ Is new queue creation declared centrally through RabbitMQ JSON definition files, Terraform, or CI/CD configuration scripts?
  □ Are temporary queues used for request-reply (RPC) patterns configured with 'Auto-delete' or 'Exclusive' attributes so they're automatically deleted after connections drop?
  □ Do we leverage Topic Exchanges with wildcard patterns (e.g., 'domain.*') to centralize several sub-event types into one shared queue?

REMEDIATION IF NOT:
  □ Clean up orphan queues without active consumers using the CLI command: 'rabbitmqctl delete_queue'.
  □ Migrate producer and consumer code to use one centralized static queue per service domain with routing key filters in application code.

Summary #

• Mnesia Metadata Explosion — Every queue declared in RabbitMQ is recorded in the internal distributed Mnesia database. Having thousands of queues exhausts RAM memory and slows inter-node cluster synchronization.
• Erlang Process per Queue — One RabbitMQ queue is represented by one independent process in the Erlang BEAM VM. Empty processes still consume RAM memory allocations and force excessive CPU scheduler context switching if extremely numerous.
• Runtime Declaration Dangers — Don’t let producer/consumer code create dynamic queues ad-hoc using runtime dynamic IDs. Topologies must be statically designed and declared through CI/CD pipelines or IaC tools.
• Multiplexing — The best solution for avoiding queue explosions is multiplexing. Channel all domain sub-events into one single processing queue, distinguishing data types through Routing Keys in consumer application code.
• Topic Exchanges — Leverage Topic Exchanges with wildcard matching capabilities (e.g., order.*) to unite or sort message flows without needing to physically declare new queues.

Closing #

RabbitMQ’s ease of queue creation is a double-edged sword. Without structured architecture design discipline, small queues grow uncontrollably as our application features grow, eventually triggering Queue Explosion that cripples broker performance.

Remember this principle: The queue count in RabbitMQ should reflect our business domain structure and data handling SLAs — not a representation of individual runtime activities.

By designing static Multiplexing-based topologies and leveraging Topic Exchange flexibility, we can keep RabbitMQ performance high, save cluster RAM consumption, and keep our distributed systems simple and easy to monitor long-term.

← Previous: RabbitMQ as Database   Next: 1 Queue for All Events →