Topic Exchange #

In enterprise-scale event-driven architecture (EDA) systems, message routing complexity often can’t be solved with the rigid literal matching of a Direct Exchange or the unfiltered mass broadcasting of a Fanout Exchange. We need a smart, flexible, dynamic routing mechanism that still performs well, capable of directing messages based on granular business domain classification.

In the RabbitMQ ecosystem, this need is fulfilled by the Topic Exchange. The Topic Exchange acts as a dynamic filter engine that evaluates wildcard expressions on queue Binding Keys against message Routing Keys. Through the Topic Exchange, consumers can specify very specific message acceptance criteria without forcing producers to know consumer queue structures. This article dissects in depth the Topic Exchange’s wildcard pattern matching mechanism, the internal search architecture based on Trie data structures in Erlang, structured naming conventions for optimal performance, and performance risk mitigation in production environments.

Pattern Routing Mechanism (Wildcard Matching) #

The Topic Exchange routes messages based on pattern matching between the Routing Key attached by the producer to the message and the Binding Key pattern registered by the consumer for its queue. Unlike the Direct Exchange, which requires exact word-by-word string matching, the Topic Exchange allows sub-category-based routing using dot separators and special wildcard characters.

The Routing Key naming syntax and pattern matching on a Topic Exchange follow these rules:

1. Word Segments (Tokens): The Routing Key must be a string containing words or tokens separated by dots (.), e.g., asia.indonesia.jakarta. The maximum Routing Key string length is 255 characters (bytes). Each word or segment usually represents a hierarchy level in our business domain.
2. Asterisk Wildcard (*): Acts as a replacement for exactly one word between dot separators. If we place *, that segment must be filled by exactly one word in the message Routing Key to match.
3. Hash Wildcard (#): Acts as a replacement for zero or more consecutive words. This wildcard is very flexible because it can match an empty segment or many segments at once up to the end of the Routing Key.

flowchart TD
    Msg["New Message (Routing Key: 'eu.sales.order.completed')"] --> TopicEx["Topic Exchange (events.topic)"]
    TopicEx -->|"Trie Traversal"| Match{"Search Pattern"}
    Match -->|"Binding Key: '*.sales.order.*'"| Queue1["Queue A (billing-eu)"]
    Match -->|"Binding Key: 'eu.#'"| Queue2["Queue B (eu-reporting)"]
    Match -->|"Binding Key: '#.failed'"| Queue3["Queue C (dead-letter)"]
    Queue1 --> ConsA["EU Billing Service"]
    Queue2 --> ConsB["Regional Reporting Service"]
    Queue3 --> ConsC["Global DLQ Service"]

Pattern Matching Evaluation #

In the diagram above, we can trace how a message with the Routing Key "eu.sales.order.completed" is distributed to various queues:

• Queue A is bound with the Binding Key *.sales.order.*. This pattern matches the message because the first segment (eu) is matched by the first * wildcard, the second and third segments (sales.order) match literally, and the fourth segment (completed) is matched by the second * wildcard.
• Queue B is bound with the Binding Key eu.#. This pattern matches the message because the message starts with the eu segment, and the remaining segments (sales.order.completed) are all matched at once by the # wildcard.
• Queue C is bound with the Binding Key #.failed. This pattern does not match the message because the message ends with the completed segment, not failed. The message is ignored by this queue.

Routing Key Matching Comparison Table #

To provide a more comprehensive picture, let’s examine the following pattern matching evaluation table:

Pattern Matching Edge Cases #

When designing production systems, we must pay attention to several edge cases that can affect routing behavior:

• Empty String (""): If a producer sends a message with an empty Routing Key, the message only matches bindings using the # wildcard alone. The * pattern won’t match because it requires at least one word.
• Double Dots (a..b): If a Routing Key is sent in the a..b format, Erlang treats it as three segments: ["a", "", "b"]. The middle segment is an empty string. A pattern like a.*.b matches a..b because the middle segment counts as one word (even though empty). However, this is a naming anti-pattern we must avoid.
• Case Sensitivity: The Topic Exchange is case-sensitive. The Routing Key sales.Order won’t match the Binding Key sales.order. Make sure our application always normalizes Routing Keys to lowercase before publishing.

Behind the Scenes: Trie-Matching Architecture in Erlang #

For most system architects, there is a concern that wildcard pattern matching will significantly degrade broker message routing performance. To overcome the performance drop from linear string searches, RabbitMQ doesn’t use standard regex engines, which are slow and CPU-hungry. The internal rabbit_exchange_type_topic module in the Erlang runtime implements a Trie (Prefix Tree) data structure stored dynamically in RAM memory.

How Does the Trie Work in RabbitMQ? #

Every time a queue is bound to a Topic Exchange using a structured Binding Key, RabbitMQ splits that key string by dots and arranges it into Trie tree nodes in memory. These nodes hold word segment information and references to matching destination queues.

For example, if we have three binding keys registered in the broker:

1. sales.order.*
2. sales.#
3. inventory.stock.updated

Erlang arranges the following Trie tree structure in RAM:

flowchart TD
    Root["Root"] --> Sales["sales"]
    Root --> Inventory["inventory"]
    Sales --> Order["order"]
    Sales --> Hash["#"]
    Inventory --> Stock["stock"]
    Order --> Wildcard["*"]
    Stock --> Updated["updated"]

When a message with the Routing Key "sales.order.created" enters the broker, the evaluation flow Erlang performs is:

1. Tokenization: The Erlang process splits the Routing Key string into a token list: ["sales", "order", "created"].
2. Trie Traversal: The broker traverses the Trie from the Root. It jumps directly to the "sales" branch, then enters the "order" branch, and finally matches the "created" word with the * wildcard node.
3. Time Efficiency: The lookup complexity is $O(L)$, where $L$ is the number of word segments in the message Routing Key (usually only 3 to 5 levels), not $O(N)$ where $N$ is the total number of registered bindings. This keeps lookup time constant and very fast even with thousands of queues bound to the exchange.

Mnesia Storage vs ETS Tables #

In a RabbitMQ cluster, Topic Exchange binding metadata is stored in a distributed database table named Mnesia. However, for fast runtime lookups when thousands of messages flow per second, RabbitMQ projects that Trie structure into ETS (Erlang Term Storage) tables optimized for high-speed parallel read operations.

Every time a RabbitMQ node receives a message, it doesn’t query the Mnesia database; it directly reads the already-compiled Trie structure in ETS RAM. This minimizes lookup latency to under a millisecond.

The Metadata Churning Phenomenon (Dynamic Recompilation) #

Even though Trie lookups are very fast ($O(L)$), the process of modifying the Trie structure has high computational cost. Every time a new queue binds, or an old queue dynamically unbinds, RabbitMQ must rebuild part or all of the Trie tree structure in RAM and replicate it to all cluster nodes.

If our application is designed to do metadata churning (e.g., creating temporary queues for every new user connection, then binding/unbinding hundreds of times per second), broker CPU is consumed entirely by Trie recompilation. During this recompilation, message routing threads can experience a blocking state, causing message queues to stall and delivery latency to spike drastically.

Structured Naming Conventions & Versioning Strategies #

The Routing Key on a Topic Exchange acts as a logical API contract between producers and consumers. Carelessly changing the Routing Key format on the producer side silently breaks data flow to consumer queues without triggering an error on the producer side. Therefore, we must apply strict, mature naming conventions from the very start of system development.

1. The Domain-Entity-Action Pattern #

The recommended naming pattern for production event-driven systems uses the following hierarchy:

<bounded_context>.<entity_name>.<event_action>

• bounded_context (or service name): The main business domain generating the event, written in lowercase. Example: sales, billing, shipping, auth.
• entity_name: The main business object or entity experiencing a state change. Example: order, invoice, shipment, user.
• event_action: The specific event that has completed, usually using past-tense verbs. Example: created, completed, failed, cancelled, paid.

Real implementation examples:

• sales.order.created (a new order was created)
• billing.invoice.paid (an invoice payment was verified)
• shipping.shipment.dispatched (goods were shipped by the courier)

With this structured pattern, consumers can subscribe flexibly:

• The notification service wants to hear all order events: sales.order.*
• The global audit service wants to hear all billing activity: billing.#
• The analytics service wants to monitor all success events across domains: *.*.completed or *.*.paid

2. Event Versioning Strategies #

As systems evolve, event payload data structures often must change (e.g., adding a new required field). This triggers a breaking change for legacy consumers not yet ready to update their code. To handle this, we must design a versioning strategy on the Routing Key.

Strategy A: Inserting the Version in the Routing Key (Explicit Versioning) #

We can insert the schema version number at the end of the Routing Key:

• sales.order.created.v1
• sales.order.created.v2

Legacy consumers keep binding their queues to .v1, while new consumers capable of handling the new data schema can bind their queues to .v2. Producers publish events to both routes during the migration transition period (dual publishing), or only send to the new version once legacy consumers are fully migrated.

Strategy B: Using Wildcards for Parallel Transition #

If we want to minimize route changes on the producer side, producers can keep using the same Routing Key, but consumers bind queues with wildcards to capture different versions. Consumers distinguish payload versions by reading the headers metadata property (e.g., x-schema-version attached by the producer). The Topic Exchange greatly simplifies this parallel transition because consumers can write binding keys like sales.order.created.# to receive all route variations under that topic.

Code Implementation: Topic Exchange Integration in Go #

Let’s review a complete implementation example using the Go language and the github.com/rabbitmq/amqp091-go library. The example below shows how to define a Topic Exchange, create queues, bind queues with wildcard patterns, and send messages with structured Routing Keys.

package main

import (
	"context"
	"log"
	"time"

	amqp "github.com/rabbitmq/amqp091-go"
)

// Helper to handle errors consistently
func failOnError(err error, msg string) {
	if err != nil {
		log.Fatalf("%s: %s", msg, err)
	}
}

func main() {
	// 1. Open a connection to the RabbitMQ broker
	conn, err := amqp.Dial("amqp://guest:***@localhost:5672/")
	failOnError(err, "Gagal terhubung ke RabbitMQ")
	defer conn.Close()

	// 2. Open a communication channel
	ch, err := conn.Channel()
	failOnError(err, "Gagal membuka channel")
	defer ch.Close()

	// 3. Declare the Topic Exchange
	exchangeName := "sales.events.topic"
	err = ch.ExchangeDeclare(
		exchangeName, // Exchange name
		"topic",       // Exchange type must be 'topic'
		true,          // Durable: survives broker restart
		false,         // Auto-deleted: don't delete when unused
		false,         // Internal: accessible directly by producers
		false,         // No-wait: wait for broker confirmation
		nil,           // Additional arguments
	)
	failOnError(err, "Gagal mendeklarasikan Topic Exchange")

	// 4. Declare a Queue for the Notification Service (All Order Events)
	notificationQueue, err := ch.QueueDeclare(
		"notifications-service-queue", // Queue name
		true,                          // Durable
		false,                         // Auto-delete
		false,                         // Exclusive
		false,                         // No-wait
		nil,
	)
	failOnError(err, "Gagal mendeklarasikan antrean notifikasi")

	// 5. Bind the Notification Queue with a Wildcard Pattern Binding Key
	// We want to capture all actions related to the 'order' entity
	notificationBindingKey := "sales.order.*"
	err = ch.QueueBind(
		notificationQueue.Name,  // Destination queue name
		notificationBindingKey,   // Binding key pattern
		exchangeName,            // Source exchange name
		false,
		nil,
	)
	failOnError(err, "Gagal mengikat antrean notifikasi")
	log.Printf("✓ Antrean %s terikat ke %s dengan pola: %s", notificationQueue.Name, exchangeName, notificationBindingKey)

	// 6. Declare a Queue for the Global Analytics Service (All Sales Events)
	analyticsQueue, err := ch.QueueDeclare(
		"analytics-service-queue", // Queue name
		true,                       // Durable
		false,                      // Auto-delete
		false,                      // Exclusive
		false,                      // No-wait
		nil,
	)
	failOnError(err, "Gagal mendeklarasikan antrean analitik")

	// 7. Bind the Analytics Queue with the '#' Binding Key (All Events under sales)
	analyticsBindingKey := "sales.#"
	err = ch.QueueBind(
		analyticsQueue.Name,
		analyticsBindingKey,
		exchangeName,
		false,
		nil,
	)
	failOnError(err, "Gagal mengikat antrean analitik")
	log.Printf("✓ Antrean %s terikat ke %s dengan pola: %s", analyticsQueue.Name, exchangeName, analyticsBindingKey)

	// 8. Producer Publishes Message 1: sales.order.created
	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
	defer cancel()

	payload1 := []byte(`{"order_id":"ORD-98754","total":150000,"status":"created"}`)
	routingKey1 := "sales.order.created"
	err = ch.PublishWithContext(ctx,
		exchangeName,
		routingKey1,
		false,
		false,
		amqp.Publishing{
			ContentType:  "application/json",
			DeliveryMode: amqp.Persistent, // Message safely written to disk
			MessageId:    "ORD-98754",      // Unique ID placed in MessageId metadata
			Timestamp:    time.Now(),
			Body:         payload1,
		},
	)
	failOnError(err, "Gagal mempublikasikan pesan 1")
	log.Printf("✓ Pesan 1 terkirim dengan Routing Key: %s", routingKey1)

	// 9. Producer Publishes Message 2: sales.payment.processed
	payload2 := []byte(`{"order_id":"ORD-98754","payment_method":"go-pay","status":"processed"}`)
	routingKey2 := "sales.payment.processed"
	err = ch.PublishWithContext(ctx,
		exchangeName,
		routingKey2,
		false,
		false,
		amqp.Publishing{
			ContentType:  "application/json",
			DeliveryMode: amqp.Persistent,
			MessageId:    "PAY-11223",
			Timestamp:    time.Now(),
			Body:         payload2,
		},
	)
	failOnError(err, "Gagal mempublikasikan pesan 2")
	log.Printf("✓ Pesan 2 terkirim dengan Routing Key: %s", routingKey2)
}

Code Execution Result Analysis: #

1. Message 1 (sales.order.created) has three segments. It is delivered to notifications-service-queue (because it matches sales.order.*) and also to analytics-service-queue (because it matches sales.#).
2. Message 2 (sales.payment.processed) has three segments. It is only delivered to analytics-service-queue (because it matches sales.#, which captures all routes starting with sales). The message doesn’t enter notifications-service-queue because the second segment is payment, not order.

Anti-Patterns to Avoid #

The Topic Exchange offers very high flexibility. However, if not used carefully, it can backfire, degrading performance and damaging system stability.

1. Inserting Dynamic IDs (UUID/Transaction IDs) in the Routing Key #

This is the most common design mistake made by developers new to RabbitMQ. They insert unique dynamic IDs into the Routing Key string to make it look specific.

// ANTI-PATTERN: Including unique dynamic IDs in the Topic Exchange routing key
func PublishOrderEventBad(ch *amqp.Channel, orderID string, payload []byte) {
	// ✗ AVOID: Putting dynamic UUID/IDs into the routing key.
	// This makes the Trie structure bloat without limits because every transaction
	// creates a new tree branch in RAM, destroying Mnesia and ETS memory efficiency.
	routingKey := "sales.order.created." + orderID
	
	_ = ch.Publish(
		"sales.events.topic",
		routingKey,
		false,
		false,
		amqp.Publishing{
			DeliveryMode: amqp.Persistent,
			ContentType:  "application/json",
			Body:         payload,
		},
	)
}

Why is this dangerous? #

1. Trie Memory Leak: Every time a message is sent with a new unique Routing Key (e.g., containing a new UUID), RabbitMQ must evaluate that route against the Trie. If a queue is bound using a wildcard like sales.order.created.#, the broker must register a new route branch in RAM. Over time, the RAM used by ETS and the Mnesia database to store the Trie structure keeps bloating until it causes an Out of Memory (OOM) crash on the broker.
2. Lookup Performance Degradation: With new Trie nodes continuously added, the Trie tree’s depth and width become very large. This increases route lookup traversal time, reducing overall message delivery throughput.

Architectural Solution: #

Keep all dynamic IDs out of the Routing Key. The Routing Key must purely contain finite, static taxonomic classification strings. Transaction IDs or UUIDs must always go into the message’s binary payload or into the standard message_id metadata property or a custom header.

// CORRECT: Using a static classification routing key for the Topic Exchange
func PublishOrderEventGood(ch *amqp.Channel, orderID string, payload []byte) {
	// ✓ SOLUTION: Use a structured, static, finite routing key
	routingKey := "sales.order.created"
	
	_ = ch.Publish(
		"sales.events.topic",
		routingKey,
		false,
		false,
		amqp.Publishing{
			MessageId:    orderID, // Put the unique transaction ID in MessageID!
			DeliveryMode: amqp.Persistent,
			ContentType:  "application/json",
			Body:         payload,
		},
	)
}

2. Over-Wildcarding (Misusing the # Character) #

Some development teams use Topic Exchanges but register binding keys with the # (or #.) pattern on almost all their queues for practicality, to avoid managing binding keys strictly.

Why is this dangerous? #

If all queues are bound using the # wildcard, the Topic Exchange behaves exactly like a Fanout Exchange because it delivers every message to every queue. However, the CPU performance consumed is far greater because the broker must compile the Trie, tokenize the Routing Key string, and traverse the Trie tree in RAM for every message. This wastes broker CPU cycles needlessly.

Architectural Solution: #

If a data flow is truly designed to be spread to all queues without any filtering, use a Fanout Exchange explicitly. The Fanout Exchange skips the entire key matching process, delivering far higher throughput performance and CPU efficiency.

Performance Characteristic Comparison #

To help us determine when to use a Topic Exchange compared to other exchange types, let’s compare their internal performance characteristics:

From the comparison table above, we can conclude that the Topic Exchange is the best choice when we need hierarchy-based routing flexibility without sacrificing too much performance, as long as we can control the number of dynamic bindings and avoid metadata churning.

Summary #

• Wildcard Pattern Routing — The Topic Exchange matches message Routing Keys against queue Binding Key patterns using the asterisk wildcard (*) for exactly one word and the hash (#) for zero or more words.
• RAM Trie Tree Evaluation — Binding rules are stored in Mnesia RAM memory and projected to ETS tables as a Trie data structure. Route lookup is efficient with $O(L)$ complexity based on word segment length.
• Avoid Dynamic IDs in Routing Keys — Keep UUIDs, user IDs, or transaction IDs out of Routing Keys to prevent RAM memory leaks from Trie tree node bloat on the broker. Use message_id instead.
• Beware of Metadata Churning — Avoid high-speed dynamic bind and unbind operations at runtime to prevent the broker from constantly rebuilding Trie data structures on the CPU.
• Use Bounded Context Conventions — Apply the <domain>.<entity>.<action> pattern in lowercase to guarantee readable, scalable, clean API contract routing audits.
• Use Fanout If No Filtering — If messages are always distributed to all queues without route filtering, use a Fanout Exchange to save broker CPU usage.

← Previous: Fanout   Next: Headers →