RabbitMQ as Database #

RabbitMQ is a very reliable tool for flowing messages quickly and efficiently between services. However, its ability to write messages to disk (persistent messages), replicate data to multiple nodes (quorum queues), and retain messages in queues for long periods often triggers fatal architectural misunderstandings. Many developer teams, especially those newly adopting asynchronous communication patterns, are tempted to treat RabbitMQ as if it were a primary database. They assume that as long as messages are in durable queues, their data is safe and can be accessed anytime like database table rows.

This approach is one of the most dangerous anti-patterns in distributed system architecture. Making a message broker a long-term data store not only drastically degrades system performance, but also endangers data integrity itself. This article deeply dissects why treating RabbitMQ as a database is an expensive design mistake, what technical consequences lurk under the hood, and how we should design correct data storage architectures using the Transactional Outbox pattern.

Why Are Developers Tempted to Use RabbitMQ as a Database? #

In the early phases of application development, implementation speed is often the top priority. When we need a place to store activity logs, transaction histories, or audit trails, standing up a new database (whether SQL or NoSQL) feels like adding infrastructure burden and code complexity. This is where the temptation to use RabbitMQ queues appears.

Several factors usually behind this bad decision include:

1. Durable Queues and Persistent Messages: RabbitMQ features allowing queues to survive broker restarts (durable) and messages to be written to disk (persistent) make developers feel their data has safety guarantees equivalent to database storage.
2. Quorum Queues: The presence of Raft algorithm-based consensus replication on Quorum Queues gives the illusion of high availability and fault tolerance equivalent to distributed database clusters like MongoDB or Cassandra.
3. Ease of Access: Developers think that by piling messages in queues without sending acknowledgements (ACKs), those messages stay stored there and can be read by other workers at any time.
4. Reluctance to Manage State: Many developers want to avoid state synchronization complexity between databases and message brokers, so they decide to make the queue the only single source of truth.

Although it looks practical at small scales, the assumptions above ignore the fundamental principles of message broker design. RabbitMQ is designed from the ground up as a transient message processing system, not an immutable data storage engine.

Fundamental Philosophical Differences: Broker vs Database #

To understand why this anti-pattern is so damaging, we must review the fundamental philosophical differences between a message broker and a database from a software architecture perspective.

RabbitMQ’s Destructive Read philosophy is the key to its efficiency. RabbitMQ queues work optimally when their size approaches zero. That means every incoming message must be immediately pulled by consumers, processed, and deleted from broker memory. Conversely, databases are designed to hold ever-growing accumulated data over time and provide fast index search mechanisms to randomly recall specific data.

The Anatomy of Long-Term Message Storage Dangers in RabbitMQ #

When we violate the philosophical boundaries above and let millions of messages pile up in RabbitMQ queues long-term, we trigger a chain reaction degrading the entire broker cluster’s performance. Here are the technical details of what happens inside RabbitMQ:

1. Ballooning Broker RAM Consumption (Erlang GC Overhead) #

RabbitMQ is written using the Erlang programming language and runs on the Erlang BEAM Virtual Machine. For every queued message, RabbitMQ stores that message’s metadata (like message ID, index position, and delivery status) in RAM memory so the broker can flow it instantly when consumers request it.

If we let millions of messages pile up in one queue, the RAM memory used to manage these indexes balloons exponentially. The Erlang VM relies on per-process Garbage Collection (GC) mechanisms. When queues become very long, the GC process takes longer and consumes large CPU cycles just scanning memory, ultimately triggering high latency for other message publishing processes.

2. Disk Paging Slowdowns (High Memory Watermark) #

To prevent RAM memory exhaustion, RabbitMQ has a safety mechanism called the High Memory Watermark. By default, this limit is set at 40% of the machine’s total available RAM. If broker memory consumption exceeds this limit due to accumulating queue metadata:

1. RabbitMQ freezes all producer connections (block publishers). This is a form of flow control to stop new message inflow rates.
2. The Erlang VM starts paging to disk. All message contents originally in RAM are force-written to physical disk storage to free memory space.

This paging process runs very slowly because it involves synchronous disk I/O operations. As a result, broker throughput drops drastically, and our systems relying on RabbitMQ for real-time communication experience latency cascades (successive latency failures).

3. Very Slow Startup Recovery Times #

If the RabbitMQ broker crashes or is deliberately restarted for maintenance, and inside it there are queues holding millions of persistent messages:

• At startup, RabbitMQ must re-read all message index files from disk to RAM memory to reconstruct queue status.
• This index reconstruction process can take tens of minutes to hours, depending on disk speed and message count.
• During this index recovery process, the broker can’t serve new connections, resulting in very long distributed system downtime.

4. Write Amplification on Quorum Queues #

If we use Quorum Queues to store large amounts of long-term messages, the situation is much worse. Quorum Queues rely on Raft replication logs written sequentially to disk on every cluster node.

Every un-ACKed message stays in the active Raft log. This prevents the broker from performing log truncation (cutting unnecessary historical logs). As a result, disk space on all cluster nodes balloons quickly from the write amplification effect, and inter-node synchronization performance slows down because of excessively large Raft consensus metadata sizes.

Why RabbitMQ Lacks Database Key Capabilities #

Besides internal broker performance issues, from the application functionality side, RabbitMQ completely lacks the basic features a database storage system must have:

1. No Random Query Capability #

In databases, we can easily find specific data using SQL queries like:

SELECT * FROM orders WHERE user_id = 456 AND status = 'failed' ORDER BY created_at DESC;

In RabbitMQ, we can’t do random queries like that. The only way to read messages is consuming them sequentially from the queue’s front position (FIFO). We can’t ask the broker to “take message number 5,000 in the middle of the queue without taking messages 1 through 4,999”. Trying to work around this by taking all messages, filtering them in the application, and requeueing non-matching messages back to the broker is a massive performance disaster.

2. No Dynamic Indexing #

Databases use B-Tree or LSM-Tree structures to index data columns so query searches run in constant or logarithmic time. RabbitMQ has no column index concept. All message payloads are stored as raw binary blobs. The broker doesn’t know and doesn’t care what’s inside those messages (whether JSON, XML, or Protocol Buffers), so there’s no way for the broker to index internal payload properties for fast queries.

3. No Backups and Point-in-Time Recovery (PITR) #

Production databases have periodic backup mechanisms, read-only replica replication, and Write-Ahead Log (WAL) recording to recover data to a specific second before incidents occur (Point-in-Time Recovery). RabbitMQ isn’t designed for partial queue data backups. If a queue is corrupted or accidentally deleted through the Management UI by an administrator, all data inside it disappears instantly and can’t be partially recovered through standard database backup mechanisms.

The Correct Architecture Pattern: Separation of Concerns #

To avoid this architectural trap, we must apply the Separation of Concerns principle. We must treat RabbitMQ purely as a transport layer for sending instructions or notifying state changes (event notifications), while databases remain the storage layer as the only data source of truth.

The standard industry design pattern solving database and message broker integration safely is the Transactional Outbox Pattern.

How Does the Transactional Outbox Pattern Work? #

Instead of publishing messages to RabbitMQ directly in the middle of application business processes (which risks failure if the broker is down or connections drop), we do the following steps:

1. Write to the Main Table & Outbox Table: In one ACID database transaction (transactionally safe), we store the main business data (e.g., the orders table) and simultaneously write the event metadata to be sent into a special table named outbox in the same database.
2. Commit the Transaction: The database transaction is committed. Because both writes are in the same transaction, we’re absolutely guaranteed the business data and outbox event are stored together, or both fail entirely (all-or-nothing).
3. Outbox Publisher: An independent background process (Outbox Publisher or Message Relay) periodically reads the outbox table (using index query polling techniques or CDC - Change Data Capture), publishes those messages to RabbitMQ, and deletes or marks outbox messages as “sent” after receiving publication confirmations (Publisher Confirms) from RabbitMQ.
4. Consumer Processing: Consumers read messages from RabbitMQ, process them, update their own local databases, and send ACKs to RabbitMQ to delete those messages from queues as quickly as possible.

With this pattern, RabbitMQ never stores historical backlog data. RabbitMQ queues only function as dynamic temporary delivery pipes.

Data Flow Architecture Relationship: Outbox Pattern #

The following diagram illustrates the transactional write workflow using the Transactional Outbox Pattern, separating permanent state storage media (Database) from temporary message delivery media (RabbitMQ).

flowchart TD
    subgraph Aplikasi Produsen ["Producer Application (Microservice A)"]
        A[Business Logic] -->|1. Start ACID Transaction| B[(Main Database)]
        A -->|2. Write Business Table e.g. Orders| B
        A -->|3. Write Outbox Table e.g. Outbox| B
        A -->|4. Commit Transaction| B
    end

    subgraph Relay["Relay Process (Outbox Publisher)"]
        C[Outbox Publisher Loop] -->|5. Polling / CDC unsent data| B
        C -->|6. Send Event| D[RabbitMQ Exchange]
        D -->|7. Route Message| E[RabbitMQ Queue]
        D -. "8. Publisher Confirm" .-> C
        C -->|9. Mark Outbox as Sent| B
    end

    subgraph Aplikasi Konsumen ["Consumer Application (Microservice B)"]
        F[Consumer Worker] -->|10. Pull FIFO Message| E
        F -->|11. Process & Update Local DB State| G[(Consumer Database)]
        F -->|12. Send ACK| E
    end

    style B stroke:#333,stroke-width:2px
    style G stroke:#333,stroke-width:2px
    style E stroke:#0288d1,stroke-width:2px
    style D stroke:#0288d1,stroke-width:2px

Implementing the Transactional Outbox Pattern in Go Code #

Here is a complete implementation example in Go. This code demonstrates how we store order data and outbox messages into a PostgreSQL database using one safe SQL transaction, and how the background relay process reliably sends those outbox messages to RabbitMQ.

package main

import (
	"context"
	"database/sql"
	"encoding/json"
	"fmt"
	"log"
	"time"

	_ "github.com/lib/pq"
	amqp "github.com/rabbitmq/amqp091-go"
)

// Order represents our main business entity data.
type Order struct {
	ID        string    `json:"id"`
	UserID    string    `json:"user_id"`
	Amount    float64   `json:"amount"`
	CreatedAt time.Time `json:"created_at"`
}

// OutboxEvent represents the outbox table schema for asynchronous event delivery.
type OutboxEvent struct {
	ID         int64
	EventType  string
	Payload    []byte
	CreatedAt  time.Time
	IsSent     bool
}

// OrderService manages order business operations.
type OrderService struct {
	db *sql.DB
}

// CreateOrder creates a new order using an ACID database transaction.
func (s *OrderService) CreateOrder(ctx context.Context, order Order) error {
	// 1. Start a database transaction
	tx, err := s.db.BeginTx(ctx, nil)
	if err != nil {
		return fmt.Errorf("gagal memulai transaksi: %w", err)
	}
	// Ensure rollback happens if a failure occurs before commit
	defer tx.Rollback()

	// 2. Write to the main table (orders)
	queryOrder := `INSERT INTO orders (id, user_id, amount, created_at) VALUES ($1, $2, $3, $4)`
	_, err = tx.ExecContext(ctx, queryOrder, order.ID, order.UserID, order.Amount, order.CreatedAt)
	if err != nil {
		return fmt.Errorf("gagal menulis data order: %w", err)
	}

	// Serialize the order data into a JSON payload for the outbox
	payload, err := json.Marshal(order)
	if err != nil {
		return fmt.Errorf("gagal serialisasi order payload: %w", err)
	}

	// 3. Write to the outbox table in the same transaction
	queryOutbox := `INSERT INTO outbox (event_type, payload, created_at, is_sent) VALUES ($1, $2, $3, false)`
	_, err = tx.ExecContext(ctx, queryOutbox, "order.created", payload, time.Now())
	if err != nil {
		return fmt.Errorf("gagal menulis data outbox: %w", err)
	}

	// 4. Commit the database transaction atomically
	if err := tx.Commit(); err != nil {
		return fmt.Errorf("gagal commit transaksi: %w", err)
	}

	log.Printf("[OrderService] Sukses menyimpan Order ID %s dan Outbox event ke DB", order.ID)
	return nil
}

// OutboxPublisher is tasked with reading the outbox table and relaying messages to RabbitMQ.
type OutboxPublisher struct {
	db         *sql.DB
	rabbitConn *amqp.Connection
	ch         *amqp.Channel
}

// Run starts the periodic polling process to send outbox messages.
func (p *OutboxPublisher) Run(ctx context.Context, interval time.Duration) {
	ticker := time.NewTicker(interval)
	defer ticker.Stop()

	for {
		select {
		case <-ctx.Done():
			return
		case <-ticker.C:
			if err := p.publishPendingEvents(ctx); err != nil {
				log.Printf("[OutboxPublisher] Error saat memproses event tunda: %v", err)
			}
		}
	}
}

func (p *OutboxPublisher) publishPendingEvents(ctx context.Context) error {
	// Get the list of unsent events from the database
	querySelect := `SELECT id, event_type, payload FROM outbox WHERE is_sent = false ORDER BY id ASC LIMIT 10`
	rows, err := p.db.QueryContext(ctx, querySelect)
	if err != nil {
		return fmt.Errorf("gagal query pending outbox: %w", err)
	}
	defer rows.Close()

	var events []OutboxEvent
	for rows.Next() {
		var ev OutboxEvent
		if err := rows.Scan(&ev.ID, &ev.EventType, &ev.Payload); err != nil {
			return fmt.Errorf("gagal scan outbox row: %w", err)
		}
		events = append(events, ev)
	}

	if len(events) == 0 {
		return nil // No pending messages
	}

	// Send each event to RabbitMQ
	for _, ev := range events {
		// Publish the message with Publisher Confirms enabled
		err = p.ch.PublishWithContext(ctx,
			"order.events", // exchange name
			ev.EventType,    // routing key
			true,            // mandatory
			false,           // immediate
			amqp.Publishing{
				ContentType:  "application/json",
				DeliveryMode: amqp.Persistent, // Make sure the message is written to disk at the broker
				Body:         ev.Payload,
				Timestamp:    time.Now(),
			},
		)
		if err != nil {
			log.Printf("[OutboxPublisher] Gagal mengirim Event ID %d ke RabbitMQ: %v", ev.ID, err)
			continue // Continue with the next event, retry the failed one later
		}

		// After successfully sending to RabbitMQ, mark the event in the database as sent
		queryUpdate := `UPDATE outbox SET is_sent = true WHERE id = $1`
		_, err = p.db.ExecContext(ctx, queryUpdate, ev.ID)
		if err != nil {
			log.Printf("[OutboxPublisher] Peringatan: Event ID %d terkirim ke RabbitMQ tapi gagal update status DB: %v", ev.ID, err)
			// Note: This potentially causes duplicate message delivery (at-least-once).
			// Our consumers must be idempotent to handle it.
		} else {
			log.Printf("[OutboxPublisher] Event ID %d sukses direlay ke RabbitMQ dan diperbarui di DB", ev.ID)
		}
	}

	return nil
}

func main() {
	// Example connection initialization (illustrative)
	db, err := sql.Open("postgres", "postgres://user:***@localhost:5432/dbname?sslmode=disable")
	if err != nil {
		log.Fatalf("Gagal koneksi ke PostgreSQL: %v", err)
	}
	defer db.Close()

	conn, err := amqp.Dial("amqp://guest:***@localhost:5672/")
	if err != nil {
		log.Fatalf("Gagal koneksi ke RabbitMQ: %v", err)
	}
	defer conn.Close()

	ch, err := conn.Channel()
	if err != nil {
		log.Fatalf("Gagal membuka channel RabbitMQ: %v", err)
	}
	defer ch.Close()

	// Enable Publisher Confirms mode so the relay gets write certainty from the broker
	if err := ch.Confirm(false); err != nil {
		log.Fatalf("Gagal mengaktifkan Publisher Confirms: %v", err)
	}

	service := &OrderService{db: db}
	publisher := &OutboxPublisher{
		db:         db,
		rabbitConn: conn,
		ch:         ch,
	}

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	// Run the relay publisher in a background goroutine
	go publisher.Run(ctx, 2*time.Second)

	// Simulate a new order creation by the application
	newOrder := Order{
		ID:        "order-99120",
		UserID:    "user-887",
		Amount:    150000.0,
		CreatedAt: time.Now(),
	}

	if err := service.CreateOrder(ctx, newOrder); err != nil {
		log.Printf("Gagal membuat order: %v", err)
	}

	// Block the main thread so the background goroutine keeps running
	time.Sleep(5 * time.Second)
}

Comprehensive Comparison: RabbitMQ vs Database #

The table below details the operational capability differences and failure tolerance limits between the RabbitMQ message queue system and transactional databases (like PostgreSQL or MySQL).

Review Checklist: Are We Using RabbitMQ as a Database? #

When auditing system architecture, we must check whether there are indications of misusing RabbitMQ as a database. Evaluate our system using the checklist below:

BROKER MISUSE INDICATORS:
  □ Is there a queue deliberately left holding messages without active consumers for weekly/monthly periods?
  □ Do we deliberately disable the ACK confirmation mechanism (auto-ack = false) without ever calling basic.ack so messages stay "safe" in the queue?
  □ Does our application try to read messages from the middle of queues using pulling techniques then manually filtering them in application RAM memory?
  □ Does the total un-ACKed message data size in RabbitMQ exceed our broker server's RAM memory capacity?
  □ Is our only business transaction audit log history stored solely in RabbitMQ queues without a permanent copy in SQL/NoSQL databases or cold storage?

SOLUTIONS IF THE ANSWER IS YES:
  □ Apply the Transactional Outbox Pattern to separate database writes and event publications.
  □ Create consumer workers that immediately process messages, store them in relational/NoSQL databases, then send ACKs to the broker within milliseconds.
  □ Use RabbitMQ queues purely as transient transport pipes with an ideal queue size target as close to zero as possible.

Summary #

• Destructive Read — RabbitMQ’s core philosophy is flowing transient data messages. Messages are immediately deleted once successfully consumed and ACKed. RabbitMQ isn’t designed to retain long-term data.
• Disk Paging Dangers — Piling millions of messages in queues exhausts broker RAM, triggers High Memory Watermark limits, activates very slow disk paging queries, and freezes producer connections (flow control).
• Lack of Query Features — RabbitMQ has no query engine, SQL queries, dynamic index creation, random search queries, or backup and Point-in-Time Recovery (PITR) mechanisms like real databases.
• Slow Boot Recovery — Excessively large persistent message backlogs slow down post-crash broker startup processes because RabbitMQ must rebuild queue indexes from disk to RAM memory.
• Transactional Outbox Pattern — The correct architecture solution for guaranteeing event delivery without burdening the broker is writing main data and outbox data to a database in one ACID transaction, then using an independent relay worker to publish them to RabbitMQ.
• Separation of Concerns — Separate temporary message delivery media (RabbitMQ) from permanent state storage media (PostgreSQL, MySQL, MongoDB, or Object Storage).

Closing #

RabbitMQ provides extraordinary ease in connecting various distributed services asynchronously. However, this ease demands strict design discipline from system architects. Using RabbitMQ as a primary database is a shortcut trading short-term ease at the start for system fragility, data loss risks, and performance degradation disasters in the future.

Remember this basic principle: Messages in RabbitMQ should carry instructions or state-change notifications — not become the storage place for the state itself.

By respecting the responsibility boundary between message brokers (transport) and databases (storage) through design patterns like the Transactional Outbox, we can build distributed systems that not only have high performance, but are also stable, resilient, and easy to maintain long-term.

← Previous: Throughput & Scalability   Next: Over Queue →