Queue vs Topic vs Log: Picking the Right Message Broker
Explore the differences between Queue, Topic, and Log broker patterns to choose the right architecture for your software systems.
Stock photo for illustration only, not from the actual event
- Broker decisions involve much more than just throughput and price metrics.
- Queues are ideal for background jobs needing retry logic and dead-letter handling.
- Topics are designed for fan-out broadcasting to multiple independent subscribers.
- Logs suit use cases requiring historical retention and message replayability.
Most broker decisions are made in a brief ten-minute conversation that concludes with adopting Kafka or selecting SQS simply because it is cheap. Both approaches treat the broker strictly as a throughput-and-price decision, which it is not. What you are actually choosing is a contract defining message ownership after delivery, consumer offset tracking, and whether yesterday's events persist tomorrow.
Getting this contract wrong means no amount of system tuning will fix it. A team that chooses a queue and later requires a new service to bootstrap from the past 30 days of order events will have nothing to replay. Conversely, a team that adopts Kafka for simple background tasks ends up managing partition planning, rebalances, and head-of-line blocking just to achieve standard queue functionality.
Stock photo for illustration only, not from the actual event
The delivery models differ fundamentally across the three primitives:
- Queue (Competing consumers): Each message goes to one handler and is deleted upon acknowledgment.
- Topic (Fan-out): Every subscription receives its own copy, deleted per subscription upon acknowledgment.
- Log (Retained stream): Every consumer group reads the whole stream independently and messages are retained until retention expires.
The final operational detail often surprises teams. In a queue, a failing message at index 7 does not block index 8. In a log, a consumer offset means everything prior is complete, so a persistently failing message will stall its entire partition unless custom retry topics are engineered.
Comprehending these three foundational architectural patterns is vital for scalable system design. Misjudging your broker requirements early creates deep technical debt. Modern production environments frequently combine these patterns, utilizing a log as the foundational source of record while coupling queues downstream to handle isolated retries and failure handling.
Three mapping details drive practical system designs:
- SNS and Service Bus: Azure topic subscriptions function as dedicated queues with independent locks and dead-letter sub-queues, whereas AWS pushes via SNS to individual SQS queues per consumer.
- EventBridge and Event Grid: These are cloud routers rather than heavy event backbones, ideal for SaaS reactions but unsuited for high-volume internal event streams requiring strict ordering.
- Partition count: This is a permanent architectural decision rather than a simple tuning knob. Kafka partitions scale up but cannot be reduced, while Event Hubs fixed sizing must account for recovering consumer requirements.
When multiple services require the exact same event, publishing once to a topic and provisioning individual queues for every consumer guarantees isolation, dedicated dead-letter queues, and independent scaling. Reach for a log streaming architecture when historical retention is mandatory, allowing new consumers to parse historical data and stream processors to maintain ordered per-key states.
Production solutions often combine these patterns. A common architecture utilizes a log as the system-of-record backbone paired with downstream queues for consumers requiring per-message retries and dead-letter queues, ensuring a single poison message cannot stall an entire partition.
Source: Dev.to
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment