Message Queues & Async Processing
2 min read
Message Queues & Async Processing
Not everything must happen now. A message queue lets a producer hand off work and move on, while consumers process it later.
Producer ──▶ [ Queue: job1, job2, job3 ] ──▶ Consumer(s)
Why queue?
- Decoupling — producer and consumer scale and fail independently.
- Buffering / load levelling — absorb traffic spikes; workers drain the backlog at their own pace.
- Async work — send emails, transcode video, generate reports without blocking the user.
- Retries & durability — a failed job stays in the queue to try again.
Queues vs pub/sub
- Queue (point-to-point): each message is processed by one consumer. (RabbitMQ, SQS)
- Pub/Sub (topic): each message is delivered to all subscribers. (Kafka, SNS, Redis pub/sub)
Kafka in one paragraph
Kafka is a distributed, append-only commit log. Producers append to topics split into partitions; consumers read at their own offset. It's built for very high throughput and replaying history — the backbone of event-driven and streaming architectures.
Delivery guarantees
| Guarantee | Meaning | Cost |
|---|---|---|
| At-most-once | May lose messages | Simplest |
| At-least-once | May duplicate messages | Common default |
| Exactly-once | No loss, no dupes | Hardest / limited |
With at-least-once, make consumers idempotent — processing the same message twice must be safe (e.g. dedupe by a job ID).
Watch out for
- Poison messages — a job that always fails. Route it to a dead-letter queue after N retries.
- Ordering — global ordering is expensive; Kafka only guarantees order within a partition.
- Backpressure — monitor queue depth; a growing backlog means consumers can't keep up.