Message Queues & Async Processing
2 min read
Message Queues & Async Processing
Not everything must happen now. A message queue lets a producer hand off work and move on, while consumers process it later.
Producer ──▶ [ Queue: job1, job2, job3 ] ──▶ Consumer(s)
Why queue?
- Decoupling — producer and consumer scale and fail independently.
- Load levelling — absorb traffic spikes; workers drain the backlog at their own pace.
- Async work — send emails, transcode video, generate reports off the request path.
- Retries & durability — a failed job stays in the queue to try again.
Queues vs pub/sub
- Queue (point-to-point): each message goes to one consumer. (RabbitMQ, SQS)
- Pub/Sub (topic): each message goes to all subscribers. (Kafka, SNS)
Kafka in one paragraph
Kafka is a distributed, append-only commit log. Producers append to topics split into partitions; consumers read at their own offset. Built for very high throughput and replaying history.
Delivery guarantees
| Guarantee | Meaning |
|---|---|
| At-most-once | May lose messages |
| At-least-once | May duplicate messages (common default) |
| Exactly-once | No loss, no dupes (hardest) |
With at-least-once, make consumers idempotent — processing the same message twice must be safe (dedupe by a job ID).
Watch out for
- Poison messages — a job that always fails → route to a dead-letter queue.
- Ordering — Kafka only guarantees order within a partition.
- Backpressure — a growing queue depth means consumers can't keep up.