Message Queues

A buffer between producers and consumers that decouples them, absorbs spikes as backlog, and lets work happen at the consumer's pace.

10 min read Watch it move Build it

A message queue is a buffer that sits between the parts of a system that *create* work (producers) and the parts that *do* it (consumers). A producer drops a message in and moves on instead of waiting for the work to finish — asynchronous decoupling. The queue absorbs spikes as a backlog, and adding more consumers drains it faster, so work happens at the consumer's own pace rather than the producer's.

Why decouple?

1A user uploads a video → the web server drops an "encode this" message on the queue and instantly returns "upload received". The user doesn't wait for the slow encode.
2A pool of consumers (encoder workers) pulls messages off and processes them in the background.
3A traffic spike — 10,000 uploads at once — just grows the backlog; nothing crashes. The queue is a shock absorber.
4Add more consumers to drain the backlog faster. Producers and consumers scale independently.

The broker runs the queue

The broker is the service that accepts messages, stores them durably, and hands them to consumers. Common ones: Kafka (high-throughput distributed log), RabbitMQ (flexible routing), and Amazon SQS (managed). Producers and consumers only ever talk to the broker — never directly to each other.

Delivery guarantees

What happens if a consumer crashes mid-processing? The answer is the queue's delivery guarantee. At-most-once: deliver and forget — a crash means the message is lost (fast, lossy). At-least-once: the broker redelivers until the consumer acknowledges success — nothing is lost, but a crash *after* the work but *before* the ack causes a duplicate. Exactly-once is the ideal but is genuinely hard end-to-end; in practice you get at-least-once delivery plus idempotent consumers, so processing a duplicate has no extra effect.

Design consumers to be idempotent

Because at-least-once is the practical default, assume every message may arrive more than once. Make handlers idempotent — dedupe on a message ID, or use INSERT ... ON CONFLICT DO NOTHING — so a redelivered "charge the card" message doesn't charge twice.

Backpressure

If producers persistently outrun consumers, the backlog grows without bound and eventually the broker runs out of storage. Backpressure is the set of mechanisms that push back: bounded queues that block or reject producers when full, slowing them to the consumers' pace; or a lag signal that triggers autoscaling of consumers. The goal is to keep the backlog *finite* rather than letting it silently balloon into an outage.

Kafka: partitions and offsets

Kafka models a queue as a topic split into partitions — each an append-only, ordered log. A message's position in a partition is its offset, a monotonically increasing number. Consumers don't have messages pushed and deleted; instead each consumer group tracks its own offset — the position it has read up to — and advances it as it processes. This is why Kafka can replay: rewind the offset and reread old messages.

topic "orders", partition 0 (append-only log):

 offset:  0    1    2    3    4    5    6
         [m0] [m1] [m2] [m3] [m4] [m5] [m6]
                        ^                  ^
              consumer committed        log end
              offset = 3 (read 0..2)

- ordering is guaranteed WITHIN a partition, not across them
- messages with the same key go to the same partition
- parallelism = number of partitions (one consumer per partition)

The partition key sets ordering and parallelism

Kafka guarantees order only *within* a partition. Route related messages (e.g. all events for one user) to the same partition by giving them the same key, so their order is preserved — while unrelated keys spread across partitions for parallelism. More partitions = more consumers working in parallel.

OperationTimeSpace

Producer enqueue · never blocks (until backpressure)O(1)O(backlog)

Drain rate · add consumers to clear backlogscales with # consumers—

Check yourself

A message queue gives at-least-once delivery. What must consumers handle, and how?