Scalability, Latency & Throughput
1 min read
Speaking the Language of Scale
Before designing anything, you need precise words for "fast" and "big".
Latency vs throughput
- Latency — how long a single request takes (ms). Lower is better.
- Throughput — how many requests you handle per second (QPS). Higher is better.
They are not the same. A system can have high throughput and high latency (a busy batch pipeline), or low latency and low throughput (a fast single-threaded service).
Optimise for tail latency (p99, p99.9), not the average. A p50 of 20 ms means nothing if 1% of users wait 3 seconds.
Vertical vs horizontal scaling
| Vertical (scale up) | Horizontal (scale out) | |
|---|---|---|
| How | Bigger machine | More machines |
| Ceiling | Hardware limit | Practically unlimited |
| Complexity | Low | High (coordination, state) |
| Fault tolerance | Single point of failure | Redundant by design |
Real systems scale horizontally — which forces the hard questions: how do you split the data, route requests and stay consistent?
Back-of-the-envelope numbers to memorise
| Operation | Rough time |
|---|---|
| Main memory reference | ~100 ns |
| SSD random read | ~150 µs |
| Round trip within a datacenter | ~500 µs |
| Read 1 MB from SSD | ~1 ms |
| Round trip across the Atlantic | ~150 ms |
A day has ~86,400 seconds (≈ 10⁵). So 1M writes/day ≈ 12 writes/second average — but plan for peaks 5–10× higher.