Amazon S3 launched in 2006. By 2013 it held two trillion objects; by 2021, over a hundred trillion.

It is the substrate under a remarkable amount of this series — routing tiles, video segments, email attachments, Kafka’s tiered storage. Every time an earlier design said “put it in object storage and forget about it,” this is what it was leaning on.

So the interesting question is: how do you promise eleven nines of durability on hardware that fails constantly?

A message queue sits between two services so they don’t have to know about each other. The producer writes and moves on; the consumer reads when it’s ready. Neither has to be up when the other is.

That buys you four things: decoupling, independent scaling of each side, availability when one side is down, and asynchronous communication so nobody blocks.

Simple enough to describe in a sentence. The design is not simple at all, and the reason is that we’re going to build the harder version: not just a queue that hands messages over and forgets them, but one that retains everything for two weeks and lets consumers read it again from any point.

You have a 4 GB video file in your Drive folder. You change its title — a few dozen bytes near the start.

How many bytes should cross the network?

The naive answer is 4 GB. The right answer is a few hundred kilobytes, and getting from one to the other is what this chapter is about. Everything else — the API servers, the metadata database, the notification service — is machinery you have already seen. The distinctive problems here are three:

The news feed we built last chapter can be seconds stale and nobody notices. Chat inverts every one of those assumptions.

A message that arrives thirty seconds late is a broken product. A message that arrives twice is a visible bug. A message that arrives out of order makes a conversation nonsensical — the reply shows up before the question. And unlike a feed, where the user pulls, the server must push, to a client that may be behind a firewall, on a train, or asleep.

Every social product has the same question at its centre, and it has exactly two answers:

Do you build a user’s feed when someone posts, or when that user opens the app?

That is it. Everything else — the caches, the queues, the graph database — follows from which side you pick. And the reason this is a great interview question is that both answers are wrong, in ways that only become visible when you do the arithmetic.

Sending a notification is one HTTP call to Apple or Google. That is the entire mechanism, and you can demo it in five minutes.

The system design question is not about that call. It is about what happens when you make it sixteen million times a day, to devices that may have been wiped, through providers that fail in ways you do not control, for users who will uninstall your app if you get it wrong.

The algorithm for a web crawler fits on a napkin:

1. Take a URL off a queue.
2. Download the page.
3. Extract its links.
4. Put the new ones back on the queue. Repeat.

Write that and you have a crawler. Point it at the open web and within about ten minutes you will have been rate-limited, IP-banned, trapped in an infinitely deep calendar page, and served the same article eleven times under eleven different URLs.

A URL shortener looks like the easiest system design question you will ever get. Store a mapping, hand back a short string, redirect. You could write it in an afternoon.

That is exactly why it gets asked. The naive version really is trivial — so the interview is not about whether you can build it. It is about whether you notice the four decisions hiding inside the triviality:

1. How short can the code be? Not a guess — an arithmetic answer from the traffic estimate.
2. How do you generate the code? Hash the URL, or encode a counter? They fail in completely different ways.
3. 301 or 302? One of these silently destroys your analytics and makes links impossible to change. Most candidates pick it.
4. What stops your service becoming a phishing tool? Every real shortener spends more engineering effort here than on the shortening.

We will build it properly, in the order an interviewer expects, and then cover the production concerns the textbook treatment leaves out.

Every row in your database needs a name. For years that name came from one line of SQL:

CREATE TABLE orders (
  id BIGINT PRIMARY KEY AUTO_INCREMENT,
  ...
);

The database hands out 1, 2, 3, 4. They are unique, they are sortable, they are small. It is a solved problem — right up until the moment you have two databases.

Then it stops working, quietly and catastrophically. Both databases happily hand out ID 1. Two different orders, same identifier. Your foreign keys now point at the wrong rows, and no error was raised anywhere.

Amazon DynamoDB stores hundreds of trillions of items and handles tens of millions of requests per second at peak. Netflix uses it to keep track of what you were watching. Airbnb uses it for availability calendars. Discord uses it for message storage.

What do all of these have in common? They all need to store and retrieve data by a simple key — blazingly fast, at global scale, with near-zero downtime. That’s what a key-value store does.