System Design Handbook · llms.txt
Picture a website where people post short entries, about 1 KB of text each, and read each other's. On launch day it has one user: you. Three years later it has 10 million users, who write a billion entries a month and read a hundred billion. This chapter grows that site on Amazon Web Services (AWS), one stage at a time, from a single virtual machine to a design that survives the loss of a data center and adds servers on its own when traffic rises.
The question is different from the other designs in this handbook. Nobody asks you to invent a clever algorithm. The interviewer wants to see how you move a system from small to large: measure it, find the part that breaks first, fix that part, and repeat. Each fix has a price, in money and in complexity, and the skill being tested is knowing when a fix is worth paying for.
Scalability tells the general story of growing from one server to many, with a recipe site called Crumb: vertical and horizontal scaling, load balancing, caching, replication, partitioning and queues. Read it first if those ideas are new. This chapter assumes them and focuses on the AWS services that implement each stage, how they behave, and what each one costs. The stages follow a well-known AWS conference talk from 2015, "Scaling Up to Your First 10 Million Users," updated to today's services.
The chapter follows the four steps from The system design interview, step by step. Step 3 builds the first box. Step 4 grows it through five stages.
The method for this whole question is a loop:
You don't need deep AWS knowledge to answer this question well unless the job asks for it. Almost every idea here has a non-AWS equivalent, and the chapter names the general concept next to each service.
A 30-day month has 30 × 86,400 = 2,592,000 seconds, about 2.6 million. Rounding it to 2.5 million makes the arithmetic easy and is off by about 4%.
A conversion table worth remembering, using 2.5 million seconds a month:
| Per second | Per month |
|---|---|
| 1 | 2.5 million |
| 40 | 100 million |
| 400 | 1 billion |
| 40,000 | 100 billion |
What these numbers tell us. The system is very read-heavy, so the big wins come from serving reads without touching the main database: caches, replicas and a content delivery network (CDN). The average write rate is modest. The trouble with writes comes later, from storage growth: 36 TB of content plus indexes, copied to every replica, eventually outgrows one database server. And 100 TB a month of outbound data is a real line on the bill. The estimation chapter covers this kind of arithmetic in more depth.
The design starts as small as it can be. One virtual server runs everything, and a DNS name points at it.
Everything after this is added in response to a measured problem. Step 4 walks this road:
| Stage | Rough users | What gets added | Problem it solves |
|---|---|---|---|
| Start | 1 | One EC2 instance, Elastic IP, Route 53 | Getting online |
| 1 | 100 | S3 for files, RDS for MySQL, a VPC | One box doing every job |
| 2 | 1,000 | Load balancer, two Availability Zones, web and app tiers, CloudFront | One web server is a bottleneck and a single point of failure |
| 3 | 100,000 | ElastiCache, read replicas | The database drowning in reads |
| 4 | 500,000 | Auto Scaling, automation, deeper monitoring | Paying for peak capacity all night |
| 5 | 10 million | Archiving, sharding, federation, DynamoDB, SQS and workers | Data and writes outgrowing one database |
The user counts are rough guides. Real systems hit these walls at very different sizes, depending on how heavy each request is. Measurements decide when to move to the next stage.
A read is a request such as GET /api/v1/posts/8812, answered with the post as JSON. A write is POST /api/v1/posts with about 1 KB of text in the body, answered with 201 Created and the new post's id. Behind both sits a small schema:
CREATE TABLE users (
id BIGINT PRIMARY KEY,
name VARCHAR(100) NOT NULL,
created_at TIMESTAMP NOT NULL
);
CREATE TABLE posts (
id BIGINT PRIMARY KEY,
user_id BIGINT NOT NULL REFERENCES users(id),
body VARCHAR(1024) NOT NULL, -- the ~1 KB of content per write
created_at TIMESTAMP NOT NULL,
INDEX idx_user_time (user_id, created_at)
);
With one or two users, the goals are simple: a single box for simplicity, a bigger box when that one fills up, and monitoring to show which resource is running out.
Run the web server on one EC2 instance, and store user data on the same instance in a MySQL database.
When the box gets busy, use vertical scaling: choose a bigger box. On AWS you stop the instance, change its type, for example from m5.large to m5.2xlarge, and start it again. From then on you pay the new type's rate. Plan for a few minutes of downtime while it's stopped. Your code doesn't change.
To know when to scale up, and which way, watch the basic metrics: CPU, memory (is the database's working set still in RAM?), disk I/O (are reads and writes queueing?) and network. CloudWatch collects metrics such as CPU and network traffic for every instance, every five minutes by default, or every minute if you turn on detailed monitoring. The CloudWatch agent, installed on the instance, adds memory and disk-space metrics from inside it. On the box itself, top and open-source tools such as Nagios, StatsD and Graphite do the same job.
Vertical scaling has two costs, covered in the Scalability chapter. Big instances get very expensive, and eventually there's no bigger one. Worse, one box has no redundancy and no failover: when it reboots for patches, the site is down. The alternative is horizontal scaling, adding more boxes, which is where Step 4 goes.
The data is relational, so start with MySQL on the same box. It gives you joins, transactions and mature tools, and your team almost certainly knows SQL. The 2015 talk gives the same advice: start with SQL, and move to NoSQL only when you need to, such as when you expect more than 5 TB of data in the first year or have an extremely data-intensive workload. Our numbers reach 12 TB in year one, so the question returns in stage 5. NoSQL and choosing a database covers the decision.
An EC2 instance can get a public IPv4 address automatically, but AWS releases it when the instance stops and assigns a new one when it starts. A reboot keeps the address. A stop and start, which is what a resize does, loses it.
An Elastic IP address is a static public IPv4 address that belongs to your account until you release it. It stays the same however often the instance it's attached to is stopped, started or replaced. That helps with failover: if the server dies, you start a replacement, move the Elastic IP to it, and users follow without any DNS change.
Since February 1, 2024, AWS charges $0.005 an hour for every public IPv4 address, attached or idle, about $3.65 a month each. Accounts get a default quota of five Elastic IPs per Region, and AWS recommends using them mainly for this remap-on-failure purpose.
Add a DNS record that maps the domain name to the Elastic IP. On AWS that's Route 53: a plain A record such as example.com → 203.0.113.10, with the simple routing policy. Route 53 also offers failover, weighted, latency-based and geolocation routing, which later stages can use. DNS and CDNs explains how a lookup works and why DNS changes take time to spread.
Open only the ports the site needs, using a security group, a virtual firewall attached to the instance. Anything without an allow rule is blocked:
| Direction | Port | Source or destination | Why |
|---|---|---|---|
| Inbound | 80 (HTTP) | Anywhere | Redirect visitors to HTTPS |
| Inbound | 443 (HTTPS) | Anywhere | The site itself |
| Inbound | 22 (SSH) | Your office's IP range only | Administration |
| Outbound | none | The server never starts connections |
Security groups are stateful: if a request is allowed in, its response is allowed out, so a server with no outbound rules still answers every visitor. A new security group starts with an outbound rule that allows everything, and you must delete it to stop the server from starting connections of its own. That limits what an attacker can do from a compromised server, such as downloading tools or sending your data elsewhere. The cost is that the server can't fetch patches either, so a common compromise is to allow outbound traffic only through a gateway you control, as stage 1 does.
Port 22 can go too. AWS Systems Manager Session Manager gives administrators a shell without any open inbound port, bastion host or SSH key. Access is granted with AWS Identity and Access Management (IAM) policies, and sessions can be logged. Security covers firewalls and least privilege in general.
Here's where the site ends up after all five stages:
Before adding anything, go back to the step 1 numbers and ask what breaks first:
The stages below add components in the order a growing site usually needs them. At each stage, your load tests might show that you need only one or two of the changes. In an interview, tie each component to the bottleneck it fixes and the price you pay for it.
What the measurements show. Load on the single box is rising. Profiling points at MySQL, which takes more and more memory and CPU, while user uploads fill the disk. Bigger instances have kept up so far, but they're getting expensive, and they can't scale the database and the web server separately: a database that needs memory forces you to buy CPU for the web server too.
Goals. Lighten the load on the box and let each part scale on its own: store static content in an object store, and move MySQL to its own server.
What it costs. The web server's code and configuration must change to point at the object store and the new database. Each new component needs securing. The AWS bill grows, which you should weigh against the cost of running, patching and backing up the same systems yourself.
Amazon S3 is a managed object store: you put files ("objects") into a container ("bucket") and fetch them by name over HTTP. Move user uploads, JavaScript, CSS, images and videos off the web server's disk and into S3.
S3's Standard storage class keeps every object on multiple devices across at least three Availability Zones. It's designed for 99.999999999% durability and 99.99% availability over a year. Since January 5, 2023, every new object is encrypted at rest by default, at no extra cost. Reads are strongly consistent, so an object is readable the moment its upload succeeds. You pay for what you use and never size a disk again.
Buckets are private by default, with Block Public Access turned on. Letting browsers fetch files straight from S3 at this stage means making a deliberate public exception. Stage 2 removes it by putting a CDN in front.
You could run MySQL yourself on a second EC2 instance. Amazon RDS (Relational Database Service) is usually the better default. It runs MySQL for you and handles backups, software patching, failure detection and recovery, while you still own the schema and query tuning. In return:
db.m7g.2xlarge to db.m7g.4xlarge, is a setting change.Encrypt data in transit (HTTPS from users, TLS from the web server to MySQL) and at rest (S3 and RDS encryption). Security explains both.
Then put everything inside a Virtual Private Cloud (VPC), a private network for your account, divided into subnets. Each subnet lives in one Availability Zone.
Apply the same pattern to every component that later stages add.
Servers in private subnets still need outbound access sometimes, for patches. A NAT gateway lets them start connections out while blocking any connection started from outside. It costs about $0.045 an hour (roughly $33 a month) plus $0.045 per gigabyte processed, in US East at the time of writing, and AWS suggests one per Availability Zone to avoid cross-zone traffic. Traffic to S3 or DynamoDB can skip it: a gateway endpoint for those services has no extra charge.
What the measurements show. The single web server is now the bottleneck at peak hours. Responses slow down, and at the worst moments the site goes down. The business also wants the site to survive a server failure, which it can't while there's one of everything.
Goals. Use horizontal scaling to handle the load and remove single points of failure: a load balancer, several web servers, two Availability Zones, separate web and application tiers, and a CDN.
A load balancer receives every request and passes it to one of several servers. On AWS that's Elastic Load Balancing, and for HTTP you'd pick an Application Load Balancer (ALB). If you run your own, such as HAProxy on EC2, you need at least two in an active-passive or active-active pair across zones, or the load balancer becomes the new single point of failure. Load balancers explains both setups.
A managed ALB is highly available by design. It runs a node in each Availability Zone you enable, and it requires at least two. It health-checks every server and stops routing to failed ones. It scales itself as traffic grows by changing its IP addresses, published in DNS with a 60-second time to live (TTL). So you point example.com at it with a Route 53 alias record, which follows those changes automatically, works at the bare domain where a CNAME can't, and is free to query.
Terminate TLS on the load balancer. Its HTTPS listener holds the certificate, which can come from AWS Certificate Manager (ACM), and decrypts requests before passing them on. The servers stop spending CPU on encryption, and certificates live in one place. TLS termination covers when to re-encrypt behind it.
The web servers can now move to private subnets, since the load balancer reaches them on private addresses, which also saves $0.005 an hour per public IP. The ALB itself costs $0.0225 an hour in US East, about $16 a month, plus $0.008 per Load Balancer Capacity Unit (LCU) hour. An LCU measures new connections, active connections, bytes processed and rule evaluations, and you're billed on whichever you use most.
An Availability Zone (AZ) is one or more data centers with separate power and networking. A Region's zones are many kilometers apart, all within 100 km of each other, so two servers in two zones rarely fail together.
The standby can't serve reads. Synchronous replication adds some latency to every commit. And you pay for the standby as a full instance, so Multi-AZ roughly doubles the database bill. Availability covers the pattern and its pitfalls.
So far each server both handles HTTP and runs business logic. Split them:
Now each tier is sized and configured on its own, and you can add application servers without adding web servers. You can split the application tier further, for example servers for the read APIs and others for the write APIs, so a burst of heavy writes can't slow reads. An internal load balancer sits between the tiers. Its nodes have only private addresses, so only the web tier can reach it. Later diagrams leave it out to reduce clutter.
A CDN keeps copies of files close to users. Amazon CloudFront sends each request to the edge location with the lowest latency. If that location has the file, it answers at once. If not, it fetches the file from the origin, here the S3 bucket, and keeps a copy, by default for 24 hours unless the origin's headers say otherwise. CloudFront can cache some dynamic content too, such as pages that are the same for every visitor, which takes load off the web tier and cuts latency.
Lock the bucket down with origin access control (OAC), so its policy allows reads only from your CloudFront distribution and browsers can't skip the CDN. CloudFront charges for data sent to users and for requests, and fetches from AWS origins such as S3 are free. DNS and CDNs covers caching rules and invalidation.
What the measurements show. The site is read-heavy, 100 reads per write, and the database suffers under the reads. Query latency climbs at peak and the primary's CPU stays high.
Goals. Take load off MySQL with a memory cache and read replicas, and add web and application servers to keep responses fast. Again, profiling may show you need only some of these.
Memory is much faster than storage. In the widely shared table of latency numbers every programmer should know, reading 1 MB sequentially takes about 250 microseconds from memory, about 1 millisecond from an SSD (4 times as long) and about 20 milliseconds from a spinning disk (80 times as long).
First, give MySQL's own cache enough memory. MySQL's InnoDB engine keeps recently used table and index pages in its buffer pool, and its manual notes that dedicated servers often give it up to 80% of physical memory. If the hot data fits, reads stop touching disk, and that may be enough. Older advice to tune MySQL's query cache no longer applies: MySQL 8.0 dropped it because it didn't scale on multi-core machines, as Relational databases explains.
Then add a cache cluster. Amazon ElastiCache runs Valkey, Redis OSS or Memcached for you, as a serverless cache that scales itself or as nodes you size. Put two kinds of data in it:
A rough size: new content arrives at 1 TB ÷ 30 ≈ 33 GB a day. If most reads hit the last week's posts, the hot set is about 7 × 33 GB ≈ 230 GB, plus sessions, so plan for a cluster that spreads keys over several nodes. Choose Valkey or Redis OSS for replicas with automatic failover, which Memcached lacks, so one failed node doesn't empty its share of the cache.
The cost is another system to run and pay for, plus new bugs: stale data after a write, and a flood of misses to the database when the cache is cold or fails. Caching covers both.
Alongside the cache, read replicas take reads off the primary, the one server that accepts writes. RDS builds each replica from a snapshot and keeps it current with MySQL's asynchronous replication. An RDS for MySQL instance can have up to 15 replicas in its Region, in any zones, and RDS recommends giving each one the same compute and storage as the source. You add and remove them yourself, since RDS doesn't autoscale replicas. Each one is billed like a regular instance, though replication traffic within a Region is free.
Three changes make replicas work:
import time
PRIMARY_HOST = "app-db.example.internal" # accepts reads and writes
REPLICA_HOST = "app-db-read.example.internal" # one name in front of all replicas
STICKY_SECONDS = 5 # longer than the lag we normally see
def pick_db_host(user_id: int, is_write: bool, cache) -> str:
"""Send writes, and reads that follow a recent write, to the primary."""
if is_write:
# Stored in the shared cache, because any web server may get the next request.
cache.set(f"last_write:{user_id}", time.time(), ex=STICKY_SECONDS)
return PRIMARY_HOST
if cache.get(f"last_write:{user_id}") is not None:
return PRIMARY_HOST # read your own write
return REPLICA_HOST
For a rough count, suppose the cache answers 90% of the 120,000 peak reads, leaving 12,000 a second. If your load tests show one replica handles 4,000 of your queries a second with headroom, you need three, plus a spare so that losing one doesn't overload the rest. The per-replica figure is made up; measure your own. Relational databases covers replication and lag in depth.
Finally, add web and application servers as needed. With sessions in the cache, that's just launching more instances behind the load balancers.
What the measurements show. Traffic peaks during US business hours and drops sharply when people leave work. Running peak capacity all day wastes money all night. The team is small and wants to automate as much of the operations work as possible.
Goals. Add autoscaling to keep up with spikes and to power down idle instances. Automate DevOps. Keep monitoring to find the next bottleneck.
Amazon EC2 Auto Scaling manages Auto Scaling groups of identical instances. Each group has a minimum size it never goes below, a maximum it never exceeds, and a desired capacity it maintains. It replaces instances that fail health checks and registers new instances with the load balancer. It has no charge of its own; you pay for the instances and alarms.
CloudWatch provides the trigger, in two broad ways:
import boto3
autoscaling = boto3.client("autoscaling")
# Thermostat: keep the web tier's average CPU near 50%.
autoscaling.put_scaling_policy(
AutoScalingGroupName="web-tier",
PolicyName="cpu-50",
PolicyType="TargetTrackingScaling",
TargetTrackingConfiguration={
"PredefinedMetricSpecification": {"PredefinedMetricType": "ASGAverageCPUUtilization"},
"TargetValue": 50.0,
},
)
# Calendar: at least 8 web servers before the US workday starts.
autoscaling.put_scheduled_update_group_action(
AutoScalingGroupName="web-tier",
ScheduledActionName="weekday-morning",
Recurrence="0 8 * * MON-FRI",
TimeZone="America/New_York",
MinSize=8,
)
Autoscaling adds complexity: servers must be stateless, each new instance must configure itself, and you debug a fleet whose members change hourly. It also lags. A metric must cross its threshold, and a new instance must boot, pass health checks and warm up before it helps. With basic monitoring, EC2 metrics arrive only every five minutes. A sudden spike can outrun all that, and target tracking also scales in gradually after a drop. Scheduled and predictive scaling cover the spikes you can see coming.
Autoscaling only works if a new instance goes from nothing to serving traffic with no human involved:
Each stage moves the bottleneck, and only measurement shows where:
Monitoring, logging and tracing explains what to measure and when to alert.
What the measurements show. As the service grows toward the step 1 numbers, each round of load testing and profiling turns up a new bottleneck. (Autoscaling groups are left out of the diagram.)
Goals. Keep fixing the problems those numbers predict.
If the MySQL database grows too large, keep only a limited window, such as the last year, and move the rest to a data warehouse. Amazon Redshift is a managed, petabyte-scale warehouse, so 1 TB of new content a month is comfortably within its range. Analysts query history there, and the transactional database stays small enough to back up, restore and replicate quickly.
At 40,000 reads a second on average, mostly for popular content, the answer to read traffic is to scale the memory cache, which also absorbs uneven traffic and spikes. The read replicas may still struggle with the misses: at 120,000 reads a second at peak, even a 5% miss rate is 6,000 queries a second.
Writes are harder. 400 a second on average, with much higher peaks, all land on one primary. A large MySQL server may handle that rate, and your load tests will tell you, but its storage also grows 1 TB a month toward the 64 TiB limit. One primary runs out of room before it runs out of speed. Four SQL patterns help:
posts by user_id over 8 shards gives each about 36 TB ÷ 8 = 4.5 TB after three years and 1,200 ÷ 8 = 150 writes a second at peak. Queries across users, such as a global "latest posts" feed, must then ask every shard or come from somewhere else.To absorb more of the reads and writes, move data that doesn't need relations to a NoSQL database such as Amazon DynamoDB. It's a fully managed key-value and document store that AWS describes as giving single-digit millisecond performance at any scale. It replicates every table across three Availability Zones by default and scales tables to virtually any size, with no servers or shards for you to manage. In on-demand mode you pay per request.
The trade-off is the query model. DynamoDB has no joins, so you design tables around the questions the application asks and denormalize to answer each in one lookup. Sessions, counters and per-user lists like "this user's posts, newest first" fit well, and the core relational data can stay in MySQL. NoSQL and choosing a database shows how.
Separate the application servers further so each job scales on its own. Work that doesn't need to finish while the user waits, such as batch jobs and heavy computations, runs asynchronously through a queue and workers.
In a photo feature, the upload and the thumbnail can be split:
import json
import boto3
sqs = boto3.client("sqs")
s3 = boto3.client("s3")
BUCKET = "example-photos"
QUEUE_URL = "https://sqs.us-east-1.amazonaws.com/123456789012/thumbnails"
def upload_photo(user_id: int, image: bytes) -> dict:
"""In the request: store, record, enqueue, return."""
photo_id = new_photo_id()
key = f"originals/{photo_id}.jpg"
s3.put_object(Bucket=BUCKET, Key=key, Body=image)
insert_photo(photo_id, user_id, key) # thumbnail_key starts out NULL
sqs.send_message(QueueUrl=QUEUE_URL, MessageBody=json.dumps({"photo_id": photo_id, "key": key}))
return {"photo_id": photo_id, "status": "processing"}
def run_worker() -> None:
"""On a worker: take jobs, do the slow part, delete each job when done."""
while True:
resp = sqs.receive_message(QueueUrl=QUEUE_URL, MaxNumberOfMessages=10, WaitTimeSeconds=20)
for msg in resp.get("Messages", []):
job = json.loads(msg["Body"])
if not thumbnail_done(job["photo_id"]): # a job can arrive twice
original = s3.get_object(Bucket=BUCKET, Key=job["key"])["Body"].read()
thumb_key = f"thumbs/{job['photo_id']}.jpg"
s3.put_object(Bucket=BUCKET, Key=thumb_key, Body=make_thumbnail(original))
set_thumbnail_key(job["photo_id"], thumb_key)
sqs.delete_message(QueueUrl=QUEUE_URL, ReceiptHandle=msg["ReceiptHandle"])
Three SQS behaviors shape that code:
Workers can autoscale on a custom metric of queue messages per instance. With Lambda, AWS polls the queue and runs your function per batch, so there are no workers to manage. The costs: the client must show "processing" until the thumbnail is ready, and you need to watch queue depth and keep a dead-letter queue for jobs that keep failing.
Beyond 10 million users the loop continues. The 2015 talk ends with multiple AWS Regions and custom solutions where managed services no longer fit.
Managed services or your own on EC2? RDS, ElastiCache and DynamoDB cost more per hour than the same software on plain instances, but AWS handles backups, patching, failover and much of the scaling. For a small team that's usually the right trade, because engineers cost more than instance hours. Run your own when you need a version, extension or setting the managed service lacks, or at a scale where the savings pay for a database team.
Multi-AZ standby, read replicas, or both? A Multi-AZ standby is for availability: synchronous, unreadable, and takes over in a minute or two. A read replica is for capacity: asynchronous, can lag, and promoting it is a manual step. At this size you usually want both. RDS also offers Multi-AZ DB clusters, whose two standbys in other zones also serve reads. See Availability and Relational databases.
Cache first, or replicas first? A cache is cheaper per read and much faster, but brings stale data and a new failure mode. Replicas speed up every query, including the long tail a cache misses, but each is a full database instance. At 100 reads per write with popular items, the cache usually pays off first, after giving the buffer pool enough memory. See Caching.
What should trigger autoscaling? CPU target tracking is simple and works when CPU is what runs out. ALB requests per target tracks web load more directly. Scheduled scaling handles the rush you can see coming, and predictive scaling learns it. A high minimum costs money every hour; a low one risks slow mornings. Here, target tracking on requests per target plus a scheduled weekday minimum is a reasonable pick.
Sharding, federation or NoSQL? Federation is gentlest: each database stays ordinary MySQL. Sharding scales one huge table but moves cross-shard queries and resharding into your code. DynamoDB removes the sharding work but needs access-pattern design and has no joins. A reasonable path: federate first, move sessions and counters to DynamoDB, and shard only the tables that still don't fit. See Relational databases and NoSQL.
Workers on EC2 or Lambda? Lambda has nothing to patch or scale and suits short, bursty jobs like thumbnails. EC2 workers suit steady, heavy or long-running jobs with special libraries. Both need idempotent handlers. See Queues and asynchronous work.
More than one Region? This design survives losing an Availability Zone, not a whole Region. Going multi-Region means copying data across Regions (cross-Region read replicas, DynamoDB global tables) and steering users with Route 53 latency or failover routing. It roughly doubles the infrastructure and raises hard consistency questions, so most products wait until a regulation or an availability promise requires it. See DNS routing policies and CAP and consistency.
RDS fails over by changing the database endpoint's DNS record to point at the standby. Some JVM configurations cache DNS lookups until the process restarts, so the servers kept connecting to the dead primary.
Set the JVM's DNS cache TTL (networkaddress.cache.ttl) to 60 seconds or less, as AWS recommends, and make sure the connection pool drops broken connections. Then rehearse it: rebooting an RDS instance with failover triggers the same switch on purpose.
The web servers are stateful: each keeps sessions in its own memory, probably with sticky sessions on the load balancer. Removing a server destroys its sessions.
Move sessions to a shared store such as ElastiCache or DynamoDB. Then any server can serve any user, Auto Scaling can remove any of them safely, and sticky sessions can go.
Stage 3. Writes go to the primary, reads go to a replica, and replication is asynchronous, so the reload hit a replica that hadn't caught up.
Send a user's reads to the primary for a few seconds after they write, as in pick_db_host, or write the new entry into the cache along with the database. Also watch replica lag in CloudWatch: lag of minutes means the replicas are overloaded.
At 95%, misses are 120,000 × 0.05 = 6,000 a second, so 6,000 ÷ 3,000 = 2 replicas, plus a spare or two for failures: about 3 to 4.
At 80%, misses jump to 120,000 × 0.20 = 24,000 a second, which needs 8 replicas. Small changes in hit rate cause large swings in database load. That's why the cache needs replicas and automatic failover (Valkey or Redis OSS), and why you can't rely on adding replicas in a hurry: you create them yourself, and RDS for MySQL allows at most 15 per source.
RDS for MySQL tops out at 64 TiB, about 70 TB, and you're at roughly 72 TB. The standby and every replica hold full copies too, so the storage bill multiplies.
From least to most work: archive old posts to Redshift or S3; federate posts into their own database; shard posts by user_id across several RDS instances; or move posts to DynamoDB. With 1 TB arriving every month, archiving only buys time. Sharding or DynamoDB is the long-term answer for the largest table.
Traffic from private subnets to S3 goes through the NAT gateway, which charges per gigabyte processed, about $0.045 per GB in US East at the time of writing. Every photo read and every thumbnail written pays it.
Add a gateway endpoint for S3 to the VPC. Gateway endpoints have no extra charge, and S3 traffic then bypasses the NAT gateway. For traffic that still needs NAT, keep a NAT gateway in each Availability Zone so it doesn't cross zones.
Target tracking reacts after the fact. CPU must rise, an alarm must fire, and new instances must boot, pass health checks and warm up. With basic monitoring, EC2 metrics arrive only every five minutes. That adds up to about ten minutes.
The spike is predictable, so scale before it: a scheduled action that raises the minimum before 9 a.m., or predictive scaling. Turn on one-minute detailed monitoring, and consider tracking ALB requests per target, which rises as soon as traffic does. Keep target tracking for the days the pattern breaks.
Previous: Design a personalized recommendation system · Next: Design a hash map