What is a load balancer?
A load balancer is a traffic cop that stands in front of several identical servers and spreads incoming requests across them. Clients only ever see ONE address — the balancer's — and never know (or care) which server actually answered.
It exists because of a simple wall: one server has a ceiling. When traffic goes past that ceiling, you have two choices: buy an ever-bigger machine (vertical scaling — quickly gets absurdly expensive and still ends at some ceiling), or run many normal machines side by side (horizontal scaling) with a load balancer sharing the work. Horizontal is how every large system scales.
- vertical scaling
- — making one machine bigger (more CPU/RAM)
- horizontal scaling
- — adding more identical machines that share the work
- health check
- — the balancer's regular "are you alive?" ping to each server
Words to know
The night SliceNow hit the ceiling
SliceNow's one server comfortably handles 500 requests per second. Then a TV spot airs during a football game, and 2,500 requests per second arrive for ten straight minutes.
What actually happens at the ceiling is ugly: requests queue up, response times stretch from 200ms to 8 seconds, memory fills, and finally the server starts refusing connections outright. Users do not see "we are at capacity" — they see a broken app, and they order elsewhere.
The fix: run five identical servers (each still handling 500/sec) and put a load balancer in front. 5 × 500 = 2,500. The TV spike is now just a busy night. And when the spike ends, you can turn three of them off — horizontal scaling works in both directions.
One server's ceiling
~500 req/sec
TV-spot traffic
~2,500 req/sec
Servers needed
5 × 500 = 2,500 ✓
Quick check
Traffic doubles every few months. Why do engineers prefer horizontal scaling (more machines) over vertical (a bigger machine) in the long run?
How it decides who gets the request
The simplest strategy is round-robin: deal requests out like cards. With servers A, B and C: request 1 → A, request 2 → B, request 3 → C, request 4 → back to A, and around again. Every server gets an equal share, no thinking required.
Round-robin's blind spot: it counts requests, not effort. If server A gets stuck with three slow, heavy requests, round-robin keeps dealing it new ones anyway. Least-connections fixes that: send each new request to whichever server currently has the fewest conversations open — busy servers naturally get skipped until they recover.
A third strategy, IP-hash, always sends the same visitor to the same server (their address picks the server). Useful when a server keeps per-visitor state in memory — though as you will see, it is usually better to not keep state on web servers at all.
Choosing a strategy
| Strategy | How it picks | Shines when | Watch out |
|---|---|---|---|
| Round-robin | Deal requests out in a circle | Requests cost roughly the same | Keeps feeding a server stuck on slow requests |
| Least-connections | Fewest open conversations wins | Request costs vary a lot | Slightly more bookkeeping for the balancer |
| IP-hash | Visitor's address picks the server | A visitor must stick to one server | Uneven spread; a busy office = one hot server |
Health checks: the reliability superpower
Every few seconds, the balancer asks each server "are you alive?" (a health check — literally a tiny request like GET /health). A server that fails a few checks in a row is quietly pulled out of the rotation; when it recovers, it is eased back in.
Sit with what that means: a server can catch fire at 2 a.m. and no customer notices. Requests just flow to the survivors. This is the deeper reason load balancers matter — capacity is what they are named for, but surviving failures without downtime is what makes them non-negotiable in serious systems.
Common trap — Wait — is the balancer itself a single point of failure?
Yes — you moved the "one thing that can die" from the server to the balancer. Real deployments run a second balancer on standby (or lean on cloud/DNS tricks) so the cop itself has a backup. If an interviewer asks "what if the load balancer dies?", that is the expected answer: balancers are made redundant too.
Match them up
Match each term to what it really means.
0/5 matched — tap a term on the left.
Build it yourself
SliceNow needs to survive the next TV spot. Put a load balancer in front of TWO app servers, and connect both servers to the shared database. One palette piece solves a problem you do not have yet — leave it off.
Quick check
Your design has a balancer and two app servers. Server 2 crashes during the lunch rush. What do customers experience?
Apply it in an interview
The URL-shortener interview problem is a classic load-balancing story: huge read traffic, one public address, identical servers behind it. Take what you just built into a full mock interview.