System design2 min

Load Balancing

When you scale horizontally (adding more servers), you need a way to distribute incoming traffic across them. This is the job of the Load Balancer (LB).

A Load Balancer sits between the client and the server pool, acting as a traffic cop. It ensures no single server bears too much demand, which improves responsiveness and availability.

Types of Load Balancers

Layer 4 (Transport Layer) Load Balancers

L4 load balancers operate at the network level (TCP/UDP).

  • How they work: They look only at the IP address and Port. They do not inspect the contents of the HTTP request.
  • Pros: Extremely fast and require very little CPU overhead.
  • Cons: "Dumb" routing. They can't route traffic based on URL paths, cookies, or HTTP headers.

Layer 7 (Application Layer) Load Balancers

L7 load balancers operate at the HTTP level.

  • How they work: They decrypt the HTTPS traffic, inspect the HTTP headers, URLs, and cookies, and then make intelligent routing decisions.
  • Pros: "Smart" routing. You can route /api/video/* to powerful video-processing servers, and /api/text/* to cheap, lightweight servers. They can also terminate SSL/TLS.
  • Cons: Slower and more CPU-intensive than L4 because they must decrypt and inspect every packet.

Routing Algorithms

How does the LB decide which server gets the next request?

  1. Round Robin: The simplest method. Requests are distributed sequentially (Server A, then B, then C, then A again). Works well if all servers are identical and requests have similar processing times.
  2. Weighted Round Robin: Administrators assign "weights" to servers. A powerful server gets a weight of 3, a weak one gets 1. The powerful server receives 3x as many requests.
  3. Least Connections: Routes the request to the server with the fewest active connections. Excellent when request processing times vary wildly (e.g., some queries take 10ms, others take 5 seconds).
  4. IP Hash: The LB hashes the client's IP address to determine which server gets the request. This ensures a specific user always hits the exact same server (useful for sticky sessions, though external session stores are preferred).

Health Checks

A load balancer is useless if it sends traffic to a crashed server. LBs continuously send "pings" or HTTP requests (e.g., to /health) to all servers in the pool. If a server fails to respond consecutively, the LB removes it from the rotation until it recovers.