Who load balances the load balancer?

Scaling load balancers. Ever thought about that?

We are all used to load balancers scaling the things behind them. Autoscaling groups, health checks, instances coming and going. That part is routine.

But what happens when the load balancer itself needs to scale?

At Layer 7 this is easy

Application load balancers hand you a hostname. Clients resolve that name, and whatever IPs sit behind it can shuffle freely. Add capacity, remove capacity, replace a failed node, and DNS absorbs it. The client never knows.

That indirection is the whole trick, and it is free.

At Layer 4 the indirection is gone

Network load balancers cannot hide behind a hostname in the same way. Clients connect directly to IP addresses, and those addresses have to stay stable even as the fleet behind them grows, shrinks, and loses nodes.

You cannot renumber your way out of it. A client with an open TCP connection to 203.0.113.10 is going to keep sending packets to 203.0.113.10.

And stable addressing is only half the problem. A Layer 4 balancer has to maintain connection state and session affinity. Every packet in a TCP connection must reach the same backend that saw the handshake, or the connection resets. That is straightforward when one machine sees every packet. It stops being straightforward when the balancer is itself a distributed fleet, and any node in that fleet might receive any packet.

So the real constraint is this: any node, receiving any packet, at any time, must independently route it to the same backend as every other node would.

No shared session table lookup on the hot path. At line rate, you do not get to ask anyone.

How Google solved it: Maglev

Maglev is Google's software network load balancer, described in their 2016 paper.

Several Maglev nodes announce the same virtual IP into the network. Upstream routers use ECMP, equal-cost multi-path, to spread incoming packets across all of them. From the client's perspective there is one address. From the network's perspective there are many equal paths to it.

That solves stable addressing. It also creates the affinity problem, because ECMP gives no guarantee that two packets in the same connection land on the same Maglev node. Routers rehash when the set of paths changes.

The answer is consistent hashing. Each Maglev node computes the backend for a packet from the connection's 5-tuple using a shared hash table that every node builds identically. Same input, same table, same answer, regardless of which node does the computing. No coordination needed on the hot path.

Maglev's contribution is the specific hashing scheme it uses to build that table: it spreads connections evenly across backends while minimizing how many connections get reassigned when the backend set changes. Even distribution and minimal disruption are in tension, and the paper is largely about the tradeoff between them.

Each node also keeps a local connection tracking table as a fast path, so established connections stay pinned even through a backend set change.

AWS solved the same problem with Hyperplane

AWS Hyperplane is the internal distributed system underneath Network Load Balancer, NAT Gateway, EFS, and PrivateLink. There is no public paper that I could find, which is a little frustrating given how much of AWS sits on top of it.

From the outside the shape is the same: client-facing addresses decoupled from the physical nodes handling packets, with distributed state that keeps flows pinned to backends.

The idea underneath both

Both solutions share one move: decouple the client-facing IP from the physical node that handles the packet.

The address becomes a virtual thing that many machines can answer for. Routing becomes a pure function of the packet, computed identically everywhere, instead of a lookup into shared state. Stable for clients. Scalable for infrastructure.

Which is the recurring shape of scaling problems generally. The abstraction you built to hide a pool of machines eventually becomes a pool of machines that needs hiding.

Every abstraction eventually needs its own abstraction.

Why this is worth knowing

Most of us will never write a load balancer. That is fine. The transferable part is the constraint:

When you cannot coordinate on the hot path, make the decision a pure function of the input.

That is the same reason consistent hashing shows up in sharded caches, in partition assignment, in request routing inside a service mesh. Any time you have N nodes that must agree without talking to each other, you are solving a version of the Maglev problem.

The alternative, shared state on every request, is the thing that quietly caps your throughput long before your CPU does.