I’ve spent a lot of time on the operating side of load balancers. I’ve read NGINX configs, stared at ALB target group health, and wondered why one pod was getting all the traffic. But I’d never actually written one. So I’ve started building a small HTTP load balancer in Go called steer, in phases, so I can understand each piece before adding the next one.

This post is about phase 1, which is the smallest possible version: a load balancer with exactly one backend. That sounds like it shouldn’t count as a load balancer at all. But it turned out to be where most of the interesting stuff lives, so I wanted to write down what I learned.

the plan Link to heading

Here’s roughly how the phases go:

  1. forward requests to a single backend (this post!)
  2. round robin across several backends
  3. health checks, so we stop sending traffic to dead backends
  4. least-connections routing
  5. metrics and logging
  6. timeouts, graceful shutdown, TLS, sticky sessions
  7. break it on purpose and see what happens

step 0: something to load balance Link to heading

First I needed some backends. I wrote a tiny Go server that replies with its own port number, so I can tell which backend answered:

mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
	time.Sleep(*delay)
	fmt.Fprintf(w, "hello from %s (path=%s)\n", name, r.URL.Path)
})

mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
	w.WriteHeader(http.StatusOK)
	fmt.Fprintln(w, "ok")
})

It also has a -delay flag that adds artificial latency. That’s for later: I want to make one backend slow on purpose and watch round robin do the wrong thing.

There’s a /healthz endpoint too, even though nothing calls it yet. The reason to have a separate health endpoint instead of just checking / is that / might be expensive (talk to a database, render a page) and health checks run constantly. You want something cheap with no side effects.

step 1: the load balancer is… one line? Link to heading

Here’s the core of the load balancer:

target, _ := url.Parse("http://localhost:8081")
proxy := httputil.NewSingleHostReverseProxy(target)
http.ListenAndServe(":8080", proxy)

That’s it. I ran it, ran curl localhost:8080/foo, and got back hello from backend-8081 (path=/foo).

I was a little disappointed, honestly. I expected to have to open sockets and copy bytes around. So I went and read about what httputil.ReverseProxy actually does, because clearly it’s doing a lot.

what a reverse proxy actually does Link to heading

A reverse proxy takes a request from a client and makes a brand new request to a backend on the client’s behalf, then relays the answer back. The client never talks to the backend directly.

(A forward proxy is the other kind: it acts on behalf of clients, like a corporate proxy you have to go through to reach the internet. A reverse proxy acts on behalf of servers. I always forget which is which, so I’m writing it down.)

Here’s what ReverseProxy does for every request:

  • it rewrites the URL. The incoming request is for localhost:8080/foo, and it changes the scheme and host to localhost:8081 but keeps the path. That’s why the backend sees /foo.
  • it adds X-Forwarded-For. Otherwise the backend would think every request came from the load balancer’s IP address. This is the header your app reads when it wants the “real” client IP.
  • it strips hop-by-hop headers. Headers like Connection and Keep-Alive only describe the single TCP connection between two machines, so forwarding them to the next hop would be wrong. I didn’t know HTTP had a category of headers like this!
  • it reuses connections. It sends requests through Go’s default http.Transport, which keeps a pool of open TCP connections to the backend. So it isn’t doing a new TCP handshake on every request.
  • it streams the response. The body gets copied back as it arrives, so if the backend sends a 1GB file, the load balancer doesn’t hold 1GB in memory.
  • it returns a 502 when things go wrong. More about that in a second.

The thing that really clicked for me is that there are two separate HTTP conversations happening: curl talks to the load balancer, and the load balancer talks to the backend. The load balancer is a server on one side and a client on the other.

curl  ──── HTTP ────▶  steer (:8080)  ──── HTTP ────▶  backend (:8081)
      ◀───────────────                ◀───────────────

That’s what makes everything a load balancer does possible. Because it owns both conversations, it can pick a different backend, retry when one fails, add headers, and time how long the backend took.

the HandlerFunc trick Link to heading

I wanted to log every request, so I wrapped the proxy:

handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
	log.Printf("%s %s -> %s", r.Method, r.URL.Path, target)
	proxy.ServeHTTP(w, r)
})

I’d written http.HandlerFunc(...) lots of times without thinking about it. It turns out it isn’t a function call at all, it’s a type conversion. The standard library has this:

type HandlerFunc func(ResponseWriter, *Request)

func (f HandlerFunc) ServeHTTP(w ResponseWriter, r *Request) {
	f(w, r)
}

http.Handler is an interface with one method, ServeHTTP. HandlerFunc is a function type that has a ServeHTTP method, and that method just calls the function. So converting a plain function to HandlerFunc turns it into a Handler. I think this is a really neat trick.

This wrapper is also where all the actual load balancing is going to go. Right now it always calls the same proxy. In phase 2 it’ll pick one from a list instead, and that choice is the entire job of a load balancer. Everything else is plumbing.

experiment: kill the backend Link to heading

Next I wanted to see what happens when the backend goes away. I killed it and curled again:

$ curl -s -o /dev/null -w "HTTP %{http_code}\n" localhost:8080/foo
HTTP 502

and the load balancer logged:

http: proxy error: dial tcp [::1]:8081: connect: connection refused

502 Bad Gateway means “I’m a proxy, and the thing behind me didn’t give me a valid response”. That’s ReverseProxy’s default error handler. It’s a totally reasonable thing to do with one backend, but with several backends, the right move is to try another one instead of handing the failure to the user. That’s phase 3.

Also, [::1] is the IPv6 loopback address. localhost resolved to IPv6 first, which I didn’t expect.

a small surprise: my first test failed Link to heading

The very first time I tested this, the “healthy” curl printed nothing at all. It turned out my test script started the backend and the load balancer and then only waited 1 second before curling. That wasn’t long enough for both servers to be listening. With a 2 second wait it worked fine.

This is a toy version of a real problem! “The process has started” and “the process is ready to serve traffic” are different things. That gap is exactly what Kubernetes readiness probes exist for, and it’s the same gap health checks are going to cover in phase 3.

some Go things I learned along the way Link to heading

  • flag.Int returns a pointer (*int), not an int, because the command line hasn’t been parsed yet when you call it. The value gets filled in when you call flag.Parse().
  • An HTTP handler that writes a body without calling WriteHeader automatically sends a 200 OK.
  • You have to set headers and the status code before writing the body. Once the first byte of the body goes out, the headers are already gone.
  • http.ListenAndServe starts a new goroutine for each connection. So if 100 requests come in at once, my wrapper function runs 100 times concurrently. That’s fine right now because it only reads shared state, but it won’t stay fine for long.

next: round robin, and my first race condition Link to heading

Phase 2 is round robin: keep a list of backends and a counter, and send each request to the next one. The obvious version looks like this:

backend := backends[counter % len(backends)]
counter++

and because of that goroutine-per-connection thing, two requests can read and increment counter at the same time. I’m looking forward to figuring out how to deal with that (and to running Go’s race detector on it).

The code is at github.com/balakrishnan-vasudevan/steer if you want to follow along.