Advanced Reverse Proxying with Caddy

A reverse proxy is the most common reason to run Caddy in production: the server accepts the public connection, terminates TLS, and forwards the request to an application server that never sees the internet directly. The reverse_proxy directive is far more than a one-line pass-through. It can spread traffic across several backends, take a backend out of rotation the moment it starts failing, rewrite paths and headers on the way in and out, and tune the transport that carries the bytes. By the end of this page you will be able to configure a production proxy with load balancing, health checks, and header rewriting, and you will know which knob to turn when an upstream misbehaves.

1. The basics

The shortest useful proxy names one or more upstreams. You can also list them in a to subdirective, which reads better once the list grows.

example.com {
	reverse_proxy {
		to localhost:9000 localhost:9001
	}
}

By default Caddy passes the request through almost unchanged, but it does add the X-Forwarded-For, X-Forwarded-Proto, and X-Forwarded-Host headers so the application can learn the real client address and the original scheme. Caddy also changes the Host header to the upstream address, because most applications expect to see their own hostname.

2. Load balancing

When several upstreams are listed, Caddy distributes requests between them, and lb_policy chooses the strategy.

PolicyBehaviour
round_robinThe default. Requests are handed out in rotation
least_connThe backend with the fewest active connections wins
randomA random backend, useful when backends are heterogeneous
ip_hashThe client address decides, so a client always reaches the same backend
header, cookie, queryA header, cookie, or query parameter decides
client_ip_hashThe client address decides, using a privacy-preserving hash
weighted_round_robinBackends are weighted relative to each other
firstThe first healthy backend always wins, a failover-only strategy
example.com {
	reverse_proxy {
		to backend-1:9000 backend-2:9000 backend-3:9000
		lb_policy least_conn
	}
}

For weighted_round_robin, a number after each address is that backend’s weight.

example.com {
	reverse_proxy {
		to backend-1:9000 3 backend-2:9000 1
		lb_policy weighted_round_robin
	}
}

A session that must stay on one backend, such as a chat application with local state, needs a sticky policy such as lb_policy cookie session_id.

3. Health checks and failover

An active health check is a request Caddy sends to a backend on its own, independent of client traffic. Without it, a dead backend keeps receiving one out of every N requests until a client happens to notice.

example.com {
	reverse_proxy {
		to backend-1:9000 backend-2:9000
		lb_policy round_robin
		health_uri /healthz
		health_interval 10s
		health_timeout 5s
		health_status 200
		health_passes 2
		health_fails 3
		lb_try_duration 5s
		lb_try_interval 250ms
	}
}

health_uri is the path probed, so a dedicated cheap endpoint is much better than a full page. health_interval is how often the probe runs, health_timeout bounds a single probe, and health_status lists the codes that count as healthy. health_passes is how many consecutive successes are needed before a backend rejoins rotation, and health_fails how many consecutive failures remove it. The lb_try_duration and lb_try_interval settings make a request that lands on a failing backend retry against another one inside the same client request.

A passive check instead watches real traffic. When a backend returns an error, Caddy marks it unhealthy for a while.

example.com {
	reverse_proxy {
		to backend-1:9000 backend-2:9000
		fail_duration 30s
		max_fails 2
		unhealthy_status 500 502 503 504
		unhealthy_latency 10s
	}
}

The two mechanisms complement each other: active checks detect a backend that is already broken before any client notices, and passive checks detect a backend that broke in the middle of a request.

4. Header manipulation

header_up changes the request that is sent to the backend, and header_down changes the response returned to the client. Both accept the same prefixes as the header directive: no prefix sets the value, + appends, and - removes the field entirely.

example.com {
	reverse_proxy {
		to backend-1:9000 backend-2:9000
		header_up Host {upstream_hostport}
		header_up X-Real-IP {http.request.remote.host}
		header_down -Server
	}
}

The {upstream_hostport} placeholder expands to the address of the backend that was actually chosen, which is the standard way to restore the original Host header. Removing Server stops the upstream from advertising its own software.

The X-Forwarded-For header is a comma-separated list. Because a client can send its own X-Forwarded-For header, Caddy only honours and appends to it when the request came from a proxy you trust. Declare those ranges with the trusted_proxies global option, otherwise a client can spoof its own address in your logs.

5. Path rewriting

Two different tools are involved and they solve different problems.

handle_path strips the matched prefix before the request is forwarded, which is the usual way to mount an application that does not know it lives under a sub-path.

example.com {
	handle_path /api/* {
		reverse_proxy localhost:9000
	}
}

A request for /api/orders reaches the backend as /orders. The rewrite subdirective of reverse_proxy behaves differently: it sets the path to the matched value rather than removing it, so the same request arrives as /api/orders.

example.com {
	reverse_proxy {
		to localhost:9000
		rewrite /api/*
	}
}

When the mapping is a simple prefix swap, the uri directive is the clearest tool.

example.com {
	uri strip_prefix /api
	reverse_proxy localhost:9000
}

6. Transport tuning

The transport subdirective configures how Caddy talks to the backend. The default http transport exposes buffer sizes, timeouts, and connection reuse.

example.com {
	reverse_proxy {
		to backend-1:9000 backend-2:9000
		transport http {
			read_buffer 64KiB
			max_response_header 32KiB
			dial_timeout 5s
			response_header_timeout 30s
			keepalive 30s
			keepalive_idle_conns 32
			keepalive_idle_conns_per_host 8
		}
	}
}

read_buffer and write_buffer size the buffers used for each connection, and the default of 4KiB is often too small for large response headers or slow clients. max_response_header caps the size of the headers the backend may send, which protects Caddy from a runaway or hostile upstream. dial_timeout bounds the TCP connection attempt and response_header_timeout bounds the wait for the backend’s response headers. The keepalive settings reuse upstream connections instead of paying for a new TCP handshake on every request.

To talk to a backend over TLS, add tls to the transport block. Add tls_insecure_skip_verify only when the backend uses a self-signed certificate that you cannot install a trust anchor for.

7. WebSockets and streaming

WebSockets need no special configuration. Caddy detects the HTTP upgrade request and passes the connection through, so a normal reverse_proxy line handles a WebSocket application correctly.

Streaming responses, such as server-sent events, are buffered by default. The flush_interval subdirective sets how often the buffer is pushed to the client, and a negative value flushes immediately after every write.

example.com {
	reverse_proxy {
		to localhost:9000
		flush_interval -1
	}
}

8. Common mistakes

MistakeWhat happensCorrection
Probing / for healthThe check costs as much as a real requestAdd a cheap endpoint such as /healthz
Expecting lb_policy to retryA failed request is returned to the clientAdd lb_try_duration
Trusting X-Forwarded-For from anyoneClients can spoof their addressConfigure trusted_proxies
Using rewrite to strip a prefixThe prefix is preservedUse handle_path or uri strip_prefix
reverse proxyload balancinghealth checksheader rewrite

Approach:

The requirement is a production proxy in front of three application backends, one of which is larger than the others, with a health endpoint the backends already expose, and with the original Host header preserved so the application keeps generating correct links. The design has to tolerate a backend dying without returning errors to users.

Steps:

  1. List the backends and give them a weighting, because the third backend can take twice the traffic of the first. The weighted_round_robin policy with a number after each address expresses that.
  2. Add active health checks against the existing /healthz endpoint, with a short interval and a timeout, so a dead backend is removed before a client is routed to it.
  3. Require two consecutive successes before a recovered backend rejoins rotation, which stops a flapping backend from taking traffic in and out repeatedly.
  4. Add lb_try_duration so a request that hits a failing backend is retried against another one within the same client request.
  5. Set header_up Host {upstream_hostport} so the backend sees its own address, and remove the Server header from the response with header_down.
  6. Raise the transport read buffer and the maximum response header size, because the default buffer is small and the application sends a large header set.
  7. Enable upstream connection reuse with keepalive, so repeated requests reuse warm connections instead of opening a new TCP session each time.

Answer:

The finished configuration is:

example.com {
	reverse_proxy {
		to backend-1:9000 1 backend-2:9000 1 backend-3:9000 2
		lb_policy weighted_round_robin
		health_uri /healthz
		health_interval 10s
		health_timeout 5s
		health_status 200
		health_passes 2
		health_fails 3
		lb_try_duration 5s
		lb_try_interval 250ms
		header_up Host {upstream_hostport}
		header_down -Server
		transport http {
			read_buffer 64KiB
			max_response_header 32KiB
			dial_timeout 5s
			response_header_timeout 30s
			keepalive 30s
			keepalive_idle_conns 32
			keepalive_idle_conns_per_host 8
		}
	}
}

With this configuration, traffic is distributed with the third backend receiving roughly twice the share of the other two, a backend that stops answering /healthz is removed from rotation after three consecutive failures, a request that lands on a failing backend is retried against a healthy one within five seconds, and the application receives the correct Host header on every request.

Found an issue with this content?

Tell us what could be improved.