Monitoring and Logging in Caddy
Caddy is good at emitting structured logs and at exposing metrics, but it does neither until you ask. There is no access log unless a log directive is configured, no metrics endpoint until the metrics option is enabled, and the default destination for both is standard error. That design keeps a small deployment simple, and it puts the work of building an observable service on the operator, which is where it belongs. By the end of this page you will have access logs that are structured and safe to ship, a rotation policy that will not fill a disk, a Prometheus endpoint that a scraper can reach, and a short list of signals worth alerting on.
Caddy produces two distinct streams, and confusing them is the most common reason people cannot find what they are looking for.
Runtime logs describe the server itself: certificates being obtained, configurations being loaded, upstreams being marked unhealthy, errors during provisioning. They are produced by the core and by modules, and they are written to standard error by default, which means they land in the journal under systemd.
Access logs describe individual HTTP requests. They are not enabled by default, because they are only useful once there is somewhere to put them and something to read them. A server with no log directive produces no access log at all.
The two are configured in different places, which is the main source of confusion. Access logging uses the log directive, inside a site block. Runtime logging uses the log global option, inside the global options block.
{
log default {
output stdout
format json
}
}
example.com {
log {
output file /var/log/caddy/access.log
format json
}
respond "ok"
}
Every log message travels through the same three stages, and understanding them explains most of the configuration.
Gathering collects the relevant context. Caddy logs are structured: the information stays strongly typed and keyed by field name rather than being flattened into a string.
Encoding turns that structure into bytes. This is where the format is chosen, and it is a separate stage so that the same message can be written as JSON to one destination and as console text to another.
Writing sends the bytes somewhere. The destination is a writer module, and there may be several.
Around that pipeline sit two more controls. The level filters by severity, with DEBUG, INFO, WARN, ERROR, and rarely PANIC and FATAL. The sampling ratio keeps a hot path from producing more log messages than the pipeline can write.
This is why the structured representation matters. A log written in the legacy Common Log Format cannot be converted into JSON later, because the information is not there in the first place: the format does not even record the Host header. Caddy can produce the same information in any of several encodings, because the fields survive until the encoding step.
example.com {
log
respond "ok" 200
}
A bare log writes to standard error. The minimum useful configuration adds a destination and a format.
example.com {
log {
output file /var/log/caddy/example.com.access.log
format json
}
respond "ok" 200
}
The block accepts an optional logger name, which is what lets several loggers coexist.
example.com {
log main {
output file /var/log/caddy/main.log
}
respond "ok" 200
}
Two subdirectives control which requests a logger covers. hostnames overrides the set of hosts the logger applies to, which matters inside a wildcard site block. no_hostname detaches the logger from the hostnames of the site block entirely, which is what makes the split-file pattern below possible.
| Output module | Behaviour |
|---|---|
stderr | Standard error. The default. |
stdout | Standard output. |
discard | Nothing is written. Useful for silencing a logger. |
file | A file on disk, with rotation. |
net | A network socket, for shipping to a collector. |
log {
output stdout
}
A net output is how you ship logs straight to a collector without a local file in between. The destination is a network address in the usual Caddy form, which means a TCP, a UDP, or a unix socket.
log {
output net 10.0.0.5:9000
}
A log file that is never rotated will eventually fill the disk, so rotation is on by default. The settings below are the ones worth knowing.
log {
output file /var/log/caddy/access.log {
mode 0640
roll_size 100mb
roll_keep 10
roll_keep_for 720h
}
}
roll_size is the size at which the file is rolled. The implementation has megabyte resolution, and a fractional value is rounded up. The default is 100 MiB.
roll_keep is how many rolled files are kept. roll_keep_for is how long they are kept, and it accepts the usual duration syntax including d for days, so 720h is thirty days.
mode sets the permissions of the log file. 0640 keeps the file readable by the group that collects logs and unreadable by everyone else, which is the right default for anything that contains request data.
roll_interval rolls by time rather than by size, and roll_disabled turns rotation off entirely. Disabling it is only reasonable when something else rotates the file.
When a file is rolled, it is renamed using the pattern name-timestamp-reason.log, where the reason is either size or time. If the rolled file is compressed, .gz is appended.
One limitation is worth knowing: a change to the options of an existing file output requires a server restart to take effect. At reload time the change applies only if a new file name is added.
| Format module | Output |
|---|---|
console | Human-readable text |
json | One JSON object per line |
filter | Another encoder, with rules applied to fields |
delete | Removes a field |
rename | Renames a field |
replace | Replaces the value of a field |
ip_mask | Masks IP addresses with a CIDR |
query | Manipulates the query string of a URL field |
cookie | Manipulates the value of a Cookie header |
set_cookie | Manipulates the value of a Set-Cookie header |
regexp | Applies a regular expression replacement |
hash | Replaces the value with a short hash |
append | Adds a field |
The default is console when standard error is a terminal and json otherwise, which means a systemd service produces JSON by default and an interactive session produces readable text.
log {
output file /var/log/caddy/access.log
format json
}
The filter encoder is how you keep sensitive data out of a log while keeping the log useful. It wraps another encoder and applies rules to named fields.
log {
output file /var/log/caddy/access.log
format filter {
wrap json
fields {
request>headers>Authorization delete
request>headers>Cookie hash
request>uri query {
delete token
}
}
}
}
The field names are paths into the structured message, separated by >. The common ones are request>remote_ip for the directly connecting client, request>client_ip for the parsed real client when trusted_proxies is configured, and request>headers>X-Forwarded-For when there is a proxy in front.
Masking an IP address is often more useful than deleting it, because it preserves the ability to group by client.
fields {
request>remote_ip ip_mask 24 64
}
log_append adds a custom field to the log entry. The value may be a placeholder, which is how you capture something the standard fields do not carry.
example.com {
log_append tenant {http.request.header.X-Tenant}
log_append upstream_status {http.reverse_proxy.upstream.status_code}
reverse_proxy localhost:9000
}
By default the field is added on the way back up the middleware chain, after the response has been written, so it captures the final state. Prefixing the key with < marks it as early, which records the value before the request is modified by later handlers.
log_append <original_uri {http.request.uri}
For debugging, the value may also be one of the body placeholders, but the documentation is explicit that this is not for production use.
log_name chooses which logger handles a request, and the no_hostname option is what makes the split possible.
example.com {
log {
output file ./caddy.access.log
}
log health_check_log {
output file ./caddy.access.health.log
no_hostname
}
handle /healthz* {
log_name health_check_log
respond "Healthy"
}
handle {
respond "Hello"
}
}
The no_hostname option prevents the second logger from being associated with the site block hostnames, so it only receives requests whose logger name was set explicitly.
log_skip excludes requests from the access log. It is the right tool for health checks, which would otherwise dominate the volume.
example.com {
log
@health path /healthz /metrics
log_skip @health
respond "ok"
}
The matcher is optional when the directive is already inside a block that matched.
handle_path /static/* {
root * /srv/static
log_skip
file_server
}
The log global option configures the runtime loggers. Without a name it changes the logger called default.
{
log default {
output stdout
format json
include http.log.access admin.api
}
}
include and exclude filter by logger name. By default every logger is included and nothing is excluded. Names are easiest to discover from the logs themselves.
{
log default {
output stdout
format json
exclude http.log.access
}
}
Two levels are therefore available independently: a runtime logger that excludes access logs, and access loggers that write wherever you want them.
For a request that is failing and you need to see everything, raise the level temporarily.
{
debug
}
Debug logging is extremely verbose. Remove it once the problem is understood.
Access logs redact headers that carry credentials: Cookie, Set-Cookie, Authorization, and Proxy-Authorization appear as REDACTED. This is on by default and should stay that way.
{
servers {
log_credentials
}
}
The option above turns redaction off. It exists, and it is a bad idea in production.
Metrics collection is off until it is enabled.
{
metrics
}
The endpoint then appears on the admin API address.
curl -s http://localhost:2019/metrics
To label metrics with the host they belong to, enable the per-host option. The documentation warns about the cardinality this introduces, and it is only safe when the set of hosts is bounded.
{
metrics {
per_host
}
}
If the admin API is disabled, or you want the endpoint on a different port or path, expose it with the metrics directive.
metrics.example.com {
metrics
}
A Prometheus server scrapes the endpoint with a configuration like this.
global:
scrape_interval: 15s
scrape_configs:
- job_name: caddy
static_configs:
- targets: ['localhost:2019']
The metrics fall into four families.
| Prefix | What it covers |
|---|---|
caddy_admin_* | Requests to the admin API, counted per handler and status |
caddy_http_* | The HTTP middleware chain, including request duration histograms |
go_* | Runtime metrics from the Go collector |
process_* | Process metrics, on Linux and Windows |
The duration histogram is the one that earns its place on a dashboard, because it turns “the site feels slow” into a number per route.
histogram_quantile(0.95, sum(rate(caddy_http_request_duration_seconds_bucket[5m])) by (le))
Caddy can push metrics to an OpenTelemetry endpoint instead of, or as well as, serving them for scraping.
{
metrics {
otlp
}
}
The exporter is configured with environment variables, in the same style as the tracing configuration.
OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4318 \
OTEL_METRICS_EXPORTER=otlp \
caddy run --config /etc/caddy/Caddyfile
The default protocol is OTLP over HTTP with protobuf. Setting OTEL_EXPORTER_OTLP_PROTOCOL=grpc switches to gRPC. Headers, endpoints, and the collection interval are controlled by variables such as OTEL_EXPORTER_OTLP_METRICS_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS, and OTEL_METRIC_EXPORT_INTERVAL. Setting OTEL_METRICS_EXPORTER=none disables the export without editing the configuration.
A short list of signals that correspond to real incidents.
Non-2xx responses. A sustained rise in 5xx is the clearest signal that something is broken upstream.
sum(rate(caddy_http_requests_total{code=~"5.."}[5m]))
Latency. Watch the 95th percentile rather than the mean, which a single slow request barely moves.
Certificate expiry. Caddy renews on its own, but a renewal that fails is visible in the logs and worth alerting on before the certificate lapses.
Upstream health. The caddy_http_reverse_proxy_* metrics and the GET /reverse_proxy/upstreams endpoint both show whether a backend is being marked unhealthy.
Disk. Log files and the data directory are the two things that grow without bound.
| Mistake | What happens | Correction |
|---|---|---|
Expecting access logs without configuring log | No request log is produced at all | Add the log directive |
| Changing file output options and reloading | The change needs a restart | Restart, or use a new file name |
| Leaving rotation disabled | The disk fills up | Set roll_size and roll_keep |
| Shipping raw access logs to a collector | Credentials may appear in query strings | Use the filter encoder to redact |
| Enabling debug logging permanently | The volume and the cost are enormous | Turn it on while diagnosing, then remove it |
| Assuming the metrics endpoint is public | It exposes internals | Keep the admin endpoint on loopback |
Approach:
The goal is a service whose behaviour can be observed after the fact and whose health can be measured while it runs, without turning on debug logging and leaving it there. That splits into two independent halves. Logging answers the question of what happened to a specific request, and it needs a destination, a format, a rotation policy, and a redaction policy. Metrics answer the question of how the service is doing overall, and they need collection enabled and an endpoint a scraper can reach.
Steps:
- Separate the two streams in your head before configuring anything. Access logs come from the
logdirective inside a site block; runtime logs come from thelogglobal option. Configuring the wrong one is the usual reason a log never appears. - Enable access logging for the site with an explicit destination and an explicit format, rather than relying on the defaults, because the default destination is standard error and the default format depends on whether a terminal is attached.
- Set the rotation policy explicitly, including
roll_size,roll_keep,roll_keep_for, and the file mode, so that the file cannot fill the disk and cannot be read by every account on the machine. - Decide what happens to sensitive data before the first request is written. Use the
filterencoder to delete authorization headers, hash cookie values, and strip sensitive query parameters. The default redaction covers headers only, not the query string. - Exclude the noisy requests with
log_skip, at minimum the health checks, so that the signal is not drowned by the probes. - Enable metrics collection in the global options, and decide whether to label by host. Per-host labels are useful but increase cardinality, so they are only safe with a bounded set of hosts.
- Point the scraper at the admin endpoint, or expose the metrics on their own path with the
metricsdirective if the admin endpoint is not reachable from the scraper. - Write down the handful of numbers that will trigger a page, with the queries that produce them, so that the dashboard exists before the incident.
Answer:
The configuration looks like this.
{
admin localhost:2019
metrics {
per_host
}
log default {
output stdout
format json
exclude http.log.access
}
}
example.com {
log {
output file /var/log/caddy/example.com.access.log {
mode 0640
roll_size 100mb
roll_keep 10
roll_keep_for 720h
}
format filter {
wrap json
fields {
request>headers>Authorization delete
request>headers>Cookie hash
request>uri query {
delete token
}
request>remote_ip ip_mask 24 64
}
}
}
@health path /healthz
log_skip @health
log_append tenant {http.request.header.X-Tenant}
respond "ok" 200
}And the Prometheus scrape configuration looks like this.
global:
scrape_interval: 15s
scrape_configs:
- job_name: caddy
static_configs:
- targets: ['localhost:2019']The result is that every request is recorded in a rotated, permission-restricted file with credentials removed, that the health-check probes do not fill it, that the endpoint is reachable at http://localhost:2019/metrics for a scraper, and that the two queries worth watching are the rate of 5xx responses and the 95th percentile of the request duration histogram.
