Observability for Go Services
A service that misbehaves in production will always tell you something - the question is whether it tells you in a form you can act on. The three signals that matter are metrics (what is the service doing right now), traces (where did this one request spend its time), and logs (what exactly happened at this moment). Go has first-class support for all three, and the wiring is not hard once you know what each signal is for.
A metric is a number over time. The Prometheus client library for Go makes exporting them nearly free.
var (
requestTotal = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "http_requests_total",
Help: "Total number of HTTP requests handled.",
},
[]string{"method", "path", "status"},
)
requestDuration = prometheus.NewHistogramVec(
prometheus.HistogramOpts{
Name: "http_request_duration_seconds",
Help: "Latency of HTTP requests.",
Buckets: prometheus.DefBuckets,
},
[]string{"method", "path"},
)
)
func init() {
prometheus.MustRegister(requestTotal, requestDuration)
}
Instrumenting a handler is then two lines:
func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
start := time.Now()
recorder := promhttp.NewResponseController(w)
h.next.ServeHTTP(w, r)
status := recorder.Status()
requestTotal.WithLabelValues(r.Method, r.URL.Path, strconv.Itoa(status)).Inc()
requestDuration.WithLabelValues(r.Method, r.URL.Path).Observe(time.Since(start).Seconds())
}
Then expose the registry on a separate port, so that scraping never competes with real traffic:
go func() {
mux := http.NewServeMux()
mux.Handle("/metrics", promhttp.Handler())
if err := http.ListenAndServe(":9100", mux); err != nil {
log.Printf("metrics server: %v", err)
}
}()
Two rules keep a metrics endpoint useful rather than fatal:
- Never use unbounded label values. A label such as
pathmust be the route template (/orders/{id}), not the concrete path, or the number of time series grows without limit and takes the Prometheus server down with it. - Choose buckets for your latency distribution. The default buckets are tuned for typical web services; a service with a 200 ms target needs different ones.
A trace follows one request across the services it touches. The OpenTelemetry Go SDK is the current standard.
tracer := otel.Tracer("orders")
func (s *Service) Charge(ctx context.Context, amount int64) error {
ctx, span := tracer.Start(ctx, "charge")
defer span.End()
span.SetAttributes(attribute.Int64("amount", amount))
if err := s.gateway.Charge(ctx, amount); err != nil {
span.RecordError(err)
span.SetStatus(codes.Error, err.Error())
return err
}
return nil
}
The context carries the trace, so any function that accepts a context.Context participates without extra plumbing. The important discipline is to add attributes that let you group spans - a customer identifier, a shard, a cache hit - because a trace you cannot slice is a picture rather than an answer.
span.SetAttributes(
attribute.Bool("cache.hit", hit),
attribute.String("db.system", "postgres"),
)
Sample intelligently. A service that receives ten thousand requests per second cannot store every trace; the usual pattern is to sample all errors and a small fraction of successful requests, and to keep the sampling decision consistent across services so a trace is not half-recorded.
A log line that says error: failed tells you nothing at 3 a.m. A structured log tells you what failed, for which request, and where.
logger := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
Level: slog.LevelInfo,
}))
func (s *Service) Handle(w http.ResponseWriter, r *http.Request) {
logger.Info("request started",
"method", r.Method,
"path", r.URL.Path,
"request_id", RequestIDFromContext(r.Context()),
)
if err := s.do(r.Context()); err != nil {
logger.Error("request failed",
"error", err,
"request_id", RequestIDFromContext(r.Context()),
)
}
}
JSON output is worth the small loss in readability because it is what log aggregation tools can query. A log line should carry the same correlation identifier as the trace, so that a single request can be reconstructed by searching for one id.
The log/slog package in the standard library covers this since Go 1.21; the third-party zap and zerolog libraries are faster and remain a reasonable choice at very high volume.
A load balancer needs to know whether your instance should receive traffic. A readiness probe answers “can this instance serve requests right now?”, and a liveness probe answers “is this process still healthy?”. They are different questions and should be different endpoints.
mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
})
mux.HandleFunc("/readyz", func(w http.ResponseWriter, r *http.Request) {
ctx, cancel := context.WithTimeout(r.Context(), 2*time.Second)
defer cancel()
if err := s.db.PingContext(ctx); err != nil {
http.Error(w, "database unreachable", http.StatusServiceUnavailable)
return
}
w.WriteHeader(http.StatusOK)
})
A liveness probe that checks the database is a common and expensive mistake: when the database has a bad minute, every instance is restarted, and the outage gets worse.
Rather than sending metrics, traces, and logs to three different backends, most teams run the OpenTelemetry Collector as a single agent: your services export to it, and it fans out to the destinations.
receivers:
otlp:
protocols:
grpc:
http:
processors:
batch:
timeout: 5s
memory_limiter:
limit_percentage: 80
exporters:
prometheus:
otlp:
endpoint: collector-backend:4317
service:
pipelines:
metrics:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [prometheus]
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlp]
The collector gives you a place to redact sensitive attributes, to resample, and to switch backends without touching application code.
- Metrics answer “what is happening now”; keep their labels bounded.
- Traces answer “where did the time go”; attach the context and group by useful attributes.
- Logs answer “what exactly happened”; make them structured and carry the request id.
- Separate readiness from liveness, and never let a liveness probe depend on a remote service.
- Use the OpenTelemetry Collector to keep the fan-out and the redaction out of your code.
Approach: The question describes a service that is “slow” and asks which signal to reach for, so classify the symptom first: a service that is slow for all users is a capacity or dependency problem, a service that is slow for some requests is a code or data problem, and a service that is sometimes wrong is a correctness problem that only logs and traces can pin down.
Steps:
- If the slowdown is uniform across all users, the aggregate rate and latency histograms in Prometheus answer the question directly - compare the p95 and p99 latency before and after the deploy, and look at whether the error rate moved with it.
- If the slowdown affects only some requests, a per-route breakdown in the metrics narrows it down, and then a trace of one slow request shows exactly which span consumed the time.
- The correlation between “the deploy at 14:02” and “the p99 went up at 14:03” is usually the fastest path to the answer, so dashboards should be able to show a deploy marker.
- Traces are only useful if the service is instrumented end to end; if the slow call crosses a boundary that is not traced, the trace stops at the edge and the remaining time is unattributed - which is a strong argument for tracing every outbound call.
- Logs do not answer a latency question - they answer “what happened in this one instance” - so they are the last resort for performance, but the first resort for correctness.
Answer: Start with metrics to establish that the slowdown is real and to find which route and which percentile is affected, use the deploy timeline to correlate the change, then pull a trace of a slow request to locate the span that consumed the time, and only then read the structured logs of that request. Metrics tell you the scope, traces tell you the location, and logs tell you the cause.
