Kong Mastery

Every chapter so far has added something: objects, plugins, environments, deployment shapes. This chapter adds nothing new to configure. What it adds is a way to hold the whole thing in your head, and a way to act when it breaks.

The output is three things you can reuse. A mental model of the request lifecycle, so that a status code tells you where to look. A troubleshooting playbook organised by symptom rather than by technology, so that you do not have to remember which plugin produces which error. And a production readiness checklist, so that the things that are easy to forget are checked once, deliberately, rather than during an incident.

The mental model: six stages of a request

Every request through Kong passes through the same six stages, in this order. Most surprising behaviour comes from a stage that did not do what you expected.

client
  |
  v
[ 1. listener ]      Kong accepts the connection on 8000 (HTTP) or 8443 (HTTPS)
  |
  v
[ 2. router ]        Matches the request against every Route and picks one Service
  |                  No match -> 404, and no plugin ever runs
  v
[ 3. plugin chain ]  Access phase, in priority order: auth, rate limit, transform
  |                  A plugin can reject here with 401, 403 or 429
  v
[ 4. balancer ]      Chooses a target in the Upstream, using health and weights
  |                  No healthy target -> 503
  v
[ 5. upstream ]      Your application answers
  |
  v
[ 6. log phase ]     Response phase: transforms, then the access log line
StageWhat must be trueTypical symptom when it is not
ListenerThe connection reached a proxy port and the TLS handshake completedConnection refused, or a TLS handshake error
RouterExactly one Route matches the path, host, method and headers404 with no Route matched with those values
Plugin chainThe plugins for the matched scope are enabled and their handlers return401, 403, 429, or 500
BalancerAt least one target of the Upstream is healthy503 with no live upstreams
UpstreamThe application accepts the connection and answers in time502, 504, or a slow response
Log phaseThe access log line is writtenNothing in the log, even though traffic flowed

Three facts about this diagram explain most surprises.

The router runs before any plugin. A 404 therefore proves the request never reached authentication, whatever the body suggests. Check the Route before you check the credential.

Plugins run in priority order, not in the order you wrote them. Scope decides which instance runs: a plugin on a Route and a plugin on the Service it belongs to both run, and the more specific one takes precedence where their behaviour overlaps.

The response phase runs after the upstream has already answered. Anything a plugin does in that phase is too late to influence the upstream, and anything the upstream did is already decided.

The sixty-second triage

When something is broken, run these before you change anything.

# 1. Is the node alive, ready and reachable?
kong health
curl -s http://localhost:8001/status | jq '.server'

# 2. What does the gateway return for the failing request?
curl -i http://localhost:8000/orders

# 3. What does the configuration say?
curl -s http://localhost:8001/routes | jq '.[] | {name, paths, methods, service}'
curl -s http://localhost:8001/plugins | jq '.[] | {name, service, route, consumer}'

# 4. What does the upstream say, when called from the gateway host?
curl -i http://orders.internal:8080/healthz

The jq calls are a convenience - the Admin API returns plain JSON, so you can read it in a browser as well.

The fourth command is the one people skip and the one that most often finds the real cause, because it separates “the gateway is broken” from “the upstream is broken”. A gateway is the only component in the path that can see every request, which makes it the natural suspect for every problem and, most of the time, the wrong one.

Troubleshooting by symptom

404: the router did not match

Symptom. 404, with the body {"message":"no Route matched with those values"}.

Locate it.

curl -s http://localhost:8001/routes | jq '.[] | {name, paths, methods, hosts, protocols, service}'

Common causes.

  • The client calls a path no Route declares. Paths are prefix matches, so /orders also matches /orders/42, but /api/orders does not match a route on /orders.
  • The Route restricts methods or headers and the request differs. A browser CORS preflight is an OPTIONS request, and a Route listing only GET and POST will not match it.
  • The request arrived over HTTPS while the Route declares only http protocols, or on a listener the Route does not belong to.
  • The path is a regular expression - a path beginning with ~ - and the request does not satisfy the expression.
  • The upstream itself answered 404. At the client this looks identical to a routing failure, and it is the most common misdiagnosis. Check the upstream_status field in the access log, or call the upstream directly.

Fix. Correct the Route’s paths, methods and protocols. If you edited a declarative file, run kong config parse on it and then kong reload the node.

503: no healthy upstream

Symptom. 503 with {"message":"no live upstreams"}, which Kong generates itself, or a 502 when the upstream closes the connection.

Locate it.

curl -s http://localhost:8001/upstreams/orders-pool/health | jq '.[] | {target, health}'
curl -s http://localhost:8001/services/orders | jq '{host, port, protocol, retries}'

Common causes.

  • Active health checks fail: the http_path points at an endpoint that does not exist, or at one that needs a credential the check does not send.
  • The upstream is unreachable from the gateway - DNS, network policy, or the service is down.
  • The Service points at an Upstream that does not exist, or at one whose targets are all weighted to zero.
  • The upstream is answering but too slowly, and the request times out on the way back.

Fix. Fix the health check path or the network path, then confirm from the gateway host that the health endpoint answers. Check that the Service’s host and port are the ones the upstream actually listens on.

429: the rate limit rejected the request

Symptom. 429 with {"message":"API rate limit exceeded"}.

Locate it.

curl -s "http://localhost:8001/plugins?name=rate-limiting" \
  | jq '.[] | {id, service, route, consumer, limit: .config.minute, by: .config.limit_by, policy: .config.policy}'

Common causes.

  • The limit really was reached.
  • limit_by: ip while many callers share one address behind NAT, so they share one bucket.
  • policy: local on a gateway with several nodes: each node keeps its own counter, so the effective limit is the configured limit multiplied by the number of nodes.
  • policy: redis with fault_tolerant: true while Redis is unreachable: the limit is quietly not applied.
  • A global plugin attached to the whole gateway, throttling routes nobody intended to throttle.

Fix. Attach the plugin to the Route or Service that owns the quota, use limit_by: consumer when callers authenticate, and move to a shared counter when the gateway runs on more than one node.

CORS headers are missing

Symptom. The browser console reports a CORS error while curl shows a perfectly good response. The response carries no Access-Control-Allow-Origin header.

Locate it. Replay the preflight by hand:

curl -i -X OPTIONS http://localhost:8000/orders \
  -H "Origin: https://app.example.com" \
  -H "Access-Control-Request-Method: POST" \
  -H "Access-Control-Request-Headers: apikey"

Common causes.

  • The cors plugin is not attached to the Route the browser is calling. Scope matters: on the Service it covers every Route of that Service, on the Route it covers only that one.
  • origins does not list the browser’s origin. The plugin matches an exact origin or the single value *; a pattern such as *.example.com is not supported.
  • credentials: true together with origins: ["*"], which browsers reject.
  • The Route restricts methods to GET and POST, so the OPTIONS preflight matches no Route and comes back as 404 before the CORS plugin runs at all.
  • The browser sends a request header that the plugin does not allow.

Fix. Attach the cors plugin to the Route, add OPTIONS to the Route’s methods if you restrict them, and list the exact origins the browser uses.

401 and 403

Symptom. 401 with {"message":"Unauthorized"} from an authentication plugin, or 403 with {"message":"Forbidden"} from the ACL plugin.

Locate it.

curl -s "http://localhost:8001/plugins?name=key-auth" | jq '.[] | {id, service, route, consumer}'
curl -s http://localhost:8001/consumers/payments-service/key-auth | jq '.[] | {id, key}'

Common causes for 401.

  • The credential is in the wrong header. key_names defaults to apikey, so a client sending X-Api-Key is rejected.
  • The credential exists but is not attached to a consumer. The plugin looks the key up through consumers, so an orphaned credential matches nobody.
  • The plugin is attached at a scope the request does not pass through, or a second plugin on a narrower scope has replaced it.

Common causes for 403.

  • The consumer’s credential is not in a group named in the ACL plugin’s allow list. allow and deny are mutually exclusive.

Fix. Align key_names with what the client actually sends, move the credential under its consumer, and add the consumer to the allowed group.

The TLS handshake fails

Symptom. curl reports an SSL error, or the browser says the connection is not private.

Locate it.

curl -v https://api.example.com/orders
openssl s_client -connect api.example.com:443 -servername api.example.com </dev/null

Read the certificate’s subject, its alternative names and its expiry date.

Common causes.

  • The certificate does not cover the requested host name. The name the client asks for must appear in the certificate’s subject alternative names.
  • The certificate expired, or the chain is incomplete because an intermediate certificate is missing.
  • The proxy listener on 8443 has no certificate configured. The default listener needs ssl_cert and ssl_cert_key, or the KONG_SSL_CERT and KONG_SSL_CERT_KEY environment variables.
  • The client and the gateway disagree about the TLS version. Kong’s default ssl_protocols allows TLS 1.2 and TLS 1.3.
  • SNI selects a different server block than the one you expect when several certificates share a listener.

Fix. Issue a certificate for the exact host name, serve the full chain, and include the TLS version the client needs in ssl_protocols.

The Admin API refuses connections

Symptom. curl reports connection refused on port 8001, while the proxy port 8000 answers normally.

Locate it.

curl -s http://127.0.0.1:8001/status
kong health

Common causes.

  • The Admin API listens on the loopback address by default. The shipped configuration binds admin_listen to 127.0.0.1:8001, so nothing outside the node can reach it - and a container run without the port published behaves exactly the same way.
  • The container runtime does not publish port 8001.

Fix. Set KONG_ADMIN_LISTEN=0.0.0.0:8001, or better a specific internal address, publish the port, and restrict it with a firewall or a security group. The Admin API is unauthenticated by default, so it must never be reachable from the public internet. In DB-less mode there is no database behind it, so it is effectively read-only.

The declarative configuration did not take effect

Symptom. The file on disk is correct, but the running gateway does not behave as the file describes, or the Admin API does not list the objects at all.

Locate it.

kong config parse kong.yml
ls -l "$KONG_DECLARATIVE_CONFIG"
grep -i declarative /var/log/kong/error.log

Common causes.

  • KONG_DECLARATIVE_CONFIG points at a path that does not exist inside the container, or at a file the user Kong runs as cannot read. The node logs the failure and never becomes ready.
  • The file was edited but the node was never told to re-read it. Kong reads the declarative file at startup and on a reload; an edit on disk changes nothing by itself. Run kong reload.
  • The node runs in database mode with a declarative file. The file is imported at startup, and after that the database is the source of truth. The file is ignored until you import it again with kong config db_import kong.yml.

Fix. Parse, then reload in DB-less mode; parse, then import in database mode.

A plugin timed out

Symptom. 500 from the gateway, and the error log mentions a plugin that did not finish.

Locate it.

grep -i "timed out" /var/log/kong/error.log
grep -i "plugin" /var/log/kong/error.log | tail -20

Common causes.

  • A plugin handler blocked on an external call - a slow identity provider, a Redis round trip, a DNS lookup - and exceeded the plugin iteration budget set by KONG_PLUGIN_ITERATION_TIMEOUT.
  • The plugin’s own timeout setting is larger than that budget, so it cannot fail fast.
  • The work belongs in the log phase but is being done in the access phase, where it holds the request.

Fix. Give the plugin a timeout shorter than the iteration budget, move logging out of the access phase, and put any external call in a queue or in the log phase.

Performance and capacity

Workers, connections and file descriptors

Kong runs on NGINX, which uses one process per core with an event-driven worker inside each process.

  • KONG_NGINX_WORKER_PROCESSES defaults to auto, one worker per core. On a container with a CPU limit, set it to the limit rather than to the core count of the host, or the workers fight over the limit.
  • KONG_NGINX_EVENTS_WORKER_CONNECTIONS controls how many simultaneous connections one worker can hold. The shipped default is auto.
  • The file descriptor limit has to sit above the connection count. nginx_main_worker_rlimit_nofile defaults to auto, which means it follows the limit the process was given.
KONG_NGINX_WORKER_PROCESSES=4
KONG_NGINX_EVENTS_WORKER_CONNECTIONS=4096

Measure before you tune: curl -s http://localhost:8001/status | jq '.server' reports how many connections the node is currently handling.

Keepalive

Opening a connection to the upstream for every request is expensive when the network has latency. Kong keeps a pool of idle upstream connections per worker, controlled by KONG_UPSTREAM_KEEPALIVE_POOL_SIZE, KONG_UPSTREAM_KEEPALIVE_MAX_REQUESTS and KONG_UPSTREAM_KEEPALIVE_IDLE_TIMEOUT. The upstream has to cooperate: an upstream that closes every connection makes the pool worthless.

Buffers and timeouts

Request and response buffering is on by default, because it is what makes a retry safe and lets plugins work with the whole body. For large payloads it is pure overhead. KONG_PROXY_REQUEST_BUFFERING and KONG_PROXY_RESPONSE_BUFFERING turn it off - per route rather than globally, and only once you know that no plugin on that route needs the buffered body.

The service-level timeouts are connect_timeout, read_timeout and write_timeout, each 60000 milliseconds by default. A gateway that waits a full minute for a dead upstream holds a worker connection for that whole minute. A few seconds is usually the better trade for an API - and the upstream’s own health is the most common cause of a slow gateway, so measure it directly with curl -s -o /dev/null -w "ttfb:%{time_starttransfer} total:%{time_total}\n" http://localhost:8000/orders before tuning the gateway.

Separating the Admin API from the data plane

In hybrid mode the control plane owns the configuration and the Admin API, and the data planes handle the traffic. Data planes are stateless and interchangeable, so one can be replaced or scaled at any time. The control plane is what needs a database, a backup and a careful network position - and it is the thing that must never sit on a public address.

DB-less or database mode

DB-lessDatabase
Source of truthOne YAML file in GitPostgreSQL
How it changesEdit the file, reload the nodeAdmin API calls, or a file imported with kong config db_import
Audit trailThe commit historyThe database, plus whatever logs you keep
Several nodesEach node loads the file itselfOne control plane pushes to every data plane
Best forKubernetes, small deployments, configuration as codeSeveral editors at runtime, an operator console, an internal portal

The choice is not really about features. It is about who is allowed to change the gateway at three in the morning, and where that change is recorded.

Production readiness checklist

Work through this before the gateway carries real traffic.

  • kong config parse passes on the exact file that will be deployed, and that command runs in CI.
  • The deployment pipeline syncs the configuration - deck gateway sync or kong config db_import - and deck gateway diff against production fails the build when the live gateway has drifted.
  • Every Route has an explicit paths value, an explicit strip_path decision, and an owner recorded in tags.
  • The plugin set is as small as the requirement allows, and each plugin is attached at the narrowest scope that satisfies it.
  • Authentication and rate limiting are on by default rather than opt-in, so a route published without a credential is a mistake the gateway refuses.
  • The Admin API is not reachable from outside the trusted network: admin_listen is bound to an internal address and the port is blocked at the firewall.
  • Certificates cover every host name the gateway answers on, the chain is complete, and the expiry date is monitored.
  • DNS for the API host name points at the gateway, and the load balancer’s health check targets a port the gateway actually listens on.
  • GET /status answers on every node, and the orchestrator’s readiness probe uses it.
  • Access logs, metrics and traces all carry the same correlation identifier.
  • The upstream’s health endpoint is checked from the gateway host, not only from a laptop.
  • Timeouts are set deliberately on every service, and they are shorter than the client’s own timeout.
  • There is a rollback: the previous configuration file is tagged, the previous image is available, and somebody has actually performed a rollback once.

Anti-patterns

Kong as a business logic layer. The moment a plugin contains a rule about your domain, the gateway becomes a place where business behaviour lives - and it is the worst possible place for it: hard to test, hard to change, and shared by everyone. If a rule can live in the service, it must live in the service.

A long plugin chain. Every plugin adds work to the request path, and every plugin is a place where the request can be rejected. Ten plugins on one route is usually ten plugins too many. Split the routes instead, so each carries only the policy it needs.

The Admin API exposed without a boundary. Port 8001 is unauthenticated by default and can rewrite every route and plugin on the gateway. Bind it internally and treat network access to it as a privileged operation.

Two sources of truth. Editing the Admin API by hand while also keeping a declarative file in Git produces drift, and drift is discovered during an incident. Pick one - the file or the API - and make the other read-only.

Health checks that only check the process. A node that answers /status can still be routing every request to a dead upstream. The upstream’s health matters more than the gateway’s.

Practice

A gateway has been running for a year. Since this morning it has returned 503 for one API only; the other APIs through the same gateway are fine. The team has already changed the rate limits, the authentication plugin and the declarative file, without effect.

Walk through the troubleshooting playbook: state which stage of the request lifecycle the symptom points at, which three commands you would run first, and what you would conclude from each. Then say which part of the configuration you would check before changing anything.

kongtroubleshootingoperations

Approach:

The symptom is the clue. One API returns 503 while every other API on the same gateway is healthy, which rules out all the stages that every request shares - the listener, the node itself, the plugin chain - and points at the two stages that belong to this API alone: the balancer and the upstream. A 503 from Kong is generated before the request reaches the application, so the application’s own behaviour cannot explain the status code the client sees. What the application can explain is why Kong stopped sending traffic to it.

The second half of the question is discipline. The team already changed three things, and every change was made without evidence. The playbook is ordered so that each step narrows the search and the fix comes last. The important move is to resist changing configuration while the cause is unknown, because a 503 caused by a failed health check gets worse, not better, when you edit the rate limit.

Steps:

  1. Establish who produced the status code. curl -i http://localhost:8000/orders and read the body. Kong’s own failure is {"message":"no live upstreams"}; an application that fails returns its own body and its own status code. This one command separates “Kong refused” from “the upstream refused”.

  2. Check target health. curl -s http://localhost:8001/upstreams/orders-pool/health | jq '.[] | {target, health}'. If every target is unhealthy the balancer has nothing to send to, and the answer is in the health check configuration rather than in the plugins.

  3. Check the health check against the upstream itself, from the gateway host: curl -i http://orders-1.internal:8080/healthz. The usual cause is a health path the application no longer serves, or one that requires a credential the active check does not send.

  4. Check that the Service points at the right place. curl -s http://localhost:8001/services/orders | jq '{host, port, protocol}'. A stale port after a deployment is the second most common cause.

  5. Look at the change, not at the last change. The rate limit produces 429 and the authentication plugin produces 401; neither produces 503. Only a change that touched the upstream, its targets or the health checks can explain this symptom, and deck gateway diff against production shows exactly that - and at the same time shows whether the live gateway has drifted from the file.

  6. Change something only now. Fix the health check path, the port or the target, and watch the target health output until it reports healthy before calling it fixed.

Answer:

StageEvidenceConclusion
Listener and routerOther APIs on the same gateway workThe node, the listener and the router are healthy
Plugin chainThe body is no live upstreams, not API rate limit exceeded or UnauthorizedThe plugins are not the cause; the request was rejected after the plugin chain ran
BalancerEvery target in the upstream health output is unhealthyThe balancer has no healthy target to pick
Upstreamcurl to the health path from the gateway host failsThe upstream is unhealthy or unreachable, and the health check is telling the truth

The configuration to check before touching anything is the health check on the upstream: its http_path, whether that endpoint still exists on the deployed version of the application, and whether it needs a credential the check does not send. Then the Service’s host and port, and finally the upstream’s target list.

The team’s three changes are a red herring, and the point of saying so explicitly is that they were made without evidence. The rate limit produces 429, the authentication plugin produces 401, and neither produces 503. The only change that could explain this symptom is one that touched the upstream, its targets or the health checks - and deck gateway diff against production will show whether it did, and whether the live gateway has drifted from the committed file at the same time.

More courses

Recommended courses

Found an issue with this content?

Tell us what could be improved.