Kong Mastery
Every chapter so far has added something: objects, plugins, environments, deployment shapes. This chapter adds nothing new to configure. What it adds is a way to hold the whole thing in your head, and a way to act when it breaks.
The output is three things you can reuse. A mental model of the request lifecycle, so that a status code tells you where to look. A troubleshooting playbook organised by symptom rather than by technology, so that you do not have to remember which plugin produces which error. And a production readiness checklist, so that the things that are easy to forget are checked once, deliberately, rather than during an incident.
Every request through Kong passes through the same six stages, in this order. Most surprising behaviour comes from a stage that did not do what you expected.
client
|
v
[ 1. listener ] Kong accepts the connection on 8000 (HTTP) or 8443 (HTTPS)
|
v
[ 2. router ] Matches the request against every Route and picks one Service
| No match -> 404, and no plugin ever runs
v
[ 3. plugin chain ] Access phase, in priority order: auth, rate limit, transform
| A plugin can reject here with 401, 403 or 429
v
[ 4. balancer ] Chooses a target in the Upstream, using health and weights
| No healthy target -> 503
v
[ 5. upstream ] Your application answers
|
v
[ 6. log phase ] Response phase: transforms, then the access log line
| Stage | What must be true | Typical symptom when it is not |
|---|---|---|
| Listener | The connection reached a proxy port and the TLS handshake completed | Connection refused, or a TLS handshake error |
| Router | Exactly one Route matches the path, host, method and headers | 404 with no Route matched with those values |
| Plugin chain | The plugins for the matched scope are enabled and their handlers return | 401, 403, 429, or 500 |
| Balancer | At least one target of the Upstream is healthy | 503 with no live upstreams |
| Upstream | The application accepts the connection and answers in time | 502, 504, or a slow response |
| Log phase | The access log line is written | Nothing in the log, even though traffic flowed |
Three facts about this diagram explain most surprises.
The router runs before any plugin. A 404 therefore proves the request never reached authentication, whatever the body suggests. Check the Route before you check the credential.
Plugins run in priority order, not in the order you wrote them. Scope decides which instance runs: a plugin on a Route and a plugin on the Service it belongs to both run, and the more specific one takes precedence where their behaviour overlaps.
The response phase runs after the upstream has already answered. Anything a plugin does in that phase is too late to influence the upstream, and anything the upstream did is already decided.
When something is broken, run these before you change anything.
# 1. Is the node alive, ready and reachable?
kong health
curl -s http://localhost:8001/status | jq '.server'
# 2. What does the gateway return for the failing request?
curl -i http://localhost:8000/orders
# 3. What does the configuration say?
curl -s http://localhost:8001/routes | jq '.[] | {name, paths, methods, service}'
curl -s http://localhost:8001/plugins | jq '.[] | {name, service, route, consumer}'
# 4. What does the upstream say, when called from the gateway host?
curl -i http://orders.internal:8080/healthz
The jq calls are a convenience - the Admin API returns plain JSON, so you can read it in a browser as well.
The fourth command is the one people skip and the one that most often finds the real cause, because it separates “the gateway is broken” from “the upstream is broken”. A gateway is the only component in the path that can see every request, which makes it the natural suspect for every problem and, most of the time, the wrong one.
404: the router did not match
Symptom. 404, with the body {"message":"no Route matched with those values"}.
Locate it.
curl -s http://localhost:8001/routes | jq '.[] | {name, paths, methods, hosts, protocols, service}'
Common causes.
- The client calls a path no Route declares. Paths are prefix matches, so
/ordersalso matches/orders/42, but/api/ordersdoes not match a route on/orders. - The Route restricts
methodsorheadersand the request differs. A browser CORS preflight is anOPTIONSrequest, and a Route listing onlyGETandPOSTwill not match it. - The request arrived over HTTPS while the Route declares only
httpprotocols, or on a listener the Route does not belong to. - The path is a regular expression - a path beginning with
~- and the request does not satisfy the expression. - The upstream itself answered
404. At the client this looks identical to a routing failure, and it is the most common misdiagnosis. Check theupstream_statusfield in the access log, or call the upstream directly.
Fix. Correct the Route’s paths, methods and protocols. If you edited a declarative file, run kong config parse on it and then kong reload the node.
503: no healthy upstream
Symptom. 503 with {"message":"no live upstreams"}, which Kong generates itself, or a 502 when the upstream closes the connection.
Locate it.
curl -s http://localhost:8001/upstreams/orders-pool/health | jq '.[] | {target, health}'
curl -s http://localhost:8001/services/orders | jq '{host, port, protocol, retries}'
Common causes.
- Active health checks fail: the
http_pathpoints at an endpoint that does not exist, or at one that needs a credential the check does not send. - The upstream is unreachable from the gateway - DNS, network policy, or the service is down.
- The Service points at an Upstream that does not exist, or at one whose targets are all weighted to zero.
- The upstream is answering but too slowly, and the request times out on the way back.
Fix. Fix the health check path or the network path, then confirm from the gateway host that the health endpoint answers. Check that the Service’s host and port are the ones the upstream actually listens on.
429: the rate limit rejected the request
Symptom. 429 with {"message":"API rate limit exceeded"}.
Locate it.
curl -s "http://localhost:8001/plugins?name=rate-limiting" \
| jq '.[] | {id, service, route, consumer, limit: .config.minute, by: .config.limit_by, policy: .config.policy}'
Common causes.
- The limit really was reached.
limit_by: ipwhile many callers share one address behind NAT, so they share one bucket.policy: localon a gateway with several nodes: each node keeps its own counter, so the effective limit is the configured limit multiplied by the number of nodes.policy: rediswithfault_tolerant: truewhile Redis is unreachable: the limit is quietly not applied.- A global plugin attached to the whole gateway, throttling routes nobody intended to throttle.
Fix. Attach the plugin to the Route or Service that owns the quota, use limit_by: consumer when callers authenticate, and move to a shared counter when the gateway runs on more than one node.
CORS headers are missing
Symptom. The browser console reports a CORS error while curl shows a perfectly good response. The response carries no Access-Control-Allow-Origin header.
Locate it. Replay the preflight by hand:
curl -i -X OPTIONS http://localhost:8000/orders \
-H "Origin: https://app.example.com" \
-H "Access-Control-Request-Method: POST" \
-H "Access-Control-Request-Headers: apikey"
Common causes.
- The
corsplugin is not attached to the Route the browser is calling. Scope matters: on the Service it covers every Route of that Service, on the Route it covers only that one. originsdoes not list the browser’s origin. The plugin matches an exact origin or the single value*; a pattern such as*.example.comis not supported.credentials: truetogether withorigins: ["*"], which browsers reject.- The Route restricts
methodstoGETandPOST, so theOPTIONSpreflight matches no Route and comes back as404before the CORS plugin runs at all. - The browser sends a request header that the plugin does not allow.
Fix. Attach the cors plugin to the Route, add OPTIONS to the Route’s methods if you restrict them, and list the exact origins the browser uses.
401 and 403
Symptom. 401 with {"message":"Unauthorized"} from an authentication plugin, or 403 with {"message":"Forbidden"} from the ACL plugin.
Locate it.
curl -s "http://localhost:8001/plugins?name=key-auth" | jq '.[] | {id, service, route, consumer}'
curl -s http://localhost:8001/consumers/payments-service/key-auth | jq '.[] | {id, key}'
Common causes for 401.
- The credential is in the wrong header.
key_namesdefaults toapikey, so a client sendingX-Api-Keyis rejected. - The credential exists but is not attached to a consumer. The plugin looks the key up through consumers, so an orphaned credential matches nobody.
- The plugin is attached at a scope the request does not pass through, or a second plugin on a narrower scope has replaced it.
Common causes for 403.
- The consumer’s credential is not in a group named in the ACL plugin’s
allowlist.allowanddenyare mutually exclusive.
Fix. Align key_names with what the client actually sends, move the credential under its consumer, and add the consumer to the allowed group.
The TLS handshake fails
Symptom. curl reports an SSL error, or the browser says the connection is not private.
Locate it.
curl -v https://api.example.com/orders
openssl s_client -connect api.example.com:443 -servername api.example.com </dev/null
Read the certificate’s subject, its alternative names and its expiry date.
Common causes.
- The certificate does not cover the requested host name. The name the client asks for must appear in the certificate’s subject alternative names.
- The certificate expired, or the chain is incomplete because an intermediate certificate is missing.
- The proxy listener on
8443has no certificate configured. The default listener needsssl_certandssl_cert_key, or theKONG_SSL_CERTandKONG_SSL_CERT_KEYenvironment variables. - The client and the gateway disagree about the TLS version. Kong’s default
ssl_protocolsallows TLS 1.2 and TLS 1.3. - SNI selects a different server block than the one you expect when several certificates share a listener.
Fix. Issue a certificate for the exact host name, serve the full chain, and include the TLS version the client needs in ssl_protocols.
The Admin API refuses connections
Symptom. curl reports connection refused on port 8001, while the proxy port 8000 answers normally.
Locate it.
curl -s http://127.0.0.1:8001/status
kong health
Common causes.
- The Admin API listens on the loopback address by default. The shipped configuration binds
admin_listento127.0.0.1:8001, so nothing outside the node can reach it - and a container run without the port published behaves exactly the same way. - The container runtime does not publish port
8001.
Fix. Set KONG_ADMIN_LISTEN=0.0.0.0:8001, or better a specific internal address, publish the port, and restrict it with a firewall or a security group. The Admin API is unauthenticated by default, so it must never be reachable from the public internet. In DB-less mode there is no database behind it, so it is effectively read-only.
The declarative configuration did not take effect
Symptom. The file on disk is correct, but the running gateway does not behave as the file describes, or the Admin API does not list the objects at all.
Locate it.
kong config parse kong.yml
ls -l "$KONG_DECLARATIVE_CONFIG"
grep -i declarative /var/log/kong/error.log
Common causes.
KONG_DECLARATIVE_CONFIGpoints at a path that does not exist inside the container, or at a file the user Kong runs as cannot read. The node logs the failure and never becomes ready.- The file was edited but the node was never told to re-read it. Kong reads the declarative file at startup and on a reload; an edit on disk changes nothing by itself. Run
kong reload. - The node runs in database mode with a declarative file. The file is imported at startup, and after that the database is the source of truth. The file is ignored until you import it again with
kong config db_import kong.yml.
Fix. Parse, then reload in DB-less mode; parse, then import in database mode.
A plugin timed out
Symptom. 500 from the gateway, and the error log mentions a plugin that did not finish.
Locate it.
grep -i "timed out" /var/log/kong/error.log
grep -i "plugin" /var/log/kong/error.log | tail -20
Common causes.
- A plugin handler blocked on an external call - a slow identity provider, a Redis round trip, a DNS lookup - and exceeded the plugin iteration budget set by
KONG_PLUGIN_ITERATION_TIMEOUT. - The plugin’s own
timeoutsetting is larger than that budget, so it cannot fail fast. - The work belongs in the log phase but is being done in the access phase, where it holds the request.
Fix. Give the plugin a timeout shorter than the iteration budget, move logging out of the access phase, and put any external call in a queue or in the log phase.
Workers, connections and file descriptors
Kong runs on NGINX, which uses one process per core with an event-driven worker inside each process.
KONG_NGINX_WORKER_PROCESSESdefaults toauto, one worker per core. On a container with a CPU limit, set it to the limit rather than to the core count of the host, or the workers fight over the limit.KONG_NGINX_EVENTS_WORKER_CONNECTIONScontrols how many simultaneous connections one worker can hold. The shipped default isauto.- The file descriptor limit has to sit above the connection count.
nginx_main_worker_rlimit_nofiledefaults toauto, which means it follows the limit the process was given.
KONG_NGINX_WORKER_PROCESSES=4
KONG_NGINX_EVENTS_WORKER_CONNECTIONS=4096
Measure before you tune: curl -s http://localhost:8001/status | jq '.server' reports how many connections the node is currently handling.
Keepalive
Opening a connection to the upstream for every request is expensive when the network has latency. Kong keeps a pool of idle upstream connections per worker, controlled by KONG_UPSTREAM_KEEPALIVE_POOL_SIZE, KONG_UPSTREAM_KEEPALIVE_MAX_REQUESTS and KONG_UPSTREAM_KEEPALIVE_IDLE_TIMEOUT. The upstream has to cooperate: an upstream that closes every connection makes the pool worthless.
Buffers and timeouts
Request and response buffering is on by default, because it is what makes a retry safe and lets plugins work with the whole body. For large payloads it is pure overhead. KONG_PROXY_REQUEST_BUFFERING and KONG_PROXY_RESPONSE_BUFFERING turn it off - per route rather than globally, and only once you know that no plugin on that route needs the buffered body.
The service-level timeouts are connect_timeout, read_timeout and write_timeout, each 60000 milliseconds by default. A gateway that waits a full minute for a dead upstream holds a worker connection for that whole minute. A few seconds is usually the better trade for an API - and the upstream’s own health is the most common cause of a slow gateway, so measure it directly with curl -s -o /dev/null -w "ttfb:%{time_starttransfer} total:%{time_total}\n" http://localhost:8000/orders before tuning the gateway.
Separating the Admin API from the data plane
In hybrid mode the control plane owns the configuration and the Admin API, and the data planes handle the traffic. Data planes are stateless and interchangeable, so one can be replaced or scaled at any time. The control plane is what needs a database, a backup and a careful network position - and it is the thing that must never sit on a public address.
DB-less or database mode
| DB-less | Database | |
|---|---|---|
| Source of truth | One YAML file in Git | PostgreSQL |
| How it changes | Edit the file, reload the node | Admin API calls, or a file imported with kong config db_import |
| Audit trail | The commit history | The database, plus whatever logs you keep |
| Several nodes | Each node loads the file itself | One control plane pushes to every data plane |
| Best for | Kubernetes, small deployments, configuration as code | Several editors at runtime, an operator console, an internal portal |
The choice is not really about features. It is about who is allowed to change the gateway at three in the morning, and where that change is recorded.
Work through this before the gateway carries real traffic.
-
kong config parsepasses on the exact file that will be deployed, and that command runs in CI. - The deployment pipeline syncs the configuration -
deck gateway syncorkong config db_import- anddeck gateway diffagainst production fails the build when the live gateway has drifted. - Every Route has an explicit
pathsvalue, an explicitstrip_pathdecision, and an owner recorded intags. - The plugin set is as small as the requirement allows, and each plugin is attached at the narrowest scope that satisfies it.
- Authentication and rate limiting are on by default rather than opt-in, so a route published without a credential is a mistake the gateway refuses.
- The Admin API is not reachable from outside the trusted network:
admin_listenis bound to an internal address and the port is blocked at the firewall. - Certificates cover every host name the gateway answers on, the chain is complete, and the expiry date is monitored.
- DNS for the API host name points at the gateway, and the load balancer’s health check targets a port the gateway actually listens on.
-
GET /statusanswers on every node, and the orchestrator’s readiness probe uses it. - Access logs, metrics and traces all carry the same correlation identifier.
- The upstream’s health endpoint is checked from the gateway host, not only from a laptop.
- Timeouts are set deliberately on every service, and they are shorter than the client’s own timeout.
- There is a rollback: the previous configuration file is tagged, the previous image is available, and somebody has actually performed a rollback once.
Kong as a business logic layer. The moment a plugin contains a rule about your domain, the gateway becomes a place where business behaviour lives - and it is the worst possible place for it: hard to test, hard to change, and shared by everyone. If a rule can live in the service, it must live in the service.
A long plugin chain. Every plugin adds work to the request path, and every plugin is a place where the request can be rejected. Ten plugins on one route is usually ten plugins too many. Split the routes instead, so each carries only the policy it needs.
The Admin API exposed without a boundary. Port 8001 is unauthenticated by default and can rewrite every route and plugin on the gateway. Bind it internally and treat network access to it as a privileged operation.
Two sources of truth. Editing the Admin API by hand while also keeping a declarative file in Git produces drift, and drift is discovered during an incident. Pick one - the file or the API - and make the other read-only.
Health checks that only check the process. A node that answers /status can still be routing every request to a dead upstream. The upstream’s health matters more than the gateway’s.
A gateway has been running for a year. Since this morning it has returned 503 for one API only; the other APIs through the same gateway are fine. The team has already changed the rate limits, the authentication plugin and the declarative file, without effect.
Walk through the troubleshooting playbook: state which stage of the request lifecycle the symptom points at, which three commands you would run first, and what you would conclude from each. Then say which part of the configuration you would check before changing anything.
Approach:
The symptom is the clue. One API returns 503 while every other API on the same gateway is healthy, which rules out all the stages that every request shares - the listener, the node itself, the plugin chain - and points at the two stages that belong to this API alone: the balancer and the upstream. A 503 from Kong is generated before the request reaches the application, so the application’s own behaviour cannot explain the status code the client sees. What the application can explain is why Kong stopped sending traffic to it.
The second half of the question is discipline. The team already changed three things, and every change was made without evidence. The playbook is ordered so that each step narrows the search and the fix comes last. The important move is to resist changing configuration while the cause is unknown, because a 503 caused by a failed health check gets worse, not better, when you edit the rate limit.
Steps:
-
Establish who produced the status code.
curl -i http://localhost:8000/ordersand read the body. Kong’s own failure is{"message":"no live upstreams"}; an application that fails returns its own body and its own status code. This one command separates “Kong refused” from “the upstream refused”. -
Check target health.
curl -s http://localhost:8001/upstreams/orders-pool/health | jq '.[] | {target, health}'. If every target is unhealthy the balancer has nothing to send to, and the answer is in the health check configuration rather than in the plugins. -
Check the health check against the upstream itself, from the gateway host:
curl -i http://orders-1.internal:8080/healthz. The usual cause is a health path the application no longer serves, or one that requires a credential the active check does not send. -
Check that the Service points at the right place.
curl -s http://localhost:8001/services/orders | jq '{host, port, protocol}'. A stale port after a deployment is the second most common cause. -
Look at the change, not at the last change. The rate limit produces
429and the authentication plugin produces401; neither produces503. Only a change that touched the upstream, its targets or the health checks can explain this symptom, anddeck gateway diffagainst production shows exactly that - and at the same time shows whether the live gateway has drifted from the file. -
Change something only now. Fix the health check path, the port or the target, and watch the target health output until it reports healthy before calling it fixed.
Answer:
| Stage | Evidence | Conclusion |
|---|---|---|
| Listener and router | Other APIs on the same gateway work | The node, the listener and the router are healthy |
| Plugin chain | The body is no live upstreams, not API rate limit exceeded or Unauthorized | The plugins are not the cause; the request was rejected after the plugin chain ran |
| Balancer | Every target in the upstream health output is unhealthy | The balancer has no healthy target to pick |
| Upstream | curl to the health path from the gateway host fails | The upstream is unhealthy or unreachable, and the health check is telling the truth |
The configuration to check before touching anything is the health check on the upstream: its http_path, whether that endpoint still exists on the deployed version of the application, and whether it needs a credential the check does not send. Then the Service’s host and port, and finally the upstream’s target list.
The team’s three changes are a red herring, and the point of saying so explicitly is that they were made without evidence. The rate limit produces 429, the authentication plugin produces 401, and neither produces 503. The only change that could explain this symptom is one that touched the upstream, its targets or the health checks - and deck gateway diff against production will show whether it did, and whether the live gateway has drifted from the committed file at the same time.
