Kong in Practice

Configuration knowledge is not the hard part of running a gateway. The hard part is the shape the gateway takes in your organisation: who owns it, what it fronts, what it is allowed to do, and how anyone finds out what it did.

This chapter takes three arrangements that show up repeatedly and works through each one. The microservice gateway fronts your own services and is mostly about reliability and consistent policy. The internal API portal gives other teams a place to discover and call your capabilities. The shared gateway is where several teams publish through one gateway, which is where ownership and blast radius become the central design questions.

Alongside them sit the three operational concerns that cut across all three: how the gateway is deployed, how configuration reaches it, and how you see what it did.

By the end of this chapter you will be able to pick a deployment shape that fits your organisation, set up the observability that makes a gateway debuggable, and design a configuration layout that stops one team’s change from breaking another’s.

Arrangement one: the microservice gateway

A platform team puts Kong in front of every service in the organisation. Clients never talk to a service directly; they go through the gateway.

_format_version: "3.0"
services:
  - name: orders
    url: http://orders.svc.cluster.local:8080
    routes:
      - name: orders-route
        paths:
          - /api/v1/orders
        strip_path: true
  - name: payments
    url: http://payments.svc.cluster.local:8080
    routes:
      - name: payments-route
        paths:
          - /api/v1/payments
        strip_path: true
consumers:
  - username: web-frontend
    keyauth_credentials:
      - key: 3f9a1c7e5b2d8046af1e3c9b7d5a2f80
  - username: mobile-app
    keyauth_credentials:
      - key: 8c2e4a1b6d3f7095ce2a8b4d6f1a3c70
plugins:
  - name: key-auth
    config:
      key_names:
        - apikey
      hide_credentials: true
  - name: rate-limiting
    service: orders
    config:
      minute: 1000
      limit_by: consumer
  - name: rate-limiting
    service: payments
    config:
      minute: 200
      limit_by: consumer
  - name: prometheus
    config:
      per_consumer: true
      status_code_metrics: true

The pattern to notice is the plugin scope. key-auth is global, because every API in the organisation is authenticated the same way and nothing should be published without it. The rate limits are per Service, because a payments API and an orders API have different capacity and different risk profiles, and a global limit would be wrong for at least one of them.

In Kubernetes this normally runs as the Kong Ingress Controller, where the routes come from Ingress resources rather than from a YAML file. The plugin model is identical; only the source of the configuration changes.

The failure mode to design against is the gateway becoming a single point of failure for everything. That is a deployment problem rather than a configuration problem, and it is covered below.

Arrangement two: the internal API portal

A team publishes a capability and other teams consume it through a portal. The gateway’s job is to make each team’s use safe and attributable.

_format_version: "3.0"
services:
  - name: catalog
    url: http://catalog.svc.cluster.local:8080
    routes:
      - name: catalog-internal-route
        paths:
          - /internal/catalog
        strip_path: true
consumers:
  - username: team-checkout
    keyauth_credentials:
      - key: a1b2c3d4e5f60718293a4b5c6d7e8f90
  - username: team-reporting
    keyauth_credentials:
      - key: 0f1e2d3c4b5a69788796a5b4c3d2e1f0
plugins:
  - name: key-auth
    service: catalog
    config:
      hide_credentials: true
  - name: acl
    service: catalog
    config:
      allow:
        - checkout
        - reporting
      deny:
        - experimental
  - name: http-log
    service: catalog
    config:
      http_endpoint: http://audit-logger.internal:9000/events
      method: POST
      content_type: application/json
      timeout: 5000
      queue:
        max_entries: 10000
        max_batch_size: 100

Two things make this an API portal rather than a plain gateway.

The ACL plugin turns consumers into groups. allow and deny are mutually exclusive - you configure one or the other - and the groups are attached to consumers through their credentials. With allow: [checkout, reporting], only consumers in those groups reach the service, and adding a new team means adding a consumer to a group, not changing the gateway’s policy.

The http-log plugin sends every request through the route to an audit endpoint. http_endpoint is required; method accepts POST, PUT and PATCH; content_type is application/json; and the queue block controls buffering, with max_entries for the queue depth and max_batch_size for how many records are sent at once. A queue matters because an audit endpoint that is briefly slow must not slow down the API.

Because a consumer identity exists for every caller, the gateway can answer “who called this API, how often, and what did they ask for” without the upstream services knowing anything about it.

Arrangement three: the gateway shared by many teams

Several teams publish through one gateway. This is where configuration layout matters, because with a single flat file one team’s change rewrites everyone else’s.

The first rule is one file per team, merged at deploy time:

kong/
  base.yaml            # global plugins: key-auth, prometheus
  team-checkout.yaml
  team-reporting.yaml
  team-payments.yaml

The second rule is tags, which give every object an owner. Tags are a first-class field on services, routes, plugins and consumers, and they let both humans and tooling filter by team:

curl "http://localhost:8001/services?tags=team-checkout"

The third rule is namespacing by path. Giving each team its own path prefix means two teams cannot accidentally create conflicting routes, and the conflict is visible in review:

_format_version: "3.0"
services:
  - name: checkout-api
    url: http://checkout.svc.cluster.local:8080
    tags:
      - team-checkout
    routes:
      - name: checkout-route
        paths:
          - /api/checkout
        strip_path: true

The fourth rule is the blast radius of a change. If team A’s rate limit is attached to a Service and team B’s routes share that Service, team A can break team B. Attaching team-specific policy to team-specific objects - which the tags make obvious - keeps the damage contained.

Merge the files and validate before deploying:

cat kong/base.yaml kong/team-*.yaml > kong/merged.yaml
kong config parse kong/merged.yaml
deck gateway diff --state kong/merged.yaml

The diff step against production is the one that catches a team’s config drifting from what was agreed.

Deployment

For a shared gateway, hybrid mode is the usual answer, and the reason is the same as in chapter one: policy needs one home, traffic needs many servers.

              load balancer
             /      |       \
      data-plane  data-plane  data-plane      <- stateless, interchangeable
             \      |       /
              control plane (PostgreSQL)      <- Admin API, the source of truth

Each data plane holds no configuration of its own beyond a cache, so a data plane can be replaced, rescheduled or scaled at any time. Run at least three of them, spread across failure domains, and a node loss is an event the load balancer absorbs.

In Kubernetes the same shape appears as a Deployment of data planes with a StatefulSet or a separate Deployment for the control plane, backed by a managed PostgreSQL instance.

Do not run the Admin API on a public address. Port 8001 is administrative and, in the default configuration, unauthenticated - it belongs on an internal network or behind your own authentication layer.

Observability

A gateway that handles every request is the best place to measure your whole system, and the worst place to leave it unmeasured.

Metrics. The Prometheus plugin exposes a /metrics endpoint. The per_consumer setting adds the consumer to the labels of the request and bandwidth counters, which is what lets you build a per-team usage dashboard; status_code_metrics adds the counters broken down by response code.

curl http://localhost:8001/metrics

Access logs. Kong writes an access log per request, and the format is configurable through the nginx access log directives. The fields worth having are the upstream service, the consumer, the status, the upstream latency and the request length.

Request tracing. The correlation-id plugin stamps every request with an identifier and propagates it to the upstream, so a log line in the gateway can be matched to a log line in the service. This is worth enabling before you need it - correlating logs after an incident, without a shared identifier, is painful.

Distributed tracing. The OpenTelemetry plugin exports spans to a collector, which is how you see the gateway’s own latency as part of a trace rather than as a separate number.

Health. GET /status on the Admin API is the readiness endpoint. It reports whether the node has finished loading its configuration and its plugins, which is what you point a Kubernetes readiness probe at.

Rolling out a change without an outage

Two mechanisms cover most cases.

Canary by route. Create a second route to the new upstream and send a small slice of traffic to it, selected by a header that only your testers send:

_format_version: "3.0"
services:
  - name: orders-v1
    url: http://orders-v1.svc.cluster.local:8080
    routes:
      - name: orders-v1-route
        paths:
          - /api/v1/orders
        strip_path: true
  - name: orders-v2
    url: http://orders-v2.svc.cluster.local:8080
    routes:
      - name: orders-v2-canary-route
        paths:
          - /api/v1/orders
        headers:
          x-canary:
            - "true"
        strip_path: true

Kong also ships a canary plugin, which is a more direct way to express “a percentage of traffic goes to the new upstream”.

Weighted targets. When the change is in the service rather than the gateway, split traffic between targets of the same upstream with different weights.

Practice

An organisation has four teams, and one of them - the payments team - wants to raise its rate limit from 50 to 200 requests per minute. The gateway is shared, the configuration is currently a single kong.yaml, and the last time somebody edited it, three unrelated routes stopped working.

Design the configuration layout that makes this change safe. Say where each team’s objects live, how the change reaches production, and what stops it from breaking the other three teams.

KongGovernanceConfiguration LayoutDeclarative ConfigTeams

Approach:

The question is really about blast radius, and it gives you the symptom: a single shared file means every change is a whole-file change, so a small edit by one team carries the risk of breaking three others. The fix is not process - it is structure. If each team’s objects live in files that are only touched by that team, and the merge is mechanical and validated, then the payments team’s change cannot structurally affect the others.

There is a second half to the answer that people forget. Splitting files is necessary but not sufficient: the merged file is still one file, and one team’s merge can still conflict with another’s at the text level. So the layout needs a stable ordering and a merge strategy that produces a deterministic result - which means concatenating in a fixed order, not globbing in filesystem order, and not merging YAML structurally.

Steps:

  1. One file per team, plus a base file. The base file holds only the configuration that is genuinely global - the shared authentication plugin, the Prometheus plugin, and anything else every team inherits. Team files hold everything team-specific.

  2. Name the files so the merge order is obvious. A prefix or a suffix that sorts teams into a stable order means cat in sorted order always produces the same result. This matters more than it sounds: if the merge order changes between deployments, the same inputs produce a different output, and you have lost the ability to reason about what is deployed.

  3. Give every team its own path prefix. /api/checkout, /api/payments, /api/reporting, /api/identity. Two teams can then never create conflicting routes, and a route collision becomes visible in review rather than at runtime.

  4. Tag every object with its owning team. Tags are a first-class field on services, routes, plugins and consumers. They make ownership machine-readable - GET /services?tags=team-payments returns exactly that team’s surface - and they let you write a check that fails when a team changes something outside their own tag.

  5. Keep team-specific policy attached to team-specific objects. The payments rate limit belongs on the payments route or service, not on a shared object. If it lived on a shared service, changing it would change it for every team using that service.

  6. Make the merge and validation a single command that anyone can run, so the pipeline and a developer are doing the same thing:

    cat kong/base.yaml kong/teams/*.yaml > kong/merged.yaml
    kong config parse kong/merged.yaml

    kong config parse is the schema check. It catches an invalid plugin name, a missing required field, a bad limit_by value - before anything reaches a running gateway.

  7. Diff against production before applying. deck gateway diff --state kong/merged.yaml exits non-zero when the live configuration differs from the file. Run it in CI against production and it becomes a drift alarm: the moment somebody changes something by hand through the Admin API, the pipeline fails and says so.

  8. Then sync. deck gateway sync --state kong/merged.yaml applies the merged file to the gateway.

  9. Do the payments change. Edit only kong/teams/team-payments.yaml, changing minute: 50 to minute: 200. Re-run the merge and validation. The other three files are byte-identical, so the merged output differs from the previous merged output in exactly one place.

Answer:

kong/
  base.yaml              # global: key-auth, prometheus, global CORS
  teams/
    10-identity.yaml
    20-checkout.yaml
    30-reporting.yaml
    40-payments.yaml
ConcernMechanismEffect on the payments change
OwnershipOne file per team under teams/Only 40-payments.yaml is edited
Merge orderNumeric filename prefixes, merged with a glob in sorted orderThe merged output is deterministic and the same on every machine
Route conflictsPer-team path prefixesThe new route cannot collide with another team’s
Machine-readable ownershiptags: [team-payments] on every objectGET /services?tags=team-payments returns that team’s surface
Blast radiusTeam policy attached to team objectsRaising the limit touches one route’s plugin instance
Syntax and schema errorskong config parse kong/merged.yaml in CIA malformed edit fails the pipeline, not production
Unintended changesdeck gateway diff --state kong/merged.yaml in CIDrift - a hand edit made outside the pipeline - is caught

The pipeline for the change becomes:

cat kong/base.yaml kong/teams/*.yaml > kong/merged.yaml
kong config parse kong/merged.yaml
deck gateway diff --state kong/merged.yaml
deck gateway sync --state kong/merged.yaml

Why the other three teams are safe: their files are not edited, so their objects are byte-identical in the merged output; each team’s policy is attached to their own routes rather than to a shared object; and the parse and diff steps catch both a malformed edit and any drift that has accumulated outside the pipeline. The failure that broke three routes last time - a whole-file edit with an unrelated mistake in it - is now structurally impossible, because no single edit touches the other teams’ files.

Found an issue with this content?

Tell us what could be improved.