Kong Advanced Practice

The last chapter gave you the vocabulary. This one is about the three things a gateway is actually judged on once it is in production: authentication, rate limiting and transformation. Each of them has a plugin, each plugin has settings that interact with the others, and the interactions are where most production incidents come from.

It closes with the unglamorous half of the job - keeping development, staging and production consistent - because a gateway whose configuration differs between environments is a gateway that will fail in the one environment where it matters.

By the end of this chapter you will be able to configure JWT and OAuth 2.0 validation, design a rate limit that is shared across a fleet, rewrite requests before they reach an upstream, and promote the same configuration from staging to production without editing it by hand.

JWT validation

The JWT plugin verifies a signed token and, critically, ties it to a Consumer so that the rest of the gateway can make decisions about the caller.

_format_version: "3.0"
consumers:
  - username: mobile-app
    jwt_secrets:
      - algorithm: HS256
        secret: 4f8c2a1b9d6e3f7a0c5b8d2e9f1a3c6d
plugins:
  - name: jwt
    config:
      header_names:
        - authorization
      uri_param_names:
        - jwt
      claims_to_verify:
        - exp
      key_claim_name: iss
      anonymous: null

The key_claim_name field is the one that confuses people. It names the claim Kong reads to find which consumer’s secret to verify the token with. The default is iss, and Kong looks in the token payload first and then in the header. If your tokens identify the caller with a different claim - sub, or a custom cid - you must set key_claim_name to match, or every token will be rejected with Invalid JWT.

claims_to_verify takes registered claims from RFC 7519 and accepts exp and nbf. Adding exp makes Kong reject expired tokens; if you omit it, an expired token is accepted as long as the signature is valid, which is rarely what you want.

anonymous is a deliberate escape hatch. Set it to a consumer’s username and any request that fails JWT validation is treated as that anonymous consumer instead of being rejected. This is how you run a plugin set that has “public” and “authenticated” access without maintaining two routes - but it means invalid tokens are silently downgraded, so use it only when the downstream policy genuinely permits anonymous access.

A rejected request returns 401 with a message that names the failure:

curl -i http://localhost:8000/orders -H "Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJtb2JpbGUtYXBwIn0.bad"
# HTTP/1.1 401 Unauthorized
# {"message":"Invalid JWT"}

If you use asymmetric algorithms - RS256, ES256, EdDSA - the credential carries an rsa_public_key and Kong validates that the key parses before accepting the credential. The gateway then verifies signatures with the public key while clients sign with the private key, so the private key never has to reach Kong at all.

OAuth 2.0

The OAuth 2.0 plugin does something different from the JWT plugin. The JWT plugin validates tokens that someone else issued. The OAuth 2.0 plugin issues them, acting as a lightweight authorization server for your APIs.

_format_version: "3.0"
services:
  - name: orders
    url: http://orders.internal:8080
    routes:
      - name: orders-route
        paths:
          - /orders
plugins:
  - name: oauth2
    service: orders
    config:
      enable_client_credentials: true
      enable_authorization_code: true
      enable_password_grant: false
      enable_implicit_grant: false
      mandatory_scope: true
      scopes:
        - read:orders
        - write:orders
      token_expiration: 3600
      refresh_token_ttl: 1209600
      auth_header_name: authorization

The four enable_ flags select which grant types are active, and each should be switched on only if you have a reason. The client credentials grant is for machine-to-machine calls and is the one you will use most; it issues a token directly for a client id and secret, with no user involved. The authorization code grant is the three-legged browser flow, where the user’s identity provider issues the code that Kong exchanges for a token. The password grant and the implicit grant are both deprecated in OAuth 2.1 because they expose user credentials to the client - leave them off unless a legacy client forces your hand.

token_expiration is the access token lifetime in seconds and defaults to 7200. refresh_token_ttl is the refresh token lifetime and defaults to 1209600, which is fourteen days. auth_header_name is the header the access token is read from, and defaults to authorization.

With mandatory_scope: true, a request whose token lacks the required scope is rejected. This is where OAuth 2.0 earns its keep over plain API keys: two clients can both be authenticated and still have different permissions, decided by the scopes attached to their tokens.

Rate limiting that survives more than one node

The rate-limiting plugin counts requests over a set of periods and rejects the ones that exceed the limit with a 429.

plugins:
  - name: rate-limiting
    service: orders
    config:
      minute: 100
      hour: 1000
      day: 10000
      limit_by: consumer
      policy: redis
      fault_tolerant: true
      hide_client_headers: false
      error_code: 429
      error_message: "API rate limit exceeded"
      redis:
        host: redis.internal
        port: 6379
        timeout: 2000
        database: 0

The limit_by field is the most consequential setting, and the available values are consumer, credential, ip, service, header and path. consumer is the default and the right answer whenever callers are authenticated, because it gives each client its own budget rather than one shared bucket.

policy decides where the counters live, and this is where a single-node gateway and a fleet behave differently:

PolicyWhere counters liveUse when
localThe node’s own memoryOne node, or an approximate limit
clusterThe Kong databaseDatabase mode, a small number of nodes
redisAn external RedisSeveral nodes, or a limit that must be exact

With local policy and three gateway nodes, a limit of 100 per minute becomes roughly 300 per minute in practice, because each node keeps its own counter. That is a real bug, not a rounding error, and it is invisible until you scale out.

Choosing redis means every node increments the same counter. The redis block needs host, port and timeout; username, password, database, ssl and ssl_verify are optional. And policy = redis requires a sync_rate that is either -1 (synchronous, the default) or a value of at least 0.02.

fault_tolerant decides what happens when Redis is unreachable. It defaults to true, which means requests are proxied anyway and the rate limit is not applied while Redis is down. That is the availability-first choice. Set it to false and a Redis outage becomes a total outage for everything behind the gateway - a single point of failure you introduced deliberately. The field hide_client_headers controls whether the X-RateLimit-* headers are returned to the caller; leaving them visible is usually helpful for clients that can back off gracefully.

Kong also ships rate-limiting-advanced, which adds sliding windows and arbitrary window sizes. The built-in rate-limiting plugin uses fixed windows, which means a client can send 100 requests at 11:00:59 and 100 more at 11:01:00 and still be within the limit - a burst of double the intended rate across the boundary.

Transforming requests

Sometimes the gateway has to adapt a request to what the upstream expects: rename a header, add one, strip one, change the method, or rewrite the body.

plugins:
  - name: request-transformer
    service: orders
    config:
      add:
        headers:
          - "X-Gateway:kong"
        querystring:
          - "source:portal"
      remove:
        headers:
          - "X-Internal-Debug"
      rename:
        headers:
          - "X-Client-Id:X-Consumer-Id"
      replace:
        headers:
          - "X-Client-Id:anonymous"

The four operations are add, remove, rename and replace, and each applies to headers, querystring and body. The add and replace operations take name:value strings - the colon separates key from value. remove takes bare names. rename takes old:new.

The replace operation on body deserves a warning: it replaces the entire request body with a fixed string, which is almost never what you want and is easy to do by accident. If you want to merge a field into a JSON body, use add with the body option instead.

The plugin also has an append operation, and an http_method field that rewrites the method.

A response transformer is the mirror image, and it is what you use to keep internal details internal:

plugins:
  - name: response-transformer
    service: orders
    config:
      remove:
        headers:
          - "X-Powered-By"
          - "Server"
      add:
        headers:
          - "X-API-Version:v2"

Stripping Server and X-Powered-By is worth doing as a matter of course: they tell an attacker the exact stack behind the gateway, and they leak through the gateway when an upstream sets them.

One configuration, several environments

A gateway that behaves differently in production than in staging will eventually break in production. The defence is to keep the configuration in one file and change only the values that differ.

_format_version: "3.0"
services:
  - name: orders
    url: "{{ ORDERS_UPSTREAM }}"
    routes:
      - name: orders-route
        paths:
          - /orders
plugins:
  - name: rate-limiting
    service: orders
    config:
      minute: "{{ RATE_LIMIT_PER_MINUTE }}"

Then substitute per environment and validate the result before it goes anywhere:

envsubst < kong.yml.tmpl > kong.yml
kong config parse kong.yml

The kong config parse step is not optional in practice. It is a schema check, so it catches a typo in a plugin name, a missing required field, or a limit_by value that does not exist - all before the file reaches a running gateway.

For the drift check that keeps environments honest, decK compares the committed file against the live configuration:

deck gateway diff --state kong.yml
deck gateway sync --state kong.yml

Run deck gateway diff in CI against production. It exits non-zero when the live configuration has drifted from the file, which catches the most common cause of an unexplained environment difference: somebody changed something by hand through the Admin API.

If you need to know whether a particular plugin is even available on the node you are talking to, ask the node directly:

curl http://localhost:8001/plugins/enabled

Practice

A public API is served by the products service. Three kinds of caller use it:

  • the web frontend, which has a JWT and needs to be limited to 60 requests per minute;
  • the mobile app, which has a JWT and needs 600 per minute;
  • an unauthenticated partner, which sends an API key and needs 30 per minute, and whose traffic must not count against either JWT consumer’s quota.

All three share the same upstream. Give the configuration, and explain where each rate limit is counted and why the partner’s traffic does not affect the others.

KongRate LimitingJWTConsumersScope

Approach:

The scenario is designed so that a single rate-limiting plugin cannot do the job. Three different callers with three different budgets means three independent counters, and the limit_by field is what decides what a counter is keyed on.

The partner is the hard case. They are unauthenticated, so they have no consumer identity to count against - but they must not consume the JWT consumers’ quota either. If you put one rate limiter on the Service with limit_by: consumer, the partner’s requests have no consumer to key on, and depending on how Kong treats an anonymous caller the partner either shares a bucket with everybody or lands in a bucket you did not intend. The clean answer is to give the partner their own route with their own limiter, and make the JWT consumers’ limit live on a scope that the partner’s route does not inherit.

Steps:

  1. Service products pointing at the upstream. One Service, shared by all three callers.

  2. Two Routes. The JWT callers and the partner need different plugin sets, so they need different Routes on the same Service.

    • products-jwt-route for /products - the JWT callers.
    • products-partner-route for /partner/products - the partner.
  3. JWT plugin on the Service, so both Routes inherit it. But the partner is unauthenticated, so on the partner Route you need to let unauthenticated requests through. The JWT plugin’s anonymous setting is the mechanism: set it to a dedicated partner consumer. Requests with a valid token are authenticated as their real consumer; requests without one become the anonymous partner consumer instead of being rejected with 401.

  4. Consumers. web-frontend, mobile-app and partner each get a JWT credential (or a key-auth credential for the partner, if the partner sends a key rather than a token). These identities are what the rate counters are keyed on.

  5. Rate limit for the JWT callers - on the JWT Route, limit_by: consumer, with minute: 60. But the two JWT callers need different budgets, and one plugin instance has one limit. So they need two Routes, or two plugin instances at different scopes.

    The clean way: two Routes - /web/products and /mobile/products - each with its own rate-limiting plugin, limit_by: consumer, minute: 60 and minute: 600 respectively. Because limit_by is consumer and each caller has exactly one consumer identity, the limit is per consumer, not per route.

  6. Rate limit for the partner - on the partner Route, limit_by: consumer, minute: 30. The partner is authenticated as the anonymous partner consumer, so their counter is keyed on that consumer.

  7. Verify the isolation. The partner’s Route has its own rate-limiting plugin and does not inherit the JWT Routes’ limiters, because plugins do not cross Route boundaries. The JWT Routes’ limiters are keyed on web-frontend and mobile-app, which the partner is not. So the three counters are genuinely independent.

  8. Choose policy: redis if more than one gateway node serves this API, otherwise each node counts separately and the effective limit is multiplied by the node count.

Answer:

_format_version: "3.0"
services:
  - name: products
    url: http://products.internal:8080
    routes:
      - name: web-products-route
        paths:
          - /web/products
        strip_path: false
      - name: mobile-products-route
        paths:
          - /mobile/products
        strip_path: false
      - name: partner-products-route
        paths:
          - /partner/products
        strip_path: false
consumers:
  - username: web-frontend
    jwt_secrets:
      - algorithm: HS256
        secret: 1a2b3c4d5e6f708192a3b4c5d6e7f809
  - username: mobile-app
    jwt_secrets:
      - algorithm: HS256
        secret: 9f8e7d6c5b4a39281706152433425160
  - username: partner
    jwt_secrets:
      - algorithm: HS256
        secret: 00112233445566778899aabbccddeeff
plugins:
  - name: jwt
    service: products
    config:
      claims_to_verify:
        - exp
      key_claim_name: iss
      anonymous: partner
  - name: rate-limiting
    route: web-products-route
    config:
      minute: 60
      limit_by: consumer
      policy: redis
      redis:
        host: redis.internal
        port: 6379
        timeout: 2000
  - name: rate-limiting
    route: mobile-products-route
    config:
      minute: 600
      limit_by: consumer
      policy: redis
      redis:
        host: redis.internal
        port: 6379
        timeout: 2000
  - name: rate-limiting
    route: partner-products-route
    config:
      minute: 30
      limit_by: consumer
      policy: redis
      redis:
        host: redis.internal
        port: 6379
        timeout: 2000
CallerRouteCounter keyed onBudgetWhy the partner is unaffected
Web frontendweb-products-routeconsumer web-frontend60/minA separate Route with its own limiter; the partner’s Route does not inherit it
Mobile appmobile-products-routeconsumer mobile-app600/minSame - the limiter is scoped to its own Route
Partnerpartner-products-routeconsumer partner (the anonymous consumer)30/minAuthenticated as partner, so it has its own counter and never appears in either JWT consumer’s bucket

Two design decisions carry the whole solution. One Route per distinct budget, because a plugin instance has one limit and limit_by: consumer only separates callers who are already on the same Route. And anonymous: partner on the JWT plugin, which turns the partner’s missing token into a real consumer identity rather than a rejection - that identity is what makes limit_by: consumer able to count the partner at all.

Found an issue with this content?

Tell us what could be improved.