Kong proxy-cache-memcached plugin

Built by Mamluk for Bahriya.

HTTP proxy caching for Kong, backed by memcached with client-side ketama sharding so the cache can be spread across a cluster of memcached nodes.

This plugin is derived from the original proxy-cache plugin shipped with Kong OSS (which stores entries in an Nginx shared dict). It keeps the same cache-key generation, cache-control semantics and admin endpoints, but the backing store has been replaced with memcached.

Compatible with Kong 3.9+.

Prerequisites: lua-resty-balancer

This plugin requires the resty.chash module from lua-resty-balancer (formerly known as lua-resty-chash). This library is not bundled with the kong/kong:3.9 Docker image and is not available on luarocks, so it must be built from source and installed into OpenResty's lualib directory.

The build produces:

• lib/resty/chash.lua + lib/resty/balancer/*.lua — pure Lua modules
• librestychash.so — a small C shared library

Installing in a custom Kong Docker image

Add this block before any COPY instructions (as root):

ARG LUA_RESTY_BALANCER_VERSION=0.05
RUN apt-get update \
 && apt-get install -y --no-install-recommends curl ca-certificates build-essential \
 && curl -fsSL "https://github.com/openresty/lua-resty-balancer/archive/refs/tags/v${LUA_RESTY_BALANCER_VERSION}.tar.gz" \
      -o /tmp/lua-resty-balancer.tar.gz \
 && tar -xzf /tmp/lua-resty-balancer.tar.gz -C /tmp \
 && make -C "/tmp/lua-resty-balancer-${LUA_RESTY_BALANCER_VERSION}" \
 && make -C "/tmp/lua-resty-balancer-${LUA_RESTY_BALANCER_VERSION}" install \
      LUA_LIB_DIR=/usr/local/openresty/lualib LUA_VERSION=5.1 \
 && rm -rf "/tmp/lua-resty-balancer-${LUA_RESTY_BALANCER_VERSION}" /tmp/lua-resty-balancer.tar.gz \
 && apt-get purge -y curl ca-certificates build-essential \
 && apt-get autoremove -y \
 && rm -rf /var/lib/apt/lists/*

This installs the modules into /usr/local/openresty/lualib/resty/, which is already on Kong's package.path and package.cpath. The build toolchain is purged afterwards so it doesn't add to the final image surface.

Installing on bare metal / VM

curl -fsSL https://github.com/openresty/lua-resty-balancer/archive/refs/tags/v0.05.tar.gz \
  | tar -xz
cd lua-resty-balancer-0.05
make
sudo make install LUA_LIB_DIR=/usr/local/openresty/lualib LUA_VERSION=5.1

Adjust LUA_LIB_DIR to match your OpenResty / Kong installation's lualib path if it differs from the default.

Why memcached + sharding

A single memcached instance is cheap to run but caps both throughput and total cache footprint at one node's resources. This plugin distributes keys across an arbitrary number of memcached nodes using ketama consistent hashing (via resty.chash from the lua-resty-balancer library). Adding or removing a node only reshuffles 1/N of the keys instead of invalidating the entire cache.

Two plugin variants: pick where it sits relative to rate-limiting

The rock installs two plugin names that share one codebase. They differ only in Kong execution priority — i.e. whether the cache lookup runs before or after the rate-limit plugin in Kong's access phase.

Both variants accept the identical configuration schema, derive the same cache keys, and target the same memcached cluster. The only difference is execution order. You enable whichever ordering you want per route or service, the same way you would attach any other Kong plugin.

When to use proxy-cache-memcached (before rate-limit, priority 902)

Use this when rate-limit is upstream-protection only. Cache hits never reach upstream, so it would be wasteful to spend a rate-limit token on them. Typical fit:

• Public read-heavy APIs where the limit exists to shield a slow backend.
• Heavy fan-out from a small number of clients hitting the same hot resource.
• You want the limit to track origin traffic, not edge traffic.

When to use proxy-cache-memcached-post-rl (after rate-limit, priority 850)

Use this when rate-limit is a client-fairness or abuse-control tool. A client that hammers a single cached endpoint should still be throttled even though no upstream load is generated. Typical fit:

• Per-consumer quotas (e.g. "1000 requests per hour, period").
• Anti-scrape / anti-abuse limits.
• Billing-driven plans where the request is the billable unit, not the upstream call.

Can I enable both on the same route?

Technically yes, but you almost certainly don't want to — they'd both attempt to serve from / write to memcached on the same key, and the higher-priority one would always win for hits. Pick one variant per route.

Note on shared cache storage

Because the cache key derivation is identical (and does not include the plugin name), an entry written by one variant is readable by the other if both point at the same memcached cluster. Practically: the DELETE /plugins/:plugin_id/<variant-name> admin endpoints operate against the same cluster, so a flush_all issued via either variant wipes everything for both. This is intentional — only one variant should be active per route.

Enabling the plugins in Kong

Both plugin names must be listed in Kong's plugins (or legacy custom_plugins) configuration. For example:

KONG_PLUGINS=bundled,proxy-cache-memcached,proxy-cache-memcached-post-rl

You only need to list the variant(s) you actually plan to use.

Cache status header

Every response carries X-Cache-Status:

• Hit — the request was cacheable and served from memcached
• Miss — the request was cacheable but not found in memcached
• Refresh — found in memcached but stale per cache_ttl or Cache-Control
• Bypass — the request or response was not cacheable

Storage TTL vs cache TTL

cache_ttl is how long the plugin will serve the cached entry. storage_ttl (optional) is how long memcached is asked to keep the entry. Setting storage_ttl > cache_ttl lets max-stale clients receive a stale copy after the cache TTL has elapsed.

vary_body_json_fields

You can include named fields from a JSON request body in the cache key. Only scalar values (string / number) are honoured — arrays and objects are collapsed.

allow_force_cache_header

Set to true and clients can pass X-Proxy-Cache-Memcached-Force: true to bypass the request-method check (and no-cache/no-store directives). The response-side checks (status code, content type) still apply.

Configuration

The same configuration schema applies to both proxy-cache-memcached and proxy-cache-memcached-post-rl. Just swap the value of name for the variant you want.

Sharding example

plugins:
- name: proxy-cache-memcached
  config:
    cache_ttl: 60
    memcached:
      hosts:
        - { host: memcached-a.svc, port: 11211, weight: 100 }
        - { host: memcached-b.svc, port: 11211, weight: 100 }
        - { host: memcached-c.svc, port: 11211, weight: 50  }

The plugin builds a ketama ring once per worker per config signature and picks the target node by hashing the cache key. Per-node weight shifts more of the key space onto bigger boxes. Adding or removing a node only shifts the slice the new/old node owned.

TLS example

config:
  memcached:
    hosts:
      - { host: memcached.internal, port: 11212 }
    tls: true
    tls_verify: true
    tls_server_name: memcached.internal

Connections are pooled separately for plain-text and TLS targets, so the same host can safely appear with both modes.

Admin API

Endpoints are mounted per plugin instance under the variant's name — i.e. proxy-cache-memcached instances expose paths under /plugins/:plugin_id/proxy-cache-memcached, and proxy-cache-memcached-post-rl instances expose paths under /plugins/:plugin_id/proxy-cache-memcached-post-rl.

For proxy-cache-memcached (before rate-limit):

For proxy-cache-memcached-post-rl (after rate-limit):

Both sets of endpoints talk to the same memcached cluster, so a flush_all issued against either variant clears the whole cluster.

Using with Kong Ingress Controller (KIC)

KIC drives Kong from Kubernetes resources. Three things have to be in place:

1. The plugin code has to be loadable inside the Kong gateway pods.
2. The gateway has to be told to load it via the KONG_PLUGINS env var.
3. Plugin configuration is delivered through a KongPlugin / KongClusterPlugin custom resource, then attached to an Ingress, Service, Route, or Consumer via annotation.

1. Make the plugin available to the gateway pods

KIC does not pull custom plugins at runtime. Three patterns, in rough order of how often KIC users reach for them:

a) ConfigMap (most common for KIC)

Each plugin directory is shipped as its own ConfigMap, and Kong is told to mount it on the Lua package path. Because this rock registers two separate Kong plugin names (proxy-cache-memcached and proxy-cache-memcached-post-rl), each gets its own ConfigMap and its own top-level entry under plugins.configMaps — they are not subdirectories of a single plugin, so the subdirectories: field on a configMaps entry does not apply here. (Use plugins.secrets: with the same shape instead of plugins.configMaps: if you'd rather ship the Lua source via a Secret.)

Create one ConfigMap per plugin directory:

kubectl create configmap -n kong kong-plugin-proxy-cache-memcached \
  --from-file=kong/plugins/proxy-cache-memcached/

kubectl create configmap -n kong kong-plugin-proxy-cache-memcached-post-rl \
  --from-file=kong/plugins/proxy-cache-memcached-post-rl/

Using the official Kong Helm chart, wire them up via plugins.configMaps — the chart handles the mount path and KONG_LUA_PACKAGE_PATH automatically, and it derives KONG_PLUGINS from the pluginName values you list here (prepending bundled). You do not need to set env.plugins yourself when using this form — the chart does it for you. Skip step 2 below.

# values.yaml
plugins:
  configMaps:
    - name: kong-plugin-proxy-cache-memcached
      pluginName: proxy-cache-memcached
    - name: kong-plugin-proxy-cache-memcached-post-rl
      pluginName: proxy-cache-memcached-post-rl

Using raw manifests / Kong Gateway Operator, mount the ConfigMaps yourself and extend KONG_LUA_PACKAGE_PATH:

# excerpt from the kong-gateway Deployment / Pod spec
containers:
  - name: proxy
    image: kong:3.9
    env:
      - name: KONG_LUA_PACKAGE_PATH
        value: "/opt/?.lua;/opt/?/init.lua;;"
      - name: KONG_PLUGINS
        value: "bundled,proxy-cache-memcached,proxy-cache-memcached-post-rl"
    volumeMounts:
      - name: plugin-base
        mountPath: /opt/kong/plugins/proxy-cache-memcached
      - name: plugin-post-rl
        mountPath: /opt/kong/plugins/proxy-cache-memcached-post-rl
volumes:
  - name: plugin-base
    configMap:
      name: kong-plugin-proxy-cache-memcached
  - name: plugin-post-rl
    configMap:
      name: kong-plugin-proxy-cache-memcached-post-rl

ConfigMap values are limited to ~1 MiB total per ConfigMap — well above the size of these plugin files. If you ever do hit that ceiling, fall back to option (b) or (c).

b) Mount via an init container

Useful when you can't rebuild the Kong image but the plugin won't fit in a ConfigMap (e.g. very large plugin bundles, or you want the plugin shipped as its own OCI image):

initContainers:
  - name: plugin-loader
    image: your-registry/proxy-cache-memcached-plugin:1.0.0-0
    command: ["sh", "-c", "cp -r /plugin/* /opt/kong-plugins/"]
    volumeMounts:
      - name: kong-plugins
        mountPath: /opt/kong-plugins
containers:
  - name: proxy
    image: kong:3.9
    env:
      - name: KONG_LUA_PACKAGE_PATH
        value: "/opt/kong-plugins/?.lua;/opt/kong-plugins/?/init.lua;;"
    volumeMounts:
      - name: kong-plugins
        mountPath: /opt/kong-plugins
volumes:
  - name: kong-plugins
    emptyDir: {}

c) Bake into a custom Kong image

Cleanest for orgs that already maintain a custom Kong image and have a container build pipeline:

FROM kong:3.9

USER root

# 1. Install lua-resty-balancer (provides resty.chash) — see "Prerequisites" above.
ARG LUA_RESTY_BALANCER_VERSION=0.05
RUN apt-get update \
 && apt-get install -y --no-install-recommends curl ca-certificates build-essential \
 && curl -fsSL "https://github.com/openresty/lua-resty-balancer/archive/refs/tags/v${LUA_RESTY_BALANCER_VERSION}.tar.gz" \
      -o /tmp/lua-resty-balancer.tar.gz \
 && tar -xzf /tmp/lua-resty-balancer.tar.gz -C /tmp \
 && make -C "/tmp/lua-resty-balancer-${LUA_RESTY_BALANCER_VERSION}" \
 && make -C "/tmp/lua-resty-balancer-${LUA_RESTY_BALANCER_VERSION}" install \
      LUA_LIB_DIR=/usr/local/openresty/lualib LUA_VERSION=5.1 \
 && rm -rf "/tmp/lua-resty-balancer-${LUA_RESTY_BALANCER_VERSION}" /tmp/lua-resty-balancer.tar.gz \
 && apt-get purge -y curl ca-certificates build-essential \
 && apt-get autoremove -y \
 && rm -rf /var/lib/apt/lists/*

# 2. Install the plugin itself.
COPY kong/plugins/proxy-cache-memcached/ /usr/local/share/lua/5.1/kong/plugins/proxy-cache-memcached/
COPY kong/plugins/proxy-cache-memcached-post-rl/ /usr/local/share/lua/5.1/kong/plugins/proxy-cache-memcached-post-rl/

USER kong

Then set KONG_PLUGINS as in step 2 below. No volume mounts required — the files are placed where Kong's package path already looks.

2. Enable the variant(s) you intend to use

Skip this step if you used the Helm chart's plugins.configMaps (option 1a, first sub-form). The chart's kong.plugins template auto-derives KONG_PLUGINS from the pluginName values you listed there.

Otherwise — i.e. for the raw-manifest form of (1a), or for options (1b) and (1c) — set KONG_PLUGINS on the Kong gateway container yourself:

env:
  - name: KONG_PLUGINS
    value: "bundled,proxy-cache-memcached,proxy-cache-memcached-post-rl"

List only the variants you plan to use.

3. Deploy memcached as a sharded cluster (StatefulSet + headless Service)

Important: if you deploy memcached as a Deployment behind a regular ClusterIP service, every Kong worker connects to the same DNS name and the service round-robins. Client-side ketama sharding then breaks — keys land on whichever pod kube-proxy picks, defeating the consistent hash. Use a StatefulSet plus a headless service so each pod gets a stable DNS name, and list those pod DNS names individually in the plugin config.

apiVersion: v1
kind: Service
metadata:
  name: memcached
spec:
  clusterIP: None              # headless — gives stable per-pod DNS
  selector:
    app: memcached
  ports:
    - port: 11211
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: memcached
spec:
  serviceName: memcached
  replicas: 3
  selector:
    matchLabels: { app: memcached }
  template:
    metadata:
      labels: { app: memcached }
    spec:
      containers:
        - name: memcached
          image: memcached:1.6-alpine
          args: ["-m", "256"]
          ports:
            - containerPort: 11211

Each pod is then addressable as memcached-0.memcached.<namespace>.svc.cluster.local, memcached-1.memcached.<namespace>.svc.cluster.local, etc. Those are the names you put into the plugin's memcached.hosts array.

4. Configure the plugin with a KongClusterPlugin (or KongPlugin)

Cluster-scoped (recommended when many namespaces need the same config):

apiVersion: configuration.konghq.com/v1
kind: KongClusterPlugin
metadata:
  name: cache-before-ratelimit
  annotations:
    kubernetes.io/ingress.class: kong
plugin: proxy-cache-memcached      # <-- variant name
config:
  cache_ttl: 60
  memcached:
    hosts:
      - { host: memcached-0.memcached.default.svc.cluster.local, port: 11211, weight: 100 }
      - { host: memcached-1.memcached.default.svc.cluster.local, port: 11211, weight: 100 }
      - { host: memcached-2.memcached.default.svc.cluster.local, port: 11211, weight: 100 }

For the post-rate-limit variant, change only the plugin: field:

apiVersion: configuration.konghq.com/v1
kind: KongClusterPlugin
metadata:
  name: cache-after-ratelimit
  annotations:
    kubernetes.io/ingress.class: kong
plugin: proxy-cache-memcached-post-rl   # <-- variant name
config:
  cache_ttl: 60
  memcached:
    hosts:
      - { host: memcached-0.memcached.default.svc.cluster.local, port: 11211 }
      - { host: memcached-1.memcached.default.svc.cluster.local, port: 11211 }
      - { host: memcached-2.memcached.default.svc.cluster.local, port: 11211 }

Namespaced variant uses kind: KongPlugin with the same body and is referenced from resources in that namespace only.

5. Attach the plugin to an Ingress, Service, or other resource

Apply the plugin to an Ingress via the konghq.com/plugins annotation. Note that on standard Ingress resources you bind to Kong via spec.ingressClassName: kong (the kubernetes.io/ingress.class annotation is deprecated as of Kubernetes 1.18):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-api
  annotations:
    konghq.com/plugins: cache-before-ratelimit       # name of the KongClusterPlugin / KongPlugin
spec:
  ingressClassName: kong
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: my-api
                port:
                  number: 80

The same konghq.com/plugins annotation can be applied to a Service, an HTTPRoute (Gateway API), a KongConsumer, or a KongConsumerGroup. The annotation value is a comma-separated list of KongPlugin / KongClusterPlugin names, so multiple plugins can stack on one resource. To run both cache variants on different routes, reference different KongPlugin resources from the relevant Ingress objects.

(Kong's own custom resources — including KongClusterPlugin and KongPlugin themselves — still use the kubernetes.io/ingress.class: kong annotation to bind to a controller instance, as shown in step 4. Only standard Ingress resources have migrated to spec.ingressClassName.)

6. Admin API from inside the cluster

The admin endpoints listed above (/plugins/:plugin_id/<variant>/...) live on the Kong admin listener, which KIC normally keeps cluster-internal. To flush or purge from a kubectl shell:

kubectl port-forward -n kong svc/kong-admin 8001:8001
curl -X DELETE http://127.0.0.1:8001/plugins/<plugin_id>/proxy-cache-memcached

The <plugin_id> is the UUID Kong assigns to the KongPlugin / KongClusterPlugin instance — fetch it from GET /plugins.

Cache-Control behaviours (when cache_control = true)

Standard RFC 7234 semantics:

• no-cache / no-store — request / response is not cached
• private — response not cached, prior cached value may still be served
• max-age=<s> — fresh window relative to request time
• max-stale[=<s>] — client accepts stale within bounds
• min-fresh=<s> — client requires the response stay fresh for <s> more seconds
• only-if-cached — client refuses to hit upstream; 504 if not cached

More: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control

About

This plugin is built by Mamluk — engineering the infrastructure that the region cannot afford to get wrong — for Bahriya, which lets you deploy globally distributed container infrastructure without the AWS degree.