Kubernetes ingress-nginx

ingress-nginx terminates mTLS with its auth-tls annotations and calls mt-enforcer through auth-url for every request. The Helm chart deploys the enforcer with probes, a PodDisruptionBudget, a NetworkPolicy and an optional ServiceMonitor.

Install the chart

Pairing Secret and Helm installmt-enforcer pair --api https://api.mutualtrust.io --state ./mt-state
kubectl create namespace mt-system
kubectl -n mt-system create secret generic mt-enforcer-pairing --from-file=gateway-state.json=./mt-state/gateway-state.json
helm install mt-enforcer services/enforcer/integrations/kubernetes/charts/mt-enforcer -n mt-system \
  --set mode=forward-auth --set forwardAuth.format=ingress-nginx \
  --set forwardAuth.trustedProxies='{10.0.0.0/8}'

charts/mt-enforcer/values.yaml# Values for the mt-enforcer chart. Every key is documented in
# https://mutualtrust.io/docs/gateways/kubernetes-ingress-nginx and
# services/enforcer/README.md.

image:
  repository: mt-enforcer            # push the image built from services/enforcer/Dockerfile to your registry
  tag: "1.0.0"
  digest: ""                          # sha256:..., takes precedence over tag
  pullPolicy: IfNotPresent

# proxy | envoy-grpc | forward-auth
mode: envoy-grpc

# Control plane URL.
api: https://api.mutualtrust.io

replicaCount: 2

pairing:
  # Secret holding gateway-state.json, created from `mt-enforcer pair --state ./mt-state`:
  #   kubectl -n mt-system create secret generic mt-enforcer-pairing --from-file=gateway-state.json=./mt-state/gateway-state.json
  existingSecret: mt-enforcer-pairing
  # Alternatively the chart creates the Secret from this value (keep it out of version control).
  stateJson: ""

envoyGrpc:
  port: 9191

forwardAuth:
  port: 9191
  # Header format of the calling gateway: ingress-nginx, nginx, traefik, haproxy, kong, caddy, xfcc.
  format: ingress-nginx
  # Source CIDRs allowed to call the endpoint (the gateway pods). Combine with the NetworkPolicy below.
  trustedProxies: ["10.0.0.0/8"]

proxy:
  port: 8443
  # Origin the proxy forwards allowed requests to.
  origin: http://127.0.0.1:9001
  # Secret of type kubernetes.io/tls with the server certificate (tls.crt, tls.key).
  tlsSecret: ""
  clientAuth: request                 # request | require
  maxStreamAge: 5m
  # Behind a load balancer that terminates mTLS (aws-alb, cloudflare): set peerSource to forwarded.
  peerSource: tls
  peerFormat: ""
  trustedProxies: []

admin:
  port: 9464

logLevel: info

# Extra command-line flags, for example ["--bundle-out", "/var/run/mt/bundle/client-ca.pem"].
extraArgs: []

resources:
  requests: { cpu: 50m, memory: 64Mi }
  limits: { memory: 256Mi }

podDisruptionBudget:
  enabled: true
  minAvailable: 1

serviceMonitor:
  enabled: false                      # requires the Prometheus Operator CRDs
  interval: 15s
  labels: {}

networkPolicy:
  enabled: true
  # Namespaces whose pods may call the authorization or proxy port (the gateway's namespace).
  allowedNamespaces: ["ingress-nginx"]

podAnnotations: {}
nodeSelector: {}
tolerations: []
affinity: {}

Ingress

services/enforcer/integrations/kubernetes/ingress-nginx/ingress.yaml# Mutual Trust with the Kubernetes ingress-nginx controller.
#
# ingress-nginx terminates mTLS against the client CA bundle (Secret mt-client-ca,
# kept current by bundle-sync.yaml) and calls mt-enforcer (chart in
# ../charts/mt-enforcer, mode forward-auth, format ingress-nginx) for every
# request through auth-url. auth-tls-pass-certificate-to-upstream makes the
# controller send the escaped client certificate (ssl-client-cert) with the
# sub-request; X-Original-URL and X-Original-Method are set by the controller.
# auth-response-headers copies the verified identity onto the proxied request,
# replacing any client-supplied value.
#
#   helm install mt-enforcer ../charts/mt-enforcer -n mt-system \
#     --set mode=forward-auth --set forwardAuth.format=ingress-nginx
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: orders-api
  namespace: orders
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
    nginx.ingress.kubernetes.io/auth-tls-secret: "orders/mt-client-ca"
    nginx.ingress.kubernetes.io/auth-tls-verify-client: "on"
    nginx.ingress.kubernetes.io/auth-tls-verify-depth: "2"
    nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "true"
    nginx.ingress.kubernetes.io/auth-url: "http://mt-enforcer.mt-system.svc.cluster.local:9191/authz"
    nginx.ingress.kubernetes.io/auth-response-headers: "X-Mt-Principal, X-Mt-Grant, X-Mt-Request-Id"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      more_clear_input_headers "X-Mt-*";
spec:
  ingressClassName: nginx
  tls:
    - hosts: [orders-api.example.com]
      secretName: orders-api-tls           # your server certificate
  rules:
    - host: orders-api.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: orders-api
                port: { number: 9001 }
---
# The backend accepts traffic only from the ingress controller.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: orders-api-from-ingress-only
  namespace: orders
spec:
  podSelector:
    matchLabels: { app: orders-api }
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels: { kubernetes.io/metadata.name: ingress-nginx }

With auth-tls-pass-certificate-to-upstream the controller sends the escaped client certificate as ssl-client-cert on the authorization sub-request, together with X-Original-URL and X-Original-Method; all three are set by the controller itself. The enforcer re-validates the certificate and answers 200 with the identity headers, which auth-response-headers copies onto the proxied request, or 401/403.

Keep the CA Secret current

services/enforcer/integrations/kubernetes/bundle-sync.yaml# Keeps Kubernetes Secrets holding the client CA bundle in step with the signed
# policy manifest. Gateways that read the bundle from a Secret (ingress-nginx
# auth-tls-secret, Istio credentialName) pick up issuer additions and rotations
# from here; mt-enforcer re-validates every client certificate against the
# signed bundle either way.
#
# Edit TARGET_NAMESPACE / TARGET_SECRET / TARGET_KEY:
#   ingress-nginx: orders / mt-client-ca / ca.crt
#   Istio:         istio-system / orders-api-credential-cacert / ca.crt
apiVersion: v1
kind: ServiceAccount
metadata: { name: mt-bundle-sync, namespace: mt-system }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: mt-bundle-sync, namespace: orders }
rules:
  - apiGroups: [""]
    resources: [secrets]
    resourceNames: [mt-client-ca]
    verbs: [get, patch, update]
  - apiGroups: [""]
    resources: [secrets]
    verbs: [create]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: mt-bundle-sync, namespace: orders }
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: Role, name: mt-bundle-sync }
subjects: [{ kind: ServiceAccount, name: mt-bundle-sync, namespace: mt-system }]
---
apiVersion: batch/v1
kind: CronJob
metadata: { name: mt-bundle-sync, namespace: mt-system }
spec:
  schedule: "*/5 * * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          serviceAccountName: mt-bundle-sync
          restartPolicy: OnFailure
          securityContext: { runAsNonRoot: true, runAsUser: 65532, seccompProfile: { type: RuntimeDefault } }
          initContainers:
            - name: bundle
              image: mt-enforcer:1.0.0
              args: ["bundle", "--state", "/pairing", "--out", "/work/ca.crt"]
              volumeMounts:
                - { name: pairing, mountPath: /pairing, readOnly: true }
                - { name: work, mountPath: /work }
          containers:
            - name: apply
              image: registry.k8s.io/kubectl:v1.34.2
              env:
                - { name: TARGET_NAMESPACE, value: orders }
                - { name: TARGET_SECRET, value: mt-client-ca }
                - { name: TARGET_KEY, value: ca.crt }
              command: ["/bin/sh", "-c"]
              args:
                - >-
                  kubectl create secret generic "$TARGET_SECRET" -n "$TARGET_NAMESPACE"
                  --from-file="$TARGET_KEY"=/work/ca.crt --dry-run=client -o yaml | kubectl apply -f -
              volumeMounts:
                - { name: work, mountPath: /work, readOnly: true }
          volumes:
            - name: pairing
              secret: { secretName: mt-enforcer-pairing }
            - name: work
              emptyDir: {}

mt-enforcer bundle fetches and verifies the current manifest and writes its client CA bundle; the job applies it to the Secret the Ingress references.

Verify

Verify with the enrolled workload identitymt enroll --token <token from portal> --api https://api.mutualtrust.io
mt call https://orders-api.example.com:443/orders/1
# 200, and your origin receives x-mt-principal: urn:mutualtrust:workload:...
mt call https://orders-api.example.com:443/admin/export
# 403 {"code":"route_not_permitted", ...}
# any other client: the files are in the workload directory that mt status prints
W="$(dirname "$(mt status --json | jq -r .files.chain)")"
curl --cert "$W/chain.pem" --key "$W/key.pem" https://orders-api.example.com:443/orders/1