Enroll a workload
Enrollment turns an attested workload identity plus a locally generated key into a 24-hour client certificate. The private key never leaves the workload; Mutual Trust only ever sees the CSR and a signature.
How enrollment works
- Attest. The workload presents an attestation to
POST /v1/enrollments: a single-use sandbox token, or a short-lived OIDC JWT from an approved workload issuer. - Challenge. The control plane checks the attestation and that the grant is active, then returns a single-use challenge valid for 5 minutes and the exact
proof_inputstring to sign. - Prove possession. The workload generates a P-256 key, signs a CSR with it, and signs
proof_inputwith the same key. - Issue.
POST /v1/enrollments/{id}/completeverifies the CSR signature and the proof, binds challenge, identity, environment, grant and public key in one single-use authorization, and sends an issuance intent to the isolated signer. - Deliver. The response carries the leaf, the issuing CA and root, and a renewal hint, or
202with a pollable issuance request when signing is pending.
What the server derives
Choose an attestation
| Kind | Environments | Who establishes it | Use when |
|---|---|---|---|
sandbox_token | Sandbox only | A partner admin approves enrollment in the portal; the CLI redeems a single-use mtenr_… token within 30 minutes. | Trying things out on a laptop or CI. |
oidc_jwt | Sandbox and production | The workload’s own identity provider (for example the Kubernetes API server) signs a JWT for an exact issuer, audience and subject approved by the partner organization. | Anything that runs unattended, and always in production. |
A human browser login can approve a sandbox enrollment, but it never establishes production workload provenance.
Sandbox token enrollment
After accepting an invitation with attestation_kind: "sandbox_token", a partner admin approves enrollment for the workload and grant. In the portal this is Approve enrollment; over the API:
POST /v1/workloads/wl_01jcqb3p1y2z4a6b8c0d2e4f6g/enrollment-tokens
Content-Type: application/json
X-CSRF-Token: <csrf_token from GET /v1/me>
{ "grant_id": "gr_01jcqb3p1n3p5q7r9s1t3v5w7x" }{
"token": "mtenr_q8V0…",
"expires_at": "2026-11-14T16:01:00Z",
"grant_id": "gr_01jcqb3p1n3p5q7r9s1t3v5w7x",
"workload_id": "wl_01jcqb3p1y2z4a6b8c0d2e4f6g",
"label": "SANDBOX: not valid for production workload provenance"
}Hand the token to the workload and run:
mt enroll --token mtenr_q8V0… --api https://api.mutualtrust.ioThe token is consumed on first use. Reusing it returns 401 attestation_invalid (or 409 attestation_replayed if two attempts race).
OIDC workload identity
With OIDC attestation the workload proves who it is with a JWT minted by its platform. Three things must line up exactly:
- The issuer is approved for the partner (consumer) organization, together with its HTTPS JWKS URI.
- The workload binding records that issuer, an audience dedicated to Mutual Trust, and a stable subject.
- Each enrollment presents a fresh token: signed by a key from that JWKS (
RS256orES256), with matchingiss, anaudthat includes the bound audience,subequal to the bound subject, unexpired (30-second skew), and issued at most 3600 seconds ago. - If the workload is bound to a service account UID, the token’s
kubernetes.io.serviceaccount.uidclaim must equal it (see below).
1. Approve the workload issuer
A partner admin of the consumer organization approves the issuer once. Discovery is restricted: the control plane fetches keys only from the JWKS URI you approve, never from arbitrary issuers.
POST /v1/organizations/org_01jcqa7m3a2b7c9d4e6f8g1h3j/workload-issuers
Content-Type: application/json
{
"issuer_url": "https://oidc.eks.eu-west-1.amazonaws.com/id/5C1E9A7B3D2F48A6B0C4E8D1F3A5B7C9",
"jwks_uri": "https://oidc.eks.eu-west-1.amazonaws.com/id/5C1E9A7B3D2F48A6B0C4E8D1F3A5B7C9/keys"
}{
"id": "wli_01jcqc2e4f6g8h0j2k4m6n8p0q",
"issuer_url": "https://oidc.eks.eu-west-1.amazonaws.com/id/5C1E9A7B3D2F48A6B0C4E8D1F3A5B7C9",
"jwks_uri": "https://oidc.eks.eu-west-1.amazonaws.com/id/5C1E9A7B3D2F48A6B0C4E8D1F3A5B7C9/keys",
"state": "approved"
}Both URLs must be https://. The issuer string you approve must be byte-for-byte the one you later bind in the workload (the JWT’s iss may differ only by a trailing slash).
2. Bind the workload when accepting the invitation
{
"name": "harbor-order-sync",
"attestation_kind": "oidc_jwt",
"issuer": "https://oidc.eks.eu-west-1.amazonaws.com/id/5C1E9A7B3D2F48A6B0C4E8D1F3A5B7C9",
"audience": "mutualtrust",
"subject": "system:serviceaccount:orders:order-sync",
"subject_uid": "6f1a2b3c-4d5e-4f60-8a7b-9c0d1e2f3a4b"
}subject_uid is optional and recommended for Kubernetes. It applies to oidc_jwt workloads only.
Production acceptance rejects sandbox_token with 422: “Production workloads must bind an OIDC workload identity (for example a Kubernetes service account).” See Accepting an invitation.
Kubernetes service accounts
Kubernetes is the pilot default. The cluster’s service-account issuer signs projected tokens; the subject is system:serviceaccount:<namespace>:<name>. Find the issuer and JWKS URI from the cluster:
kubectl get --raw /.well-known/openid-configuration | jq '{issuer, jwks_uri}'The JWKS URI must be reachable over HTTPS from the Mutual Trust control plane. Managed clusters publish it publicly (EKS: https://oidc.eks.<region>.amazonaws.com/id/<id>; GKE: https://container.googleapis.com/v1/projects/<project>/locations/<location>/clusters/<cluster>). For self-managed clusters, publish the discovery document and keys at an HTTPS location and set the API server’s --service-account-issuer accordingly.
Mount a token with a dedicated audience and a lifetime of one hour or less, so it is always younger than the 3600-second limit (the kubelet refreshes it before expiry):
apiVersion: v1
kind: Pod
metadata:
name: order-sync
namespace: orders
spec:
serviceAccountName: order-sync
containers:
- name: app
image: registry.example/harbor/order-sync:1.4.2
volumeMounts:
- name: mutualtrust-token
mountPath: /var/run/secrets/mutualtrust
readOnly: true
volumes:
- name: mutualtrust-token
projected:
sources:
- serviceAccountToken:
path: token
audience: mutualtrust
expirationSeconds: 3600mt enroll \
--workload wl_01jcqb3p1y2z4a6b8c0d2e4f6g \
--grant gr_01jcqb3p1n3p5q7r9s1t3v5w7x \
--oidc-token-file /var/run/secrets/mutualtrust/token \
--api https://api.mutualtrust.ioFind the workload and grant IDs in the portal (the partner’s grant view) or with GET /v1/grants?org_id=<partner org>.
Bind the service account UID
The sub claim names a service account by namespace and name. If the account is deleted and re-created with the same name, the subject is unchanged, so a subject-only binding would accept the replacement. Bind the account’s UID to close that gap: Kubernetes assigns every service account a new UID, and projected tokens carry it at kubernetes.io > serviceaccount > uid.
kubectl get serviceaccount order-sync -n orders -o jsonpath='{.metadata.uid}'Pass it as workload.subject_uid when accepting the invitation, or set it later. With a UID bound, a token without the claim or with a different UID is refused with 401 attestation_invalid, and the control plane logs enrollment.subject_uid_mismatch.
After replacing a service account, a partner administrator re-approves the workload by rebinding it to the new UID. The change is recorded in the partner’s audit log and in the audit log of every API owner with a live grant to the workload (workload.binding_updated, with the previous and new UID). Certificates already issued stay valid until they expire or are revoked.
PATCH /v1/workloads/wl_01jcqb3p1y2z4a6b8c0d2e4f6g/binding
Content-Type: application/json
{ "subject_uid": "0a0b0c0d-1111-4222-8333-944455556666", "note": "order-sync service account recreated" }Send "subject_uid": null to remove the UID binding and accept any token with the bound issuer, audience and subject. In the portal, the grant page shows the bound UID and partner administrators get Rebind service account.
Step 1: create the enrollment
POST https://api.mutualtrust.io/v1/enrollments
Content-Type: application/json
{
"workload_id": "wl_01jcqb3p1y2z4a6b8c0d2e4f6g",
"grant_id": "gr_01jcqb3p1n3p5q7r9s1t3v5w7x",
"attestation": { "kind": "oidc_jwt", "token": "eyJhbGciOiJSUzI1NiIsImtpZCI6…" }
}For a sandbox token send {"attestation":{"kind":"sandbox_token","token":"mtenr_…"}}; the workload and grant come from the token.
{
"enrollment_id": "enr_01jcqb5q2h4j6k8m0n2p4q6r8s",
"challenge": "Xq3m7Yf0b2RkT9vW1sZpLc8nHd4aGe6uJ5iKo0tQyB8",
"expires_at": "2026-11-14T15:36:00Z",
"proof_input": "mt-enroll-v1:enr_01jcqb5q2h4j6k8m0n2p4q6r8s:Xq3m7Yf0b2RkT9vW1sZpLc8nHd4aGe6uJ5iKo0tQyB8",
"workload_id": "wl_01jcqb3p1y2z4a6b8c0d2e4f6g",
"grant_id": "gr_01jcqb3p1n3p5q7r9s1t3v5w7x"
}Step 2: key, CSR and proof of possession
Generate an ECDSA P-256 key on the workload, build a PKCS#10 CSR signed with it (ECDSA with SHA-256; the subject is ignored), and sign the exact proof_input bytes (UTF-8) with the same key.
Proof encoding
# key and CSR (the key stays on this machine)
openssl ecparam -name prime256v1 -genkey -noout | openssl pkcs8 -topk8 -nocrypt -out key.pem
openssl req -new -key key.pem -subj "/CN=mt-workload" -outform DER -out csr.der
# proof over proof_input, raw r||s, base64url
PROOF=$(node -e '
const { sign, createPrivateKey } = require("node:crypto");
const key = createPrivateKey(require("node:fs").readFileSync("key.pem"));
process.stdout.write(sign("sha256", Buffer.from(process.argv[1]), { key, dsaEncoding: "ieee-p1363" }).toString("base64url"));
' "mt-enroll-v1:enr_01jcqb5q2h4j6k8m0n2p4q6r8s:Xq3m7Yf0b2RkT9vW1sZpLc8nHd4aGe6uJ5iKo0tQyB8")
curl -sS https://api.mutualtrust.io/v1/enrollments/enr_01jcqb5q2h4j6k8m0n2p4q6r8s/complete \
-H 'content-type: application/json' \
-d "{\"csr\":\"$(base64 < csr.der | tr -d '\n')\",\"proof\":\"$PROOF\"}"Step 3: complete and collect the certificate
{
"state": "issued",
"issuance_request_id": "isr_01jcqb5q9t1v3w5x7y9z1a3b5c",
"certificate_id": "crt_01jcqb5r0d2e4f6g8h0j2k4m6n",
"grant_id": "gr_01jcqb3p1n3p5q7r9s1t3v5w7x",
"principal": "urn:mutualtrust:workload:sandbox:org_01jcqa7m3a2b7c9d4e6f8g1h3j:wl_01jcqb3p1y2z4a6b8c0d2e4f6g",
"serial": "5c1e9a7b3d2f48a6b0c4e8d1f3a5b7c9",
"fingerprint": "2e7a…",
"not_before": "2026-11-14T15:30:30Z",
"not_after": "2026-11-15T15:30:30Z",
"renewal_hint": {
"renew_after": "2026-11-15T07:26:41Z",
"strategy": "two_thirds_with_jitter"
},
"certificate_pem": "-----BEGIN CERTIFICATE-----\n…\n-----END CERTIFICATE-----\n",
"chain_pem": "-----BEGIN CERTIFICATE-----\n… issuing CA …\n-----END CERTIFICATE-----\n",
"root_pem": "-----BEGIN CERTIFICATE-----\n… root …\n-----END CERTIFICATE-----\n",
"root_fingerprint": "9f2c41d8e3b7a6051c4e8d2f7a9b3c6e1d5f0a8b2c7e4d9f3a6b1c8e5d2f7a40",
"ignored_csr_extensions": [],
"note": "Subject, SAN, key usage, EKU and validity are derived by policy; CSR extensions are ignored."
}Present certificate_pem followed by chain_pem (leaf + issuing CA) as your TLS client chain, with your private key. Pin root_fingerprint against the trust fingerprint the API owner shows if you want an out-of-band check.
When signing is pending: 202 and polling
If the signer’s response is ambiguous or it is temporarily unavailable, the approved intent is kept and retried, and complete answers 202 Accepted:
{
"state": "pending",
"issuance_request_id": "isr_01jcqb5q9t1v3w5x7y9z1a3b5c",
"poll_url": "/v1/issuance-requests/isr_01jcqb5q9t1v3w5x7y9z1a3b5c",
"retry_after_seconds": 5
}Poll with the enrollment challenge as proof that you are the enrolling workload. Each poll also retries signing; the response is 200 with "state":"pending" until the certificate is issued, then the same body as above. Only one certificate can ever be committed per issuance request, even across retries.
GET https://api.mutualtrust.io/v1/issuance-requests/isr_01jcqb5q9t1v3w5x7y9z1a3b5c
X-Enrollment-Challenge: Xq3m7Yf0b2RkT9vW1sZpLc8nHd4aGe6uJ5iKo0tQyB8The CLI polls with exponential backoff from 2 seconds up to 30 seconds.
Validity and files
- Leaves are valid for 24 hours from 30 seconds before issuance, cut short by the grant’s
expires_ator the issuing CA’s validity. If less than a minute would remain, issuance fails with409 grant_expiring. mt enrollwritesworkload/key.pem(mode 0600, or the macOS Keychain with--key-store keychain),cert.pem,chain.pem(leaf + issuing CA),root.pemandidentity.jsonunder its home directory (per platform), committed together.- A lost key cannot be recovered: enroll again (a new attestation) and revoke the old leaf.
Enrollment errors
| Status | Code | Cause and fix |
|---|---|---|
| 401 | attestation_invalid | Sandbox token unknown, expired or used; OIDC workload unknown, issuer not approved, or the token’s service account UID does not match the bound UID. Create a new token, approve the issuer, or rebind the workload. |
| 401 | unauthenticated | The OIDC token failed verification: signature, iss, aud, sub, expiry, or older than 3600 seconds. |
| 409 | attestation_replayed | A sandbox token was redeemed concurrently. |
| 409 | grant_not_active | The grant is pending the API owner’s approval, or suspended, revoked or expired. Wait for approval or contact the API owner. |
| 409 | enrollment_used | This challenge was already used. Start a new enrollment. |
| 409 | challenge_expired | More than 5 minutes passed since the challenge. Start a new enrollment. |
| 409 | grant_expiring | The grant expires within a minute. |
| 409 | issuance_refused | The signer refused the intent (for example the issuer was suspended during issuance). |
| 422 | validation_failed | Malformed or non-P-256 CSR, bad CSR signature, or a proof that does not verify with the CSR key. |
| 503 | issuer_unavailable / signer_unavailable | The issuing CA is suspended, or the signer is temporarily unavailable. Existing certificates keep working. |
All codes: Error catalogue.