Troubleshooting

Every denial carries a stable code, a human-readable title and a request ID. Start with the code; quote the request ID if you contact support.

Read a denial

A gateway denial is a 403 problem document:

Gateway 403 body{
  "type": "https://mutualtrust.io/problems/route_not_permitted",
  "title": "This workload is not permitted to call this route or method.",
  "status": 403,
  "code": "route_not_permitted",
  "request_id": "req_7b2e9d4a1f6c3e8b5a0d",
  "tls": "verified",
  "policy_revision": 2
}

tls is the separate TLS verdict: verified, failed or absent.
policy_revision is the revision the gateway enforced (null if it has none). The x-mt-policy-revision header carries epoch.revision.
The same decision is recorded on a signed receipt visible to the API owner, so both sides can look at the same evidence.

Gateway deny reasons

Checks run in this order; the first failure wins: policy present → policy fresh → client certificate → trusted issuer → exactly one workload URI for this environment → clientAuth EKU → not expired → serial not revoked → path unambiguous → grant and route.

Gateway deny reasons
Code	Message shown	Cause	Fix
`allowed`	None	The request was authorized (shown on receipts).	None needed.
`uninitialized`	This gateway has not received a signed policy yet; all protected routes are denied.	The gateway has never applied a manifest (not paired, or cannot reach the API since start).	Check pairing and outbound HTTPS to `api.mutualtrust.io`. Look for `manifest_fetch_failed` or `manifest_rejected` in the gateway log.
`manifest_expired`	Cached policy expired and the control plane is unreachable; failing closed.	No successful refresh for more than 5 minutes 30 seconds.	Restore connectivity; the next successful refresh re-enables traffic. Check the gateway clock.
`no_client_certificate`	No client certificate presented. Configure your client with the enrolled certificate and key.	No certificate presented, or (with `"tls":"failed"` in the body) the chain did not verify against the trust bundle (wrong space, expired chain, or missing issuing CA in the presented chain).	Send `chain.pem` (leaf + issuing CA) and `key.pem`. If TLS failed, re-enroll against this service’s space.
`tls_verification_failed`	The client certificate chain did not verify against this service’s trust bundle.	Reserved for chain failures reported separately from authorization; gateways report chain failures as `no_client_certificate` with `"tls":"failed"`.	As above.
`unknown_issuer`	The certificate was issued by a CA this service does not trust.	The chain verified, but the issuing CA is not in this service’s signed trust list.	Enroll against the service you are calling.
`malformed_san`	The certificate does not carry exactly one valid workload identity.	Zero or several workload URIs, a malformed URI, or a URI for another environment.	Use a certificate issued by Mutual Trust for this environment.
`missing_client_auth_eku`	The certificate is not valid for client authentication.	The leaf lacks the clientAuth extended key usage.	Use the Mutual Trust workload certificate, not a server certificate.
`certificate_expired`	The client certificate has expired; renew it.	`not_after` has passed.	Run `mt renew`; if it reports `certificate_expired`, re-enroll.
`certificate_revoked`	The client certificate was revoked.	The serial is in the deny set: revoked individually or by grant withdrawal.	Re-enroll if your grant is still active; otherwise contact the API owner.
`grant_revoked`	Access for this workload was withdrawn by the API owner.	The grant was withdrawn.	Contact the API owner (the service contact is shown in your grant view).
`grant_suspended`	Access for this workload is temporarily suspended by the API owner.	The grant is suspended.	Contact the API owner.
`grant_expired`	The access grant for this workload has expired.	The grant’s `expires_at` passed.	Ask the API owner for a new invitation.
`no_matching_grant`	No active grant authorizes this workload for this service.	Valid identity, but no rule for this principal and issuer on this service.	Check you are calling the right service with the right workload.
`route_not_permitted`	This workload is not permitted to call this route or method.	A live grant exists but its scope does not cover this method and path.	Ask the API owner to change the scope; check the method (HEAD needs GET).
`ambiguous_route`	The request path is ambiguous and was refused.	Encoded slashes or dots, backslashes, dot or empty segments.	Send a normalized path.

Other gateway responses

Other gateway responses
Response	Meaning
`502 {"code":"origin_unavailable"}`	The request was allowed but the gateway could not reach your origin.
`401 {"error":"origin_requires_gateway"}`	From the quickstart sample origin: a request reached the origin without the gateway’s verified principal header. Your real origin should behave the same way.
Connection closed	The gateway drains connections when policy goes stale, when a principal loses all rules, and after 5 minutes of connection age. Clients should reconnect.

Manifest rejections

If a gateway refuses a manifest it keeps its current policy, logs manifest_rejected: <code>, and (except for scope errors) sends a signed rejected acknowledgment with that code.

Manifest rejection codes
Code	Meaning
`jws_alg_rejected`	The JWS algorithm is not ES256.
`jws_kid_mismatch`	Signed with a key other than the one pinned at pairing.
`jws_typ_mismatch`	Wrong JWS `typ` (must be `mt.manifest.v1+jws`).
`jws_signature_invalid`	The signature does not verify.
`wrong_scope`	Tenant, environment, service, gateway, epoch or revision do not match the pinned target or the embedded policy.
`expired`	Received after `exp` + 30 s.
`not_yet_valid`	`iat` is more than 30 s in the future; check the gateway clock.
`hash_mismatch`	The embedded policy does not hash to `policy_hash`.
`downgrade`	Lower epoch or revision than the gateway’s high-water mark.
`replay_conflict`	Same epoch and revision as applied, but a different hash.

API problems

Common API problems
Status	Code	Typical fix
401	`unauthenticated`	Sign in again, or check the bearer credential (revoked, expired, wrong host for cookies).
403	`forbidden`	Missing role, missing CSRF header on a cookie session, or a production action as a sandbox guest. The title says which.
404	`not_found`	Wrong ID, or a resource in an organization you are not a member of.
409	`invalid_transition`	The resource is in a state that does not allow this action.
412	`revision_mismatch`	Re-read the grant and retry with its current revision in `If-Match`.
415	`unsupported_media_type`	Send `Content-Type: application/json`.
422	`validation_failed`	Check `errors[]` for the failing JSON pointer.
428	`precondition_required` / `idempotency_key_required`	Add `If-Match` or `Idempotency-Key`.
429	`rate_limited`	Wait for `Retry-After` seconds.
503	various	A dependency is unavailable (`Retry-After: 5`). Existing certificates keep working.

The full list is the error catalogue.

Request IDs

API: every response has an X-Request-Id header, and problem bodies include request_id. On the hosted service it is req_ followed by the edge ray ID.
Gateway: every response has x-mt-request-id (req_ + 20 hex characters); the same ID is on the receipt and is forwarded to your origin.
CLI: mt prints the request ID with every error and every call.
Audit entries record the API request_id and an operation_id, so support can trace a request from the edge to the database transition.

Health and readiness

Control plane healthcurl -sS https://api.mutualtrust.io/healthz
{"ok":true,"deployment":"sandbox"}

GET /v1/services/{id} returns a readiness checklist. Each step shows exactly what is missing:

Service readiness steps
Step	Passes when
`trust_anchor`	The space’s private root exists.
`gateway_paired`	At least one gateway is paired with a pinned policy key.
`policy_applied`	A gateway acknowledged the current policy revision.
`allow_receipt`	An allowed request was observed.
`deny_receipt`	A denied control request with verified TLS was observed.

Gateway status in the same response is current, pending (connected, not on the latest revision), offline (no check-in for 5 minutes), never_connected or disabled, with policy_age_seconds since its last acknowledgment.

Frequently seen problems

Frequently seen problems
Symptom	Likely cause
Partner blocked right after renewal	The client kept the old key with the new certificate, or still presents the predecessor after it expired. Reload both `chain.pem` and `key.pem` together, then confirm reload.
Renewal returns 404	The proof was signed with the wrong key or over the wrong CSR digest or timestamp.
Enrollment 422 “Proof of possession failed”	The proof is DER-encoded or signs something other than `proof_input`. Use raw `r‖s`, base64url.
OIDC enrollment 401	Token older than one hour, audience mismatch, or `sub` differs from the binding.
Every request `manifest_expired`	Gateway lost outbound access or its clock is far off.
Invitation accept 403	You are signed in as a member of a different organization, or as a sandbox guest on a production invitation.