Issuer and key rollover

Replace an issuing CA, a root or the manifest-signing key in stages: the successor is published first, issuance switches only after every gateway acknowledged it, and the old material is retired after a canary and the remaining leaf lifetime plus an overlap window.

What rolls and who starts it

Every trust space has a private root, an online issuing CA (pathLen 0) and a separate manifest-signing key that gateways pin at pairing. Each can be replaced without re-enrolling workloads or re-pairing gateways. A rollover never re-keys material in place: the signer creates a new key version, and the old one stays trusted until it is safe to retire.

Rollover kinds
Kind	Started by	Control	Successor
`issuing_ca`	Your `trust_admin` (portal or `POST /v1/spaces/{id}/rollovers`) or Sectigo trust operations	Step-up in production; production staging runs as a PKI Operations ceremony	New issuing CA under the same root
`root`	Sectigo PKI Operations	Two independent people at the start and again before the old root is retired	New root with a new issuing CA (no cross-signature); both chains are in the bundle during the overlap
`manifest_key`	Sectigo PKI Operations	Two independent people at the start	New manifest-signing key, introduced to gateways by a statement signed with the key they already pin

Stages

Each rollover is a small state machine. Every transition is a compare-and-swap on the rollover revision (send it as If-Match), is written with its audit entry and an issuer.rollover_* event in one transaction, and appears in the rollover timeline. The control plane moves a rollover on by itself once a stage's condition holds (every minute); POST /v1/rollovers/{id}/advance checks immediately.

started (issuer.rollover_started): the signer creates the successor as a new key version. Root and manifest-key rollovers wait in requested until a second person approves.
staged (issuer.rollover_staged): the successor chain is added to the signed trust bundle of every service in the space, next to the current one. Certificates are still issued by the current issuing CA. The rollover waits until every paired gateway in the space acknowledged a policy revision that carries the successor.
issuing (issuer.rollover_issuing): new certificates come from the successor. Workloads move over at their next renewal (24 h leaves renew around two thirds of their lifetime). The rollover waits for a canary: on every service with enrolled workloads, an allowed request with a successor-issued certificate.
draining (issuer.rollover_draining): the previous issuing CA stops issuing. The rollover waits until the last certificate of the previous issuer that a workload may still present has expired (a certificate replaced by a renewal whose reload was confirmed no longer counts), plus the overlap window (default 24 hours).
retired (issuer.rollover_retired): the previous issuer leaves the bundle, its final CRL is published and its key version is disabled. Certificates it issued are now refused: mt-enforcer builds client chains only from the bundle and denies them with tls_verification_failed (TLS reason chain_invalid); gateways that verify the chain before the policy check, such as the Cloudflare Worker, deny with unknown_issuer.

A rollover can be aborted before it drains (issuer.rollover_aborted): issuance returns to the previous issuer and the successor leaves the bundle. Certificates the successor already issued keep working until they expire. A manifest-key rollover cannot be aborted once a gateway pins the new key.

Start an issuing CA rollover

Start a rollovercurl -sS -X POST https://api.mutualtrust.io/v1/spaces/spc_01jcqb3p1n3p5q7r9s1t3v5w7x/rollovers \
  -H 'Authorization: Bearer mtk_…' -H 'Content-Type: application/json' \
  -d '{"kind": "issuing_ca", "overlap_seconds": 86400, "reason": "annual rollover"}'

The response is the rollover with next.waiting_for (awaiting_gateway_acks, awaiting_canary, draining and so on), next.pending (the gateway or service ids still outstanding), the timeline, per-gateway acknowledgment status and workloads_on_old_issuer. The portal shows the same on the space page (Services, open a service, Issuers and rollover). Only one rollover per space can be in progress (rollover_in_progress).

Offline gateways

A gateway that never acknowledges blocks the rollover in staged: issuing from the successor before a gateway trusts it would deny every renewed workload there. If a gateway is decommissioned or unreachable for good, continue explicitly:

Continue without the gateways that have not acknowledgedcurl -sS -X POST https://api.mutualtrust.io/v1/rollovers/trl_…/ack-override \
  -H 'Authorization: Bearer mtk_…' -H 'If-Match: "3"' -H 'Content-Type: application/json' \
  -d '{"reason": "edge-eu-2 decommissioned"}'

The override records which gateways it covered and who decided, in the audit log and the timeline. In production it needs a recent step-up (or an API credential created with high_risk_allowed). For rollovers that Sectigo runs, the override is a two-person staff decision.

Manifest-signing key rotation

A gateway pins the manifest verification key (kid and public key) when it pairs, and accepts a different key only through a key-rotation statement signed by the key it already pins. During the rotation every manifest response carries key_rotation next to a manifest signed by the new key:

Rotation statement (JWS payload, header kid = from_kid, typ mt.key-rotation.v1+jws){"typ": "mt.key-rotation.v1", "rotation_id": "trl_…", "tenant_id": "org_…", "environment": "production",
 "space_id": "spc_…", "from_kid": "kv_old…", "to_kid": "kv_new…",
 "to_jwk": {"kty": "EC", "crv": "P-256", "x": "…", "y": "…"},
 "iat": 1794200000, "nbf": 1794200000, "exp": 1794891200}

The gateway verifies the statement with its pinned key, checks tenant, environment, that it starts from the pinned kid and introduces exactly the kid of the manifest, and its validity window (30 seconds of clock allowance), then verifies the manifest with the new key. Only when everything passes does it persist the new pin atomically (mt-enforcer: policy-key.json in its data directory) and apply the manifest. From then on manifests signed by the previous key are refused like any unpinned key (jws_kid_mismatch), and rotating back to a previously pinned kid is refused (rotation_replayed). Each acknowledgment reports the pinned kid (policy_kid), and the control plane retires the previous key once every gateway pins the new one and the overlap has passed. A gateway offline beyond the statement's exp (overlap plus seven days) must be paired again.

Key rotation rejection codes
Code in the rejected acknowledgment	Meaning
`rotation_signature_invalid`	The statement is not a valid JWS signed by the pinned key.
`rotation_malformed`	Missing or invalid fields (key, times).
`rotation_wrong_scope`	Wrong statement type, tenant or environment.
`rotation_not_from_pinned`	The statement starts from a different kid (for example an old statement replayed after the rotation).
`rotation_kid_mismatch`	The statement introduces a different kid than the manifest is signed with.
`rotation_replayed`	The statement would pin a kid this gateway already retired.
`rotation_not_yet_valid`, `rotation_expired`	Outside the statement's validity window.
`state_not_persisted`	The gateway could not store the new pin; it keeps the old one.

Compromised issuing CA

Sectigo incident command can suspend an issuing CA at once and start an expedited rollover with a short overlap (default one hour). A suspended issuer leaves the bundle on the next manifest, so its certificates are denied everywhere; renewal of those certificates is refused with reenrollment_required. Partners with affected workloads see a portal alert and an entry in their audit log (issuer.reenrollment_required) asking them to re-enroll with a new enrollment token. The expedited rollover does not wait for acknowledgments or canaries, because the old issuer is already out of service.

Evidence

The rollover timeline and audit log record every transition, decision, override and approval policy used.
Per-gateway acknowledgments show the first revision that carried the successor and the revision each gateway applied.
GET /v1/spaces/{id}/trust-bundle.pem exports every issuer of the space with its state.
The final CRL of the retired issuer is published like every CRL.