Atom PKCS#11 CA Operations and Recovery Runbook

This runbook covers Atom-managed online CA keys held by a PKCS#11 token. It does not apply to the production root, which remains offline. Substitute the HSM vendor's supported commands and quorum procedures for the SoftHSM examples.

Security invariants

• Set ATOM_PKI_CA_KEY_BACKEND=pkcs11 only through operator deployment configuration. No GraphQL, HTTP, EST, ACME, or database request selects a backend or supplies a key reference.
• Inject ATOM_PKI_PKCS11_USER_PIN through the deployment secret mechanism. Do not place it in source control, an image, logs, or a database row.
• Require token-generated P-256 keys with CKA_SENSITIVE=true and CKA_EXTRACTABLE=false. Atom refuses a key that does not retain that policy.
• Back up or replicate the token with the vendor's protected mechanism. Never export a plaintext CA private key.

Configuration and preflight

Required configuration:

ATOM_PKI_CA_KEY_BACKEND=pkcs11
ATOM_PKI_PKCS11_MODULE_PATH=/opt/vendor/lib/libpkcs11.so
ATOM_PKI_PKCS11_TOKEN_LABEL=atom-production-ca
ATOM_PKI_PKCS11_USER_PIN=<secret injection>

Bounded-failure defaults are a 2-second operation timeout, one retry, eight in-flight operations, a three-failure circuit threshold, and a 30-second half-open interval. Tune them only from measured HSM latency and capacity.

Before rollout:

1. Confirm the module and token are mounted into every Atom replica and the service identity can open a read/write session.
2. Confirm the user PIN can generate and sign with an EC P-256 key without permitting private-value reads or key wrapping.
3. Take a protected token backup or confirm the vendor replication quorum.
4. Start one Atom replica. Startup must report PKI CA key providers validated. A missing token, wrong PIN, missing object, or certificate mismatch must stop startup.
5. Provision a canary intermediate, issue and independently verify a leaf, then verify CRL and OCSP publication before increasing replica count.

SoftHSM pre-production exercise

SoftHSM is for interoperability and recovery drills, not production key protection:

export SOFTHSM2_CONF=/secure/test/softhsm2.conf
softhsm2-util --init-token --free --label atom-pki-test \
  --so-pin "$TEST_SO_PIN" --pin "$TEST_USER_PIN"

Point Atom at the installed libsofthsm2.so, the atom-pki-test label, and the user PIN. CI exercises signing, authentication failure, non-exportability, idempotent key lookup, provider rotation, destruction, and a file-store backup restore on a disposable token.

Provider rotation

Changing ATOM_PKI_CA_KEY_BACKEND affects only newly provisioned authorities. It never rewrites an existing authority row.

• Encrypted database to PKCS#11: keep the old CA KEK configured, configure the PKCS#11 token, switch the provisioning backend, and provision the replacement authority. Retiring and retired encrypted authorities continue signing their retained CRL/OCSP artifacts.
• PKCS#11 to encrypted database: keep the PKCS#11 module, token, and PIN configured while any retained PKCS#11 authority must publish artifacts. Set the provisioning backend to encrypted_database and retain the CA KEK.
• Token replacement: use vendor replication or restore so the replacement token contains the same object IDs and public keys before changing the module or token label. Atom compares every retained certificate to its token public key at startup.

Complete normal authority activation and retirement checks before disabling an old provider. Do not delete old keys while unexpired certificates, CRLs, or OCSP responses still depend on them.

Outage and throttling response

The provider times out, retries only bounded idempotent operations, caps in-flight work, and opens a circuit after consecutive transient failures. It never falls back to another authority or to a database key.

1. Confirm HSM health, connectivity, session capacity, and the Atom metrics atom_pki_key_provider_operations_total{provider="pkcs11",...}.
2. Stop new authority automation or enrollment traffic at the deployment edge if failures persist. Do not alter issuer or key-reference rows.
3. Restore token access. Wait for the circuit reset interval or restart one replica and confirm startup validation.
4. Issue a canary certificate and verify the chain, CRL, and OCSP response.
5. Record the affected authority and operation identifiers. Never copy the PIN, opaque reference, CSR, certificate private material, or vendor session data into the incident record.

Backup recovery drill

1. Stop Atom so no signing session is active.
2. Preserve the failed token/store for forensics. Restore the last protected backup into an isolated replacement token using the vendor procedure.
3. Verify the replacement token label and object inventory without exporting private values.
4. Start one Atom replica with the replacement module/token configuration. Startup validates every opaque reference and certificate/public-key pair.
5. Independently verify a canary signature, one retained CRL, and one OCSP response, then return other replicas to service.

Do not "recover" by generating a new object under an old reference. Its public key will not match the retained CA certificate and Atom will fail startup. If an exact protected backup or replica is unavailable, treat the CA key as lost, follow the authority-compromise/replacement procedure, and preserve old public certificates for validation only.

Destruction

Destroy a token key only after the authority retention policy proves that no unexpired credential or publication artifact requires it, a current protected backup exists if policy requires one, and the action has the required operator quorum. Provider destruction removes both private and public token objects and is fail-closed; a database lifecycle transition must not be used as a substitute for successful provider destruction.