Backup Restoration Test Log (Evidence 7.3.5)

Policy Requirement: Full system restore test conducted at least annually. CheckTick Standard: Quarterly restoration drills (once in production). Current Status: Pre-production. Full end-to-end restoration drill scheduled as a mandatory pre-launch action.


Test Log

ID Date Type Scenario Environment Result Time Taken Verified By
CT-01 March 2026 Partial โ€” Vault unseal & key reconstruction Simulated Vault loss requiring unseal and platform master key reconstruction from Shamir custodian shares via YubiKey hardware tokens Development โœ… Pass < 30 mins Dr Simon Chapman
CT-02 Pre-launch TBC Full system restore Simulated complete infrastructure loss โ€” DB + Vault + App redeployment Staging โณ Pending โ€” โ€”
CT-03 2026-11-06 Partial โ€” Vault nightly snapshot restoration Nightly encrypted raft snapshot restored to a scratch Vault; unsealed with production 3-of-4 shares; production AppRole login + platform master key verified Scratch in-cluster (production snapshot) โœ… Pass ~45 mins Dr Simon Chapman

CT-01 Detail: Vault Unseal & Platform Key Reconstruction

Date: March 2026 Environment: Development Conducted by: Dr Simon Chapman (CTO)

Scope: Validated the complete Vault unseal and platform master key reconstruction workflow using YubiKey hardware tokens. This is the most technically complex component of the full restoration procedure and a prerequisite for any encrypted data recovery.

Configuration: - 2 YubiKeys each holding 2 unseal keys and 2 shares of the platform custodian component - 2 YubiKey backups stored at separate secure locations - Shamir threshold: 3 of 4 shares required to reconstruct the platform key

Steps completed:

Step Result Notes
Vault unseal using YubiKey hardware tokens โœ… Pass Both primary YubiKeys tested
Platform custodian component reconstruction from Shamir shares โœ… Pass Threshold met with 3 of 4 shares
Vault health check post-unseal โœ… Pass All secrets accessible
Full end-to-end key derivation workflow โœ… Pass Survey KEK successfully derived

Outcome: Workflow confirmed viable. No issues identified.

Limitations: Development environment only, against test data. Full end-to-end test including database restoration and survey decryption is planned pre-launch (CT-02).


CT-03 Detail: Vault Nightly Snapshot Restoration

Date: 2026-11-06 Environment: Scratch in-cluster Vault (production snapshot) Conducted by: Dr Simon Chapman (CTO)

Scope: First restoration test of the nightly encrypted Vault raft snapshot pipeline (live since 2026-11-06 โ€” see the infrastructure change log, 06/11/2026). Proves the off-cluster backup chain is genuinely recoverable end to end: object store โ†’ age decryption โ†’ raft snapshot restore โ†’ production barrier unseal โ†’ escrow material readable.

Chain exercised:

  1. Newest vault-raft-*.tar.age downloaded from s3://checktick-backups/vault/ (sha256-verified)
  2. Decrypted on the operator laptop with the Bitwarden-held age private key (the key never entered the cluster)
  3. vault operator raft snapshot inspect โ€” full key inventory present (core/master, seal-config, policies, auth, both KV mounts)
  4. Restored into a fresh scratch Vault (vault operator raft snapshot restore -force โ€” -force is required for a cross-cluster restore; the default hash check correctly rejects a snapshot sealed under different unseal keys, which is itself a custody property)
  5. Unsealed with the PRODUCTION 3-of-4 Shamir shares โ€” the restored barrier is the production one
  6. Verified: production cluster identity (vault-cluster-978f63d7), checktick-app AppRole login with the production credentials, and a count-only read of secret/data/platform/master-key (values never printed)

Result: โœ… Pass. The stored object requires BOTH the Shamir shares AND the operator's age private key โ€” the custody model held at every layer.

Operational findings:

  • A fresh raft Vault does not create its own storage path โ€” mkdir first
  • Post-restore, the running process serves its OLD in-memory seal config; a process restart is required before the restored 4-of-3 config is served (unsealing before the restart fails with invalid key size)
  • snapshot restore needs -force for cross-cluster restores

Teardown: the scratch StatefulSet + PVC (which held a copy of production Vault data) were deleted in the same session; laptop working files deleted.


CT-02 Plan: Full System Restoration Drill

Target Date: Prior to live clinical deployment Environment: Staging

Scope: Full end-to-end system restoration from Northflank snapshots, covering all components required to restore the essential service.

Steps planned:

Step Target Time Pass Criteria
Restore MFA-protected admin access to Northflank and GitHub < 15 mins Console accessible
Restore Vault volume from Northflank snapshot < 20 mins Vault service running
Unseal Vault using YubiKey tokens (as per CT-01) < 15 mins Vault unsealed, health check passes
Restore PostgreSQL from Northflank addon snapshot < 60 mins Database accessible
Redeploy application container from GitHub < 20 mins Application boots successfully
Verify DNS resolution and platform reachability < 10 mins Platform responding on expected domain
End-to-end decryption of test survey record < 15 mins Test record successfully decrypted
Total RTO < 4 hours All steps passed within target

Results: To be completed and documented here prior to go-live.


Pre-Production Statement

CheckTick is currently in pre-production. No clinical patient data is held at this time. CT-01 confirms the most critical recovery component โ€” Vault unseal and key reconstruction โ€” has been successfully validated in a development environment using production-equivalent YubiKey hardware and Shamir key distribution.

CT-02 (full system restoration drill) is a mandatory documented pre-launch action and will be completed and recorded here before any health data is processed.

Next scheduled review: Upon completion of CT-02, or June 2026, whichever is sooner.