Printed from CheckTick DSPT Compliance Documentation
Backup Restoration Test Log (Evidence 7.3.5)
Policy Requirement: Full system restore test conducted at least annually. CheckTick Standard: Quarterly restoration drills (once in production). Current Status: Pre-production. Full end-to-end restoration drill scheduled as a mandatory pre-launch action.
Test Log
| ID | Date | Type | Scenario | Environment | Result | Time Taken | Verified By |
|---|---|---|---|---|---|---|---|
| CT-01 | March 2026 | Partial โ Vault unseal & key reconstruction | Simulated Vault loss requiring unseal and platform master key reconstruction from Shamir custodian shares via YubiKey hardware tokens | Development | โ Pass | < 30 mins | Dr Simon Chapman |
| CT-02 | Pre-launch TBC | Full system restore | Simulated complete infrastructure loss โ DB + Vault + App redeployment | Staging | โณ Pending | โ | โ |
| CT-03 | 2026-11-06 | Partial โ Vault nightly snapshot restoration | Nightly encrypted raft snapshot restored to a scratch Vault; unsealed with production 3-of-4 shares; production AppRole login + platform master key verified | Scratch in-cluster (production snapshot) | โ Pass | ~45 mins | Dr Simon Chapman |
CT-01 Detail: Vault Unseal & Platform Key Reconstruction
Date: March 2026 Environment: Development Conducted by: Dr Simon Chapman (CTO)
Scope: Validated the complete Vault unseal and platform master key reconstruction workflow using YubiKey hardware tokens. This is the most technically complex component of the full restoration procedure and a prerequisite for any encrypted data recovery.
Configuration: - 2 YubiKeys each holding 2 unseal keys and 2 shares of the platform custodian component - 2 YubiKey backups stored at separate secure locations - Shamir threshold: 3 of 4 shares required to reconstruct the platform key
Steps completed:
| Step | Result | Notes |
|---|---|---|
| Vault unseal using YubiKey hardware tokens | โ Pass | Both primary YubiKeys tested |
| Platform custodian component reconstruction from Shamir shares | โ Pass | Threshold met with 3 of 4 shares |
| Vault health check post-unseal | โ Pass | All secrets accessible |
| Full end-to-end key derivation workflow | โ Pass | Survey KEK successfully derived |
Outcome: Workflow confirmed viable. No issues identified.
Limitations: Development environment only, against test data. Full end-to-end test including database restoration and survey decryption is planned pre-launch (CT-02).
CT-03 Detail: Vault Nightly Snapshot Restoration
Date: 2026-11-06 Environment: Scratch in-cluster Vault (production snapshot) Conducted by: Dr Simon Chapman (CTO)
Scope: First restoration test of the nightly encrypted Vault raft snapshot pipeline (live since 2026-11-06 โ see the infrastructure change log, 06/11/2026). Proves the off-cluster backup chain is genuinely recoverable end to end: object store โ age decryption โ raft snapshot restore โ production barrier unseal โ escrow material readable.
Chain exercised:
- Newest
vault-raft-*.tar.agedownloaded froms3://checktick-backups/vault/(sha256-verified) - Decrypted on the operator laptop with the Bitwarden-held age private key (the key never entered the cluster)
vault operator raft snapshot inspectโ full key inventory present (core/master, seal-config, policies, auth, both KV mounts)- Restored into a fresh scratch Vault (
vault operator raft snapshot restore -forceโ-forceis required for a cross-cluster restore; the default hash check correctly rejects a snapshot sealed under different unseal keys, which is itself a custody property) - Unsealed with the PRODUCTION 3-of-4 Shamir shares โ the restored barrier is the production one
- Verified: production cluster identity (
vault-cluster-978f63d7),checktick-appAppRole login with the production credentials, and a count-only read ofsecret/data/platform/master-key(values never printed)
Result: โ Pass. The stored object requires BOTH the Shamir shares AND the operator's age private key โ the custody model held at every layer.
Operational findings:
- A fresh raft Vault does not create its own storage path โ
mkdirfirst - Post-restore, the running process serves its OLD in-memory seal config;
a process restart is required before the restored 4-of-3 config is
served (unsealing before the restart fails with
invalid key size) snapshot restoreneeds-forcefor cross-cluster restores
Teardown: the scratch StatefulSet + PVC (which held a copy of production Vault data) were deleted in the same session; laptop working files deleted.
CT-02 Plan: Full System Restoration Drill
Target Date: Prior to live clinical deployment Environment: Staging
Scope: Full end-to-end system restoration from Northflank snapshots, covering all components required to restore the essential service.
Steps planned:
| Step | Target Time | Pass Criteria |
|---|---|---|
| Restore MFA-protected admin access to Northflank and GitHub | < 15 mins | Console accessible |
| Restore Vault volume from Northflank snapshot | < 20 mins | Vault service running |
| Unseal Vault using YubiKey tokens (as per CT-01) | < 15 mins | Vault unsealed, health check passes |
| Restore PostgreSQL from Northflank addon snapshot | < 60 mins | Database accessible |
| Redeploy application container from GitHub | < 20 mins | Application boots successfully |
| Verify DNS resolution and platform reachability | < 10 mins | Platform responding on expected domain |
| End-to-end decryption of test survey record | < 15 mins | Test record successfully decrypted |
| Total RTO | < 4 hours | All steps passed within target |
Results: To be completed and documented here prior to go-live.
Pre-Production Statement
CheckTick is currently in pre-production. No clinical patient data is held at this time. CT-01 confirms the most critical recovery component โ Vault unseal and key reconstruction โ has been successfully validated in a development environment using production-equivalent YubiKey hardware and Shamir key distribution.
CT-02 (full system restoration drill) is a mandatory documented pre-launch action and will be completed and recorded here before any health data is processed.
Next scheduled review: Upon completion of CT-02, or June 2026, whichever is sooner.