Civo Kubernetes Operations
Operational reference for running CheckTick on managed Kubernetes (Civo, London LON1).
The durable knowledge here was extracted from the Northflank β Civo migration
(production cutover 2026-11-03). The migration runbook and its session log
were removed at close-out (2026-11-06) β the permanent historical record
lives in
docs/compliance/infrastructure-technical-change-log.md,
with the full documents preserved in git history.
For a walkthrough of the manifests and day-to-day validation commands, see
k8s/README.md.
Cluster Overview
- Managed k3s on Civo, UK London (LON1) β UK-sovereign hosting for a medical application
- Node pools: 2Γ
g4s.kube.large(poolslarge2+large3, ~$87/mo) since 2026-11-08 β the 1-large+2-medium layout was redesigned because the ~1.1 GiB/node platform tax is 47% of a medium and drove a week of placement crises; nodes idle ~48β60%. The log-collector DaemonSet is LIVE. DB + Vault PVCs are oncivo-volume(provider-managed block storage); Longhorn serves only the RWX trio (media/SNOMED/OpenObserve) at 2 replicas. - Production namespace
checktick; staging (staging.checktick.uk) runs in the SAME namespace β see Staging environment below - Components:
- web Deployment (image pinned to a published release tag, never
latest) - CloudNativePG PostgreSQL 17 β
checktick-dbcluster (primary + one replica), in-cluster - HashiCorp Vault β single-replica StatefulSet (
vault-0), integrated Raft storage on a PVC (migrated from file storage 2026-11-06) - Traefik ingress (443 only) + cert-manager TLS
- Longhorn volumes:
checktick-media,checktick-snomed-data - 13 CronJobs (scheduled tasks β see
docs/self-hosting-scheduled-tasks.md) - Backups: CNPG
ScheduledBackup(nightly 02:00 UTC base backup) + continuous WAL archiving to Civo Object Store (UK region), 120-day retention per the Backup & Data Retention Policy; nightly ENCRYPTED Vault raft snapshots (03:45 UTC, 14-snapshot retention β see Vault Operations below) - Deploys: GitHub Actions Deploy (Civo) workflow (pre-deploy Job runs migrations +
idempotent syncs pinned to the same tag, then a
maxUnavailable: 0rolling update gated on/healthz; thedeploy-stagingjob then repeats the sequence against staging β see Staging environment). Migrations must stay expansion/contract β a rolled-back rollout briefly runs the previous image against the new schema.
Staging environment
Staging (staging.checktick.uk) runs in the same checktick namespace as
prod β deliberately, so it shares the namespace-scoped checktick-media +
checktick-snomed-data RWX PVCs (SNOMED arrives free, no re-import) and the
Vault (VAULT_ADDR + AppRole shared; the vault-airgap policy admits
app=checktick-staging-web alongside the prod web pods). Stood up
2026-11-08 β apply order and runbooks in k8s/README.md β
Staging environment:
- Web:
deploy/checktick-staging-webβ 1 replica, no PDB (a single-replica PDB would veto every voluntary drain) and no topology spread (pointless at 1 replica; a staleapp=checktick-webselector would count PROD pods). Deployed by the Deploy (Civo) workflow'sdeploy-stagingjob (staging migrate Job βset imageβ rollout β smoke); manualkubectl set imageafter re-running the staging migrate Job at the same tag remains the fallback.DATABASE_URLcomes from theSTAGING_DATABASE_URLkey ofchecktick-secrets(anenventry overrides the sharedenvFrom);ENVIRONMENT=staginglabels the JSON logs (kept OUT of OpenObserve β see Logs below);SITE_URLcomes fromSTAGING_SITE_URLin the config overlay so emailed links point at staging, never prod. - DB:
checktick-staging-db(CloudNativePG, 1 instance, no ScheduledBackup, no WAL archiving β staging data is disposable) oncivo-volume, with its ownchecktick-staging-db-ingressNetworkPolicy. The staging batch label ischecktick.io/staging-db-client, deliberately NOT the prodchecktick.io/db-clientgrant: prod CronJobs must never reach the staging DB, and staging jobs must never reach the prod DB. - Cron: exactly ONE staging CronJob β
prune-unconfirmed-users-staging(same 02:40 slot as prod, different database).EMAIL_BOUNCE_PROCESSING_ENABLED=falsekeeps the shared prod mailbox out of staging's pruning, and it has no healthchecks.io ping (a staging run must never mark prod's check seen). Re-imaged by thedeploy-stagingjob AFTER the staging migrate Job and web rollout are verified β the prod job's roll-every-CronJob step skips it (ordering: the cron must never run a tag the staging schema has not migrated to). - Seeding:
k8s/staging-migrate-job.yaml(Jobchecktick-staging-predeployβ a distinct name from the prod pre-deploy Job the workflow manages) run once per staging release. - Ingress:
staging.checktick.ukβ servicechecktick-staging-web, in the git-ignoredk8s/ingress.local.yaml(the cert is unchanged since the POC). - SSO: no new credentials β the same Google/Azure client IDs + secrets
are shared (
checktick-config-local+checktick-secrets); the only console-side addition is the redirect URIhttps://staging.checktick.uk/oidc/callback/in both consoles.settings.pyderives the OIDC redirect base fromSITE_URL/STAGING_SITE_URL(previously hardcoded tohttps://checktick.ukin every non-DEBUG env β fixed 2026-11-08, shipped in 0.27.1 and verified live: staging sendshttps://staging.checktick.uk/oidc/callback/). hCaptcha needs no further console-side change for staging: the sitekey's hostname list is organisational in hCaptcha, not an enforcement mechanism (the siteverifyhostnamefield is a statistical metric).checktick.uk/www.checktick.ukandstaging.checktick.uk/www.staging.checktick.ukare listed for hygiene; verification binds to the accountHCAPTCHA_SECRET, andverify_captchaalso sends thesitekeyparameter so a token issued on another sitekey in the same account cannot be redeemed here. Billing:settings.pyselects the GoCardless API byENVIRONMENTβ onlyproductionuses the live API; development AND staging use the sandbox (CTO decision 2026-11-08; the formerDEBUG-based rule sent staging checkout to the live API, since staging runsDEBUG=False). Shipped in 0.27.1 and verified live: staging resolvesPAYMENT_ENVIRONMENT: sandbox. Sandbox checkout needsPAYMENT_PROCESSING_SANDBOX_API_KEY(+PAYMENT_PROCESSING_SANDBOX_WEBHOOK_SECRET) inchecktick-secrets. - Logs: staging stdout is deliberately NOT shipped to OpenObserve β
the collector's
Exclude_Pathdrops every*staging*container log (web, cron and the staging CNPG pod) so the proddefault/defaultstream stays unmixed (CTO decision 2026-11-08: separate streams; a dedicated staging stream was considered and skipped β it would need a second tail input + output on every node). Staging logs stay viewable viakubectl logs; containerd keeps them on the node. The records shipped between staging going live (~11:45 UTC) and the exclusion applying (~12:50 UTC) remain in the stream β purge via the OpenObserve UI (searchkubernetes.pod_namematches*staging*, then Delete) if wanted. - Capacity (measured 2026-11-08): staging adds ~600 Mi steady / 1.5 GiB
requests (web 512 Mi + db 1 Gi) against ~2β2.8 GiB actually free per node
on the 2Γ
g4s.kube.largelayout β comfortable, no resize.
Critical Safety Rules
- Never
vault operator initon migrated/restored Vault storage. A fresh init makes every encrypted survey in PostgreSQL permanently undecryptable. Restore the storage directory first, then unseal with the EXISTING shares. - Vault and PostgreSQL move together or not at all. Never roll back one component alone.
- Major PostgreSQL upgrades go via dump/restore with the TARGET version's
binaries β never
pg_upgradein place, and never dump with newer binaries than the restore target. - Probe rules: never point liveness OR startupProbe at
/healthz. A deep-probed startup gate restart-loops the web pods every 150 s while the Vault is sealed β restarts cannot fix a seal (found live 2026-11-05 when the interim Vault backup's sealed window took the site down; both probes now use the trivial/api/health, which itself 500s if the DB is unreachable β DRF's throttle cache is DB-backed). - A ~2 GiB node is a full node. The smallest node ran primary DB + CNPG
operator + traefik (floating IP) + longhorn + a web pod at ~79% of
requests; adding one small DaemonSet (~150β250 Mi/node) caused
memory-reclaim stalls that liveness-killed EVERY pod on the node in
lockstep, including the primary. Stabilisation that worked: cordon β CNPG
auto-switchover β move Deployment pods off. Verify
kubectl top nodeheadroom before adding any per-node workload (found live 2026-11-05). Standing decision (2026-11-06, CTO β k8s/POC-STATUS.md session 15): pool resize tog4s.kube.largeis DEFERRED while real users number 1β2 ($65β$130/mo not justified; the tolerated risk is availability, never data). All three nodes are back in service (8ciwhuncordoned). Updated 2026-11-07 (CTO): ladder step 1 GO β ONE node tog4s.kube.largevia a second pool + 2Γ medium (~$86/mo) to unblock the OpenObserve log collector (runbook: Node Pool Resize below); 3Γ Large remains the sustained-growth step. Ladder step 1 EXECUTED + REDISTRIBUTED 2026-11-07 (see Node Pool Resize β final steady state ~45%/68%/76%, the primary on the large node after a clean operator-coordinated switchover); the collector DaemonSet stays OFF pending the trimmed-footprint decision. No per-node DaemonSet untilkubectl top nodeheadroom is verified on every node. - Public hostnames are owned by the config overlay via
ALLOWED_HOSTS_EXTRA(k8s/config.env β checktick-config-local), which settings.py appends on top of the in-cluster ALLOWED_HOSTS. The web Deployment's env entry carries only the in-cluster hosts (pod IP + service names), sokubectl apply -f k8s/web.yaml/-k k8s/can never 400 the public site. (Shipped in 0.25.0.) Interim live env patch RETIRED 2026-11-08 β the documented sequence executed: overlay verified βweb.yamlre-applied β both hostnames verified green through the ingress βk8s/scratch-allowhosts-patch.yamldeleted. Two lessons from the retirement: (a) re-applyingweb.yamlagainst a rolled-forward deployment needs the LIVE tag sed-substituted β the manifest pin (0.25.3) vs the last-applied annotation (0.24.2) would have three-way-merged the image BACKWARDS (live was0.27.0); (b) the "apply wipes scratch patches" rule is precise only for patches that modify apply-managed fields (the env VALUE β reverted); affinity patches added viakubectl patchSURVIVE applies (verified live β the web large-pool preference came through the re-apply untouched). Re-applying the affinity patches after a manifest re-apply remains the safe habit.
Node Maintenance (kernel / node-image upgrades)
Proven during the CVE-2026-98432 patch window (2026-11-04):
civo k8s upgradeis IN-PLACE β it swaps the k3s binary only (~4 min) and does not re-image the nodes. One APIServiceUnavailableblip + a transientdegradedhealthz while the k3s daemon bounces. The node image and kernel are unchanged by it.civo k8s recycle CLUSTER --node NODEis the image path: cordon + drain β DELETE the instance β build a fresh node from the current image (~4β8 min each; the PVC reattach dominates). Serial per node β never recycle two at once.- The external IP is a cluster-managed floating IP. When the node holding it is deleted, Civo moves the IP to a surviving node automatically. Across a full 3-node recycle it never changed (no kubeconfig / DNS / Namecheap changes needed) β but verify after every recycle.
- Expect ~2 minutes of 404s after each recycle while the floating IP settles and the new node's traefik syncs its routers β a fresh traefik answers TLS on the IP before it has the ingress routes. Self-heals; do not panic.
- Take a CNPG base backup before each recycle. Draining the primary's node triggers a CNPG switchover (~40 s, healthz degradedβok) β data-safe because the operator coordinates it via each instance's status API (see the NetworkPolicy note below).
- After any node loss, Longhorn RWX volumes serve from one surviving replica
until replenishment completes (600 s
replica-replenishment-wait-interval); both volumes returned to 3 replicas a few minutes after the last recycle.
Node Pool Resize (1 Large + 2 medium β log-collector enabler)
Decision (2026-11-07, CTO): ladder step 1 GO β 3Γ g4s.kube.medium
(~$65/mo) β ONE g4s.kube.large in a second pool + 2Γ medium (~$86/mo),
to create the per-node headroom the log collector DaemonSet needs
(k8s/log-collector.yaml: 128 Mi request / 256 Mi limit per node). The
collector stays OFF until the go/no-go below. civo CLI syntax (v1.5.5):
node-pool ls / instance-ls / create / instance-delete / scale.
- Pre-flight (read-only):
s/k8s-backup-verifygreen;kubectl top nodebaseline;civo kubernetes node-pool ls <cluster>β original pool ID;kubectl get pods -A -o wideβ pick the retire target: a medium hosting neithervault-0(a drain re-seals Vault β 503 + unseal ceremony) nor the CNPG primary (else accept the ~40 s switchover blip, rehearsed 2026-11-04). Take a fresh CNPG base backup, as before each recycle. - Add the large node:
civo kubernetes node-pool create <cluster> --name large -n 1 -s g4s.kube.large; waitkubectl get nodesReady. The new node joins schedulable (Civo pools carry no taints). β civo CLI β€ v1.5.5 PANICS on--nameunder 6 chars (slice bounds out of range [:6]) β AFTER the API call succeeds, so the pool IS created. Verify withcivo kubernetes node-pool ls <cluster>before retrying, or a duplicate pool bills twice (found live 2026-11-07 β a retry after the panic created a second large pool; deleted viacivo kubernetes node-pool delete <cluster> <pool-name> -y). Upgrade the CLI or use a β₯6-char pool name to avoid the panic. - Rebalance: cordon + drain the chosen medium (
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data) β its pods land on the large node; Longhorn rebuilds replicas (600 s replenishment interval). - Retire it:
civo kubernetes node-pool instance-delete <cluster> -p <original-pool-id> -i <instance-id>(alternative:node-pool scale <cluster> <pool-id> -n 2β check withinstance-lswhich instance each path removes first). Verify the floating IP settled on a surviving node (~2 min of 404s expected):curl -sf https://checktick.uk/healthz. β 2026-11-07 pre-flight found the resize target is ALSO the floating-IP holder AND the node that restart-storms on every web rollout (99% memory) β retire it promptly after the drain; do not leave a cordoned 4th node running. - Post-change verification (standing rules):
s/k8s-backup-verifyagain; Longhorn volumes back to 3 replicas;/healthzfully ok. - GO/NO-GO for the collector:
kubectl top nodeβ every node needs β₯ ~400 Mi headroom (the 2026-11-05 rule: the collector adds ~150β250 Mi/node). NO-GO β use its footprint trims (β header ink8s/log-collector.yaml) or defer to 3Γ Large. - GO:
kubectl apply -f k8s/log-collector.yaml(out-of-band β neverapply -k k8s/),kubectl -n checktick rollout status ds/log-collector, then port-forward OpenObserve (5080:5080) and confirm live logs land in streamdefault/default(search the last 5 minutes). Watchkubectl top nodeovernight β rollback is deleting the DaemonSet (the 2026-11-05 stabiliser that worked). - Record: Β§2c row in the Infrastructure Technical Change
Log with measured
timings; flip the collector wording here (Monitoring, backlog item 4)
and the β header in
k8s/log-collector.yamlto applied-live.
Executed 2026-11-07 (measured timings): pre-flight green (backup chain +
fresh base backup pre-pool-resize-20261107 17:47:48Z). Retire target
β¦-b966-8ciwh (floating-IP holder + CNPG replica β the only node hosting
neither vault-0 nor the primary): cordoned + drained 17:48, instance
deleted 17:49, floating IP re-announced on the large node automatically;
CNPG 2/2 throughout (no switchover β primary untouched); backup chain
re-verified green. The 404 window ran ~4β5 min (17:49β17:54), NOT the ~2
min the recycle path predicts: the drain evicts the Longhorn RWX
share-manager for checktick-media too β both web pods died on the NFS
blip and their replacements stalled Init:0/2 on FailedMount β¦
DeadlineExceeded until the share-manager rebuilt on the large node.
All three volumes returned to healthy robustness by ~18:05. Post-resize
layout defeated the resize's intent: requests-based packing + the web
anti-affinity put ONE web pod on EACH medium β 88%/88% working set while
the large node idles at ~52β55% β collector go/no-go FAILED. Options
on the table were (a) pin OpenObserve to the large node + nudge one web
pod there (β mediums ~70β75%, collector GO), (b) 3Γ large at sustained
growth (~$162/mo), (c) hold. Also caught: civo CLI β€ v1.5.5 panics on
--name
under 6 chars AFTER the API call succeeds β verify with node-pool ls
before retrying or you double-bill a duplicate pool.
Redistribution (same day, ~18:25β18:40 UTC) β option (a) executed:
git-ignored scratch patches (the scratch-allowhosts pattern β any
apply -k k8s/ WIPES them; re-apply after):
- k8s/scratch-cronpool-affinity-patch.yaml β ALL 14 CronJobs hard-pinned
to kubernetes.civo.com/civo-node-pool=large (every batch Django boot
lands on the roomy node; mediums never see cron spikes)
- k8s/scratch-openobserve-affinity-patch.yaml β OpenObserve β large
- k8s/scratch-web-affinity-patch.yaml β web soft-PREFERS large
(weight 100; the hard topologySpreadConstraints in web.yaml still
force one replica per node, so steady state is always large + one
medium β the preference only picks WHICH nodes win)
During the web rollout the boot spike on the primary's medium tripped
db-1's status probe β CNPG operator coordinated a CLEAN switchover to
db-2 (exit 0, ~40 s, data-safe β the rehearsed 04/11 path; WAL chain
unbroken, backup chain re-verified green): the primary now runs on the
large node. Final placement: large = web + primary + OpenObserve +
crons (~45%), f985n = web + vault (~68%), hwp2f = db-1 replica +
operator + tax (~76%). Lesson: web replica moves are deterministic ONLY
while the primary's node stays cordoned until the replacement is Ready
(a timed uncordon let one replacement land back on it).
Collector verdict at this layout: the TRIMMED footprint (~100 Mi/node)
takes the mediums to ~79β80% β the edge of the 11-05 storm band β and the
full footprint does not fit. Keep it OFF until watched overnight, or GO
trimmed + overnight kubectl top node watch + rollback
(kubectl delete ds) ready.
Redesign to 2Γ large (executed 08/11, ~09:30β10:30 UTC): the 1L+2M
layout still left the mediums at 76β89% β the CTO approved 2Γ
g4s.kube.large at the same ~$87/mo. Executed: pool large2 created
(CLI quirk: node-pool scale rejects the 5-char large pool ID β
create a β₯6-char pool instead), hwp2f retired (zero impact), f985n
retired with the rehearsed Vault re-seal ceremony, medium pool deleted.
INCIDENT: civo kubernetes node-pool delete for the medium pool ALSO
deleted the large pool (third CLI ID bug β the empty pool name in
the response was the tell); Civo tore down mbuc5 (repeated re-cordons
+ pod deletions); recovered via pool large3 (~5 min to a replacement
node; all HA pairs rescheduled; floating IP auto-moved; no data loss β
DB + Vault are on civo-volume, not Longhorn). CLI RULE: verify with
node-pool ls before AND after every civo pool operation; prefer the
Civo dashboard for pool deletions; three CLI ID bugs are now documented
(create panic on <6-char names, scale rejecting <6-char IDs, delete
hitting the wrong pool). CLI upgraded to v1.6.0 on 09/11 β the
wrong-pool delete did NOT reproduce there (v1.5.5 remains the
suspected version; its empty-parens response is the tell). Also this window: cilium-operator lease
patched 15sβ90s (item 12); Longhorn volumes set to 2 replicas (2-node
reality); web re-spread one-per-node after a rollout left both replicas
collapsed on one node (topology spread is evaluated at scheduling time,
not end-state β check placement after every web rollout).
Overnight 07β08/11: a ~06:05 UTC blip liveness-killed BOTH web pods
(their replacements coin-flipped again β the large-pool preference
cannot help the SECOND replica, the spread constraint forbids it) and one
landed back on hwp2f β 91% β 07:15/07:30 readiness blips on the fresh
pod (DB connect EPERM, then a cluster-wide DNS outage β coredns was a
SINGLE replica on hwp2f; when that node stalls, in-cluster AND
external resolution fail everywhere; the 07:30 "Vault is sealed" healthz
line was a FALSE ALARM from this β Vault stayed unsealed throughout).
Fixes: web re-moved off hwp2f (cordon dance), coredns scaled to 2
replicas (hwp2f + mbuc5). Standing hazard: after ANY web pod churn,
check placement (kubectl get pods -l app=checktick-web -o wide) β the
second replica coin-flips between the mediums and hwp2f cannot host it.
Morning 08/11 β collector apply + rollback + root-cause fix (found
live): overnight BOTH RWX volume engines (media + SNOMED) + their
share-managers drifted from the large node onto f985n (+~250 Mi β
82%), which liveness-killed its web pod at 08:38 β BEFORE any collector.
The trimmed collector then passed the headroom gate (all nodes β₯ ~400 Mi
free) and shipped 526,717 records / ~324 MB to OpenObserve in ~20 min β
but at 89% f985n killed the web pod again, so the documented rollback
fired. Fix executed 09:00 UTC: both engines moved back to the large
node (cordon both mediums β delete the two share-manager pods β they
reschedule to the large node β uncordon; ~3 min user-visible NFS/404
window, web pods replaced cleanly, healthz 200, volumes back to healthy
robustness, backup chain re-verified green). Collector re-applied
09:06 UTC β LIVE and shipping; steady state large 53% / f985n 82% /
hwp2f 78%. The tail DBs are hostPath-persistent, so the backlog drain
RESUMED from 11-05 21:35 (no duplicate re-ship of the 526,717 records
already ingested); live logs appear in the stream once the remaining
~2.5-day backlog drains (~30 min). Overnight watch + rollback
(kubectl -n checktick delete ds log-collector) per the runbook.
CNPG (PostgreSQL) Operations
- The db-ingress NetworkPolicy must allow the CNPG operator to reach the
instances on port 8000 (status API) as well as 5432. Without it:
Instance Status Extraction Erroron the Cluster andCannot extract Pod status ... :8000: i/o timeoutevery ~4 s in the operator log β and draining the primary's node STALLS against thechecktick-db-primaryPDB. - Verify the backup chain after ANY intervention (node recycles, operator
restarts, schedule changes β this caught two real outages). The runner is
s/k8s-backup-verify(read-only, exits non-zero on any failed check); it performs all of the following: kubectl get scheduledbackups -n checktickβ schedule armed,nextScheduleTimein the futurekubectl get backups -n checktickβ every row must be terminal (completed/failed). A stuck non-terminal Backup silently blocks the whole schedule (delete it to unblock)- Check the scheduled backup's status for a dead pod IP β a backup pinned to a deleted pod's IP never runs
ContinuousArchiving=Trueon the Cluster; WAL chain unbroken in the object store- Before any CNPG operator upgrade past 1.30: migrate backups from the in-tree Barman object store to the Barman Cloud Plugin β in-tree support is removed in 1.31 and the operator emits a deprecation warning on every reconcile.
- Restore path: latest base backup + WAL archive from Civo Object Store into a
fresh CNPG cluster. Step-by-step procedure: Quarterly Restore
Rehearsal, run by
s/k8s-restore-rehearsal; results feeddocs/compliance/restoration-test-log.md. - Post-restore Vault ceremony (generate-root, fixed-policy verification,
AppRole mint, token revoke):
s/k8s-vault-ceremony(drill pod by default; production requires an explicit--pod vault-0+ confirmation).
Vault Operations
- After node loss,
vault-0's storage reattaches in ~5.5 min and the pod comes upInitialized: true, Sealed: trueβ storage survives; do not re-init. Unseal with 3-of-4 production shares (s/k8s-vault-unsealβ hidden prompts only, shares never appear on a command line); healthz is fully ok ~10 s later. - Storage backup: nightly ENCRYPTED raft snapshots β LIVE since 2026-11-06.
A 03:45 UTC CronJob (
k8s/vault-snapshot.yaml, applied out-of-band) logs in via a least-privilege AppRole (sys/storage/raft/snapshotonly β no unseal shares ever touch a scheduled job), runsvault operator raft snapshot save, wraps the snapshot withageusing the operator's public key, and uploads tos3://checktick-backups/vault/(retain 14). The stored object needs BOTH the Shamir shares AND the operator's age private key (Bitwarden). The restore chain is proven end-to-end β CT-03 inrestoration-test-log.md(2026-11-06). The pre-migration alternative (stop-copy-restart on a cron) would put unseal shares on a scheduled job and breaks the custody model β do not use it. - 2026-11-08 incident (found + fixed same morning): the deploy
workflow's roll-every-CronJob step set the out-of-band
vault-snapshotjob'suploadcontainer to the app image (no aws CLI) β the 03:45 nightly failedaws: not found(both backoff attempts; no upload, no HC ping β healthchecks.io alerts on the missed ping). Live image restored; manual re-run completed 08:06 UTC (vault-raft-2026-11-08T080625.tar.age);deploy-civo.ymlnow skipsvault-snapshotin the roll. The init containers (hashicorp/vault, alpine) were untouched β the roll only coverscontainers, which is why the job reached the upload step before dying. RECURRENCE 08/11 13:19 UTC, found + fixed within the hour: the skip's case pattern (cronjob/vault-snapshot) never matchedkubectl get cronjobs -o nameoutput (cronjob.batch/vault-snapshotβ resource.group format), so the skip silently no-op'd. The 0.27.1 deploy β the FIRST since the skip was added β rolleduploadto the app image again. Live image restored toamazon/aws-cli:latestbefore the 03:45 run; both skip patterns fixed to globs (*/vault-snapshot,*/prune-unconfirmed-users-staging). Lesson: a skip that has never fired is untested β grep the roll step's log for theSkippinglines on the first deploy after adding one. tls_disable = truein-cluster is the migration baseline; the backlog adds cert-manager TLS on the in-cluster endpoint +VAULT_TLS_VERIFY=true.- The Vault container image defaults to root β pin
runAsUser: 100(backlog).
Monitoring & Alerting
- StatusCake (5-minute polling) on
https://checktick.uk/healthzβ returns 503 if Vault is sealed, the database is unreachable, or SNOMED is misconfigured. - healthchecks.io per-CronJob pings: each CronJob pings its
HC_PING_*capability URL only on success (command && curl ...), so a failed or non-running job triggers an email alert. The URLs live in the git-ignoredk8s/config.env(checktick-config-localConfigMap) and never enter the repo. Unset = no ping (self-hosters are unaffected). - OpenObserve (self-hosted,
k8s/openobserve.yaml): log retention layer for the app's structured stdout JSON (Infrastructure Logging). ClusterIP only β reach the UI with./s/k8s-forward(orkubectl -n checktick port-forward svc/openobserve 5080:5080); credentials in theopenobserve-authSecret (seek8s/openobserve.example.env). Orgdefault/ streamdefaultcarries the exported Northflank-era archive (June 2026 week β the in-app exporter only ran that long) and would carry live logs; 180-day stream retention per the Logging & Audit Policy. The collector DaemonSet (k8s/log-collector.yaml): applied 2026-11-08 with the trimmed footprint, rolled back the same morning β it shipped 526,717 records / ~324 MB in ~20 min (proof the pipeline works), but f985n had drifted ~250 Mi heavier overnight (both RWX engines + share-managers landed there) and liveness-killed its web pod with the collector aboard β the documented rollback (kubectl -n checktick delete ds log-collector) fired. Re-apply after the engine move (Node Pool Resize). It stays out ofkustomization.yamlsokubectl apply -k k8s/can never silently recreate it. The Platform Admin log-viewer API (HOSTING_*) is an unrelated opt-in feature.
OpenObserve log queries (cut-and-paste)
Access: ./s/k8s-forward port-forwards svc/openobserve 5080 to
localhost. In the Logs page select stream default, switch the editor
to SQL mode, paste. Saved views are per-user metadata in
OpenObserve's internal DB on the openobserve-data PVC (covered by the
nightly Longhorn backup; not a k8s object) β an ERR_CONNECTION_REFUSED
toast mid-session just means the tunnel dropped: rerun
./s/k8s-forward, hard-refresh the tab, retry.
One timeline, three record shapes:
- Northflank archive (June 2026 week, seeded into the data dir):
NF shipper schema β
dt/entity/host/job; no app fields. - Civo, pre-fix (everything the collector shipped before the
2026-11-08 parser fix, ~15:24 UTC): the whole CRI line sits flat in
log, no app fields. Immutable β ages out via retention; exclude withAND message IS NOT NULL. - Civo, current (since the fix): the CRI parser +
Merge_Logunwrap the app JSON into top-level fields βlevel,logger,message,request_id,user_id,remote_addr,service(django),environment(production),module,funcName,line.
Scope: the collector ships EVERY container log except its own, the CNPG
checktick-db pods and *staging* β kube-system (coredns, traefik,
metrics-serverβ¦) and the CNPG operator are in the stream too. Filter by
kubernetes_container_name: web = Django app container, run =
every CronJob (same JSON shape), media-chown / snomed-copy = init
containers, vault / openobserve = infra.
Django web logs β time, level, message:
SELECT _timestamp, level, message
FROM "default"
WHERE kubernetes_container_name = 'web'
ORDER BY _timestamp DESC
LIMIT 200
Errors only β the 3am view:
SELECT _timestamp, level, logger, message
FROM "default"
WHERE kubernetes_container_name = 'web'
AND level IN ('ERROR', 'CRITICAL')
ORDER BY _timestamp DESC
LIMIT 200
One request end-to-end, any container, chronological:
SELECT _timestamp, kubernetes_container_name, level, logger, message
FROM "default"
WHERE request_id = '<request_id>'
ORDER BY _timestamp ASC
All Django JSON including CronJobs (service comes from
LoggingContextFilter):
SELECT _timestamp, kubernetes_container_name, level, message
FROM "default"
WHERE service = 'django'
ORDER BY _timestamp DESC
LIMIT 200
Volume by level (set the UI time range first):
SELECT level, count(*) AS n
FROM "default"
WHERE kubernetes_container_name = 'web'
GROUP BY level ORDER BY n DESC
Ingestion health check β newest records in the stream. _timestamp is
microseconds since epoch (date -r $(( TS / 1000000 )) converts on
macOS). If the newest is stale while pods are logging, the collector is
down β the query is fine:
SELECT _timestamp, kubernetes_container_name
FROM "default"
ORDER BY _timestamp DESC
LIMIT 5
Hardening Backlog
Moved from the migration runbook's Phase 6 list; in rough priority order
(ALLOWED_HOSTS_EXTRA shipped in 0.25.0 and is off the list):
- ~~Vault storage backup β Raft migration + nightly snapshot CronJob~~
DONE 2026-11-06 β migration executed, snapshot pipeline live, and
the restore test PASSED (CT-03,
restoration-test-log.md) - Barman Cloud Plugin migration before any CNPG operator upgrade past 1.30
- TLS on the in-cluster Vault endpoint (cert-manager) +
VAULT_TLS_VERIFY=true - ~~OpenObserve + collector DaemonSet for infrastructure log retention~~ β
DONE 2026-11-08: OpenObserve deployed (2026-11-05, NF-era history
restored); collector LIVE since 09:06 UTC 2026-11-08 (trimmed profile,
after the resize + the engine-move fix β Node Pool Resize above); see
the β note in
k8s/log-collector.yaml - Least-privilege ServiceAccount kubeconfig for the deploy workflow (today it is cluster-admin)
- Longhorn recurring snapshots on
checktick-media; Velero/PVC backups - k3s Secrets encryption-at-rest (verify feasibility); non-root Vault
(
runAsUser: 100); staging namespace separation; alerting on/healthz - Open SNOMED readers as
file:...?mode=ro&immutable=1(needs an image release; non-urgent given the init-container conversion) - Quarterly restore rehearsal β step-by-step procedure in docs/restore-rehearsal.md
Open Items from the Retired Migration Records (2026-11-06)
The migration session records (k8s/POC-STATUS.md, the NorthflankβCivo runbook,
docs/vault-raft-migration.md) were retired at close-out β the durable posture
lives in this document and the compliance folder, full history in git. Items
they left open, carried here so nothing is lost with them:
- Vault rekey β
vault operator rekeyon the migrated Vault (Shamir 3-of-4, operator ceremony; also decide the share medium) + revoke the drill API token. Nothing records it as done; keys were minted pre-migration. - Clinical-safety docs review β
docs/clinical-safety/*still carry Northflank-era live-state claims ("hosted on Northflank infrastructure", "Northflank SLA 99.9%") that are now factually wrong; assessment records, so they need a deliberate review pass, not an automated sweep. - OpenObserve root password is still the Northflank-era credential β rotate
in the UI (
kubectl -n checktick port-forward svc/openobserve 5080:5080) and update theopenobserve-authSecret. - Graph email 403s β ROOT-CAUSED 2026-11-08 (probe via
k8s/scratch-graph-diag.py, git-ignored):Mail.SendIS granted (token roles claim verified) and plain sends from theadmin@checktick.ukmailbox succeed (HTTP 202). The 403 isErrorSendAsDenied: Django'sAdminEmailHandlersends error mail fromSERVER_EMAIL(server@checktick.uk), the backend then sets an explicit Graphfrom, andserver@is NOT an alias of the sending mailbox β so every error-alert email is rejected while all user-facing mail (fromDEFAULT_FROM_EMAIL= admin@) works. Fix (M365 admin, ~30 s): addserver@checktick.ukas an alias of the admin@ mailbox, or setSERVER_EMAIL=admin@checktick.ukink8s/config.envβ recreate thechecktick-config-localConfigMap. Verify by re-running the probe (expect probe B β 202). **VERIFIED 2026-11-08: alias added β probe B VERIFIED 2026-11-08: alias added β probe B 202 β the Django error-email alerting path is live again; revisit item 5 (the ERROR-email storm) before the next outage.Mail.Read(application) consented + verified same day β token roles['Mail.Read', 'Mail.Send'],fetch_bounce_messagesreads the inbox (0 NDRs in the 24h window, as expected for current traffic). Gotcha found on the way: the first add was the DELEGATEDMail.Readβ invisible to the client-credentials flow, which only sees APPLICATION permissions (hence "admin consent required: No" with nothing to grant). Remaining: the Exchange Online ApplicationAccessPolicy scopingMail.Readto the sending mailbox (see access-control.md Β§12). - AdminEmailHandler emails on every ERROR log β an outage (a healthz 503
loop) turns into an outbound-email storm that amplifies load. FIX
WRITTEN 2026-11-08 (ships with the next release):
AdminEmailRateLimitFilter(checktick_app/core/logging/ admin_email_rate_limit.py) β one admin email per 15 min window (envADMIN_EMAIL_RATE_LIMIT_SECONDS, 0 = off), mounted on themail_adminshandler only; dropped records still reach stdout/ OpenObserve. Until the release: filter them in the mail client. - Traefik access logging is OFF β no request-level logs, only internal CRD-watch noise; a monitoring gap worth a deliberate decision.
s/k8s-vault-unsealstdin piping is broken on this kubectl ("Unseal submission failed") β use the interactive formkubectl -n checktick exec -it vault-0 -- vault operator unseal; port the captured-output verification pattern (see the ceremony scripts) into it before relying on the non-interactive path.- Nothing schedules
s/k8s-backup-verifyβ a stuck non-terminal Backup silently blocks the nightly schedule (the 2026-11-04 outage); the runner is read-only and exit-coded, so wire it to cron/CI for the "phase != completed after 02:10 UTC" check, and keep running it after any cluster intervention. - The raft peer registers as
127.0.0.1:8201β the migrate helper's loopbackcluster_addr; harmless while single-node, revisit if a raft peer is ever added. logs.checktick.ukis a STALE Northflank-era DNS record (found 2026-11-07): the Namecheap CNAME still points atβ¦dns.northflank.appβ a Northflank LB returning 503 β it has been dead since the 2026-11-03 cutover, it does NOT route to Civo. The OpenObserve instance on Civo is healthy and ClusterIP-only BY DESIGN (port-forward to reach it). Action: delete the DNS record β or, if a public logs UI is ever wanted, that is a deliberate decision requiring an ingress route + cert AND rotating the still-Northflank-era root password (item 3) first.LOGS_BASE_URL/LOGS_ORGANIZATION/LOGS_STREAM_NAMEare stale Northflank-era values (found 2026-11-07):http://openobservelogs:5080(no such service on Civo β it isopenobserve) and org/streamprod_logs(the Civo instance servesdefault/default). Dormant β noHOSTING_*keys are set, so the Platform Admin log-viewer feature is off β but updatek8s/config.envbefore ever enabling it.- cilium-operator CrashLoopBackOff β ROOT CAUSE of the 08/11 DNS
blips + email storm (pre-existing since 04/11): both replicas
fatal with
"Leader election lost"(75/83 restarts by 09:20 UTC, 08/11) β the lease (cilium-operator-resource-lock) hasleaseDurationSeconds: 15and renewals land at the 14 s edge, so any API latency spike kills the holder. Millisecond correlation with app failures: operator stop 09:11:10.520Z β first web-pod DNS failure 09:11:10.863Z; each crash cycle blips in-cluster DNS/ service routing for ~1β2 min (cilium v1.18.4, in-cluster API endpoint 192.168.1.2:6443). cnpg-controller-manager (~119 restarts) and civo-ccm (~86) churn the same way β all lease-based, pointing at intermittent API-server latency from the worker nodes. Lease reset attempted 09:16 UTC (deleted the resource-lock; re-acquired, then lost again). ACTION: Civo support ticket (managed control plane + CNI) β evidence bundle in the ops session log 08/11; interim: the item-5 email rate-limit makes the blips quiet. INTERIM FIX APPLIED 08/11 09:53 UTC: cilium-operator leader-election lease patched 15sβ90s (--leader-election-lease-duration=90s --leader-election-renew-deadline=60s --leader-election-retry-period=5son the kube-system deployment β Civo-managed, may be reverted by their reconciliation; re-apply if the churn returns). Operators stable since (0 restarts vs a crash every 5β15 min); the DNS blips and their error-email storms stopped. The Civo ticket should cover BOTH the lease root cause AND the CLI's wrong-pool delete (see Node Pool Resize). RESOLVED 09/11 (Civo support): root cause was an UNDERSIZED CONTROL PLANE. Civo updated the control-plane spec (~03:30 UTC 09/11 β visible as a single bounce of every controller) and the lease thrash stopped: 15+ h of ZERO restarts across the patched cilium operators AND the unpatched cnpg-controller-manager / civo-ccm (previously every 10β30 min), zero DNS blips since, first night's scheduled jobs all green. The 90s lease patch is now redundant but harmless β keep it as belt-and-braces (Civo reconciliation may revert it; either way is fine). CLI upgraded to v1.6.0: the wrong-pool delete was v1.5.5 β its response printed an EMPTY pool name ("The node pool () has been deleted"), unlike 1.6.0's correct output; Civo could not reproduce on 1.6.0.
Related
k8s/README.mdβ manifest walkthrough and validation commandsdocs/self-hosting-scheduled-tasks.mdβ CronJob referencedocs/infrastructure-logging.mdβ structured loggingdocs/versioning-and-deployment.mdβ release flow.github/workflows/deploy-civo.ymlβ deploy pipeline