Skip to main content

DR Drill Runbook

Executable local drill (ENT-05, automated)​

The drill runs in the integration gate as go test -tags=integration -run TestDR_BackupWipeRestoreVerify ./tests/integration/dr/:

  1. Disposable Postgres (testcontainers) migrates with the production migrations/*.up.sql set.
  2. Audit rows are seeded through the production auditlog.Writer.
  3. pg_dump --format=custom produces the backup (RPO artifact).
  4. The schema is dropped — the "disaster".
  5. pg_restore rebuilds it (RTO window).
  6. Production auditlog.Verifier re-walks the hash-chain and row counts.

Latest measured evidence (local scale, 2026-09-13): backup 149,719 B in 121 ms; restore + hash-chain verify in 202 ms; chain intact (0 broken seq). Shell helpers for host-side runs against a live database: scripts/dr/backup_all.sh, scripts/dr/restore_all.sh (fail-closed: refuse to restore into a non-empty database). RPO in production equals the backup schedule of the host cron that invokes backup_all.sh.

Purpose​

Quarterly disaster recovery drill to verify backup/restore procedures, measure RTO/RPO, and produce evidence for SOC 2 Type II and ISO 27001 compliance.

Pre-Drill Checklist​

  • Full backup taken (scripts/dr/backup_all.sh)
  • Fresh cluster provisioned (kind/eks with name dr-test-YYYYMMDD)
  • All on-call engineers notified of drill window
  • Status page maintenance window posted

Drill Procedure​

Step 1: Record Start Time​

echo "DRILL_START=$(date -u +%Y-%m-%dT%H:%M:%SZ)" >> dr-evidence.md

Step 2: Simulate Disaster​

  • Declare "production cluster unavailable" (do NOT actually destroy production).
  • In a separate terminal, record time until first on-call page.

Step 3: Provision Fresh Cluster​

# For cloud: create new EKS cluster in same region
aws eks create-cluster --name dr-test-$(date +%Y%m%d) --region $(REGION)

# For local: kind create cluster --name dr-test-$(date +%Y%m%d)
kind create cluster --name dr-test-$(date +%Y%m%d)

Step 4: Restore from Backup​

scripts/dr/restore_all.sh --backup-ts <BACKUP_TS>

Step 5: Deploy Platform​

helm install yabanin deploy/helm/yabanin-onprem \
    -f deploy/helm/yabanin-onprem/values-airgap.yaml \
    --set image.tag=<latest-tag>

Step 6: Verify Service​

kubectl -n yabanin rollout status deploy/yabanin-gateway
curl -sf http://<node-ip>:30082/healthz        # expect {"status":"ok"}
curl -sf http://<node-ip>:30082/healthz/license # expect {"active":true}

Step 7: Verify Data Integrity​

  • Audit-log hash-chain: Run pkg/auditlog/verifier.Verify() — should return 0 (intact).
  • Lineage attestations: Query request_lineage table — data should be intact.
  • Model registry: Check model entries in models_registry.

Step 8: Record End Time​

echo "DRILL_END=$(date -u +%Y-%m-%dT%H:%M:%SZ)" >> dr-evidence.md

Evidence Recording​

Append to the internal DR evidence log (private/docs/compliance/dr-evidence.md locally):

DateRTORPOIssuesSign-off
2026-Q3TBD minTBD minNone/Issues listedCTO Name

Target Metrics​

  • RTO (Recovery Time Objective): < 60 minutes from disaster declaration to service availability.
  • RPO (Recovery Point Objective): < 1 hour of data loss (time between last backup and "disaster" point).

Post-Drill Actions​

  1. Review any issues found during the drill.
  2. Update runbooks if procedures were unclear or failed.
  3. Schedule next quarterly drill date.
  4. Share results with CTO and Head of Security.