Skip to main content

Runbook: backup restore rehearsal (compose stack)

Quarterly, ~20 minutes. Proves that the daily pg_dump produced by the backup compose sidecar can actually rebuild the database, and produces the RTO/RPO numbers for the compliance evidence table. The automated in-CI version of this drill is go test -tags=integration -run TestDR_BackupWipeRestoreVerify ./tests/integration/dr/ (see docs/runbooks/dr-drill.md); this runbook is the hands-on doc-test against a live compose deployment.

What produces the backups​

The backup service in deploy/docker-compose.prod.yml runs scripts/dr/backup_compose.sh daily at 02:00 UTC into the backup_data volume (/data/backups), custom format (pg_dump -F c), gzipped, 7-day retention.

Step 1 — take a fresh backup​

cd deploy
docker compose -f docker-compose.prod.yml exec backup /bin/sh /app/backup.sh
docker compose -f docker-compose.prod.yml exec backup ls -lh /data/backups
# e.g. pg_20261005_021500.dump.gz

Step 2 — copy the dump off the volume​

DUMP=pg_20261005_021500.dump.gz   # pick the newest
docker compose -f docker-compose.prod.yml cp backup:/data/backups/$DUMP /tmp/$DUMP
ls -lh /tmp/$DUMP                 # record the size for the evidence table

Step 3 — restore into a disposable Postgres​

Never restore into the live pg_data volume. Use a throwaway container:

docker run -d --name restore-drill -e POSTGRES_PASSWORD=drill \
  -e POSTGRES_DB=yabanin_drill postgres:16-alpine
sleep 5                                        # let it initialize
docker cp /tmp/$DUMP restore-drill:/tmp/$DUMP
gunzip -c /tmp/$DUMP > /tmp/$DUMP.raw          # custom format: gunzip first
START=$(date +%s)
docker exec restore-drill psql -U postgres -c 'SELECT count(*) FROM information_schema.tables WHERE table_schema='"'"'public'"'"''
# ^ must print 0 — the drill target is empty (same fail-closed check as
#   scripts/dr/restore.sh; refuse to continue otherwise)
docker exec restore-drill pg_restore -U postgres --no-owner --no-acl \
  --exit-on-error -d yabanin_drill /tmp/$DUMP.raw
echo "restore took $(( $(date +%s) - START ))s"   # RTO core component

Step 4 — verify the restored data​

docker exec restore-drill psql -U postgres -d yabanin_drill -c \
  "SELECT count(*) AS orgs FROM organizations;
   SELECT count(*) AS keys  FROM api_keys;
   SELECT count(*) AS reqs  FROM request_logs;"

Expect org/key counts to match the live system (docker compose -f docker-compose.prod.yml exec postgres psql -U yabanin -d yabanin -c "<same queries>"). request_logs lags by the backup age — that delta IS the RPO.

Integrity checks beyond row counts:

  • Audit hash chain: the production verifier runs in the automated drill (pkg/auditlog/verifier, step 6 of docs/runbooks/dr-drill.md); for the manual rehearsal, spot-check that the newest admin_audit_logs sequences are contiguous:

    docker exec restore-drill psql -U postgres -d yabanin_drill -c \
      "SELECT max(seq), count(DISTINCT seq) FROM admin_audit_logs;"

    (max(seq) == count(DISTINCT seq) means no gaps; chain-hash verification stays with the automated drill.)

Step 5 — record evidence​

Append to the team DR log:

DateDump sizeRestore timeRPO (backup age)IssuesSign-off
2026-Q4

Targets (from docs/runbooks/dr-drill.md): RTO < 60 min end-to-end, RPO < 24 h for the daily sidecar (tighten by adding host-cron scripts/dr/backup_all.sh runs if needed).

Step 6 — cleanup​

docker rm -f restore-drill
rm -f /tmp/$DUMP /tmp/$DUMP.raw

The dump contains customer data — delete it from the rehearsal host.

If the restore fails​

  1. --exit-on-error stops at the first bad item — read the first error, not the last.
  2. Common cause: dump from a newer migration set restored into an older Postgres — use the same major version (postgres:16-alpine).
  3. If backups have been failing silently, check the sidecar log: docker compose -f docker-compose.prod.yml logs backup (cron output goes to the container log; the script is fail-fast via set -euo pipefail).
  4. A real incident restore follows the same commands against the (recreated) live volume — fail-closed wrapper for host-side runs: scripts/dr/restore_all.sh --backup-ts <TS>.