Skip to main content

On-prem upgrade runbook

Pre-flight​

  1. Confirm license valid for new version: curl gateway:8082/healthz.
  2. Disk space on backup PVC ≥ 2× DB size.
  3. Notify users of maintenance window.

Upgrade​

helm upgrade yabanin deploy/helm/yabanin-onprem \
    -f deploy/helm/yabanin-onprem/values-airgap.yaml \
    --set image.tag=1.1.0

The chart auto-runs:

  1. pre-upgrade Job: pg_dump → /backup
  2. rolling-update gateway/loggerd/admin (zero downtime with replicas≥2)
  3. post-upgrade Job: YABANIN_REGION=onprem yabanin-migrate

Rollback (if migration failed)​

helm rollback yabanin

The post-rollback Job restores the latest /backup dump.

Migration discipline​

  • New columns must be nullable/defaulted (backward-compat during rolling update).
  • Destructive changes wrap IF EXISTS / IF NOT EXISTS.
  • Never rename a column in one step; add new, dual-write, backfill, drop old.

Automated pre-upgrade drill (ENT-06)​

Before touching a real deployment, run the offline drill:

bash scripts/airgap/upgrade_drill.sh \
    "postgres://yabanin:…@localhost:5432/yabanin?sslmode=disable" /tmp/upgrade-drill

It proves, without touching the application:

  1. Migration mirror — migrations/ and pkg/migrate/migrations/ carry the identical set (a divergence means the standalone migrate binary would drift from the unified binary).
  2. Pre-upgrade backup — ENT-05 backup_all.sh (RPO artifact) with sha256.
  3. Migrate up — yabanin-migrate built fresh from source; a no-op on an up-to-date schema, applies pending files otherwise.
  4. Offline license activation — full PKI round-trip (gen-root → issue-intermediate → issue → verify): LICENSE VALID proves air-gapped activation without any network.

Latest local evidence (2026-09-13): all four steps green, backup 152,416 B, migrate no-op, license chain verified.

Known failure mode: stale migrate binary​

A yabanin-migrate binary built from a pre-squash tree embeds the old 37-file catalog and writes a bogus schema_migrations (37, dirty) record (observed in the 2026-09-13 drill). The drill therefore builds the migrate binary from current source into the work directory. Repair of a corrupted tracking table: bring the schema to the true applied set, then UPDATE schema_migrations SET version=<head>, dirty=false.

Known failure mode: manual psql migration​

Databases migrated by piping .sql files manually carry no schema_migrations row — the runner then treats them as unmigrated and fails on the first object. Always migrate through yabanin-migrate.