Skip to main content

Yabanin Production Runbook

Architecture Overview​

                    ┌─────────────────────────────────────────┐
                    │              Internet / CDN              │
                    └────────────────┬────────────────────────┘
                                     │
                    ┌────────────────▼────────────────────────┐
                    │        Gateway (:8082)                   │
                    │  - Auth middleware (API key → org/team)  │
                    │  - Cache lookup (Redis + pgvector)       │
                    │  - Classifier routing (ML + rule-tier)   │
                    │  - Budget enforcement                  │
                    │  - Upstream proxy (OpenAI/Anthropic)     │
                    └──┬──────┬──────┬──────┬──────────────────┘
                       │      │      │      │
            ┌──────────▼┐  ┌──▼──┐ ┌▼────────┐ ┌──────────────┐
            │ PostgreSQL │  │Redis│ │Classifier│ │ BGE Embedder │
            │  (:5432)   │  │(:6379)│ (:8080) │ │   (:6006)    │
            │  pgvector  │  │     │ │  ML V1  │ │              │
            └─────┬─────┘  └──┬──┘ └─────────┘ └──────────────┘
                  │           │
            ┌─────▼─────┐    │ Redis Stream: events:queue
            │  Admin    │    │
            │ (:8086)   │◄───┘
            │ Dashboard │         ┌──────────────┐
            │ (:3000)   │         │   Loggerd    │
            └───────────┘         │   (:8085)    │
                                │ DB writer    │
                                └──────────────┘

Services​

ServicePortPurposeHealth Check
Gateway8082Main API proxy (OpenAI-compatible)GET /healthz
Admin8086Admin panel + REST APIGET /health
Loggerd8085Redis stream → PostgreSQL writerGET /healthz
PostgreSQL5432Primary database (pgvector)pg_isready
Redis6379Cache + event streamredis-cli ping
Classifier8080ML task complexity (bge-small)GET /health
BGE Embedder6006Embedding service for semantic cacheGET /health
Dashboard3000React cost analytics dashboardHTTP 200

Deployment​

Initial Deployment​

# 1. Create secrets (one-time setup)
cd deploy/secrets
echo "postgresql://yabanin:SECRET@postgres:5432/yabanin?sslmode=disable" > pg_uri
openssl rand -hex 32 > session_key
openssl rand -hex 32 > cache_hmac

# 2. Build and start all services
cd deploy/
docker compose up -d

# 3. Verify migrations ran
docker compose logs migrate | tail -20

# 4. Verify all services healthy
docker compose ps
curl http://localhost:8082/healthz
curl http://localhost:8086/health

Updating a Service​

# Rebuild and restart a single service
docker compose up -d --build gateway

# Zero-downtime rolling restart (for config-only changes)
docker compose restart gateway

Rollback​

# Roll back to previous image
docker compose down gateway
docker tag yabanin-gateway:latest yabanin-gateway:rollback
docker tag yabanin-gateway:previous yabanin-gateway:latest
docker compose up -d gateway

# Database rollback (if migration caused issues)
# CAUTION: only run .down.sql if you understand the consequences
psql -h localhost -U yabanin -d yabanin -f migrations/0029_team_budgets.down.sql

Environment Variables​

Gateway​

VariableRequiredDescription
YABANIN_GATEWAY_LISTENYesListen address (:8082)
YABANIN_PG_URI_FILEYesPath to PG connection string secret
YABANIN_REDIS_ADDRYesRedis address (redis:6379)
YABANIN_EVENT_STREAMYesRedis stream name (events:queue)
YABANIN_CACHE_HMAC_SECRET_FILEYesPath to HMAC signing secret
YABANIN_EMBEDDER_BASE_URLNoEmbedder service URL
YABANIN_EMBEDDER_ENABLEDNoEnable semantic cache (true/false)

Admin​

VariableRequiredDescription
YABANIN_PG_URI_FILEYesPath to PG connection string
YABANIN_SESSION_KEY_FILEYesPath to session signing key
YABANIN_ADMIN_LISTENYesListen address (:8086)
YABANIN_ADMIN_DOMAINYesDomain for cookie (localhost)
YABANIN_ADMIN_API_KEYNoEnables read-only /internal service API (see below)

Admin: Internal service API​

  • Disabled by default: without YABANIN_ADMIN_API_KEY there is no standalone /internal surface.
  • Endpoints: GET /internal/api-keys?org_id=<id>&page=1&page_size=50, GET /internal/analytics/savings?org_id=<id>&days=7.
  • Auth: constant-time header check; send the key as X-Admin-Key: <value> (trcli uses --api-key).
  • Read-only + org-scoped: both routes require org_id and expose no mutation operations.

Scaling​

Vertical (increase CPU/memory)​

# In docker-compose.yml, under deploy.resources:
deploy:
  resources:
    limits:
      memory: 512M  # was 256M
      cpus: "2.0"   # was 1.0

Horizontal (add replicas)​

Gateway and loggerd are stateless — safe to scale horizontally:

docker compose up -d --scale gateway=3 --scale loggerd=2

Put a load balancer (nginx/HAProxy) in front of the gateway replicas.

Redis Scaling​

  • Read replicas: Add Redis replicas for read-heavy workloads (cache lookups)
  • Memory: Monitor used_memory — increase container memory limit before OOM
  • Persistence: appendonly yes is enabled for durability; monitor AOF rewrite frequency

PostgreSQL Scaling​

  • Connection pooling: Use PgBouncer if gateway connections exceed max_connections
  • pgvector: Monitor HNSW index size — reindex if query latency degrades
  • Read replicas: Add for analytics/dashboard queries (admin service)

Incident Response​

P0: Gateway Down (all requests failing)​

# 1. Check gateway health
curl http://localhost:8082/healthz

# 2. Check recent logs
docker compose logs gateway --tail=100

# 3. Check dependencies
docker compose ps  # Are postgres/redis/classifier healthy?

# 4. Quick restart
docker compose restart gateway

# 5. If still failing, check DB connectivity
docker compose exec gateway wget -qO- http://postgres:5432 || echo "DB unreachable"

# 6. If Redis is down, gateway can still serve (cache miss fallback)
#    but loggerd cannot write to DB. Fix Redis first.

P1: High Latency (p95 > 2s)​

# 1. Check upstream provider latency
docker compose logs gateway --tail=50 | grep "duration"

# 2. Check Redis latency
docker compose exec redis redis-cli --latency

# 3. Check DB connection pool
docker compose exec postgres psql -U yabanin -c \
  "SELECT count(*), state FROM pg_stat_activity GROUP BY state;"

# 4. Check if classifier is slow (>100ms timeout)
docker compose logs gateway --tail=50 | grep "classifier"

# 5. Mitigation: disable classifier temporarily (fallback to rule-tier)
curl -X PATCH -H "Authorization: Bearer $ADMIN_TOKEN" \
  http://localhost:8086/api/organizations/$ORG_ID/routing \
  -d '{"routing_mode": "always_cheap"}'

P1: Classifier Service Down​

# Gateway automatically falls back to rule-tier classifier (circuit breaker)
# Verify fallback is working:
docker compose logs gateway --tail=50 | grep "circuit.*open"

# Check classifier health
curl http://localhost:8080/health

# Restart classifier
docker compose restart classifier

# Verify recovery
docker compose logs gateway --tail=50 | grep "circuit.*closed"

P1: Database Connection Exhausted​

# Check active connections
docker compose exec postgres psql -U yabanin -c \
  "SELECT count(*), state, application_name FROM pg_stat_activity GROUP BY state, application_name;"

# If idle connections are high, restart services to release them
docker compose restart gateway admin

# Increase max_connections if needed
docker compose exec postgres psql -U yabanin -c "SHOW max_connections;"
# Edit postgresql.conf or set in docker-compose env

P2: Redis Memory Full​

# Check memory usage
docker compose exec redis redis-cli INFO memory | grep used_memory_human

# Check evicted keys
docker compose exec redis redis-cli INFO stats | grep evicted_keys

# Flush cache if needed (will cause temporary cache miss spike)
docker compose exec redis redis-cli FLUSHDB
# Or flush specific patterns:
docker compose exec redis redis-cli --scan --pattern "cache:*" | xargs redis-cli DEL

P2: Loggerd Falling Behind (stream backlog)​

# Check stream length
docker compose exec redis redis-cli XLEN events:queue

# Check consumer group lag
docker compose exec redis redis-cli XINFO GROUPS events:queue

# Scale loggerd
docker compose up -d --scale loggerd=3

# If backlog is huge, trim old messages
docker compose exec redis redis-cli XTRIM events:queue MAXLEN 100000

Monitoring​

Key Metrics (Prometheus at :2112)​

MetricAlert Threshold
http_request_duration_seconds{quantile="0.95"}> 2s for 5m
cache_hits_total{type="miss"} rate> 80% for 15m
circuit_breaker_state1 (open) for 5m
redis_stream_length> 10,000
pg_connections_active> 80% of max

Grafana Dashboard​

Available at deploy/grafana/dashboards/ — import via Grafana UI.

Key panels:

  • Request volume + latency percentiles
  • Cache hit rate (exact vs semantic vs miss)
  • Classifier distribution (easy/medium/hard)
  • Cost savings (actual vs counterfactual)
  • Upstream error rate by provider

Alerting caveat (read this)​

deploy/alertmanager/alertmanager.yml ships with receivers commented out (SMTP/Slack/PagerDuty are TODO placeholders). Until a real receiver/webhook is configured, NO alerts will be delivered to anyone — Alertmanager will evaluate rules and route them to the file-log receiver only. Wire at least one real receiver before relying on alerting in production.

Metrics exposure caveat​

metrics_enabled: true serves /metrics on the admin listener (:9090). By default there is no authentication — an intentional trade-off so Prometheus can scrape it. Set metrics_bearer_key (env YABANIN_METRICS_BEARER_KEY) to require Authorization: Bearer <key> instead; note the check is on the bearer token only, with no localhost restriction, since scrapers usually sit in another pod/host. Prefer the key when the admin port cannot be fully network-restricted. The metrics carry model names, org ids, and per-route traffic volumes, which is operational data you may not want exposed to a segment that reaches the port. In production, restrict it at the network level: Kubernetes NetworkPolicy or a firewall rule scoped to the cluster's Prometheus pod/metric range, and do NOT publish the admin port on the host in compose deployments. Keep scraping inside the cluster (PodMonitor / ServiceMonitor), never from outside it.

Gated pprof​

When debug_pprof: true, the gateway admin listener exposes five profiles — /debug/pprof/ (index), cmdline, profile, symbol, trace — all behind the same gate as /debug/vars: localhost callers only with Authorization: Bearer <YABANIN_API_KEY>. Capture a 30s CPU profile:

curl -H "Authorization: Bearer $YABANIN_API_KEY"
"http://127.0.0.1:9090/debug/pprof/profile?seconds=30" -o cpu.pprof go tool pprof cpu.pprof

The named profile endpoints (goroutine, heap, block, mutex, threadcreate) are deliberately NOT exposed on the admin listener; use the index page to pick what is available, and take heap/goroutine data from SIGQUIT dumps or a sidecar if needed. Every rejection returns 403 without a stack trace (both non-localhost callers and bad bearer tokens).

Backup and Recovery​

Database Backup​

# Full backup (run daily via cron)
docker compose exec postgres pg_dump -U yabanin yabanin | gzip > backup_$(date +%Y%m%d).sql.gz

# Restore
gunzip -c backup_20260723.sql.gz | docker compose exec -T postgres psql -U yabanin yabanin

Redis Backup​

# AOF persistence is enabled (appendonly yes)
# Manual snapshot:
docker compose exec redis redis-cli BGSAVE
# Copy dump.rdb from volume
docker compose cp redis:/data/dump.rdb ./redis_backup_$(date +%Y%m%d).rdb

Model Checkpoint Backup​

# Classifier checkpoint
cp artifacts/classifier-v1.pt backups/classifier-v1-$(date +%Y%m%d).pt

Source Backup (git bundle)​

# Local bundle of ALL branches + tags into .backups/ (default retention: last 5)
make backup
# Options: BACKUP_DIR=/other/path RETAIN=10 make backup
# Restore/clone from a bundle:
git clone .backups/yabanin-<stamp>.bundle

No remote or credentials are used; bundles are gitignored (.backups/). Copy the bundle off-box (e.g. to encrypted storage) to get a real off-machine backup.

Common Operations​

Add a New Organization​

curl -X POST -H "Authorization: Bearer $ADMIN_TOKEN" \
  http://localhost:8086/api/organizations \
  -d '{"name": "Acme Corp", "slug": "acme"}'

Rotate API Key​

curl -X POST -H "Authorization: Bearer $ADMIN_TOKEN" \
  http://localhost:8086/api/api-keys/{id}/rotate

Configure Spend Alert​

curl -X PATCH -H "Authorization: Bearer $ADMIN_TOKEN" \
  http://localhost:8086/api/organizations/{id}/spend-alert \
  -d '{"threshold_usd": 100.0, "webhook_url": "https://hooks.slack.com/..."}'

Set Team Budget​

curl -X PUT -H "Authorization: Bearer $ADMIN_TOKEN" \
  http://localhost:8086/api/teams/{team_id}/budget \
  -d '{"monthly_limit_usd": 500.0, "hard_limit": true}'

Retrain Classifier​

# 1. Generate new dataset
docker run --rm -v $(pwd)/datasets:/app/datasets yabanin-train \
  python scripts/build_dataset.py --output /app/datasets/classifier/

# 2. Train
docker run --rm \
  -v $(pwd)/datasets:/app/datasets \
  -v $(pwd)/artifacts:/output \
  yabanin-train \
  --train /app/datasets/classifier/prompts_train.jsonl \
  --test /app/datasets/classifier/prompts_test.jsonl \
  --output /output/classifier-v2.pt \
  --epochs 5

# 3. Evaluate
docker run --rm --entrypoint python \
  -v $(pwd)/artifacts:/app/artifacts \
  -v $(pwd)/datasets:/app/datasets \
  -v $(pwd)/scripts:/app/scripts \
  yabanin-train \
  /app/scripts/evaluate_classifier.py \
  --checkpoint /app/artifacts/classifier-v2.pt

# 4. Deploy new model
docker compose build classifier
docker compose up -d classifier

On-Call Quick Reference​

SymptomFirst ActionEscalate If
5xx errors spikingCheck gateway logs, restart if neededPersists >5 min
Latency p95 > 2sCheck upstream + Redis latencyPersists >10 min
Cache hit rate < 25%See cache runbookPersists >30 min
Classifier errorsCircuit breaker auto-fallbackAll classifiers down
Budget rejections spikeCheck if limits are misconfiguredOrgs locked out
DB disk fullVacuum + archive old logs< 10% free space
Redis OOMFlush cache + add memoryRecurring