Budget Reservations — Alerts Runbook (ADR-0001 §7)

Operator reference for the durable budget reservation observability: metrics (§7.2), the reconcile worker's invariant checks (§7.1.3/§7.1.4), the settlement-lag watchdog (§5.4) and the D17 shadow-exit gate inputs (§7.3). The rules below are copy-paste ready for deploy/prometheus/alert-rules.yml.

Sources:

• loggerd reconcile worker (pkg/budget/reconcile.go, component budget-reconcile) — releases lapsed escrow (§7.1.1/§7.1.2), exports the invariant gauges and the settlement lag on every pass (default every 5 min). It runs wherever the settlement worker runs; when loggerd is down nothing is released (§5.4, conservative) — that blind spot is exactly what the settlement-lag alert watches.
• gateway admission layer (pkg/budget/admission.go, component budget-admission) — grant/admit outcomes, added latency and bound violations (§4.2).

1. Metric reference (§7.2)

Series	Type	Producer	Meaning
budget_reserved_micro_usd{scope_kind}	gauge	reconcile	Capacity frozen by reservations per hard scope (wallet/team_month/agent_day/session).
budget_allowance_released_total{reason}	counter	reconcile	Releases: expired_safety_window (replica died, lease lapsed past 2 h) / zero_remainder (draining/expired, nothing to return).
budget_reserved_divergence_micro_usd{scope_kind}	gauge	reconcile	§7.1.3 breach: Σ|reserved(row) − Σ remainders of the row's non-released allowances|. Money-conserving paths keep this at 0.
budget_wallet_headroom_micro_usd	gauge	reconcile	Worst-case §7.1.4 LHS across orgs: min(balance − remainder − floor − reserved).
budget_wallet_invariant_violations	gauge	reconcile	Org count with negative §7.1.4 headroom (orgs named in loggerd logs, never in labels).
budget_stranded_micro_usd	gauge	reconcile	Capacity frozen by lapsed non-released allowances (§12): dead replicas block money up to 2 h + TTL.
settlement_lag_seconds	gauge	reconcile	Age of the oldest unsettled settlement job (pending/retry/processing).
budget_grants_total{scope_kind,result}	counter	gateway	granted / insufficient / no_source / error (store outage).
budget_grant_duration_seconds	histogram	gateway	PG grant latency.
budget_local_admit_total{result}	counter	gateway	How Reserve funded a request: local / grant / refused.
budget_admission_overhead_seconds	histogram	gateway	Synchronous admission latency added by the layer (Plan + Reserve). D17 gate: p95 ≤ 5 ms.
budget_bound_violation_total{scope_kind}	counter	gateway	Finalized requests with actual > bound (§4.2) — the only legitimate overspend source.
budget_bound_overshoot_micro_usd_total{scope_kind}	counter	gateway	Accumulated (actual − bound)+ in µUSD.
gateway_reservations_would_deny_total{scope_kind}	counter	gateway (NEXT-18c)	ADR's budget_would_deny_total: refusals observed in shadow (D17 exit-report input).
gateway_reservations_denied_total{scope_kind}	counter	gateway (NEXT-18c)	Refusals applied in enforce (402/429/403).
gateway_would_bill_partial_micro_usd_total{scope_kind}	counter	settlement (NEXT-18d)	ADR's would_bill_partial_micro_usd_total: partial-usage cost enforce would bill, observed in shadow.
gateway_uncharged_unknown_total{scope_kind}	counter	settlement (NEXT-18d)	Reserved bound of unknown-usage events — never billed, visible here.
gateway_settlement_batch_size	histogram	settlement (NEXT-18d)	Jobs per org-batch settlement TX.
gateway_settlement_jobs_backlog{state}	gauge	settlement (AF-06)	Backlog by state; requires_attention is durable debt.

No org/key/session identifiers ever appear in labels (cardinality and privacy rule AF-16); per-org values are the admin API's job.

2. Alert rules

BudgetSettlementLagHigh — warning → critical (§5.4)

- alert: BudgetSettlementLagHigh
  expr: settlement_lag_seconds > 1800
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Settlement lag above 30 minutes"
    description: "Oldest unsettled settlement job is {{ $value | humanizeDuration }} old."

§5.4: lag > 30 min means the settlement worker is stalled. While it stalls, reconcile conservatively releases nothing (the reserve stays frozen — safe direction); a stalled worker also delays billing. Escalate to critical after 30 min (for: 30m clone) — a double outage (dead replica + stalled loggerd beyond the 2 h safety window) can temporarily overstate available capacity by the unaccounted spend of the dead replica.

Action: check gateway_settlement_jobs_backlog{state=~"pending|retry|processing"} and settlement worker logs; restart loggerd if the worker is down. Reconcile resumes automatically; no manual release is ever needed.

BudgetScopeReserveDivergence — critical (§7.1.3)

- alert: BudgetScopeReserveDivergence
  expr: max by (scope_kind) (budget_reserved_divergence_micro_usd) > 0
  for: 10m
  labels:
    severity: critical
  annotations:
    summary: "Reserved-money invariant divergence ({{ $labels.scope_kind }})"
    description: "reserved_µ(S) ≠ Σ remainders of non-released allowances; {{ $value }} µUSD unexplained."

The money invariant reserved_µ(S) = Σ remainders of non-released allowances holds on every code path (grant/return/settle/release commit under the per-org advisory lock). Any non-zero value sustained for 10 min is drift: a bug or manual tampering. Release paths re-check under the lock, so reconcile never widens a divergence; releasing "over" a drift freezes capacity in the safe direction and logs reconcile reserve decrement failed; releasing anyway.

Action: grep loggerd logs for scope reserve invariant divergence and reconcile reserve decrement failed (rows carry scope/key), then budget_allowances vs the scope row's reserved_micro_usd for the named key. Escalate to the billing-owner; do not hand-edit reserves — fix the diverging writer and re-run reconcile.

BudgetWalletInvariantBreach — critical (§7.1.4)

- alert: BudgetWalletInvariantBreach
  expr: budget_wallet_invariant_violations > 0
  for: 15m
  labels:
    severity: critical
  annotations:
    summary: "Wallet invariant breached on {{ $value }} org(s)"
    description: "balance − floor − reserved < 0 sustained beyond the bound-overshoot window (§7.1.4)."

Transient dips are legal: a bound overshoot (estimation error, §4.2) can pull balance − floor − reserved below zero until the overshoot settles and reconcile clears the escrow — the §7.1.4 window. 15 min of continuous violation exceeds any normal settle latency and means either overspend beyond recorded overshoot or reserve accounting drift.

Action: cross-check increase(budget_bound_overshoot_micro_usd_total[30m]) against budget_wallet_headroom_micro_usd; find the org in the loggerd log (wallet invariant violation (§7.1.4), carries org id). If covered by overshoot: bound estimation for that model needs review (see D17 inputs). If not: treat as reserve drift (see the divergence alert).

BudgetWalletHeadroomNegative — warning (precursor)

- alert: BudgetWalletHeadroomNegative
  expr: budget_wallet_headroom_micro_usd < 0
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Worst-case wallet headroom is negative"

Early signal of the breach above (one or more orgs inside the transient window). Watch, do not page.

BudgetStrandedCapacityHigh — warning (§12)

- alert: BudgetStrandedCapacityHigh
  expr: budget_stranded_micro_usd > 10000000
  for: 30m
  labels:
    severity: warning
  annotations:
    summary: "Stranded reservation capacity {{ $value | humanize }}µUSD"
    description: "Dead/lost replicas keep freezing capacity (released after TTL 10m + safety 2h)."

Action: identify dead replicas (budget_allowances non-released rows with lease_until < now(); replica_id names host:boot), restart or drain them; reconcile releases automatically after the safety window. Sustained stranded capacity with no dead replicas → renewal loop broken (check gateway logs reservation renewal failed).

BudgetExpiredReleaseSpike — warning (replicas dying)

- alert: BudgetExpiredReleaseSpike
  expr: increase(budget_allowance_released_total{reason="expired_safety_window"}[1h]) > 10
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Multiple allowances released past the safety window"
    description: "Replicas are dying (or losing renewal) without graceful drain."

Action: correlate with deploys/OOM kills of gateway replicas. A gentle background rate is normal for autoscaling; a spike is a stability signal.

BoundViolationRateHigh — D17 gate monitor (§7.3)

- alert: BoundViolationRateHigh
  expr: >
    sum(increase(budget_bound_violation_total[7d]))
      / clamp_min(sum(increase(budget_local_admit_total{result=~"local|grant"}[7d])), 1) > 0.001
  for: 1h
  labels:
    severity: warning
  annotations:
    summary: "Bound-violation rate above 0.1% of reserved requests (D17 gate)"

D17 shadow-exit requires ≤ 0.1% for 7 consecutive days. This alert does not page for money (overshoot is absorbed by grace, and budget_bound_overshoot_micro_usd_total sizes it) — it blocks the shadow exit report while firing.

Action: pull the bound-estimate fields from request events (in_est, out_cap, model) for the violating models; review InputMargin/default_output_cap_tokens (§4.3) before re-attempting exit.

AdmissionOverheadP95High — D17 gate monitor (§7.3)

- alert: AdmissionOverheadP95High
  expr: histogram_quantile(0.95, sum by (le) (rate(budget_admission_overhead_seconds_bucket[1h]))) > 0.005
  for: 30m
  labels:
    severity: warning
  annotations:
    summary: "Reservation admission p95 overhead above 5ms (D17 gate)"

Action: check budget_grant_duration_seconds p95 first (grants are the PG round trip inside the overhead) and PG health; admission only adds latency on the reserve path (cache miss), so a rise with stable grants points at the org-settings/probe caches (5 s TTL) being cold.

ReservationGrantErrors — warning (§5.4)

- alert: ReservationGrantErrors
  expr: sum(rate(budget_grants_total{result="error"}[10m])) > 0
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "Reservation store errors during grant"
    description: "PG unreachable for grants — enforce orgs get 503 budget_store_unavailable once local escrow drains (D9/D14)."

Action: restore PG connectivity. result="insufficient" and result="no_source" are normal refusals (exhaustion / missing budget row), not outages.

3. D17 shadow-exit report inputs (§7.3)

For the org under shadow, 7 consecutive days of:

1. sum(increase(budget_bound_violation_total[7d])) / sum(increase(budget_local_admit_total{result=~"local|grant"}[7d])) ≤ 0.001;
2. cumulative overshoot increase(budget_bound_overshoot_micro_usd_total[7d]) ≤ org's |grace_floor| (per-org value via admin API);
3. p95 budget_admission_overhead_seconds ≤ 0.005;
4. gateway_reservations_would_deny_total and gateway_would_bill_partial_micro_usd_total broken down and every reason dispositioned (unpriced models priced or consciously denied).

4. Notes

• The reconcile pass interval (5 min default) quantizes all reconcile-side gauges; for: durations in the rules above already account for it.
• Retention of released allowances defaults to 90 days (same horizon as the settled settlement-jobs retention, AF-16). It deletes only terminal released rows in bounded batches; live states are never touched.
• Global kill switch YABANIN_RESERVATIONS=off stops new reservations (mode clamp, §9); existing allocations drain via reconcile.