Skip to main content

Gateway alerts runbook

The runbook_url annotation of every alert in deploy/prometheus/alert-rules.yml points at a section of this file (anchor = alert name). The scripts/quality/alert_check.py checker verifies that metrics used in expressions are declared in pkg/metrics, that runbook anchors exist, and that the SLO metrics (latency/error/queue) are present.

Test-firing owner channels (webhook + Telegram)​

Owner delivery is wired in deploy/alertmanager/alertmanager.yml: the default route goes to the generic webhook receiver (ALERT_WEBHOOK_URL), severity=" critical" alerts fan out to a receiver that carries BOTH the webhook and the Telegram bot (ALERT_TG_BOT_TOKEN / ALERT_TG_CHAT_ID). All three are env vars rendered into the config by deploy/docker-compose.prod.yml at start (placeholder + sed); unset values stay on dud black-holes so the stack still boots.

Fire a synthetic critical alert and expect it in the Telegram chat and the webhook within ~1 minute (group_wait 10s):

curl -sS -X POST http://127.0.0.1:9093/api/v2/alerts \
  -H 'Content-Type: application/json' -d '[{
    "labels": {
      "alertname": "TestFireCritical",
      "severity": "critical",
      "job": "manual-test"
    },
    "annotations": {
      "summary": "Manual delivery test — ignore",
      "runbook_url": "https://example.com/test"
    }
  }]'

Verify:

  • Telegram: the bot posts to the configured chat (and a "resolved" message ~5 min later, resolve_timeout).
  • Webhook: one POST per receiver hit.
  • Config errors or send failures: docker compose -f deploy/docker-compose.prod.yml logs alertmanager (send failures show as telegram/api or webhook errors with 401/400 — usually a wrong token/chat id, or the dud black-hole defaults).
  • Config changes require a container restart (the file is rendered at start).

A severity: "warning" test alert exercises the default webhook-only route — useful to confirm critical-vs-warning routing.

HighGatewayLatencyP99​

P99 latency above 100 ms. Look at: which models/providers top gateway_http_request_duration_seconds_bucket; check upstream health (UpstreamDown) and the circuit breaker. Action: move the problem model with a rule/tier change, verify network reachability of the provider.

HighGatewayErrorRate​

Above 5% 5xx over 5 minutes. Look at gateway_http_requests_total by path/status and the gateway logs. Common causes: provider 401 (key), 429 (quota — see cooldown), upstream 5xx (fallback should have fired — see HighFallbackRate).

UpstreamDown​

An upstream has been unhealthy for over 1 minute. Check gateway_upstream_health_state per url; curl the provider with the key from env. Action: disable the model (enabled=false) or raise its fallback priority.

CircuitBreakerOpen​

The breaker is open: the upstream fails systematically. gateway_circuit_breaker_*. Action: do not "reset" the breaker — fix the upstream; temporarily take the model out of rotation.

HighFallbackRate​

Above 20% of requests go to fallback. The primary channel is unstable. Check the latency/errors of the primary model; the fallback model may be more expensive — monitor the budget impact.

LowClassifierAccuracy​

The hard share is above 30% — suspiciously many complex requests (or traffic drift). Check gateway_classifier_class_total by source; recalibrate.

DiskSpaceLow​

Under 10% free on the partition. Clean wal/logs; check retention.

HighDeductionFailureRate​

More than 0.5 deductions/sec are skipped for insufficient funds. Check organization balances (top negative), possibly sudden credit exhaustion (see the overshoot measurements, NEXT-17).

HighGatewayErrorRateV2​

Stricter threshold (10%). Handle as HighGatewayErrorRate.

ServiceDown​

A scrape target is unreachable. Check the service container/process and the availability of /metrics (Prometheus bearer key).

SLO​

  • Latency: gateway_http_request_duration_seconds (histogram).
  • Errors: gateway_http_requests_total{status=~"5.."}.
  • loggerd queue: gateway_loggerd_events_in_flight / events_consumed.

ShadowHighErrorRate​

Shadow mirroring shows a high error rate (above 50% over 15 minutes while shadow traffic flows). Shadow never affects clients (traffic is mirrored), but the comparison data becomes unreliable. Check yabanin_shadow_*: availability of the mirrored upstream, its key.

ShadowBudgetBurn​

The shadow experiment budget is exhausted (shadow_budget_exhausted_total is growing). Action: stop or narrow the shadow share, or top up the experiment budget — see the shadow configuration.