Operations Guide

Overview

Yabanin is a three-service system:

Service	Role	Default Port
gateway	HTTP API proxy + routing	8082 / 9090
admin	Admin panel + analytics API	8086
loggerd	Event ingestion → Postgres writer	8085

Module: github.com/reireirei/yabanin
Go version: 1.25.0
Build: go build ./... from the monorepo root.

---

Service Endpoints

Gateway (pkg/gateway/gateway.go)

Address	Purpose
:8082	HTTP API (forwarding proxy)
:9090	Admin server (metrics, tracing)

• Health: GET http://localhost:8082/healthz — returns JSON with Redis connectivity status.
• Metrics: GET http://localhost:9090/metrics — Prometheus exposition format.
• Tracing: OTLP/HTTP exporter. Configured via environment variables (see below).

Admin Service (services/admin/cmd/main.go)

Address	Purpose
:8086	HTTP API (admin + analytics)

• Routes: /auth/*, /api/analytics/*
• Health: GET http://localhost:8086/health — returns "ok" with 200.
• Metrics: GET http://localhost:8086/metrics — Prometheus exposition format.
• Port override: Set via listen_addr configuration key.

Loggerd Service (services/loggerd/cmd/main.go)

Address	Purpose
:8085	HTTP API (event ingestion)

• Health: GET http://localhost:8085/healthz — returns 200 OK.
• Tracing: OTLP/HTTP exporter. Configured via environment variables (see below).

---

Health Checks

Service	Endpoint	Response	Dependencies Checked
Gateway	GET /healthz	JSON: {"status":"ok","redis":{"status":"up"}} or "down"	Redis connectivity
Admin	GET /health	Plaintext: ok	None
Loggerd	GET /healthz	HTTP 200 OK	None

Gateway Health Detail

The gateway health endpoint performs a PING to Redis. The response includes:

{
  "status": "ok",
  "redis": {
    "status": "up"
  }
}

If Redis is unreachable, the redis.status field becomes "down" and the overall status reflects a degraded state. This is useful for Kubernetes liveness/readiness probes or load balancer health checks.

---

Metrics

All metrics are registered under subsystem gateway via Prometheus client-go (github.com/prometheus/client_golang/prometheus). They are exposed in standard Prometheus exposition format at each service's /metrics endpoint.

Request Metrics (Gateway)

Metric	Type	Labels	Description
gateway_requests_total	Counter	method, path, status	Total HTTP requests handled
gateway_request_duration_seconds	Histogram	method, path, status	Request latency in seconds
gateway_upstream_first_token_seconds	Histogram	provider, model	Time to first streamed token (SSE requests only; 0.005s–10s buckets)
gateway_upstream_requests_total	Counter	provider, model, status	Upstream requests after routing
gateway_retry_total	Counter	attempt, status	Intra-model retries (transport/429/5xx) per attempt number
gateway_fallback_total	Counter	from, to, reason	Fallback-chain hop completions
cache_hits_total	Counter	type, org_id	Response-cache lookups by type (exact/semantic/miss, incl. misses)

The first-token histogram and the retry/fallback counters back the first_token_ms, attempt_count and retry_count columns stored in request_logs (migration 0011) and surfaced in the admin request explorer (/api/organizations/{id}/requests[/stats]).

Loggerd Metrics

Metric	Type	Labels	Description
gateway_events_in_flight	Gauge	(none)	Number of messages currently being processed
gateway_events_consumed_total	Counter	event_type	Total events successfully consumed

Event types for events_consumed_total:

• request.completed — Successful proxy request logged
• request.failed — Failed proxy request logged
• Any other string from the event_type field in the Redis stream (generic events)

Circuit Breaker Metrics (Gateway)

Metric	Type	Labels	Description
gateway_circuit_state	Gauge	(none)	Current circuit breaker state (see below)
gateway_circuit_failures	Counter	(none)	Total circuit breaker tripped events
gateway_circuit_recoveries	Counter	(none)	Total circuit breaker recovery events

Circuit breaker states (encoded as integer values on the gauge):

• 0 — CLOSED: Normal operation, requests flow through.
• 1 — OPEN: Circuit is tripped; requests are rejected without forwarding.
• 2 — HALF_OPEN: Probe request allowed; if successful, transitions to CLOSED.

---

Tracing

Both the Gateway and Loggerd services support distributed tracing via OpenTelemetry.

Configuration

Environment Variable	Description	Default
YABANIN_OTEL_ENDPOINT	OTLP/HTTP endpoint URL (e.g., http://localhost:4318)	(disabled)
YABANIN_OTEL_SAMPLE_RATE	Trace sampling rate, float in [0.0, 1.0]	1.0

Exporter

• Protocol: OTLP/HTTP (not gRPC)
• The tracing package (pkg/tracing) initializes an otlptracehttp exporter and registers a global TracerProvider.
• Span sampling is applied at the provider level using the configured sample rate.

Usage in Code

Services call the tracing initialization function at startup:

// Gateway (pkg/gateway/gateway.go)
if err := tracing.Init(ctx, logger); err != nil {
    // log warning but do not fail startup
}

// Loggerd (services/loggerd/cmd/main.go)
if err := tracing.Init(ctx, logger); err != nil {
    // log warning but do not fail startup
}

The Admin service does not have tracing instrumentation.

---

Response Cache Behavior

Lookup order

1. Redis exact — SHA-256 of the request body (key includes org + model).
2. pg exact (opt-in, YABANIN_CACHE_PG_EXACT_LOOKUP) — Postgres fallback after Redis loss.
3. Semantic — pgvector ANN search over cache_entries embeddings.

Clients may bypass lookup and storage per request with Cache-Control: no-store.

Similarity semantics

pgvector <=> is cosine distance (1 − similarity, range 0..2). The gateway reports X-Cache-Similarity as cosine similarity = 1 − distance (range −1..1); an entry is a hit only when similarity ≥ threshold (default 0.95, so distance ≤ 0.05). Exact hits report 1.0000. Junk prompts must never score above the threshold — an X-Cache-Similarity value greater than 1 in logs/metrics indicates a build older than the distance-inversion fix.

Streaming clients

Cache hits for stream: true requests are replayed as canonical chat.completion.chunk SSE frames (Content-Type: text/event-stream, data: ... lines, data: [DONE] terminator) — OpenAI-compatible SDKs see a normal stream. Cache hits carry X-Cache: EXACT|PG_EXACT|SEMANTIC.

What gets stored from a streamed response: the first parseable completion frame, with the usage object from the final frame carried over. A malformed cache entry is treated as a miss (the request is served fresh), never replayed broken.

Savings semantics

request_logs.counterfactual_cost_usd is the per-row savings baseline and depends on how the request was served:

Request class	Counterfactual baseline
Cache hit	Priced usage of the cached completion (what the request would have cost upstream); 0 when usage is unknown
Tier-routed (classifier served)	Priced usage at the model the client requested (routing_decision.from)
Everything else	The actual spend (no savings claimed without an optimization)

Consequences for interpreting the dashboards:

• Savings percentages are (SUM(counterfactual) − SUM(actual)) / SUM(counterfactual); rows with unknown baselines contribute 0/0 — they dilute toward 0 rather than fabricating savings.
• Negative routing savings are meaningful: the classifier routed to a pricier model than the client requested.
• Tier routing (and therefore routing savings) requires the org's routing_mode to be something other than disabled.
• Lens differs by screen: the savings report groups top models by model_requested (per-request outcome — routed spend lands on the model the client asked for), the cost report groups by model_used (spend attribution to the model that actually served).

Grouping requests into tasks

Requests are grouped into logical tasks by the caller: send the X-Yabanin-Task header (format [A-Za-z0-9._:-]{1,128}, invalid values are rejected with 400; YABANIN_TASK sets a deployment-level default for headerless callers). Details and per-commit trailers: docs/agent-governance.md.

Where to analyze a task group: the Tasks screen (per-task cost / counterfactual / savings, CSV export) and the Request Explorer's Task filter (scopes the table and the stats strip to one task; the scope is part of the URL, so a filtered view is shareable).

Tool calls

Two views of the tool/function calls models request:

• per request — the Explorer inspector parses the stored response sample (non-streaming completions and streamed SSE samples alike; delta fragments are reassembled into one readable call). Arguments render as stored; privacy mode masks them.
• across requests — the Tool calls screen aggregates the tool_calls table loggerd populates from the same samples: totals, top tools with share and last-used, per-model usage (org-scoped, 30-day window). Rows exist only for parsed calls: truncated or unparseable samples read as absence, never as zero-usage. Historical streamed samples (pre-table) were not backfilled and stay inspector-only.

---

Configuration Variables

Environment Variables

Variable	Service	Description	Default
YABANIN_OTEL_ENDPOINT	Gateway, Loggerd	OTLP/HTTP trace exporter endpoint	(disabled)
YABANIN_OTEL_SAMPLE_RATE	Gateway, Loggerd	Trace sampling rate (0.0–1.0)	1.0
LISTEN_ADDR	Admin	HTTP listen address (e.g., :8086)	:8086

Redis Streams Configuration (Loggerd)

Constant	Value	Description
consumerGroup	loggerd-workers	Redis Streams consumer group name
consumerName	loggerd-0	Consumer identifier within the group
batchSize	100	Max messages per XREADGROUP call
idleTimeout	1000ms	Block duration for empty streams
defaultStream	events:queue	Default Redis stream name
maxRetries	3	Retry attempts before DLQ
dlqTTL	7d	TTL on retry counter keys
pelReclaimTimeout	5m	Interval for PEL stale message reclaim
msgSeenTTL	24h	Deduplication key expiry

DLQ (Dead Letter Queue)

• Stream name: events:queue:dlq
• Messages exceeding maxRetries are moved here with fields: original_stream, retry_count, error, and all original message values.
• Retry counter keys follow the pattern dlq:retry:{stream}:{group}:{msg_id} with a 7-day TTL.

PEL (Pending Entries List) Reclamation

Every 5 minutes, loggerd scans the consumer group's pending entries for messages idle longer than 5 minutes and reclaims them via XCLAIM to the "reclaimer" consumer. Failed reclaimed messages are sent to the DLQ after another processing failure.

---

Runbooks

Service Won't Start

1. Check logs for initialization errors (Redis connection, DB pool).
2. Verify environment variables — ensure YABANIN_OTEL_ENDPOINT and YABANIN_OTEL_SAMPLE_RATE follow the expected format if tracing is desired.
3. Check port conflicts: each service binds to a distinct port (gateway 8082/9090, admin 8086, loggerd 8085).

Gateway Health Failing

1. curl http://localhost:8082/healthz — check the response JSON.
2. If redis.status is "down":
   • Verify Redis is running and reachable from the gateway host.
   • Check Redis connection parameters (host, port, password) in configuration.
   • Look for network/firewall issues between gateway and Redis.

High Request Latency

1. Query Prometheus for gateway_request_duration_seconds:

   histogram_quantile(0.95, rate(gateway_request_duration_seconds_bucket[5m]))

2. Filter by method and path labels to identify slow endpoints.
3. Check upstream service health — the gateway forwards requests to configured backends.

Events Not Appearing in Postgres (Loggerd)

1. Check health: curl http://localhost:8085/healthz
2. Verify Redis stream has messages:

   redis-cli XRANGE events:queue - + COUNT 10

3. Check consumer group status:

   redis-cli XINFO CONSUMERS events:queue loggerd-workers

4. Check pending entries for stuck messages:

   redis-cli XPENDING events:queue loggerd-workers

5. Check Postgres connectivity — the loggerd service requires a DB pool to insert request_logs.
6. Inspect logs for XREADGROUP errors or XACK failures.

Messages Stuck in Pending Entries List (PEL)

Messages accumulate in the PEL when they are delivered to a consumer but not acknowledged (XACK). The loggerd service auto-reclaims stale messages every 5 minutes.

If you see a large backlog:

1. Check for application errors in logs (e.g., DB write failures).
2. Verify Postgres is accepting connections and has sufficient capacity.
3. Messages that fail processing after 3 retries are moved to the DLQ (events:queue:dlq). Inspect the DLQ:

   redis-cli XRANGE events:queue:dlq - + COUNT 10

Circuit Breaker Tripping

If gateway_circuit_state reads 1 (OPEN):

1. The upstream service is likely unhealthy or experiencing high error rates.
2. Check upstream service health directly.
3. Monitor gateway_circuit_failures counter for the total trip count.
4. When the circuit transitions to HALF_OPEN (2), a probe request will be sent. If successful, it closes (0).

DLQ Inspection

To view poison messages:

redis-cli XRANGE events:queue:dlq - + COUNT 50

Each DLQ entry contains: original_stream, retry_count, error, and all original message fields. Messages can be reprocessed by reading from the DLQ stream and republishing to the main stream, or by fixing the underlying issue and letting the system resume normal processing.

---

Alerting Recommendations

Prometheus Rule Examples

# Gateway requests exceeding 95th percentile 2s latency over 5 minutes
- alert: HighRequestLatency
  expr: histogram_quantile(0.95, rate(gateway_request_duration_seconds_bucket[5m])) > 2
  for: 5m
  labels:
    severity: warning

# Circuit breaker open
- alert: CircuitBreakerOpen
  expr: gateway_circuit_state == 1
  for: 0m
  labels:
    severity: critical

# Events queue backlog (in-flight > threshold)
- alert: HighEventBacklog
  expr: gateway_events_in_flight > 500
  for: 5m
  labels:
    severity: warning

# Event consumption rate drops to zero
- alert: NoEventsConsumed
  expr: rate(gateway_events_consumed_total[5m]) == 0 and gateway_events_in_flight == 0
  for: 10m
  labels:
    severity: critical

# Circuit breaker failure rate increasing
- alert: CircuitBreakerTrippingFrequently
  expr: increase(gateway_circuit_failures[1h]) > 5
  for: 0m
  labels:
    severity: warning

Logging Levels

All services use Go's log/slog. Production deployments should run at INFO level. Set the log level via the SLOG_LEVEL environment variable (or equivalent slog handler configuration).

• DEBUG: Verbose message-level details, useful for DLQ debugging.
• INFO: Service lifecycle events, successful operations.
• WARN: Recoverable issues (XACK warnings, dedup SETNX failures).
• ERROR: Failures requiring attention (DB write errors, Redis connectivity loss).

---

Docker Images

Image	Base Digest	Description
golang:1.25.0-alpine3.22	@sha256:...	Go build image
alpine:3.22	@sha256:...	Runtime (multi-stage)

Docker images are pinned by SHA-256 digest in the GitHub Actions workflow (.github/workflows/ci.yml). See the workflow file for exact digests and rebuild dates.

---

CI/CD

GitHub Actions Workflows

Workflow	Trigger	Description
ci.yml	Push to main, PRs	Build, lint, test, Docker push
vulncheck.yml	Weekly (Monday 00:00 UTC)	govulncheck on dependencies

Dependabot

Dependency updates are managed via Dependabot with two ecosystems:

• gomod: Weekly schedule, auto-merge for patch versions.
• docker: Weekly schedule, auto-merge for patch versions (images pinned by digest).

---

Quick Reference

# Run all services locally (requires Redis + Postgres)
go run ./cmd/yabanin gateway
go run ./services/admin/
go run ./services/loggerd/

# Health checks
curl http://localhost:8082/healthz   # Gateway
curl http://localhost:8086/health    # Admin
curl http://localhost:8085/healthz   # Loggerd

# Metrics
curl http://localhost:9090/metrics   # Gateway admin server
curl http://localhost:8086/metrics   # Admin service

# Redis DLQ inspection
redis-cli XRANGE events:queue:dlq - + COUNT 10