Gateway performance

What a routing layer costs on top of the provider's own latency. Measured, not asserted.

Methodology

• Mock provider isolation. The demo stack (make demo) points the gateway at an in-stack mock OpenAI provider. Its latency is flat and tiny, so what the harness measures is the gateway overlay (auth, cache, routing, lineage) plus one loopback hop to the mock — not any real provider network.
• Deterministic traffic. scripts/loadtest/http replays a fixed set of prompts, so cache state evolves identically across runs and hit-rate comparisons are meaningful.
• Honest caveat. Numbers below come from one laptop (Apple silicon, Docker). For a valid comparison, rerun both sides on the same hardware with the same commands.

Reproduction

make demo                                            # bring up the demo stack
(cd scripts/loadtest/http && go build -o /tmp/loadtest .)
/tmp/loadtest -api-key demo-key-1234567890 -rps 50 -duration 20s

The harness prints avg latency and the X-Cache breakdown (exact / semantic / miss) and passes at ≥ 25% hit-rate (the cache control point).

Matched benchmark: direct vs gateway (WEB-03)

The same traffic straight at the provider mock (direct) and through the gateway, back-to-back on the same host — the delta is the gateway overlay (auth, credit guard, cache miss-path, routing, lineage, event publish).

Setup (scripts/loadtest/matched, reproducible):

make demo                                            # fresh disposable stack
(cd scripts/loadtest/matched && go build -o bench .)
./bench -n 200 -c 1 -rps 40   # JSON: per-cell avg/p50/p95/p99, error codes

• 200 requests per cell, concurrency 1, paced at 40 rps — deliberately below the gateway's per-IP token bucket (50 rps, burst 100, pkg/gateway/gateway.go): an unpaced benchmark trips the limiter and measures 429 rejections, not the overlay.
• Unique prompt per request: the exact cache never shortcuts the measured path (a cache-hit cell would measure ~1 ms and tell you nothing about the full path).
• SSE caveat (honest): the demo mock does not implement streaming — it answers stream:true with a single JSON body — so the stream cells exercise the same non-SSE path and TTFT is omitted (there is no first frame to time). The tool measures TTFT from request start when an SSE-speaking upstream is configured.

Measured (laptop, Docker Desktop, gateway restarted before the runs; median of 3 runs, 2026-09-27):

Side	Mode	p50	p95	p99
direct	non-stream	1.0 ms	1.9 ms	2.1 ms
direct	stream*	1.0 ms	1.8 ms	2.4 ms
gateway	non-stream	5.5 ms (runs: 5.1–75.4)	6.8–81.6 ms	7.9–87.0 ms
gateway	stream*	5.8 ms (runs: 5.2–77.6)	6.5–86.2 ms	7.4–94.2 ms

* stream cells against the non-streaming mock — see the caveat above.

Reading it honestly:

• On a quiet host the overlay is ~5 ms p50, ~7 ms p95/p99 at the median request (auth + guard + cache miss-path + routing + event publish), against a ~1 ms mock answer.
• The run-to-run ranges above are not gateway variance: this laptop also runs the agent session that produced the benchmark, and the degraded runs coincide with concurrent host load (one of three runs landed at p50 75 ms). Treat ~5 ms as the floor on quiet hardware and rerun on a dedicated host for capacity numbers.
• At concurrency 4 on this shared host, p50 swung 70–379 ms across runs — not publishable as a stable number; the reproduction command above is the artifact, not a promise.
• Errors were 0 in every published cell; the tool records error status codes per cell so a red run is diagnosable from its JSON.

v12 before/after

50 RPS, 20 s, 1,000 requests, same mock provider, same prompt set.

Metric	Before (v12-1)	After (v12)
Avg latency	3 ms	3 ms
Hit-rate	60.4% (exact 579 / semantic 19 / miss 392)	59.7% (exact 597 / semantic 0 / miss 403)
Cache control point (≥ 25%)	✅	✅

The hit-rate delta (±1 point) is within run-to-run cache-state variance — two post-v12 runs measured 59.7% and 58.9%. The semantic column is a warm-up-phase artifact: once Redis is warm, every repeat prompt is caught by the exact layer before the semantic lookup runs (the semantic path itself was verified live: a near-duplicate probe returned X-Cache: SEMANTIC at 0.961 similarity).

Why the overlay numbers barely move on this harness: the mock provider sits on loopback, so each eliminated round-trip is a sub-millisecond local call — below the harness's 1 ms rounding. The v12 changes remove serial network round-trips from the hot path, which is where they pay off under real deployment conditions (Postgres on another host, providers over the internet, concurrency beyond a handful of in-flight requests):

• Upstream transport. HTTP/2 to providers (a custom http.Transport does not enable h2 implicitly) and 100 idle connections per host (was 4 — at concurrency > 4 every extra in-flight request paid a fresh TCP+TLS handshake).
• pg-exact lookup gated off by default. Every cache miss no longer costs a Postgres round-trip; Redis is the exact layer, and the pg lookup is re-enabled with YABANIN_CACHE_PG_EXACT_LOOKUP=true after Redis data loss.
• Lineage key cache. The per-request SELECT for the Ed25519 signing key is now a 5-minute in-memory lookup (revocation bound: one TTL window, same trade-off as the model resolver cache).
• Batched async ledger appends. Signed attestations leave the handler immediately and are written in pgx batches (one round-trip per batch). Queue overflow and batch failure attempt synchronous single-row appends. This is not crash-durable: pending memory can be lost on process failure, and failed fallback writes are logged. Strict evidence durability needs a separate durable delivery design.
• Per-request logging demoted to debug and the chi request logger removed (slog + structured fields remain at info for errors/warnings).
• automaxprocs. GOMAXPROCS now tracks container CPU limits instead of the host core count (fewer context switches, steadier P99 under K8s limits).