Yabanin — Offline Spend Audit (yb audit)

yb audit is a self-contained, offline cost analyzer. Point it at a customer's own LLM request log, and it reports estimated spend from logged usage and bundled prices, and what semantic caching plus tier-routing would have saved on that traffic. It needs no gateway, no admin URL, no API key, no organization, and makes no network calls — it runs from a single binary against a local file (or stdin).

The analyzer is honest by construction: every dollar figure is traceable to the input rows or the pricing table, every derived number is labeled measured or estimated, and a projection that is conditional (cache equivalence, routing acceptability) carries an explicit caveat. Where the sample is too small to be reliable, the report says so in a banner instead of pretending precision.

Quickstart: clean checkout → HTML report

Verified end-to-end on this tree (2026-09-13):

# 1. Build the operator CLI (contains `yb audit`).
go build -o bin/yb ./cmd/yb

# 2. Run it against the bundled synthetic traffic fixture…
bin/yb audit datasets/audit/sample_traffic.jsonl \
    --format html --output /tmp/my-report.html

# 3. …or against your own export (JSONL, one request per line):
bin/yb audit your-traffic.jsonl --format html --output your-report.html

Expected result: a self-contained HTML file (opens without JS/CDN/network) with per-model spend, cache and routing projections. Unknown model prices surface as unknown — never zero; malformed input lines are counted and reported (the command fails when nothing could be parsed).

Usage

# Markdown report (default) to stdout
yb audit path/to/traffic.jsonl

# JSON
yb audit path/to/traffic.jsonl --format json

# Self-contained HTML (no CDN, no JS) to a file
yb audit path/to/traffic.jsonl --format html --output report.html

# Exact mode is the supported offline default.
# Semantic simulation is a library capability; the CLI has no embedder wiring.

# Share the report without leaking prompt content
yb audit path/to/traffic.jsonl --no-prompts

# Read from stdin
cat traffic.jsonl | yb audit -

Flags:

Flag	Default	Meaning
--format, -f	md	md, json, or html
--cache-mode	exact	exact, semantic, or off
--output, -o	(stdout)	write the report to this file
--max-records	5,000,000	cap on ingested records
--no-prompts	false	drop raw prompt text after hashing
--provider	(inferred)	override provider inference for the whole file

An invalid --format or --cache-mode is rejected, never silently defaulted.

Input formats

The ingest auto-detects the format, or you can rely on the sniffing:

• OpenAI-style JSONL — one JSON object per line, in either the flat shape (model, usage.prompt_tokens, usage.completion_tokens, messages, optional timestamp, status_code, duration_ms) or the wrapped shape ({"request": {...}, "response": {...}}).
• CSV — header-driven; column names are matched to fields.
• This gateway's format — the JSONL/CSV emitted by Yabanin's own logger.

Malformed lines are counted and skipped, never fatal. A symlinked input is reported, not silently followed. When token counts are absent they are estimated from text length at a fixed 4 chars/token, and the report discloses the share of rows whose counts were estimated.

The six sections

1. Summary — request count, total cost (input/output split), unpriced requests (excluded from all dollar totals, never priced at $0), the window, top models, top providers.
2. What we found — the measured findings: spend concentration (p50/p90/p99 + top-1%/10% share), repeated-prefix bloat, top prompt clusters, retry storms, error waste, max_tokens over-provision, tail latency, model-mix drift. Only findings above threshold are listed; none is padded with adjectives.
3. Cache simulation — hit rate, tokens avoided, dollars avoided, per-model hit rates. In semantic mode it carries the caveat that a semantic hit is a similarity match, not a verified-equivalent answer.
4. Routing simulation — the class distribution (easy/medium/hard) from the in-process rule-tier classifier, routing-only savings, routing-on-misses savings, and the combined figure with the overlap stated explicitly.
5. Data quality — parsed/skipped counts, share with real token counts, share with timestamps, unpriced request count, and the unpriced model names.
6. Recommendations — every rule that fired, verbatim, anchored to the number that triggered it. No recommendation is printed unless its finding is in the data.

Cache and routing: production order, no double-counting

The combined savings figure applies the two optimizations in the same order the gateway applies them in production: cache first, routing to cache-misses only. The report states three numbers so a reader cannot double-count:

• Cache-only — dollars the cache alone would have avoided.
• Routing-only — dollars tier-routing would have saved if every request were routed.
• Combined — cache-only + routing-on-misses. The overlap (routing-only − routing-on-misses) is the amount that would be double-counted if you naively added cache + routing, and it is stated explicitly.

A request never contributes a routing saving when its target model has no known price (it is dropped and counted), when the target equals the original model, or when the target is not cheaper. Unknown models are never silently priced at $0.

Cache modes

• exact (default) — keys on the production prompt hash. No dependencies, fully offline, no network.
• semantic — the library simulator requires an injected EmbedFunc. The CLI does not wire an embedder and rejects this mode explicitly; starting an embedder service alone does not enable it.
• off — skip the simulation; routing-on-misses then equals routing-all.

Determinism

For a given input and mode the report is byte-identical run to run, with one exception: the generated line, which carries the timestamp. Everything else — every table, every figure, the JSON field order (struct declaration order, maps sorted by key) — is stable. This is what makes the report safe to diff, archive, and hand to a customer.

Insufficient sample

The report carries a hard insufficient sample banner when the input has fewer than 50 requests, or a window shorter than 24 hours (or no window at all). The banner is a fact about the sample, not a suggestion: the figures are still shown, but the report flags that they are not statistically reliable.

Example

A sample request log and its generated report live in datasets/audit/:

yb audit datasets/audit/sample_traffic.jsonl --format md
# or open the pre-generated HTML:
open datasets/audit/sample_report.html

The sample is a deterministic 600-request log (repeated system prompts, a mix of easy/medium/hard workloads, retries, errors, and an unpriced internal model) that exercises every section.

What it does not do

• It does not call the model, the gateway, or any network service in exact mode.
• It does not modify the input file.
• It does not attempt to verify that a cached answer is equivalent to a fresh one (that is a cache-architecture decision); it reports the hit rate and the caveat.
• It does not price an unknown model at $0 or claim a saving it cannot back with the pricing table.

First report from a clean checkout

Install the Go version declared in the root go.mod (currently Go 1.27.1 or a compatible newer toolchain). From the repository root:

go build -o /tmp/yabanin-audit ./cmd/yabanin
/tmp/yabanin-audit audit datasets/audit/sample_traffic.jsonl \
  --cache-mode exact --format html --no-prompts \
  --output /tmp/yabanin-sample-report.html

Open /tmp/yabanin-sample-report.html in a browser. The bundled 600-request fixture is synthetic, not customer evidence. View the synthetic sample report. No gateway, provider key, Redis or database is needed for the audit. Building from source may download Go modules; the resulting exact audit itself is offline.

The report identifies the bundled pricing source and its SHA-256 fingerprint. The original upstream verification date was not recorded and is shown as unknown. These are usage-based list-price estimates; cached-input discounts, negotiated rates and service tiers are not reconciled against provider invoices. Unknown model prices stay excluded, not counted as free.

Invalid formats and unwired semantic mode exit with an error. If every input row is invalid, the command fails; otherwise parse failures appear in Data quality. Missing token usage may be estimated and disclosed. --no-prompts drops raw prompt text after hashing; source paths, model names, timestamps and derived findings can still identify a workload. Inspect the entire export before sharing. Dropping prompts also reduces the information available to prompt-based classification; interpret routing projections accordingly.

To reproduce the public example exactly, use the same command with --output web/site/static/examples/audit-sample.html. A clearly marked synthetic-data banner is added to the public copy. Generated timestamps vary; all underlying figures must match a new run with the same source and binary.