Skip to main content

Evaluation Suites

The yb CLI ships two offline evaluation commands so you can measure routing quality and guard-hook detection quality before and after a change, on a fixed labeled dataset. Both are deterministic and runnable in CI.

yb evaluate routing --golden datasets/eval/golden_routing.jsonl [--save] [--against datasets/eval/routing_baseline.json] [--max-mismatch N]
yb evaluate guards  --corpus datasets/eval/guards_labeled.jsonl [--min-precision 0.98] [--min-recall 0.90]
yb evaluate guards  --revision datasets/eval/guards-revision.json [--min-precision 0.98] [--min-recall 0.90]

Make targets wrap them:

make evaluate-routing
make evaluate-guards

The make evaluate-guards target continues to use the legacy --corpus input. Use --revision directly when you need the result to identify the exact local revision document evaluated.

Routing regression (yb evaluate routing)​

Routing is a classifier + tier-router pipeline. A change to the classifier, the rules, the pricing, or the router can silently shift which model a prompt lands on. The routing evaluation pins that behavior.

Dataset. datasets/eval/golden_routing.jsonl is a labeled set of requests. Each line is one request (prompt/metadata) plus the routing tier it should map to under the current routing policy. The set is generated to match the production rule_tier behavior, so it is a regression baseline, not a "correctness oracle".

What it does.

  1. Runs the current routing pipeline over every row.
  2. Compares the resulting tier against the golden label.
  3. Reports the mismatch count and rate.
  4. With --save, writes a snapshot (datasets/eval/routing_baseline.json) of the current mapping. With --against <file>, compares the current run to a saved baseline and reports which rows changed.
  5. --max-mismatch N makes the command fail (non-zero exit) when more than N rows disagree — wire this into CI to catch routing regressions.

A clean run is one where the current pipeline reproduces the golden tiers (or, when used with --against, reproduces the saved baseline).

Guard detection (yb evaluate guards)​

Guard hooks (see Agent Governance) are deterministic detectors. The guard evaluation measures their precision and recall against a labeled corpus.

Dataset. datasets/eval/guards_labeled.jsonl is a set of labeled samples. Positive rows contain a real, check-digit/Luhn-validated sensitive datum (INN, SNILS, PAN, secret, or injection phrase) that a detector should flag. Negative rows are clean — including decoys (numbers and strings that look sensitive but fail the checksum, e.g. a 10-digit run that is not a valid INN) that a detector should not flag. The corpus is generated by scripts/gen_guards_corpus.py, which computes valid checksums so the positive labels are genuine.

What it does.

  1. Runs the detectors over every row.
  2. Scores against the labels: true positives, false positives, false negatives.
  3. Reports precision (detected-and-correct / detected) and recall (detected / should-detect).
  4. Fails (non-zero exit) when precision < --min-precision (default 0.98) or recall < --min-recall (default 0.90).

The defaults are deliberately strict: a guard that runs at deny mode must have very high precision (to avoid blocking legitimate traffic) and high recall (to avoid leaking sensitive data).

Pinning a local guard dataset revision​

--revision FILE is an opt-in alternative to --corpus FILE. It reads a local JSON document with an explicit schema version, dataset ID and sample IDs. Each sample payload uses the same text, type and expect fields as the guard evaluator.

Use a yb build whose commit includes the --revision option; there is no separate released minimum version to install. Check the binary with yb version. If an older binary reports an unknown --revision flag, rebuild or upgrade to a build containing this option, or continue with the legacy --corpus input.

For example, a small synthetic revision looks like this:

{
  "schema_version": 1,
  "dataset_id": "demo-guards",
  "samples": [
    {
      "id": "sample-detect-1",
      "payload": {
        "text": "<SYNTHETIC_SECRET>",
        "type": "secret",
        "expect": "detect"
      }
    },
    {
      "id": "sample-clean-1",
      "payload": {
        "text": "ordinary synthetic text",
        "type": "clean",
        "expect": "clean"
      }
    }
  ]
}

Save it as guards-revision.json. Generate the synthetic secret fixture before evaluating it; do not use real credentials:

python3 - <<'PY'
import json
from pathlib import Path

path = Path("guards-revision.json")
revision = json.loads(path.read_text())
revision["samples"][0]["payload"]["text"] = "synthetic marker " + "sk-" + "abcdefghijklmnop12345678"
path.write_text(json.dumps(revision, indent=2) + "\n")
PY
yb evaluate guards --revision guards-revision.json

The report includes the dataset ID and a lowercase SHA-256 revision_id for the exact bytes read. Keep the file that produced a result and record this ID with the metrics when you need to compare runs. Editing the file changes its revision ID. An invocation evaluates and hashes the same in-memory byte snapshot; if the file is replaced after that snapshot has been read, the active result still identifies the bytes it evaluated. A later invocation reads the current file and may report a different revision ID.

The revision document is limited to 8 MiB and 10,000 samples. The schema version must be 1; dataset and sample IDs must be non-empty, and sample IDs must be unique within the document. Each payload must be an object with non-empty text and expect set to detect or clean. Unknown fields, repeated JSON keys, malformed JSON, a second trailing JSON document, and duplicate sample IDs are rejected before metrics are printed. If validation fails, correct the document and run the command again. Supplying --corpus and --revision together is an error; choose one input mode per invocation.

The legacy --corpus option and its output remain available. Revision input is local to this CLI flow: it does not upload or persist samples, add a server-side catalog, or change existing run-manifest formats. The revision ID identifies the exact input used for this evaluation; it does not certify dataset quality or claim that another run producer has adopted revision pinning.

Baseline policy​

The evaluation suites exist to keep a baseline and to compare against it:

  • Before a routing- or guard-affecting change, run both evaluations and note the numbers (or --save the routing baseline).
  • After the change, run again and compare. A routing regression shows up as a mismatch-rate increase; a guard regression shows up as a precision or recall drop below threshold.
  • In CI, run make evaluate-routing -- --max-mismatch 0 and make evaluate-guards as gates. (The exact --max-mismatch budget is a policy choice: 0 is the strictest and only allows exact reproduction.)

Evals as a PR gate (yb-eval)​

The reusable composite action at .github/actions/yb-eval/action.yml wires these suites into pull requests:

  • builds the yb binary in the runner from the repo (no network — the routing and guard evals are in-process and deterministic);
  • runs yb evaluate routing --against <baseline> --max-mismatch 0 and yb evaluate guards --min-precision … --min-recall …;
  • writes a machine-readable summary (pass/fail + counters) to a yb-eval artifact;
  • posts a single PR summary comment (replaced on re-run, not duplicated);
  • fails the step (red check) when either gate is red, so a routing or guard regression breaks CI.

Wire it into a pull_request workflow with path filters on the routing / pricing / guard-hook source and the eval datasets, e.g.:

on:
  pull_request:
    paths:
      - 'pkg/classifier/**'
      - 'pkg/router/**'
      - 'pkg/guardhooks/**'
      - 'pkg/pricing/**'
      - 'datasets/eval/**'
      - 'cmd/yb/**'
jobs:
  eval-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/yb-eval
        with:
          comment: true

All inputs have repo defaults, so the action is drop-in; override routing-max-mismatch, guards-min-precision, or guards-min-recall to relax the gate per policy.

This is a regression harness, not a benchmark suite: it does not chase absolute "best-in-class" numbers, it ensures a change does not move behavior you did not intend to move.

The numbers reported here are measured against the shipped datasets. They are not quoted from any external whitepaper or third-party evaluation.