Cache precision
The most common objection to a semantic cache: "how do you know the cached answer is right for this question — and doesn't judging that require another LLM call?"
The second part first: no LLM is in the judgment loop. That would be self-defeating. The cache judgment is a fixed, self-hosted embedding model plus a cosine threshold — deterministic and cheap:
- Exact layer — SHA-256 of the normalized prompt against Redis. Identical bytes: hit, zero model involvement.
- Semantic layer — embed the prompt with a 137M-parameter bi-encoder (bge-small, ~20 ms on CPU), then a nearest-neighbor search over pgvector. Cosine similarity ≥ 0.95 (default threshold): hit.
The open question is empirical: how good is that judgment? This page exists to answer it with measurements instead of adjectives.
Methodology
The harness is scripts/bench_cache_precision.py, run against paraphrase
pairs (dataset format in datasets/cache/README.md):
- positive pairs (
same: true) — paraphrases where serving the cached answer is acceptable; - negative pairs (
same: false) — near-misses: same topic, different intent, different entity, negation (where serving the cached answer would be a wrong answer).
Per threshold it reports precision (share of hits that are truly the same question), recall (share of true duplicates caught), and F1.
Results
EN and JA measured. Reproduce with:
docker compose --profile bge up -d embedder-bge # Xenova/bge-small-en-v1.5 (amd64 hosts)
docker compose --profile multilingual up -d embedder-multilingual # intfloat/multilingual-e5-small
python3 datasets/cache/gen_pairs.py --lang en --pos 60 --neg 31 --out datasets/cache/pairs_en.jsonl
python3 scripts/bench_cache_precision.py datasets/cache/pairs_en.jsonl \
--embedder http://localhost:6008 --model Xenova/bge-small-en-v1.5 --batch-size 8
python3 datasets/cache/gen_pairs.py --lang ja --pos 60 --neg 31 --out datasets/cache/pairs_ja.jsonl
python3 scripts/bench_cache_precision.py datasets/cache/pairs_ja.jsonl \
--embedder http://localhost:6007 --model intfloat/multilingual-e5-small --batch-size 8
| Model | Dataset | Threshold | Precision | Recall | F1 |
|---|---|---|---|---|---|
| bge-small-en-v1.5 | pairs_en (N=91: 60 same / 31 different) | 0.95 | 0.0000 | 0.0000 | 0.0000 |
| multilingual-e5-small | pairs_ja (N=91: 60 same / 31 different) | 0.95 | 0.2857 | 0.0667 | 0.1081 |
Full threshold sweep (same runs):
bge-small-en-v1.5 / pairs_en:
| Threshold | Precision | Recall | F1 | FP | FN |
|---|---|---|---|---|---|
| 0.85 | 0.4643 | 0.2167 | 0.2955 | 15 | 47 |
| 0.90 | 0.0833 | 0.0167 | 0.0278 | 11 | 59 |
| 0.92 | 0.0000 | 0.0000 | 0.0000 | 9 | 60 |
| 0.95 | 0.0000 | 0.0000 | 0.0000 | 3 | 60 |
| 0.97 | 0.0000 | 0.0000 | 0.0000 | 1 | 60 |
| 0.98 | 0.0000 | 0.0000 | 0.0000 | 1 | 60 |
Best F1: 0.2955 at threshold 0.85. Similarity distribution: same-questions p10=0.7549 p50=0.8282 p90=0.8816; different-questions p50=0.8449 p90=0.9437 p99=0.9874.
multilingual-e5-small / pairs_ja:
| Threshold | Precision | Recall | F1 | FP | FN |
|---|---|---|---|---|---|
| 0.85 | 0.6988 | 0.9667 | 0.8112 | 25 | 2 |
| 0.90 | 0.6346 | 0.5500 | 0.5893 | 19 | 27 |
| 0.92 | 0.5588 | 0.3167 | 0.4043 | 15 | 41 |
| 0.95 | 0.2857 | 0.0667 | 0.1081 | 10 | 56 |
| 0.97 | 0.0000 | 0.0000 | 0.0000 | 6 | 60 |
| 0.98 | 0.0000 | 0.0000 | 0.0000 | 3 | 60 |
Best F1: 0.8112 at threshold 0.85. Similarity distribution: same-questions p10=0.8683 p50=0.9114 p90=0.9440; different-questions p50=0.9164 p90=0.9794 p99=0.9986.
Honest reading: the synthetic templated FAQ pairs are paraphrased far more
freely than typical repeated customer prompts, and both models sit between
the two distributions only weakly — at the production 0.95 threshold
bge-small-en-v1.5 catches zero of the 60 EN paraphrases, and
multilingual-e5-small catches 4 of 60 JA paraphrases at 0.286 precision,
while near-misses reach p99=0.9874 (EN) and 0.9986 (JA). The JA sweep also
shows the real operating point is much lower than the default: multilingual-e5
separates the JA pair space best around 0.85 (F1 0.81), and even there
precision is 0.70 — roughly 3 in 10 hits would serve a wrong answer on this
adversarial set. Two methodology caveats: e5 models are trained with
query: / passage: prefixes this harness does not add (raw-text numbers
are a lower bound), and templated near-misses are harder than real duplicate
traffic. Treat these numbers as a conservative floor: the semantic layer's
value shows up on real duplicate traffic (typos, word order, trivial
rephrasings), which the templated set does not model. The exact layer remains
the high-precision backbone; the semantic layer is a recall supplement, and
per-org exact privacy mode is the safe posture when precision matters more
than savings.
Until your own numbers are in, treat cache savings as something to measure
on your own traffic, which is exactly what the offline
audit and the live dashboards (cache_similarity_score
histogram, per-threshold hit accounting) are for.
Tuning
- Raise precision (high-stakes traffic): threshold 0.97+, or privacy mode
exact— hits only on byte-identical prompts. - Raise recall (cost-sensitive traffic): threshold 0.92–0.95; on the measured JA set the separating point is lower still — 0.85 (see sweep above), so tune from measurement, not from defaults.
- Per-org privacy modes (
off/exact/semantic) let a single deployment serve both postures; see Cache architecture. - Non-English traffic needs a multilingual embedder — see Multilingual traffic.
Relationship to the A/B flow
Numbers above are lab measurements. The gateway's built-in
A/B experiments (control =
cache off, treatment = cache on, compared on real savings_usd) are how you
verify the same question on production traffic before committing.