Semantic cache vs vendor prompt caching
AWS Bedrock, OpenAI, Anthropic and Vertex all ship native prompt caching now.
This page says plainly what each mechanism does, so you can decide — or,
better, run yb audit on your log and let it split the two for
your traffic.
| Vendor prompt caching | Yabanin semantic cache | |
|---|---|---|
| Matched | exact prefix (token-identical) | meaning: cosine ≥ 0.95, bge-small embedding |
| Scope | one model, one provider, one account | cross-model (tier-keyed) and cross-provider |
| Lifetime | minutes (Anthropic default 5 min) | 24 h TTL, per-org policy |
| Economics | discounted read tokens (order of ~90% off cached input) | the request is not paid at all |
| Sits on top of | nothing | tier routing, fallback, lineage |
| Best at | a long repeated system prefix on a fixed model | repeated user questions, model-mix optimization, multi-provider setups |
When the vendor cache wins
Stable multi-K system prompts against a single pinned model. The offline audit says exactly this in its recommendations: repeated-prefix bloat is a provider prompt-cache problem, and the audit reports cache savings separately from routing so nothing is double-counted.
When the semantic cache wins
- The same question arrives in different phrasings (support, internal FAQ, agent loops).
- Traffic fans out across models and providers — vendor caches never cross that boundary.
- You want a measured hit rate on your own logs before believing any savings number.
They compose
Turn on the vendor cache where it fits; the gateway passes provider usage
through (cache_read_tokens / cache_creation_tokens are captured per
request), so both savings show up separately in the dashboard. No either/or.
Precision
The semantic match is not an LLM call — it is a fixed embedding model plus a cosine threshold. How precise that judgment is, and how the threshold trades false hits against recall, is measured, not asserted: see Cache precision.
Pricing and TTL figures above reflect public mid-2026 vendor mechanics and change often — re-verify against the vendor's current pricing page before quoting them externally.