Semantic Cache

The semantic cache system provides multi-layer caching for LLM responses, combining exact-match Redis caching with pgvector-based semantic search.

Architecture

Request
  │
  ├─→ [Redis Exact Cache] ─── hit → Return cached response
  │     key: cache:{org_id}:{model_tier}:{sha256(prompt)}
  │
  ├─→ [pgvector Exact Cache] ─── hit → Return + warm Redis
  │     unique index on (org_id, model_tier, prompt_hash, model_version)
  │     (disabled by default — see Read Path below)
  │
  ├─→ [Embedding Service] → vector(384)
  │     bge-small-en via text-embeddings-inference sidecar
  │
  ├─→ [pgvector Semantic Search] ─── hit → Return + warm Redis
  │     HNSW index with cosine similarity threshold (≥0.95)
  │
  └─→ [Upstream LLM API]
        ↓ async write to both Redis and pgvector

Components

Component	Package	Description
Redis exact cache	pkg/cache/redis.go	SHA-256 keyed Redis store, 24h TTL
pgvector storage	pkg/cache/store.go	CRUD operations, vector I/O helpers
Semantic search	pkg/cache/semantic.go	ANN lookup via HNSW cosine distance
Write pipeline	pkg/cache/writer.go	Async worker pool for cache writes
SSE replay	pkg/cache/replay.go	RecordingWriter + 3 replay modes
Key normalization	pkg/cache/normalize.go	Canonical JSON for consistent hashing
Embedding service	pkg/embeddings/client.go	HTTP client for embedding API
Unified service	pkg/cache/service.go	Orchestrates all layers with privacy controls
Privacy policies	pkg/cache/service.go	Per-org enabled/disabled/read-only modes
TTL cleaner	pkg/cache/cleaner.go	Periodic expired entry purge
Prometheus metrics	pkg/cache/metrics.go	8 cache-specific metric vectors
Analytics + A/B	pkg/cache/analytics.go	Hit rate tracking, traffic splitting, warming

Multilingual traffic (Japanese and others)

bge-small-en-v1.5 is English-trained: on Japanese (and other non-English) traffic both semantic-cache precision and classifier accuracy degrade. For multilingual deployments swap the embedder for a multilingual model via config — no code change:

YABANIN_EMBEDDER_MODEL=intfloat/multilingual-e5-small
YABANIN_EMBEDDER_QUERY_PREFIX="query: "

Notes:

• multilingual-e5-small keeps the same 384 dimensions as bge-small-en, so the existing pgvector schema (vector(384)) needs no migration (verify the dimension on first boot — a mismatch fails the insert loudly rather than corrupting the cache).
• e5-family models are trained with input prefixes. The configured YABANIN_EMBEDDER_QUERY_PREFIX is prepended to every embedded text — lookups and writes alike — which keeps cosine comparisons symmetric. Changing the prefix invalidates existing semantic entries; flush the cache (POST /api/cache/flush) after switching.
• The task classifier is a separate model with its own (English) training data; routed tiering on Japanese traffic falls back to the rule-tier V0. Retraining the classifier on Japanese prompts is tracked separately.

A compose profile with the multilingual embedder sidecar lives in deploy/docker-compose.yml (embedder-multilingual).

Privacy Controls

Each organization has a cache policy:

• enabled — full read/write cache access
• disabled — no caching at all (privacy-first)
• read-only — can look up cached responses but never write new ones

Policies are cached in-memory with 60s TTL to avoid DB overhead on every request.

SSE Streaming

For streaming responses, the cache captures individual SSE frames with timing:

rw := cache.NewRecordingWriter(w)
// ... upstream response is written through rw ...
chunks := rw.Chunks() // []SSEChunk{Data, DelayMS}

Replay modes:

• faithful — preserve original timing (default)
• burst — send all chunks immediately
• paced — equal 10ms intervals

Docker Compose

The bge embedding sidecar is included in deploy/docker-compose.yml:

bge-embedder:
  image: ghcr.io/huggingface/text-embeddings-inference:latest
  command: --model-id BAAI/bge-small-en-v1.5 --dtype float32
  ports: ["6006:80"]

Prometheus Metrics

Metric	Type	Labels	Description
cache_hits_total	Counter	type, org_id	Cache hits by lookup type
cache_writes_total	Counter	status	Write attempts (written/dropped)
cache_purged_total	Counter	backend	Expired entries removed
cache_size	Gauge	backend	Current entry count
cache_lookup_duration_seconds	Histogram	type	Lookup latency
cache_write_queue_size	Gauge	—	Pending write jobs
cache_write_queue_drops_total	Counter	—	Dropped writes (queue full)
cache_embedding_errors_total	Counter	—	Embedding failures

Migration

Run migration 0009 to create the cache tables:

-- migrations/0009_pgvector.up.sql
CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE cache_entries (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    org_id UUID NOT NULL, model_tier TEXT NOT NULL,
    prompt_hash TEXT NOT NULL, embedding vector(384) NOT NULL,
    response JSONB NOT NULL, sse_chunks JSONB,
    tokens_in INT DEFAULT 0, tokens_out INT DEFAULT 0,
    hit_count INT DEFAULT 0, model_version TEXT DEFAULT '1',
    created_at TIMESTAMPTZ DEFAULT now(),
    expires_at TIMESTAMPTZ DEFAULT (now() + interval '24 hours')
);

Tests

• Unit tests: go test ./pkg/cache/...
• Benchmarks: go test -bench=. ./pkg/cache/...
• Integration (requires Docker): go test -race ./services/gateway/integration/...