Skip to main content

Runbook: Kadar Hit Cache Jatuh di Bawah 25%

Langkah Diagnosis​

1. Semak Metrik Serta-merta​

# From Grafana dashboard or Prometheus:
# - cache_hits_total{type="miss"} spike?
# - cache_lookup_duration_seconds p99 > 15ms?
# - cache_embedding_errors_total increasing?
# - cache_write_queue_drops_total increasing?

curl http://localhost:2112/metrics | grep "cache_"

2. Semak Kesihatan Embedder​

# bge-embedder health check (port 6006)
curl http://localhost:6006/health

# Check embedder logs
docker logs deploy-bge-embedder-1 --tail=50

3. Semak Konfigurasi Cache​

# Verify org cache settings haven't been accidentally changed
curl -H "Authorization: Bearer $ADMIN_TOKEN" \
http://localhost:2112/api/organizations/$ORG_ID/cache-settings

# Check if any orgs were switched to "exact" or "off" mode
curl -H "Authorization: Bearer $ADMIN_TOKEN" \
http://localhost:2112/api/organizations?cache_mode=off

4. Semak Perubahan Skema​

# Verify pgvector extension is still valid
docker exec deploy-postgres-1 psql -U yabanin -d yabanin \
-c "SELECT extversion FROM pg_extension WHERE extname='vector';"

# Check cache_entries table size
docker exec deploy-postgres-1 psql -U yabanin -d yabanin \
-c "SELECT count(*), min(expires_at), max(expires_at) FROM cache_entries;"

5. Semak Perubahan Corak Trafik​

# Look at request volume vs cache hits in the last hour
curl -H "Authorization: Bearer $ADMIN_TOKEN" \
http://localhost:2112/api/analytics/$ORG_ID/cache?from=$(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%SZ)

# Check if new request patterns don't match existing cache entries
docker exec deploy-postgres-1 psql -U yabanin -d yabanin \
-c "SELECT count(*) FROM request_logs WHERE created_at > now() - interval '1 hour' AND cache_hit_type IS NULL;"

Tindakan Pembetulan​

Pembetulan Pantas: Siram​

Jika cache mempunyai entri lapuk atau model embedding berubah:

# Flush entire cache (root admin only)
curl -X POST -H "Authorization: Bearer $ADMIN_TOKEN" \
"http://localhost:2112/api/cache/flush?org_id=$ORG_ID&reason=low_hit_rate"

# NOTE: cache warming was removed (W-CACHE); POST /api/cache/warm answers 410.
# The semantic cache re-populates from live traffic only.

Pembetulan Sederhana: Laraskan Ambang​

Jika cache semantik memulangkan terlalu sedikit hit (ambang terlalu tinggi):

  • Sunting pkg/cache/semantic.go: ubah DefaultSimilarityThreshold daripada 0.95 kepada 0.92
  • Mulakan semula gerbang: docker compose restart gateway

Jika terlalu banyak positif palsu (ambang terlalu rendah):

  • Tingkatkan ambang kepada 0.97

Pembetulan Jangka Panjang: Tambah Entri Cache Baharu​

Pemanasan cache telah dibuang (W-CACHE); entri ditulis daripada trafik langsung sahaja.

  1. Kenal pasti prompt teratas yang tiada daripada request_logs
  2. Jalankan prompt itu melalui gerbang (trafik sebenar atau yang dimainkan semula)
  3. Pantau peningkatan kadar hit selama 30 minit seterusnya

Eskalasi​

GejalaTindakan
Embedder terhenti >5 minInsiden P1 — mulakan semula embedder, undur ke mod Redis sahaja
Guguran barisan tulis cache >10/minSemak I/O cakera pada Redis; pertimbangkan meningkatkan cache_write_queue_size
Ekstensi pgvector tiadaKRITIKAL — migrasi pgvector belum dijalankan; jalankan migrasi 0009 dengan segera
Kadar hit bawah 10% selama >1 jam selepas siramSemak dasar penormalan; semak sama ada corak permintaan berubah dengan ketara

Pencegahan​

  • Tetapkan amaran Grafana: cache_hits_total{type="miss"} / (cache_hits_total{type="exact"} + cache_hits_total{type="semantic"} + cache_hits_total{type="miss"}) > 0.75 for 15m
  • Semakan mingguan trend savings_usd dalam papan pemuka
  • Audit dasar penormalan bulanan: semak laporan perlanggaran palsu