Runbook: Kadar Hit Cache Jatuh di Bawah 25%
Langkah Diagnosis
1. Semak Metrik Serta-merta
# From Grafana dashboard or Prometheus:
# - cache_hits_total{type="miss"} spike?
# - cache_lookup_duration_seconds p99 > 15ms?
# - cache_embedding_errors_total increasing?
# - cache_write_queue_drops_total increasing?
curl http://localhost:2112/metrics | grep "cache_"
2. Semak Kesihatan Embedder
# bge-embedder health check (port 6006)
curl http://localhost:6006/health
# Check embedder logs
docker logs deploy-bge-embedder-1 --tail=50
3. Semak Konfigurasi Cache
# Verify org cache settings haven't been accidentally changed
curl -H "Authorization: Bearer $ADMIN_TOKEN" \
http://localhost:2112/api/organizations/$ORG_ID/cache-settings
# Check if any orgs were switched to "exact" or "off" mode
curl -H "Authorization: Bearer $ADMIN_TOKEN" \
http://localhost:2112/api/organizations?cache_mode=off
4. Semak Perubahan Skema
# Verify pgvector extension is still valid
docker exec deploy-postgres-1 psql -U yabanin -d yabanin \
-c "SELECT extversion FROM pg_extension WHERE extname='vector';"
# Check cache_entries table size
docker exec deploy-postgres-1 psql -U yabanin -d yabanin \
-c "SELECT count(*), min(expires_at), max(expires_at) FROM cache_entries;"
5. Semak Perubahan Corak Trafik
# Look at request volume vs cache hits in the last hour
curl -H "Authorization: Bearer $ADMIN_TOKEN" \
http://localhost:2112/api/analytics/$ORG_ID/cache?from=$(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%SZ)
# Check if new request patterns don't match existing cache entries
docker exec deploy-postgres-1 psql -U yabanin -d yabanin \
-c "SELECT count(*) FROM request_logs WHERE created_at > now() - interval '1 hour' AND cache_hit_type IS NULL;"
Tindakan Pembetulan
Pembetulan Pantas: Siram
Jika cache mempunyai entri lapuk atau model embedding berubah:
# Flush entire cache (root admin only)
curl -X POST -H "Authorization: Bearer $ADMIN_TOKEN" \
"http://localhost:2112/api/cache/flush?org_id=$ORG_ID&reason=low_hit_rate"
# NOTE: cache warming was removed (W-CACHE); POST /api/cache/warm answers 410.
# The semantic cache re-populates from live traffic only.
Pembetulan Sederhana: Laraskan Ambang
Jika cache semantik memulangkan terlalu sedikit hit (ambang terlalu tinggi):
- Sunting
pkg/cache/semantic.go: ubahDefaultSimilarityThresholddaripada0.95kepada0.92 - Mulakan semula gerbang:
docker compose restart gateway
Jika terlalu banyak positif palsu (ambang terlalu rendah):
- Tingkatkan ambang kepada
0.97
Pembetulan Jangka Panjang: Tambah Entri Cache Baharu
Pemanasan cache telah dibuang (W-CACHE); entri ditulis daripada trafik langsung sahaja.
- Kenal pasti prompt teratas yang tiada daripada request_logs
- Jalankan prompt itu melalui gerbang (trafik sebenar atau yang dimainkan semula)
- Pantau peningkatan kadar hit selama 30 minit seterusnya
Eskalasi
| Gejala | Tindakan |
|---|---|
| Embedder terhenti >5 min | Insiden P1 — mulakan semula embedder, undur ke mod Redis sahaja |
| Guguran barisan tulis cache >10/min | Semak I/O cakera pada Redis; pertimbangkan meningkatkan cache_write_queue_size |
| Ekstensi pgvector tiada | KRITIKAL — migrasi pgvector belum dijalankan; jalankan migrasi 0009 dengan segera |
| Kadar hit bawah 10% selama >1 jam selepas siram | Semak dasar penormalan; semak sama ada corak permintaan berubah dengan ketara |
Pencegahan
- Tetapkan amaran Grafana:
cache_hits_total{type="miss"} / (cache_hits_total{type="exact"} + cache_hits_total{type="semantic"} + cache_hits_total{type="miss"}) > 0.75 for 15m - Semakan mingguan trend
savings_usddalam papan pemuka - Audit dasar penormalan bulanan: semak laporan perlanggaran palsu