The problem
A retrieval change can look better on a few hand-tried questions while regressing exact-term lookup or a slice nobody sampled. Shipping it because an example “feels” better is how procedure search silently degrades.
Retrieval · Embeddings · Evaluation
Prism Bench
A retrieval evaluation bench for a synthetic security knowledge base. Compare lexical search, character n-grams, and text-embedding-3 dense vectors, alone or fused, slice the queries, and decide PROMOTE or HOLD from deterministic metrics. The promoted BM25 + dense hybrid is the retriever the Aegis agent searches with.
- 32 synthetic documents
- 28 labeled queries
- 5 retrievers · 1 gate
POST /api/retrieval
Personal project on synthetic data · Not an employer system
Architecture
- BM25
- Dense embeddings
- Hybrid RRF
- nDCG / MRR
- Release gate
- Judged corpus32 documents and 28 graded queries, sliced by query type
- RetrieversBM25, character trigrams, text-embedding-3-large (256d, precomputed)
- FusionWeighted reciprocal rank fusion of BM25 with vectors
- IR metricsRecall@k, MRR, nDCG, and per-slice recall vs frozen BM25
- Release gatePROMOTE or HOLD by fixed rules; the winner serves Aegis
Run it
- Switch the candidate retriever
- Toggle query expansion
- Read the PROMOTE or HOLD verdict and why
What this showsSearch & RAG retrieval · Embeddings · Offline evaluation · Data-driven release decisions
- Recall@5
- 86.3%+0.173 vs BM25
- Precision@5
- 37.1%+0.071 vs BM25
- MRR
- 0.982+0.111 vs BM25
- nDCG@5
- 0.907+0.118 vs BM25
PROMOTECandidate clears the release gate.
Deterministic IR metrics against the frozen BM25 baseline.
- Recall@k improves by at least 0.03Δ recall +0.173
- MRR does not dropΔ MRR +0.111
- No slice regresses by more than 0.10No slice below −0.10
Recall by query slice
Gate fails if any slice drops more than 0.10
BM25 baseline · candidate
Query inspector
Q07 · paraphrase · “stop a stolen login from keeping a session”
BM25 top 5
- KB-32Operations status log1
- KB-05Backup network segmentation2
- KB-14Attack path enumeration3
- KB-15Conditional access for finance SSO4
- KB-02Session revocation after takeover · grade 25
Hybrid RRF · BM25 + dense top 5
- KB-02Session revocation after takeover · grade 21
- KB-32Operations status log2
- KB-10Identity compromise procedure · grade 13
- KB-06Endpoint detection isolation4
- KB-05Backup network segmentation5
All retrievers at @5
Same 28 queries · with query expansion · gated against BM25
| Retriever | Recall@5 | MRR | nDCG@5 | Gate |
|---|---|---|---|---|
| 78.0% | 0.954 | 0.879 | PROMOTE | |
| 77.4% | 0.929 | 0.869 | PROMOTE | |
| 78.0% | 0.954 | 0.881 | PROMOTE | |
| 80.4% | 0.973 | 0.881 | PROMOTE | |
| 86.3% | 0.982 | 0.907 | PROMOTE |
Per-query recall
Select a row to inspect rankings
| Query | Slice | BM25 | Candidate | Status |
|---|---|---|---|---|
| exact | 1.00 | 1.00 | unchanged | |
| exact | 1.00 | 1.00 | unchanged | |
| exact | 1.00 | 1.00 | unchanged | |
| exact | 1.00 | 1.00 | unchanged | |
| exact | 0.50 | 0.50 | still failing | |
| exact | 0.50 | 1.00 | fixed | |
| paraphrase | 0.50 | 1.00 | fixed | |
| paraphrase | 1.00 | 1.00 | unchanged | |
| paraphrase | 1.00 | 1.00 | unchanged | |
| paraphrase | 0.50 | 1.00 | fixed | |
| paraphrase | 0.00 | 0.50 | fixed | |
| acronym | 0.00 | 1.00 | fixed | |
| acronym | 0.50 | 0.50 | still failing | |
| acronym | 0.50 | 1.00 | fixed | |
| acronym | 1.00 | 1.00 | unchanged | |
| acronym | 1.00 | 1.00 | unchanged | |
| typo | 1.00 | 1.00 | unchanged | |
| typo | 0.50 | 0.50 | still failing | |
| typo | 1.00 | 1.00 | unchanged | |
| typo | 1.00 | 1.00 | unchanged | |
| typo | 0.50 | 0.50 | still failing | |
| typo | 0.50 | 1.00 | fixed | |
| multi | 0.67 | 0.67 | still failing | |
| multi | 0.67 | 0.67 | still failing | |
| multi | 1.00 | 1.00 | unchanged | |
| multi | 0.67 | 1.00 | fixed | |
| multi | 0.33 | 0.33 | still failing | |
| multi | 0.50 | 1.00 | fixed |
Remaining misses
Still incomplete at k=5
- Q05 · exactpopulation stability index monitor
Missed KB-24. Top distractor: KB-09 · Population stability index monitor
- Q11 · paraphrasekeep a retrieval change from quietly getting worse
Missed KB-27. Top distractor: KB-29 · Release gate for retrieval changes
- Q13 · acronymPSI drift alerting
Missed KB-24. Top distractor: KB-09 · Population stability index monitor
- Q18 · typoendpont isolation
Missed KB-12. Top distractor: KB-06 · Endpoint detection isolation
- Q21 · typopopultion stability index
Missed KB-24. Top distractor: KB-09 · Population stability index monitor
- Q23 · multicontain an identity takeover
Missed KB-01. Top distractor: KB-10 · Identity compromise procedure
Synthetic judgments. 32 documents and 28 queries. Dense vectors are text-embedding-3-large embeddings (256 dimensions), precomputed offline for every document and labeled query. N-gram vectors are character-trigram TF-IDF. Graded nDCG, MRR, and recall against hand-labeled judgments. Baseline is BM25 without expansion.
PROMOTE. Recall 86.3%, MRR 0.982.
Decisions, evidence, and limits
How it works
Score BM25, character-trigram TF-IDF, dense embeddings, and weighted reciprocal rank fusion against graded relevance judgments. Dense vectors come from text-embedding-3-large at 256 dimensions, computed offline for every document and query by scripts/embed-prism.ts and stored with a hash of their source text, so the page ranks with real embeddings at no runtime cost and the tests fail if the corpus changes without re-embedding. Optional query expansion adds a curated acronym lexicon.
Baseline
BM25 without expansion is the frozen production retriever. Every candidate is compared with that baseline on the same 28 queries. The gate promotes only when recall@k rises by at least three points, MRR does not drop, and no slice regresses by more than ten points.
What is measured
Recall@k, precision@k, MRR, nDCG@k with graded gains, per-slice recall, a leaderboard of all five retrievers, per-query wins and regressions, and the PROMOTE / HOLD verdict. Two findings stand out: dense vectors without the curated lexicon beat BM25 with it, and BM25 fused with dense vectors is the strongest configuration once expansion is on.
Failure modes
The corpus and judgments are small and synthetic. Dense vectors can blur exact rare identifiers, which is why BM25 stays in the fusion. The hybrid weight is hand-set, and a gate on twenty-eight queries cannot certify production quality; it shows the evaluation pattern.
Production tradeoffs
Build a larger judged set from search logs, add a cross-encoder reranker as the next candidate, add statistical paired tests, monitor live recall proxies, and require the same gate in CI before replacing a retriever. Aegis already consumes the promoted hybrid, so a retrieval regression here would show up in its eval.