Does the shipped hybrid retriever beat simpler ones?
Hybrid search (semantic + keyword) with a reranker, the default I shipped, retrieves the right page more often than any single method.
- Kind
- Comparison
- State
- Done
- Run on
- 2026-09-26
- Code / data
- /alex-os-evaluation
Why I ran it
It was on by default and had never been measured.
Setup & method
31 hand-labelled questions against the site's real content (140 items). Five configurations, scored on recall@1, recall@3, MRR and nDCG@5 with the retrieval-eval harness.
What happened
The shipped default came last on every ranking metric. Semantic search alone and semantic with a reranker scored highest.
Result (one line)
Hybrid + rerank, the shipped default, scored worst: MRR 0.882 against 0.925 for semantic search alone.
Results table
What surprised me
Keyword search did far better than I expected, which said more about my test questions than about keyword search.
Limits — what this does not prove
31 questions, mostly title lookups, which favours keyword matching. The differences are a few questions either way, so this is not enough to crown a winner; a harder graded set comes before changing the default.