Keenable AI Opens Sources NEEDLE Live Search Benchmark
Keenable AI open-sources NEEDLE, a live search benchmark rebuilding its query set hourly across five verticals, detailing benchmark scores for major search engines.
Stock photo for illustration only, not from the actual event
- Keenable AI open-sources NEEDLE, a live search benchmark generating a new query set every hour.
- Covers five core verticals including news, everyday, expert, deep-tail, and legal evaluation.
- Employs diverse scoring methods such as LLM judges, answer recall, and identifier matching.
- Data shows finance is nearly solved, while deep-tail queries remain a major challenge for agents.
Keenable AI has open-sourced NEEDLE, an evaluation harness designed for live search performance that reconstructs its entire query set on an hourly basis. Operating as a Python CLI installed via uv sync, it relies on two subcommands—generate and run—running seamlessly on laptops or within continuous integration pipelines while requiring an OpenRouter key for evaluation and individual API keys for each tested search engine.
The acronym NEEDLE stands for News, Everyday, Expert, Deep-tail, and Legal Evaluation, where each vertical is structured to emulate distinct agent intents. For instance, the news vertical pulls breaking items from approximately 124 curated RSS feeds alongside Google Trends into keyword queries, while the finance vertical gathers registry facts from Wikidata and GLEIF together with single-quarter 10-Q disclosures from SEC XBRL, and the legal vertical extracts recent opinions across 14 federal courts from CourtListener.
Photo by Teemu Paananen / Unsplash
Scoring methodologies align directly with the nature of each task. Because news and deep-tail queries lack a single definitive answer, an LLM judge evaluates results on a scale from 0 to 4 while the harness reports nDCG@5 with a duplicate-URL penalty. Finance relies on answer-recall@5 to check if the fact reaches the agent within top results, whereas scholarly and legal tasks depend on known-item identifier matching while engines are evaluated sequentially to ensure comparable latency metrics.
A particularly insightful metric provided by the framework is the ultimate ceiling, which aggregates results returned by every participating engine into a synthetic oracle engine ordered by relevance. This establishes an empirical baseline derived from the collective retrieval capability of the entire field, allowing researchers to distinguish between ranking failures and market-wide retrieval limitations.
Rebuilding evaluation query sets every hour effectively counters benchmark data contamination, a persistent issue where AI models memorize static evaluation sets. By utilizing live data streams, the framework offers a rigorous assessment of how search engines handle dynamic, real-world agent tasks.
Published 7-day mean figures ending August 28, 2026, indicate that finance is nearing completion, with Exa scoring 0.910, Keenable at 0.872, and Perplexity at 0.871 against a 0.965 ceiling. Conversely, deep-tail queries remain the closest mirror to real agent traffic and the hardest to solve, where Exa leads at 0.557 and Bing trails at 0.199, while latency tests show Keenable-realtime achieving 193 milliseconds at p50 compared to 1,876 milliseconds for Exa.
Source: MarkTechPost
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment