Technology
Keenable developers introduce NEEDLE, a live benchmark for agentic search engines
The continuous evaluation tool aims to prevent overfitting and data leakage, which developers say corrupt standard static search benchmarks.
The short version
- Developers from search startup Keenable have launched NEEDLE, an open-source, live benchmark designed to measure search engine quality for AI agents.
- The benchmark continuously runs in public, updating its query sets hourly or daily with fresh data to prevent AI models from memorizing static answer keys.
- Initial evaluations on NEEDLE highlight performance gaps in AI-targeted search queries, particularly when comparing traditional human-optimized search engines with agent-optimized indexes.
Key facts
- Keenable developers Ilya Gusev, Matthias Petri, and Andrey Styskin introduced NEEDLE, an open-source, continuous benchmark for agentic search engines.[Hacker News]
- Static benchmarks are highly vulnerable to overfitting and data leakage because AI agents can access test datasets and answer keys directly from repositories like HuggingFace during evaluations.[Hacker News]
- NEEDLE generates continuous, fresh queries across five categories: News, Finance, Scholar, AgenticRare, and Legal, sourcing data from RSS feeds, APIs, and public agent search logs.[Hacker News]
- The benchmark compares Keenable's performance against traditional search engines like Google, Bing, and Brave, alongside AI search startups such as Tavily, Parallel, and Exa.[Hacker News]
- According to the authors, traditional search engines fall short for AI agents because they optimize for human preferences like video links and dwell time, whereas agents require raw, authoritative factual extraction.[Hacker News]