Boosting RAG Accuracy to 81% With Four Retrieval Fixes
A developer improved local RAG search for coding agents on EnterpriseRAG-Bench, raising correct answers from 66.3% to 81.0% across 499 questions.
Stock photo for illustration only, not from the actual event
- RAG is a local search tool designed for coding agents, running via command line and MCP server.
- The tool was tested on Onyx's public EnterpriseRAG-Bench dataset containing over 100,000 documents.
- Four retrieval adjustments increased correct answers from 66.3% to 81.0% on 499 benchmark questions.
- Source code, experiment logs, and exact figures are openly shared on GitHub for developers.
RAG operates as a local search utility tailored for coding agents. It indexes designated files, executes exact-word searches (BM25) alongside semantic searches, and retrieves relevant passages complete with file names and line numbers. The tool runs seamlessly through the rag command line interface and functions as an MCP server, allowing Claude Code, Codex, and other agents to query it directly.
Installation is straightforward via package managers using commands such as uv tool install ragmcp-cli, pipx install ragmcp-cli, or npm install -g ragmcp-cli. Users can then establish connections via rag connect claude or integrate it with codex, gemini, and copilot. This technical overview focuses specifically on measuring and optimizing retrieval performance.
The developer evaluated rag against a public benchmark dataset consisting of mock corporate documentation including Slack threads, emails, tickets, general documents, and source code. By reviewing failures from the initial benchmark run (rag 0.2) individually, it became apparent that two primary root causes accounted for the vast majority of missed answers.
Stock photo for illustration only, not from the actual event
Evaluating Retrieval-Augmented Generation (RAG) frameworks against standardized enterprise benchmarks provides critical insights into how coding agents ingest local codebases. Systematically tracking failure modes and validating performance increments through rigorous statistical testing—such as McNemar's test—establishes a reliable methodology for iterative software development in AI tooling.
Each sequential modification was evaluated against the prior baseline on identical questions utilizing a paired McNemar test. Step 1 successfully resolved 57 questions while introducing 13 regressions, whereas step 2 fixed 35 questions and caused 6 regressions. Both outcomes proved statistically significant well beyond random chance (p < 0.001).
While certain experimental iterations yielded poorer performance, all quantitative records were preserved for transparency. Furthermore, exact-word search operates independently without requiring accounts or API keys, returning optimal passages and line references directly to consuming agents. Users authenticated with Codex via ChatGPT plans or utilizing OpenRouter API keys can additionally leverage the system to synthesize fully generated answers.
Source: Dev.to
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment