Comparing AI Code Search Tools: Ripgrep, Serena, and More
Benchmarking 4 code search tools for AI agents on Flask 3.1.3, comparing speed, setup complexity, and accuracy by Semlith developer.
Stock photo for illustration only, not from the actual event
- Benchmarked 4 code search tools (Ripgrep, Serena, claude-context, Semlith) on Flask 3.1.3
- Ripgrep excels at exact identifier searches but fails with natural language queries
- claude-context indexing took roughly 21 minutes running on CPU via Docker
- Semlith developer tool took 84.3 seconds to index and supports PDF, Word, and Excel files
There are numerous ways to help coding agents find code efficiently today, ranging from plain text grep, language servers behind MCP, vector databases, to local indexes. Because every project claims its approach is superior, the creator of Semlith decided to test them all on the exact same codebase to see how they perform.
The benchmarking environment utilized Flask version 3.1.3 running on an M1 MacBook Air with 8 GB of RAM. The tested tools included ripgrep 15.2.0, Serena 2.1.0, claude-context 0.1.15 (integrated with Milvus 3.0.3 and Ollama running the nomic-embed-text model), and semlith 0.50.0, cloned via git clone --depth 1 --branch 3.1.3 https://github.com/pallets/flask
When searching with a known identifier, such as looking up max_content_length using ripgrep, the tool instantly returned eight correct lines within milliseconds. However, when querying the same concept using natural language without knowing the exact name, standard text search failed to find the target because wrappers.py never explicitly mentions those words, forcing the agent to guess and open multiple files.
Stock photo for illustration only, not from the actual event
Serena provides symbol tools backed by actual language servers supporting over 40 languages. Operating as a REPL since version 2, it allows agents to write brief Python scripts interacting with Serena's API. Setup involves creating a virtual environment and installing via uv, with project indexing taking 30.9 seconds for high precision when names are known. Without explicit names, it falls back to regex pattern matching. Notably, Serena stands out by supporting code edits, such as replacing function bodies or renaming symbols across files.
This benchmarking comparison highlights that no single tool is universally ideal for every workflow. The optimal choice depends entirely on development needs: exact text search remains unbeatable for known identifiers, whereas vector embeddings and local indexes become crucial when AI agents need to understand semantic meaning from natural language descriptions.
claude-context, Zilliz's MCP server, chunks code, generates embeddings, and stores vectors in Milvus with hybrid search capabilities. Running a fully local setup required deploying Milvus via Docker Compose and configuring Ollama while capping CPU usage at four cores to prevent system locks. Although status logs reported the codebase fully indexed at 92 seconds, actual server logs finished at 1,266.9 seconds, meaning agents searching early would query an incomplete index unknowingly. The 21-minute duration reflected CPU-only embedding inside Docker rather than optimized native hardware.
Meanwhile, Semlith indexed 211 files (comprising 2,778 chunks and 5 images) in 84.3 seconds. While slower than Serena due to embedding overhead, it successfully processed documentation formats including .rst docs, PDF, Word, and Excel files which other tools skipped by default.
Source: Dev.to
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment