LLM Wiki vs. RAG: What Each One Is and Benchmarks
A benchmark comparing LLM Wiki and RAG on a 1,176-line Python library reveals wikis are slower and costlier, yet win on unrecorded decisions.
Stock photo for illustration only, not from the actual event
- LLM Wikis are markdown page folders written and maintained by an agent with citations.
- RAG retrieves document chunks into prompts without writing anything back.
- On a small 1,200-line codebase, using a wiki is slower and more expensive than no wiki.
- Wikis only win when answering questions about decisions never written in source code.
In software development, people often treat "LLM wiki" and "RAG" as two ways of achieving the same goal. In reality, they serve different functions. A recent experiment built a wiki for a real codebase and tested four identical questions with and without it. The results showed that operating without a wiki was faster, cheaper, and just as accurate, except for a fifth question whose answer never existed in the source files at all, highlighting the core distinction between the two approaches.
To define the terms properly, RAG (retrieval-augmented generation) is when a retriever finds the most relevant document chunks from a document store upon a query and pastes them into the prompt for the model to answer. The store remains a static copy of sources with no write-back mechanism. Meanwhile, agentic retrieval is what most coding agents use today without a vector database, where the agent runs shell tools to grep and read files in a loop until it finds the answer, matching how Claude Code operates on a bare repository.
Stock photo for illustration only, not from the actual event
An LLM wiki consists of a folder of markdown pages written and maintained by an agent, featuring one page per concept, decision, or module, complete with source citations and internal wikilinks. When a question cannot be answered from these pages, the agent reads the source files and writes down what it learned so subsequent queries can skip them. The benchmark targeted pallets/itsdangerous, the signing library under Flask sessions, consisting of 1,176 lines of Python across 8 modules, a 292-line CHANGES.rst file, and 7 documentation pages.
Testing began with four typical integration questions answered against the codebase sources, yielding correct outputs with valid citations. Building the wiki via initialization prompts produced varying results across runs, demonstrating that wiki builds are non-deterministic. Querying the wiki read the index and relevant pages, falling back to sources only when necessary while updating its notes for future use.
"RAG retrieves what was written. A wiki also records what was decided."
Vellum Labs
Deeper analysis reveals that on a 1,200-line codebase, a wiki functions as a worse RAG than having no RAG at all. The initial pass is 55% slower and 34% more expensive per question. Even after write-backs, repeat queries cost roughly the same as reading cold source files, meaning the retrieval savings never amortize the initial build cost on this scale.
The turning point arrives with the fifth question concerning a team review decision—such as keeping at most three secret keys and dropping the oldest every 30 days—which was never documented in source code, changelogs, or documentation. Because RAG and agentic retrieval can only find what someone previously wrote down, only a tool with a capture command like a wiki can record and retrieve such decisions, making it the deciding factor between a system that can answer and one that cannot.
Source: Dev.to
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment