Meet LEANN: Open-Source Vector Index Shrinks RAG to 6GB
UC Berkeley introduces LEANN, an open-source vector index using a pruned graph and on-the-fly embedding recomputation to shrink a 60M-chunk RAG index to 6GB.
Stock photo for illustration only, not from the actual event
- UC Berkeley unveils LEANN open-source vector index
- Reduces a 60M-chunk RAG index from 201GB to 6GB
- Utilizes a pruned graph and recomputes embeddings at query time
- Matches HNSW accuracy at 90% recall with seconds-level search
Researchers at UC Berkeley have introduced LEANN, an open-source vector index engineered to address the heavy memory overhead associated with large Retrieval-Augmented Generation (RAG) systems. This innovative approach successfully shrinks a massive 60 million-chunk RAG index from its original 201 GB footprint down to just 6 GB.
The core mechanism behind LEANN relies on storing a pruned graph combined with recomputing embeddings dynamically at query time. Instead of keeping all vectors resident in memory like traditional methods, this architecture drastically slashes hardware resource requirements without sacrificing core retrieval effectiveness.
Stock photo for illustration only, not from the actual event
In terms of performance benchmarks, LEANN achieves HNSW accuracy at a 90% recall rate while utilizing only 4 GB of memory, a sharp contrast to the 188 GB previously required under similar configurations.
The introduction of index structures like LEANN highlights the broader industry push to mitigate infrastructure costs in AI deployments. As vector databases scale to accommodate massive enterprise knowledge bases, memory consumption remains a primary bottleneck. While the trade-off involves seconds-level search latency due to on-the-fly recomputation—making it less ideal for sub-second conversational bots—it presents a game-changing solution for deep data analytics and large-scale batch processing where slight latency is an acceptable compromise for massive storage savings.
The primary trade-off for achieving these unprecedented space savings is search latency, which shifts to a seconds-level timeframe due to the computational overhead of recomputing embeddings during queries. Nevertheless, this efficiency leap makes large-scale RAG deployments vastly more accessible and cost-effective for organizations with limited memory infrastructure.
Source: MarkTechPost
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment