Meet Redis LangCache to Cut LLM API Costs by 90%
Redis launches LangCache on Redis Cloud as a public preview, cutting LLM API costs by up to 90% with 15x faster response times.
Stock photo for illustration only, not from the actual event
- Redis has announced LangCache as a public preview on Redis Cloud
- Cuts LLM API costs by up to 90% and boosts speed up to 15x
- Works seamlessly with any LLM provider and programming language
- Includes customizable similarity thresholds, TTLs, and eviction policies
Redis has officially introduced Redis LangCache, a managed semantic cache engineered to help developers and enterprises slash Large Language Model (LLM) API expenses while significantly accelerating response times. The new service is currently available as a public preview on Redis Cloud, accessible via a REST API accompanied by Python and JavaScript SDKs.
In standard customer support or AI assistant workflows, users frequently phrase inquiries differently while asking the exact same fundamental question. Without a semantic cache, every single variant triggers a complete, redundant generation cycle encompassing input token processing, output token decoding, and unnecessary waiting times for the user.
Traditional prefix caching only mitigates a fraction of those expenses. While prefix caching allows inference engines to reuse computed KV states for shared system prompts or contexts, the request still reaches the LLM, new tokens still undergo processing, and full answers still require decoding. A prefix-cache hit represents a cheaper generation call rather than an entirely avoided one.
Stock photo for illustration only, not from the actual event
LangCache re-architects this flow by positioning the cache externally from the model, directly storing the finalized generated response. The service manages embedding generation automatically—supporting default models or custom user-provided ones—while cache behavior is governed by adjustable similarity thresholds, time-to-live (TTL) settings, eviction policies, and adaptive precision controls.
During a comparative demo run evaluating paraphrased queries, direct inference took 2.232 seconds and consumed 514 input tokens alongside 250 output tokens. In contrast, LangCache retrieved the prior response in just 0.37 seconds with zero LLM input or output tokens consumed, achieving approximately a 6x speedup in that specific execution.
The introduction of semantic caching mechanisms like Redis LangCache addresses a critical scaling bottleneck for modern AI applications, where inference costs scale linearly with user adoption. By intercepting semantically equivalent requests before they reach the foundation model, organizations can drastically curtail computational overhead. However, engineering teams must carefully calibrate similarity thresholds in production to balance cache efficiency and prevent mismatched answers across nuanced queries.
Redis's public preview announcement highlighted performance metrics of up to 15x faster responses and up to 70% lower token usage, whereas the current product page notes overall savings reaching up to 90%. Early adopters such as Mangoes.ai have reported a 70% cache hit rate on their patient-care voice application, successfully reducing their total LLM expenditures by 70% and accelerating responses fourfold.
Source: MarkTechPost
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment