Perplexity Details Its GPU Embedding Stack: Ivy, Tulip and ROSE
An in-depth look at Perplexity's GPU embedding infrastructure, featuring re-used LLM kernels, Tulip scheduling, ROSE backends, and LazyTensor.
Stock photo for illustration only, not from the actual event
- Perplexity reuses LLM prefill and decode kernels for embedding models without building a separate engine.
- The Tulip scheduler processes sequences first-come, first-served, ideal for sub-billion-parameter models.
- Full-model CUDA graphs eliminate CPU-side kernel launch bottlenecks on small batches.
- LazyTensor enables asynchronous Rust tasks to overlap with CUDA operations efficiently.
The Perplexity team has detailed its GPU-side embedding inference architecture, noting that runtime execution has converged across mature Hopper and Blackwell hardware. Key performance gains stem from the surrounding runtime harness, including CUDA graph management, asynchronous tracking abstractions, and a Rust request path.
Embedding workloads are categorized into two types: batch embedding for vector database creation and indexing where throughput minimizes cost, and online embedding for rapid query-time processing. Scoring tasks operate between these two paradigms.
By reusing prefill and decode kernels from its existing LLM stack, Perplexity avoids the overhead of maintaining a separate embedding engine. Small Transformer embedding models exhibit compute and memory patterns analogous to LLM workloads, making this architectural unification highly efficient.
Stock photo for illustration only, not from the actual event
Perplexity's Tulip scheduler processes sequences on a first-come, first-served basis. Measurements indicate that for small embedding models at targeted sequence lengths, the linear cost of dense layers outweighs the quadratic attention cost, tying latency directly to token count rather than sequence count.
Because CPU-side kernel launching can surpass GPU execution time on small batches, Perplexity constructs whole-model CUDA graphs. To manage capture overhead, the system utilizes lazy capture via eager warmup runs, while LazyTensor tracks pinned host buffers alongside asynchronous CUDA events to decouple CPU preparation from device execution.
"Perplexity benchmarks against vLLM v0.22.0 in BF16 on real weights and eval-derived inputs, with warmup runs verifying cosine similarity divergence within 0.1%."
MarkTechPost
Meanwhile, ROSE supports multiple ragged attention backends including FlashInfer 2, FlashInfer 3, and FlashAttention 4. Selection is handled on a case-by-case basis, with FlashInfer 3 favored for Qwen-based models at extreme sequence lengths, operating without a KV cache to avoid unnecessary padding.
Source: MarkTechPost
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment