Perplexity Open Sources Lily: Rust & Metal Engine for Qwen3.6
Perplexity open-sources Lily, a Rust and Metal inference engine for running Qwen3.6-35B-A3B models on Apple Silicon, achieving 4,156 prefill tokens/s.
Stock photo for illustration only, not from the actual event
- Perplexity open-sources Lily, a Rust and Metal inference engine built for Apple Silicon.
- Tailored for the 4-bit Qwen3.6-35B-A3B model requiring a minimum of 32 GB unified memory.
- Achieves an average of 4,156 prefill tokens/s and 170 decode tokens/s on M5 Max.
- Boosts performance through kernel fusion and direct GPU command buffer execution.
Perplexity has open-sourced Lily, a standalone inference engine and server written in Rust and Metal designed specifically to run large language models on Apple Silicon hardware. A public standalone demo is currently available in the pplx-garden repository on GitHub.
Because the 4-bit quantized checkpoint weighs 19.4 GB, an Apple Silicon Mac with 32 GB or more of unified memory serves as the realistic hardware floor. Perplexity's shipping Hybrid Compute product lists macOS 15+ as a requirement, with 24 GB as a minimum and 32 GB recommended for optimal results.
Stock photo for illustration only, not from the actual event
While the default Mac stack typically relies on MLX and MLX-LM, Lily sacrifices cross-architecture reusability to bundle model structure, execution plans, and kernel selection into a single optimized runtime, significantly cutting down operational overhead.
The Qwen3.6-35B-A3B model houses 35 billion parameters while activating roughly 3 billion per token. It mixes 10 full-attention layers with 30 Gated DeltaNet layers and utilizes groupwise affine 4-bit quantization, compressing 70 GB of bfloat16 weights down to 19.4 GB. Lily reconstructs these weights tile by tile inside threadgroup memory without ever spilling the expanded array back to unified memory.
The introduction of specialized engines like Lily highlights the industry shift toward hardware-specific domain runtimes. On Apple Silicon, where unified memory architecture bridges CPU and GPU, eliminating round-trips and managing cache at the threadgroup level are critical strategies for accelerating Mixture of Experts (MoE) models.
Tested on a 40-core, 128 GB M5 Max at batch size 1 against MLX-LM, Lily averaged 4,156 prefill tokens/s compared to 3,388 (1.23x speedup) and 170.0 decode tokens/s versus 126.4 (1.35x speedup), while maintaining a teacher-forced perplexity difference of just 0.04%.
Source: MarkTechPost
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment