Liquid AI Open-Sources Pipette Benchmarking Suite

Liquid AI has made waves in the artificial intelligence community by officially releasing Pipette, an open-source benchmarking suite designed to evaluate on-device AI model performance. Distributed under the Apache 2.0 license, the infrastructure includes pipette-mgmt, pipette-clients, and pipette-scores, alongside a public results dataset, a hosted dashboard, and native iOS and Android benchmark apps with no waitlist required, though community-submitted results remain in beta.

The release was developed in partnership with Artificial Analysis, an independent validator that reviewed and verified the benchmarking methodology. The core premise is both narrow and practical: on-device behavior is a property of the deployed system rather than the model in isolation. The initial launch dataset covers five on-device performance metrics across more than 1,000 configurations combining models, quantization formats, runtimes, devices, and context lengths.

The suite supports over 30 models, multiple quantization formats, llama.cpp builds for macOS, iOS, Windows, and Android, and context lengths ranging from 256 to 8,192 tokens. Initial published results stem from high-end hardware including an M5 Max MacBook Pro, an iPhone 17 Pro, and a Galaxy S26 Ultra, with benchmarks for the AMD Ryzen AI Max+ 395 and Radeon 8060S marked as coming soon.

Stock photo for illustration only, not from the actual event

Within Pipette, the core unit of measurement is a deployment configuration comprising model, quantization, runtime, and device. Benchmarks then evaluate specific metrics and token shapes to produce latency, throughput, or memory outcomes. Model quality is tracked separately via IFBench, GPQA Diamond, and MATH-500, with quality scores derived from llama.cpp evaluation runs on NVIDIA H100 80GB reference systems before being matched to on-device runs sharing identical models and quantization.