Meet Needle 2: An Open 45M-Parameter Tool-Calling Model That Ships as a 14MB Binary
Cactus introduces Needle 2, a compact 45M tool-calling model designed to run efficiently offline with minimal RAM consumption.
Stock photo for illustration only, not from the actual event
- Needle 2 features 45 million parameters and ships as a 14MB prebuilt binary.
- Maintains a steady RAM usage of around 28MB regardless of conversation length.
- Utilizes a Simple Attention Network architecture without FFN layers.
- Achieves 98.3 function-name accuracy on Mobile Actions benchmark splits.
The artificial intelligence community is buzzing with the release of Needle 2, a compact tool-calling model developed by the Cactus team. The model is distributed as prebuilt binaries and static libraries supporting a wide range of platforms including macOS, Linux across x86-64, ARM64, ARMv7, RISC-V, and MIPS32el, as well as Windows, Android, iOS, watchOS, tvOS, and WebAssembly. Furthermore, Cactus reports that Pebble already runs Needle locally within the Index 01 application for offline voice actions.
Under the hood, the Cactus team implemented what they term a Simple Attention Network. This recipe replaces traditional FFN layers with a Hadamard MLP, retains GQA attention, integrates engram key-value memory sourced from hashed n-gram tables, and applies multi-lane hyper-connections. The network consists of 27 layers with a width of 512, supported by underlying research published on arXiv as A Controlled Study of Attention-Only Transformers. Pretraining relied on a proprietary 115B-token corpus accompanied by 38B tokens of post-training, contrasting with models like LFM2.5-230M which were pretrained on 19 trillion tokens.
Efficiency is a core pillar of Needle 2, consuming only 70 MFLOPs per token with 35 million out of 45 million parameters being matmul-active. For comparison, LFM2.5 230M utilizes 460 MFLOPs, FunctionGemma 270M requires 540 MFLOPs, and Apple FM approaches 6,000 MFLOPs. Model weights never decompress into RAM; instead, 2-bit codes expand within vector registers and fuse directly into integer dot products, keeping the arithmetic path strictly int8. Upon startup, a single binary probes the CPU to select the optimal kernel tier among SDOT, NEON, AVX2, RISC-V vectors, wasm SIMD, or scalar.
Stock photo for illustration only, not from the actual event
To enforce structural integrity, a byte-level grammar compiled from JSON schemas constrains every emitted token. Because the matcher identifies legal tokens before logits are generated, the engine bypasses up to 98% of vocabulary projection on structural tokens. Attention mechanisms utilize a 256-token sliding window while system turns and tool declarations remain pinned as KV sinks, keeping memory consumption anchored near 28MB irrespective of how long the conversation extends.
The ability to pack a capable tool-calling model into a 14MB binary while capping RAM usage at 28MB marks a major milestone for edge AI deployment. By processing data locally without cloud dependency, devices can achieve ultra-low latency and heightened user privacy. The combination of weight quantization and grammar-constrained decoding demonstrates sophisticated systems engineering tailored for resource-constrained hardware.
When handling tool declarations, declaring five or fewer tools renders them directly, while exceeding five triggers a contrastive retrieval head that embeds each schema once, scores queries per turn, and admits only the top five candidates. Unselected tools remain entirely unreachable rather than merely improbable. Every response includes a calibrated confidence value derived from a post-hoc head and decoding probability, establishing a strict performance threshold where actions proceed above it while off-topic requests return an empty call.
"Needle 2 leads both Seal-Tools splits and posts 98.3 function-name accuracy on Mobile Actions."
Cactus Team
The Cactus team evaluated their model across five public function-calling benchmarks using ordered strict exact match criteria, where names, call order, and all arguments must align perfectly. Needle 2 outperformed competitors across both Seal-Tools splits and achieved a 98.3 function-name accuracy score on Mobile Actions. Although it trailed on BFCL v4—a discrepancy attributed by Cactus to dataset distribution differences favoring consumer device actions over enterprise APIs—it maintained a well-formed output rate of 93.4 across all 3,641 BFCL rows.
Source: MarkTechPost
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment