FreeToken Runs 753B MoE Model on Workstation GPU
FreeToken, an edge-native MoE serving engine released on PyPI and GitHub, enables single workstation hardware to run massive models.
Stock photo for illustration only, not from the actual event
- FreeToken is an edge-native MoE engine running large models on a single workstation GPU
- Achieves 77-83 tok/s on Qwen3.6-35B and 22-25 tok/s on DeepSeek-V4-Flash using an RTX 5090
- Exposes OpenAI- and Anthropic-compatible API endpoints on port 1919
- Ideal for solo developers and enterprises requiring strict data privacy
The artificial intelligence community welcomes FreeToken, an edge-native Mixture-of-Experts (MoE) serving engine designed to make running large-scale language models on a single workstation hardware practically feasible. The software is available under the Apache-2.0 license on GitHub, published on PyPI as freetoken v0.1.2, and also shipped as a one-click desktop application for Windows and Linux at flashml.ai.
Regarding command-line interface (CLI) usage, it targets Linux x86_64 systems equipped with an NVIDIA GPU running driver r580+ (CUDA 13). The ft serve command exposes endpoints on port 1919 that are compatible with OpenAI and Anthropic specifications, while the ft launch claude command hooks up tools like Claude Code, Codex, OpenCode, or OpenClaw directly to the user's local machine.
Mixture-of-Experts architecture makes local frontier inference arithmetically feasible. For instance, DeepSeek-V4-Flash activates only 6 out of 256 routed experts across each of its 43 layers, resulting in just 13 billion of its 284 billion parameters participating in any single token. Despite this sparsity, the full expert pool in FP4 format is roughly 140 GB, meaning inactive experts reside in host memory and enter the execution path dynamically on demand.
Stock photo for illustration only, not from the actual event
Mixture-of-Experts (MoE) architecture tackles the memory bottleneck of running massive language models by selectively activating only the relevant subset of parameters per token. This computational efficiency allows developers to run frontier-class models on local desktop hardware or cost-effective workstations rather than expensive server clusters.
In terms of performance benchmarks, FreeToken running on an RTX 5090 sustains between 77 and 83 tokens per second on Qwen3.6-35B-A3B (BF16), and 22 to 25 tokens per second on DeepSeek-V4-Flash (MXFP4), reaching 1.5 to 2.3 times the speed of the strongest baseline. Furthermore, it maintains lower global LRU cache miss rates compared to competing engines under equal cache capacities.
The primary target audience includes solo developers, startups, and SMB engineering teams whose AI agent token expenses already surpass the hardware cost of owning a GPU. Enterprises can leverage it as an air-gapped or regulated-workload path, with top industry fits spanning healthcare, legal, finance, and IP-heavy research and development sectors where keeping data strictly localized is paramount.
Source: MarkTechPost
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment