Alibaba Releases Qwen3.8-Flash-Next 125B MoE Model
Alibaba's Qwen team has released Qwen3.8-Flash-Next, a 125B multimodal MoE model featuring 6B active parameters per token, previewing the upcoming Qwen4 architecture.
Stock photo for illustration only, not from the actual event
- Alibaba launches Qwen3.8-Flash-Next, a 125B multimodal MoE with 6B active parameters
- Features 512 experts in the MoE layer and supports up to 1M native context with YaRN
- Training costs reduced to roughly one-ninth compared to the previous Qwen3.7-Plus
- Comes with default thinking mode and broad framework support including vLLM and SGLang
Alibaba's Qwen team has officially rolled out its latest advanced model, Qwen3.8-Flash-Next, serving as a direct preview of the upcoming Qwen4 architecture. The new model combines a 125B main model, 51B N-gram embedding parameters, and a 4B multi-token prediction module, resulting in a total disk footprint of 180 billion parameters.
Despite its massive total size, the model activates only 6 billion parameters per token through sparse activation techniques, which cuts down compute requirements without sacrificing storage. Running the model requires substantial hardware resources: the FP8 checkpoint takes 172.78 GiB and the BF16 version requires 335.28 GiB. According to vLLM recipes, TP2 is the minimum validated FP8 configuration on GB300 chips, with TP4 recommended, while an 8×H200 node requires TEP8.
Stock photo for illustration only, not from the actual event
Four primary structural updates drive the model's performance. Its Mixture of Experts (MoE) layer carries 512 total experts, activating 10 routed experts alongside 1 shared expert at an intermediate dimension of 640. Furthermore, the model handles a native context length of 262,144 tokens, which is seamlessly extensible up to 1,000,000 tokens using YaRN technology.
In benchmark evaluations, Qwen3.8-Flash-Next achieved 58.7 on DeepSWE 1.1, 62.5 on SWE-bench Pro, 81.0 on SWE-bench Multilingual, and 91.9 on LiveCodeBench v6. For agentic capabilities, it scored 73.9 on CoWorkBench, 55.7 on JobBench, and 73.5 on Toolathlon Verified. Multimodal test results include 84.5 on AndroidWorld, 76.6 on LVBench, 88.5 on RealWorldQA, and an impressive 95.7 on MathVision when paired with a code interpreter.
The widespread adoption of Mixture of Experts (MoE) architectures paired with sparse activation highlights an industry-wide shift toward balancing immense model capacity with operational efficiency. By scaling total parameter counts while keeping active computation per token low, developers can achieve breakthrough speeds and reasoning capabilities without triggering exponential hardware costs during inference.
However, the model does not lead across every single category. Frontier reasoning benchmarks show Claude Opus 4.6 (Max) leading HLE at 40.0 compared to Qwen's 35.9, while DeepSeek-V4-Flash-0731 outpaces it on NL2Repo-Bench with 54.2 versus 48.1. On the economic front, Alibaba reports that training costs dropped to roughly one-ninth of Qwen3.7-Plus, while serving speeds demonstrate QSA kernel speedups of up to 7.6× for prefill and 4.9× for decoding at 1M tokens.
For deployment, the model integrates natively with serving engines like vLLM, SGLang, TokenSpeed, transformers serve, and llama.cpp for GGUF quantizations. Fine-tuning is fully supported through frameworks such as Unsloth, Swift, and LLaMA-Factory. It already powers the Standard mode on QwenWork and operates directly alongside Qwen Code, with thinking mode enabled by default across customizable reasoning effort levels.
Source: MarkTechPost
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment