StepFun Launches Step 5 Preview 600B MoE Model
StepFun introduces Step 5 Preview, a 600B-parameter MoE model with 27B active parameters per token and a 1 million context window.
Stock photo for illustration only, not from the actual event
- Step 5 Preview is a sparse Mixture-of-Experts model with 600B total parameters.
- It activates approximately 27 billion parameters per token, about 4.5% of the total.
- The model supports a 1-million-token context window for long-horizon agentic workflows.
- Open weights are scheduled for release on October 15, 2026.
StepFun has officially announced the launch of Step 5 Preview, an advanced model tailored for complex, long-horizon agentic workloads. Built on a sparse Mixture-of-Experts (MoE) architecture, the model boasts roughly 600 billion total parameters while activating approximately 27 billion parameters per token, which accounts for about 4.5% of the model's total weight per token.
Rather than widening the neural network, the research team stacked 92 Transformer layers in a narrow-and-deep configuration. According to reports, this deeper structural layout provides extended pathways for implicit multi-hop reasoning, proving vital during extensive prefill phases when agents perform web searches, execute code, and process tool returns.
The training pipeline heavily relies on on-policy, long-horizon reinforcement learning. StepFun also cited bit-wise train and inference alignment across MoE routing, alongside other optimizations such as MTP-3 speculative decoding, FP8 MoE quantization, and KV-cache offloading, delivering over a 3x end-to-end speedup for long-horizon tasks.
Stock photo for illustration only, not from the actual event
Mixture-of-Experts (MoE) architecture represents a crucial paradigm shift in scaling large language models efficiently. By routing tokens to specific subnetworks and activating only a fraction of total parameters per computation, MoE models achieve massive capacity without incurring the proportional inference latency of dense models of the same overall size.
Regarding deployment options, the model is currently available through a hosted API and on the StepFun platform. Self-hosting enthusiasts must wait for the open weights release, which StepFun has slated for October 15, 2026. Basic arithmetic suggests that 600 billion parameters require roughly 1.2 terabytes in BF16 precision before accounting for KV caches, requiring multi-GPU server setups.
Source: MarkTechPost
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment