AMD Releases Instella-MoE-16B-A3B: A Fully Open Mixture-of-Experts LLM
AMD introduces Instella-MoE-16B-A3B, a fully open-source Mixture-of-Experts LLM featuring 2.8B active parameters and optimized for Instinct GPUs.
Stock photo for illustration only, not from the actual event
- Instella-MoE is a 16B parameter Mixture-of-Experts model with 2.8B active parameters.
- It incorporates structural choices like Gated MLA and FarSkip-Collective to accelerate processing.
- The training corpus spans 7.1T tokens with a context window extended up to 64K tokens.
- The base checkpoint achieves an average score of 76.7, leading fully open-source models.
AMD has officially released Instella-MoE-16B-A3B, a fully open-source Mixture-of-Experts (MoE) large language model trained on its Instinct GPUs. The model features a total of 16 billion parameters while utilizing 2.8 billion active parameters per forward pass.
Regarding licensing, the model weights are shipped under a ResearchRAIL license strictly for academic and research purposes, meaning it is not a drop-in commercial model. However, the training codebase is released under the more reusable MIT license.
Stock photo for illustration only, not from the actual event
Architecturally, Instella-MoE is a decoder-only MoE featuring 27 layers, a hidden size of 2048, 16 attention heads, and a 128,896-token vocabulary. Each MoE layer utilizes 2 shared experts alongside 6 routed experts selected from a pool of 64, applying a Multi-Token Prediction objective during pre-training and mid-training.
Two critical structural choices drive the model's efficiency. Gated MLA introduces a lightweight learned output gate to Multi-head Latent Attention, while FarSkip-Collective passes outdated and partial activations into the MoE and attention layers to overlap expert-parallel communication with computation. AMD reports a 12.7% pre-training speedup and up to a 39.2% reduction in time to first token when serving with expert parallelism.
Mixture-of-Experts (MoE) architecture represents a pivotal shift in modern LLM design, allowing models to scale their total parameter count for vast knowledge retention while keeping active compute costs remarkably low per token. AMD's release of Instella-MoE paired with its hardware optimizations highlights a strategic push to empower developers and researchers with high-performance, cost-effective open AI infrastructure.
Pre-training covers 7.1 trillion tokens drawn from open corpora including Nemotron-CC-v2, MegaMath, FineMath, RefineCode, and TxT360. This is followed by mid-training phases and a long-context stage extending the sequence window from 4K to 64K using YaRN, increased RoPE theta, and document masking.
In benchmark evaluations, the base checkpoint achieves an average score of 76.7, ranking as the strongest among fully open models—outperforming Moonlight-16B-A3B (76.2) and OLMo-3-7B (70.1). It also records 86.5 on WinoGrande and 65.7 on HumanEval+.
Source: MarkTechPost
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment