Meta Releases Llama 4 Scout Open-Weights 105B Model

Meta's Fundamental AI Research (FAIR) team has officially rolled out Llama 4 Scout, marking the open-source community's initial frontier-class foundation model engineered natively on dynamic Mixture-of-Depths or MoD routing. Boasting a total parameter capacity of 105 billion alongside just 24 billion active parameters per token pass, Llama 4 Scout achieves complete parity with leading closed-source reasoning engines across synthetic coding, advanced mathematics, and long-context synthesis benchmarks while executing efficiently on enterprise dual-GPU setups.

Standard dense transformer architectures allocate an identical compute budget to every single token, evaluating routine punctuation and filler words with the exact same 80-layer depth assigned to complex mathematical proofs. Llama 4 Scout discards this uniform resource distribution by embedding a learned top-k routing gate directly into each transformer block.

Stock photo for illustration only, not from the actual event

During runtime inference, the internal router evaluates token complexity and dynamically assigns computational resources. Simple tokens bypass deep self-attention and feed-forward layers via residual shortcut paths, whereas highly informative tokens secure maximal compute across specialized reasoning blocks. This dynamic depth allocation reduces generation latency by 58% compared to standard dense 70B models while expanding total model capacity to 105 billion parameters.

Regarding architectural mechanics, the model integrates RingAttention coupled with sliding-window multi-head latent attention or MLA. Official FP8 and INT4 AWQ quantization checkpoints are directly accessible for download via Hugging Face, requiring a mere 24GB VRAM inference footprint in 4-bit mode for single GPU execution.

Stock photo for illustration only, not from the actual event

Addressing the challenge of maintaining native 1M-token context windows without performance loss, previous open-weights models claiming multi-hundred-thousand token support often relied on post-training RoPE interpolation techniques like YaRN, which triggered sharp perplexity spikes beyond 64K tokens. Conversely, Llama 4 Scout underwent pre-training from step zero utilizing an 8-stage progressive context expansion curriculum scaling up to 1,048,576 tokens.