Baidu Releases ERNIE 5.5 Flash Base 280B MoE Model
Baidu launches ERNIE 5.5 Flash Base, a 280B sparse MoE base model with 14B active parameters, trained on 33.5T tokens.
Stock photo for illustration only, not from the actual event
- Baidu introduces the new ERNIE 5.5 Flash Base AI model
- Features a sparse Mixture of Experts architecture totaling 280B parameters
- Utilizes 14B active parameters per computation task
- Trained extensively on a massive dataset of 33.5T tokens
Tech giant Baidu has made a significant advancement in artificial intelligence by officially releasing ERNIE 5.5 Flash Base, a new large language model built on a Mixture of Experts (MoE) architecture designed for high-speed performance and agility.
This newly released model stands out with a total parameter count of 280 billion (280B), while intelligently routing workloads to activate only 14 billion parameters (14B Active Parameters) during actual inference. This sparse design dramatically reduces computational overhead while maintaining robust processing capabilities.
Stock photo for illustration only, not from the actual event
Regarding its training phase, ERNIE 5.5 Flash Base was fed an enormous dataset comprising 33.5 trillion tokens (33.5T Tokens). This extensive pre-training equips the model with a profound comprehension of language patterns and contexts prior to undergoing any mid- or post-training refinements.
Releasing the model directly as a base version before post-training indicates Baidu's intention to empower developers and the AI community to customize and fine-tune the weights for specific use cases. Combining an MoE framework with a low active-to-total parameter ratio enables organizations to run large models with drastically reduced hardware constraints, making it highly suitable for real-time applications.
Making this raw base model publicly available provides researchers and developers with a powerful foundation to build specialized artificial intelligence applications, ultimately driving innovation for business and consumer solutions.
Source: MarkTechPost
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment