H Company Releases NeoMME Multimodal Encoders
H Company has launched NeoMME, a family of 260M and 800M single-tower multimodal encoders dropping vision towers and causal decoders under Apache 2.0.
Stock photo for illustration only, not from the actual event
- H Company releases NeoMME 260M and 800M models
- Single Transformer processes text and 32x32 image patches
- NeoMME-Retriever scores 0.523 nDCG@10 on ViDoRe v3
- Available under Apache 2.0 on Hugging Face
H Company has released NeoMME, a family of 260-million and 800-million parameter bidirectional encoders that completely omit both vision towers and causal decoders. The architecture relies on a single Transformer to process multilingual text tokens and raw 32x32 RGB image patches through identical layers, trained entirely from random initialization.
Deployable out of the box, every checkpoint ships under the Apache 2.0 license with day-zero support in Hugging Face Transformers. The 260M model indexes 51.3 pages per second on a single NVIDIA L40S GPU and encodes queries in 78.3 milliseconds on a CPU-only host environment.
Stock photo for illustration only, not from the actual event
Text enters through an ALBERT-style factorized embedding utilizing a 256-dimensional lookup projected to model width. Images are partitioned into non-overlapping 32x32 patches and projected via a 2-layer MLP trained from scratch without any patch-merging modules. Both models natively support a 16,384-token context window, utilizing symmetric sliding-window attention alongside global attention layers.
The pretraining process applies discrete masked diffusion over text data, optionally conditioned on visible image patches. Text-only segments utilize corruption rates from 0 to 1, whereas multimodal segments draw from 0.30 to 1, compelling the model to thoroughly read document pages. Cross-modal ablation probes indicate that visible page patches elevate masked-token accuracy by 38.4 points for the 260M variant and 40.5 points for the 800M variant.
"H Company has released NeoMME, a family of 260M and 800M bidirectional encoders that drops both components."
H Company
Removing separate vision towers and causal decoders represents a notable simplification in multimodal machine learning architectures. By unifying text and image patch processing into a single backbone, NeoMME achieves high throughput and efficient document indexing capabilities, establishing a strong foundation for resource-constrained enterprise deployment.
Source: MarkTechPost
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment