Princeton Proposes Recurrent Looped Transformer (RLT)
A Princeton researcher introduces Recurrent Looped Transformer (RLT), pairing an encoder with a recurrent decoder to execute 96 blocks per token.
Stock photo for illustration only, not from the actual event
- A Princeton researcher proposes Recurrent Looped Transformer (RLT)
- Pairs a causal encoder with a recurrent decoder carrying state across tokens
- Features 48 encoder and 48 decoder layers totaling 96 logical blocks per token
A researcher from Princeton University has introduced a novel architecture named Recurrent Looped Transformer (RLT), designed to enhance language model processing by pairing a causal encoder with a recurrent decoder. This approach allows decoder states to be carried across every token with unbounded temporal depth.
The system operates by having the encoder process tokens in parallel under a causal mask, producing representations e_t. From these representations, key-value memory M≤t is projected, where memory groups can either be shared across all decoder layers or kept specific to individual layers.
Stock photo for illustration only, not from the actual event
The reference tied configuration utilizes 48 encoder layers and 48 decoder layers with compatible attention and FFN weights shared between them. Consequently, each token executes through 96 logical blocks, though decoder blocks incorporate cross-attention which makes per-block FLOPs unequal. The researcher characterizes this mechanism as parameter reuse rather than activation copying.
"Zhang calls this parameter reuse, not activation copying."
MarkTechPost
Pretraining relies on full-sequence next-token prediction with full backpropagation through time. During Supervised Fine-Tuning (SFT), the loss is masked for assistant targets while state updates are never masked, ensuring assistant losses backpropagate through user and tool tokens.
Traditional transformer architectures often struggle with scaling context lengths and maintaining efficient state representations over long sequences. By introducing recurrent loops into a transformer framework, RLT bridges the gap between recurrent neural networks and attention-based models, offering a promising solution for handling extensive temporal dependencies without exploding parameter counts.
For multi-turn serving, an exact prefix snapshot encompasses the encoder cache, memory, complete decoder state, position metadata, window convention, and model version. This enables the reuse of fixed-weight snapshots since the state remains independent of the serving split.
Source: MarkTechPost
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment