Recurrent Looped Transformer

Authors: Yifan Zhang, Jichen Feng, Shihan Qin
Venue: arXiv preprint arXiv:2610.07591 (2026)

A causal encoder provides token representations and KV memory to a recurrent decoder that carries state across prompt and response tokens

Overview

RLT combines parallel causal encoding with token-by-token recurrent decoding. A gated connection carries the previous decoder output into the next token, extending the model’s computation path as sequences grow while keeping the number of blocks evaluated for each token fixed. The paper studies six algorithmic tasks across encoder/decoder depth splits and feedback schedules. On selected parity and permutation-tracking settings, recurrent feedback substantially improves length extrapolation over an eight-layer Transformer. The ablations also show a trade-off: updating feedback less frequently enables parallel chunk processing, but hurts tasks that require fine-grained state updates.

Citation

@misc{zhang2026recurrentlooped,
  title        = {Recurrent Looped Transformer},
  author       = {Zhang, Yifan and Feng, Jichen and Qin, Shihan},
  year         = {2026},
  eprint       = {2610.07591},
  archivePrefix= {arXiv},
  primaryClass = {cs.CL},
  url          = {https://arxiv.org/abs/2610.07591}
}