MIT’s Recursive Language Models Let LLMs Process 10 Million Tokens Without Context Rot

RLM (Retrieval-Augmented Language Model) Performance Summary:

This article details the performance of a new framework called Retrieval-Augmented Language Models (RLMs) designed⁤ to ⁣handle extremely long context windows ⁢(10 million+ tokens). Here’s a breakdown of the key findings:

* Problem ‍Addressed: Standard ⁤language models⁣ struggle with‍ very long contexts, frequently enough failing to process information effectively. RLMs aim to overcome this limitation. A key aspect is the ability‍ to perform⁣ problem decomposition, which ‍is crucial for handling complex tasks with long inputs.
* Key Advantage: RLMs substantially‍ outperform base models (like GPT-5 without the RLM framework) adn‍ other agentic approaches (CodeAct, Summary Agents) when dealing with long-context tasks.
* Benchmark Results:

⁢ ‍ ⁢ * BrowseComp-Plus (6-11 million tokens): RLM (GPT-5 powered) – 91.33%, Summary Agent – ‍70.47%, CodeAct – 51%, Base Models‍ – 0%.
* OOLONG-Pairs (information-dense reasoning): RLM – 58% F1 score, Base GPT-5 – 0.04%.
⁣ * CodeQA (code understanding): RLM – 62%, Base GPT-5 – 24%.
* emergent Capabilities: RLMs⁣ demonstrate an ability to handle dense,computationally ⁣complex tasks that “paralyze” standard models.
* Technology: The RLM framework utilizes GPT-5 as its underlying language model.

In essence, the RLM framework represents a substantial advancement in the ‍ability of language models to process and reason over extremely large amounts of text.