# Deterministic Aggregation vs. Recursive Language Models — an OOLONG replication

A replication study of Recursive Language Models (RLM; Zhang et al., [arXiv:2512.24601](https://arxiv.org/abs/2512.24601)) on the OOLONG benchmark, testing whether a simple deterministic-aggregation baseline can match or beat recursive sub-calls on long-context aggregation tasks.

## Correction — please read first

An earlier version of this repository reported much stronger numbers — **87.8% exact match** and **"zero query-time LLM calls."** Those results were inflated by a methodology error, and I'm grateful to the RLM authors (Alex Zhang et al., MIT) for flagging it directly:

- The earlier run fed `context_window_text_with_labels` into the context — i.e. the ground-truth labels. With the labels present, a regex trivially scores ~100%. The point of OOLONG is to learn the labels *semantically*, which is exactly what makes the model calls necessary. Using the labels was not the real task.
- OOLONG is scored with exact match / the paper's numeric metric, **not F1**, and the paper's numbers are on the **`trec_coarse`** split.

This version corrects all three: labels withheld, `trec_coarse` only, exact match, GPT-5 for an apples-to-apples comparison. The corrected numbers are below. The original commits remain in git history for transparency.

**Second correction (Jul 28, 2026), caught in our own re-audit:** our first corrected run evaluated 250 short-context tasks (1K–16K tokens), but the RLM paper's OOLONG numbers are **50 tasks at 131K tokens** ("the trec_coarse split, a set of 50 tasks", Table 1). Putting those side by side was still not apples-to-apples, so this repo now ships a runner ([`run_oolong_gpt5.py`](run_oolong_gpt5.py)) that reproduces **both** settings, and the paper comparison below is on the paper's exact setting. One forced substitution: `gpt-5-chat-latest` (used in the first corrected run) has since been deprecated by OpenAI, so the runner uses `gpt-5` — the model the paper itself benchmarked.

## Corrected results

**Paper's setting — OOLONG `trec_coarse`, 131K-token contexts, 50 tasks (validation split, labels withheld):**

| Method | OOLONG score | Exact match | LLM calls |
|---|---|---|---|
| GPT-5 base (paper) | 44.00% | — | — |
| GPT-5 RLM depth=2 (paper) | 56.50% | — | ~10–1000s per query |
| GPT-5 RLM depth=3 (paper) | 58.00% | — | ~10–1000s per query |
| **Ours — classify + `Counter()` (gpt-5, minimal reasoning)** | **63.38%** | **62.00%** | **320 total** |

**Short-context setting — 1K–16K tokens, 250 tasks (the setting of our first corrected run):**

| Method | OOLONG score | Exact match | LLM calls |
|---|---|---|---|
| First corrected run (gpt-5-chat-latest, now deprecated) | 73.24% | 63.60% | 1,900 total |
| This repo's runner (gpt-5, one classification pass per distinct context window) | 74.82% | 66.80% | 76 total |

The short-context reproduction lands within ~1.6 points of the first corrected run on a different model — the earlier numbers replicate. The call-count drop (1,900 → 76) is window caching: OOLONG reuses each context window across many questions, so records are classified once per *window*, not once per *question*.

Approach: classify each record independently, then aggregate deterministically with Python's `Counter()` — **no recursion**. On the paper's own setting it beats the paper's RLM at every reported depth while making ~320 total calls, where RLM makes tens to thousands *per query*.

### By task type (paper's setting, 131K)

| Task | Score | N |
|---|---|---|
| RELATIVE_FREQ | 82.8% | 29 |
| MOST_FREQ | 80.0% | 5 |
| LEAST_FREQ | 75.0% | 4 |
| NUMERIC_ONE_CLASS | 5.8% | 12 |

The NUMERIC_ONE_CLASS row is the honest weakness: exact-count questions score `0.75^|error|`, and per-record classification noise across ~3,182 records compounds into counts off by tens — partial credit collapses. Order-statistics questions (most/least/relative frequency) are robust to the same noise. Deterministic sub-questions that need no classification at all (e.g. "which user appears most often") score 1.0 with zero LLM involvement.

## What holds, and what doesn't

- **Holds:** OOLONG's aggregation tasks are largely classify-then-count over structured records, so a deterministic-aggregation baseline is genuinely competitive with — here, ahead of — recursive sub-calls. That's the real finding.
- **Doesn't:** the earlier "zero query-time inference / 0.26ms / 6-orders-of-magnitude" framing. That was an artifact of the labels. The corrected approach still calls the model to classify each record (320 calls for the paper's 131K setting; 76 for the short setting with window caching); the advantage over RLM is in *total* calls and determinism, **not** in eliminating inference.

## Reproducing

```bash
pip install duckdb
export OPENAI_API_KEY=sk-...
python3 run_oolong_gpt5.py --tier 131k    # paper's setting: 50 tasks @ 131K context
python3 run_oolong_gpt5.py --tier short   # 250 tasks @ 1K-16K context
```

The runner streams the dataset from HuggingFace (`oolongbench/oolong-synth`, validation split) with the labeled column never loaded, classifies records in batches of 20, answers every question with plain Python counting, and scores with the official OOLONG scorer (vendored verbatim from [abertsch72/oolong](https://github.com/abertsch72/oolong)). Full outputs: [`benchmark_data/oolong_gpt5_results_131k.json`](benchmark_data/oolong_gpt5_results_131k.json), [`benchmark_data/oolong_gpt5_results_short.json`](benchmark_data/oolong_gpt5_results_short.json). Classification is cached per context window (`benchmark_data/clf_cache_*/`, regenerated on first run), so re-runs are free.

## Limitations

- One benchmark, one dataset (OOLONG `trec_coarse`): 50 tasks at 131K, 250 tasks at 1K–16K.
- Aggregation-style tasks only; not tested on BrowseComp or other RLM benchmarks.
- Exact-count (NUMERIC) questions are a real weakness of this method at long context — see the by-task table.
- [GPT5_BENCHMARK_REPORT.md](GPT5_BENCHMARK_REPORT.md) documents the first corrected run (short-context, gpt-5-chat-latest); the tables above supersede its paper comparison. The older `oolong_benchmark.py` here is the **superseded, labeled** version (see its header) and does not reproduce any corrected number.

## RLM limitations documented in the paper (unchanged, from arXiv:2512.24601)

> "RLM(Qwen3-Coder) made hundreds to thousands of recursive sub-calls for a single simple task" (Section 3.1)

> "Distinguishing between a final answer and a thought is brittle for RLMs" (Section 5)

## References

Zhang, A., et al. (2025). *Recursive Language Models.* arXiv:2512.24601.

## License

MIT