REVIEW 2 major objections 5 minor 1 cited by
A minimal log-and-search memory matches specialized agent harnesses on long-horizon games.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 10:54 UTC pith:UOENZJ54
load-bearing objection PRO-LONG's core result—an 18-point gain from a lossless log searched with code—is real and carefully measured; the 'matches or exceeds SOTA' framing survives only on one of three comparisons. the 2 major comments →
PRO-LONG: Programmatic Memory Enables Long-Horizon Reasoning
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that 'programmatic memory'—an append-all write to logs.txt and a code-based read over it—is enough to match or beat specialized, hand-built agent harnesses on ARC-AGI-3. Across three frontier models, PRO-LONG improves over the same coding agent without the log by 15.7–21.0 points (average 18.0), reaching 41.2–76.1% pass@1 and up to 97.4% best@2 while using 4.2–5.8× fewer billed tokens. The authors attribute this to simplicity (write is trivial), losslessness (nothing is compressed), and compatibility with coding agents (programmatic search stays tractable for logs over 100k lines). Ablations support this: removing log persistence costs only 0.5 points, while truncating t
What carries the argument
The mechanism is PRO-LONG's programmatic memory: a single append-only structured log (logs.txt) that records every action header, plan, action, and board state, plus a read interface consisting of the coding agent's own tools—grep, sed, awk, and Python—used to search, parse, and even replay the log. The paper frames this as a write/read split: write is trivial (append all), and read is tractable because the agent can execute code to query the log (e.g., grep for score changes, write a script to replay actions, or build a transition model). This turns a memory design problem into a code-search problem and exploits the fact that coding agents are already strong at structured text processing.
Load-bearing premise
The head-to-head claim against the specialized harnesses depends on rescoring their publicly released runs under selection rules the original authors did not disclose (single released runs treated as pass@1, versus PRO-LONG's best-of-k), so those comparisons could be unfair; the internal no-log-vs-log comparison does not depend on this premise.
What would settle it
Re-run each prior specialized harness under its own original protocol for many fresh, independent games with a published selection rule, then compare pass@1 and best@5 against PRO-LONG on the same 25 games with the same 500-action budget and scoring code; if PRO-LONG no longer matches or exceeds them, or the token savings shrink, the headline claim fails. A cheaper check: run the no-log vs. PRO-LONG comparison on a second long-horizon benchmark; if the ~18-point gain disappears, the effect is not a general property of programmatic memory.
If this is right
- Agent memory design does not require lossy write heuristics: an append-only log with programmatic search matches specialized harnesses on a difficult continual-learning benchmark.
- Adding a log file to a standard coding agent yields a consistent, large gain (15.7–21.0 points across models) on long-horizon exploratory tasks.
- Because write costs are trivial, PRO-LONG uses 4.2–5.8× fewer billed tokens than the compared harnesses, and the full benchmark can be run for roughly $1,750.
- The benefit concentrates on games where the current board does not determine the dynamics; for games solvable from the current view, the log adds little.
- Lossless memory expands the set of solvable games: best@k curves rise faster with the full log than with a truncated log, and the gap widens as more runs are averaged.
Where Pith is reading between the lines
- A direct testable extension is to apply the same append-all-and-search recipe to other text-representable long-horizon benchmarks (e.g., NetHack-style environments); the paper's own ablations suggest the gain should transfer as long as the observation stream can be logged and replayed.
- If the result holds, it implies that harness engineering effort can shift away from purpose-built memory indexes and world-model scaffolds toward simply giving coding agents raw, lossless histories and letting them write their own queries.
- Because logs grow without bound, a hybrid design that keeps a lossless log but periodically summarizes only the oldest segments would test whether the apparent benefit is from full-history access or merely from recent history plus occasional search.
- The strong dependence on independent runs (best@k) suggests that variance, not memory alone, limits current agents; an inference is that PRO-LONG's advantage might be amplified by techniques that reduce exploration variance, such as self-verification of learned rules.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes PRO-LONG, a minimal context-management framework for LLM agents in long-horizon exploratory tasks. Instead of compressing or selectively storing observations, PRO-LONG appends every action, observation, and outcome to a structured log (logs.txt) and lets the agent read it with programmatic tools (grep, Python, regular expressions). The method is evaluated on the full ARC-AGI-3 public game set (25 games) with GPT-5.5, Opus 4.6, and Fable 5. The paper reports pass@1 gains of 15.7–21.0 percentage points over a matched no-log coding-agent baseline (average 18.0), with ablations showing the gains come from full log access and increasingly programmatic tools. It also claims to match or exceed specialized harnesses (WorldModeler, Arcgentica, Schema) while using 4.2–5.8× fewer billed tokens, and reports up to 97.4% best@2 with Fable 5 at a 2,000-action budget ($1,750 total cost).
Significance. The internal result is a clean, well-controlled empirical claim: adding a lossless log and code-based read to a coding agent yields a large and consistent improvement across three frontier models. The evaluation hygiene is a genuine strength: game versions are pinned (Table 4), the official engine 0.9.7 is used, scoring follows Eq. (1), and bootstrapped CIs are provided for multi-run pass@1. The no-log/PRO-LONG comparison is a matched ablation on the same models and tools, and the tool-ladder ablation (Table 1) supports the attribution to programmatic search. If the primary claim is the 18-point gain, it is convincing and useful. However, the broader 'matches or exceeds state-of-the-art' framing is not supported by the reported numbers (PRO-LONG trails on two of three model comparisons) and rests on rescoring released runs whose selection distributions are unknown. The paper is a valuable empirical contribution if the claims are calibrated to 'competitive' and the caveats about rescored runs are made prominent.
major comments (2)
- [§3.1, Fig. 2, footnotes 2–3] The abstract and §1 state that PRO-LONG 'matches or exceeds state-of-the-art specialized harnesses.' The matched-budget numbers in §3.1 do not support this for two of three models: on GPT-5.5, PRO-LONG pass@1 is 41.2 vs WorldModeler's 45.1; on Fable 5 at 500 actions, PRO-LONG best@2 is 82.1 vs Schema's 84.4; at the extended budget, PRO-LONG's 97.4 best@2 is below Schema's 99.0, and the action budgets differ (2,000 vs up to 3,000). Only the Opus 4.6 pass@1 comparison (42.4 vs 39.0) is favorable, and that is against a single released run. The phrase 'matches or exceeds' overstates the evidence; it should either be replaced by a precise statement such as 'competitive with, and within 2–4 points of, published results,' or the analysis must be extended with a sensitivity check that justifies the stronger claim.
- [§3.1, footnotes 2–3; Appendix A] The rescored comparison to prior harnesses depends on unverifiable assumptions about run selection. WorldModeler and Arcgentica each release one run per game, and the paper labels these pass@1 without knowing whether those runs were selected from many attempts; Schema releases only its retained best run, so its score distribution is a maximum-of-two that cannot be cross-checked. If the prior runs are lucky or filtered, the reported SOTA scores are not estimates of the harnesses' true pass@1/best@2, and the comparison is not a matched or distributional one. The paper should explicitly frame the comparison as 'released runs,' provide a sensitivity analysis (e.g., treating a single prior run as best@1 vs best@k; or conservatively as an upper bound), and state in the abstract that the SOTA claim is based on rescored public trajectories whose selection procedure is unknown.
minor comments (5)
- [Table 1 caption] Typo: 'hierarchy of progammatic access' should be 'programmatic.' Also, the plus signs in the first column are cumulative; make this explicit so the reader does not interpret each row as a standalone configuration.
- [§2.2] The log contains 'the agent’s summarized plan.' The paper calls the write operation 'lossless' in §1; this holds for environment observations/actions/outcomes but not for the agent's own summaries. Please qualify the losslessness claim.
- [§3.1, Footnote 2] The sentence 'Schema uses a best-of-two heuristic over Opus 4.8 and Fable 5' is confusing because Opus 4.8 results are never reported. Clarify that this refers to the models Schema was run on, and that only Fable 5 is used in the comparison.
- [Figure 2, right panel] The legend labels 'PRO-LONG best@5' on the Codex panel; the comparison to WorldModeler is pass@1. Since the curves use different selection criteria, consider plotting PRO-LONG pass@1 as well, or add an explicit note that the token-cost comparison holds at different operating points.
- [§5 Conclusion] 'two of which were released after the benchmark' is vague; specify which models.
Circularity Check
No circular derivation; central result is a measured comparison on an external benchmark, with one non-load-bearing self-citation.
full rationale
PRO-LONG is an empirical systems paper, not a derivation. The headline improvement (15.7–21.0 pp over no-log, 18.0 average) is obtained by running both configurations on the same public ARC-AGI-3 games under the same 500-action protocol; neither quantity is solved for from the other, and no parameter is fitted to the outcome. The SOTA comparison rescoring (footnotes 2–3, Appendix A) is a fairness/validity concern, not circularity: it compares against externally released runs and is disclosed, and even if the rescored comparison is unfair, the internal no-log comparison stands independently. The only overlap with prior work is Cao et al. 2026, co-authored by co-author Bhuwan Dhingra, cited for the motivational claim that coding agents are effective long-context processors; this citation is not used as a uniqueness theorem, an ansatz, or a substitute for measurement, so it is not load-bearing. No equation reduces to its input; tool-ladder ablations are empirical attributions rather than tautologies. Verdict: no significant circularity.
Axiom & Free-Parameter Ledger
free parameters (4)
- evaluation action budget =
500 actions/game (2,000 for Fable 5 full-budget run)
- actions per turn =
20
- best@k selection protocol =
k=2 default; k=5 for GPT-5.5 best@5
- log-window truncation (ablation) =
full log default; last-25-actions in the Figure 3 ablation
axioms (5)
- domain assumption ARC-AGI-3 games and scoring (Eq. 1) implemented by the official arc_agi engine v0.9.7 with game versions pinned in Table 4 are a valid benchmark of long-horizon reasoning.
- domain assumption Rescored public runs of WorldModeler, Arcgentica, and Schema are representative of those harnesses' true performance, and Schema's retained-best-run logs suffice for cost comparison.
- domain assumption Billed-token formulas (Eqs. 2-3) with stated price weights and API usage reports reflect real economic cost.
- domain assumption 64x64 ASCII grids with a 16-entry hex color map are an information-preserving enough observation representation; disabling vision does not handicap the agent.
- domain assumption Frontier API models (GPT-5.5, Opus 4.6, Fable 5) behave as the versions available at evaluation time; base model capability is an uncontrolled variable, not a controlled one.
read the original abstract
Long-horizon tasks require sustained perception, reasoning, and exploration, and are a persistent challenge for large language model (LLM) agents. This gap is reflected in their limited performance on continual learning benchmarks such as ARC-AGI-3, especially when models are evaluated out of the box. Various agent harnesses have been proposed to close this gap, and each commits to a strategy for handling long sequences of observations, i.e., what information to save from the environment and how to load it into model context, a choice we argue is particularly consequential. Existing methods for context management face a significant tradeoff, as preserving more information makes retrieving relevant details less tractable. We propose PRO-LONG, a minimal context management framework built around programmatic memory for LLM agents in long-horizon, exploratory settings. PRO-LONG addresses the tradeoff by keeping a complete, structured interaction log and capitalizing on recent progress in coding agents to search this history efficiently. On the full ARC-AGI-3 public game set, PRO-LONG improves over a base coding agent by an average of 18.0 percentage points across frontier models, and matches or exceeds state-of-the-art specialized harnesses (up to 76.1% pass@1) while using 4.2-5.8x fewer tokens. With Fable 5, PRO-LONG achieves 97.4% best@2 at a total cost of \$1,750. Relevant code and logs are available at https://github.com/alexisfox7/PRO-LONG.
Forward citations
Cited by 1 Pith paper
-
Tycho: Active Abstraction with Programmatic World Models for ARC-AGI-3
Selective programmatic world modeling (actor-requested builder) yields 100 RHAE on all 183 public ARC-AGI-3 levels, while automatic repair is more transition-exact but weaker at play.
Reference graph
Works this paper leans on
-
[1]
ARC-AGI-3 reference agents
ARC Prize Foundation . ARC-AGI-3 reference agents. https://github.com/arcprize/ARC-AGI-3-Agents, 2026
2026
-
[2]
Coding agents are effective long-context processors, 2026
Weili Cao, Xunjian Yin, Bhuwan Dhingra, and Shuyan Zhou. Coding agents are effective long-context processors, 2026. URL https://arxiv.org/abs/2603.20432
arXiv 2026
-
[3]
Mem0: Building production-ready ai agents with scalable long-term memory, 2025
Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. Mem0: Building production-ready ai agents with scalable long-term memory, 2025. URL https://arxiv.org/abs/2504.19413
Pith/arXiv arXiv 2025
-
[4]
ARC-AGI-3 : A new challenge for frontier agentic intelligence
Fran c ois Chollet, Mike Knoop, and Greg Kamradt. ARC-AGI-3 : A new challenge for frontier agentic intelligence. arXiv preprint arXiv:2603.24621, 2026 a . URL https://arxiv.org/abs/2603.24621
Pith/arXiv arXiv 2026
-
[5]
Arc-agi-2: A new challenge for frontier ai reasoning systems, 2026 b
Francois Chollet, Mike Knoop, Gregory Kamradt, Bryan Landers, and Henry Pinkard. Arc-agi-2: A new challenge for frontier ai reasoning systems, 2026 b . URL https://arxiv.org/abs/2505.11831
Pith/arXiv arXiv 2026
-
[6]
On the measure of intelligence, 2019
François Chollet. On the measure of intelligence, 2019. URL https://arxiv.org/abs/1911.01547
Pith/arXiv arXiv 2019
-
[7]
ArcMemo : Abstract reasoning composition with lifelong LLM memory, 2025
Matthew Ho et al. ArcMemo : Abstract reasoning composition with lifelong LLM memory, 2025. URL https://arxiv.org/abs/2509.04439
arXiv 2025
-
[8]
Context rot: How increasing input tokens impacts llm performance
Kelly Hong, Anton Troynikov, and Jeff Huber. Context rot: How increasing input tokens impacts llm performance. https://www.trychroma.com/research/context-rot, 2025
2025
-
[9]
Schema: Frontier models with our harness achieve 99\ ARC-AGI-3 public
Impossible Research . Schema: Frontier models with our harness achieve 99\ ARC-AGI-3 public. https://schema-harness.github.io, 2026. Released 2026-07-15
2026
-
[10]
Less is more: Recursive reasoning with tiny networks, 2025
Alexia Jolicoeur-Martineau. Less is more: Recursive reasoning with tiny networks, 2025. URL https://arxiv.org/abs/2510.04871
Pith/arXiv arXiv 2025
-
[11]
u ttler, Nantas Nardelli, Alexander H. Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rockt \
Heinrich K \"u ttler, Nantas Nardelli, Alexander H. Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rockt \"a schel. The NetHack learning environment. In Advances in Neural Information Processing Systems (NeurIPS), 2020
2020
-
[12]
u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks, 2020. URL https://arxiv.org/abs/2005.11401
Pith/arXiv arXiv 2020
-
[13]
ARC-AGI without pretraining, 2025
Isaac Liao and Albert Gu. ARC-AGI without pretraining, 2025. URL https://arxiv.org/abs/2512.06104
arXiv 2025
-
[14]
Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts, 2023. URL https://arxiv.org/abs/2307.03172
Pith/arXiv arXiv 2023
-
[15]
A survey of context engineering for large language models, 2025
Lingrui Mei, Jiayu Yao, Yuyao Ge, Yiwei Wang, Baolong Bi, Yujun Cai, Jiazhi Liu, Mingyu Li, Zhong-Zhi Li, Duzhen Zhang, Chenlin Zhou, Jiayi Mao, Tianze Xia, Jiafeng Guo, and Shenghua Liu. A survey of context engineering for large language models, 2025. URL https://arxiv.org/abs/2507.13334
Pith/arXiv arXiv 2025
-
[16]
Show your work: Scratchpads for intermediate computation with language models, 2021
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. Show your work: Scratchpads for intermediate computation with language models, 2021. URL https://arxiv.org/abs/2112.00114
Pith/arXiv arXiv 2021
-
[17]
Siru Ouyang, Jun Yan, I-Hung Hsu, Yanfei Chen, Ke Jiang, Zifeng Wang, Rujun Han, Long T. Le, Samira Daruki, Xiangru Tang, Vishy Tirumalashetty, George Lee, Mahsan Rofouei, Hangfei Lin, Jiawei Han, Chen-Yu Lee, and Tomas Pfister. Reasoningbank: Scaling agent self-evolving with reasoning memory, 2025. URL https://arxiv.org/abs/2509.25140
Pith/arXiv arXiv 2025
-
[18]
Patil, Ion Stoica, and Joseph E
Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. Memgpt: Towards llms as operating systems, 2023. URL https://arxiv.org/abs/2310.08560
Pith/arXiv arXiv 2023
-
[19]
BALROG : Benchmarking agentic LLM and VLM reasoning on games
Davide Paglieri, Bart omiej Cupia , Samuel Coward, Ulyana Piterbarg, Maciej Wolczyk, Akbir Khan, Eduardo Pignatelli, ukasz Kuci \'n ski, Lerrel Pinto, Rob Fergus, Jakob Nicolaus Foerster, Jack Parker-Holder, and Tim Rockt \"a schel. BALROG : Benchmarking agentic LLM and VLM reasoning on games. In International Conference on Learning Representations (ICLR), 2025
2025
-
[20]
Joon Sung Park, Joseph C. O'Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior, 2023. URL https://arxiv.org/abs/2304.03442
Pith/arXiv arXiv 2023
-
[21]
Self-improving language models for evolutionary program synthesis: A case study on ARC-AGI , 2025
Julien Pourcel, C\'edric Colas, and Pierre-Yves Oudeyer. Self-improving language models for evolutionary program synthesis: A case study on ARC-AGI , 2025. URL https://arxiv.org/abs/2507.14172
arXiv 2025
-
[22]
Executable world models for ARC-AGI-3 in the era of coding agents, 2026
Sergey Rodionov. Executable world models for ARC-AGI-3 in the era of coding agents, 2026. URL https://arxiv.org/abs/2605.05138
Pith/arXiv arXiv 2026
-
[23]
StochasticGoose : 1st place in the ARC-AGI-3 agent preview competition
Dries Smit. StochasticGoose : 1st place in the ARC-AGI-3 agent preview competition. https://github.com/DriesSmit/ARC3-solution, 2025
2025
-
[24]
From 0\ https://www.symbolica.ai/blog/arc-agi-3, 2026
Symbolica AI . From 0\ https://www.symbolica.ai/blog/arc-agi-3, 2026
2026
-
[25]
It's 2026
Vaguely Aligned . It's 2026. Can LLMs Play NetHack Yet? https://kenforthewin.github.io/blog/posts/nethack-agent/, 2026
2026
-
[26]
Hierarchical reasoning model, 2025
Guan Wang, Jin Li, Yuhao Sun, Xing Chen, Changling Liu, Yue Wu, Meng Lu, Sen Song, and Yasin Abbasi Yadkori. Hierarchical reasoning model, 2025. URL https://arxiv.org/abs/2506.21734
Pith/arXiv arXiv 2025
-
[27]
Yaxiong Wu, Sheng Liang, Chen Zhang, Yichao Wang, Yongyue Zhang, Huifeng Guo, Ruiming Tang, and Yong Liu. From human memory to AI memory: A survey on memory mechanisms in the era of large language models, 2025. URL https://arxiv.org/abs/2504.15965
Pith/arXiv arXiv 2025
-
[28]
Skillrl: Evolving agents via recursive skill-augmented reinforcement learning, 2026
Peng Xia, Jianwen Chen, Hanyang Wang, Jiaqi Liu, Kaide Zeng, Yu Wang, Siwei Han, Yiyang Zhou, Xujiang Zhao, Haifeng Chen, Zeyu Zheng, Cihang Xie, and Huaxiu Yao. Skillrl: Evolving agents via recursive skill-augmented reinforcement learning, 2026. URL https://arxiv.org/abs/2602.08234
Pith/arXiv arXiv 2026
-
[29]
Zhang, Tim Kraska, and Omar Khattab
Alex L. Zhang, Tim Kraska, and Omar Khattab. Recursive language models, 2025. URL https://arxiv.org/abs/2512.24601
Pith/arXiv arXiv 2025
-
[30]
A survey on the memory mechanism of large language model based agents, 2024
Zeyu Zhang, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Quanyu Dai, Jieming Zhu, Zhenhua Dong, and Ji-Rong Wen. A survey on the memory mechanism of large language model based agents, 2024. URL https://arxiv.org/abs/2404.13501
Pith/arXiv arXiv 2024
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.