Pith. sign in

REVIEW 3 major objections 4 minor 45 references

Structurally isolating reasoning stages so only a compressed symbolic rule can pass between them is what makes few-shot induction work in frozen LLMs.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 03:47 UTC pith:GHAPRZ5G

load-bearing objection Clean topology paper with real ablations and solid cross-domain gains; the soft NL bottleneck plus unmatched multi-call compute is the main open confound, not a reason to dismiss it. the 3 major comments →

arxiv 2607.11696 v1 pith:GHAPRZ5G submitted 2026-07-13 cs.AI

Think Through a Bottleneck: Hourglass Reasoning for Rigorous Induction

Huan Zhu This is my paper
classification cs.AI
keywords hourglass reasoninginductive reasoninginformation bottleneckfew-shot learningself-refinementcontext isolationsymbolic intermediatesfrozen LLMs
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Self-refinement usually fails to improve few-shot inductive reasoning because raw examples, code, errors, and repair instructions share one dense context, so the model patches artifacts instead of recovering the rule. This paper claims the fix is structural: force every stage into a fresh context and allow only a compressed symbolic state—a parsing schema φ and a transformation rule T—to cross boundaries. The frozen model acts as a meta-constructor that builds, for each task, an encoder–decoder (Induction produces φ plus a throwaway scaffold z; Deduction derives T and discards z; Implementation compiles (φ, T); Refinement revises only (φ, T) and regenerates artifacts from scratch). Across visual abstraction, hardware synthesis, and linguistics-olympiad puzzles the same pipeline lifts accuracy substantially—up to 14 points best-of-5 on ARC-AGI-2 and nearly doubling Verilog pass rates—while ablations show the isolation topology, not prompt wording or the particular symbolic format, carries the gains. A reader cares because this reorganizes how frozen models reason without weight updates or domain-specific languages.

Core claim

Hourglass reasoning establishes that a structurally enforced isolation between induction, deduction, implementation, and refinement—so that only the compressed pair (schema φ, transformation rule T) survives as persistent state—substantially improves few-shot inductive reasoning in frozen large language models, and that how information is allowed to flow, more than the language used to express it, is the decisive factor.

What carries the argument

Hourglass reasoning: a four-stage, role-isolated pipeline in which the frozen LLM builds a per-task symbolic encoder–decoder. Induction compresses support examples into schema φ and transient scaffold z; Deduction derives rule T from them and permanently discards z; Implementation compiles (φ, T) into artifacts; Refinement revises only (φ, T) and regenerates artifacts from scratch. Only (φ, T) crosses stage boundaries.

Load-bearing premise

That natural-language descriptions of a schema and a rule, passed only through separate API calls, actually force abstraction rather than merely adding extra compute, multi-call structure, or scaffolding.

What would settle it

A compute-matched experiment that gives monolithic self-refine the same total tokens and number of isolated calls as Hourglass but without forcing the (φ, T) bottleneck, and that still matches or exceeds Hourglass accuracy on ARC-AGI-2 and ChipBench.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Monolithic self-refinement that keeps examples, code, and feedback in one context will keep favoring shortcut patches over recovered rules.
  • A domain-agnostic isolation topology can raise few-shot induction without training, reusable DSLs, or weight updates.
  • Explicit (φ, T) intermediates supply an auditable abstraction that can be inspected and corrected independently of the final artifact.
  • Structured bottlenecks can mitigate the known harm of forcing rules into language on anti-verbalization tasks such as linguistics olympiads.
  • Consensus sampling over isolated Hourglass runs can reach high ARC accuracy at moderate per-task cost.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same isolation principle may transfer to other test-time compute methods that currently share full conversational histories across roles.
  • Soft natural-language bottlenecks could later be replaced by typed or formally constrained intermediates while keeping the plug-and-play character.
  • Accumulating cross-task meta-strategies, as the discussion sketches, could turn ephemeral per-task hourglasses into a form of test-time library learning.
  • Unmatched extra tokens remain a confound; budget-matched multi-call baselines without the symbolic bottleneck would be the cleanest next measurement.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes Hourglass reasoning: a four-stage, role-isolated pipeline in which a frozen LLM induces a schema φ and transient scaffold z from few-shot support examples, deduces a transformation rule T (discarding z), compiles (φ, T) into artifacts, and revises only (φ, T) under error feedback while regenerating artifacts from scratch. Only the compressed symbolic pair (φ, T) is allowed to cross stage boundaries. Evaluated on ARC-AGI-2, ChipBench (Python reference models and Verilog), and BBEH-Linguini with GPT-5.5 and Gemini 3.1 Pro, Hourglass improves over a context-reset Self-Refine baseline (up to +14 points pass@5 on ARC-AGI-2; nearly doubling Verilog pass@1 for GPT-5.5). Ablations (Weak-Init, Code-Only, Plain, Unstructured, Struct-SR; Table 6) are used to argue that physical context isolation and initial induction quality, not prompt wording or the particular symbolic form, drive the gains.

Significance. If the topology claim holds, the work supplies a practical, domain-agnostic recipe for improving few-shot inductive reasoning in frozen LLMs without weight updates: enforce a soft information bottleneck by isolating stages and routing refinement through an explicit symbolic state. The multi-domain evaluation (visual abstraction, hardware synthesis, anti-prior linguistic induction), the clean ablation suite that isolates isolation from formatting, and the public code/prompts are genuine strengths. The result is of clear interest to test-time compute, neuro-symbolic, and program-synthesis communities, even though the bottleneck remains soft (natural-language (φ, T) plus separate API calls) rather than architecturally enforced.

major comments (3)
  1. §6.2 Limitation 3 and the token/call counts leave the central topology claim under-determined relative to unmatched compute. Hourglass uses roughly 2–3× tokens and ~2–3× API calls versus Self-Refine; the paper states that main comparisons are not compute-matched. Without a budget-matched isolation-ablated control (or an equal-token multi-call Self-Refine), it remains possible that extra search budget, rather than the enforced (φ, T) bottleneck, explains a substantial fraction of the reported lifts (Tables 2–5).
  2. Table 6 Code-Only retains most of the gain on GPT-5.5 (−2.5/−2.1) and is mixed on Gemini, while Struct-SR collapses. This is informative but double-edged: if the compiled artifact already embeds the structure of (φ, T), then the explicit symbolic bottleneck may be less load-bearing than claimed in the abstract and §5. The paper should either strengthen the leakage/audit argument (e.g., that instance-specific detail cannot re-enter via S or free-form NL) or qualify the claim that isolation of the symbolic pair, rather than multi-stage regeneration of code, is the decisive mechanism.
  3. §4.4 / Table 5: on BBEH-Linguini, Hourglass underperforms Raw Prompt at pass@1 for both models and only recovers or exceeds it at pass@5 for Gemini. Given that prior work cited by the authors shows explicit verbalization can hurt, the abstract’s claim that Hourglass “mitigates this tendency, and on Gemini 3.1 Pro, it reverses the effect entirely” should be stated more carefully (pass@5 only; GPT-5.5 still trails Raw Prompt). A clearer discussion of when the bottleneck helps versus when it merely reduces the damage of Self-Refine would strengthen the cross-domain narrative.
minor comments (4)
  1. §3.2 / Table 1: the precise content of the transient scaffold z is illustrated for ARC but left somewhat underspecified for ChipBench and BBEH; a short cross-domain example would help readers assess how much instance-specific detail is actually discarded.
  2. Appendix A competition-mode numbers are single-run point estimates on the public set; the text already notes this, but the main body should more explicitly flag that absolute ARC-AGI-2 figures may not transfer to the private leaderboard.
  3. §6.2 Limitation 1 correctly labels the bottleneck “soft”; the abstract and contribution list could mirror that language more closely so readers do not over-read “structurally enforced isolation” as an architectural guarantee.
  4. Figure 1 and Figure 2 are helpful; ensuring that the discarded-z edge is visually unambiguous in both would reduce ambiguity about what is purged versus re-supplied (S is re-supplied every stage).

Circularity Check

0 steps flagged

No circularity: empirical method paper whose gains are measured on external benchmarks with independent ablations; no derivation reduces by construction to its inputs.

full rationale

Hourglass is a test-time workflow for frozen LLMs, not a closed-form derivation. The central claim (role-isolated passage of only compressed symbolic state (φ, T) drives inductive gains) is supported by pass@k lifts on three external, unsaturated benchmarks (ARC-AGI-2 official scoring, ChipBench random test vectors, BBEH-Linguini exact-string matching) against a context-reset Self-Refine baseline and five ablations (Table 6). None of the load-bearing steps fit a parameter to data and then re-label a related quantity as a prediction; none invoke a uniqueness theorem or ansatz from overlapping authors; none rename a known empirical pattern as a new first-principles result. The Information-Bottleneck citations (§2.3) are classical external theory used only as design intuition, not as a load-bearing uniqueness proof. Limitation 1 (§6.2) explicitly calls the bottleneck soft and prompt-enforced, so the paper does not claim architectural necessity that would create definitional circularity. Token/call overhead is acknowledged but does not convert the empirical comparison into a tautology. Steps array is empty because no quoted reduction of the form Eq. X ≡ Eq. Y by construction, or fitted input called prediction, exists.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 3 invented entities

The central claim rests on empirical measurement under a soft, prompt-enforced bottleneck and on standard assumptions about frozen LLM APIs. Free parameters are operational (iteration caps, consensus thresholds). Invented entities are the pipeline constructs themselves; they have no independent existence outside the method. No hidden physical constants or fitted scientific parameters.

free parameters (3)
  • max_refinement_iterations_M = 5
    Both Hourglass and Self-Refine are capped at five refinement iterations; the cap is a design choice that affects pass@k and cost.
  • competition_mode_consensus_threshold_N = 2 (preferred)
    Pass@2 competition mode selects Slot 1 by consensus of N independent runs (swept N=2,3,4); N is chosen for cost–accuracy trade-off, not derived.
  • API_retry_limit = 3
    Modules producing invalid syntax are retried up to three times; operational hyperparameter.
axioms (4)
  • ad hoc to paper Separate API calls with no shared conversational history prevent instance-specific details from leaking across stages except via the explicitly passed (φ, T).
    Core design premise of Hourglass (§3); not architecturally enforced, only prompt- and session-enforced (Limitation 1).
  • domain assumption Natural-language schema φ and rule T are sufficiently discrete and compressible to act as an information bottleneck analogous to classical IB / autoencoder latents.
    Invoked via analogy to IB theory and DreamCoder (§2.3–2.4); no formal mutual-information bound is measured.
  • domain assumption Self-Refine with context reset and the same iteration budget is an adequate primary baseline for attributing gains to topology rather than extra reasoning steps.
    Stated in §4.1; authors note comparisons are not compute-matched (§6.2).
  • domain assumption Public ARC-AGI-2 evaluation set and cleaned BBEH-Linguini subset are valid inductive tests for the claimed mechanism.
    Standard benchmark use; authors flag possible pretraining exposure and problem filtering (§6.2, Appendix C).
invented entities (3)
  • Hourglass reasoning topology (Induction / Deduction / Implementer / schema-anchored Refiner) no independent evidence
    purpose: Enforce role-isolated symbolic state passing for inductive tasks.
    The method itself; evaluated only by end-task accuracy, no independent physical prediction.
  • Transient scaffold z no independent evidence
    purpose: Aid derivation of T then be discarded so instance details do not reach implementation/refinement.
    Defined operationally in §3; existence is prompt-defined, not independently measured.
  • Symbolic pair (φ, T) as sole persistent cross-stage state no independent evidence
    purpose: Serve as encoder–decoder bottleneck that anchors all refinement.
    Central invented interface; ablations test necessity but do not give an external falsifiable signature beyond task accuracy.

pith-pipeline@v1.1.0-grok45 · 25133 in / 3504 out tokens · 35150 ms · 2026-07-14T03:47:35.658080+00:00 · methodology

0 comments
read the original abstract

Self-refinement often fails to strengthen few-shot inductive reasoning in large language models. Prompting a model to explicitly state its inferred rule does little on its own. What actually matters is a structurally enforced isolation between reasoning stages, so that information can only pass between them as a compressed symbolic state. We introduce \textbf{Hourglass reasoning}, which enforces strict context isolation between reasoning stages. The frozen LLM acts as a meta-constructor, building for each task a symbolic encoder--decoder: an Induction module compresses the support examples into a schema $\phi$ (encoder) and a transient scaffold $z$; a Deduction module derives rule $T$ (decoder) from these and discards $z$; an Implementer compiles $(\phi, T)$ into artifacts; an error-driven Refiner revises $(\phi, T)$ and regenerates artifacts from scratch. Only $(\phi, T)$ crosses stage boundaries, so all refinement stays anchored to the rule. We evaluate Hourglass across three benchmarks spanning visual abstraction, hardware synthesis, and textual rule induction, using GPT-5.5 and Gemini 3.1 Pro. On ARC-AGI-2, it raises best-of-5 accuracy by up to 14 points over an iterative-refinement baseline. On ChipBench, it nearly doubles Verilog synthesis accuracy with GPT-5.5, from 31\% to 58\%. BBEH-Linguini draws on puzzles from the International Linguistics Olympiad, a setting where prior work has shown that explicit verbalization can hurt performance. Hourglass mitigates this tendency, and on Gemini 3.1 Pro, it reverses the effect entirely. Ablations confirm that these gains come from the isolation between stages and the quality of the initial induction, not from prompt wording or the particular symbolic form used. It is how information flows through the reasoning process, rather than the language used to express it, that drives inductive reasoning in frozen LLMs.

Figures

Figures reproduced from arXiv: 2607.11696 by Huan Zhu.

Figure 1
Figure 1. Figure 1: Monolithic self-refinement vs. Hourglass Reasoning. Left: [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The four-stage Hourglass Reasoning pipeline (illustrated for ARC-AGI-2). [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

45 extracted references · 34 linked inside Pith

  1. [1]

    A., Fischer, I., Dillon, J

    Alemi, A. A., Fischer, I., Dillon, J. V., & Murphy, K. (2017). Deep variational information bottleneck.International Conference on Learning Representations (ICLR 2017). arXiv:1612.00410

    Pith/arXiv arXiv 2017

  2. [2]

    Chen, L., Li, Z., Lyu, K., Peng, B., & Wu, H. (2026). The information bottleneck of chain-of-thought and how latent CoT overcomes it.International Conference on Learning Representations (ICLR 2026)

    2026

  3. [3]

    Chollet, F. (2019). On the measure of intelligence.arXiv. arXiv:1911.01547

    Pith/arXiv arXiv 2019

  4. [4]

    Choudhary, M., Srivatsa, K. V. A., Aeron, G., Bhattacharya, A. R., Dinh, D. K. D., Hanif, I. A., Kotova, D., Kochmar, E., & Choudhury, M. (2025). UNVEILING: What makes linguistics olympiad puzzles tricky for LLMs? arXiv. arXiv:2508.11260

    Pith/arXiv arXiv 2025

  5. [5]

    Dahl, M., Magesh, V., Suzgun, M., & Ho, D. E. (2024). Large legal fictions: Profiling legal hallucinations in large language models.Journal of Legal Analysis, 16(1), 64–93. arXiv:2401.01301

    Pith/arXiv arXiv 2024

  6. [6]

    R., Falorsi, L., De Cao, N., Kipf, T., & Tomczak, J

    Davidson, T. R., Falorsi, L., De Cao, N., Kipf, T., & Tomczak, J. M. (2018). Hyperspherical variational auto- encoders.arXiv preprint arXiv:1804.00891. arXiv:1804.00891

    Pith/arXiv arXiv 2018

  7. [7]

    DeepSeek-AI. (2025). DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv. arXiv:2501.12948

    Pith/arXiv arXiv 2025

  8. [8]

    B., & Mordatch, I

    Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., & Mordatch, I. (2023). Improving factuality and reasoning in language models through multiagent debate.International Conference on Machine Learning (ICML 2023). arXiv:2305.14325 12

    Pith/arXiv arXiv 2023

  9. [9]

    Ellis, K., Wong, C., Nye, M., Sabl ´e-Meyer, M., Cary, L., Morales, L., Hewitt, L., Solar-Lezama, A., & Tenen- baum, J. B. (2021). DreamCoder: Bootstrapping inductive program synthesis with wake-sleep library learn- ing.ACM-SIGPLAN Symposium on Programming Language Design and Implementation (PLDI 2021). DOI: 10.1145/3453483.3454080

    doi:10.1145/3453483.3454080 2021

  10. [10]

    Franzen, D., Disselhoff, J., & Hartmann, D. (2025). Product of experts with LLMs: Boosting performance on ARC is a matter of perspective.International Conference on Machine Learning (ICML 2025). arXiv:2505.07859

    Pith/arXiv arXiv 2025

  11. [11]

    C., Ameisen, E., Chen, J., Kishylau, D., Pearce, A., Tarng, J., Wu, A., Wu, J., Zhang, Y., Ziegler, D

    Fraser-Taliente, K., Kantamneni, S., Ong, E., Mossing, D., Lu, C., Bogdan, P. C., Ameisen, E., Chen, J., Kishylau, D., Pearce, A., Tarng, J., Wu, A., Wu, J., Zhang, Y., Ziegler, D. M., Hubinger, E., Batson, J., Lindsey, J., Zimmerman, S., & Marks, S. (2026). Natural language autoencoders produce unsupervised explanations of LLM activations.Transformer Cir...

    2026

  12. [12]

    A., & Brendel, W

    Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F. A., & Brendel, W. (2019). ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness.International Conference on Learning Representations (ICLR 2019). arXiv:1811.12231

    Pith/arXiv arXiv 2019

  13. [13]

    Goyal, S., & Dan, S. (2025). IOLBench: Benchmarking LLMs on linguistic reasoning.arXiv. arXiv:2501.04249

    arXiv 2025

  14. [14]

    Gu, Y., Tafjord, O., Kim, H., Moore, J., Le Bras, R., Clark, P., & Choi, Y. (2024). SimpleToM: Exposing the gap between explicit ToM inference and implicit ToM application in LLMs.arXiv. arXiv:2410.13648

    arXiv 2024

  15. [15]

    Hong, S., Zhuge, M., Chen, J., Zheng, X., Cheng, Y., Wang, J., Zhang, C., Wang, Z., Yau, S. K. S., Lin, Z., Zhou, L., Ran, C., Xiao, L., Wu, C., & Schmidhuber, J. (2024). MetaGPT: Meta programming for a multi-agent collaborative framework.International Conference on Learning Representations (ICLR 2024). arXiv:2308.00352

    Pith/arXiv arXiv 2024

  16. [16]

    S., Yu, A

    Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Chi, E. H., & Le, Q. V. (2023). Large language models cannot self-correct reasoning yet.International Conference on Learning Representations (ICLR 2024). arXiv:2310.01798

    Pith/arXiv arXiv 2023

  17. [17]

    V., Jain, L

    Kazemi, M., Fatemi, B., Bansal, H., Palowitch, J., Anastasiou, C., Mehta, S. V., Jain, L. K., Aglietti, V., Jindal, D., Chen, P., Dikkala, N., Tyen, G., Liu, X., Shalit, U., Chiappa, S., Olszewska, K., Tay, Y., Tran, V. Q., Le, Q. V., & Firat, O. (2025). BIG-Bench Extra Hard.Proceedings of the 63rd Annual Meeting of the Association for Computational Lingu...

    Pith/arXiv arXiv 2025

  18. [18]

    P., & Welling, M

    Kingma, D. P., & Welling, M. (2014, April). Auto-encoding variational bayes. InProceedings of the International Conference on Learning Representations (ICLR). arXiv:1312.6114

    Pith/arXiv arXiv 2014

  19. [19]

    Lei, S., Cheng, Z., Jia, K., & Tao, D. (2025). Revisiting LLM reasoning via information bottleneck.arXiv. arXiv:2507.18391

    Pith/arXiv arXiv 2025

  20. [20]

    Li, X., & Eisner, J. (2019). Specializing word embeddings (for parsing) by information bottleneck.Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP 2019). arXiv:1910.00163

    Pith/arXiv arXiv 2019

  21. [21]

    Lian, D.-C., Huang, R.-S., Chen, P.-E., Lim, C., Lin, Y.-K., Tseng, G.-Y., Yang, T.-C., Lin, Z.-Y., Chen, P.-C., & Hsieh, S.-K. (2025). LingBench++: A linguistically-informed benchmark and reasoning framework for multi-step and cross-cultural inference with LLMs.arXiv. arXiv:2507.16809

    Pith/arXiv arXiv 2025

  22. [22]

    Lin, Z.-L., Shih, Y.-F., & Hsieh, S.-K. (2025). Probing large language models in reasoning and translating complex linguistic puzzles.arXiv. arXiv:2502.00817

    Pith/arXiv arXiv 2025

  23. [23]

    F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P

    Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). Lost in the middle: How language models use long contexts.Transactions of the Association for Computational Linguistics, 12, 277–294. arXiv:2307.03172

    Pith/arXiv arXiv 2024

  24. [24]

    Liu, Y., Xu, C., Zhou, Y., Li, Z., & Xu, Q. (2025). DeepRTL: Bridging Verilog understanding and genera- tion with a unified representation model.International Conference on Learning Representations (ICLR 2025). arXiv:2502.15832

    Pith/arXiv arXiv 2025

  25. [25]

    P., Hermann, K., Welleck, S., Yazdanbakhsh, A., & Clark, P

    Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., Gupta, S., Majumder, B. P., Hermann, K., Welleck, S., Yazdanbakhsh, A., & Clark, P. (2023). Self-refine: Iterative refinement with self-feedback.Advances in Neural Information Processing Systems 36 (NeurIPS 2023). arXiv:2303.17651 13

    Pith/arXiv arXiv 2023

  26. [26]

    B., & Moskvichev, A

    Mitchell, M., Palmarini, A. B., & Moskvichev, A. (2023). Comparing humans, GPT-4, and GPT-4V on abstraction and reasoning tasks.arXiv. arXiv:2311.09247

    Pith/arXiv arXiv 2023

  27. [27]

    V., & Mitchell, M

    Moskvichev, A., Odouard, V. V., & Mitchell, M. (2023). The ConceptARC benchmark: Evaluating cognitive capabilities of language models.Transactions on Machine Learning Research

    2023

  28. [28]

    OpenAI. (2024). Learning to reason with LLMs. OpenAI Blog

    2024

  29. [29]

    Qian, C., Liu, W., Liu, H., Chen, N., Dang, Y., Li, J., Yang, C., Chen, W., Su, Y., Cong, X., Xu, J., Li, D., Liu, Z., & Sun, M. (2024). ChatDev: Communicative agents for software development.Annual Meeting of the Association for Computational Linguistics (ACL 2024). arXiv:2307.07924

    Pith/arXiv arXiv 2024

  30. [30]

    Sanz-Guerrero, M., & Von Der Wense, K. (2025). Corrective in-context learning: Evaluating self-correction in large language models.Workshop on Insights from Negative Results in NLP (co-located with NAACL 2025). arXiv:2503.16022

    Pith/arXiv arXiv 2025

  31. [31]

    H., Sch ¨arli, N., & Zhou, D

    Shi, F., Chen, J., Misra, K., Scales, N., Dohan, D., Chi, E. H., Sch ¨arli, N., & Zhou, D. (2023). Large language models can be easily distracted by irrelevant context.International Conference on Machine Learning (ICML 2023). arXiv:2302.00093

    Pith/arXiv arXiv 2023

  32. [32]

    Snell, C., Lee, J., Xu, K., & Kumar, A. (2024). Scaling LLM test-time compute optimally can be more effective than scaling model parameters.arXiv. arXiv:2408.03314

    Pith/arXiv arXiv 2024

  33. [33]

    Suzgun, M., Yuksekgonul, M., Bianchi, F., Jurafsky, D., & Zou, J. (2026). Dynamic Cheatsheet: Test-time learning with adaptive memory.Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2026). arXiv:2504.07952

    Pith/arXiv arXiv 2026

  34. [34]

    C., & Bialek, W

    Tishby, N., Pereira, F. C., & Bialek, W. (1999). The information bottleneck method.Proceedings of the 37th Annual Allerton Conference on Communication, Control, and Computing. arXiv:physics/0004057

    Pith/arXiv arXiv 1999

  35. [35]

    Tishby, N., & Zaslavsky, N. (2015). Deep learning and the information bottleneck principle.IEEE Information Theory Workshop. arXiv:1503.02406

    Pith/arXiv arXiv 2015

  36. [36]

    Tsui, K. (2025). Self-correction bench: Uncovering and addressing the self-correction blind spot in large language models.arXiv. arXiv:2507.02778

    arXiv 2025

  37. [37]

    Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P. A. (2008, July). Extracting and composing robust features with denoising autoencoders. InProceedings of the 25th international conference on Machine learning(pp. 1096–1103). DOI: 10.1145/1390156.1390294

    doi:10.1145/1390156.1390294 2008

  38. [38]

    Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2023). Self-consistency improves chain of thought reasoning in language models.International Conference on Learning Representations (ICLR 2023)

    2023

  39. [39]

    Wen, X., Liu, Z., Zheng, S., Xu, Z., Ye, S., Wu, Z., Liang, X., Wang, Y., Li, J., Miao, Z., Bian, J., & Yang, M. (2026). Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base LLMs. International Conference on Learning Representations (ICLR 2026). arXiv:2506.14245

    Pith/arXiv arXiv 2026

  40. [40]

    West, P., Holtzman, A., Buys, J., & Choi, Y. (2019). BottleSum: Unsupervised and self-supervised sentence summarization using the information bottleneck principle.Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP 2019). arXiv:1909.07405

    Pith/arXiv arXiv 2019

  41. [41]

    B., & Andreas, J

    Wong, C., Ellis, K., Tenenbaum, J. B., & Andreas, J. (2021). Leveraging language to learn program abstractions and search heuristics.International Conference on Machine Learning (ICML 2021). arXiv:2106.11053

    Pith/arXiv arXiv 2021

  42. [42]

    Yang, M., Piao, J., Xia, X., Lan, X., Chen, J., Gong, Y., & Li, Y. (2026). SkillMaster: Toward autonomous skill mastery in LLM agents.arXiv. arXiv:2605.08693

    Pith/arXiv arXiv 2026

  43. [43]

    Yin, Z., Sun, Q., Guo, Q., Wu, J., Qiu, X., & Huang, X. (2023). Do large language models know what they don’t know?Findings of the Association for Computational Linguistics: ACL 2023. arXiv:2305.18153

    Pith/arXiv arXiv 2023

  44. [44]

    Yu, Z., Zhou, C., Lin, Y., Zhang, H., Ye, H., Cui, J., Pan, Z., Zhao, J., & Ding, Y. (2026). ChipBench: A next-step benchmark for evaluating LLM performance in AI-aided chip design.arXiv. arXiv:2601.21448 14

    arXiv 2026

  45. [45]

    do not copy absolute coordinates,

    Zhang, Q., Wang, D., Qian, H., Li, Y., Zhang, T., Huang, M., Xu, K., Li, H., Yan, L., & Qiu, H. (2025). Understanding the dark side of LLMs’ intrinsic self-correction.Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025). arXiv:2412.14959 15 A Alignment with ARC-AGI-2 Competition Rules (Competition Mode) The of...

    Pith/arXiv arXiv 2025