Pith. sign in

REVIEW 5 major objections 4 minor 2 cited by

Reinforcement learning improves LLM factual recall by improving navigation of knowledge hierarchies, not by adding facts.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 23:23 UTC pith:JEHTGC2W

load-bearing objection The abstract promises a controlled RL experiment that never appears in the body, and Table 2 contradicts the 'consistently outperform' line; the structured-prompt result is still worth a look but the paper needs major revision before it supports the navigation claim. the 5 major comments →

arxiv 2511.05933 v2 pith:JEHTGC2W submitted 2025-11-08 cs.CL cs.AI

Reinforcement Learning Improves Traversal of Parametric Knowledge in LLMs

Renfei Zhang , Manasa Kaniselvan , Rylan Schaeffer abd Niloofar Mireshghallah This is my paper
classification cs.CL cs.AI
keywords reinforcement learningparametric knowledgehierarchical traversalknowledge navigationstructured promptingalignment taxlayer-wise representation analysisfactual recall
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Reinforcement-learning-trained reasoning models consistently outperform their instruction-tuned counterparts on recall of hierarchical factual knowledge, such as medical and patent codes, by as much as 24 percentage points. The paper argues these gains come from improved procedural navigation of knowledge structures already stored in the model, not from learning new facts. Structured prompts that explicitly guide hierarchical traversal recover most of the gap, layer-wise activations show answer representations stay aligned while query representations diverge, and retrieval-depth experiments show reasoning models keep better path accuracy as traversal gets deeper. If correct, post-training can improve factual recall without expanding stored knowledge, and the 'alignment tax' does not apply uniformly across knowledge types.

Core claim

The paper's central claim is that RL-based post-training improves performance on structured knowledge recall by teaching models to traverse existing parametric knowledge hierarchies, rather than by encoding new information. On the MedConceptsQA medical-code benchmark, the reasoning-enhanced variant of a leading base model outperforms its instruction-tuned base by 23.7 percentage points under standard QA prompting, but a hand-crafted structured prompt that forces stepwise hierarchical recall narrows the gap to 7.5 points — evidence that the facts were already present and the bottleneck was access. The authors back this with three experiments: a depth-stratified patent-classification task wher

What carries the argument

The mechanism is hierarchical navigation: systematically moving through a taxonomy (e.g., ICD-9 volume → chapter → code range → specific code) rather than attempting direct recall. The paper's key probe is the structured prompt, a two-stage instruction that forces models to first recall the hierarchical structure and then evaluate each option by elimination; it recovers most of the instruct-versus-reasoning gap. The layer-wise representational asymmetry — answers align (~0.85–0.92 cosine similarity) while queries diverge (0.65–0.73) — is the internal-states signature of the claimed effect, and the path-matching score (harmonic mean of F1 and common-subsequence score) quantifies traversal qua

Load-bearing premise

The paper's interpretation depends on its benchmarks being 'pure information recall without multi-step computation or logical deduction' (Section 2.1); if multiple-choice answering involves comparing or eliminating options in a way that constitutes reasoning, the structured-prompt closure could reflect added reasoning rather than navigation, and the knowledge-versus-navigation distinction collapses.

What would settle it

Compare reasoning-enhanced and instruction-tuned models on a flat, non-hierarchical fact-recall task (e.g., list-style trivia with no taxonomic structure) with the same prompt templates; if the reasoning model still holds a large advantage, the navigation-specific explanation fails. Alternatively, provide the complete hierarchical path inside the prompt so no navigation is required — if the reasoning advantage persists, the claim that navigation is the mechanism is refuted.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Structured prompting can close most of the accuracy gap between instruction-tuned and reasoning models on straightforward hierarchical recall, implying the knowledge is already stored but hard to reach.
  • On deep-retrieval tasks, reasoning models retain superior path-level accuracy even when final-answer accuracy is similar, so RL instills a navigation skill distinct from answer correctness.
  • Distilled reasoning models do not acquire this exploratory navigation; they imitate self-correction and remain behind, so distillation should not be assumed to transfer traversal skills.
  • The alignment tax is not universal: RL can improve structured factual recall instead of degrading memorized knowledge.
  • Post-training methods that explicitly reward traversal could further improve factual recall without adding data.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the navigation hypothesis: on a flat, non-hierarchical fact-recall benchmark, the reasoning advantage should shrink or vanish; if it does not, the explanation is not specific to hierarchy.
  • The query-versus-answer representation asymmetry suggests that cheap interventions on query encoding (e.g., query reformulation or prompt optimization) could substitute for full RL on shallow retrieval tasks.
  • The results imply that standard knowledge benchmarks that ignore hierarchy underestimate what instruction-tuned models know, and evaluation suites should include deep, structured retrieval tasks to separate knowledge from access.
  • Because distilled models fail at exploration, distillation pipelines for reasoning models may need to reward path diversity or intermediate traversal fidelity, not just final-answer correctness.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper argues that reinforcement learning (RL) improves language models' recall of hierarchical parametric knowledge not by adding new facts, but by teaching better traversal of existing knowledge structures. The authors support this with three lines of evidence: structured prompting that narrows the accuracy gap between instruction-tuned and reasoning models on MedConceptsQA and IPC; a depth-stratified IPC dataset with a path-matching metric showing reasoning models improve on deeper retrieval; and layer-wise activation analyses suggesting question representations diverge while factual/answer representations remain similar. The abstract also promises a controlled RL experiment on unseen, non-extractable facts, but no such experiment appears in the full text. The central claim—that RL improves navigation rather than knowledge—is therefore not directly tested.

Significance. If established, the distinction between knowledge content and knowledge navigation would be practically important for post-training design, suggesting that RL could improve factual recall without expanding parametric knowledge. The paper proposes a concrete, falsifiable hypothesis and includes useful instrumentation: structured prompt templates, a depth-stratified retrieval task, and an activation-similarity methodology. However, as submitted, the central causal claim is unsupported because the key controlled experiment is missing, and the reported results contain internal contradictions that weaken the empirical foundation.

major comments (5)
  1. [Abstract; Sections 2–3] The abstract and introduction promise a 'controlled RL experiment on unseen, non-extractable facts' that would directly test whether RL adds knowledge or improves navigation. No such experiment is reported anywhere in Sections 2–3 or the appendices. Without it, the paper cannot distinguish 'RL improves traversal of existing knowledge' from 'RL adds knowledge.' The structured-prompt and activation evidence are indirect and do not fill this gap. This is a load-bearing omission for the paper's central claim.
  2. [Section 2.1] The claim that MedConceptsQA and IPC tasks are 'pure information recall without multi-step computation or logical deduction' is not defended. These are multiple-choice tasks requiring option comparison and elimination; indeed, the structured prompt (Template 3) explicitly instructs stepwise evaluation and elimination. If such steps constitute reasoning, then the structured-prompt closure may reflect added reasoning rather than knowledge navigation. This assumption is central to RQ1 and the overall navigation-vs-knowledge interpretation, but it is asserted rather than tested.
  3. [Table 2] The paper repeatedly states that reasoning models 'consistently outperform' their instruction-tuned counterparts. This is contradicted by Table 2. For Qwen3-235B-A22B under Structured prompting, the Instruct model outperforms the Reasoning model: majority-vote 0.631 vs 0.580 (Δ=-0.051) and mean 0.589 vs 0.554 (Δ=-0.035). Similar inversions occur for Qwen2.5-32B on IPC (mean QA: 0.759 vs 0.713; structured mean: 0.774 vs 0.769) and for DeepSeek-V3 IPC CoT (mean: 0.882 vs 0.867). The 'consistently outperform' claim is an overgeneralization that undermines the motivation.
  4. [Section 2.3, Eq. (1), Figure 3] The inter-model answer-answer (A-A) cosine similarity is computed by feeding identical declarative strings to the base and specialized models. High A-A similarity is largely expected when the input text is the same, and it does not establish that the models' factual knowledge representations are unchanged. Similarly, the Q-Q divergence could reflect differences in prompt formatting or instruction following rather than 'navigation' mechanisms. This activation analysis is at best suggestive; as presented, it does not provide strong evidence for the knowledge-preservation claim.
  5. [Section 2.2, Path Matching Score] The Path Matching Score combines F1 and the Common Subsequence Score via an unweighted harmonic mean. No justification or sensitivity analysis is given for this aggregation. Table 5's conclusion that the R1-V3 gap widens with retrieval depth is based entirely on this metric. A different aggregation could yield different qualitative conclusions, so the depth-dependent traversal claim is not robustly grounded.
minor comments (4)
  1. [Figure 1] Typo: 'unary catheter' should be 'urinary catheter.'
  2. [References] Reference list includes a note about citation key 'xu2024gepa' but first author is Agrawal; please reconcile citation keys.
  3. [Section 2.1] No dataset sizes, construction details, or licensing information are given for MedConceptsQA or the extended IPC dataset. Please include these for reproducibility.
  4. [Section 3.1] The phrase 'consistently' in the abstract and Section 1 should be softened to 'generally' or qualified, given the counterexamples in Table 2.

Circularity Check

0 steps flagged

No circular derivation: central claim rests on independent interventions, though the promised controlled RL experiment is absent from the body.

full rationale

The paper's central inference (RL improves navigation of existing hierarchical knowledge rather than adding knowledge) is not a self-referential reduction. The structured-prompt experiment is an external intervention: the prompt template does not encode the dataset answers or the hierarchy content; it only instructs the model to recall the category structure and evaluate options. The 24pp-to-7pp gap reduction is a contingent empirical outcome that could have failed, so it is not a fitted parameter renamed as a prediction. The layer-wise analysis is also not circular by construction: although the inter-model A-A comparison feeds identical declarative strings, identical inputs do not guarantee high cosine similarity across models (the paper itself shows query representations diverge under the same setup), so the high A-A similarity is a substantive, if confounded, measurement. No equation in the paper reduces the conclusion to its inputs, and the Path Matching Score is a defined metric compared across models, not optimized to fit the claim. The only self-citation (Sorensen et al. 2025, a coauthor paper on alignment tax) is background and not load-bearing. The abstract's controlled RL experiment on unseen, non-extractable facts is not present in Sections 2-3 or the appendices; this is a serious missing-evidence problem for the causal claim, but absence of an experiment is not a circular identity. Overall, the derivation chain is independent rather than circular; the score is low, with the caveat that the headline causal result is under-supported by missing evidence rather than by circularity.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The central inference that RL improves navigation rather than knowledge rests on three untested premises: task purity, knowledge-presence-before-prompting, and representational faithfulness. The structured prompt is also a hand-fitted instrument, and the path-matching metric involves unvalidated design choices.

free parameters (2)
  • Structured Prompt Template 3 = Appendix B.1, Template 3
    Hand-crafted to elicit hierarchical traversal and stepwise elimination; the paper notes it 'minimized' the gap, so it may have been selected after observing test-set performance rather than independently specified.
  • Path Matching Score harmonic-mean weights = Equal weights on F1 and CSS
    The equal-weight harmonic combination of F1 and CSS is a design choice, not derived from theory or validated against human judgment.
axioms (4)
  • domain assumption The test tasks (MedConceptsQA, IPC) require pure recall with no reasoning confound.
    Section 2.1 describes them as testing factual recall 'without reasoning over patient cases or performing calculations,' but multiple-choice answering still requires option comparison and elimination.
  • domain assumption Instruction-tuned/base models contain the target facts; failure is due to access, not absence.
    This is the premise of RQ1 and the interpretation of the structured-prompt result; it is asserted rather than directly proven in the manuscript.
  • domain assumption Final-token cosine similarity of declarative answer strings reflects factual knowledge content.
    Section 2.3 uses answer-answer similarity to conclude knowledge is preserved, but both models receive identical answer strings, so high similarity may reflect identical input text rather than shared factual knowledge.
  • domain assumption Path Matching Score measures hierarchical traversal quality.
    Section 2.2 assumes the harmonic mean of F1 and CSS captures navigation skill, but no validation against human judgments or alternative metrics is provided.

pith-pipeline@v1.3.0-alltime-deepseek · 17207 in / 13856 out tokens · 132854 ms · 2026-08-03T23:23:41.581358+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning (RL) is often credited with improving language model reasoning at the expense of knowledge. We challenge this narrative by showing that reasoning models consistently outperform their instruction-tuned versions on pure knowledge recall tasks. These gains do not reflect newly acquired information, but rather an improved procedural skill in navigating and searching existing knowledge hierarchies within the model parameters. Structured prompting, which explicitly guides models through hierarchical traversal -- recovers most of the instruct-reasoning gap across five model families. A controlled RL experiment on unseen, non-extractable facts improves recall of held-out frequent but previously inaccessible facts, ruling out simple data exposure. On depth-stratified retrieval tasks, reasoning models exhibit superior traversal as retrieval depth grows. Layerwise activation analysis further shows that while factual representations maintain high cosine similarity between instruct and reasoning models, query representations diverge noticeably, indicating that reasoning primarily reshapes how models traverse knowledge rather than the knowledge representation itself. Finally, we find that distilled models often fail to match reasoning models on knowledge recall because they imitate self-correction without acquiring the exploratory behavior needed for hierarchical navigation. Together, these findings suggest that improving factual recall in LLMs depends not only on expanding what models know but also on teaching them to navigate it -- motivating future post-training methods that optimize traversal.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Preference Tuning as Spectral Update Reorganization

    cs.CL 2026-05 conditional novelty 6.0

    Preference tuning reorganizes a model's weight update into a dominant spectral head that drives visible behavior and a residual tail that supports learning breadth; the update is structured, not monolithic.

  2. Cram Less to Fit More: Training Data Pruning Improves Memorization of Facts

    cs.CL 2026-04 conditional novelty 6.0

    Loss-based pruning of training data to limit facts and flatten their frequency distribution enables a 110M-parameter GPT-2 model to memorize 1.3 times more entity facts than standard training, matching a 1.3B-paramete...

Reference graph

Works this paper leans on

55 extracted references · 37 linked inside Pith · cited by 2 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny ...

    Pith/arXiv arXiv 2023

  3. [3]

    On-policy distillation of language models: Learning from self-generated mistakes

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In The twelfth international conference on learning representations, 2024

    2024

  4. [5]

    Gepa: Reflective prompt evolution can outperform reinforcement learning

    Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, et al. Gepa: Reflective prompt evolution can outperform reinforcement learning. arXiv preprint arXiv:2507.19457, 2025

    Pith/arXiv arXiv 2025

  5. [6]

    A general language assistant as a laboratory for alignment, 2021

    Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. A general language assistant as a labora...

    Pith/arXiv arXiv 2021

  6. [7]

    Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, K...

    Pith/arXiv arXiv 2022

  7. [8]

    A is B

    Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: LLMs trained on " A is B " fail to learn " B is A ". arXiv preprint arXiv:2309.12288, 2024

    Pith/arXiv arXiv 2024

  8. [9]

    The relationship between recall and precision

    Michael Buckland and Fredric Gey. The relationship between recall and precision. Journal of the American society for information science, 45 0 (1): 0 12--19, 1994

    1994

  9. [10]

    Sft or rl? an early investigation into training r1-like reasoning large vision-language models

    Hardy Chen, Haoqin Tu, Fali Wang, Hui Liu, Xianfeng Tang, Xinya Du, Yuyin Zhou, and Cihang Xie. Sft or rl? an early investigation into training r1-like reasoning large vision-language models. arXiv preprint arXiv:2504.11468, 2025 a

    Pith/arXiv arXiv 2025

  10. [11]

    Learning to reason with search for llms via reinforcement learning

    Mingyang Chen, Linzhuang Sun, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang, Jeff Z Pan, Wen Zhang, Huajun Chen, et al. Learning to reason with search for llms via reinforcement learning. arXiv preprint arXiv:2503.19470, 2025 b

    Pith/arXiv arXiv 2025

  11. [12]

    Sft memorizes, rl generalizes: A comparative study of foundation model post-training

    Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. arXiv preprint arXiv:2501.17161, 2025

    Pith/arXiv arXiv 2025

  12. [13]

    Measuring retrieval complexity in question answering systems

    Matteo Gabburo, Nicolaas Paul Jedema, Siddhant Garg, Leonardo FR Ribeiro, and Alessandro Moschitti. Measuring retrieval complexity in question answering systems. arXiv preprint arXiv:2406.03592, 2024

    Pith/arXiv arXiv 2024

  13. [14]

    Does fine-tuning llms on new knowledge encourage hallucinations? arXiv preprint arXiv:2405.05904, 2024

    Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. Does fine-tuning llms on new knowledge encourage hallucinations? arXiv preprint arXiv:2405.05904, 2024

    Pith/arXiv arXiv 2024

  14. [15]

    A closer look at the limitations of instruction tuning

    Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar, Deepali Aneja, Zeyu Jin, Ramani Duraiswami, Dinesh Manocha, et al. A closer look at the limitations of instruction tuning. arXiv preprint arXiv:2402.05119, 2024

    Pith/arXiv arXiv 2024

  15. [16]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  16. [18]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025 b

    Pith/arXiv arXiv 2025

  17. [19]

    Does math reasoning improve general llm capabilities? understanding transferability of llm reasoning

    Maggie Huan, Yuetai Li, Tuney Zheng, Xiaoyu Xu, Seungone Kim, Minxin Du, Radha Poovendran, Graham Neubig, and Xiang Yue. Does math reasoning improve general llm capabilities? understanding transferability of llm reasoning. arXiv preprint arXiv:2507.00432, 2025

    Pith/arXiv arXiv 2025

  18. [20]

    Search-r1: Training llms to reason and leverage search engines with reinforcement learning

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516, 2025

    Pith/arXiv arXiv 2025

  19. [21]

    Mistral--a journey towards reproducible language model training, 2021

    Siddharth Karamcheti, Laurel Orr, Jason Bolton, Tianyi Zhang, Karan Goel, Avanika Narayan, Rishi Bommasani, Deepak Narayanan, Tatsunori Hashimoto, Dan Jurafsky, et al. Mistral--a journey towards reproducible language model training, 2021

    2021

  20. [22]

    Dspy: Compiling declarative language model calls into self-improving pipelines

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, et al. Dspy: Compiling declarative language model calls into self-improving pipelines. arXiv preprint arXiv:2310.03714, 2023

    Pith/arXiv arXiv 2023

  21. [23]

    Reinforcement learning vs

    Minwu Kim, Anubhav Shrestha, Safal Shrestha, Aadim Nepal, and Keith Ross. Reinforcement learning vs. distillation: Understanding accuracy and capability in llm reasoning. arXiv preprint arXiv:2505.14216, 2025

    arXiv 2025

  22. [24]

    Understanding the effects of rlhf on llm generalisation and diversity

    Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the effects of rlhf on llm generalisation and diversity. arXiv preprint arXiv:2310.06452, 2023

    Pith/arXiv arXiv 2023

  23. [25]

    Knowledge-level consistency reinforcement learning: Dual-fact alignment for long-form factuality

    Junliang Li, Yucheng Wang, Yan Chen, Yu Ran, Ruiqing Zhang, Jing Liu, Hua Wu, and Haifeng Wang. Knowledge-level consistency reinforcement learning: Dual-fact alignment for long-form factuality. arXiv preprint arXiv:2509.23765, 2025

    Pith/arXiv arXiv 2025

  24. [26]

    Ehr-r1: A reasoning-enhanced foundational language model for electronic health record analysis

    Yusheng Liao, Chaoyi Wu, Junwei Liu, Shuyang Jiang, Pengcheng Qiu, Haowen Wang, Yun Yue, Shuai Zhen, Jian Wang, Qianrui Fan, et al. Ehr-r1: A reasoning-enhanced foundational language model for electronic health record analysis. arXiv preprint arXiv:2510.25628, 2025

    arXiv 2025

  25. [27]

    Let's verify step by step, 2023

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step, 2023. URL https://arxiv.org/abs/2305.20050

    Pith/arXiv arXiv 2023

  26. [28]

    Mitigating the alignment tax of RLHF

    Yong Lin, Hangyu Lin, Wei Xiong, Shizhe Diao, Jianmeng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang, Wenbin Hu, Hanning Zhang, Hanze Dong, Renjie Pi, Han Zhao, Nan Jiang, Heng Ji, Yuan Yao, and Tong Zhang. Mitigating the alignment tax of RLHF . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Association for Computat...

    2024

  27. [29]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    Pith/arXiv arXiv 2024

  28. [30]

    Hierarchical label-wise attention transformer model for explainable icd coding

    Leibo Liu, Oscar Perez-Concha, Anthony Nguyen, Vicki Bennett, and Louisa Jorm. Hierarchical label-wise attention transformer model for explainable icd coding. Journal of biomedical informatics, 133: 0 104161, 2022

    2022

  29. [31]

    Factscore: Fine-grained atomic evaluation of factual precision in long form text generation

    Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. arXiv preprint arXiv:2305.14251, 2023

    Pith/arXiv arXiv 2023

  30. [32]

    Reinforcement learning finetunes small subnetworks in large language models

    Sagnik Mukherjee, Lifan Yuan, Dilek Hakkani-T \"u r, and Hao Peng. Reinforcement learning finetunes small subnetworks in large language models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=0NdS4xCngO

    2025

  31. [33]

    Fine-tuning or retrieval? comparing knowledge injection in LLMs

    Oded Ovadia, Menachem Brief, Moshik Mishaeli, and Oren Elisha. Fine-tuning or retrieval? comparing knowledge injection in LLMs . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 237--250, 2024

    2024

  32. [34]

    Longest common subsequences

    Mike Paterson and Vlado Dan c \' k. Longest common subsequences. In International symposium on mathematical foundations of computer science, pages 127--142. Springer, 1994

    1994

  33. [35]

    Beyond reasoning gains: Mitigating general capabilities forgetting in large reasoning models

    Hoang Phan, Xianjun Yang, Kevin Yao, Jingyu Zhang, Shengjie Bi, Xiaocheng Tang, Madian Khabsa, Lijuan Liu, and Deren Lei. Beyond reasoning gains: Mitigating general capabilities forgetting in large reasoning models. arXiv preprint arXiv:2510.21978, 2025

    Pith/arXiv arXiv 2025

  34. [36]

    Procedural knowledge in pretraining drives reasoning in large language models

    Laura Ruis, Maximilian Mozes, Juhan Bae, Siddhartha Rao Kamalakara, Dwarak Talupuru, Acyr Locatelli, Robert Kirk, Tim Rockt \"a schel, Edward Grefenstette, and Max Bartolo. Procedural knowledge in pretraining drives reasoning in large language models. arXiv preprint arXiv:2411.12580, 2024

    Pith/arXiv arXiv 2024

  35. [37]

    From extreme multi-label to multi-class: A hierarchical approach for automated icd-10 coding using phrase-level attention

    Cansu Sen, Bingyang Ye, Javed Aslam, and Amir Tahmasebi. From extreme multi-label to multi-class: A hierarchical approach for automated icd-10 coding using phrase-level attention. arXiv preprint arXiv:2102.09136, 2021

    Pith/arXiv arXiv 2021

  36. [38]

    Reflexion: Language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023). Curran Associates Inc., 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/hash/1b44b878bb782e6954cd888628510e90-Abstract-Conf...

    Pith/arXiv arXiv 2023

  37. [39]

    Layer by layer: Uncovering hidden representations in language models

    Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. Layer by layer: Uncovering hidden representations in language models. arXiv preprint arXiv:2502.02013, 2025

    Pith/arXiv arXiv 2025

  38. [40]

    Spectrum tuning: Post-training for distributional coverage and in-context steerability

    Taylor Sorensen, Benjamin Newman, Jared Moore, Chan Park, Jillian Fisher, Niloofar Mireshghallah, Liwei Jiang, and Yejin Choi. Spectrum tuning: Post-training for distributional coverage and in-context steerability. arXiv preprint arXiv:2510.06084, 2025

    arXiv 2025

  39. [41]

    Qwen2 technical report

    Qwen Team et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2: 0 3, 2024

    Pith/arXiv arXiv 2024

  40. [42]

    Hierarchical reasoning model

    Guan Wang, Jin Li, Yuhao Sun, Xing Chen, Changling Liu, Yue Wu, Meng Lu, Sen Song, and Yasin Abbasi Yadkori. Hierarchical reasoning model. arXiv preprint arXiv:2506.21734, 2025 a

    Pith/arXiv arXiv 2025

  41. [43]

    Emergent hierarchical reasoning in llms through reinforcement learning

    Haozhe Wang, Qixin Xu, Che Liu, Junhong Wu, Fangzhen Lin, and Wenhu Chen. Emergent hierarchical reasoning in llms through reinforcement learning. arXiv preprint arXiv:2509.03646, 2025 b

    arXiv 2025

  42. [44]

    Octothinker: Mid-training incentivizes reinforcement learning scaling

    Zengzhi Wang, Fan Zhou, Xuefeng Li, and Pengfei Liu. Octothinker: Mid-training incentivizes reinforcement learning scaling. arXiv preprint arXiv:2506.20512, 2025 c

    Pith/arXiv arXiv 2025

  43. [45]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 24824--24837. Curran Associ...

    2022

  44. [46]

    The invisible leash: Why rlvr may or may not escape its origin

    Fang Wu, Weihao Xuan, Ximing Lu, Mingjie Liu, Yi Dong, Zaid Harchaoui, and Yejin Choi. The invisible leash: Why rlvr may or may not escape its origin. arXiv preprint arXiv:2507.14843, 2025

    arXiv 2025

  45. [47]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025 a

    Pith/arXiv arXiv 2025

  46. [48]

    Reasonflux: Hierarchical llm reasoning via scaling thought templates

    Ling Yang, Zhaochen Yu, Bin Cui, and Mengdi Wang. Reasonflux: Hierarchical llm reasoning via scaling thought templates. ArXiv, abs/2502.06772, 2025 b . URL https://api.semanticscholar.org/CorpusID:276250066

    Pith/arXiv arXiv 2025

  47. [49]

    Process-supervised reinforcement learning for code generation

    Yufan Ye, Ting Zhang, Wenbin Jiang, and Hua Huang. Process-supervised reinforcement learning for code generation. arXiv preprint arXiv:2502.01715, 2025

    Pith/arXiv arXiv 2025

  48. [50]

    Dapo: An open-source llm reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025

    Pith/arXiv arXiv 2025

  49. [51]

    Towards a holistic evaluation of llms on factual knowledge recall

    Jiaqing Yuan, Lin Pan, Chung-Wei Hang, Jiang Guo, Jiarong Jiang, Bonan Min, Patrick Ng, and Zhiguo Wang. Towards a holistic evaluation of llms on factual knowledge recall. arXiv preprint arXiv:2404.16164, 2024

    Pith/arXiv arXiv 2024

  50. [52]

    Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D. Goodman. Quiet- STaR : Language models can teach themselves to think before speaking, 2024. URL https://arxiv.org/abs/2403.09629

    Pith/arXiv arXiv 2024

  51. [53]

    Least-to-most prompting enables complex reasoning in large language models

    Denny Zhou, Nathanael Sch\" a rli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, and Ed Chi. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=WZH7099tgfM

    2023

  52. [54]

    Multi-module grpo: Composing policy gradients and prompt optimization for language model programs

    Noah Ziems, Dilara Soylu, Lakshya A Agrawal, Isaac Miller, Liheng Lai, Chen Qian, Kaiqiang Song, Meng Jiang, Dan Klein, Matei Zaharia, et al. Multi-module grpo: Composing policy gradients and prompt optimization for language model programs. arXiv preprint arXiv:2508.04660, 2025

    Pith/arXiv arXiv 2025

  53. [55]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  54. [56]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  55. [57]

    A is B

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    Pith/arXiv arXiv doi:10.48550/arxiv.2501.12948 2006