Pith. sign in

REVIEW 4 major objections 4 minor 7 cited by

AutoTIR: Autonomous Tools Integrated Reasoning via Reinforcement Learning

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read RL teaches models when to search, code, or think alone

desk verdict Useful RL recipe for tool-integrated reasoning with a real but fixable circularity in the autonomy claim; deserves review after revision. read the letter →

arxiv 2507.21836 v1 pith:6PEQMMMA submitted 2025-07-29 cs.CL

Yifan Wei , Xiaoyan Yu , Yixuan Weng , Tengfei Pan , Angsheng Li , Li Du This is my paper
classification cs.CL
keywords tool-integratedreasoningreinforcementlearningautonomoustoolselectionlargemodelsinstructionfollowinghybridrewardretrievaltoolscodeinterpreter
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

AutoTIR sets out to show that a language model can learn, through reinforcement learning alone, when to reason in text, when to call a search engine, and when to call a code interpreter, instead of following a fixed tool-use template. The paper argues that existing tool-integrated reasoning methods win accuracy in one domain at the cost of general instruction-following, and that the damage comes from not letting the model choose its tools. Its hybrid reward scores the tool-invocation action separately from the final answer: correct tool choices are rewarded, wrong tool choices are penalized even when the answer is correct, and tool choice is left open on general instructions. Across ten benchmarks the trained model reaches the highest average score, covering knowledge-intensive, mathematical, and instruction-following tests, which the paper presents as evidence that autonomous tool selection is learnable and that the balance with language modeling can be preserved.

What carries the argument

The load-bearing object is a hybrid, advantage-based reward that separates the question 'did the model choose the right tool?' from the question 'did the model give the right answer?'. The action reward assigns $+1$ for the correct tool in rewarded domains, a penalty for the wrong tool even when the answer is correct, and $+1$ for open-domain instructions where tool choice is left free; the output reward uses task-specific scoring on answers in a required $\boxed{}$ format, with F1 for question answering, a binary score for mathematics, and an instruction-following score for IFEval. The two rewards combine as $r = 0.1 \times r_{\text{act}} + 0.9 \times r_{\text{out}}$ and are optimized by group-relative policy optimization, which normalizes advantages within a group of rollouts against a reference instruct model. This separation is what allows one policy to learn search, code, and no-tool reasoning at the same time rather than inheriting a single fixed tool-use pattern.

What would settle it

Run AutoTIR on a task family where the prescribed mapping is inverted, such as a mathematical question whose answer is best obtained by search or a knowledge question best solved by code, and measure both tool selection and final accuracy. If the model follows the rewarded domain-to-tool mapping rather than the task's actual needs, the autonomy claim fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that large language models can balance tool-integrated reasoning with language modeling and can learn, through reinforcement learning, to decide both whether an external tool is needed and which tool to invoke. AutoTIR, trained on a mix of multi-hop question answering, mathematical reasoning, and instruction-following data, achieves an average score of 46.01 across ten benchmarks, ahead of the strongest baseline ReSearch at 28.65, while keeping IFEval instruction-following at 51.02 SAcc compared with 21.84 for the base instruct model. The framework treats tool use as part of an advantage-based reward: an action reward pushes the model toward search on knowledge-intensive tasks and code on mathematics, penalizes the wrong tool even when the final answer is correct, and leaves tool choice free on open-domain instructions; an output reward then grades the format-compliant answer against task-specific metrics. Trained this way, the model invokes tools selectively, shows high tool-selection accuracy and tool productivity across domains, and retains instruction-following, which the paper reads as the desired balance between the two objectives.

Load-bearing premise

The load-bearing premise is that the reward designer's mapping, search for knowledge-intensive tasks, code for mathematics, and a penalty for the other tool, is the correct inductive bias and transfers to unseen tasks, so the autonomy the model learns is only as general as that mapping.

Editorial extensions

If this is right

  • If the claim holds, tool-augmented reasoning no longer requires a separate specialist model per tool; one policy can serve retrieval-heavy, computation-heavy, and plain-language questions.
  • Autonomous tool selection can be trained from rule-based rewards without supervised tool-use traces, so the same recipe can extend to more tools and more task families by adding reward terms.
  • Including instruction-following data in the reinforcement learning mix prevents the usual degradation, so deploying a tool-integrated reasoning model need not sacrifice general usefulness for domain accuracy.
  • The learned policy transfers to out-of-distribution test sets: all benchmarks except MuSiQue come from sources not used in training, yet tool selection and final accuracy improve.
  • The tool-productivity metric indicates the model converts tool calls into correct answers at a higher rate than baselines, meaning the learned behavior is cost-aware rather than tool-happy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to replace the fixed domain-to-tool mapping in the action reward with a measured, per-task estimate of which tool actually helps; that would upgrade 'autonomy within the designer's prior' to autonomy based on task need.
  • The same reward split could be applied to a larger tool set beyond search and code, such as calculators, database queries, or image generators, letting the open-domain free-exploration term discover useful tools the designer did not preselect.
  • The instruction-following gain from 21.84 to 51.02 SAcc suggests RL with IF data can actively improve adherence; testing this at larger scale with more diverse instruction distributions would show whether the effect is robust.
  • The test-time cost analysis hints at a per-sample difficulty gate that skips tool calls on easy questions, a latency-saving direction the paper sketches but does not implement.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces AutoTIR, a reinforcement-learning framework that trains a Qwen2.5-7B-Instruct model to decide, during a reasoning rollout, whether to invoke a tool and which tool (Wikipedia search or code interpreter) to use. Training uses GRPO with a hybrid reward that combines an action reward for tool choice (Eq. 4), an output reward for answer correctness and formatting (Eq. 5), and a penalty for wrong-tool usage. The training mixture includes MuSiQue, ToRL, Math-DAPO, Natural Questions, and instruction-following data. The model is evaluated on ten benchmarks spanning knowledge-intensive QA, mathematics, logical reasoning, and instruction following. The authors report the highest average score (46.01) versus the best baseline ReSearch (28.65), while preserving IFEval soft accuracy at 51.02, and present ablations, tool-usage metrics, training curves, and test-time-cost analysis supporting the claim that tool-integrated reasoning can be learned without sacrificing instruction following.

Significance. If the empirical results are sound, AutoTIR is a useful contribution to tool-integrated reasoning: it directly addresses the known trade-off between tool-augmented reasoning and instruction-following degradation, uses a simple GRPO-based RLVR recipe, reports multiple ablations (w/o Tools, w/o IF, w/o Penalty, w/ Prior), and releases code and data. The training curves and the ablation showing that free exploration on open-domain instructions outperforms a hand-crafted tool-use prior are genuinely informative. However, the central 'autonomous tool selection' claim currently rests on a metric that shares its ground truth with the training reward, and the generalization claims are weakened by training/evaluation overlap and the absence of any variance or significance analysis. These issues are load-bearing and need to be addressed before the paper's main claims can be accepted.

major comments (4)
  1. [Reward Design, Eq. (4); Performance Evaluation Metrics, Eq. (7)] The 'autonomous tool selection' claim is measured with a reward-circular metric. Eq. (4) defines ract = +1 for 'correct tool invocation,' where correct is hard-coded as search on knowledge-intensive tasks and code on math problems, with rpenalty for the opposite tool. Eq. (7) then defines Tool Selection Accuracy using a ground-truth tool E*(t), but no independent source for E*(t) is provided; the description implies E*(t) is exactly the domain-tool mapping used in the reward. Consequently, the high TS scores in Tables 3 and 4 on knowledge-intensive and math benchmarks largely measure whether the policy memorized the reward designer's mapping rather than whether it learned to choose tools autonomously. This is compounded by the statement in §Tool-Utilization Efficiency Analysis that TS is not calculated on open-domain instructions, the only setting where tool choice is left free. Please supply an external ground-truth tool label (e.g., human-annotated labels on held-out tasks) or a separate tool-selection benchmark and report TS there; alternatively, soften the claims from 'autonomous tool selection' to 'adherence to a learned domain-tool policy.'
  2. [Experiment Setup: Training Settings; Table 1; Main Results] The generalization claim is partly confounded by training/evaluation overlap. The MuSiQue training split is used during RL training and MuSiQue also appears as a test benchmark in Table 1 (43.20 EM); the paper itself acknowledges in §Main Results that 'all benchmarks besides MuSiQue have different sources compared to the training data.' Moreover, the math training sets ToRL and Math-DAPO may overlap with MATH500, GSM8K, or AIME, but no contamination analysis or deduplication is reported. Because AutoTIR's largest margins over retrieval baselines are on math benchmarks (e.g., AIME24 33.33 vs 26.67; GSM8K 88.48 vs 84.76) and because the out-of-distribution claim is central, please compute exact n-gram or question-level overlap rates with all eval sets, remove overlapping items, and re-report Table 1 on the cleaned eval sets.
  3. [Experiments: Main Results and Table 1; Ablation Analysis and Table 2] All headline comparisons rest on a single run without error bars or significance tests. Table 1 reports one point estimate per model, and several differences are small (HotpotQA 43.15 vs ReSearch 42.17; MuSiQue 43.20 vs ReSearch 41.60; MATH500 62.60 vs Eurus-2-7B-PRIME 62.00). Since the central claim is the highest average score, report means and standard deviations over at least three training seeds (or bootstrap CIs over evaluation samples) and, where feasible, paired significance tests. The same applies to the ablation rows in Table 2, where w/o Penalty is within one point of the full model on several benchmarks and the IFEval drop from 51.02 to 47.13 could be within run-to-run noise.
  4. [Tool-Utilization Efficiency Analysis, Tables 3 and 4] The claim that AutoTIR shows 'comparable or higher tool selection accuracy and tool productivity compared to SoTA baselines' is contradicted by Table 4. For AIME25, AutoTIR TP is 6.67 versus ToRL 8.57, and for GSM8K AutoTIR TP is 76.15 versus ToRL 80.59, while TS is higher for AutoTIR in both cases. Either revise the claim to say that AutoTIR has higher tool-selection accuracy with comparable but lower tool productivity, or explain why the lower TP is acceptable (e.g., a trade-off between more frequent correct tool use and higher answer accuracy). Also note that TP is not directly comparable across models with different tool-call frequencies if the denominator counts all invocations, so a breakdown of correct-tool versus incorrect-tool invocations would help.
minor comments (4)
  1. [Appendix: Implementation Details] The appendix states 'We employ Qwen2-7B-instruct as the base model,' while the main text and Table 1 use Qwen2.5-7B-Instruct; please resolve the discrepancy.
  2. [Methodology, Eq. (3)] The reward weights 0.1 and 0.9 are fixed without a sensitivity analysis; a short grid or a statement that the results are robust to moderate changes would strengthen the method section.
  3. [Performance Evaluation Metrics, Eq. (7)] In Eq. (7), T is described as 'the set of all tool invocation contents,' but E(t) is referred to as 'the tool selected by the model'; clarify whether the metric is computed per invocation or per content token, and define E*(t) explicitly.
  4. [Reward Design, Eq. (4)] In Eq. (4), the 'open domain instances' branch assigns ract = 1 even if a tool is invoked; the text says the model is left free to explore, but the equation as written gives a constant reward. Please align the formal definition with the prose.

Circularity Check

1 steps flagged · score 6.0 of 10

The tool-selection metric (Eq. 7) is scored against the same hard-coded domain-to-tool mapping that Eq. (4) uses to define 'correct tool invocation', so the reported autonomous tool-selection accuracy partly measures fidelity to the reward prior.

  1. self definitional [Reward Design, Eq. (4); Performance Evaluation Metrics, Eq. (7); Tool-Utilization Efficiency Analysis]
    "On complex knowledge-intensive tasks, the model is rewarded to utilize the searching engine tool; On mathematical problems with massive numerical calculation, the model is rewarded to invoke the code interpreter ... ract = 1, For correct tool invocation; rpenalty, For wrong tool usage; 1, For open domain instances (4) ... TS = 1/|T| Σ_{t∈T} I(E(t)=E*(t)) (7) ... On open domain instructions, TS is not calculated, as there is no clear standard for judging if tool(s) should be invoked."

    The reward in Eq. (4) defines 'correct tool invocation' by a fixed domain-to-tool mapping: search for knowledge-intensive tasks, code for math, with a penalty for the opposite tool even when the final answer is correct. Eq. (7) scores Tool Selection Accuracy against a ground-truth tool E*(t) that the paper never defines independently; on the reported knowledge and math benchmarks, E*(t) is exactly the Eq. (4) mapping. Thus a high TS mainly indicates that the policy learned the reward designer's prescribed mapping, not that it autonomously discovered when to search or code. The paper also withholds TS on open-domain instructions, the only condition where Eq. (4) leaves tool choice free, so the one setting that could test emergent selection is not scored.

full rationale

AutoTIR's headline claim is that RL lets the model autonomously decide whether and which tool to invoke. Part of the evidence is independent: the output reward (Eq. 5) is tied to external answer correctness, the open-domain branch of Eq. (4) allows free tool exploration, and the final benchmark scores are externally grounded. However, the tool-selection evidence is partially circular. Eq. (4) prescribes search for knowledge-intensive tasks and code for math, and Eq. (7)'s E*(t) is presented without any independent source; on the reported domains it is the same mapping. Therefore the TS metric measures adherence to the reward prior, not emergent generalization. The paper's decision not to compute TS on open-domain instructions removes the one split where selection is truly free. Because the central 'which tool' claim leans on TS, while the 'whether to use tools' and answer-quality claims do not, this is a partial reduction by construction rather than full circularity. No load-bearing self-citation chain or imported uniqueness theorem appears in the derivation.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central behavior of AutoTIR is shaped by hand-set reward weights, a hand-set penalty, and a per-domain tool mapping. The paper pulls these from the authors' prior, not from data, so they are the main 'unpaid' inputs. The GRPO objective and the use of retrieval and code execution are standard tools pulled from prior work. No new physical or conceptual entities are introduced.

free parameters (5)
  • action_reward_weight = 0.1
    Weight in the total reward r = 0.1 * ract + 0.9 * rout (Eq. 3). Chosen by hand with no sensitivity analysis.
  • output_reward_weight = 0.9
    Weight in the total reward (Eq. 3). Chosen by hand.
  • tool_penalty_rpenalty = -1
    Penalty for wrong tool usage in Eq. (4). The authors state 'simply setting r_penalty = -1 could be acceptable.'
  • output_reward_floor = 0.1
    Floor in rout = max[0.1, feva(...)] in Eq. (5). Hand-chosen minimum reward for formatted but possibly incorrect outputs.
  • per_domain_tool_rule = search for knowledge-intensive, code for math, free for open-domain
    The action reward in Eq. (4) hard-codes which tool is correct for each task domain, which shapes the entire learned tool-use policy.
assumptions (5)
  • standard math GRPO objective in Eq. (6) provides a valid policy-gradient update with clipped importance weights and KL penalty.
    Standard RLVR machinery from Shao et al. 2024; the paper relies on it without proof.
  • domain assumption e5-base-v2 retrieval over a Wikipedia 2018 corpus is a faithful search tool for knowledge-intensive QA.
    Used as the search environment in all experiments; results depend on corpus coverage and retriever quality.
  • domain assumption Tool execution results can be masked during loss computation without biasing the policy gradient.
    Stated in Methodology without analysis of whether masking affects credit assignment for tool-invocation decisions.
  • domain assumption IFEval rule-based scoring adequately measures instruction-following ability.
    Used both as the output reward for open-domain data and as the evaluation metric; if the score is brittle, the claim of preserved instruction-following is weakened.
  • domain assumption The MuSiQue test split is disjoint from the MuSiQue training split used in RL.
    The paper trains on MuSiQue and evaluates on MuSiQue without explicitly describing split boundaries, creating a potential train/eval overlap.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AutoTIR: Autonomous Tools Integrated Reasoning via Reinforcement Learning." pith.science (2026). https://pith.science/paper/6PEQMMMA

@misc{pith2026250721836,
  author       = {Pith},
  title        = {Pith review of: AutoTIR: Autonomous Tools Integrated Reasoning via Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6PEQMMMA}},
  note         = {Machine review of arXiv:2507.21836}
}
read the original abstract

Large Language Models (LLMs), when enhanced through reasoning-oriented post-training, evolve into powerful Large Reasoning Models (LRMs). Tool-Integrated Reasoning (TIR) further extends their capabilities by incorporating external tools, but existing methods often rely on rigid, predefined tool-use patterns that risk degrading core language competence. Inspired by the human ability to adaptively select tools, we introduce AutoTIR, a reinforcement learning framework that enables LLMs to autonomously decide whether and which tool to invoke during the reasoning process, rather than following static tool-use strategies. AutoTIR leverages a hybrid reward mechanism that jointly optimizes for task-specific answer correctness, structured output adherence, and penalization of incorrect tool usage, thereby encouraging both precise reasoning and efficient tool integration. Extensive evaluations across diverse knowledge-intensive, mathematical, and general language modeling tasks demonstrate that AutoTIR achieves superior overall performance, significantly outperforming baselines and exhibits superior generalization in tool-use behavior. These results highlight the promise of reinforcement learning in building truly generalizable and scalable TIR capabilities in LLMs. The code and data are available at https://github.com/weiyifan1023/AutoTIR.

Figures

Figures reproduced from arXiv: 2507.21836 by the authors.

Figure 1
Figure 1. AutoTIR balances the tool-integrated reasoning [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overall framework of AutoTIR. Top: Comparison between AutoTIR and existing paradigms (fixed reasoning strategy vs. autonomous decision). Bottom: GRPO training pipeline that incorporates multiple reasoning actions. ∀k ∈ [1, N], tk = ∅, ok = ∅. The iterative generation pro￾cess proceeds as follows: (sk, tk) = M (Q ⊕ τk−1) ok = E (tk) τk = τk−1 ⊕ Ak (2) This cycle continues until the model M produces a final an￾swer or… view at source ↗
Figure 3
Figure 3. Avg. reward score and response length during training. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Model Performance and Tool Advantage Across [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: System prompt template for training and inference from AutoTIR. [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: System prompt template for text-based inference from AutoTIR. [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Performance metrics across different datasets during training stage. [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Test-Time comparison [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TurnSight: Turn-Level Hindsight Self-Distillation for Tool-Integrated Reasoning

    cs.CL 2026-08 conditional novelty 6.0 of 10

    TurnSight uses a frozen reference model plus future tool outputs to score each reasoning turn, then uses those scores to fine-tune an RL agent's credit assignment, beating existing methods on three benchmarks.

  2. Reasoning and Tool-use Compete in Agentic RL:From Quantifying Interference to Disentangled Tuning

    cs.AI 2026-02 conditional novelty 6.0 of 10

    Jointly optimizing reasoning and tool use in agentic RL interferes with both; separating the two into disjoint LoRA adapters (DART) improves retrieval-augmented QA.

  3. Lost in Execution: On the Multilingual Robustness of Tool Calling in Large Language Models

    cs.CL 2026-01 conditional novelty 6.0 of 10

    A new benchmark shows multilingual tool-calling errors in LLMs are mostly parameter-language mismatches at the execution boundary, not failures of intent understanding.

  4. Learning How to Use Tools, Not Just When: Pattern-Aware Tool-Integrated Reasoning

    cs.AI 2025-09 reject novelty 6.0 of 10

    A two-stage pattern-aware tool-integrated reasoning method raises code usage and code-plus-correct metrics on math benchmarks, but the paper conflates Code@1 with problem-solving accuracy in its headline claims.

  5. Chain-of-Agents: End-to-End Agent Foundation Models via Multi-Agent Distillation and Agentic RL

    cs.AI 2025-08 conditional novelty 6.0 of 10

    A single LLM is trained with multi-agent distilled trajectories plus agentic RL, and the resulting Chain-of-Agents models set state-of-the-art Pass@1 scores among tool-integrated reasoning methods on GAIA, BrowseComp,...

  6. Toward Efficient Agents: Memory, Tool learning, and Planning

    cs.AI 2026-01 conditional novelty 3.0 of 10

    A survey that organizes efficiency techniques for LLM agents into memory, tool learning, and planning, and consolidates benchmarks and metrics for measuring cost-performance trade-offs.

  7. SIGMA: Search-Augmented On-Demand Knowledge Integration for Agentic Mathematical Reasoning

    cs.AI 2025-10 reject novelty 3.0 of 10

    SIGMA uses four specialized retrieval agents plus a moderator to improve math reasoning, reporting up to 7.4 absolute MATH500 points over Search-o1 at 1.5B scale.

Reference graph

Works this paper leans on

71 extracted references · 34 canonical work pages · cited by 7 Pith papers

  1. [1]

    H.; Meade, N.; and Reddy, S

    Adlakha, V.; BehnamGhader, P.; Lu, X. H.; Meade, N.; and Reddy, S. 2024. Evaluating correctness and faithfulness of instruction-following models for question answering. Transactions of the Association for Computational Linguistics, 12: 681--699

    work page 2024

  2. [2]

    Z.; Zhang, W.; Chen, H.; Yang, F.; et al

    Chen, M.; Li, T.; Sun, H.; Zhou, Y.; Zhu, C.; Wang, H.; Pan, J. Z.; Zhang, W.; Chen, H.; Yang, F.; et al. 2025. Learning to reason with search for llms via reinforcement learning. arXiv preprint arXiv:2503.19470

    arXiv 2025

  3. [3]

    V.; Levine, S.; and Ma, Y

    Chu, T.; Zhai, Y.; Yang, J.; Tong, S.; Xie, S.; Schuurmans, D.; Le, Q. V.; Levine, S.; and Ma, Y. 2025. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. arXiv preprint arXiv:2501.17161

    arXiv 2025

  4. [4]

    Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    arXiv 2021

  5. [5]

    Cui, G.; Yuan, L.; Wang, Z.; Wang, H.; Li, W.; He, B.; Fan, Y.; Yu, T.; Xu, Q.; Chen, W.; et al. 2025. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456

    arXiv 2025

  6. [6]

    Das, D.; Banerjee, D.; Aditya, S.; and Kulkarni, A. 2024. MATHSENSEI: a tool-augmented large language model for mathematical reasoning. arXiv preprint arXiv:2402.17231

    arXiv 2024

  7. [7]

    Dong, G.; Chen, Y.; Li, X.; Jin, J.; Qian, H.; Zhu, Y.; Mao, H.; Zhou, G.; Dou, Z.; and Wen, J.-R. 2025. Tool-Star: Empowering LLM-Brained Multi-Tool Reasoner via Reinforcement Learning. arXiv preprint arXiv:2505.16410

    arXiv 2025

  8. [8]

    Du, M.; He, F.; Zou, N.; Tao, D.; and Hu, X. 2023. Shortcut learning of large language models in natural language understanding. Communications of the ACM, 67(1): 110--120

    work page 2023

Show all 71 references
  1. [9]

    Feng, J.; Huang, S.; Qu, X.; Zhang, G.; Qin, Y.; Zhong, B.; Jiang, C.; Chi, J.; and Zhong, W. 2025 a . Retool: Reinforcement learning for strategic tool use in llms. arXiv preprint arXiv:2504.11536

    2025 arXiv

  2. [10]

    Feng, Z.; Cao, S.; Ren, J.; Su, J.; Chen, R.; Zhang, Y.; Xu, Z.; Hu, Y.; Wu, J.; and Liu, Z. 2025 b . Mt-r1-zero: Advancing llm-based machine translation via r1-zero-like reinforcement learning. arXiv preprint arXiv:2504.10160

    2025 arXiv

  3. [11]

    Fu, T.; Gu, J.; Li, Y.; Qu, X.; and Cheng, Y. 2025. Scaling reasoning, losing control: Evaluating instruction following in large reasoning models. arXiv preprint arXiv:2505.14810

    2025 arXiv

  4. [12]

    Gehring, J.; Zheng, K.; Copet, J.; Mella, V.; Cohen, T.; and Synnaeve, G. 2025. RLEF : Grounding Code LLM s in Execution Feedback with Reinforcement Learning. In Forty-second International Conference on Machine Learning

    2025

  5. [13]

    Gou, Z.; Shao, Z.; Gong, Y.; Shen, Y.; Yang, Y.; Huang, M.; Duan, N.; and Chen, W. 2023. Tora: A tool-integrated reasoning agent for mathematical problem solving. arXiv preprint arXiv:2309.17452

    2023 arXiv

  6. [14]

    Guo, D.; Yang, D.; Zhang, H.; Song, J.; Zhang, R.; Xu, R.; Zhu, Q.; Ma, S.; Wang, P.; Bi, X.; et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    2025 arXiv

  7. [15]

    Havrilla, A.; Raparthy, S.; Nalmpantis, C.; Dwivedi-Yu, J.; Zhuravynski, M.; Hambro, E.; and Raileanu, R. 2024. GLoRe: when, where, and how to improve LLM reasoning via global and local refinements. In Proceedings of the 41st International Conference on Machine Learning, 17719--17733

    2024

  8. [16]

    Hendrycks, D.; Burns, C.; Kadavath, S.; Arora, A.; Basart, S.; Tang, E.; Song, D.; and Steinhardt, J. 2021. Measuring Mathematical Problem Solving With the MATH Dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)

    2021

  9. [17]

    D.; Sugawara, S.; and Aizawa, A

    Ho, X.; Nguyen, A.-K. D.; Sugawara, S.; and Aizawa, A. 2020. Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps. In Proceedings of the 28th International Conference on Computational Linguistics, 6609--6625

    2020

  10. [18]

    D.; Phan, D.; Dohan, D.; Douglas, S.; Le, T

    Hoffman, M. D.; Phan, D.; Dohan, D.; Douglas, S.; Le, T. A.; Parisi, A.; Sountsov, P.; Sutton, C.; Vikram, S.; and Saurous, R. A. 2023. Training chain-of-thought via latent-variable inference. In NeurIPS

    2023

  11. [19]

    T.; Rahman, S.; Morol, M

    Hosain, M. T.; Rahman, S.; Morol, M. K.; and Parvez, M. R. 2025. Xolver: Multi-Agent Reasoning with Holistic Experience Learning Just Like an Olympiad Team. arXiv preprint arXiv:2506.14234

    2025 arXiv

  12. [20]

    Huang, Z.; Yuan, X.; Ju, Y.; Zhao, J.; and Liu, K. 2025. Reinforced Internal-External Knowledge Synergistic Reasoning for Efficient Adaptive Search Agent. arXiv preprint arXiv:2505.07596

    2025 arXiv

  13. [21]

    Jin, B.; Zeng, H.; Yue, Z.; Yoon, J.; Arik, S.; Wang, D.; Zamani, H.; and Han, J. 2025 a . Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516

    2025 arXiv

  14. [22]

    Jin, J.; Zhu, Y.; Dou, Z.; Dong, G.; Yang, X.; Zhang, C.; Zhao, T.; Yang, Z.; and Wen, J.-R. 2025 b . Flashrag: A modular toolkit for efficient retrieval-augmented generation research. In Companion Proceedings of the ACM on Web Conference 2025, 737--740

    2025

  15. [23]

    Kazemnejad, A.; Aghajohari, M.; Portelance, E.; Sordoni, A.; Reddy, S.; Courville, A.; and Le Roux, N. 2025. VinePPO: Refining Credit Assignment in RL Training of LLMs. In Forty-second International Conference on Machine Learning

    2025

  16. [24]

    Kwiatkowski, T.; Palomaki, J.; Redfield, O.; Collins, M.; Parikh, A.; Alberti, C.; Epstein, D.; Polosukhin, I.; Devlin, J.; Lee, K.; et al. 2019. Natural Questions: A Benchmark for Question Answering Research. Transactions of the Association for Computational Linguistics, 7: 452--466

    2019

  17. [25]

    Lambert, N.; Morrison, J.; Pyatkin, V.; Huang, S.; Ivison, H.; Brahman, F.; Miranda, L. J. V.; Liu, A.; Dziri, N.; Lyu, S.; et al. 2024. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124

    2024 arXiv

  18. [26]

    Li, C.; Liang, J.; Zeng, A.; Chen, X.; Hausman, K.; Sadigh, D.; Levine, S.; Fei-Fei, L.; Xia, F.; and Ichter, B. 2024. Chain of code: reasoning with a language model-augmented code emulator. In Proceedings of the 41st International Conference on Machine Learning, 28259--28277

    2024

  19. [27]

    Li, M.; Zhao, Y.; Yu, B.; Song, F.; Li, H.; Yu, H.; Li, Z.; Huang, F.; and Li, Y. 2023. API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs. In The 2023 Conference on Empirical Methods in Natural Language Processing

    2023

  20. [28]

    Li, X.; Yu, Z.; Zhang, Z.; Chen, X.; Zhang, Z.; Zhuang, Y.; Sadagopan, N.; and Beniwal, A. 2025. When thinking fails: The pitfalls of reasoning for instruction-following in llms. arXiv preprint arXiv:2505.11423

    2025 arXiv

  21. [29]

    Li, X.; Zou, H.; and Liu, P. 2025. Torl: Scaling tool-integrated rl. arXiv preprint arXiv:2503.23383

    2025 arXiv

  22. [30]

    Liu, J.; Cui, L.; Liu, H.; Huang, D.; Wang, Y.; and Zhang, Y. 2021. LogiQA: a challenge dataset for machine reading comprehension with logical reasoning. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, ...

    2021

  23. [31]

    Liu, W.; Huang, X.; Zeng, X.; xinlong hao; Yu, S.; Li, D.; Wang, S.; Gan, W.; Liu, Z.; Yu, Y.; WANG, Z.; Wang, Y.; Ning, W.; Hou, Y.; Wang, B.; Wu, C.; Xinzhi, W.; Liu, Y.; Wang, Y.; Tang, D.; Tu, D.; Shang, L.; Jiang, X.; Tang, R.; Lian, D.; Liu, Q.; and Chen, E. 2025. Tool A...

    2025

  24. [32]

    Lu, P.; Chen, B.; Liu, S.; Thapa, R.; Boen, J.; and Zou, J. 2025. OctoTools: An Agentic Framework with Extensible Tools for Complex Reasoning. In ICLR 2025 Workshop on Foundation Models in the Wild

    2025

  25. [33]

    Ma, Y.; Gou, Z.; Hao, J.; Xu, R.; Wang, S.; Pan, L.; Yang, Y.; Cao, Y.; and Sun, A. 2024. SciAgent: Tool-augmented Language Models for Scientific Reasoning. In EMNLP

    2024

  26. [34]

    Mai, X.; Xu, H.; Wang, W.; Zhang, Y.; Zhang, W.; et al. 2025. Agent rl scaling law: Agent rl with spontaneous code execution for mathematical problem solving. arXiv preprint arXiv:2505.07773

    2025 arXiv

  27. [35]

    Meng, Y.; Xia, M.; and Chen, D. 2024. Simpo: Simple preference optimization with a reference-free reward. Advances in Neural Information Processing Systems, 37: 124198--124235

    2024

  28. [36]

    Minaee, S.; Mikolov, T.; Nikzad, N.; Chenaghlu, M.; Socher, R.; Amatriain, X.; and Gao, J. 2024. Large language models: A survey. arXiv preprint arXiv:2402.06196

    2024 arXiv

  29. [37]

    OpenAI . 2024. Learning to Reason with LLMs

    2024

  30. [38]

    Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 27730--27744

    2022

  31. [39]

    A.; and Lewis, M

    Press, O.; Zhang, M.; Min, S.; Schmidt, L.; Smith, N. A.; and Lewis, M. 2023. Measuring and Narrowing the Compositionality Gap in Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2023, 5687--5711

    2023

  32. [40]

    Qin, Y.; Hu, S.; Lin, Y.; Chen, W.; Ding, N.; Cui, G.; Zeng, Z.; Zhou, X.; Huang, Y.; Xiao, C.; et al. 2024 a . Tool learning with foundation models. ACM Computing Surveys, 57(4): 1--40

    2024

  33. [41]

    Qin, Y.; Liang, S.; Ye, Y.; Zhu, K.; Yan, L.; Lu, Y.; Lin, Y.; Cong, X.; Tang, X.; Qian, B.; Zhao, S.; Hong, L.; Tian, R.; Xie, R.; Zhou, J.; Gerstein, M.; dahai li; Liu, Z.; and Sun, M. 2024 b . Tool LLM : Facilitating Large Language Models to Master 16000+ Real-world API s. ...

    2024

  34. [42]

    Qu, C.; Dai, S.; Wei, X.; Cai, H.; Wang, S.; Yin, D.; Xu, J.; and Wen, J.-r. 2025. Tool learning with large language models: a survey. Frontiers of Computer Science, 19(8)

    2025

  35. [43]

    D.; Ermon, S.; and Finn, C

    Rafailov, R.; Sharma, A.; Mitchell, E.; Manning, C. D.; Ermon, S.; and Finn, C. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36: 53728--53741

    2023

  36. [44]

    Schick, T.; Dwivedi-Yu, J.; Dess \` , R.; Raileanu, R.; Lomeli, M.; Hambro, E.; Zettlemoyer, L.; Cancedda, N.; and Scialom, T. 2023. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36: 68539--68551

    2023

  37. [45]

    Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and Klimov, O. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347

    2017 arXiv

  38. [46]

    Shao, Z.; Gong, Y.; Shen, Y.; Huang, M.; Duan, N.; and Chen, W. 2023. Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy. In Findings of the Association for Computational Linguistics: EMNLP 2023, 9248--9274

    2023

  39. [47]

    Shao, Z.; Wang, P.; Zhu, Q.; Xu, R.; Song, J.; Bi, X.; Zhang, H.; Zhang, M.; Li, Y.; Wu, Y.; et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300

    2024 arXiv

  40. [48]

    Sheng, G.; Zhang, C.; Ye, Z.; Wu, X.; Zhang, W.; Zhang, R.; Peng, Y.; Lin, H.; and Wu, C. 2025. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, 1279--1297

    2025

  41. [49]

    X.; Fang, L.; and Wen, J.-R

    Song, H.; Jiang, J.; Min, Y.; Chen, J.; Chen, Z.; Zhao, W. X.; Fang, L.; and Wen, J.-R. 2025. R1-searcher: Incentivizing the search capability in llms via reinforcement learning. arXiv preprint arXiv:2503.05592

    2025 arXiv

  42. [50]

    Team, Q. 2024 a . Qwen2.5: A Party of Foundation Models

    2024

  43. [51]

    Team, Q. 2024 b . Qwq: Reflect deeply on the boundaries of the unknown. Hugging Face

    2024

  44. [52]

    Trivedi, H.; Balasubramanian, N.; Khot, T.; and Sabharwal, A. 2022. MuSiQue: Multihop Questions via Single-hop Question Composition. Transactions of the Association for Computational Linguistics, 10: 539--554

    2022

  45. [53]

    Trivedi, H.; Balasubramanian, N.; Khot, T.; and Sabharwal, A. 2023. Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long P...

    2023

  46. [54]

    Wang, H.; Qian, C.; Zhong, W.; Chen, X.; Qiu, J.; Huang, S.; Jin, B.; Wang, M.; Wong, K.-F.; and Ji, H. 2025. Otc: Optimal tool calls via reinforcement learning. arXiv e-prints, arXiv--2504

    2025

  47. [55]

    Wang, K.; Ren, H.; Zhou, A.; Lu, Z.; Luo, S.; Shi, W.; Zhang, R.; Song, L.; Zhan, M.; and Li, H. 2024. MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning. In ICLR

    2024

  48. [56]

    Wang, L.; Yang, N.; Huang, X.; Jiao, B.; Yang, L.; Jiang, D.; Majumder, R.; and Wei, F. 2022. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533

    2022 arXiv

  49. [57]

    Wei, Y.; Su, Y.; Ma, H.; Yu, X.; Lei, F.; Zhang, Y.; Zhao, J.; and Liu, K. 2023. Menatqa: A new dataset for testing the temporal comprehension and reasoning abilities of large language models. arXiv preprint arXiv:2310.05157

    2023 arXiv

  50. [58]

    Wei, Y.; Yu, X.; Pan, T.; Li, A.; and Du, L. 2025. Structural Entropy Guided Agent for Detecting and Repairing Knowledge Deficiencies in LLMs. arXiv preprint arXiv:2505.07184

    2025 arXiv

  51. [59]

    Xu, S.; Fu, W.; Gao, J.; Ye, W.; Liu, W.; Mei, Z.; Wang, G.; Yu, C.; and Wu, Y. 2024. Is DPO superior to PPO for LLM alignment? a comprehensive study. In Proceedings of the 41st International Conference on Machine Learning, 54983--54998

    2024

  52. [60]

    Yang, R.; Song, L.; Li, Y.; Zhao, S.; Ge, Y.; Li, X.; and Shan, Y. 2023. Gpt4tools: Teaching large language model to use tools via self-instruction. Advances in Neural Information Processing Systems, 36: 71995--72007

    2023

  53. [61]

    Yang, Z.; Qi, P.; Zhang, S.; Bengio, Y.; Cohen, W.; Salakhutdinov, R.; and Manning, C. D. 2018. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2369--2380

    2018

  54. [62]

    Yao, S.; Zhao, J.; Yu, D.; Du, N.; Shafran, I.; Narasimhan, K.; and Cao, Y. 2023. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR)

    2023

  55. [63]

    Ye, J.; Li, G.; Gao, S.; Huang, C.; Wu, Y.; Li, S.; Fan, X.; Dou, S.; Ji, T.; Zhang, Q.; et al. 2025 a . ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarios. In Proceedings of the 31st International Conference on Co...

    2025

  56. [64]

    C.; Kaddar, Y.; Blunsom, P.; Staton, S.; and Gal, Y

    Ye, Z.; Melo, L. C.; Kaddar, Y.; Blunsom, P.; Staton, S.; and Gal, Y. 2025 b . Uncertainty-Aware Step-wise Verification with Generative Reward Models. In ICLR Workshop: Quantify Uncertainty and Hallucination in Foundation Models: The Next Frontier in Reliable AI

    2025

  57. [65]

    Yu, Q.; Zhang, Z.; Zhu, R.; Yuan, Y.; Zuo, X.; Yue, Y.; Dai, W.; Fan, T.; Liu, G.; Liu, L.; et al. 2025. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476

    2025 arXiv

  58. [66]

    Zelikman, E.; Wu, Y.; Mu, J.; and Goodman, N. 2022. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35: 15476--15488

    2022

  59. [67]

    Zeng, W.; Huang, Y.; Liu, Q.; Liu, W.; He, K.; Ma, Z.; and He, J. 2025. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503.18892

    2025 arXiv

  60. [68]

    Zhou, J.; Lu, T.; Mishra, S.; Brahma, S.; Basu, S.; Luan, Y.; Zhou, D.; and Hou, L. 2023. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911

    2023 arXiv

  61. [69]

    Zhou, Z.; Qu, A.; Wu, Z.; Kim, S.; Prakash, A.; Rus, D.; Zhao, J.; Low, B. K. H.; and Liang, P. P. 2025. MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents. arXiv preprint arXiv:2506.15841

    2025 arXiv

  62. [70]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  63. [71]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.