pith. machine review for the scientific record. sign in

arxiv: 2604.17433 · v1 · submitted 2026-04-19 · 💻 cs.CL · cs.AI· cs.LG

Recognition: unknown

Self-Consistency from Only Two Samples: CoT-PoT Ensembling for Efficient LLM Reasoning

Majd Hawasly, Md Rizwan Parvez, Mohammad Raza, Raman Saparkhan

Authors on Pith no claims yet

Pith reviewed 2026-05-10 05:36 UTC · model grok-4.3

classification 💻 cs.CL cs.AIcs.LG
keywords self-consistencychain-of-thoughtprogram-of-thoughtLLM reasoningensemblingearly stoppingefficient inferencelarge language models
0
0 comments X

The pith

CoT-PoT ensembling cuts the samples needed for LLM self-consistency by 9.3 times while raising accuracy.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces a hybrid approach that combines Chain-of-Thought and Program-of-Thought reasoning inside the self-consistency process for large language models. This method uses the complementary strengths of verbal step-by-step reasoning and executable program-style reasoning to reach consistent answers. It reports both higher overall accuracy and a sharp drop in the number of samples required, with most tasks handled by only two samples. A reader would care because the approach lowers the high computational cost that has limited self-consistency in practice.

Core claim

The authors establish that ensembling Chain-of-Thought and Program-of-Thought outputs within self-consistency improves accuracy and reduces the required samples by a factor of 9.3. In particular, 78.6 percent of tasks can be solved correctly with only two samples through agreement-based early stopping. The framework supports both full sampling and early-stopping strategies that exploit the two distinct reasoning modes.

What carries the argument

The CoT-PoT ensembling framework that aggregates outputs from Chain-of-Thought and Program-of-Thought reasoning paths and stops early when they agree.

If this is right

  • Accuracy on reasoning benchmarks rises above standard self-consistency baselines.
  • The average number of samples per task falls by a factor of 9.3.
  • 78.6 percent of tasks reach correct answers with only two samples.
  • Early stopping based on mode agreement becomes practical for many problems.
  • Computational cost for inference drops while maintaining or improving reliability.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Pairs of other reasoning formats might produce similar sample reductions if they remain complementary.
  • The method suggests that format diversity can substitute for sample quantity in self-consistency.
  • Real-time applications with tight latency budgets could adopt dual-mode sampling as a default.
  • Extending the approach to additional reasoning styles or domains would test its generality.

Load-bearing premise

That Chain-of-Thought and Program-of-Thought outputs are sufficiently complementary so their agreement reliably signals the correct answer without new error modes.

What would settle it

A dataset where CoT and PoT outputs agree on wrong answers at a high rate, causing accuracy to fall below that of standard self-consistency with more samples.

Figures

Figures reproduced from arXiv: 2604.17433 by Majd Hawasly, Md Rizwan Parvez, Mohammad Raza, Raman Saparkhan.

Figure 1
Figure 1. Figure 1: CoT-PoT consistency provides the highest accuracy, the highest efficiency and can solve most problems [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Percentage of problems solved with only two [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Efficiency vs. sampling budget across differ [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
read the original abstract

Self-consistency (SC) is a popular technique for improving the reasoning accuracy of large language models by aggregating multiple sampled outputs, but it comes at a high computational cost due to extensive sampling. We introduce a hybrid ensembling approach that leverages the complementary strengths of two distinct modes of reasoning: Chain-of-Thought (CoT) and Program-of-Thought (PoT). We describe a general framework for combining these two forms of reasoning in self-consistency, as well as particular strategies for both full sampling and early-stopping. We show that CoT-PoT ensembling not only improves overall accuracy, but also drastically reduces the number of samples required for SC by a factor of 9.3x. In particular, the majority of tasks (78.6%) can be addressed with only two samples, which has not been possible with any prior SC methods.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper introduces a hybrid CoT-PoT ensembling approach within the self-consistency framework for LLMs. It combines Chain-of-Thought and Program-of-Thought reasoning modes, with strategies for both full sampling and early-stopping on agreement, claiming not only higher overall accuracy but also a 9.3x reduction in required samples, such that 78.6% of tasks can be solved with only two samples.

Significance. If the efficiency claims hold with preserved accuracy, the work could meaningfully reduce the computational overhead of self-consistency, making it more practical for deployment. The empirical reporting of measured sample reductions is a strength, but the absence of conditional accuracy breakdowns on early-stopped cases weakens the ability to assess whether the gains are achieved without new error modes.

major comments (2)
  1. [Abstract] Abstract: The headline efficiency claim (9.3x sample reduction and 78.6% of tasks solved with exactly two samples via early-stopping on CoT-PoT agreement) is load-bearing for the contribution, yet the abstract provides no indication that accuracy is broken out for the early-stopped subset versus the continued-sampling subset. Without this, it is impossible to verify that agreement after one CoT and one PoT is a correctness signal comparable to full majority vote.
  2. [Early-stopping strategy] Early-stopping strategy: The premise that CoT and PoT outputs are sufficiently complementary for their agreement to serve as a reliable stopping criterion (without introducing new error modes or requiring additional samples) is not supported by any reported conditional accuracy or error analysis on the 78.6% early-stopped cases. This risks the efficiency gains being achieved by selectively accepting lower-confidence answers on the majority of examples.
minor comments (1)
  1. [Abstract] The abstract refers to 'particular strategies for both full sampling and early-stopping' without sufficient detail on implementation or pseudocode; adding a concise algorithmic description would improve reproducibility.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their thoughtful comments, which identify key areas where additional clarity on our efficiency claims would strengthen the paper. We address each major comment below and indicate the revisions made.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The headline efficiency claim (9.3x sample reduction and 78.6% of tasks solved with exactly two samples via early-stopping on CoT-PoT agreement) is load-bearing for the contribution, yet the abstract provides no indication that accuracy is broken out for the early-stopped subset versus the continued-sampling subset. Without this, it is impossible to verify that agreement after one CoT and one PoT is a correctness signal comparable to full majority vote.

    Authors: We agree that the abstract should better contextualize the efficiency results with respect to accuracy preservation. In the revised manuscript, we have updated the abstract to state that accuracy on the early-stopped subset remains comparable to full self-consistency, with a reference to the new conditional analysis added in the experiments section. This makes the load-bearing claim more transparent. revision: yes

  2. Referee: [Early-stopping strategy] Early-stopping strategy: The premise that CoT and PoT outputs are sufficiently complementary for their agreement to serve as a reliable stopping criterion (without introducing new error modes or requiring additional samples) is not supported by any reported conditional accuracy or error analysis on the 78.6% early-stopped cases. This risks the efficiency gains being achieved by selectively accepting lower-confidence answers on the majority of examples.

    Authors: We acknowledge that the original manuscript did not report conditional accuracy or error analysis specifically for the early-stopped cases, which limits the ability to fully validate the stopping criterion. We have added this analysis to the revised version, including accuracy breakdowns and error comparisons for the 78.6% of tasks. The new results confirm that agreement after one CoT and one PoT does not introduce new error modes and yields accuracy comparable to full sampling on those instances. revision: yes

Circularity Check

0 steps flagged

No circularity in empirical ensembling method

full rationale

The paper is an empirical study proposing CoT-PoT hybrid ensembling for self-consistency, with full-sampling and early-stopping strategies. It reports measured outcomes such as 9.3x sample reduction and 78.6% of tasks solved with two samples. No equations, derivations, or self-referential definitions exist that would make any result equivalent to its inputs by construction. Claims rest on experimental validation rather than fitted parameters renamed as predictions or self-citation chains. The work is self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central efficiency claim rests on the unstated premise that CoT and PoT reasoning paths produce sufficiently independent errors so that their early agreement is a reliable stopping signal; no free parameters or invented entities are mentioned.

axioms (1)
  • domain assumption CoT and PoT outputs are complementary enough that their agreement indicates correctness with high probability after only two samples.
    This premise is required for the early-stopping strategy to preserve accuracy while reducing sample count.

pith-pipeline@v0.9.0 · 5464 in / 1318 out tokens · 35144 ms · 2026-05-10T05:36:51.832748+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

101 extracted references · 43 canonical work pages · 17 internal anchors

  1. [1]

    Aho and Jeffrey D

    Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

    1972

  2. [2]

    Publications Manual , year = "1983", publisher =

    1983

  3. [3]

    Chandra and Dexter C

    Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243

    work page doi:10.1145/322234.322243 1981

  4. [4]

    Scalable training of

    Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of

  5. [5]

    Dan Gusfield , title =. 1997

    1997

  6. [6]

    Tetreault , title =

    Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =

    2015

  7. [7]

    A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =

    Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =

  8. [8]

    Langley , title =

    P. Langley , title =. Proceedings of the 17th International Conference on Machine Learning (ICML 2000) , address =. 2000 , pages =

    2000

  9. [9]

    Scaling test-time compute with open models , author=

  10. [10]

    T. M. Mitchell. The Need for Biases in Learning Generalizations. 1980

    1980

  11. [11]

    M. J. Kearns , title =

  12. [12]

    Machine Learning: An Artificial Intelligence Approach, Vol. I. 1983

    1983

  13. [13]

    R. O. Duda and P. E. Hart and D. G. Stork. Pattern Classification. 2000

    2000

  14. [14]

    Suppressed for Anonymity , author=

  15. [15]

    ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving

    Gou, Zhibin and Shao, Zhihong and Gong, Yeyun and Shen, Yelong and Yang, Yujiu and Huang, Minlie and Duan, Nan and Chen, Weizhu , biburl =. ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving. , url =. ICLR , ee =

  16. [16]

    Newell and P

    A. Newell and P. S. Rosenbloom. Mechanisms of Skill Acquisition and the Law of Practice. Cognitive Skills and Their Acquisition. 1981

    1981

  17. [17]

    A. L. Samuel. Some Studies in Machine Learning Using the Game of Checkers. IBM Journal of Research and Development. 1959

    1959

  18. [18]

    2024 , howpublished =

    OpenAI , title =. 2024 , howpublished =

    2024

  19. [19]

    2025 , howpublished =

    OpenAI , title =. 2025 , howpublished =

    2025

  20. [20]

    Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters

    Scaling llm test-time compute optimally can be more effective than scaling model parameters , author=. arXiv preprint arXiv:2408.03314 , year=

    work page Pith review arXiv

  21. [21]

    2024 , howpublished =

    DeepSeek , title =. 2024 , howpublished =

    2024

  22. [22]

    2024 , howpublished =

    Qwen , title =. 2024 , howpublished =

    2024

  23. [23]

    Measuring Coding Challenge Competence With APPS

    Measuring coding challenge competence with apps , author=. arXiv preprint arXiv:2105.09938 , year=

    work page internal anchor Pith review arXiv

  24. [24]

    GPT-4 Technical Report

    Gpt-4 technical report , author=. arXiv preprint arXiv:2303.08774 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  25. [25]

    Qwen2.5-Coder Technical Report

    Qwen2. 5-Coder Technical Report , author=. arXiv preprint arXiv:2409.12186 , year=

    work page internal anchor Pith review arXiv

  26. [26]

    Qwen2 Technical Report

    Qwen2 Technical Report , author=. arXiv preprint arXiv:2407.10671 , year=

    work page internal anchor Pith review arXiv

  27. [27]

    Let's Verify Step by Step

    Let's verify step by step , author=. arXiv preprint arXiv:2305.20050 , year=

    work page internal anchor Pith review arXiv

  28. [28]

    Advances in neural information processing systems , volume=

    Chain-of-thought prompting elicits reasoning in large language models , author=. Advances in neural information processing systems , volume=

  29. [29]

    DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines

    Dspy: Compiling declarative language model calls into self-improving pipelines , author=. arXiv preprint arXiv:2310.03714 , year=

    work page internal anchor Pith review arXiv

  30. [30]

    Advances in Neural Information Processing Systems , volume=

    Self-refine: Iterative refinement with self-feedback , author=. Advances in Neural Information Processing Systems , volume=

  31. [31]

    Physics of language models: Part 2.2, how to learn from mistakes on grade-school math problems

    Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems , author=. arXiv preprint arXiv:2408.16293 , year=

    work page arXiv

  32. [32]

    Advances in Neural Information Processing Systems , volume=

    Judging llm-as-a-judge with mt-bench and chatbot arena , author=. Advances in Neural Information Processing Systems , volume=

  33. [33]

    arXiv preprint arXiv:2207.10397 , year=

    Codet: Code generation with generated tests , author=. arXiv preprint arXiv:2207.10397 , year=

    work page arXiv

  34. [34]

    Teaching Large Language Models to Self-Debug

    Teaching large language models to self-debug , author=. arXiv preprint arXiv:2304.05128 , year=

    work page internal anchor Pith review arXiv

  35. [35]

    Transactions of the Association for Computational Linguistics , volume=

    Lost in the middle: How language models use long contexts , author=. Transactions of the Association for Computational Linguistics , volume=. 2024 , publisher=

    2024

  36. [36]

    LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code

    Livecodebench: Holistic and contamination free evaluation of large language models for code , author=. arXiv preprint arXiv:2403.07974 , year=

    work page internal anchor Pith review arXiv

  37. [37]

    Evaluating Large Language Models Trained on Code

    Evaluating large language models trained on code , author=. arXiv preprint arXiv:2107.03374 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  38. [38]

    NeurIPS , year=

    Measuring Coding Challenge Competence With APPS , author=. NeurIPS , year=

  39. [39]

    Science , volume=

    Competition-level code generation with alphacode , author=. Science , volume=. 2022 , publisher=

    2022

  40. [40]

    Large Language Monkeys: Scaling Inference Compute with Repeated Sampling

    Large language monkeys: Scaling inference compute with repeated sampling , author=. arXiv preprint arXiv:2407.21787 , year=

    work page internal anchor Pith review arXiv

  41. [41]

    s1: Simple test-time scaling

    s1: Simple test-time scaling , author=. arXiv preprint arXiv:2501.19393 , year=

    work page Pith review arXiv

  42. [42]

    arXiv preprint arXiv:2501.14723 , year=

    CodeMonkeys: Scaling Test-Time Compute for Software Engineering , author=. arXiv preprint arXiv:2501.14723 , year=

    work page arXiv

  43. [43]

    NovaSky Team , title =

  44. [44]

    Self-Consistency Improves Chain of Thought Reasoning in Language Models

    Self-consistency improves chain of thought reasoning in language models , author=. arXiv preprint arXiv:2203.11171 , year=

    work page Pith review arXiv

  45. [45]

    Advances in neural information processing systems , volume=

    Deep reinforcement learning from human preferences , author=. Advances in neural information processing systems , volume=

  46. [46]

    On the Measure of Intelligence

    On the measure of intelligence , author=. arXiv preprint arXiv:1911.01547 , year=

    work page internal anchor Pith review arXiv 1911

  47. [47]

    Rewarding chatbots for real-world engagement with millions of users, 2023

    Rewarding chatbots for real-world engagement with millions of users , author=. arXiv preprint arXiv:2303.06135 , year=

    work page arXiv

  48. [48]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Math-shepherd: Verify and reinforce llms step-by-step without human annotations , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  49. [49]

    The larger the better? improved llm code-generation via budget reallocation, 2024

    The Larger the Better? Improved LLM Code-Generation via Budget Reallocation , author=. arXiv preprint arXiv:2404.00725 , year=

    work page arXiv

  50. [50]

    Inference scaling f laws: The limits of llm resampling with imperfect verifiers.arXiv preprint arXiv:2411.17501, 2024

    Inference Scaling fLaws: The Limits of LLM Resampling with Imperfect Verifiers , author=. arXiv preprint arXiv:2411.17501 , year=

    work page arXiv

  51. [51]

    Show Your Work: Scratchpads for Intermediate Computation with Language Models

    Show your work: Scratchpads for intermediate computation with language models , author=. arXiv preprint arXiv:2112.00114 , year=

    work page internal anchor Pith review arXiv

  52. [52]

    Advancing language model reasoning through reinforcement learning and inference scaling.arXiv preprint arXiv:2501.11651, 2025

    Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling , author=. arXiv preprint arXiv:2501.11651 , year=

    work page arXiv

  53. [53]

    Evolving deeper LLM thinking

    Evolving Deeper LLM Thinking , author=. arXiv preprint arXiv:2501.09891 , year=

    work page arXiv

  54. [54]

    Huang, S

    Large language models can self-improve , author=. arXiv preprint arXiv:2210.11610 , year=

    work page arXiv

  55. [55]

    Imitate, explore, and self-improve: A reproduction report on slow-thinking reasoning systems.arXiv preprint arXiv:2412.09413, 2024

    Imitate, explore, and self-improve: A reproduction report on slow-thinking reasoning systems , author=. arXiv preprint arXiv:2412.09413 , year=

    work page arXiv

  56. [56]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning , author=. arXiv preprint arXiv:2501.12948 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  57. [57]

    Kimi k1.5: Scaling Reinforcement Learning with LLMs

    Kimi k1. 5: Scaling Reinforcement Learning with LLMs , author=. arXiv preprint arXiv:2501.12599 , year=

    work page internal anchor Pith review arXiv

  58. [58]

    Interpretable contrastive monte carlo tree search reasoning

    Interpretable contrastive monte carlo tree search reasoning , author=. arXiv preprint arXiv:2410.01707 , year=

    work page arXiv

  59. [59]

    arXiv preprint arXiv:2405.18634 , year=

    A Theoretical Understanding of Self-Correction through In-context Alignment , author=. arXiv preprint arXiv:2405.18634 , year=

    work page arXiv

  60. [60]

    InAdvances in Neural Information Processing Systems, volume 35, pages 24804–24817

    Archon: An architecture search framework for inference-time techniques , author=. arXiv preprint arXiv:2409.15254 , year=

    work page arXiv

  61. [61]

    RethinkMCTS: Refining Erroneous Thoughts in Monte Carlo Tree Search for Code Generation,

    RethinkMCTS: Refining Erroneous Thoughts in Monte Carlo Tree Search for Code Generation , author=. arXiv preprint arXiv:2409.09584 , year=

    work page arXiv

  62. [62]

    arXiv preprint arXiv:2305.14992 , year=

    Reasoning with language model is planning with world model , author=. arXiv preprint arXiv:2305.14992 , year=

    work page arXiv

  63. [63]

    nature , volume=

    Mastering the game of Go with deep neural networks and tree search , author=. nature , volume=. 2016 , publisher=

    2016

  64. [64]

    arXiv preprint arXiv:2408.00724 , year=

    Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models , author=. arXiv preprint arXiv:2408.00724 , year=

    work page arXiv

  65. [65]

    MathPrompter: Mathematical Reasoning using Large Language Models

    Imani, Shima and Du, Liang and Shrivastava, Harsh , biburl =. MathPrompter: Mathematical Reasoning using Large Language Models. , url =. ACL (industry) , editor =

  66. [66]

    Plan, Verify and Switch: Integrated Reasoning with Diverse X-of-Thoughts

    Liu, Tengxiao and Guo, Qipeng and Yang, Yuqing and Hu, Xiangkun and Zhang, Yue and Qiu, Xipeng and Zhang, Zheng , biburl =. Plan, Verify and Switch: Integrated Reasoning with Diverse X-of-Thoughts. , url =. EMNLP , editor =

  67. [67]

    Large Language Model Cascades with Mixture of Thought Representations for Cost-Efficient Reasoning

    Yue, Murong and Zhao, Jie and Zhang, Min and Du, Liang and Yao, Ziyu , biburl =. Large Language Model Cascades with Mixture of Thought Representations for Cost-Efficient Reasoning. , url =. ICLR , ee =

  68. [68]

    ArXiv , year=

    AceCoder: Acing Coder RL via Automated Test-Case Synthesis , author=. ArXiv , year=

  69. [69]

    Jon Saad-Falcon and 1 others

    Code generation with alphacodium: From prompt engineering to flow engineering , author=. arXiv preprint arXiv:2401.08500 , year=

    work page arXiv

  70. [70]

    2024 , publisher=

    Introducing swe-bench verified , author=. 2024 , publisher=

    2024

  71. [71]

    TACO: Topics in algorithmic code generation dataset.arXiv preprint arXiv:2312.14852, 2023

    Taco: Topics in algorithmic code generation dataset , author=. arXiv preprint arXiv:2312.14852 , year=

    work page arXiv

  72. [72]

    Bd , volume=

    Foundations and trends in programming languages , author=. Bd , volume=

  73. [73]

    Advances in Neural Information Processing Systems , volume=

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation , author=. Advances in Neural Information Processing Systems , volume=

  74. [74]

    Hugging Face repository , volume=

    Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions , author=. Hugging Face repository , volume=

  75. [75]

    arXiv preprint arXiv:2309.17272 , year=

    Enhancing large language models in coding through multi-perspective self-consistency , author=. arXiv preprint arXiv:2309.17272 , year=

    work page arXiv

  76. [76]

    Preference optimiza- tion for reasoning with pseudo feedback

    Preference Optimization for Reasoning with Pseudo Feedback , author=. arXiv preprint arXiv:2411.16345 , year=

    work page arXiv

  77. [77]

    CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution

    Cruxeval: A benchmark for code reasoning, understanding and execution , author=. arXiv preprint arXiv:2401.03065 , year=

    work page internal anchor Pith review arXiv

  78. [78]

    Findings of the Association for Computational Linguistics: EMNLP 2023 , editor =

    Zhao, James Xu and Xie, Yuxi and Kawaguchi, Kenji and He, Junxian and Xie, Michael Qizhe , title =. Findings of the Association for Computational Linguistics: EMNLP 2023 , editor =. 2023 , url =

    2023

  79. [79]

    SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

    Swe-bench: Can language models resolve real-world github issues? , author=. arXiv preprint arXiv:2310.06770 , year=

    work page internal anchor Pith review arXiv

  80. [80]

    2024 , journal =

    Training Software Engineering Agents and Verifiers with SWE-Gym , author =. 2024 , journal =

    2024

Showing first 80 references.