Pith. sign in

REVIEW 2 major objections 2 minor 45 references

Bayesian control for coding agents

T0 review · 2 major / 2 minor · reviewed 2026-06-25 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A Bayesian controller maintains belief over code correctness to decide dynamically when to verify or stop.

desk verdict The paper turns coding-agent tool orchestration into cost-sensitive Bayesian sequential testing with a maintained belief state, but the independence and binary-correctness assumptions are the load-bearing part that needs checking. read the letter →

arxiv 2606.24453 v1 pith:T2KXPEYM submitted 2026-06-23 cs.AI cs.CL

Theodore Papamarkou , Vladislav Smirnov , Viktor Mazanov , Artem Vazhentsev , Preslav Nakov , Timothy Baldwin , Artem Shelmanov This is my paper · ORCID
classification cs.AIcs.CL
keywords BayesiancontrolcodingagentssequentialhypothesistestinguncertaintyquantificationLLMorchestrationcost-sensitivedecisionsagenttooluse
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper formulates orchestration of coding agents as cost-sensitive sequential hypothesis testing. A Bayesian controller updates its belief about whether a generated candidate is correct using signals from cheap diagnostics and costly verifiers. It then chooses among actions such as gathering more evidence, refining the candidate, running verification, or stopping. This policy is shown to reduce overall cost most effectively when verification is expensive and the available critics supply useful but imperfect information. The same belief state also supplies an interpretable probability of correctness that outperforms token-level probabilities and raw tool-success rates as an uncertainty measure.

What carries the argument

Bayesian controller that maintains and updates a belief distribution over binary correctness using conditionally independent signals from diagnostics and verifiers to minimize expected total cost.

What would settle it

A new benchmark or setting in which the Bayesian policy incurs higher total verification cost than a fixed-rule baseline while achieving equal or lower final accuracy.

Watch

Extended reading notes

Core claim

A Bayesian controller maintains a belief over the binary hypothesis of candidate correctness and uses cost-sensitive sequential testing to decide dynamically whether to gather evidence, refine, verify, or stop, yielding better performance on six generators and nine benchmarks when verification costs are high and critics are informative but imperfect.

Load-bearing premise

The diagnostics and verifiers supply signals whose informativeness can be captured by a simple Bayesian update over a binary correctness hypothesis.

Editorial extensions

If this is right

  • Agents incur lower total verification cost while preserving solution accuracy across multiple generators and benchmarks.
  • The maintained belief serves as a calibrated correctness score superior to token-probability and raw success baselines.
  • Gains appear largest precisely when verification is costly and individual critics are informative but imperfect.
  • Orchestration shifts from fixed rules to sequential, cost-aware decisions that stop early when belief is sufficiently high or low.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same belief-maintenance structure could guide tool-use decisions in non-coding agent domains where actions carry different costs.
  • If signal dependence is stronger than assumed, replacing the simple update with a joint model might further reduce cost.
  • The correctness probability could be exposed to users or downstream systems as an explicit uncertainty flag.
  • Extending the state to track multiple candidate solutions at once might allow parallel refinement under a shared budget.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper formulates orchestration of LLM coding agents as cost-sensitive sequential hypothesis testing. A Bayesian controller maintains a posterior belief over a binary 'correct' hypothesis for candidate solutions and uses it to decide dynamically whether to collect more diagnostic evidence, refine the candidate, invoke an expensive verifier, or stop. Experiments across six generators and nine coding benchmarks show the controller is most valuable when verification is costly and critics are informative but imperfect; the resulting belief state also yields an interpretable correctness score that outperforms token-probability and raw tool-success baselines for uncertainty quantification.

Significance. If the modeling assumptions and empirical claims hold after validation, the work supplies a principled, uncertainty-aware alternative to fixed-rule orchestrators for tool-using coding agents. The explicit separation of control from generation and the use of the belief state for both stopping and UQ are potentially reusable contributions beyond the specific benchmarks.

major comments (2)
  1. [§3.2] §3.2 (Bayesian update): The controller relies on a simple sequential update that treats diagnostic and verifier signals as conditionally independent given the binary correctness hypothesis. No diagnostic checks, sensitivity analysis, or empirical tests for dependence or for the adequacy of the binary state are reported; if signals share latent failure modes or if partial correctness matters, both the cost-sensitive policy and the claimed UQ superiority would be miscalibrated.
  2. [§5] §5 (Experiments): The abstract states that Bayesian control 'proves to be most valuable' under specific conditions, yet the manuscript provides no pre-registered analysis plan, multiple-testing correction, or ablation isolating the contribution of the independence assumption versus other design choices. Without these, it is unclear whether the reported gains survive controls for benchmark selection and hyper-parameter tuning.
minor comments (2)
  1. Notation for the belief state and likelihood functions is introduced without a compact reference table; a single summary table would improve readability.
  2. Figure captions for the cost-sensitivity plots do not state the exact cost ratios used, making it hard to reproduce the 'most valuable when verification is costly' claim.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive comments. We address each major point below and indicate the revisions we will make to strengthen the manuscript.

read point-by-point responses

  1. Referee: [§3.2] §3.2 (Bayesian update): The controller relies on a simple sequential update that treats diagnostic and verifier signals as conditionally independent given the binary correctness hypothesis. No diagnostic checks, sensitivity analysis, or empirical tests for dependence or for the adequacy of the binary state are reported; if signals share latent failure modes or if partial correctness matters, both the cost-sensitive policy and the claimed UQ superiority would be miscalibrated.

    Authors: We agree that the conditional independence assumption merits explicit validation. In the revised version we will add a dedicated sensitivity analysis subsection that (i) derives the effect of pairwise signal dependence on the posterior trajectory under a simple correlation model and (ii) reports empirical posterior calibration on two benchmarks where both diagnostic and verifier outcomes are available for the same candidates. We will also clarify that the binary hypothesis is an operational abstraction chosen because the controller’s cost-sensitive stopping rule is defined with respect to the probability of a fully correct solution; partial correctness is already handled upstream by the generators and is not claimed to be modeled by the belief state itself. revision: partial

  2. Referee: [§5] §5 (Experiments): The abstract states that Bayesian control 'proves to be most valuable' under specific conditions, yet the manuscript provides no pre-registered analysis plan, multiple-testing correction, or ablation isolating the contribution of the independence assumption versus other design choices. Without these, it is unclear whether the reported gains survive controls for benchmark selection and hyper-parameter tuning.

    Authors: The experiments were exploratory and the primary claims rest on consistent qualitative patterns across six generators and nine benchmarks rather than on formal hypothesis tests. We acknowledge that a pre-registered analysis plan was not used. In revision we will (i) add an explicit ablation that replaces the independence assumption with a simple joint likelihood model on a subset of tasks and (ii) include a supplementary table that recomputes all headline metrics after re-tuning the controller hyper-parameters on a held-out benchmark split. We will also add a limitations paragraph discussing the absence of pre-registration and multiple-testing correction. revision: partial

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: standard Bayesian update applied to tool orchestration without self-referential fitting or load-bearing self-citations.

full rationale

The abstract formulates orchestration as cost-sensitive sequential hypothesis testing with a Bayesian belief over binary correctness, updated from diagnostics and verifiers. No equations appear that define a parameter from data and then rename its output as a prediction. No self-citation chains, uniqueness theorems, or ansatzes are invoked. The claimed superiority is presented as an empirical result across generators and benchmarks rather than a tautological consequence of the modeling assumptions. The reader's assessment of score 2 aligns with the absence of any load-bearing reduction; the modeling choice (conditional independence, binary state) is an explicit assumption open to falsification, not a hidden circularity.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

Abstract-only; no free parameters, axioms, or invented entities are mentioned. The Bayesian update itself is treated as a standard tool rather than derived in the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Bayesian control for coding agents." pith.science (2026). https://pith.science/paper/T2KXPEYM

@misc{pith2026260624453,
  author       = {Pith},
  title        = {Pith review of: Bayesian control for coding agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T2KXPEYM}},
  note         = {Machine review of arXiv:2606.24453}
}
read the original abstract

Modern coding agents pair LLM generators with various tools, including cheap diagnostics and expensive verifiers. The tool-use decisions are typically governed by orchestrators that often use fixed rules and ignore uncertainty. We formulate orchestration as cost-sensitive sequential hypothesis testing: a Bayesian controller maintains a belief over candidate correctness and dynamically decides whether to gather more evidence, refine the candidate, verify it, or stop. Across six generators and nine coding benchmarks, Bayesian control proves to be most valuable when verification is costly and critics are informative but imperfect. Beyond control, the belief state yields an interpretable correctness score that outperforms token-probability and raw tool-success baselines for uncertainty quantification.

Figures

Figures reproduced from arXiv: 2606.24453 by the authors.

Figure 1
Figure 1. Illustration of the Bayesian control loop. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Per-policy ∆ utility against always_verify on SWE-Bench Lite / claude-haiku-4.5. Gold stars mark per-cell winners; numbers above bars are abso￾lute mean utility; error bars are paired-bootstrap 95% CIs (B=1000). Discussion in §6.2; per-cell evaluation￾set bars across all (benchmark, generator) pairs in Appendix D.1; full train-vs-evaluation panel in Ap￾pendix D.2. saving that does not justify the loss. 6.2 Policy Co… view at source ↗
Figure 2
Figure 2. Empirical decision regions over the (P(Y =1), Cver/R) plane: k-NN (k=300) over 7,020 sweep points. Colors: A (green) Bayesian; B (blue) gate(Crtest); C (yellow) always_verify; other (coral). See §6.1 for the full regime description. cost, the optimal Bayes decision often is a gate on the public-test critic. On cells where Crtest is highly informative (γtest ≳ 0.7 on LCB and on most Hu￾manEvalFix and CodeContests cel… view at source ↗
Figures from the paper (30 more)
Figure 4
Figure 4. Figure 4: sweeps Cver ∈ [1, 100] on MBPP+ / claude-haiku-4.5 at fixed R=100, with the other costs held at their default values. The sweep traverses all three regimes within a single cell, illustrating the C→ B →A handoff: at low Cver, always_verify wins because the prior is high…
Figure 5
Figure 5. Figure 5: Critic informativeness γi = Pi(z=pass | Y =1) − Pi(z=pass | Y =0) vs prior P(Y =1) across the available benchmark–generator pairs, one subplot per critic layer. Point color indicates the Bayesian ∆ for each benchmark–generator pair. SWE-Bench cells do not appear in the…
Figure 6
Figure 6. Figure 6: Per-benchmark location of each generator in ( [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Held-out eval policy comparison on LCB-hard, all generators on one panel. Bayesian variants dominate [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]
Figure 8
Figure 8. Figure 8: Held-out eval policy comparison on LCB-medium, all generators on one panel. [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]
Figure 9
Figure 9. Figure 9: Held-out eval policy comparison on LCB-easy, all generators on one panel. [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Held-out eval policy comparison on MBPP+, all generators on one panel. [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: Held-out eval policy comparison on HumanEval+, all generators on one panel. [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: Held-out eval policy comparison on HumanEvalFix, all generators on one panel. [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]
Figure 13
Figure 13. Figure 13: Held-out eval policy comparison on CodeContests, all generators on one panel. [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]
Figure 14
Figure 14. Figure 14: Held-out eval policy comparison on SWE-Bench Lite, all generators on one panel (the [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]
Figure 15
Figure 15. Figure 15: Held-out eval policy comparison on SWE-Bench Verified, all generators on one panel. [PITH_FULL_IMAGE:figures/full_fig_p022_15.png]
Figure 16
Figure 16. Figure 16: Train-vs-eval policy comparison on LCB-hard. Left column: 75% train (in-sample fit). Right column: [PITH_FULL_IMAGE:figures/full_fig_p024_16.png]
Figure 17
Figure 17. Figure 17: Train-vs-eval policy comparison on LCB-medium. [PITH_FULL_IMAGE:figures/full_fig_p025_17.png]
Figure 18
Figure 18. Figure 18: Train-vs-eval policy comparison on LCB-easy. [PITH_FULL_IMAGE:figures/full_fig_p026_18.png]
Figure 19
Figure 19. Figure 19: Train-vs-eval policy comparison on MBPP+. [PITH_FULL_IMAGE:figures/full_fig_p027_19.png]
Figure 20
Figure 20. Figure 20: Train-vs-eval policy comparison on HumanEval+. [PITH_FULL_IMAGE:figures/full_fig_p028_20.png]
Figure 21
Figure 21. Figure 21: Train-vs-eval policy comparison on HumanEvalFix. [PITH_FULL_IMAGE:figures/full_fig_p029_21.png]
Figure 22
Figure 22. Figure 22: Train-vs-eval policy comparison on CodeContests. [PITH_FULL_IMAGE:figures/full_fig_p030_22.png]
Figure 23
Figure 23. Figure 23: Train-vs-eval policy comparison on SWE-Bench Lite. The [PITH_FULL_IMAGE:figures/full_fig_p031_23.png]
Figure 24
Figure 24. Figure 24: Train-vs-eval policy comparison on SWE-Bench Verified. [PITH_FULL_IMAGE:figures/full_fig_p032_24.png]
Figure 25
Figure 25. Figure 25: Cver sweep on MBPP+, one sub-panel per generator (companion to [PITH_FULL_IMAGE:figures/full_fig_p034_25.png]
Figure 26
Figure 26. Figure 26: Cver sweep on LCB-hard, one sub-panel per generator. LCB-hard’s low prior (P(Y =1) ≈ 0.17) places the analytic crossover at C ∗ ver ≈ 17, so the cell sits in regime A for nearly the entire sweep – the Bayesian variants lead above Cver ≈ 20. The two Bayesian curves ove…
Figure 27
Figure 27. Figure 27: Cver sweep on LCB-medium, one sub-panel per generator [PITH_FULL_IMAGE:figures/full_fig_p035_27.png]
Figure 28
Figure 28. Figure 28: Cver sweep on LCB-easy, one sub-panel per generator. 35 [PITH_FULL_IMAGE:figures/full_fig_p035_28.png]
Figure 29
Figure 29. Figure 29: Cver sweep on HumanEval+, one sub-panel per generator [PITH_FULL_IMAGE:figures/full_fig_p036_29.png]
Figure 30
Figure 30. Figure 30: Cver sweep on HumanEvalFix, one sub-panel per generator. 36 [PITH_FULL_IMAGE:figures/full_fig_p036_30.png]
Figure 31
Figure 31. Figure 31: Cver sweep on CodeContests, one sub-panel per generator [PITH_FULL_IMAGE:figures/full_fig_p037_31.png]
Figure 32
Figure 32. Figure 32: Cver sweep on SWE-Bench Lite, one sub-panel per generator. 37 [PITH_FULL_IMAGE:figures/full_fig_p037_32.png]
Figure 33
Figure 33. Figure 33: Cver sweep on SWE-Bench Verified, one sub-panel per generator. 38 [PITH_FULL_IMAGE:figures/full_fig_p038_33.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 2 canonical work pages

  1. [1]

    Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2023. CodeT : C ode generation with generated tests. In International Conference on Learning Representations (ICLR)

    2023

  2. [2]

    Deepro Choudhury, Sinead Williamson, Adam Goli \'n ski, Ning Miao, Freddie Bickford Smith, Michael Kirchhof, Yizhe Zhang, and Tom Rainforth. 2026. BED-LLM : I ntelligent information gathering with LLM s and B ayesian experimental design. In International Conference on Learning Representations (ICLR)

    2026

  3. [3]

    Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Fr \'e d \'e ric Blain, Francisco Guzm \'a n, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. 2020. Unsupervised quality estimation for neural machine translation. Transactions of the Association for Computational Linguistics, 8:539--555

    2020

  4. [4]

    Ronald A. Howard. 1966. Information value theory. IEEE Transactions on Systems Science and Cybernetics, 2(1):22--26

    1966

  5. [6]

    Lahiri, Madanlal Musuvathi, and Jianfeng Gao

    Jeevana Priya Inala, Chenglong Wang, Mei Yang, Andres Codas, Mark Encarnaci \'o n, Shuvendu K. Lahiri, Madanlal Musuvathi, and Jianfeng Gao. 2022. Fault-aware neural code rankers. In Advances in Neural Information Processing Systems (NeurIPS)

    2022

  6. [7]

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2025. Live C ode B ench: H olistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations (ICLR)

    2025

  7. [8]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. 2024. SWE-bench : C an language models resolve real-world Github issues? In International Conference on Learning Representations (ICLR)

    2024

  8. [9]

    Littman, and Anthony R

    Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. 1998. Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101(1--2):99--134

    1998

Show all 45 references
  1. [10]

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, R\' e mi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal,...

    2022

  2. [11]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation. In International Conference on Neural Information Processing Systems (NeurIPS)

    2023

  3. [12]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self- R efine: ...

    2023

  4. [13]

    Andrey Malinin and Mark Gales. 2021. Uncertainty estimation in autoregressive structured prediction. In International Conference on Learning Representations (ICLR)

    2021

  5. [14]

    Niklas Muennighoff, Qian Liu, Armel Randy Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre. 2024. Octo P ack: I nstruction tuning code large language models. In International Conference on Learning Representat...

    2024

  6. [15]

    Theodore Papamarkou, Pierre Alquier, Matthias Bauer, Wray Buntine, Andrew Davison, Gintare Karolina Dziugaite, Maurizio Filippone, Andrew Y. K. Foong, Vincent Fortuin, Dimitris Fouskakis, Jes Frellsen, Eyke Hüllermeier, Theofanis Karaletsos, Mohammad Emtiyaz Khan, Nikita Kotel...

    2026

  7. [16]

    Narasimhan, and Shunyu Yao

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R. Narasimhan, and Shunyu Yao. 2023. Reflexion: L anguage agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS)

    2023

  8. [18]

    Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev, Lyudmila Rvanova, Daniil Vasilev, Akim Tsvigun, Sergey Petrakov, Rui Xing, Abdelrahman Sadallah, Kirill Grishchenkov, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov, and Artem Shelmanov. 2025. Benchmarking ...

    2025

  9. [19]

    Abraham Wald. 1947. Sequential analysis. John Wiley & Sons

    1947

  10. [20]

    Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, and 5 others. 2025. Open H...

    2025

  11. [21]

    Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R

    John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R. Narasimhan, and Ofir Press. 2024. SWE-agent : A gent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems (NeurIPS)

    2024

  12. [22]

    Hashimoto, Mike Lewis, Wen-Tau Yih, Daniel Fried, and Sida I

    Tianyi Zhang, Tao Yu, Tatsunori B. Hashimoto, Mike Lewis, Wen-Tau Yih, Daniel Fried, and Sida I. Wang. 2023. Coder reviewer reranking for code generation. In International Conference on Machine Learning (ICML)

    2023

  13. [23]

    Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. 2024. Language agent tree search unifies reasoning, acting, and planning in language models. In International Conference on Learning Representations (ICLR)

    2024

  14. [24]

    2024 , publisher =

    Du, Xueying and Liu, Mingwei and Wang, Kaixin and Wang, Hanlin and Liu, Junwei and Chen, Yixuan and Feng, Jiayi and Sha, Chaofeng and Peng, Xin and Lou, Yiling , title =. 2024 , publisher =

    2024

  15. [25]

    and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik R

    Jimenez, Carlos E. and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik R. , title =. International Conference on Learning Representations (ICLR) , year =

  16. [26]

    2023 , booktitle =

    Liu, Jiawei and Xia, Chunqiu Steven and Wang, Yuyao and Zhang, Lingming , title =. 2023 , booktitle =

    2023

  17. [27]

    Niklas Muennighoff and Qian Liu and Armel Randy Zebaze and Qinkai Zheng and Binyuan Hui and Terry Yue Zhuo and Swayam Singh and Xiangru Tang and Leandro Von Werra and Shayne Longpre , booktitle=. Octo

  18. [28]

    Competition-level code generation with

    Li, Yujia and Choi, David and Chung, Junyoung and Kushman, Nate and Schrittwieser, Julian and Leblond, R\'. Competition-level code generation with. Science , publisher =. 2022 , pages =

    2022

  19. [29]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Madaan, Aman and Tandon, Niket and Gupta, Prakhar and Hallinan, Skyler and Gao, Luyu and Wiegreffe, Sarah and Alon, Uri and Dziri, Nouha and Prabhumoye, Shrimai and Yang, Yiming and Gupta, Shashank and Majumder, Bodhisattwa Prasad and Hermann, Katherine and Welleck, Sean and Y...

  20. [30]

    and Yao, Shunyu , title =

    Shinn, Noah and Cassano, Federico and Gopinath, Ashwin and Narasimhan, Karthik R. and Yao, Shunyu , title =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  21. [31]

    Wald, Abraham , title =

  22. [32]

    , title =

    Howard, Ronald A. , title =. IEEE Transactions on Systems Science and Cybernetics , volume =

  23. [33]

    Naman Jain and King Han and Alex Gu and Wen-Ding Li and Fanjia Yan and Tianjun Zhang and Sida Wang and Armando Solar-Lezama and Koushik Sen and Ion Stoica , booktitle=. Live

  24. [34]

    Papamarkou, Theodore and Alquier, Pierre and Bauer, Matthias and Buntine, Wray and Davison, Andrew and Dziugaite, Gintare Karolina and Filippone, Maurizio and Foong, Andrew Y. K. and Fortuin, Vincent and Fouskakis, Dimitris and Frellsen, Jes and Hüllermeier, Eyke and Karaletso...

  25. [35]

    and Cassandra, Anthony R

    Kaelbling, Leslie Pack and Littman, Michael L. and Cassandra, Anthony R. , title =. Artificial Intelligence , volume =

  26. [36]

    International Conference on Learning Representations (ICLR) , year =

    Choudhury, Deepro and Williamson, Sinead and Goli. International Conference on Learning Representations (ICLR) , year =

  27. [37]

    and Manocha, Dinesh , title =

    Suri, Manan and Mathur, Puneet and Lipka, Nedim and Dernoncourt, Franck and Rossi, Ryan A. and Manocha, Dinesh , title =. arXiv preprint arXiv:2511.08798 , year =

    arXiv

  28. [38]

    International Conference on Learning Representations (ICLR) , year =

    Chen, Bei and Zhang, Fengji and Nguyen, Anh and Zan, Daoguang and Lin, Zeqi and Lou, Jian-Guang and Chen, Weizhu , title =. International Conference on Learning Representations (ICLR) , year =

  29. [39]

    Fault-aware neural code rankers , booktitle =

    Inala, Jeevana Priya and Wang, Chenglong and Yang, Mei and Codas, Andres and Encarnaci. Fault-aware neural code rankers , booktitle =

  30. [40]

    and Lewis, Mike and Yih, Wen-Tau and Fried, Daniel and Wang, Sida I

    Zhang, Tianyi and Yu, Tao and Hashimoto, Tatsunori B. and Lewis, Mike and Yih, Wen-Tau and Fried, Daniel and Wang, Sida I. , title =. International Conference on Machine Learning (ICML) , year =

  31. [41]

    and Luck, Michael and Bu, Qingwen and Qing, Yuhao and Cui, Heming , title =

    Huang, Dong and Zhang, Jie M. and Luck, Michael and Bu, Qingwen and Qing, Yuhao and Cui, Heming , title =. arXiv preprint arXiv:2312.13010 , year =

    arXiv

  32. [42]

    International Conference on Learning Representations (ICLR) , year =

    Zhou, Andy and Yan, Kai and Shlapentokh-Rothman, Michal and Wang, Haohan and Wang, Yu-Xiong , title =. International Conference on Learning Representations (ICLR) , year =

  33. [43]

    and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik R

    Yang, John and Jimenez, Carlos E. and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik R. and Press, Ofir , title =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  34. [44]

    Xu and Xiangru Tang and Mingchen Zhuge and Jiayi Pan and Yueqi Song and Bowen Li and Jaskirat Singh and Hoang H

    Xingyao Wang and Boxuan Li and Yufan Song and Frank F. Xu and Xiangru Tang and Mingchen Zhuge and Jiayi Pan and Yueqi Song and Bowen Li and Jaskirat Singh and Hoang H. Tran and Fuqiang Li and Ren Ma and Mingzhang Zheng and Bill Qian and Yanjun Shao and Niklas Muennighoff and Y...

  35. [45]

    Benchmarking uncertainty quantification methods for large language models with

    Vashurin, Roman and Fadeeva, Ekaterina and Vazhentsev, Artem and Rvanova, Lyudmila and Vasilev, Daniil and Tsvigun, Akim and Petrakov, Sergey and Xing, Rui and Sadallah, Abdelrahman and Grishchenkov, Kirill and Panchenko, Alexander and Baldwin, Timothy and Nakov, Preslav and P...

    2025

  36. [46]

    International Conference on Learning Representations (ICLR) , year =

    Andrey Malinin and Mark Gales , title =. International Conference on Learning Representations (ICLR) , year =

  37. [47]

    Transactions of the Association for Computational Linguistics , volume =

    Unsupervised quality estimation for neural machine translation , author =. Transactions of the Association for Computational Linguistics , volume =. 2020 , publisher =

    2020

Pith tools

Reviewed June 25, 2026 · model on record in the stance chip above.