Pith. sign in

REVIEW 3 major objections 5 minor 69 references

LangChoiceBench: Measuring and Explaining Programming-Language Choice in LLMs

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Across 25 LLMs, Python accounts for 35.3% of implementations on tasks where it is typically a poor default, even though models recommend it in only 10.7% of cases.

desk verdict A solid, well-released benchmark; the headline over-selection claim is partly protected by the recommendation-implementation gap, but the paper would be stronger with error bars and an external audit of task suitability. read the letter →

arxiv 2608.06041 v1 pith:2MIYVJPK submitted 2026-08-06 cs.SE cs.CL

Lukas Twist , Twm Stone , Helen Yannakoudakis , Jie M. Zhang This is my paper · ORCID
classification cs.SEcs.CL
keywords programming-languagepreferencePythondefaultbiasLLMcodegenerationproject-levelbenchmarkrecommendation-implementationconsistencylanguagediversityreasoningtracesphantomevidence
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LangChoiceBench is a project-level code-generation benchmark: 28 language-neutral tasks across seven software areas where Python is often a poor default, each paired with implementation and recommendation prompts. The paper evaluates 25 LLMs on it and claims that Python remains heavily over-selected, appearing in 35.3% of implementations but only 10.7% of top-three recommendations, while fewer than half of implementations use a language the model itself recommended in its top three. An analysis of 9,826 reasoning traces finds that 69.8% of Python choices are automatic, 20.5% are driven by ease, 7.8% fabricate contextual support (phantom evidence), and 1.9% contradict the model's own stated language choice. The authors argue that language choice in agentic code generation is therefore not just a preference problem but also a deliberation, grounding, and consistency problem. If true, this gives benchmark builders and model developers a concrete target: force the language decision to be explicit and check it against the project requirements.

What carries the argument

The load-bearing machinery is the benchmark's paired prompt design and metric suite. Each of 28 tasks appears in three implementation prompts and three recommendation prompts, so the same project description elicits implicit language choice (code) and explicit language choice (recommendation). Five metrics -- PyIR, PyRR, RIR, RIA, and ELD -- turn those responses into comparable preference, consistency, and diversity scores, with ELD using exponential entropy (the categorical Vendi Score) and RIA using Spearman rank correlation. The reasoning-trace analysis adds a five-way taxonomy (phantom evidence, language mismatch, Python for ease, automatic Python, unclear/other) that is applied by a validated LLM judge to 9,826 traces, so the benchmark explains not only that models choose Python but the justification they present for doing so.

What would settle it

Re-run the task-suitability annotation with a larger panel of domain practitioners using the same strong/acceptable/weak scale; if a majority of the tasks labelled weak by the authors receive acceptable or strong, the headline over-selection claim loses its normative base, and the 35.3% implementation rate would need reinterpretation as a generic default rather than an error.

Watch

Extended reading notes

Core claim

The central claim, stated on the paper's terms, is that modern LLMs systematically over-select Python when asked to implement new projects whose requirements make Python a weak fit, and do so even when their own recommendations point elsewhere. On LangChoiceBench, average PyIR is 35.3% versus 10.7% PyRR; average RIR is 48.8%, mean RIA is 0.17, and average ELD is 3.05, with smaller open-weight models leaning hardest on Python. Reasoning-trace classification shows 69.8% automatic, 20.5% ease, 7.8% phantom evidence, and 1.9% language mismatch. The paper treats this as evidence that the chain of thought starts too late: models reason about implementation details before deciding whether the default stack is appropriate.

Load-bearing premise

The benchmark's normative premise is that Python is a poor default for all 28 tasks; that premise rests on two authors' ratings with raw agreement 75.0% and quadratic weighted kappa 0.56, so if practitioners would rate many tasks as acceptable for Python, the over-selection interpretation weakens.

Editorial extensions

If this is right

  • If LangChoiceBench becomes a standard evaluation, model releases can be checked for Python over-selection and recommendation-implementation divergence before deployment in agentic coding tools.
  • Because most Python choices are automatic, prompting or fine-tuning that forces an explicit language-comparison step before code generation could measurably reduce PyIR without changing task difficulty.
  • Because code-specialised models also show strong Python preference, code specialisation alone is not a remedy; training objectives need to incorporate project-level suitability.
  • The phantom-evidence and language-mismatch categories give concrete defects to target with grounding checks and reasoning-output consistency verification.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An extension left implicit by the paper: apply the same paired-prompt protocol to other dominant defaults, such as JavaScript or TypeScript for frontend tasks, to test whether the bias is specifically Python or a broader most-frequent-language-in-pretraining effect.
  • The between-model variation, with ELD ranging from about 1.7 to 3.94 and phantom-evidence rates from 0% to 16.4%, suggests the benchmark could double as a diagnostic for training-data composition, a link the paper does not pursue.
  • If the phantom-evidence cases are genuine, they imply models can produce confident false claims about their own context; varying the system prompt experimentally could test whether this fabrication is context-sensitive.
  • A reader should weigh the benchmark's normative premise carefully: it rests on two authors' ratings with quadratic weighted kappa 0.56, so a larger expert panel would harden the over-selection interpretation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces LangChoiceBench, a project-level code-generation benchmark with 28 tasks across seven software areas where Python is claimed to be a poor default, together with three implementation-prompt and three recommendation-prompt variants. The evaluation protocol is transparent: 20 implementation samples and 5 recommendation samples per prompt variant, yielding 2,100 responses per model, with extraction validated on 100 samples and an LLM judge for reasoning traces validated at 96% accuracy on a held-out set. Across 25 LLMs, the authors report that Python is used in 35.3% of implementations but appears in only 10.7% of top-three recommendations, that recommendation-implementation consistency is low (mean RIR 48.8%, mean RIA 0.17), and that smaller open-weight models show stronger Python preference and lower language diversity. An analysis of 9,826 reasoning traces classifies 69.8% of Python choices as automatic, 20.5% as driven by ease, 7.8% as phantom evidence, and 1.9% as language mismatch. The paper argues that LLMs remain unreliable decision-makers for project-level programming-language choice.

Significance. If the results hold, the benchmark and trace taxonomy are useful contributions: they provide a reproducible measurement protocol for a known but under-measured behavior, cover 25 models across 10 families with both reasoning and non-reasoning variants, and include a transparent, validated extraction and classification pipeline. The release of data, prompts, and tools is a concrete strength, as is the effort to audit the LLM judge against held-out human labels. The main significance risk is that the headline 'Python is heavily over-selected' rests on the premise that the 28 tasks are settings where Python is a poor default; that premise is currently supported only by two authors' ratings with moderate agreement. The reasoning-trace findings are interesting but their interpretation depends on the breadth of the 'automatic Python' category. Overall, the empirical protocol is solid and the paper is likely to be a useful reference for future evaluations, provided the normative ground truth is strengthened and the interpretation of the trace categories is calibrated.

major comments (3)
  1. [§3.2 / Appendix A.3, Table 4] The central claim that Python is 'heavily over-selected' (Abstract; §7) presupposes that the 28 tasks are settings where Python is a poor default. This premise is validated only by two authors' independent ratings, with raw agreement 75.0%, unweighted Cohen's kappa 0.40, and quadratic weighted kappa 0.56; moreover, one rater never uses the 'strong' label, and five of the seven disagreements are between 'weak' and 'acceptable'. Because the 'acceptable' category explicitly permits Python 'under some consideration, such as understandability or rapid prototyping', and the implementation prompts ask for 'initial', 'minimal working', or 'minimal' versions, these weak/acceptable disagreements are precisely the cases where a Python prototype may be a legitimate answer. The paper should report the per-task ratings, perform a sensitivity analysis of PyIR, PyRR, and RIR under a conservative relabeling (for example, treating all weak/acceptable disagreements as acceptable), or add more raters. Without this, the headline over-selection interpretation is exactly as strong as a moderate-agreement two-person judgment.
  2. [§5.1 / Appendix C.4, Table 7] The 69.8% 'automatic Python' figure drives the conclusion that models 'do not meaningfully consider' language choice, but the operational definition of this category is broad: it includes traces that never mention a language, traces that assume Python immediately, traces that praise a Python library without reflecting on Python itself, and traces that justify Python only by calling it common or popular. These sub-behaviors have different implications, and the judge prompt in Appendix C.4 explicitly instructs that several of them count as automatic. Please report the distribution of these automatic sub-cases (for example, from the judge's evidence quotes or a targeted re-annotation) or soften the headline interpretation to 'no explicit cross-language comparison in the trace'. As written, the 'automatic' label conflates distinct reasoning patterns and overstates the strength of the finding.
  3. [§3.4 / Table 3] All five metrics are reported as point estimates without confidence intervals or variance information, even though they are computed from finite samples (20 implementation samples and 5 recommendation samples per prompt variant). The headline comparison PyIR 35.3% vs. PyRR 10.7% is large and unlikely to be sampling noise, but per-model comparisons such as RIA values near zero or differences among mid-ranked models may be within sampling variability. Please report standard errors or confidence intervals for the main metrics, or at least state the sampling uncertainty explicitly for the reader.
minor comments (5)
  1. [§3.4] The top-three cutoff for PyRR and RIR is a reasonable design choice, but the paper does not report any sensitivity analysis; a sentence noting whether the conclusions hold under top-one or top-five cutoffs would strengthen the metrics section.
  2. [§4.1 / Appendix B] Generation settings differ across models (for example, temperature ranges from 0.3 to 1.0 and top_p from 0.95 to 1.0), so cross-model comparisons conflate model identity with decoding configuration. The appendix documents this, but the main text should state it as a limitation of the cross-model comparisons.
  3. [Appendix A.3] The specific agreement of 87.8% on the 'weak' label is the strongest evidence for the suitability premise, but the fact that one rater never assigns 'strong' means the scale is effectively binary for that rater; this should be noted when interpreting the kappa values.
  4. [§5.2] The phrase 'nearly one in ten traces contains fabricated evidence or a direct mismatch' is slightly misleading because the two categories are mutually exclusive in the hierarchical labeling; the underlying numbers (7.8% + 1.9%) are correct, but the phrasing could be read as implying overlap.
  5. [Throughout] There are several formatting and typographical issues, including missing spaces in 'LANGCHOICEBENCHcovers' and inconsistent use of 'Python' vs. 'Python.' in table captions; a careful copyedit would improve readability.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the benchmark metrics are direct measurements, the two-author suitability ratings are an explicit input assumption rather than a fitted quantity, and the reasoning-trace judge is validated against held-out human labels.

full rationale

The paper's derivation chain does not reduce to its inputs. The benchmark is constructed by selecting seven software areas where Python is often a poor default; this is a stated design premise, supported by evidence-based reference languages and independent two-author suitability ratings (75.0% raw agreement, quadratic weighted Cohen's kappa 0.56, 87.8% specific agreement on the weak label). This premise is an input assumption, not a quantity fitted from the model outputs, and the paper explicitly acknowledges in its Limitations that suitability depends on context. The headline metrics PyIR, PyRR, RIR, RIA, and ELD are direct counts, rank correlations, and exponential entropy computed from extracted model responses; no parameter is fitted to a subset of data and then used to predict a closely related quantity. The reasoning-trace taxonomy defines categories such as 'automatic Python' independently of the headline statistics, and the LLM judge was selected on a held-out development set and evaluated on a held-out test set (96.0% accuracy, Cohen's kappa 0.93), so the 69.8% automatic-Python finding is a measurement, not a construction. Self-citations (primarily Twist et al. 2026a) motivate the benchmark but are not load-bearing: the empirical results come from 25 models run on the new benchmark, and the prior work is used only to frame the problem. The normative claim that Python is over-selected is conditional on the benchmark's task selection; whether that premise is correct is a validity concern, not a circularity concern. Overall, the paper is self-contained in its measurement pipeline, with only a minor motivational self-citation that does not affect the derivation chain.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

The central claims are empirical measurements, not derivations, so the only free parameters are analytic thresholds and protocol counts. The load-bearing assumptions are task suitability, prompt language-neutrality, judge validity, and trace representativeness.

free parameters (2)
  • top-three recommendation cutoff = 3
    PyRR and RIR count a language only if it appears in the first three recommended languages. Hand-chosen; top-1 or top-5 would shift the reported consistency rates.
  • sample counts per prompt variant = 20 implementation / 5 recommendation
    The default protocol draws 20 implementation and 5 recommendation responses per prompt variant. These counts set the precision of the rate metrics, and no confidence intervals are reported.
assumptions (4)
  • domain assumption Python is often a poor default in the seven chosen software areas.
    The over-selection interpretation depends on this. Grounded in citations and two-author ratings with 87.8% specific agreement on weak but quadratic weighted kappa 0.56. Sections 3.2, Table 1, Appendix A.3.
  • domain assumption Prompts are language-neutral and do not leak language names.
    A build check verifies absence of language leakage; if prompts leaked names, measured choices would reflect instruction following rather than preference. Appendix A.2.
  • domain assumption The LLM judge (GPT-5.4 Mini) classifies reasoning traces accurately beyond the validation set.
    All 9,826 traces are labeled by the judge after 96.0% accuracy and kappa 0.93 on a 150-trace held-out set. Sections 5.1 and Appendix C.4.
  • domain assumption Reasoning traces provide usable evidence about the model's language decision.
    The paper itself notes traces may omit latent influences (Section 2.3 and Limitations), so this is a real assumption even though acknowledged.
invented entities (1)
  • phantom evidence independent evidence
    purpose: A named failure mode where a model fabricates instructions, examples, or requirements to justify choosing Python.
    It is falsifiable in any single-turn, language-neutral prompt: if the trace cites prior context that was never provided, it qualifies. The paper provides verbatim examples and per-model prevalence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LangChoiceBench: Measuring and Explaining Programming-Language Choice in LLMs." pith.science (2026). https://pith.science/paper/2MIYVJPK

@misc{pith2026260806041,
  author       = {Pith},
  title        = {Pith review of: LangChoiceBench: Measuring and Explaining Programming-Language Choice in LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2MIYVJPK}},
  note         = {Machine review of arXiv:2608.06041}
}
read the original abstract

Large language models (LLMs) have been shown to exhibit strong Python preferences when generating project-level code, but there is currently no systematic way to measure this behaviour across new models. To bridge this gap, we introduce LangChoiceBench, a project-level code-generation benchmark for measuring Python preference, recommendation-implementation consistency, and language diversity. LangChoiceBench covers 28 projects across seven software areas where Python is often a poor default. We evaluate 25 diverse LLMs and find that Python remains heavily over-selected, recommendation-implementation consistency is low, and smaller open-weight models generally show stronger Python preference and lower language diversity. We further analyse 9,826 reasoning traces and find that most Python choices are automatic or driven primarily by ease, rather than explicit consideration of project requirements. In a smaller but important set of cases, models fabricate contextual support for choosing Python - a failure mode we call phantom evidence - or produce code that contradicts the language selected in their own reasoning.

Figures

Figures reproduced from arXiv: 2608.06041 by the authors.

Figure 1
Figure 1. Overall language preferences. Distribution of primary implementation languages across all models and LANGCHOICEBENCH tasks. reveal a strong concentration around a small num￾ber of familiar languages, clearly led by Python. 5 Reasoning-Trace Analysis The benchmark results show that many models select Python even for tasks where it is suboptimal, often clashing with their own recommendations. To better understand this… view at source ↗
Figure 2
Figure 2. Why models choose Python. Distribution of mutually exclusive reasoning-trace classifications for all responses with Python as the primary language, from the 15 models that expose raw reasoning traces. the final response unexpectedly switches to Python. This is a separate consistency error between the model’s stated decision and generated code. To￾gether, phantom evidence and language mismatch affect 9.7% of traces, … view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

69 extracted references · 40 canonical work pages

  1. [1]

    and Harman, Mark and Syme, Don and Noppen, Joost and Yannakoudakis, Helen and Nauck, Detlef , booktitle =

    Twist, Lukas and Zhang, Jie M. and Harman, Mark and Syme, Don and Noppen, Joost and Yannakoudakis, Helen and Nauck, Detlef , booktitle =. 2026 , month =. doi:10.18653/v1/2026.findings-acl.15 , url =

    work page doi:10.18653/v1/2026.findings-acl.15 2026

  2. [2]

    Studying

    Donato, Benedetta and Mariani, Leonardo and Micucci, Daniela and Riganelli, Oliviero , year = 2025, month = feb, eprint =. Studying. The. doi:10.48550/arXiv.2502.17450 , urldate =

    arXiv doi:10.48550/arxiv.2502.17450 2025

  3. [3]

    and Harman, Mark and Wang, Meng , year = 2025, month = jan, journal =

    Ouyang, Shuyin and Zhang, Jie M. and Harman, Mark and Wang, Meng , year = 2025, month = jan, journal =. An. doi:10.1145/3697010 , urldate =

    doi:10.1145/3697010 2025

  4. [4]

    Spearman

    Dodge, Yadolah , year = 2008, pages =. Spearman. The. doi:10.1007/978-0-387-32833-1_379 , urldate =

    work page doi:10.1007/978-0-387-32833-1_379 2008

  5. [5]

    Friedman, Dan and Dieng, Adji Bousso , year = 2023, month = feb, journal =. The

    work page 2023

  6. [6]

    Android's

    Google , journal =. Android's

    work page

  7. [7]

    Bilokon, Paul and Gunduz, Burak , year = 2023, month = sep, number =. C++. doi:10.48550/arXiv.2309.04259 , urldate =. arXiv , keywords =:2309.04259 , primaryclass =

    work page Pith review arXiv doi:10.48550/arxiv.2309.04259 2023

  8. [8]

    Embedded

    Python , year = 2026, journal =. Embedded

    work page 2026

Show all 69 references
  1. [9]

    Wei, Kai , journal =. How

  2. [10]

    Apple Developer , urldate =

    Swift , author =. Apple Developer , urldate =

  3. [11]

    Mozilla , year = 2026, month = may, journal =

    2026

  4. [12]

    A Memory Footprint Optimization Framework for

    Katsaragakis, Manolis and Papadopoulos, Lazaros and Konijnenburg, Mario and Catthoor, Francky and Soudris, Dimitrios , year = 2023, month = sep, journal =. A Memory Footprint Optimization Framework for. doi:10.1016/j.sysarc.2023.102936 , urldate =

    2023

  5. [13]

    The Cloudflare Blog , urldate =

    Kernel Bypass , author =. The Cloudflare Blog , urldate =

  6. [14]

    Nam, Chung Tran , year = 2025, month = sep, journal =. The

    2025

  7. [15]

    Gross, Sam , journal =

  8. [16]

    Programming with

    Epic , journal =. Programming with

  9. [17]

    Scripting in

    Unity , journal =. Scripting in

  10. [18]

    Noble , year = 2025, month = jul, journal =. Top 5

    2025

  11. [19]

    VMSOIT , year = 2025, month = apr, journal =. Why

    2025

  12. [20]

    MicroPython , urldate =

  13. [21]

    Howarth, Jesse , year = 2020, urldate =. Why

    2020

  14. [22]

    Biswas, Andrei , year = 2025, month = dec, journal =. Why

    2025

  15. [23]

    Singh, Pampreet , journal =. Key

  16. [24]

    Chen, Zhifei and Chen, Lin and Yang, Yibiao and Feng, Qiong and Li, Xuansong and Song, Wei , year = 2024, month = jun, journal =. Risky. doi:10.1145/3649593 , urldate =

    2024 doi

  17. [25]

    Angular vs

    Raghunathan, Venkatesh , journal =. Angular vs

  18. [26]

    Godot Engine documentation , urldate =

    Scripting Languages , author =. Godot Engine documentation , urldate =

  19. [27]

    Bugden, William and Alahmar, Ayman , year = 2022, month = may, journal =. The. doi:10.1142/S0218194022500231 , urldate =

    2022 doi

  20. [28]

    Performance vs

    Costanzo, Manuel and Rucci, Enzo and Naiouf, Marcelo and Giusti, Armando De , year = 2021, month = oct, pages =. Performance vs. 2021. doi:10.1109/CLEI53233.2021.9640225 , urldate =

    2021

  21. [29]

    and Hollenbeck, Celeste and Maj, Petr and Vitek, Olga and Vitek, Jan , year = 2019, month = oct, journal =

    Berger, Emery D. and Hollenbeck, Celeste and Maj, Petr and Vitek, Olga and Vitek, Jan , year = 2019, month = oct, journal =. On the. doi:10.1145/3340571 , urldate =

    2019 doi

  22. [30]

    Assistance or

    Pu, Kevin and Lazaro, Daniel and Arawjo, Ian and Xia, Haijun and Xiao, Ziang and Grossman, Tovi and Chen, Yan , year = 2025, month = apr, series =. Assistance or. Proceedings of the 2025. doi:10.1145/3706598.3713357 , urldate =

    2025

  23. [31]

    and Bellomo, Stephany and Ozkaya, Ipek and Nord, Robert L

    Ernst, Neil A. and Bellomo, Stephany and Ozkaya, Ipek and Nord, Robert L. and Gorton, Ian , year = 2015, month = aug, series =. Measure It?. Proceedings of the 2015 10th. doi:10.1145/2786805.2786848 , urldate =

    2015

  24. [32]

    Jiang, Juyong and Wang, Fan and Shen, Jiasi and Kim, Sungju and Kim, Sunghun , year = 2024, month = jun, journal =. A. arXiv , keywords =:2406.00515 , urldate =

    2024 arXiv

  25. [33]

    Li, Lehui and Jia, Ruixuan and Yang, Guo-Ye and Li, Jia , year = 2026, month = jun, urldate =. An. Deep

    2026

  26. [34]

    Rasnayaka, Sanka and Wang, Guanlin and Shariffdeen, Ridwan and Iyer, Ganesh Neelakanta , year = 2024, month = sep, series =. An. Proceedings of the 1st. doi:10.1145/3643795.3648379 , urldate =

    2024

  27. [35]

    The GitHub Blog , urldate =

    Octoverse:. The GitHub Blog , urldate =

  28. [36]

    , booktitle =

    Twist, Lukas and Zhang, Jie M. , booktitle =. 2026 , month =. doi:10.1145/3793302.3793562 , url =

    2026

  29. [37]

    Pujar, Saurabh and Ceka, Ira and Manotas, Irene and Kaiser, Gail and Ray, Baishakhi and Ramji, Shyam , year = 2026, month = jan, journal =. Code. arXiv , keywords =:2506.13932 , primaryclass =

    2026 arXiv

  30. [38]

    Hou, Xinyi and Zhao, Yanjie and Liu, Yue and Yang, Zhou and Wang, Kailong and Li, Li and Luo, Xiapu and Lo, David and Grundy, John and Wang, Haoyu , year = 2024, month = dec, journal =. Large. doi:10.1145/3695988 , urldate =

    2024 doi

  31. [39]

    and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik R

    Jimenez, Carlos E. and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik R. , year = 2023, month = oct, urldate =. The

    2023

  32. [40]

    Vibe Coding: Programming through Conversation with Artificial Intelligence , shorttitle =

    Sarkar, Advait and Drosos, Ian , year = 2025, month = jun, eprint =. Vibe Coding: Programming through Conversation with Artificial Intelligence , shorttitle =. Proceedings of the 36th

    2025

  33. [41]

    Vangala, Bhanu Prakash and Adibifar, Ali and Gehani, Ashish and Malik, Tanu , year = 2026, month = mar, eprint =

    2026

  34. [42]

    and Tang, Xiangru and Zhuge, Mingchen and Pan, Jiayi and Song, Yueqi and Li, Bowen and Singh, Jaskirat and Tran, Hoang H

    Wang, Xingyao and Li, Boxuan and Song, Yufan and Xu, Frank F. and Tang, Xiangru and Zhuge, Mingchen and Pan, Jiayi and Song, Yueqi and Li, Bowen and Singh, Jaskirat and Tran, Hoang H. and Li, Fuqiang and Ma, Ren and Zheng, Mingzhang and Qian, Bill and Shao, Yanjun and Muennigh...

    2024

  35. [43]

    and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik and Press, Ofir , year = 2024, month = dec, series =

    Yang, John and Jimenez, Carlos E. and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik and Press, Ofir , year = 2024, month = dec, series =. Proceedings of the 38th

    2024

  36. [44]

    Evaluating

    Chen, Mark and Tworek, Jerry and Jun, Heewoo and Yuan, Qiming and Pinto, Henrique Ponde de Oliveira and Kaplan, Jared and Edwards, Harri and Burda, Yuri and Joseph, Nicholas and Brockman, Greg and Ray, Alex and Puri, Raul and Krueger, Gretchen and Petrov, Michael and Khlaaf, H...

    doi arXiv

  37. [45]

    Security

    Fu, Yujia and Liang, Peng and Tahir, Amjed and Li, Zengyang and Shahin, Mojtaba and Yu, Jiaxin and Chen, Jinfu , year = 2025, month = oct, journal =. Security. doi:10.1145/3716848 , urldate =

    2025 doi

  38. [46]

    doi:10.52202/079017-0367 , urldate =

    Huang, Dong and Qing, Yuhao and Shang, Weiyi and Cui, Heming and Zhang, Jie , year = 2024, month = dec, journal =. doi:10.52202/079017-0367 , urldate =

    2024 doi

  39. [47]

    Benchmarks and

    Paul, Debalina Ghosh and Zhu, Hong and Bayley, Ian , year = 2024, month = jul, pages =. Benchmarks and. 2024. doi:10.1109/AITest62860.2024.00019 , urldate =

    2024

  40. [48]

    Asleep at the

    Pearce, Hammond and Ahmad, Baleegh and Tan, Benjamin and. Asleep at the. Communications of the ACM , volume =. doi:10.1145/3610721 , urldate =

    doi

  41. [49]

    Rabbi, Fazle and Ling, Lin and Wang, Song and Yang, Jinqiu , year = 2026, month = may, number =. Social. doi:10.48550/arXiv.2605.00382 , urldate =. arXiv , keywords =:2605.00382 , primaryclass =

    2026 doi arXiv

  42. [50]

    Proceedings of the

    Tian, Yuchen and Yan, Weixiang and Yang, Qian and Zhao, Xuandong and Chen, Qian and Wang, Wen and Luo, Ziyang and Ma, Lei and Song, Dawn , year = 2025, month = feb, series =. Proceedings of the. doi:10.1609/aaai.v39i24.34717 , urldate =

    2025 doi

  43. [51]

    and Harman, Mark and Yannakoudakis, Helen , year = 2026, month = may, number =

    Twist, Lukas and Zhang, Jie M. and Harman, Mark and Yannakoudakis, Helen , year = 2026, month = may, number =. Library. doi:10.48550/arXiv.2509.22202 , urldate =. arXiv , keywords =:2509.22202 , primaryclass =

    2026 doi arXiv

  44. [52]

    Zhang, Yuanliang and Xie, Yifan and Li, Shanshan and Liu, Ke and Wang, Chong and Jia, Zhouyang and Huang, Xiangbing and Song, Jie and Luo, Chaopeng and Zheng, Zhizheng and Xu, Rulin and Liu, Yitong and Zheng, Si and Liao, Xiangke , year = 2025, month = sep, series =. Unseen. P...

    2025

  45. [53]

    Fang, Hanpei and Tao, Sijie and Chen, Nuo and Chang, Kai-Xin and Sakai, Tetsuya , year = 2025, month = dec, series =. Do. Proceedings of the 2025. doi:10.1145/3767695.3769493 , urldate =

    2025

  46. [54]

    and Rossi, Ryan A

    Gallegos, Isabel O. and Rossi, Ryan A. and Barrow, Joe and Tanjim, Md Mehrab and Kim, Sungchul and Dernoncourt, Franck and Yu, Tong and Zhang, Ruiyi and Ahmed, Nesreen K. , year = 2024, month = sep, journal =. Bias and. doi:10.1162/coli_a_00524 , urldate =

    2024 doi

  47. [55]

    Zhang, Jie and Bu, Qingwen and Xie, Xiaofei and Chen, Junjie and Cui, Heming , year = 2025, month = dec, journal =

    Huang, Dong and M. Zhang, Jie and Bu, Qingwen and Xie, Xiaofei and Chen, Junjie and Cui, Heming , year = 2025, month = dec, journal =. Bias. doi:10.1145/3724117 , urldate =

    2025 doi

  48. [56]

    Ling, Lin and Rabbi, Fazle and Wang, Song and Yang, Jinqiu , year = 2025, month = apr, journal =. Bias. doi:10.1609/aaai.v39i26.34961 , urldate =

    2025 doi

  49. [57]

    Uncovering and

    Liu, Yan and Chen, Xiaokang and Gao, Yan and Su, Zhe and Zhang, Fengji and Zan, Daoguang and Lou, Jian-Guang and Chen, Pin-Yu and Ho, Tsung-Yi , year = 2023, month = may, eprint =. Uncovering and. Advances in. doi:10.48550/arXiv.2305.15377 , urldate =

    2023 doi arXiv

  50. [58]

    Programming

    Moumoula, Micheline B. Programming. doi:10.48550/arXiv.2503.13620 , urldate =. arXiv , keywords =:2503.13620 , primaryclass =

    doi

  51. [59]

    Exploring

    Wang, Chaozheng and Li, Zongjie and Gao, Cuiyun and Wang, Wenxuan and Peng, Ting and Huang, Hailiang and Deng, Yuetang and Wang, Shuai and Lyu, Michael , year = 2026, month = jan, journal =. Exploring. doi:10.1145/3786793 , urldate =

    2026 doi

  52. [60]

    Zhang, Xiaoyu and Zhai, Juan and Ma, Shiqing and Bao, Qingshuang and Jiang, Weipeng and Wang, Qian and Shen, Chao and Liu, Yang , editor =. The. Proceedings of the 63rd. doi:10.18653/v1/2025.acl-long.1038 , urldate =

    2025 doi

  53. [61]

    Chain-of-

    Arcuschin, Iv. Chain-of-. Forty-Third

  54. [62]

    and Leike, Jan and Kaplan, Jared and Perez, Ethan , year = 2025, month = may, number =

    Chen, Yanda and Benton, Joe and Radhakrishnan, Ansh and Uesato, Jonathan and Denison, Carson and Schulman, John and Somani, Arushi and Hase, Peter and Wagner, Misha and Roger, Fabien and Mikulik, Vlad and Bowman, Samuel R. and Leike, Jan and Kaplan, Jared and Perez, Ethan , ye...

    2025 doi arXiv

  55. [63]

    OpenAI , year = 2025, journal =

    2025

  56. [64]

    Proceedings of the 41st

    Gu, Alex and Roziere, Baptiste and Leather, Hugh James and. Proceedings of the 41st

  57. [65]

    IEEE Transactions on Software Engineering , volume =

    Cassano, Federico and Gouwar, John and Nguyen, Daniel and Nguyen, Sydney and. IEEE Transactions on Software Engineering , volume =. doi:10.1109/TSE.2023.3267446 , urldate =

    2023

  58. [66]

    Findings of the

    Liu, Kaiyuan and Pan, Youcheng and Xiang, Yang and He, Daojing and Li, Jing and Du, Yexing and Gao, Tianrun , editor =. Findings of the. doi:10.18653/v1/2025.findings-acl.1036 , urldate =

    2025 doi

  59. [67]

    Proceedings of the 29th

    Zheng, Qinkai and Xia, Xiao and Zou, Xu and Dong, Yuxiao and Wang, Shan and Xue, Yufei and Shen, Lei and Wang, Zihan and Wang, Andi and Li, Yang and Su, Teng and Yang, Zhilin and Tang, Jie , year = 2023, month = aug, series =. Proceedings of the 29th. doi:10.1145/3580305.35997...

    2023

  60. [68]

    Guo, Daya and Yang, Dejian and Zhang, Haowei and Song, Junxiao and Wang, Peiyi and Zhu, Qihao and Xu, Runxin and Zhang, Ruoyu and Ma, Shirong and Bi, Xiao and Zhang, Xiaokang and Yu, Xingkai and Wu, Yu and Wu, Z. F. and Gou, Zhibin and Shao, Zhihong and Li, Zhuoshu and Gao, Zi...

    2025

  61. [69]

    Advances in Neural Information Processing Systems , author =

    Language Models Dont Always Say What They Think:. Advances in Neural Information Processing Systems , author =

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.