Pith. sign in

REVIEW 3 major objections 4 minor 2 cited by

Copilot Arena: A Platform for Code LLM Evaluation in the Wild

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Pairwise votes collected inside a real IDE produce a code-model ranking that static benchmarks do not reproduce.

desk verdict The platform and Snip-It prompting are real contributions, but the headline ranking-divergence claim rests on underpowered correlations. read the letter →

arxiv 2502.09328 v1 pith:BVNPT4Z3 submitted 2025-02-13 cs.SE

Wayne Chi , Valerie Chen , Anastasios Nikolas Angelopoulos , Wei-Lin Chiang , Aditya Mittal , Naman Jain , Tianjun Zhang , Ion Stoica
show 2 more authors
Chris Donahue Ameet Talwalkar
This is my paper · ORCID
classification cs.SE
keywords codeLLMevaluationhumanpreferencesin-the-wildcompletionfill-in-the-middleBradley-TerrymodelleaderboardIDEintegration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Copilot Arena is a Visual Studio Code extension that shows developers two code completions from different LLMs and lets them accept one, converting real editing behavior into pairwise preference votes. Using 11,604 votes from 1,642 users across 10 models, the paper builds a Bradley-Terry leaderboard and reports that its rankings differ sharply from static coding benchmarks (Spearman rank correlation at most 0.1) while correlating more strongly with human-preference leaderboards from chat platforms (0.48 general, 0.62 coding). The paper attributes this divergence to a more realistic data distribution: 103 programming languages, 24 natural languages, long real contexts, 65% infilling tasks, and day-to-day development tasks rather than competition problems. If correct, the result matters because it suggests that benchmark rankings may not reflect which model helps a working developer, and that evaluation should be stratified by task category.

What carries the argument

The load-bearing mechanism is the Copilot Arena platform itself: a VSCode extension that displays two completions stacked like a Git diff, randomizes which model appears on top, and counts an accepted completion as a vote. Three components carry the argument: a latency-aware sampling strategy that models the maximum latency of a model pair as a log-normal CDF and minimizes expected latency while preserving pair coverage; a prompting scheme called Snip-It that asks instruction-tuned chat models to generate a snippet repeating part of the prefix and suffix and then strips the repeated text, turning left-to-right chat models into usable fill-in-the-middle completers; and a Bradley-Terry logistic-regression model that converts pairwise votes into model strengths, with bootstrap confidence intervals for the leaderboard.

What would settle it

Re-run the Bradley-Terry logistic regression with an added covariate for whether the winning model appeared in the top position. If the position coefficient is non-negligible or the model ordering changes materially, the leaderboard is distorted and the comparison against static benchmarks would need revision.

Watch

Extended reading notes

Core claim

The paper's central claim is that model rankings obtained from pairwise preferences collected in a developer's IDE differ from rankings produced by existing evaluations, and that this difference traces to the distribution of tasks and data. Copilot Arena's leaderboard places DeepSeek Coder and Claude-3.5-Sonnet at the top, followed closely by Codestral, while smaller models such as GPT-4o mini and Qwen-2.5-Coder-32B do substantially better on static benchmarks than they do in the arena; the Spearman rank correlation between Copilot Arena and each of LiveBench, LiveCodeBench, and BigCodeBench is at most 0.1, versus 0.48 with the general Chatbot Arena leaderboard and 0.62 with its coding subset. The paper also establishes that user preferences are stable across programming languages (only 6.6% of win-rate comparisons change substantially) but vary by task category (31.1% change), and that models explicitly trained for infilling show no special advantage on infilling tasks, which the paper reads as evidence that the votes track code usefulness rather than formatting.

Load-bearing premise

The whole leaderboard assumes that the 11,604 votes, collected after randomizing which completion appears on top, estimate model preference without a position bias, yet users chose the top completion 86% of the time and the Bradley-Terry model does not include a position covariate.

Editorial extensions

If this is right

  • Static coding benchmark rankings should not be read as rankings of real coding-assistant usefulness: Copilot Arena's rankings correlate at at most 0.1 with LiveBench, LiveCodeBench, and BigCodeBench.
  • Smaller models such as GPT-4o mini and Qwen-2.5-Coder-32B look better on static benchmarks than they do to in-IDE users, so benchmark gains for small models may not transfer to workflow value.
  • Task category matters more than programming language: 31.1% of win-rate comparisons changed substantially between frontend/backend and algorithm-design tasks, versus 6.6% between non-Python and Python, so evaluations should report task-stratified rankings.
  • Models trained for infilling do not gain a detectable edge on infilling tasks in the arena, suggesting the preference signal reflects content quality rather than completion formatting.
  • The Snip-It prompting scheme improves fill-in-the-middle pass@1 for instruction-tuned chat models (for example, Claude-3.5-Sonnet from 56.1% to 73.0% on HumanEval-infilling), which is what made the multi-model evaluation possible.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If in-IDE preference is the right ground truth, benchmark builders could test the paper's attribution directly: sample static-benchmark tasks matched to Copilot Arena's task mix and see whether static rankings shift toward the arena's.
  • The reported 86% top-completion rate is large enough that even with randomized placement, model-specific differences in how often a completion is accepted as top versus bottom could perturb the leaderboard; a position-stratified analysis would settle whether the divergence from benchmarks is robust.
  • The Snip-It equalization of formatting means the arena may underweight formatting or stylistic differences that matter in real products; a follow-up comparing Snip-It-processed and native-FiM outputs for the same models would quantify this.
  • The general pattern of embedding pairwise comparison into a real tool could extend beyond code to other AI-assisted workflows where the environment itself shapes what a good answer is.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces Copilot Arena, a VSCode extension that collects pairwise human preferences between code completions from ten LLMs in the context of real developer workflows. The system includes a novel side-by-side completion interface, a latency-aware model-pair sampling strategy, and a prompting scheme (Snip-It) that adapts instruction-tuned chat models to fill-in-the-middle tasks. The authors report a leaderboard based on Bradley-Terry analysis of 11,604 votes, compare their ranking against static benchmarks and Chatbot Arena, and analyze how user preferences vary across programming languages, task categories, context lengths, and completion structures. They conclude that Copilot Arena rankings differ from those of existing evaluations, attributing this to the more realistic and diverse distribution of tasks and data in their platform.

Significance. If the central claim is sound, the paper makes a valuable contribution: it provides a public platform and dataset for in-the-wild evaluation of coding assistants, and it raises the possibility that static coding benchmarks miss aspects of model quality that matter to real developers. The platform design is thoughtful, especially the latency-aware sampling strategy and the Snip-It prompting scheme, which is evaluated on a realistic infilling benchmark with clear gains. The authors ship an open-source system and a hand-curated data sample, which are concrete artifacts that the community can build on. The style-controlled BT ablation in Appendix D is a useful check. However, the headline claim that 'model rankings from Copilot Arena differ from those of existing evaluations' is statistically fragile: the rank-correlation evidence in Figure 5 is based on at most 10 models, and the 86% top-completion bias is documented but not modeled as a potential confound. These issues do not invalidate the platform contribution but they do undercut the paper's main interpretive claim as currently presented.

major comments (3)
  1. [§4.2, Figure 5] The central claim that Copilot Arena rankings differ from static benchmarks rests on Spearman correlations computed on at most 10 overlapping models. With n=10, the two-sided 5% critical value for Spearman's rho is about 0.65, so r=0.10 is not significantly different from zero, and the 95% confidence interval for r=0.62 (approximately -0.05 to 0.93) overlaps heavily with the interval for r=0.10. The paper reports neither p-values nor confidence intervals for these correlations, so the data cannot distinguish 'rankings differ' from 'rankings are similar within noise.' Please report exact n for each comparison, provide bootstrap or Fisher-z confidence intervals, and either temper the claim to a descriptive observation or strengthen it with a test that accounts for the small n.
  2. [Appendix C and §4.1] The paper documents that users selected the top completion 86% of the time, and §2.1 states that ordering is randomized. Randomization of position balances a main position effect, but it does not remove the possibility that position interacts with model attributes (e.g., latency, response length, or code style). The Bradley-Terry model in Eq. (3) includes the position indicator X_i but no position covariate or interaction term, so a systematic interaction could bias the estimated β values and, consequently, every comparison in Sections 4 and 5. I request an explicit position-controlled analysis: add a position main effect and a position-by-model interaction test, or otherwise show that the 86% first-position preference does not alter the leaderboard or the stratification results.
  3. [§1 and §4.2] The attribution 'we attribute these differences to the more realistic distribution of data and tasks' is causal and is not tested. The low correlations with static benchmarks, even if accepted, could also arise from differences in user demographics, the specific model versions/deployment dates, the UI effect of showing two completions, or the particular latency of each model. The paper does not rule out these alternatives. I recommend either running a covariate-controlled comparison (e.g., conditioning on task category and context length in both Copilot Arena and the benchmark data) or recasting the statement as a hypothesis that motivated the analysis rather than an established conclusion.
minor comments (4)
  1. [Figure 5] The notation is inconsistent: the figure captions report 'r=0.10' etc., while the text uses 'rs' for Spearman's rho. Please align the notation and clarify that these are rank correlations.
  2. [Table 1] The checkmark/cross entries for Chatbot Arena rows are visually unclear (the footnote symbol is typeset in a way that makes the cell contents ambiguous). Please use explicit symbols and a legend.
  3. [§5.2] The definition of Δ_i,j uses a one-line indicator function that is easy to misread; please define it as Δ_{i,j} = 1{W_{i,j}(X) - W_{i,j}(X̃) > ε} with a sentence explaining that ε is the 90th percentile of the absolute differences.
  4. [Appendix A.2.3] The two example error snippets are helpful, but they are rendered as plain text without syntax coloring or line numbers, which makes it hard to see the exact formatting errors described in the text. A small figure or table would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: leaderboard and comparisons are empirically grounded in newly collected preference data; self-citations are comparative or methodological, not load-bearing.

full rationale

The paper's central claims are empirically grounded rather than derived from their own outputs by construction. The Copilot Arena leaderboard is estimated from 11,604 pairwise user votes using a standard Bradley-Terry model with bootstrapped confidence intervals; no parameter of that model is fitted to the external benchmark rankings and then reported as a prediction. The comparison against LiveBench, LiveCodeBench, BigCodeBench, and Chatbot Arena is an external rank-correlation analysis of independently collected evaluations, not a quantity forced by the paper's own equations. The latency-sampling parameters θ and the win-rate threshold ε=0.166 are descriptive or data-collection tuning choices; they shape which pairs users see and how changes are summarized, but they do not define the leaderboard or the headline claim that rankings differ. The Snip-It prompting scheme is validated on the external HumanEval-infilling dataset, not on Copilot Arena votes, so it is not a self-referential prediction. Self-citations (Chatbot Arena, LiveCodeBench, BigCodeBench) appear as comparison targets or standard methodology; they are not invoked as uniqueness theorems or as the sole justification for a controversial premise. Concerns about top-position selection bias (86% top completion) and the small number of overlapping models in the Spearman comparisons are threats to validity or statistical power, not circularity: they concern whether the measurements are unbiased and precise, not whether the conclusions reduce to the inputs by definition.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claims rest on assumptions about vote validity under position bias, representativeness of self-selected users, reliability of LLM-generated task labels, and the log-normal latency model used for sampling. No new physical or mathematical entities are introduced; the paper's contribution is a measurement platform and an empirical dataset.

free parameters (3)
  • Win-rate change threshold ε = 0.166
    Set to the 90th percentile of observed win-rate changes in Section 5.2; the count of 'significant changes' depends directly on this data-derived cutoff.
  • Sampling temperature τ = 5 to 10
    Hand-chosen in Section 2.2 to trade off latency versus pair coverage; affects which model pairs users see and hence the vote distribution.
  • Latency-model parameters θ = estimated from historical data
    In Eqs. (1)-(2), θ is fit by gradient descent to minimize expected pairwise latency; it shapes the comparison graph from which the leaderboard is built.
assumptions (5)
  • domain assumption User preference votes follow a Bradley-Terry model with independent battles and no position bias after order randomization.
    Standard in arena evaluations (Chiang et al. 2024); threatened by the reported 86% first-completion bias, which the paper does not explicitly model.
  • domain assumption The pairwise maximum latency follows a log-normal distribution with parameters estimated from historical data (Section 2.2).
    Used to optimize the sampling distribution pθ; if the log-normal fit is poor, the latency-optimal sampling could still be biased.
  • domain assumption LLM-as-a-judge (GPT-4o-mini) accurately clusters code contexts into task categories (Appendix C).
    Task labels underpin the task-type win-rate comparisons; only manual sanity checks are reported.
  • domain assumption HumanEval-infilling pass@1 is a valid proxy for real-world fill-in-the-middle quality (Section 2.3).
    Used to validate Snip-It; the paper acknowledges offline evaluation is not a perfect metric.
  • domain assumption Self-selected, privacy-opt-in users and files are representative of developer workflows.
    Users volunteer and can opt out of code collection; the paper notes unclear generalizability in Limitations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Copilot Arena: A Platform for Code LLM Evaluation in the Wild." pith.science (2026). https://pith.science/paper/BVNPT4Z3

@misc{pith2026250209328,
  author       = {Pith},
  title        = {Pith review of: Copilot Arena: A Platform for Code LLM Evaluation in the Wild},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BVNPT4Z3}},
  note         = {Machine review of arXiv:2502.09328}
}
read the original abstract

Evaluating in-the-wild coding capabilities of large language models (LLMs) is a challenging endeavor with no clear solution. We introduce Copilot Arena, a platform to collect user preferences for code generation through native integration into a developer's working environment. Copilot Arena comprises a novel interface for comparing pairs of model outputs, a sampling strategy optimized to reduce latency, and a prompting scheme to enable code completion functionality. Copilot Arena has served over 4.5 million suggestions from 10 models and collected over 11k pairwise judgements. Our results highlight the importance of model evaluations in integrated settings. We find that model rankings from Copilot Arena differ from those of existing evaluations, which we attribute to the more realistic distribution of data and tasks contained in Copilot Arena. We also identify novel insights into human preferences on code such as an observed consistency in user preference across programming languages yet significant variation in preference due to task category. We open-source Copilot Arena and release data to enable human-centric evaluations and improve understanding of coding assistants.

Figures

Figures reproduced from arXiv: 2502.09328 by the authors.

Figure 1
Figure 1. Copilot Arena is a platform for conducting realistic evaluations of code LLMs, collecting [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. We introduce Copilot Arena, a VSCode extension to collect human preferences of code [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The likelihood of users accepting one of the two completions as a function of empirical [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: We evaluate the effectiveness of our prompting scheme by comparing LLM performance [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: We compare model rankings in Copilot Arena (1st column) to existing evaluations, [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Copilot Arena data is diverse in programming and natural languages, downstream [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Significant win-rate changes (∆) as a result of different data partitions: frontend/back [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Natural languages in Copilot Arena Programming Language Detection. We detect programming languages in Copilot Arena by using the file’s extension type ( [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]
Figure 9
Figure 9. Figure 9: Programming languages in Copilot Arena. For image clarity, we only show programming [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]
Figure 10
Figure 10. Figure 10: Number of votes for each language model. [PITH_FULL_IMAGE:figures/full_fig_p024_10.png]
Figure 11
Figure 11. Figure 11: Number of votes over time. Code Structure Detection. To detect the presence of FiM, we check if there exists a suffix. If there is only the prefix, we label it as ”completion-only”. To detect if there are comments, we check if any of the 5 previous lines start with co…
Figure 12
Figure 12. Figure 12: Completion similarity vs. decision time, grouped by selection of the first or second [PITH_FULL_IMAGE:figures/full_fig_p025_12.png]
Figure 13
Figure 13. Figure 13: Win-rate difference based on Task: frontend/backend versus algorithmic design [PITH_FULL_IMAGE:figures/full_fig_p031_13.png]
Figure 14
Figure 14. Figure 14: Win-rate difference based on FiM: whether the task is FiM or not. [PITH_FULL_IMAGE:figures/full_fig_p032_14.png]
Figure 15
Figure 15. Figure 15: Win-rate difference based on context length: context length in top versus bottom 20 [PITH_FULL_IMAGE:figures/full_fig_p033_15.png]
Figure 16
Figure 16. Figure 16: Win-rate difference based on programming language (PL): Non-python code versus [PITH_FULL_IMAGE:figures/full_fig_p034_16.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Nonparametric LLM Evaluation from Preference Data

    cs.LG 2026-01 conditional novelty 7.0 of 10

    A debiased, nonparametric estimator (DMLRank) yields efficient ranking scores and valid confidence intervals for LLM leaderboards from preference data, covering Bradley-Terry, Borda, and rank centrality.

  2. Structure-Aware Fill-in-the-Middle Pretraining for Code

    cs.CL 2025-05 conditional novelty 7.0 of 10

    AST-FIM masks complete syntax-tree subtrees during fill-in-the-middle pretraining, improving infilling performance on real-world code edits.

Reference graph

Works this paper leans on

67 extracted references · 37 canonical work pages · cited by 2 Pith papers

  1. [1]

    The impact of ai on developer productivity: Evidence from github copilot

    Sida Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer. The impact of ai on developer productivity: Evidence from github copilot. arXiv preprint arXiv:2302.06590 , 2023. 12

    arXiv 2023

  2. [2]

    Michael Oberst, Davis Liang, and Zachary C. Lipton. The science of AI evaluation at Abridge. https://www.abridge.com/ai/science-ai-evaluation, 2024

    work page 2024

  3. [3]

    Beyond the bar: Generative ai as a transformative component in legal document review

    Eugene Yang, Roshanak Omrani, Evan Curtin, Tara Emory, Lenora Gray, Jeremy Pickens, Nathan Reff, Cristin Traylor, Sean Underwood, David D Lewis, et al. Beyond the bar: Generative ai as a transformative component in legal document review. In 2024 IEEE International Conference on Big Data (BigData) , pages 4779–4788. IEEE, 2024

    work page 2024

  4. [4]

    Evaluation gaps in machine learning practice

    Ben Hutchinson, Negar Rostamzadeh, Christina Greer, Katherine Heller, and Vinodkumar Prabhakaran. Evaluation gaps in machine learning practice. In Proceedings of the 2022 ACM conference on fairness, accountability, and transparency , pages 1859–1876, 2022

    work page 2022

  5. [5]

    Benchmarks as microscopes: A call for model metrology

    Michael Saxon, Ari Holtzman, Peter West, William Yang Wang, and Naomi Saphra. Benchmarks as microscopes: A call for model metrology. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=bttKwCZDkm

    work page 2024

  6. [6]

    Ai agents that matter

    Sayash Kapoor, Benedikt Stroebl, Zachary S Siegel, Nitya Nadgir, and Arvind Narayanan. Ai agents that matter. arXiv preprint arXiv:2407.01502 , 2024

    arXiv 2024

  7. [7]

    Gonzalez, and Ion Stoica

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track ,

    work page

  8. [8]

    Alpacafarm: A simulation framework for methods that learn from human feedback

    Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36, 2023

    work page 2023

Show all 67 references
  1. [9]

    Chatbot arena: An open platform for evaluating llms by human preference

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating llms by human preference. arXiv preprint arXiv:2403.04132, 2024

    2024 arXiv

  2. [10]

    Hannah Rose Kirk, Alexander Whitefield, Paul R¨ ottger, Andrew Michael Bean, Kate- rina Margatina, Rafael Mosquera, Juan Manuel Ciro, Max Bartolo, Adina Williams, He He, Bertie Vidgen, and Scott A. Hale. The PRISM alignment dataset: What par- ticipatory, representative and ind...

    2024

  3. [11]

    Github copilot - your ai pair programmer, 2022

    Github. Github copilot - your ai pair programmer, 2022. URL https://github.com/ features/copilot

    2022

  4. [12]

    Evalu- ating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evalu- ating large language models trained on code. arXiv preprint arXiv:2107.03374 , 2021

    2021 arXiv

  5. [13]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 , 2021. 13

    2021 arXiv

  6. [14]

    Livecodebench: Holistic and contam- ination free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contam- ination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974 , 2024

    2024 arXiv

  7. [15]

    Livebench: A challenging, contamination-free llm benchmark

    Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, et al. Livebench: A challenging, contamination-free llm benchmark. arXiv preprint arXiv:2406.19314 , 2024

    2024 arXiv

  8. [16]

    Expectation vs

    Priyan Vaithilingam, Tianyi Zhang, and Elena L Glassman. Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. In CHI Conference on Human Factors in Computing Systems Extended Abstracts , pages 1–7, 2022

    2022

  9. [17]

    The programmer’s assistant: Conversational interaction with a large language model for software development

    Steven I Ross, Fernando Martinez, Stephanie Houde, Michael Muller, and Justin D Weisz. The programmer’s assistant: Conversational interaction with a large language model for software development. In Proceedings of the 28th International Conference on Intelligent User Interface...

    2023

  10. [18]

    The realhumaneval: Evaluating large language models’ abilities to support programmers

    Hussein Mozannar, Valerie Chen, Mohammed Alsobay, Subhro Das, Sebastian Zhao, Dennis Wei, Manish Nagireddy, Prasanna Sattigeri, Ameet Talwalkar, and David Sontag. The realhumaneval: Evaluating large language models’ abilities to support programmers. arXiv preprint arXiv:2404.0...

    2024 arXiv

  11. [19]

    Wildvision: Evaluating vision-language models in the wild with human preferences

    Yujie Lu, Dongfu Jiang, Wenhu Chen, William Yang Wang, Yejin Choi, and Bill Yuchen Lin. Wildvision: Evaluating vision-language models in the wild with human preferences. arXiv preprint arXiv:2406.11069 , 2024

    2024 arXiv

  12. [20]

    Dai, and Quoc V Le

    Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations , 2022

    2022

  13. [21]

    Incoder: A generative model for code infilling and synthesis, 2023

    Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis, 2023. URL https://arxiv.org/abs/2204.05999

    2023 arXiv

  14. [22]

    Evaluation of llms on syntax-aware code fill-in-the-middle tasks

    Linyuan Gong, Sida Wang, Mostafa Elhoushi, and Alvin Cheung. Evaluation of llms on syntax-aware code fill-in-the-middle tasks. arXiv preprint arXiv:2403.04814 , 2024

    2024 arXiv

  15. [23]

    Efficient training of language models to fill in the middle

    Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models to fill in the middle. arXiv preprint arXiv:2207.14255 , 2022

    2022 arXiv

  16. [24]

    Wildbench: Benchmarking llms with challenging tasks from real users in the wild

    Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. Wildbench: Benchmarking llms with challenging tasks from real users in the wild. arXiv preprint arXiv:2406.04770 , 2024

    2024 arXiv

  17. [25]

    Raising the bar on swe-bench verified with claude 3.5 sonnet, 2024

    Anthropic. Raising the bar on swe-bench verified with claude 3.5 sonnet, 2024. URL https://www.anthropic.com/research/swe-bench-sonnet

    2024

  18. [26]

    Rank analysis of incomplete block designs: I

    Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952

    1952

  19. [27]

    Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions, 2024

    Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, 14 Zhihan Zhang, Prateek...

    2024 arXiv

  20. [28]

    Humaneval-xl: A multilingual code gen- eration benchmark for cross-lingual natural language generalization

    Qiwei Peng, Yekun Chai, and Xuhong Li. Humaneval-xl: A multilingual code gen- eration benchmark for cross-lingual natural language generalization. arXiv preprint arXiv:2402.16694, 2024

    2024 arXiv

  21. [29]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, Soroosh Mariooryad, Yifan Ding, Xinyang Geng, Fred Alcober, Roy Frostig, and Mark Omernick and. Gemini 1.5: Unlocking multimodal understan...

    2024 arXiv

  22. [30]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022

    2022 arXiv

  23. [31]

    Fine-grained human feedback gives better rewards for language model training

    Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. Advances in Neural Information Processing Systems , 36:59008–59...

    2023

  24. [32]

    Training language models with language feedback

    J´ er´ emy Scheurer, Jon Ander Campos, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. Training language models with language feedback. arXiv preprint arXiv:2204.14146, 2022

    2022 arXiv

  25. [33]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:2773...

    2022

  26. [34]

    Gonzalez, and Wei-Lin Chiang

    Christopher Chou, Lisa Dunlap, Koki Mashita, Krishna Mandal, Trevor Darrell, Ion Stoica, Joseph E. Gonzalez, and Wei-Lin Chiang. Visionarena: 230k real world user-vlm conversations with preference labels. 2024. URL https://arxiv.org/abs/2412.08687

    2024 arXiv

  27. [35]

    CodeXGLUE: A machine learning benchmark dataset for code understanding and generation

    Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, MING GONG, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shuj...

    2021

  28. [36]

    Codegen: An open large language model for code with multi-turn program synthesis

    Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations ,

  29. [37]

    Ming Zhu, Aneesh Jain, Karthik Suresh, Roshan Ravindran, Sindhu Tipirneni, and Chan- dan K. Reddy. Xlcost: A benchmark dataset for cross-lingual code intelligence, 2022. URL https://arxiv.org/abs/2206.08474. 15

    2022 arXiv

  30. [38]

    URL https://openreview.net/forum?id=iaYcJKpY2B_

  31. [39]

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems , 36, 2023

    2023

  32. [40]

    Recode: Robustness evaluation of code generation models

    Shiqi Wang, Zheng Li, Haifeng Qian, Chenghao Yang, Zijian Wang, Mingyue Shang, Varun Kumar, Samson Tan, Baishakhi Ray, Parminder Bhatia, et al. Recode: Robustness evaluation of code generation models. In Proceedings of the 61st Annual Meeting of the Association for Computation...

    2023

  33. [41]

    xcodeeval: A large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval

    Mohammad Abdullah Matin Khan, M Saiful Bari, Xuan Long Do, Weishi Wang, Md Rizwan Parvez, and Shafiq Joty. xcodeeval: A large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval. arXiv preprint arXiv:2303.03004 , 2023

    2023 arXiv

  34. [42]

    Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations , 2023

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations , 2023

    2023

  35. [43]

    Multipl-e: a scalable and polyglot approach to benchmarking neural code generation

    Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. Multipl-e: a scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions on S...

    2023

  36. [44]

    Codescope: An execution-based multilingual multitask multidimensional benchmark for evaluating llms on code understanding and generation

    Weixiang Yan, Haitian Liu, Yunkun Wang, Yunzhe Li, Qian Chen, Wen Wang, Tingyu Lin, Weishan Zhao, Li Zhu, Shuiguang Deng, et al. Codescope: An execution-based multilingual multitask multidimensional benchmark for evaluating llms on code understanding and generation. arXiv prep...

    2023 arXiv

  37. [45]

    Large language models of code fail at completing code with potential bugs

    Tuan Dinh, Jinman Zhao, Samson Tan, Renato Negrinho, Leonard Lausen, Sheng Zha, and George Karypis. Large language models of code fail at completing code with potential bugs. Advances in Neural Information Processing Systems , 36, 2023

    2023

  38. [46]

    Octopack: Instruction tuning code large language models

    Niklas Muennighoff, Qian Liu, Armel Randy Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre. Octopack: Instruction tuning code large language models. In The Twelfth International Conference on Learning Represen...

    2023

  39. [47]

    R2e: Turning any github repository into a programming agent environment

    Naman Jain, Manish Shetty, Tianjun Zhang, King Han, Koushik Sen, and Ion Stoica. R2e: Turning any github repository into a programming agent environment. In Forty-first International Conference on Machine Learning , 2024

    2024

  40. [48]

    Intercode: Stan- dardizing and benchmarking interactive coding with execution feedback

    John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. Intercode: Stan- dardizing and benchmarking interactive coding with execution feedback. Advances in Neural Information Processing Systems, 36, 2024

    2024

  41. [49]

    Grounded copilot: How programmers interact with code-generating models

    Shraddha Barke, Michael B James, and Nadia Polikarpova. Grounded copilot: How programmers interact with code-generating models. Proceedings of the ACM on Programming Languages, 7(OOPSLA1):85–111, 2023. 16

    2023

  42. [50]

    Bernstein, and Percy Liang

    Mina Lee, Megha Srivastava, Amelia Hardy, John Thickstun, Esin Durmus, Ashwin Paranjape, Ines Gerard-Ursin, Xiang Lisa Li, Faisal Ladhak, Frieda Rong, Rose E Wang, Minae Kwon, Joon Sung Park, Hancheng Cao, Tony Lee, Rishi Bommasani, Michael S. Bernstein, and Percy Liang. Evalu...

    2023

  43. [51]

    Ai-assisted code authoring at scale: Fine-tuning, deploying, and mixed methods evaluation

    Vijayaraghavan Murali, Chandra Maddila, Imad Ahmad, Michael Bolin, Daniel Cheng, Negar Ghorbani, Renuka Fernandez, Nachiappan Nagappan, and Peter C Rigby. Ai-assisted code authoring at scale: Fine-tuning, deploying, and mixed methods evaluation. Proceedings of the ACM on Softw...

    2024

  44. [52]

    Reading between the lines: Modeling user behavior and costs in ai-assisted programming

    Hussein Mozannar, Gagan Bansal, Adam Fourney, and Eric Horvitz. Reading between the lines: Modeling user behavior and costs in ai-assisted programming. In Proceedings of the CHI Conference on Human Factors in Computing Systems , pages 1–16, 2024

    2024

  45. [53]

    The productivity effects of generative ai: Evidence from a field experiment with github copilot

    Kevin Zheyuan Cui, Mert Demirer, Sonia Jaffe, Leon Musolff, Sida Peng, and Tobias Salz. The productivity effects of generative ai: Evidence from a field experiment with github copilot. 2024

    2024

  46. [54]

    Need help? designing proactive ai assistants for programming

    Valerie Chen, Alan Zhu, Sebastian Zhao, Hussein Mozannar, David Sontag, and Ameet Talwalkar. Need help? designing proactive ai assistants for programming. arXiv preprint arXiv:2410.04596, 2024

    2024 arXiv

  47. [55]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder: When the large language model meets programming – the rise of code intelligence, 2024. URL https://arxiv.o...

    2024 arXiv

  48. [56]

    Language models for code completion: A practical evaluation, 2024

    Maliheh Izadi, Jonathan Katzy, Tim van Dam, Marc Otten, Razvan Mihai Popescu, and Arie van Deursen. Language models for code completion: A practical evaluation, 2024. URL https://arxiv.org/abs/2402.16197

    2024 arXiv

  49. [57]

    A long way to go: Investi- gating length correlations in rlhf

    Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investi- gating length correlations in rlhf. arXiv preprint arXiv:2310.03716 , 2023. 17 A Additional System Details We describe further implementation details and considerations for each of the thre...

    2023 arXiv

  50. [58]

    Peter M. Stahl. Lingua: The most accurate natural language detection library for Java, Kotlin, Rust, Swift, Python and Go. https://github.com/pemistahl/lingua, 2024

    2024

  51. [60]

    PSM presents the code context in the order of prefix and then suffix, using XML notation to demarcate prefix, suffix, and middle segments (e.g., <PREFIX> and </PREFIX>)

    Prefix-Suffix-Middle (PSM) . PSM presents the code context in the order of prefix and then suffix, using XML notation to demarcate prefix, suffix, and middle segments (e.g., <PREFIX> and </PREFIX>). The LLM is then asked to output the middle segment given the prompt

  52. [61]

    SPM is identical to PSM except that the suffix appears before the prefix, which may be more natural than having the suffix appear directly before the output as is the case with PSM

    Suffix-Prefix-Middle (SPM) . SPM is identical to PSM except that the suffix appears before the prefix, which may be more natural than having the suffix appear directly before the output as is the case with PSM

  53. [62]

    sentinel

    Mask . Rather than using start and end tokens to denote the prefix and suffix, the Mask prompt uses a special “sentinel” token to indicate the masked (i.e. middle) code segment [ 55]. The LLM is then requested to fill in the masked code segment

  54. [63]

    pre-fill

    Instructed Prefix Feeding (IPF) . IPF begins with the Mask prompt and then repeats the prefix as a “prefill” of the completion for the language model. 2 This is similar to IPF in Guo et al. [55], except with instructions adjusted to better align with chat models. This approach...

    2024

  55. [64]

    Identify 5-10 main functional clusters or themes

  56. [65]

    Assign each description to the most appropriate cluster

  57. [66]

    Provide a brief name and description for each cluster

  58. [67]

    clusters

    Format the response as valid JSON with the following structure: { "clusters": [ { "name": "cluster_name", "description": "brief cluster description", "descriptions": ["description", "description2"] } ] } Finally, we provide the full code context and ask the LLM to categorize t...

  59. [2023]

    URL https://openreview.net/forum?id=uccHPGDlao

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.