Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Tests as Prompt: A Test-Driven-Development Benchmark for LLM Code Generation

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that on test-driven coding tasks, instruction following and in-context learning matter more than general programming ability, and that prompt length is the main bottleneck affecting all models.

desk verdict WebApp1K is a genuinely new and potentially useful TDD benchmark, but the paper's headline claims about instruction following and context length rest on unvalidated GPT-generated tests and confounded comparisons. read the letter →

arxiv 2505.09027 v1 pith:UV3OZGZH submitted 2025-05-13 cs.SE cs.AI

Yi Cui This is my paper · ORCID
classification cs.SEcs.AI
keywords test-drivendevelopmentLLMcodegenerationbenchmarkinstructionfollowingin-contextlearningcontextlengthWebApp1Kpass@k
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces WebApp1K, a benchmark of 1000 test-driven development (TDD) tasks in which each prompt is a pair of unit tests (one success, one failure) and the model must generate a single web-app component that passes both. The paper's central claim is that on such tasks, instruction following and in-context learning are the decisive capabilities, more so than general coding proficiency or pretraining knowledge. The evidence is that models with low TDD pass rates jump to high pass rates when the tests are retrofitted to their generated code, and that all models' performance drops sharply when the prompt doubles in length. The authors conclude that the practical bottleneck for LLMs in TDD is faithfully absorbing all coded expectations, especially in longer prompts.

What carries the argument

The central object is the TDD task itself: a prompt composed entirely of test code, which simultaneously specifies the feature and verifies the solution. Each WebApp1K scenario pairs one success test and one failure test for a small web-app feature, and the model must emit a single component passing both. The argument's load-bearing comparison is the TLD experiment, in which failing tests are edited—without changing their structure—to match the generated code; the resulting pass-rate jump isolates instruction following from coding ability. The duo-feature upgrade, which doubles the prompt to about 1K tokens, supplies the context-length bottleneck evidence. The paper also builds a seven-category error taxonomy and shows that 93% of error logs contain a single error or two errors of the same type, tracing them to capabilities such as instruction following, in-context learning, and pretraining knowledge.

What would settle it

Randomly sample 100 scenarios, give independent human developers the one-sentence feature description without the tests, and run their implementations against the original test pairs; if many reasonable implementations fail because the tests encode idiosyncratic expectations (for example, requiring backend validation where frontend validation is standard practice), then the benchmark would be measuring test-convention guessing rather than TDD capability.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that LLM performance on TDD code generation is governed by the model's ability to extract and obey the specifications encoded in test code, not by its raw programming strength. WebApp1K contains 1000 scenarios across 20 application domains; each scenario is a small web-app component tested by a success/failure unit-test pair. Across 19 frontier models, pass@1 ranges from about 0.07 to 0.95. When violated tests are minimally modified to match the code the model actually wrote (the TLD experiment), the weakest models' pass@1 more than doubles, showing their code is functional but non-conforming. A duo-feature version with four tests per task cuts pass@1 for all tested models, and a case study shows a strong reasoning model failing only the last of four coded instructions, which the authors attribute to instruction loss.

Load-bearing premise

The claim stands on the assumption that the success/failure test pairs generated by a large language model for each of the 1,000 scenarios faithfully and unambiguously encode the intended feature, since the paper reports no human verification that the tests are correct or unambiguous.

Editorial extensions

If this is right

  • Models ranked high on conventional coding benchmarks can rank low on TDD tasks, so TDD-specific evaluation measures a distinct capability that should be tracked separately.
  • Improving instruction following and in-context learning for code-based instructions should lift TDD pass rates more than additional algorithm or syntax training.
  • Doubling the number of test cases roughly halves many models' pass@1, so prompt length and instruction density are first-order constraints for practical use of LLMs in TDD.
  • Since 93% of error logs contain one or two same-type errors, targeted fixes to a small set of root causes could recover most failures.
  • The benchmark's construction recipe—human-authored scenarios plus model-generated tests—offers a low-cost way to build large TDD benchmarks for other application domains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the central claim holds, then the quality and ambiguity of the test pairs become the main validity risk: because the tests are generated by a single model without human verification, part of what is being measured may be the ability to predict that model's peculiar conventions, such as preferring backend validation over the more common frontend validation.
  • A direct testable extension would be to vary the position of the last test in the prompt; if attention decay is the cause, moving that test earlier should raise pass@1 under the duo-feature format.
  • The TDD-versus-TLD gap could be used as a general "instruction-following quotient" on any code-generation benchmark, giving a cheap way to separate specification-adherence from coding skill.
  • Multi-turn TDD, where tests are added incrementally and the model sees failures, may be a more practical and less bottleneck-prone setting than the single-shot transaction used here.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces WebApp1K, a benchmark of 1000 test-driven development (TDD) tasks for React web applications, in which each task consists of a success and a failure unit test that serve as both the prompt and the verification for code generation. The authors evaluate 19 LLMs and report pass@k scores, an error taxonomy of seven failure types, a 'TLD' experiment in which tests are modified to match generated code, and a duo-feature variant in which two single-feature tasks are merged into one prompt of roughly double length. The paper's central claims are that instruction following and in-context learning are the decisive capabilities for TDD success, more important than general coding proficiency or pretraining knowledge, and that input context length is the main bottleneck to TDD success.

Significance. The WebApp1K benchmark, if carefully validated, would be a useful new evaluation resource for studying code generation from formal test specifications rather than natural language, and the paper has several concrete strengths: the scale of the benchmark (1000 tasks, 19 models), the clear task formulation, the public leaderboard, the structured error analysis, and the explicit consideration of reasoning models. The qualitative observations in Appendix D, such as the ticketSubmission example and the duo-feature export-name trap, are interesting and testable. However, the central interpretative claims are not yet supported: the unvalidated GPT-4o-generated tests, the confounded duo-feature design, and the nonstandard TLD operationalization mean the paper's headline conclusions about capability differentiators and context-length bottleneck go beyond the evidence. These issues are fixable, but they require additional validation experiments and more careful framing.

major comments (4)
  1. [Appendix A; §2.2; Appendix D.1; Table 7] The benchmark's ground truth consists of GPT-4o-generated unit tests with no reported human verification of correctness or unambiguity (Appendix A: 'we prompt GPT-4o to generate a success test and failure test for each scenario'). Appendix D.1's ticketSubmission task shows the risk concretely: the test requires backend validation ('Title is required' after a 400 response), while common practice is frontend validation, and the paper reports that all non-reasoning models are 'misled' into frontend validation; DeepSeek-R1's trace even states 'Maybe the test is written incorrectly, but I have to write the component to pass the tests as given.' If a substantial fraction of the 1000 test pairs are similarly idiosyncratic or under-specified, then failures attributed in Table 7 to 'instruction following' (types B, C, D, G) or 'in-context learning' may instead reflect the impossibility of inferring the intended behavior from an ambiguous test. The TLD experiment (§3.6, Table 8) does not resolve this: modifying tests to match the code only shows proximity between code and test, not that the original test was the correct specification. The central claim that instruction following and in-context learning are the key capabilities for TDD success therefore needs support from a validation study of the test set, e.g., human annotation of a sample of tasks for ambiguity and consensus, and possibly the removal or replacement of non-consensus tasks.
  2. [§4, Table 9] The duo-feature experiment changes both the prompt length and the number of features to be implemented, so the observed drop in pass@1 cannot be attributed to input context length as claimed ('the input context length [is] the main bottleneck to TDD success rate'). To substantiate this claim, the experiment needs a control that increases context length without adding functional requirements (e.g., additional tests for the same feature, or a longer prompt with the same tests) or, conversely, implements two features in two separate single-feature prompts of the same total length. Without such a control, the 'instruction loss' interpretation in §4.2 is not uniquely supported.
  3. [§3.6, Table 8] The TLD experiment is not a test-last development baseline in the sense defined in §1 and Table 1. In §3.6, TLD is operationalized as modifying the failed tests to accommodate the generated code ('we modify the violated tests to accommodate the verbatim code output'). This is a test-adaptation procedure, not the natural-language prompt-based TLD task defined earlier. Consequently, the comparison between TDD and TLD pass rates in Table 8 does not directly support the claim that 'LLMs of low TDD success rate have high success rate on sibling TLD tasks' as the term TLD is used in the introduction. The experiment also only includes models with low TDD pass rates, so the inference about capability differentiators is limited.
  4. [§3.3, Table 7; §3.5] The error taxonomy is presented as a 'conjecture' about connections to model capabilities, yet the abstract and §3.5 treat it as evidence that 'instruction following and in-context learning are critical capabilities... surpassing the importance of general coding proficiency or pretraining knowledge' and that 'all LLMs possess the same knowledge and capabilities.' The mapping from verbatim errors to root causes and capabilities is not validated (e.g., by human annotation, inter-annotator agreement, or an ablation). The claim that all models 'have the same inherent vulnerabilities' (Section 3.5) is an unsupported generalization from error distributions that could be confounded by test quality (see the first major comment). At minimum, the paper should report how the taxonomy was applied and its reliability.
minor comments (6)
  1. [Abstract vs. §6] The abstract and Section 6 refer to '18 frontier LLMs' while Table 6 reports 19 models; please reconcile the count.
  2. [Throughout] There are numerous typos, e.g., 'essemble' (Section 1), 'bewteen' (Section 3.5), 'Performnaces' (Section 4.1), 'valiation' (Appendix D.1), 'instructoin' (Section 5.4), and 'Unfortunatelly' (Appendix D.1).
  3. [§3.1] The statement that 'since the value of pass@k asymptotically increases with k, there is no doubt that the top reasoning models lead other LLMs by an obvious gap' is not justified; pass@k curves of different models can cross, and reasoning models were measured with n=1 while other models used n=10, so their pass@1 estimates have larger variance.
  4. [Appendix B, Table 14] The parameter-tuning table reports pass@1 values on a 100-task subset but the chosen parameters are applied to the full benchmark; please clarify whether the tuning set is disjoint from the evaluation set to avoid optimism in the reported results.
  5. [Appendix D.2, Table 21] The raw-format duo-feature experiment (Table 19) is dramatic, but the explanation that reasoning models commit to the wrong first planning step is anecdotal; the paper would benefit from quantifying how often the first reasoning step matches the eventual (wrong) implementation.
  6. [References] References contain malformed URLs, e.g., 'https://https://developer.mozilla.org/', and the WebApp1K leaderboard link points to a leaderboard rather than to the benchmark dataset; please provide a stable data release with an explicit license.

Circularity Check

1 steps flagged · score 2.0 of 10

Headline capability attribution is partly built into the error taxonomy, though the TLD control gives the central claim independent empirical content.

  1. self definitional [Section 2.3 (success criterion); Section 3.3, Table 7 (error taxonomy); Section 3.5 (conclusion)]
    "If all expectations in Tab. 4 and 5 are met, the tests pass, and the task succeeds. ... In-context Learning: mismatching string or integer values specified in the model input ... Instruction Following: misunderstanding or missing the feature requested in test cases ... The key differentiator here is that top LLMs meet test instructions, where others fail instructions."

    TDD success is defined as passing the given tests. The error taxonomy then classifies a large share of test failures as 'Instruction Following' or 'In-context Learning' by definitional labels rather than by an independent capability measure. Counting these labeled failures and concluding that instruction following and in-context learning are the key differentiators is therefore partly a restatement of the benchmark's own success criterion. This is not a full circularity: the TLD experiment and the raw error distribution provide separable evidence that weak models can produce functional code, so the 'coding proficiency is not sufficient' part has independent support. The circularity is confined to the interpretive step that equates test-passing with these two named capabilities.

full rationale

The paper is an empirical benchmark study with no fitted parameters or equations whose outputs are fed back as predictions. The TLD experiment manipulates tests after seeing model outputs, which is a control condition rather than a circular prediction. The only self-citation (WebApp1K leaderboard, Ref. web, 2024) is not load-bearing. The unvalidated GPT-4o-generated tests are a validity and reproducibility concern, but they do not make a derivation circular. The one partially circular element is the error taxonomy in Table 7, where the headline capabilities are defined in terms of the same test-passing behavior the benchmark measures; however, the empirical ranking and the TLD control give the central claim independent content, so the overall circularity is mild.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The report relies on the paper's reported numbers and appendix; we cannot independently verify the benchmark quality or model API runs. The numeric claims depend on inference hyperparameters chosen on a subset, and the validity assumptions are enumerated as axioms.

free parameters (5)
  • temperature = 0.2
    Chosen by grid search on 100 random tasks to maximize pass@1 (Appendix B, Table 14).
  • top_p = 0.8
    Same grid search procedure as temperature (Appendix B, Table 14).
  • top_k = 40
    Fixed value, not tuned in grid search (Appendix B).
  • presence_penalty = 0
    Fixed across all APIs (Appendix B).
  • frequency_penalty = 0
    Fixed across all APIs (Appendix B).
assumptions (3)
  • domain assumption GPT-4o-generated tests for each scenario are correct and unambiguous specifications of the intended feature
    Appendix A describes test generation by GPT-4o with no reported human validation; the ticketSubmission case shows tests can encode nonstandard behavior.
  • domain assumption The 1000 scenarios are effectively unseen by the evaluated models
    Section 1 claims application-layer tasks are untapped, but no decontamination or overlap analysis is provided.
  • domain assumption pass@k computed with n=10 (n=1 for reasoning models) yields stable model rankings
    No confidence intervals are reported; reasoning models were evaluated once per task, so their pass@1 estimates have high variance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Tests as Prompt: A Test-Driven-Development Benchmark for LLM Code Generation." pith.science (2026). https://pith.science/paper/UV3OZGZH

@misc{pith2026250509027,
  author       = {Pith},
  title        = {Pith review of: Tests as Prompt: A Test-Driven-Development Benchmark for LLM Code Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UV3OZGZH}},
  note         = {Machine review of arXiv:2505.09027}
}
read the original abstract

We introduce WebApp1K, a novel benchmark for evaluating large language models (LLMs) in test-driven development (TDD) tasks, where test cases serve as both prompt and verification for code generation. Unlike traditional approaches relying on natural language prompts, our benchmark emphasizes the ability of LLMs to interpret and implement functionality directly from test cases, reflecting real-world software development practices. Comprising 1000 diverse challenges across 20 application domains, the benchmark evaluates LLMs on their ability to generate compact, functional code under the constraints of context length and multi-feature complexity. Our findings highlight instruction following and in-context learning as critical capabilities for TDD success, surpassing the importance of general coding proficiency or pretraining knowledge. Through comprehensive evaluation of 19 frontier models, we reveal performance bottlenecks, such as instruction loss in long prompts, and provide a detailed error analysis spanning multiple root causes. This work underscores the practical value of TDD-specific benchmarks and lays the foundation for advancing LLM capabilities in rigorous, application-driven coding scenarios.

Figures

Figures reproduced from arXiv: 2505.09027 by the authors.

Figure 1
Figure 1. Incremental TDD by Human On the other hand, LLM approaches TDD in a transactional manner ( [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Transactional TDD by LLM The goal of this paper is to identify key LLM capabilities for TDD task success, in other words, what cause LLMs to fail at the task. To get these insights, we need a greenfield benchmark, with following considerations. • Untapped Semantic Space: The key for an LLM to succeed on TDD is to capture and understand the in￾structions from tests, instead of reciting from pretrain￾ing knowledge. Un… view at source ↗
Figure 3
Figure 3. Failures per problem they are difficult. 3.3. Error Types We study error logs and find LLMs make seven types of errors, coded to A through G. They are summarized in Tab. 7. The verbatim errors are the original error messages or codes captured by the log. Each of them is broadly scoped to contain a wide array of behaviors. However, in the context of our benchmark, we find all verbatim errors are projected to a narrow… view at source ↗
Figures from the paper (12 more)
Figure 6
Figure 6. Figure 6: TLD by LLM [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Comparison of frontend and backend validation [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: LOC distribution by model (bimodal) Next, we use violin charts to visualize LOC distribution of each model. The distributions are either bimodal or unimodal, and they are collected in [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]
Figure 9
Figure 9. Figure 9: LOC distribution by model (unimodal) E.2. Impact of Success/Failure To get more insights, we search for statistical distinction between successful model outputs and failed outputs. In [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]
Figure 10
Figure 10. Figure 10: LOC distribution by model of high pass@1: success vs failure [PITH_FULL_IMAGE:figures/full_fig_p022_10.png]
Figure 11
Figure 11. Figure 11: LOC distribution by model of low pass@1: success vs failure [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]
Figure 12
Figure 12. Figure 12: LOC distribution by model: success and failure [PITH_FULL_IMAGE:figures/full_fig_p025_12.png]
Figure 13
Figure 13. Figure 13: LOC distribution by applications: unimodal [PITH_FULL_IMAGE:figures/full_fig_p026_13.png]
Figure 14
Figure 14. Figure 14: LOC distribution by applications: multimodal [PITH_FULL_IMAGE:figures/full_fig_p027_14.png]
Figure 15
Figure 15. Figure 15: LOC Distribution by Application: Success vs Failure [PITH_FULL_IMAGE:figures/full_fig_p029_15.png]
Figure 17
Figure 17. Figure 17: Failures per problem by application 31 [PITH_FULL_IMAGE:figures/full_fig_p031_17.png]
Figure 18
Figure 18. Figure 18: Errors by applications 32 [PITH_FULL_IMAGE:figures/full_fig_p032_18.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI

    cs.SE 2025-05 conditional novelty 3.0 of 10

    A qualitative taxonomy positions vibe coding and agentic coding as complementary paradigms rather than rivals in AI-assisted software development.

Reference graph

Works this paper leans on

52 extracted references · 14 canonical work pages · cited by 1 Pith paper

  1. [1]

    https://fireship.io/, 2017

    Fireship. https://fireship.io/, 2017

    work page 2017

  2. [2]

    https://huggingface.co/spaces/onekq-ai/WebApp1K-models-leaderboard, 2024

    Webapp1k. https://huggingface.co/spaces/onekq-ai/WebApp1K-models-leaderboard, 2024

    work page 2024

  3. [3]

    Fullstack React: The Complete Guide to ReactJS and Friends

    Accomazzo, A., Murray, N., and Lerner, A. Fullstack React: The Complete Guide to ReactJS and Friends. Fullstack.io, 2017. ISBN 9780991344628. URL https://books.google.com/books?id=ppjUtAEACAAJ

    work page 2017

  4. [4]

    W., Tian, Z., and Barber, D

    Anthony, T. W., Tian, Z., and Barber, D. Thinking fast and slow with deep learning and tree search. In Neural Information Processing Systems, 2017

    work page 2017

  5. [5]

    Program synthesis with large language models

    Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., and Sutton, C. Program synthesis with large language models. https://arxiv.org/abs/2108.07732, 2021

    arXiv 2021

  6. [6]

    Test Driven Development: By Example

    Beck, K. Test Driven Development: By Example. Addison-Wesley Signature Series (Beck). Pearson Education, 2022. ISBN 9780137585236. URL https://books.google.com/books?id=zNnPEAAAQBAJ

    work page 2022

  7. [7]

    Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Ch...

    arXiv 2021

  8. [8]

    Batch prompting: Efficient inference with large language model apis, 2023

    Cheng, Z., Kasai, J., and Yu, T. Batch prompting: Efficient inference with large language model apis, 2023. URL https://arxiv.org/abs/2301.08721

    arXiv 2023

Show all 52 references
  1. [9]

    K., Hong, P., Bing, L., and Poria, S

    Chia, Y. K., Hong, P., Bing, L., and Poria, S. Instructeval: Towards holistic evaluation of instruction-tuned large language models, 2023. URL https://arxiv.org/abs/2306.04757

    2023 arXiv

  2. [10]

    DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., Zhang, X., Yu, X., Wu, Y., Wu, Z. F., Gou, Z., Shao, Z., Li, Z., Gao, Z., Liu, A., Xue, B., Wang, B., Wu, B., Feng, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., D...

    2025 arXiv

  3. [11]

    A survey on in-context learning, 2024

    Dong, Q., Li, L., Dai, D., Zheng, C., Ma, J., Li, R., Xia, H., Xu, J., Wu, Z., Liu, T., Chang, B., Sun, X., Li, L., and Sui, Z. A survey on in-context learning, 2024. URL https://arxiv.org/abs/2301.00234

    2024 arXiv

  4. [12]

    Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation

    Du, X., Liu, M., Wang, K., Wang, H., Liu, J., Chen, Y., Feng, J., Sha, C., Peng, X., and Lou, Y. Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation. https://arxiv.org/abs/2308.01861, 2023

    2023 arXiv

  5. [13]

    Text-to-sql empowered by large language models: A benchmark evaluation, 2023

    Gao, D., Wang, H., Li, Y., Sun, X., Qian, Y., Ding, B., and Zhou, J. Text-to-sql empowered by large language models: A benchmark evaluation, 2023. URL https://arxiv.org/abs/2308.15363

    2023 arXiv

  6. [14]

    F., Liang, P., and Manning, C

    Hewitt, J., Liu, N. F., Liang, P., and Manning, C. D. Instruction following without instruction tuning, 2024. URL https://arxiv.org/abs/2409.14254

    2024 arXiv

  7. [15]

    K., Hao, J., Song, L., Xu, Y., Yang, J., Liu, J

    Huang, S., Cheng, T., Liu, J. K., Hao, J., Song, L., Xu, Y., Yang, J., Liu, J. H., Zhang, C., Chai, L., Yuan, R., Zhang, Z., Fu, J., Liu, Q., Zhang, G., Wang, Z., Qi, Y., Xu, Y., and Chu, W. Opencoder: The open cookbook for top-tier code large language models, 2024. URL https:...

    2024 arXiv

  8. [16]

    Qwen2.5-coder technical report, 2024

    Hui, B., Yang, J., Cui, Z., Yang, J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., Lu, K., Dang, K., Fan, Y., Zhang, Y., Yang, A., Men, R., Huang, F., Zheng, B., Miao, Y., Quan, S., Feng, Y., Ren, X., Ren, X., Zhou, J., and Lin, J. Qwen2.5-coder technical report, 2024. URL ...

    2024 arXiv

  9. [17]

    T., Karkevandi, M

    Islam, N. T., Karkevandi, M. B., and Najafirad, P. Code security vulnerability repair using reinforcement learning with large language models, 2024 a . URL https://arxiv.org/abs/2401.07031

    2024 arXiv

  10. [18]

    T., Khoury, J., Seong, A., Karkevandi, M

    Islam, N. T., Khoury, J., Seong, A., Karkevandi, M. B., Parra, G. D. L. T., Bou-Harb, E., and Najafirad, P. Llm-powered code vulnerability repair with reinforcement learning and semantic reward, 2024 b . URL https://arxiv.org/abs/2401.03374

    2024 arXiv

  11. [19]

    Coarse-tuning models of code with reinforcement learning feedback, 2023

    Jain, A., Adiole, C., Chaudhuri, S., Reps, T., and Jermaine, C. Coarse-tuning models of code with reinforcement learning feedback, 2023. URL https://arxiv.org/abs/2305.18341

    2023 arXiv

  12. [20]

    Followbench: A multi-level fine-grained constraints following benchmark for large language models, 2024

    Jiang, Y., Wang, Y., Zeng, X., Zhong, W., Li, L., Mi, F., Shang, L., Jiang, X., Liu, Q., and Wang, W. Followbench: A multi-level fine-grained constraints following benchmark for large language models, 2024. URL https://arxiv.org/abs/2310.20410

    2024 arXiv

  13. [21]

    E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K

    Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. R. SWE -bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024

    2024

  14. [22]

    W., Fried, D., Wang, S., and Yu, T

    Lai, Y., Li, C., Wang, Y., Zhang, T., Zhong, R., Zettlemoyer, L., tau Yih, S. W., Fried, D., Wang, S., and Yu, T. Ds-1000: A natural and reliable benchmark for data science code generation. https://arxiv.org/abs/2211.11501, 2022

    2022 arXiv

  15. [23]

    B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., Liu, Q., Zheltonozhskii, E., Zhuo, T

    Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., Liu, Q., Zheltonozhskii, E., Zhuo, T. Y., Wang, T., Dehaene, O., Davaadorj, M., Lamy-Poirier, J., Monteiro, J., Shliazhko, O., Gontier, N., Meade, N., Zebaze, A., Yee...

    2023 arXiv

  16. [24]

    Li, Z., Xu, Z., Han, L., Gao, Y., Wen, S., Liu, D., Wang, H., and Metaxas, D. N. Implicit in-context learning, 2024. URL https://arxiv.org/abs/2405.14660

    2024 arXiv

  17. [25]

    Let's verify step by step, 2023

    Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step, 2023. URL https://arxiv.org/abs/2305.20050

    2023 arXiv

  18. [26]

    F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P

    Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12, 2024

    2024

  19. [27]

    Large language model instruction following: A survey of progresses and challenges, 2024

    Lou, R., Zhang, K., and Yin, W. Large language model instruction following: A survey of progresses and challenges, 2024. URL https://arxiv.org/abs/2303.10475

    2024 arXiv

  20. [28]

    Lozhkov, A., Li, R., Allal, L. B., Cassano, F., Lamy-Poirier, J., Tazi, N., Tang, A., Pykhtar, D., Liu, J., Wei, Y., Liu, T., Tian, M., Kocetkov, D., Zucker, A., Belkada, Y., Wang, Z., Liu, Q., Abulkhanov, D., Paul, I., Li, Z., Li, W.-D., Risdal, M., Li, J., Zhu, J., Zhuo, T. ...

    2024 arXiv

  21. [29]

    Mathews, N. S. and Nagappan, M. Test-driven development for code generation, 2024. URL https://arxiv.org/abs/2402.13521

    2024 arXiv

  22. [30]

    React framework

    Meta. React framework. https://reactjs.org/, 2013

    2013

  23. [31]

    Mdn web docs

    Mozilla. Mdn web docs. https://https://developer.mozilla.org/, 2005

    2005

  24. [32]

    Testing llms on code generation with varying levels of prompt specificity, 2023

    Murr, L., Grainger, M., and Gao, D. Testing llms on code generation with varying levels of prompt specificity, 2023. URL https://arxiv.org/abs/2311.07599

    2023 arXiv

  25. [33]

    Introducing swe-bench verified

    OpenAI. Introducing swe-bench verified. https://openai.com/index/introducing-swe-bench-verified/, 2024

    2024

  26. [34]

    and Sullivan, A

    Piya, S. and Sullivan, A. Llm4tdd: Best practices for test driven development using large language models, 2023. URL https://arxiv.org/abs/2312.04687

    2023 arXiv

  27. [35]

    Infobench: Evaluating instruction following ability in large language models, 2024

    Qin, Y., Song, K., Hu, Y., Yao, W., Cho, S., Wang, X., Wu, X., Liu, F., Liu, P., and Yu, D. Infobench: Evaluating instruction following ability in large language models, 2024. URL https://arxiv.org/abs/2401.03601

    2024 arXiv

  28. [36]

    Mastering chess and shogi by self-play with a general reinforcement learning algorithm, 2017

    Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., Lillicrap, T., Simonyan, K., and Hassabis, D. Mastering chess and shogi by self-play with a general reinforcement learning algorithm, 2017. URL https...

    2017 arXiv

  29. [37]

    K., Moskovitz, T., Hill, F., Chan, S

    Singh, A. K., Moskovitz, T., Hill, F., Chan, S. C. Y., and Saxe, A. M. What needs to go right for an induction head? a mechanistic study of in-context learning circuits and their formation, 2024. URL https://arxiv.org/abs/2404.07129

    2024 arXiv

  30. [38]

    Multi-task inference: Can large language models follow multiple instructions at once?, 2024

    Son, G., Baek, S., Nam, S., Jeong, I., and Kim, S. Multi-task inference: Can large language models follow multiple instructions at once?, 2024. URL https://arxiv.org/abs/2402.11597

    2024 arXiv

  31. [39]

    Reinforcement learning from automatic feedback for high-quality unit test generation, 2023

    Steenhoek, B., Tufano, M., Sundaresan, N., and Svyatkovskiy, A. Reinforcement learning from automatic feedback for high-quality unit test generation, 2023. URL https://arxiv.org/abs/2310.02368

    2023 arXiv

  32. [40]

    Hypothesis search: Inductive reasoning with language models

    Wang, R., Zelikman, E., Poesia, G., Pu, Y., Haber, N., and Goodman, N. Hypothesis search: Inductive reasoning with language models. In The Twelfth International Conference on Learning Representations, 2024

    2024

  33. [41]

    A., Khashabi, D., and Hajishirzi, H

    Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language models with self-generated instructions, 2023. URL https://arxiv.org/abs/2212.10560

    2023 arXiv

  34. [42]

    Y., Guu, K., Yu, A

    Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned language models are zero-shot learners, 2022. URL https://arxiv.org/abs/2109.01652

    2022 arXiv

  35. [43]

    M., Raghunathan, A., Liang, P., and Ma, T

    Xie, S. M., Raghunathan, A., Liang, P., and Ma, T. An explanation of in-context learning as implicit bayesian inference, 2022. URL https://arxiv.org/abs/2111.02080

    2022 arXiv

  36. [44]

    C.-J., Zhang, T., Patil, S

    Yan, F., Mao, H., Ji, C. C.-J., Zhang, T., Patil, S. G., Stoica, I., and Gonzalez, J. E. Berkeley function calling leaderboard. https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html, 2024

    2024

  37. [45]

    Codereval: A benchmark of pragmatic code generation with generative pre-trained models, 2023

    Yu, H., Shen, B., Ran, D., Zhang, J., Zhang, Q., Ma, Y., Liang, G., Li, Y., Xie, T., and Wang, Q. Codereval: A benchmark of pragmatic code generation with generative pre-trained models, 2023

    2023

  38. [46]

    Zelikman, E., Wu, Y., Mu, J., and Goodman, N. D. Star: self-taught reasoner bootstrapping reasoning with reasoning. In Proceedings of the 36th International Conference on Neural Information Processing Systems, 2022

    2022

  39. [47]

    Zelikman, E., Harik, G., Shao, Y., Jayasiri, V., Haber, N., and Goodman, N. D. Quiet-star: Language models can teach themselves to think before speaking, 2024. URL https://arxiv.org/abs/2403.09629

    2024 arXiv

  40. [48]

    A survey on self-play methods in reinforcement learning, 2024

    Zhang, R., Xu, Z., Ma, C., Yu, C., Tu, W.-W., Huang, S., Ye, D., Ding, W., Yang, Y., and Wang, Y. A survey on self-play methods in reinforcement learning, 2024. URL https://arxiv.org/abs/2408.01072

    2024

  41. [49]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  42. [50]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  43. [51]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  44. [52]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.