{"id":"b7d0c56a-f098-42f7-8392-03d00edae7d7","arxiv_id":"2505.09027","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"The paper builds a 1000-task test-driven development benchmark for LLMs and concludes that instruction following and in-context learning matter more than general coding proficiency.","lead":"WebApp1K is a new benchmark that gives language models only unit tests as a prompt and asks them to generate React code that passes them. The paper compares 19 models and argues that following coded instructions, not raw coding skill, is what separates strong from weak models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unvalidated GPT-4o-generated tests make 'instruction following' claims untrustworthy: if many of the 1000 test pairs encode idiosyncratic expectations, the benchmark measures test-guessing rather than TDD skill.","rationale":"The headline contribution of the paper is a capability claim, not just a leaderboard: TDD success is driven by instruction following and in-context learning, not general coding proficiency. That claim rests on interpreting test mismatches as instruction-following errors, which in turn assumes the GPT-4o-generated tests are correct, complete, and unambiguous specifications. The paper's own Appendix D.1 provides a concrete counterexample: a model explicitly reasons that the test may be written incorrectly but says it must pass it anyway, and the authors score this as a failure of instruction following. Without a validity check on the generated tests, the error taxonomy and the TLD experiment cannot distinguish 'model ignored a clear coded instruction' from 'model made a reasonable choice the test did not anticipate.' This is a validity threat, not a stylistic objection. I agree with the reader's weakest_assumption; it is the most load-bearing premise. I keep the verdict conditional: the benchmark may be useful and the ranking data are internally consistent, but the central capability claim should not be accepted until the gold tests are validated. I also note that the Section 4 'input context length is the main bottleneck' claim is separately confounded because duo-feature tasks double both context length and feature count; a controlled context-length experiment would be needed even if test quality were confirmed. The primary concern, however, is the unvalidated ground truth.","tokens_in":26309,"tokens_out":6258,"duration_ms":55653,"concrete_test":"Select a random sample of 100 WebApp1K tasks. Give two independent human developers only the original human scenario sentence (not the GPT-4o tests) and ask each to implement the described feature in the React template. Run the benchmark's success/failure tests against both implementations. If a meaningful fraction (e.g., >20%) of reasonable human implementations fail, the tests encode idiosyncratic expectations, directly undermining the 'instruction following' interpretation. A complementary check: have three human experts independently rate each sampled test pair for ambiguity/correctness and test whether expert-rated ambiguous tasks have disproportionately high failure rates across models.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Appendix A states that after humans wrote 1000 scenario sentences, 'we prompt GPT-4o to generate a success test and failure test for each scenario'; no human verification of test correctness or ambiguity is reported. Appendix D.1's ticketSubmission task shows the danger concretely: the test expects backend validation ('Title is required' after a 400 response), while common practice is frontend validation, and the paper reports that all non-reasoning models are 'misled' into frontend validation. DeepSeek R1's own trace says: 'Maybe the test is written incorrectly, but I have to write the component to pass the tests as given.' This is an explicit admission that the gold tests can contradict reasonable implementation choices. If a substantial fraction of the 1000 task pairs are similarly idiosyncratic or under-specified, then the central claim that instruction following and in-context learning, not coding proficiency, determine TDD success is not established: failures attributed to 'instruction following' (error types B, C, D, G) could instead reflect the impossibility of inferring a single intended behavior from an ambiguous or nonstandard test. The TLD experiment (Tab. 8) only shows that code passes after tests are modified to match it; it does not show the original tests were the correct specification. Thus the benchmark's ground truth is the load-bearing assumption and it is currently unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces WebApp1K, a benchmark of 1000 test-driven development (TDD) tasks for React web applications, in which each task consists of a success and a failure unit test that serve as both the prompt and the verification for code generation. The authors evaluate 19 LLMs and report pass@k scores, an error taxonomy of seven failure types, a 'TLD' experiment in which tests are modified to match generated code, and a duo-feature variant in which two single-feature tasks are merged into one prompt of roughly double length. The paper's central claims are that instruction following and in-context learning are the decisive capabilities for TDD success, more important than general coding proficiency or pretraining knowledge, and that input context length is the main bottleneck to TDD success.","tokens_in":26599,"tokens_out":9608,"duration_ms":79696,"significance":"The WebApp1K benchmark, if carefully validated, would be a useful new evaluation resource for studying code generation from formal test specifications rather than natural language, and the paper has several concrete strengths: the scale of the benchmark (1000 tasks, 19 models), the clear task formulation, the public leaderboard, the structured error analysis, and the explicit consideration of reasoning models. The qualitative observations in Appendix D, such as the ticketSubmission example and the duo-feature export-name trap, are interesting and testable. However, the central interpretative claims are not yet supported: the unvalidated GPT-4o-generated tests, the confounded duo-feature design, and the nonstandard TLD operationalization mean the paper's headline conclusions about capability differentiators and context-length bottleneck go beyond the evidence. These issues are fixable, but they require additional validation experiments and more careful framing.","major_comments":[{"comment":"The benchmark's ground truth consists of GPT-4o-generated unit tests with no reported human verification of correctness or unambiguity (Appendix A: 'we prompt GPT-4o to generate a success test and failure test for each scenario'). Appendix D.1's ticketSubmission task shows the risk concretely: the test requires backend validation ('Title is required' after a 400 response), while common practice is frontend validation, and the paper reports that all non-reasoning models are 'misled' into frontend validation; DeepSeek-R1's trace even states 'Maybe the test is written incorrectly, but I have to write the component to pass the tests as given.' If a substantial fraction of the 1000 test pairs are similarly idiosyncratic or under-specified, then failures attributed in Table 7 to 'instruction following' (types B, C, D, G) or 'in-context learning' may instead reflect the impossibility of inferring the intended behavior from an ambiguous test. The TLD experiment (§3.6, Table 8) does not resolve this: modifying tests to match the code only shows proximity between code and test, not that the original test was the correct specification. The central claim that instruction following and in-context learning are the key capabilities for TDD success therefore needs support from a validation study of the test set, e.g., human annotation of a sample of tasks for ambiguity and consensus, and possibly the removal or replacement of non-consensus tasks.","section":"Appendix A; §2.2; Appendix D.1; Table 7"},{"comment":"The duo-feature experiment changes both the prompt length and the number of features to be implemented, so the observed drop in pass@1 cannot be attributed to input context length as claimed ('the input context length [is] the main bottleneck to TDD success rate'). To substantiate this claim, the experiment needs a control that increases context length without adding functional requirements (e.g., additional tests for the same feature, or a longer prompt with the same tests) or, conversely, implements two features in two separate single-feature prompts of the same total length. Without such a control, the 'instruction loss' interpretation in §4.2 is not uniquely supported.","section":"§4, Table 9"},{"comment":"The TLD experiment is not a test-last development baseline in the sense defined in §1 and Table 1. In §3.6, TLD is operationalized as modifying the failed tests to accommodate the generated code ('we modify the violated tests to accommodate the verbatim code output'). This is a test-adaptation procedure, not the natural-language prompt-based TLD task defined earlier. Consequently, the comparison between TDD and TLD pass rates in Table 8 does not directly support the claim that 'LLMs of low TDD success rate have high success rate on sibling TLD tasks' as the term TLD is used in the introduction. The experiment also only includes models with low TDD pass rates, so the inference about capability differentiators is limited.","section":"§3.6, Table 8"},{"comment":"The error taxonomy is presented as a 'conjecture' about connections to model capabilities, yet the abstract and §3.5 treat it as evidence that 'instruction following and in-context learning are critical capabilities... surpassing the importance of general coding proficiency or pretraining knowledge' and that 'all LLMs possess the same knowledge and capabilities.' The mapping from verbatim errors to root causes and capabilities is not validated (e.g., by human annotation, inter-annotator agreement, or an ablation). The claim that all models 'have the same inherent vulnerabilities' (Section 3.5) is an unsupported generalization from error distributions that could be confounded by test quality (see the first major comment). At minimum, the paper should report how the taxonomy was applied and its reliability.","section":"§3.3, Table 7; §3.5"}],"minor_comments":[{"comment":"The abstract and Section 6 refer to '18 frontier LLMs' while Table 6 reports 19 models; please reconcile the count.","section":"Abstract vs. §6"},{"comment":"There are numerous typos, e.g., 'essemble' (Section 1), 'bewteen' (Section 3.5), 'Performnaces' (Section 4.1), 'valiation' (Appendix D.1), 'instructoin' (Section 5.4), and 'Unfortunatelly' (Appendix D.1).","section":"Throughout"},{"comment":"The statement that 'since the value of pass@k asymptotically increases with k, there is no doubt that the top reasoning models lead other LLMs by an obvious gap' is not justified; pass@k curves of different models can cross, and reasoning models were measured with n=1 while other models used n=10, so their pass@1 estimates have larger variance.","section":"§3.1"},{"comment":"The parameter-tuning table reports pass@1 values on a 100-task subset but the chosen parameters are applied to the full benchmark; please clarify whether the tuning set is disjoint from the evaluation set to avoid optimism in the reported results.","section":"Appendix B, Table 14"},{"comment":"The raw-format duo-feature experiment (Table 19) is dramatic, but the explanation that reasoning models commit to the wrong first planning step is anecdotal; the paper would benefit from quantifying how often the first reasoning step matches the eventual (wrong) implementation.","section":"Appendix D.2, Table 21"},{"comment":"References contain malformed URLs, e.g., 'https://https://developer.mozilla.org/', and the WebApp1K leaderboard link points to a leaderboard rather than to the benchmark dataset; please provide a stable data release with an explicit license.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solo-authored preprint on arXiv with no linked code repository or dataset release beyond a leaderboard URL. Given the load-bearing concerns about ground-truth validation, I recommend requiring the authors to submit the benchmark artifacts for review as part of the revision. The lack of human verification of the GPT-4o-generated tests is the most serious issue; even a modest human audit of a random sample (e.g., 100 tasks) with reported agreement rates would substantially strengthen the paper. The duo-feature confound is also fixable with a control experiment. I do not recommend rejection because the benchmark idea is valuable and the evaluation data, for all its limitations, are presented transparently."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real contribution as a benchmark resource, but the interpretive claims outrun the evidence. I'd send it to review and push for major revisions, not desk rejection.\n\nWhat's new and good: it is, to my knowledge, the first dedicated TDD benchmark at 1000-task scale, with tests serving as both prompt and verification. The 20-domain structure, paired success/failure tests, 19-model pass@k results, and the seven-type error taxonomy are useful scaffolding for the community. The appendices are unusually thorough: prompt experiments, LOC distributions, per-application error analysis, even a bias audit. The TLD experiment is an honest attempt to see whether failures reflect dysfunctional code or merely test mismatch; it shows that code often passes after tests are modified to match it. That is interesting, but it is not evidence for the paper's capability claim.\n\nThe soft spots, in order of severity. First and most important: the ground truth of the tests is the load-bearing assumption, and it is the least validated part. Appendix A says GPT-4o generated all success/failure tests from one-sentence human scenarios, with no reported human check for correctness or ambiguity. The ticketSubmission deep dive shows the risk concretely: the test expects backend validation, while common practice is frontend validation, and all non-reasoning models fall into the trap. DeepSeek R1's own trace says 'Maybe the test is written incorrectly, but I have to write the component to pass the tests as given.' If many of the 1000 test pairs are similarly idiosyncratic, the benchmark measures test-guessing or arbitrary API quirks, not TDD skill. That directly undermines the attribution of errors to instruction following (types B, C, D, G).\n\nSecond, the duo-feature experiment confounds context length with feature count; doubling tests doubles both, with no redundant-test control. So the claim that 'input context length is the main bottleneck' is not established. Third, there is no natural-language-prompted baseline on the same tasks, which is needed to support the claim that coded instruction following surpasses general coding proficiency. Fourth, pass@k estimates lack confidence intervals; for a 19-model comparison, that is a real omission, though minor relative to the first two.\n\nCitation pattern looks fair; they cite Self-Instruct for the pipeline and the TDD-prompt literature for related work. No fitted parameters, no circular derivation. The fix is doable: release the dataset and harness, human-audit a sample of tests for correctness and ambiguity, add a natural-language control, and deconfound length from feature count. As it stands, the benchmark deserves attention, but the capability conclusions need substantially more support before I would trust or cite them.","headline":"WebApp1K is a genuinely new and potentially useful TDD benchmark, but the paper's headline claims about instruction following and context length rest on unvalidated GPT-generated tests and confounded comparisons.","tokens_in":27116,"tokens_out":2251,"would_cite":false,"duration_ms":24432,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that on test-driven coding tasks, instruction following and in-context learning matter more than general programming ability, and that prompt length is the main bottleneck affecting all models.","keywords":["test-driven development","LLM code generation","benchmark","instruction following","in-context learning","context length","WebApp1K","pass@k"],"falsifier":"Randomly sample 100 scenarios, give independent human developers the one-sentence feature description without the tests, and run their implementations against the original test pairs; if many reasonable implementations fail because the tests encode idiosyncratic expectations (for example, requiring backend validation where frontend validation is standard practice), then the benchmark would be measuring test-convention guessing rather than TDD capability.","tokens_in":26122,"feed_emoji":"🧪","tokens_out":6606,"duration_ms":59895,"temperature":0.7,"pith_summary":"This paper introduces WebApp1K, a benchmark of 1000 test-driven development (TDD) tasks in which each prompt is a pair of unit tests (one success, one failure) and the model must generate a single web-app component that passes both. The paper's central claim is that on such tasks, instruction following and in-context learning are the decisive capabilities, more so than general coding proficiency or pretraining knowledge. The evidence is that models with low TDD pass rates jump to high pass rates when the tests are retrofitted to their generated code, and that all models' performance drops sharply when the prompt doubles in length. The authors conclude that the practical bottleneck for LLMs in TDD is faithfully absorbing all coded expectations, especially in longer prompts.","feed_headline":"Tests-as-prompt benchmark: instruction-following beats coding skill","feed_subtitle":"Across 1,000 test-driven web-app tasks, obeying coded specs matters more than raw programming strength.","key_machinery":"The central object is the TDD task itself: a prompt composed entirely of test code, which simultaneously specifies the feature and verifies the solution. Each WebApp1K scenario pairs one success test and one failure test for a small web-app feature, and the model must emit a single component passing both. The argument's load-bearing comparison is the TLD experiment, in which failing tests are edited—without changing their structure—to match the generated code; the resulting pass-rate jump isolates instruction following from coding ability. The duo-feature upgrade, which doubles the prompt to about 1K tokens, supplies the context-length bottleneck evidence. The paper also builds a seven-category error taxonomy and shows that 93% of error logs contain a single error or two errors of the same type, tracing them to capabilities such as instruction following, in-context learning, and pretraining knowledge.","core_discovery":"On its own terms, the paper establishes that LLM performance on TDD code generation is governed by the model's ability to extract and obey the specifications encoded in test code, not by its raw programming strength. WebApp1K contains 1000 scenarios across 20 application domains; each scenario is a small web-app component tested by a success/failure unit-test pair. Across 19 frontier models, pass@1 ranges from about 0.07 to 0.95. When violated tests are minimally modified to match the code the model actually wrote (the TLD experiment), the weakest models' pass@1 more than doubles, showing their code is functional but non-conforming. A duo-feature version with four tests per task cuts pass@1 for all tested models, and a case study shows a strong reasoning model failing only the last of four coded instructions, which the authors attribute to instruction loss.","pith_inferences":["If the central claim holds, then the quality and ambiguity of the test pairs become the main validity risk: because the tests are generated by a single model without human verification, part of what is being measured may be the ability to predict that model's peculiar conventions, such as preferring backend validation over the more common frontend validation.","A direct testable extension would be to vary the position of the last test in the prompt; if attention decay is the cause, moving that test earlier should raise pass@1 under the duo-feature format.","The TDD-versus-TLD gap could be used as a general \"instruction-following quotient\" on any code-generation benchmark, giving a cheap way to separate specification-adherence from coding skill.","Multi-turn TDD, where tests are added incrementally and the model sees failures, may be a more practical and less bottleneck-prone setting than the single-shot transaction used here."],"forward_implications":["Models ranked high on conventional coding benchmarks can rank low on TDD tasks, so TDD-specific evaluation measures a distinct capability that should be tracked separately.","Improving instruction following and in-context learning for code-based instructions should lift TDD pass rates more than additional algorithm or syntax training.","Doubling the number of test cases roughly halves many models' pass@1, so prompt length and instruction density are first-order constraints for practical use of LLMs in TDD.","Since 93% of error logs contain one or two same-type errors, targeted fixes to a small set of root causes could recover most failures.","The benchmark's construction recipe—human-authored scenarios plus model-generated tests—offers a low-cost way to build large TDD benchmarks for other application domains."],"supporting_citations":[{"why":"Defines pass@k and the natural-language code-generation benchmark paradigm that WebApp1K contrasts with as test-last development.","marker":"(Chen et al., 2021)"},{"why":"Defines test-driven development and the principle that tests are the system specification, which motivates the benchmark's design.","marker":"(Beck, 2022)"},{"why":"Provides the Self-Instruct methodology that WebApp1K follows when generating the 1000 test pairs from human-authored scenarios.","marker":"(Wang et al., 2023)"},{"why":"Cited as the suspected root cause of instruction loss in long prompts, supporting the claim that input context length is the main bottleneck.","marker":"(Liu et al., 2024)"},{"why":"Supplies SWE-bench, the problem-solving benchmark that the paper contrasts as partial TDD where tests are only part of the context.","marker":"(Jimenez et al., 2024)"},{"why":"Represents prior work on TDD in LLM coding whose benchmarking focus this paper distinguishes itself from.","marker":"(Mathews & Nagappan, 2024)"}],"fun_headline_variants":["Test-as-prompt: Following instructions, not coding skill","Weak models pass if tests match their code","LLM coding: obey the tests, not just write code","WebApp1K: Instruction loss hurts, not insufficient code","Tests-as-prompt: The instruction-following factor wins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim stands on the assumption that the success/failure test pairs generated by a large language model for each of the 1,000 scenarios faithfully and unambiguously encode the intended feature, since the paper reports no human verification that the tests are correct or unambiguous.","fun_headline_variants_meta":{"raw":{"variants":["Test-as-prompt: Following instructions, not coding skill","Weak models pass if tests match their code","LLM coding: obey the tests, not just write code","WebApp1K: Instruction loss hurts, not insufficient code","Tests-as-prompt: The instruction-following factor wins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000206,"raw_usage":{"total_tokens":1377,"prompt_tokens":903,"completion_tokens":474,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":393}},"tokens_in":519,"tokens_out":474,"duration_ms":5017,"temperature":1.0,"reasoning_tokens":393,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:41:40.747027+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Randomly sample 100 scenarios, give independent human developers the one-sentence feature description without the tests, and run their implementations against the original test pairs; if many reasonable implementations fail because the tests encode idiosyncratic expectations (for example, requiring backend validation where frontend validation is standard practice), then the benchmark would be measuring test-convention guessing rather than TDD capability.","supporting_citations":[{"cited_title":"Test Driven Development: By Example","cited_arxiv_id":null,"evidence_quote":"Defines test-driven development and the principle that tests are the system specification, which motivates the benchmark's design."},{"cited_title":"F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P","cited_arxiv_id":null,"evidence_quote":"Cited as the suspected root cause of instruction loss in long prompts, supporting the claim that input context length is the main bottleneck."},{"cited_title":"E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K","cited_arxiv_id":null,"evidence_quote":"Supplies SWE-bench, the problem-solving benchmark that the paper contrasts as partial TDD where tests are only part of the context."}],"review_version":1}