{"id":"9ddd1282-1cd9-480a-bfb2-8b5b250bcdf8","arxiv_id":"2502.01619","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"UTGen trains LLMs to generate error-revealing unit tests with correct expected outputs, and UTDebug uses those tests with test-time scaling and backtracking to improve automated debugging and code selection.","lead":"A new training method called UTGen teaches LLMs to write unit tests that both expose bugs in faulty code and predict the correct output, and a companion pipeline called UTDebug uses these tests to debug code. The approach improves debugging accuracy on coding benchmarks and even helps a frontier model like GPT-4o when the tests come from a smaller model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No decontamination check: Tulu-3 SFT training data may contain HumanEval/MBPP+ problems, potentially inflating UTGen's intrinsic and debugging gains.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the absence of decontamination between the Tulu-3 training data and the evaluation benchmarks. This is the most serious threat to the paper's central claim because it directly undermines the generalization story. If HumanEval/MBPP+ problems appear in UTGen's training data, the reported improvements in unit-test quality and downstream debugging could be largely memorization rather than learning to generate useful tests for unseen tasks. The paper does not provide any decontamination analysis, and these benchmarks are common enough in web corpora that the risk is nontrivial. I agree with the reader that this warrants a conditional verdict rather than rejection: the method and empirical pipeline are well-executed, and the concern is a missing robustness check rather than a demonstrated internal inconsistency. The abstract's 7.59% figure and the 12.35% comparison issue are real but secondary; they affect precision of reporting, not the core mechanism. The proposed overlap analysis directly tests the key assumption and would settle whether the concern lands.","tokens_in":88,"tokens_out":2778,"duration_ms":37703,"concrete_test":"Compute exact and fuzzy overlap (normalized n-gram similarity, function-signature match) between the 48.3K filtered Tulu-3 prompts and the HE+Fix, MBPP+Fix, and MBPP+Fix (Hard) problem descriptions and gold solutions. Then re-run the intrinsic evaluation (Table 1) and one downstream debugging condition (e.g., Qwen2.5 32B on MBPP+Fix Hard) after excluding from training all problems with any overlap. If overlap is zero, or if the gains persist on the decontaminated subset, the concern is resolved. If gains vanish on non-overlapping problems, the headline claims require qualification.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Appendix B describes filtering the Tulu-3 SFT mixture for Python functions but reports no decontamination analysis against HumanEval, MBPP+, or EvalPlus, which are exactly the evaluation benchmarks used in Tables 1, 2, 5, and 6. These benchmarks are widely distributed in web corpora and open-source repositories, so a general SFT mixture like Tulu-3 is a priori likely to contain their problem descriptions or gold solutions. This matters directly for the central claim: UTGen is supposed to learn to generate error-revealing inputs and correct outputs from task descriptions. If the training set contains the exact evaluation problems, the model can memorize expected outputs and edge-case inputs, inflating Attack Rate, Output Accuracy, Acc.∩Attack, and downstream debugging pass@1 on those benchmarks without generalizing to unseen programs. The paper's strongest evidence is therefore confounded unless decontamination is demonstrated. The absence of any overlap check or statement about this in the manuscript means the current experiments cannot distinguish genuine generalization from benchmark memorization.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes UTGen, a supervised fine-tuning recipe that teaches LLMs to generate unit-test input-output pairs conditioned on a task description and a buggy program, aiming for both high attack rate and correct expected outputs without access to a gold solution. Training data is bootstrapped from the Tulu-3 SFT mixture by corrupting gold code, sampling failing unit-test inputs, and adding post-hoc rationales whose outputs are grounded by executing the gold code. The authors then introduce UTDebug, a multi-turn debugging pipeline that uses self-consistency over UT outputs and pass-rate-based backtracking to handle noisy generated tests. Experiments cover HumanEvalFix, MBPP+Fix, and a new harder MBPP+ split; the paper reports improved debugging pass@1 across several 7-32B models, best-of-N code selection gains over a reward-model reranker, and improved GPT-4o debugging when UTGen-generated tests from a smaller model are used as feedback.","tokens_in":23858,"tokens_out":6102,"duration_ms":52508,"significance":"If the claims withstand scrutiny, this is a valuable contribution: automated, reference-free unit-test generation with error-revealing inputs and correct outputs is a real bottleneck for LLM debugging and verification. The paper's strengths include its public release of code and datasets, evaluation across multiple model families and datasets, the use of gold-code execution to ground training outputs, and a well-motivated intrinsic metric suite (attack rate, output accuracy, Accuracy∩Attack) whose downstream correlation is explicitly studied in Table 7. The UTDebug mechanisms—self-consistency for output prediction and validation/backtracking against a suite of generated tests—are sensible and are shown to be important in ablation. The consistent direction of results across models and datasets gives the central thesis initial plausibility. However, the headline numbers need correction, and the training-data overlap question must be resolved before the generalization claims can be accepted.","major_comments":[{"comment":"The abstract attributes a 12.35% improvement on MBPP+Fix (Hard) to \"other LLM-based UT generation baselines,\" but Table 2 shows that for Qwen2.5 32B the differences are 4.70% over Prompted (40.59 to 45.29) and 22.35% over Random (22.94 to 45.29); 12.35% is the gain over the No-UT baseline (32.94 to 45.29). The abstract's 7.59% Accuracy∩Attack figure is also not directly traceable to any single model-baseline pair in Table 1 or Table 6; the closest text statement in §5.1 reports 7.65% for Llama3.1, which is the gain over Random, not Prompted. Please state the exact comparison being reported or correct the numbers.","section":"Abstract; §4; §5.2, Table 2"},{"comment":"The UTGen training pipeline filters the Tulu-3 SFT mixture for Python functions but reports no decontamination check against HumanEval, MBPP+, or EvalPlus, which are exactly the evaluation benchmarks used in the paper. These benchmarks are widely distributed in web corpora and open-source repositories, so exact or near-exact overlap with a general SFT mixture is plausible. If the training data contains the evaluation problems or their gold solutions, the model can memorize expected outputs and edge-case inputs, inflating Attack Rate, Output Accuracy, Accuracy∩Attack, and downstream debugging pass@1 without demonstrating generalization to unseen programs. Please add a decontamination analysis for the Tulu-3 subset used in training, report the overlap statistics, and rerun the headline evaluations on a non-overlapping subset if any overlap is found.","section":"Appendix B; §5, Tables 1, 2, 5, 6"},{"comment":"The MBPP+Fix (Hard) split, which carries much of the paper's headline debugging claims, is defined by an arbitrary 50%-95% pass-rate filter on sampled faulty solutions. No sensitivity analysis is provided for these thresholds, and the split is self-constructed. Since the abstract and §5.2 emphasize results on this split, please report how the conclusions vary with the threshold choice or provide a principled justification for the selected range.","section":"§4; Appendix A.1"}],"minor_comments":[{"comment":"The column header \"MPBB+Fix\" appears to be a typo for \"MBPP+Fix.\"","section":"Table 2"},{"comment":"Table 1 reports averages over 3 runs but does not report standard deviations or confidence intervals; given the small absolute gaps between some conditions, please include variance information or specify which differences are stable across runs.","section":"Table 1; §5.1"},{"comment":"The ablation results in Table 8 appear to be single-run numbers; given the large drops when removing test-time scaling, it would be helpful to report significance or repeated-run variance.","section":"Appendix E, Table 8"},{"comment":"There are duplicate reference entries for Guo et al. (2024) and for Muennighoff et al. (2023/2024); please consolidate them.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the Tulu-3 overlap with the evaluation benchmarks; this needs to be addressed head-on. If the authors can provide a clean decontamination analysis and fix the abstract's comparison claims, the paper would be a solid contribution. The self-constructed hard split would also benefit from sensitivity analysis, but that is secondary."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper is worth a look for anyone working on LLM code repair or test generation. The core idea is clean: train a model to generate unit tests that both reveal a bug and carry the correct expected output, then feed those tests into a debugging loop. The trade-off between attack rate and output accuracy is a genuinely useful framing.\n\nWhat the paper does well: the UTGen recipe (perturb gold code, execute to filter failing tests, add rationales) is a practical way to bootstrap training data without needing gold tests at inference. The UTDebug pipeline, with self-consistency for output prediction and backtracking based on a test-suite pass rate, is sensible and ablated carefully. Results are consistent across three 7-8B models, a 32B model, and multiple benchmarks; the public artifact is a real plus. Comparing against CodeT and a trained reward model for best-of-N selection makes the contribution concrete.\n\nThe soft spots are real but manageable. First, the abstract says the 12.35% gain on MBPP+Fix (Hard) is \"over other LLM-based UT generation baselines,\" but that number is versus the No-UT baseline; against the prompted baseline it is about 4.7%. The 7.59% intrinsic improvement is also not directly traceable to Table 1—per-model gains range from roughly 5% to 7%. These are presentational errors, not necessarily fatal, but the headline numbers should be fixed. The more substantive concern is the lack of any decontamination check. Tulu-3 is a broad SFT mixture, and HumanEval/MBPP+ are common in such mixtures; without an overlap analysis, the intrinsic and debugging numbers on those benchmarks could be optimistic. This is a missing robustness check, not evidence of dodgy behavior, but it needs to be addressed before I'd trust the claim of generalization.\n\nMinor point: the \"hard\" split (keeping faulty code that passes 50-95% of tests) is reasonable but arbitrary; a sensitivity analysis on that threshold would strengthen it.\n\nAudience: this is for researchers and engineers working on automated debugging, feedback generation, and unit-test synthesis. It is an empirical systems paper, not a theoretical breakthrough, and it claims appropriate scope.\n\nRecommendation: yes, this deserves peer review—a serious referee can verify the reproducibility and push for the needed decontamination analysis. Fix the abstract numbers and add a training/eval overlap check, and this would be a solid contribution.","headline":"A solid, useful empirical paper on generating unit tests for debugging, but the abstract overstates one comparison and the training-data overlap with evaluation benchmarks is unexamined.","tokens_in":24327,"tokens_out":3233,"would_cite":true,"duration_ms":28013,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Trained test-writers improve LLM debugging by 15 points","keywords":["unit test generation","automated debugging","LLM code repair","attack rate","output accuracy","test-time compute","self-consistency","code generation"],"falsifier":"Run a decontamination scan of UTGen's training data against HumanEval and MBPP+ (for example, fuzzy matching problem descriptions or code tokens to benchmark solutions); if overlaps exist, retrain on the cleaned set and check whether the 7.59% Accuracy ∩ Attack gain and the debugging improvements persist. A large drop toward the zero-shot prompting baseline would show the reported gains came from memorization rather than learned unit-test generation; no overlap with stable numbers would support the generalization claim.","tokens_in":23392,"feed_emoji":"🧪","tokens_out":11253,"duration_ms":81406,"temperature":0.7,"pith_summary":"Unit tests are the standard way to check code, but writing them by hand is expensive, and LLMs usually need a gold solution to know the expected output. This paper claims that a relatively small fine-tuned model can generate unit tests that do both things at once: expose a bug in faulty code and predict the correct output, using only the problem description and the faulty code. The authors introduce UTGen, a training recipe that creates supervision data by intentionally corrupting correct code, and UTDebug, a debugging loop that applies test-time scaling and a validation-and-backtracking step to cope with noisy generated tests. If the claim holds, automated debugging and code selection no longer depend on human-written tests or gold implementations, and even frontier debugging models benefit from cheap, generated test feedback.","feed_headline":"Trained test-writers improve LLM debugging by 15 points","feed_subtitle":"Fine-tuned models generate error-revealing tests with correct outputs, beating reward models at choosing code.","key_machinery":"The machinery is a three-stage data bootstrapping recipe (UTGen) plus a robust inference loop (UTDebug). UTGen starts from gold code solutions, (I) corrupts them into faulty variants, (II) samples unit test inputs that fail on the faulty code while using the gold code to supply the correct output, and (III) relabels each output prediction with a chain-of-thought rationale generated post hoc, producing roughly 30K-70K supervised instances that jointly reward attack rate (how often the test makes faulty code fail) and output accuracy (how often the predicted test output is correct). At inference, UTDebug samples k=8 output predictions per input, keeps the majority vote only if it exceeds 50% of votes, generates n unit tests, uses one failing test as feedback, and accepts a code edit only when the pass rate over the full test suite improves; otherwise it backtracks. The combination is what allows noisy, model-generated tests to serve as reliable feedback for debugging and code selection.","core_discovery":"UTGen teaches LLMs to generate unit test input-output pairs such that the input is valid for the task, the input reveals an error in the given faulty code, and the output matches what a correct solution would return. The paper's central discovery is a trade-off: with zero-shot prompting, tests that are likely to reveal bugs tend to have harder inputs whose outputs the model cannot reliably predict, while randomly sampled tests have more accurate outputs but rarely fail. Supervised fine-tuning on bootstrapped data, built from corrupted reference solutions and chain-of-thought rationales, moves models into a sweet spot where both properties hold; UTGen reports 7.59% higher Accuracy ∩ Attack than prompting baselines. On downstream tasks, feeding these tests through UTDebug's self-consistency and backtracking improves pass@1 debugging accuracy (the share of debugged programs passing all tests) for Qwen2.5 32B by up to 15.07% absolute over no-test feedback and by over 12.35% on the hardest MBPP+ split compared with other LLM test generators, and the same tests act as a better judge of code correctness than a specialized 8B reward model.","pith_inferences":["If the same bootstrapping recipe—corrupt a reference, sample inputs that expose the corruption, relabel outputs with rationales—were applied to other programming languages or to non-functional properties such as performance or memory, the attack-rate/output-accuracy trade-off would likely re-emerge and training would be expected to resolve it there as well.","Once decontamination is confirmed, UTGen-style models could serve as a cheap 'test oracle' for reinforcement-learning loops that train code models on execution feedback, potentially replacing much of the cost of sampling gold solutions during training.","The trade-off may generalize beyond code: in any specification-to-behavior task, the hardest distinguishing examples are precisely the cases where current models are least able to predict the expected behavior, so the same training recipe may apply to data transformations or API usage."],"forward_implications":["LLM debugging pipelines can run without human-written unit tests: a fine-tuned 7B-32B model supplies the feedback, removing the test-availability bottleneck.","Unit-test-based judging of code correctness beats a trained 8B reward model on best-of-10 sampling, offering an alternative verification signal for code generation.","Frontier debuggers such as GPT-4o improve substantially when given UTGen-generated tests, showing test quality matters even for very strong debuggers.","The debugging gains persist as the number of generated unit tests grows, and UTGen remains better than random test sampling when both are scaled up.","Because UTGen conditions on the faulty code rather than sampling inputs independently, its tests localize errors better than CodeT-style generation."],"supporting_citations":[{"why":"Supplies the CodeT baseline and the random-UT generation comparison that UTGen is measured against.","marker":"(Chen et al., 2023a)"},{"why":"Provides EvalPlus's expanded private unit tests used to build HE+Fix and MBPP+ and to evaluate code accuracy.","marker":"(Liu et al., 2024b)"},{"why":"Supplies the Tulu-3 SFT mixture of problem descriptions and gold code from which UTGen training data is bootstrapped.","marker":"(Lambert et al., 2024a)"},{"why":"Provides HumanEvalFix, the source of faulty code for HE+Fix.","marker":"(Muennighoff et al., 2024)"},{"why":"Provides self-consistency, the test-time scaling method UTDebug uses to boost unit-test output accuracy.","marker":"(Wang et al., 2022)"},{"why":"Provides chain-of-thought prompting, the rationale format UTGen adds to its supervised data.","marker":"(Wei et al., 2022)"},{"why":"Provides the post-hoc rationalization (STaR) procedure used to generate the chain-of-thought rationales for training.","marker":"(Zelikman et al., 2022)"},{"why":"Supplies the Skywork 8B reward model baseline that UTGen outperforms as a code judge.","marker":"(Liu et al., 2024a)"},{"why":"Documents GPT-4o, the frontier debugger used to test the value of high-quality generated tests.","marker":"(Hurst et al., 2024)"}],"fun_headline_variants":["AI-written tests reveal bugs and boost debugging","Teaching LLMs to write bug-revealing tests","Trained tests beat reward models at judging code","Bug-catching tests improve LLM debugging accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that HumanEval and MBPP+ problems used for evaluation never appear among the Tulu-3 training problems that UTGen is fine-tuned on; the paper filters Tulu-3 for Python functions but reports no decontamination check, so leaked benchmark solutions would inflate the measured test-generation and debugging gains.","fun_headline_variants_meta":{"raw":{"variants":["AI-written tests reveal bugs and boost debugging","Teaching LLMs to write bug-revealing tests","Trained tests beat reward models at judging code","Bug-catching tests improve LLM debugging accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000299,"raw_usage":{"total_tokens":1817,"prompt_tokens":1122,"completion_tokens":695,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":738,"completion_tokens_details":{"reasoning_tokens":637}},"tokens_in":738,"tokens_out":695,"duration_ms":6934,"temperature":1.0,"reasoning_tokens":637,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T14:47:44.827482+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a decontamination scan of UTGen's training data against HumanEval and MBPP+ (for example, fuzzy matching problem descriptions or code tokens to benchmark solutions); if overlaps exist, retrain on the cleaned set and check whether the 7.59% Accuracy ∩ Attack gain and the debugging improvements persist. A large drop toward the zero-shot prompting baseline would show the reported gains came from memorization rather than learned unit-test generation; no overlap with stable numbers would support the generalization claim.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides HumanEvalFix, the source of faulty code for HE+Fix."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the post-hoc rationalization (STaR) procedure used to generate the chain-of-thought rationales for training."}],"review_version":1}