{"id":"b30e02ac-4920-4e37-96bc-51b533460228","arxiv_id":"2606.00920","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"LLM run-level pass rates on 100 LeetCode problems exceed retry-free coverage by up to 17.8 points and reverse some model rankings.","lead":"The paper finds that standard single-run pass rates for LLMs on coding tasks overstate consistent success without retries by up to 17.8 percentage points, largest for mid-performing models. A smart generalist might read it to see why current benchmarks can mislead when LLMs must deliver reliable outputs every time in deterministic settings.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Point estimates of 17.8pp gap and ranking reversals lack uncertainty quantification given only 5 runs per problem","rationale":"The reader's weakest assumption concerns whether 100 problems + 2 prompts + 5 runs suffice for a general relationship. My concern is a narrower but directly load-bearing statistical issue within the reported numbers: the absence of uncertainty quantification on the very quantities (gap size and ranking changes) used to support the headline claim. The two issues are related because the small per-problem run count is part of what makes the sufficiency question acute. If the bootstrap CIs are wide, the specific 17.8pp figure and the ranking reversals would not be reliable even on this benchmark, moving the verdict from UNVERDICTED to CONDITIONAL pending the check.","tokens_in":1709,"tokens_out":450,"duration_ms":20040,"concrete_test":"For each model, treat the 100 problems as the sampling units; compute the observed gap (run-level pass rate minus retry-free coverage) and the rank order. Then draw 10,000 bootstrap replicates by resampling the 100 problems with replacement, recompute both metrics and the gap on each replicate, and obtain 95% percentile CIs. Check whether the CI for the largest reported gap excludes zero and whether any claimed ranking reversal remains under the lower and upper bounds of the CI.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central empirical claim is that run-level pass rate exceeds retry-free coverage by up to 17.8pp (largest for mid-performing models) and that this produces ranking reversals. Both metrics are computed from exactly five independent runs per problem. Run-level pass rate is an average over 500 binary outcomes per model; retry-free coverage is the fraction of problems with success on all five runs. With n=5 the binomial variance on per-problem success probability is large, and the probability of observing all successes is highly sensitive to small fluctuations in the underlying success rate. The abstract reports no bootstrap CIs, standard errors, or paired tests on the per-problem differences, nor any assessment of whether the observed gaps or ranking changes exceed sampling variability.","agreement_with_reader":"partial"},"referee_report":{"model":"grok-4.3","summary":"The manuscript reports an empirical evaluation of 16 LLMs on 100 LeetCode-style deterministic programming problems using two prompt templates and five independent runs per problem (16,000 total instances). It claims that run-level pass rate consistently exceeds retry-free coverage (problems solved on all five runs) by up to 17.8 percentage points, with the gap largest for mid-performing models; the two metrics are strongly correlated (r=0.985) yet produce ranking reversals; and prompt effects are model-dependent rather than uniformly beneficial.","tokens_in":1881,"tokens_out":416,"duration_ms":15415,"significance":"If the central empirical findings hold after statistical scrutiny, the work provides concrete evidence that conventional single-run or eventual-success accuracy metrics can overstate reliability for deterministic text-conditioned tasks, particularly for mid-tier models. The scale of the repeated-run protocol and the direct counts of pass rates versus perfect-stability coverage constitute a useful, falsifiable dataset for the community.","major_comments":[{"comment":"Abstract and (presumed) Results section: the reported maximum gap of 17.8 percentage points and the occurrence of ranking reversals are given as point estimates with no bootstrap confidence intervals, standard errors, or paired statistical tests on the per-problem differences. With exactly five runs per problem, the binomial variance on the all-success probability is large and the metric is highly sensitive to small changes in underlying success rate; without uncertainty quantification it is impossible to determine whether the observed gaps or reversals exceed sampling variability.","section":"Abstract"},{"comment":"Evaluation protocol (described in abstract): the decision to use precisely five runs is presented without power analysis, sensitivity checks, or justification relative to the binomial variance of the retry-free coverage metric. This choice directly affects the magnitude and reliability of the 17.8 pp gap that forms the central claim.","section":"Evaluation protocol"}],"minor_comments":[],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the careful review and for highlighting the need for greater statistical rigor in reporting our empirical results. We respond to each major comment below.","responses":[{"response":"We agree that uncertainty quantification is necessary to assess whether the reported gaps and ranking reversals exceed sampling variability. In the revised manuscript we will add bootstrap confidence intervals (with 10,000 resamples) for the per-model differences between run-level pass rate and retry-free coverage, and we will apply paired McNemar tests on the per-problem binary outcomes to evaluate statistical significance of the observed differences.","revision_made":"yes","referee_comment":"[Abstract] Abstract and (presumed) Results section: the reported maximum gap of 17.8 percentage points and the occurrence of ranking reversals are given as point estimates with no bootstrap confidence intervals, standard errors, or paired statistical tests on the per-problem differences. With exactly five runs per problem, the binomial variance on the all-success probability is large and the metric is highly sensitive to small changes in underlying success rate; without uncertainty quantification it is impossible to determine whether the observed gaps or reversals exceed sampling variability."},{"response":"The choice of five runs was driven by the need to keep the total evaluation budget (16,000 instances) tractable while still permitting estimation of stability. No formal power analysis was performed prior to data collection. In revision we will add a post-hoc sensitivity analysis that recomputes the gap, correlation, and ranking reversals using the first 3 and first 7 runs of the existing data, together with a brief justification tied to the observed per-problem variance in our pilot runs.","revision_made":"partial","referee_comment":"[Evaluation protocol] Evaluation protocol (described in abstract): the decision to use precisely five runs is presented without power analysis, sensitivity checks, or justification relative to the binomial variance of the retry-free coverage metric. This choice directly affects the magnitude and reliability of the 17.8 pp gap that forms the central claim."}],"tokens_in":1408,"tokens_out":445,"duration_ms":19611,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The core observation is straightforward: on 100 LeetCode-style problems, average per-run success exceeds the fraction of problems solved correctly in every one of five runs, with the difference largest for mid-tier models and large enough to change some model orderings. They ran 16 models from five families, two prompt templates, and 16,000 total generations, which is a solid empirical scale for this kind of check.\n\nWhat stands out is the clean separation of metrics—run-level accuracy versus retry-free coverage—and the direct count of how often the two diverge. The correlation between the two is high (0.985), yet the gap is systematic rather than random. Prompt effects also vary by model instead of helping everyone equally. That part is useful for anyone who has to pick models for settings where the same prompt must produce the same correct output every time.\n\nThe soft spot is the sampling. Five repeats per problem is enough to show that variability exists, but binomial variance on per-problem success is still large, and the probability of five successes in a row is sensitive to small shifts in the underlying rate. The abstract gives point estimates for the 17.8-point maximum gap and the ranking reversals without bootstrap intervals, standard errors, or paired tests on the per-problem differences. It is therefore unclear how much of the reported gap and the ranking changes would survive additional runs or different problem samples. The 100-problem set is recent, which is good, but it remains a narrow slice of deterministic generation tasks.\n\nThis work is aimed at groups that build or select LLMs for production code generation where output consistency matters. It is worth sending to referees because the measurement protocol is simple to replicate and the basic phenomenon is worth confirming with tighter uncertainty estimates or more runs. A revised version with error bars and a sensitivity check on the number of repeats would be straightforward to produce and would strengthen the claim.","headline":"The paper quantifies how single-run pass rates overstate consistent success on code tasks by up to 17.8 points and can flip rankings, but five runs per problem leave those numbers noisy.","tokens_in":2367,"tokens_out":471,"would_cite":false,"duration_ms":13394,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Run-level pass rates for LLMs on programming tasks overstate retry-free coverage by up to 17.8 percentage points.","keywords":["LLM evaluation","code generation","run-level accuracy","retry-free coverage","model stability","repeated runs","prompt effects","deterministic tasks"],"falsifier":"Finding no systematic gap between run-level pass rate and retry-free coverage on a larger or differently constructed set of deterministic problems would falsify the claimed overstatement.","tokens_in":2615,"feed_emoji":"📊","tokens_out":644,"duration_ms":16890,"temperature":0.7,"pith_summary":"The paper measures how consistently large language models solve the same deterministic programming problems when invoked repeatedly under identical prompts. Standard accuracy reporting counts every successful run separately, but this exceeds the share of problems solved correctly on every run without any retries. The difference reaches 17.8 points and is largest for mid-tier models, enough to reverse rankings between otherwise similar systems. The evaluation uses 100 LeetCode-style problems, 16 models, two prompt templates, and five runs each to separate average success from per-problem stability. Readers care because many deployed uses require the model to produce the correct output reliably rather than sometimes.","feed_headline":"LLM pass rates overstate reliable coverage by 17.8 points","feed_subtitle":"On 100 coding problems, single-run success exceeds consistent success on every run, reversing rankings among similar models.","key_machinery":"The repeated-run evaluation protocol that separately tracks run-level accuracy, retry-free coverage (problems solved correctly on every run), and per-problem variability across five executions per problem.","core_discovery":"On a recency-based benchmark of 100 LeetCode-style problems, 16 models from five provider families were evaluated under two prompt templates with five repeated runs per problem. Run-level pass rate consistently exceeds retry-free coverage, with the gap reaching 17.8 percentage points and largest for mid-performing systems; this discrepancy reverses model rankings even among closely matched systems. Run-level pass rate and perfect stability rate correlate strongly (r=0.985), yet the systematic overstatement remains. Prompt effects are model-dependent rather than uniformly beneficial. These findings indicate that repeated-run stability analysis is a necessary complement to conventional accurac","pith_inferences":["The same run-versus-coverage gap may appear in other deterministic text-generation domains such as structured data extraction or formal verification.","Benchmark designers could adopt dual reporting of both metrics to prevent misleading single-number rankings.","Applications that penalize inconsistent outputs may favor models with smaller gaps even when their average accuracy is lower.","Increasing the number of runs per problem beyond five could sharpen estimates of per-problem variability."],"forward_implications":["Run-level pass rate and perfect stability rate remain strongly correlated yet differ by as much as 17.8 points.","The overstatement gap is largest precisely for mid-performing models.","Prompt template choice produces model-dependent rather than uniform effects on both accuracy and stability.","Stability analysis must be reported alongside accuracy for deterministic generation tasks."],"fun_headline_variants":["LLM pass rates overstate stable coverage by 17.8 points","Run accuracy exceeds consistent success by 17.8 points","Mid systems show largest stability gaps of 17.8 points","Rankings reverse when requiring run stability in LLMs"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The 100 LeetCode-style problems together with the two prompt templates and five repeated runs per problem are sufficient to reveal the general accuracy-stability relationship for deterministic text-conditioned generation tasks.","fun_headline_variants_meta":{"raw":{"variants":["LLM pass rates overstate stable coverage by 17.8 points","Run accuracy exceeds consistent success by 17.8 points","Mid systems show largest stability gaps of 17.8 points","Rankings reverse when requiring run stability in LLMs"]},"model":"grok-4.3","cost_usd":0.00888,"raw_usage":{"total_tokens":4021,"prompt_tokens":723,"num_sources_used":0,"completion_tokens":68,"cost_in_usd_ticks":88799500,"prompt_tokens_details":{"text_tokens":723,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":3230,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":723,"tokens_out":68,"duration_ms":21532,"temperature":1.0,"reasoning_tokens":3230,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-28T18:46:14.458688+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Finding no systematic gap between run-level pass rate and retry-free coverage on a larger or differently constructed set of deterministic problems would falsify the claimed overstatement.","supporting_citations":[],"review_version":1}