{"id":"1a9e5995-b6b6-4745-9005-f6c3dcedd955","arxiv_id":"2608.07437","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"P-Bench and Fisher-R1 show that small LLM agents trained with outcome-grounded reinforcement learning on synthetic hypothesis-testing tasks can outperform frontier models at statistically valid p-value reporting and decisions.","lead":"This paper introduces P-Bench, a benchmark of 425 open-ended hypothesis-testing tasks with expert-verified answer keys, and Fisher-R1, a small open-weight LLM agent trained with reinforcement learning to beat larger models like GPT-5.4 on statistical validity. It matters because LLM agents increasingly automate data analysis, and the benchmark exposes a failure mode that conclusion-only evaluations miss: correct code with invalid statistical conclusions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"P-Bench's Strict metric treats a single canonical analysis as the unique correct answer, while §4.3 acknowledges multiple defensible methods; this can penalize valid alternatives and makes the benchmark's validity, not just its reproducibility, load-bearing for the headline claim.","rationale":"The reader's weakest assumption identifies exactly the issue I find most load-bearing: the benchmark treats the canonical reference analysis as a unique gold standard. My reading of the full text supports this concern and adds one amplification: the same p-value-closeness criterion appears twice, in the RL reward (§4.3) and in the Strict evaluation metric (§5.1). That symmetry means the paper is not merely evaluating an agent's statistical reliability; it is evaluating closeness to one logged analysis, and rewarding that same closeness during training. The internal admission in §4.3 that multiple procedures may be defensible directly undercuts the 'trustworthy by construction' framing of §3.2 unless the benchmark can show that different defensible methods rarely produce p-values separated by more than |Δz| = 0.5. The concern is not that the authors are wrong about method selection mattering—the Figure 1 example is persuasive and the Raw versus Strict gap is real—but that the headline quantitative claims (21% average relative improvement, 'reliable hypothesis testing') depend on the single-key assumption. I also note the paper's own Discussion limits P-Bench to single hypothesis tests and positions the tools as oversight aids, which is honest but does not resolve the uniqueness concern. The lack of released benchmark artifacts, code, and expert-audit records makes it impossible to check this from the preprint alone. Therefore the correct disposition is the reader's CONDITIONAL: accept the contribution as promising, but require a consensus-based audit or an alternative-methods analysis before treating P-Bench as a standard reliability measure. I do not see a reason to move to REJECT, because the training pipeline, the similarity-based generalization check, and the internal ablations are all coherent and supportive; the unresolved risk is about what P-Bench measures, not about whether the experiments were run as described.","tokens_in":19032,"tokens_out":4810,"duration_ms":51522,"concrete_test":"Select 50 P-Bench tasks spanning source families and difficulty splits. Give each task to 3–5 independent statisticians blinded to the answer keys, asking them to list all defensible methods and report p-values, and to mark clearly invalid methods. Then: (1) compute the fraction of tasks where an expert-endorsed alternative differs from the canonical key by |Δz| >= 0.5; (2) recompute Strict pass@1 for Fisher-R1-14B and DeepSeek-V4-Pro on the subset with unique expert consensus. If the fraction is high or rankings change, the canonical-key assumption fails; if rankings persist, the concern is bounded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 states that answer keys are 'trustworthy by construction' because they come from logged canonical reference analyses, and Section 5.1 defines Strict accuracy as |z(p_hat) - z(p*)| < 0.5 in two-sided z-space. Yet Section 4.3 explicitly avoids an explicit method-correctness reward because 'multiple statistical procedures may be defensible for the same hypothesis-testing task.' These two commitments are in tension. For many P-Bench tasks—especially observational regressions such as logistic, Cox, mixed-effects, and IV/2SLS—the reported p-value depends on defensible modeling choices: covariate adjustment, robust versus model-based standard errors, clustering level, outlier handling, missing-data protocol, and filtering. Two valid analyses can easily differ by more than the Strict threshold; for example, p = 0.01 and p = 0.05 differ by about 0.62 in z, and both can arise from legitimate specifications on the same data. Under Strict, one is marked wrong even if both are statistically defensible. The load-bearing consequence is that 'reliable hypothesis testing' is operationalized as matching a single published analysis. The same p-value-closeness signal is used both as the RL reward (§4.3) and as the evaluation metric (§5.1), so a model trained to reproduce canonical p-values on synthetic tasks will score well even if its method choices are not the only valid ones. The paper provides no estimate of how often P-Bench tasks admit multiple defensible analyses with p-values separated by more than |Δz| = 0.5. If that fraction is substantial, both the benchmark's validity and the magnitude of Fisher-R1's measured advantage are open to question.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces P-Bench, a benchmark of 425 open-ended hypothesis-testing tasks drawn from economics, biology, and medicine, in which an agent must choose a statistical method, execute the analysis in R, and report a p-value and a reject/fail-to-reject conclusion against a hidden answer key derived from a logged canonical reference analysis. The paper also introduces Fisher-R1, an open-weight agent initialized from Qwen2.5-Coder and trained with supervised fine-tuning on teacher trajectories followed by DAPO reinforcement learning on 8,642 synthetic tasks; the reward combines p-value closeness on a two-sided z-scale with conclusion agreement. The paper reports that Fisher-R1-14B improves substantially over its backbone and matches or exceeds GPT-5.4 and DeepSeek-V4-Pro on P-Bench, and it provides ablations, a cosine-similarity analysis, and a relaxed-threshold robustness check.","tokens_in":19338,"tokens_out":7598,"duration_ms":69286,"significance":"If the answer keys are accepted as unique ground truth, P-Bench fills a real gap in evaluating inferential method selection and p-value accuracy rather than code executability or factoid answers, and the reported training gains for Fisher-R1 are substantial and practically useful. The strengths include the execution-log grounding of answer keys, the held-out evaluation set, the three-stage reproducibility pipeline, the ablation showing that SFT and DAPO are both needed, the similarity analysis that addresses memorization, and the relaxed-threshold table. The main risk is that the evaluation criterion assumes a single canonical analysis is the only correct answer, while the paper's own training design acknowledges that multiple statistical procedures may be defensible; this makes the benchmark's validity, not just its reproducibility, load-bearing for the headline conclusions.","major_comments":[{"comment":"The validity of P-Bench as a measure of reliable hypothesis testing is not established against the possibility of multiple defensible analyses. Section 3.2 calls answer keys 'trustworthy by construction,' but Section 5.1 defines Strict accuracy as |z(p_hat) - z(p*)| < 0.5, which marks a reported p-value as wrong whenever it differs from the single canonical value by more than that threshold. Section 4.3, however, explicitly states that 'multiple statistical procedures may be defensible for the same hypothesis-testing task.' For observational tasks in P-Bench (logistic, Cox, mixed-effects, IV/2SLS), legitimate choices about covariate adjustment, robust versus model-based standard errors, clustering level, and missing-data handling can produce p-values differing by more than 0.5 in z-space; for example, p = 0.05 and p = 0.01 differ by about 0.62 in z. The same z-space closeness is used as the RL reward in Section 4.3 and as the evaluation metric in Section 5.1, so 'reliability' is operationalized as agreement with one canonical implementation. The paper should provide an audit of how often P-Bench tasks admit alternative defensible analyses with p-values beyond the Strict threshold, plus a sensitivity analysis that accepts a set of reasonable alternative answers.","section":"§3.2, §5.1, §4.3"},{"comment":"The reward and the evaluation metric are the same functional object: both score the reported p-value by closeness to a single p* in two-sided z-space. Because the synthetic training answer keys are produced by running 'canonical' method code on simulated data (Appendix C.2.1) and P-Bench answer keys are produced by the same style of canonical reference analysis (Section 3.2), the training signal optimizes for reproducing a particular statistical convention. This is not direct overfitting to P-Bench, since P-Bench is held out, but it means the reported gains are evidence of alignment with the reference canon rather than, by itself, evidence of statistical validity independent of that canon. To disentangle these, the paper should add an evaluation variant in which any p-value inside an interval derived from a reasonable range of defensible specifications counts as correct, and an ablation in which the RL reward is replaced by a method-validity score that does not depend on a single p*.","section":"§4.3 and §5.1"},{"comment":"The audit and release details needed to verify P-Bench are missing. The text says every answer key is expert-audited and tasks failing review are repaired or removed, but it does not state the number of auditors, their qualifications, the audit protocol, or inter-auditor agreement; it also does not provide the benchmark itself, the answer keys, the datasets, or the reproducibility pipeline code. The automated pipeline described in Appendix B.1 is built on Claude Code, but no accuracy or failure-rate statistics are reported for the claim-parsing and reproduction steps. Without public release of the benchmark and an audit appendix, the 'expert-verified' status and the numerical scores cannot be independently checked.","section":"§3.2 and Appendix B.1"},{"comment":"The specific claim that Fisher-R1-14B outperforms GPT-5.4 on three of four Strict metrics is not supported by the reported statistics. Table 1 reports pass@1 as mean ± stdev over three independent runs; on P-Easy Strict pass@1 the values are 64.2 ± 0.6 for Fisher-R1-14B and 64.7 ± 3.3 for GPT-5.4, and on P-Hard Strict pass@1 they are 33.0 ± 1.7 and 30.5 ± 1.6. These differences are within the reported run-to-run variability, no paired significance test is given, and the relaxed-threshold results in Table 5 even show GPT-5.4 ahead on P-Easy Strict (72.6 ± 2.5 vs 70.6 ± 1.4). Please report task-level bootstrap confidence intervals and paired tests across models, or soften the superiority claim accordingly.","section":"Table 1 and §5.2"}],"minor_comments":[{"comment":"The sentence 'P-Bench targets a regime that neither captures the full open-ended hypothesis-testing loop...' appears to omit an intended contrast with existing benchmarks and should be rewritten for clarity.","section":"§2.2"},{"comment":"Model names are sometimes concatenated without spaces (e.g., 'GPT-5.492.9' and 'DeepSeekV4-Pro'); fix the formatting throughout.","section":"Table 1 and Figure 1"},{"comment":"The similarity analysis would be more informative if it reported numeric summaries of the separation between the eval-to-train and train-to-train distributions (e.g., mean and percentile values) rather than only a density plot.","section":"§5.4"},{"comment":"For the perturbed outlier tasks, the answer key is computed using 'robust standard errors or trimming' as the handling protocol; this choice should be documented per task, since trimming versus robust standard errors can change both the p-value and the conclusion.","section":"Appendix C.3.2"},{"comment":"The cross-reference 'summarised in S4.3' should be '§4.3'; there is no 'S4.3' section in the main text.","section":"Appendix C.2.1"}],"recommendation":"major_revision","confidential_remarks":"The answer-key uniqueness issue is the central risk. If the authors cannot show that the large majority of P-Bench tasks have a unique defensible p-value within the Strict threshold, both the benchmark's validity and the model comparison are in question. The paper is otherwise well structured, and the training pipeline with held-out evaluation is a strong contribution, so I would support a revision that adds the requested audit and sensitivity analyses and releases the benchmark for independent verification."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely useful benchmark plus a clean training recipe, but the evaluation's central assumption—one correct p-value per task—is doing a lot of work and needs defending before I'd trust the headline numbers.\n\nWhat's new: P-Bench is the first benchmark I know that combines open-ended method selection, executed analysis, p-value accuracy, and conclusion correctness on real scientific data. Answer keys come from logged canonical runs rather than paper text, and the expert-audit step is the right instinct. Fisher-R1's training loop is also well designed: synthetic tasks with verified rewards, P-Bench held out, SFT warm-start then DAPO. The ablations and the relaxed-threshold check are the right kind of supporting evidence.\n\nThe soft spot is exactly the one flagged in the stress test. The Strict metric requires the agent's p-value to be within |Δz|<0.5 of the canonical answer key. Yet §4.3 explicitly says multiple statistical procedures may be defensible for the same task—that's why the reward has no method-correctness term. So the paper acknowledges multiplicity but then grades against a single p-value. On observational regressions (Cox, mixed-effects, IV), defensible choices about covariates, robust vs. model-based standard errors, or clustering can easily move p-values more than 0.5 in z. The paper never quantifies how often P-Bench tasks admit multiple defensible analyses with separated p-values. That number matters because the headline gains are measured on Strict. The same p-value closeness is used as both reward and metric, so the model is explicitly trained to reproduce canonical p-values—which may be exactly what we want, or may be overfitting to one statistical culture.\n\nSecond, the paper says it releases P-Bench and Fisher-R1, but the preprint shows no code, data, or audit artifacts. For a benchmark, that's a real obstacle to independent verification. The similarity analysis is also thin—cosine similarity on prompt text only, not on the analyses themselves. Minor: the proprietary baselines may have seen some public datasets during pretraining; no contamination discussion.\n\nWho this is for: people building LLM agents for automated science and anyone designing evaluations for statistical reasoning. It deserves a serious referee slot. Required revision: either show that P-Bench tasks are effectively unambiguous, or switch to a multi-answer evaluation (e.g., any method from a pre-approved set) and re-check the main comparisons. If that survives, the contribution stands.","headline":"P-Bench and Fisher-R1 fill a real gap in evaluating statistical reasoning of LLM agents, but the Strict metric's one-correct-p-value assumption needs validation before the headline numbers can be fully trusted.","tokens_in":19924,"tokens_out":3377,"would_cite":true,"duration_ms":33547,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fisher-R1 shows that small open-weight agents, trained with verified p-value rewards, can outperform frontier models on statistical hypothesis testing.","keywords":["hypothesis testing","LLM agent","p-value","reinforcement learning","statistical reasoning","benchmark","P-Bench","Fisher-R1"],"falsifier":"Take a P-Bench biology task with outliers, have a panel of statisticians pre-register an alternative valid method (e.g., a rank-based test where the key uses linear regression), and check whether their p-value, differing by more than |Δz|=0.5, is marked wrong; if correct inferences are marked wrong, the Strict metric is not method-agnostic.","tokens_in":18807,"feed_emoji":"🧪","tokens_out":6710,"duration_ms":59163,"temperature":0.7,"pith_summary":"The paper argues that current LLM agents, even frontier ones, cannot be trusted to carry out hypothesis testing end-to-end: they often pick a plausible statistical method, run it correctly, and report a confident but wrong conclusion. To make this failure measurable it builds P-Bench, 425 open-ended tasks with expert-audited answer keys from real economic, biological, and medical analyses. It then trains Fisher-R1, an open-weight agent, on synthetic tasks whose answer keys are produced by executing the canonical method, using reinforcement learning with a reward that scores closeness of the reported p-value to the key in z-score space. On P-Bench, Fisher-R1-14B beats strong proprietary and open-source baselines, including GPT-5.4 and DeepSeek-V4-Pro, with a 21% average relative improvement in single-trial success over DeepSeek-V4-Pro. The broader claim is that reliable statistical inference is a skill distinct from code execution, and that outcome-verified reinforcement learning can instill it.","feed_headline":"Open-weight 14B agent beats GPT-5.4 at hypothesis testing","feed_subtitle":"Trained on verified p-value rewards, Fisher-R1 lifts single-trial success 21% over DeepSeek-V4-Pro.","key_machinery":"The load-bearing machinery is the z-score re-labeling of p-values, $z(p)=\\Phi^{-1}(1-p/2)$, used twice: as the training reward (exponential closeness between reported and key z-scores) and as the evaluation metric (Strict requires $|\\Delta z|<0.5$). Because raw p-values are compressed near zero, this monotonic transformation makes small differences at high significance count as large evidence gaps; it lets a single reward signal compare p-values regardless of which test produced them. The second mechanism is the synthetic task generator: the answer key is the output of running the canonical statistical method on simulated data, giving an executable, programmatically checkable reward signal that keeps P-Bench fully out of the training set.","core_discovery":"On the paper's own terms, the central discovery is that instruction-following and coding ability are not enough for statistical hypothesis testing: every evaluated model, including GPT-5.4, frequently reports a correct reject/fail-to-reject decision while producing a p-value far from the reference analysis, as shown by Raw scores far exceeding Strict scores, especially on hard tasks. The paper demonstrates that this gap can be closed substantially by training with a verified statistical reward: Fisher-R1-14B reaches 33.0% Strict pass@1 on the hard split versus 30.5% for GPT-5.4 and 26.3% for DeepSeek-V4-Pro, and the trained models also show much lower run-to-run variance. This supports the claim that targeted reinforcement learning on hypothesis-testing tasks with outcome-grounded rewards, rather than mere model scale, improves inferential reliability.","pith_inferences":["Outside the paper: if several statistical analyses are defensible for one task, the single-key Strict metric will penalize valid alternative methods; an ensemble-keys variant of P-Bench would test how much of Fisher-R1's edge is method-agnostic.","Outside the paper: the z-space reward could be lifted to other inferential quantities such as effect sizes, confidence intervals, or Bayes factors, letting the same RL recipe train agents on calibration rather than a single p-value.","Outside the paper: ablating the adversarial perturbation variants from the training corpus would isolate how much of the hard-split gain comes from explicit handling of outliers and missingness rather than from general method selection."],"forward_implications":["Conclusion-only evaluation overstates agent reliability: every baseline scores far lower on Strict than on Raw, so adding a p-value-closeness criterion to data-analysis benchmarks is a cheap and informative check.","Training a small open-weight model with an outcome-verified p-value reward can outperform much larger models, since Fisher-R1-7B/14B beat GPT-OSS-120B, Qwen3-32B, and DeepSeek-V4-Pro on every Strict metric.","Using synthetic tasks with executable answer keys gives a scalable, programmatically checkable reward that keeps the evaluation benchmark out of the training set, as the embedding-similarity analysis shows.","The same recipe can extend to multi-test pipelines and multiple-comparison correction, the next direction the authors identify in the discussion."],"supporting_citations":[{"why":"Supplies the economics task family from published field-experiment datasets used to build P-Bench.","marker":"[23]"},{"why":"Supplies the biology task family from public cancer-genomics datasets used to build P-Bench.","marker":"[24–26]"},{"why":"Supplies the medicine and biostatistics teaching datasets used to build P-Bench tasks.","marker":"[27]"},{"why":"Frames the gap P-Bench fills: prior evaluation asks for method selection without executing the analysis.","marker":"[8]"},{"why":"Provides the DAPO reinforcement-learning algorithm with asymmetric clipping and dynamic sampling used to train Fisher-R1.","marker":"[30]"},{"why":"Provides the Qwen2.5-Coder backbone models that Fisher-R1 fine-tunes.","marker":"[31]"},{"why":"GPT-5.4 is the proprietary baseline that Fisher-R1-14B matches or exceeds on Strict metrics.","marker":"[32]"},{"why":"DeepSeek-V4-Pro is the strong open-source baseline against which the 21% relative improvement is measured.","marker":"[33]"}],"fun_headline_variants":["Fisher-R1: 14B agent betters GPT-5.4 in hypothesis tests","RL on statistical reward lifts Fisher-R1 21% over DeepSeek","Open-weight Fisher-R1 beats GPT-5.4 on P-Bench rigor","P-value RL: Fisher-R1 wins hypothesis testing by 21%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's answer keys assume there is one correct statistical analysis per task, and the Strict score requires the agent's p-value to land within |Δz|<0.5 of that key, so any task with several defensible methods can mark a correct inference as wrong.","fun_headline_variants_meta":{"raw":{"variants":["Fisher-R1: 14B agent betters GPT-5.4 in hypothesis tests","RL on statistical reward lifts Fisher-R1 21% over DeepSeek","Open-weight Fisher-R1 beats GPT-5.4 on P-Bench rigor","P-value RL: Fisher-R1 wins hypothesis testing by 21%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000217,"raw_usage":{"total_tokens":1456,"prompt_tokens":985,"completion_tokens":471,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":385}},"tokens_in":601,"tokens_out":471,"duration_ms":4361,"temperature":1.0,"reasoning_tokens":385,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T04:34:50.543178+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a P-Bench biology task with outliers, have a panel of statisticians pre-register an alternative valid method (e.g., a rank-based test where the key uses linear regression), and check whether their p-value, differing by more than |Δz|=0.5, is marked wrong; if correct inferences are marked wrong, the Strict metric is not method-agnostic.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the medicine and biostatistics teaching datasets used to build P-Bench tasks."},{"cited_title":"Development economics field experiments (dfeep)","cited_arxiv_id":null,"evidence_quote":"Supplies the economics task family from published field-experiment datasets used to build P-Bench."},{"cited_title":"Are large language models good statisticians?Advances in Neural Information Processing Systems, 37:62697–62731, 2024","cited_arxiv_id":null,"evidence_quote":"Frames the gap P-Bench fills: prior evaluation asks for method selection without executing the analysis."},{"cited_title":"Deepseek-v4: Towards highly efficient million-token context intelligence","cited_arxiv_id":null,"evidence_quote":"DeepSeek-V4-Pro is the strong open-source baseline against which the 21% relative improvement is measured."}],"review_version":1}