{"id":"25b19e57-cd10-433c-a5f2-74723bfbc991","arxiv_id":"1908.08702","paper_version":4,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A profit-maximizing scientist will rationally choose small samples when effects are rare or weak, producing the bimodal power and low reproducibility seen in the literature.","lead":"This paper builds a simple economic model where scientists pick sample sizes to maximize publication income minus cost. It predicts that small samples and low reproducibility are rational responses to low odds of a true effect, small effect sizes, and low funding-per-publication.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline PPV<0.5 result hinges on the assumed low/bimodal distribution of base rates b; under uniform b the same model yields PPV≈0.8, so the irreproducibility claim is an output of an unmeasured input.","rationale":"The reader's weakest_assumption correctly identifies the dependence of the low-reproducibility prediction on the unmeasured distribution of b. This is the single most load-bearing concern because the abstract's claim of 'low overall reproducibility rates' is the empirical payoff of the model, and Fig. 3C shows it flips to PPV≈0.8 under uniform b. The internal mathematics of the model are sound, and the bimodal-power prediction is robust across b distributions, but the quantitative irreproducibility result is conditional on assuming most hypotheses have low pre-study probability. The paper is transparent about this assumption and frames the model as a formalization of an existing intuition, so the appropriate verdict is unchanged: accept as a theoretical contribution with a clearly stated caveat. The concrete test using replication-derived b distributions would determine whether the low-PPV result is empirically grounded or an artifact of the chosen prior. The reader's verdict and reasoning already capture this caveat, so no adjustment is needed.","tokens_in":20554,"tokens_out":10138,"duration_ms":111404,"concrete_test":"Recompute the Fig. 3C simulations using an empirically estimated distribution of b instead of the hand-set beta(1.1, 10). A direct source is the posterior distribution of the prior probability of true effects estimated from Open Science Collaboration replication data using the Johnson et al. (2017) mixture model. Plug that empirical b distribution (and the same d and IF distributions) into the paper's provided code and recompute the mean PPV. If the mean PPV remains below 0.5, the low-reproducibility prediction is supported by data; if it rises above 0.5, the headline result is an artifact of the assumed base-rate prior.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The model's central quantitative claim—that economically rational sample-size choice yields expected reproducibility rates below 50%—is not robust across the space of plausible base-rate distributions. In Fig. 3C, panels 1–8, for uniform or bimodal distributions of b (mean b ≈ 0.5), the same model produces mean PPV between 0.71 and 0.84. Only for the 'low' and 'low/bimodal' beta distributions (α=1.1, β=10, centered near 0.1; Fig. 3C, panels 9–16) does PPV fall to 0.26–0.40. These distributions are justified by plausibility and citations to Smaldino & McElreath and Johnson et al., but the model itself provides no measurement of b across scientific niches. Thus the 'low overall reproducibility rates' predicted in the abstract are not a robust emergent property of the economic optimization; they are largely an input assumption about the pre-study probability of true effects. The robustness claim in the paper correctly applies to the bimodal distribution of power, but not to the PPV<0.5 result, which disappears under uniform b. This matters because the causal framing 'economically rational sample-size choice drives irreproducibility' depends on the empirical fact that most tested hypotheses have low b—something the model does not establish.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper develops a one-period optimization model in which a scientist chooses sample size s to maximize expected income from publications minus sampling cost, with only statistically significant positive findings assumed publishable. The equilibrium sample size (ESS) is shown to increase with the base rate of true effects b, effect size d, and income factor IF. For assumed distributions of these parameters, simulations produce bimodal distributions of statistical power and, for low base-rate distributions, positive predictive values below 0.5; conditional equivalence testing (CET) is then explored as a policy remedy. The paper includes Python code for the ESS and CET computations.","tokens_in":20861,"tokens_out":10520,"duration_ms":116750,"significance":"If the modeling framework is accepted, this is a useful minimal formalization of the widely discussed 'publish-or-perish' explanation for underpowered studies. The main strengths are transparency (full code in the supporting information), falsifiable qualitative predictions (positive associations of b, d, and IF with sample size; a bimodal power distribution), and a clear link to the conditional-equivalence-testing policy discussion. The quantitative headline prediction of sub-50% reproducibility, however, is strongly dependent on the assumed distribution of b, and the objective function in Eq. (1) needs justification relative to a budget-constrained rational scientist. With revision, the paper could be a solid theoretical contribution; in its current form the central claims are somewhat overstated.","major_comments":[{"comment":"Profit is defined as Profit(s)=IF×TPR(s)−s, which is the expected net income of a single study. The Simulation paragraph then weights each ESS by TPR/ESS, explicitly because smaller ESS allows more studies to be conducted. This weighting is not part of the optimization: a budget-constrained scientist with total budget B would maximize B·(IF·TPR(s)/s)−B, whose maximizer generally differs from the maximizer of Eq. (1). For instance, with b=0.5, d=0.5, IF=200, the per-study objective has an interior ESS, while the per-resource objective is maximized at a substantially smaller sample size. Please either provide a fixed-number-of-studies justification (for example, a per-study overhead that makes the number of studies independent of s) or revise the objective and recompute the ESS values and the power/PPV distributions that depend on them.","section":"Eq. (1) and the 'Simulation' paragraph"},{"comment":"The claim of robustly low reproducibility rates is not supported across the probed input distributions. In panels 1–8, uniform and simple bimodal distributions of b (mean b ≈ 0.5) produce mean PPV values of 0.71–0.84; PPV falls to 0.26–0.40 only for the 'low' and 'low/bimodal' beta distributions centered near b ≈ 0.1 (panels 9–16). The Discussion ('Input parameter range estimates') acknowledges that the true distribution of b is unknown, and the low distributions are justified by plausibility and citations rather than measurement. Thus the sub-50% reproducibility result is largely an input assumption rather than an emergent property of the economic optimization. The abstract and conclusion should either conditionalize this claim on the base-rate distribution or provide an empirical calibration of b across scientific niches.","section":"Fig. 3C and 'Emergent power distributions for plausible input parameter distributions'"},{"comment":"The CET power is computed by calling TOSTER::powerTOSTtwo with N=s, whereas the main text states that s is the size of one of two equally sized samples (Materials and methods, 'Simulation'). If the TOSTER function interprets N as the total sample size across both groups, the CET calculations effectively use twice the intended sample size, which would bias ESSCET and the reported PPV improvements in Fig. 4E,F. Please confirm the N convention used by the package and rerun the CET simulations if the current call is inconsistent with the two-sample setup.","section":"S2 Model Code CET and Eq. (3)"}],"minor_comments":[{"comment":"The text states that greater d and IF shift the inflection point 'rightward', but in Fig. 2A the steep rise in ESS occurs at smaller b as d and IF increase, so the shift appears to be leftward along the b axis.","section":"Results, third paragraph"},{"comment":"There are several typographical errors, including 'probablity' in the Introduction and 'adress' and 'reproduciblity' in the Discussion; a careful proofread is needed.","section":"Introduction and Discussion"},{"comment":"The c1–c16 panel labels in a four-by-four grid are difficult to map to the input distributions; labeling rows and columns by the b and IF distributions would make the figure much easier to read.","section":"Fig. 3C"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the manuscript is within scope and the author is appropriately connected to the earlier literature on economic models of sample size. The two load-bearing issues identified above (the objective function in Eq. (1) and the sensitivity of PPV to the unmeasured b distribution) are fixable through re-specification or reframing, so a major revision rather than rejection seems appropriate. No concerns about attribution or research integrity."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper you want to know about: Braganza's simple model of sample-size choice as profit maximization. It is not a major conceptual leap—Higginson & Munafo (2016) and Campbell & Gustafson (2019) already formalized the profit-maximization account of underpowering. What this paper adds is a stripped-down three-parameter version (base rate b, effect size d, income factor IF) that makes two sharp predictions: a three-phase relationship between b and power, and a bimodal distribution of achieved power across niches. The bimodal power prediction is the real contribution; it emerges robustly across most of the parameter distributions they try and lines up with the empirical power distributions from Button et al. and others.\n\nThe paper is also honest and well built. The math is straightforward and the code is included. The author explicitly acknowledges the prior models, and the CET extension is a sensible, well-executed addition. The discussion of the winner's curse and the contrast with scientifically normative sample-size models (Miller & Ulrich) is fair and useful.\n\nThe main soft spot is exactly where the stress-test lands: the headline \"low overall reproducibility rates\" (PPV<0.5) is not a robust output of the model—it is an input. For uniform or bimodal base-rate distributions (mean b ≈ 0.5), the same model yields PPV around 0.71–0.84 (their Fig. 3C, panels 1–8). The below-50% PPV only appears when b is assumed to be low or low/bimodal, centered near 0.1, which is justified by plausibility (Smaldino & McElreath, Johnson et al.) rather than measurement. So the abstract's claim that the model \"drives irreproducibility\" overreaches: the model shows that rational sample sizes produce low reproducibility if most tested hypotheses are already unlikely to be true, which is a coherent conditional statement, not a demonstrated fact. The author does acknowledge this in the discussion, so it is a framing issue more than a hidden flaw.\n\nThere is also a mild concern that the empirical effect-size distribution is fitted to the same Szucs et al. dataset used for comparison, and the IF distribution is essentially guessed. These are not fatal—the qualitative predictions don't hinge on precise parameter values—but they mean the \"empirical match\" is weaker than it appears.\n\nBottom line: this is a useful, clean model paper that sharpens a known argument into testable predictions. The bimodal power prediction is worth taking seriously and testing directly. The \"PPV<0.5\" result should be read as a conditional prediction pending better measurement of base rates. It deserves a serious referee and, if revised to temper the causal language, I'd be happy to see it published. I'd cite it in metascience work.","headline":"A clean, transparent formalization of the economic-underpowering argument; the bimodal-power prediction is the genuine takeaway, while the PPV<0.5 headline depends on the unmeasured base-rate distribution.","tokens_in":21369,"tokens_out":2068,"would_cite":true,"duration_ms":20380,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that small sample sizes are an economically rational response to publication and funding incentives, and that making negative results publishable could fix them.","keywords":["reproducibility","statistical power","sample size","positive publication bias","economic rationality","positive predictive value","conditional equivalence testing","scientific incentives"],"falsifier":"Measure the actual distribution of $b$ across hypotheses in a field, for example by tracking preregistered predictions that are later confirmed, and measure the actual relationship between sample size, effect size, and per-publication funding. If the base-rate distribution were centered near 0.5, the model's own simulations predict a positive predictive value near 0.8 rather than below 0.5, so the headline low-reproducibility prediction fails. Alternatively, a large corrected dataset showing no positive correlation between sample size and true effect size, or no effect of funding per publication on sample size, would contradict the mechanism.","tokens_in":20335,"feed_emoji":"🔬","tokens_out":7079,"duration_ms":63473,"temperature":0.7,"pith_summary":"The paper argues that scientists who are paid only for statistically significant positive findings can rationally choose sample sizes that are too small to reproduce. It models that choice as profit maximization: income equals the expected publishable rate times grant income per publication, minus sampling cost, and the sample size at which profit peaks is the 'equilibrium sample size'. The model predicts that the equilibrium sample size shrinks when true effects are rare, when effect sizes are small, or when grant income per publication is low. For parameter distributions the paper judges plausible, the model robustly produces a bimodal distribution of statistical power and positive predictive values around 0.26 to 0.40, below the 50 percent reproducibility benchmark. A reader should care because this would mean underpowering is a structural economic equilibrium, not merely sloppy methodology.","feed_headline":"Underpowered studies can be the rational economic choice","feed_subtitle":"When only positive results pay, profit-maximizing scientists choose samples too small to replicate.","key_machinery":"The load-bearing object is the profit identity $\\text{Profit}(s,IF,d,b)=IF\\times TPR(s,d,b)-s$, with $TPR(s,d,b)=\\alpha(1-b)+(1-\\beta(s))b$. $TPR$ is the total publishable rate at which a study yields a publishable positive result: false positives occur at rate $\\alpha$ among null hypotheses, and true positives occur at rate $1-\\beta(s)$ among true hypotheses. The economic insight comes from the curve's shape: income saturates as power approaches 1, but sample cost is linear, so profit has an interior maximum. That maximum is the equilibrium sample size, and its location is the argument: lower $b$, lower $d$, or lower $IF$ all move it toward the minimum, making underpowering the rational response.","core_discovery":"The paper's central claim is formal: with positive publication bias and a fixed cost per sample, a researcher's profit is $\\text{Profit}(s,IF,d,b)=IF\\times TPR(s,d,b)-s$, where $TPR(s,d,b)=\\alpha(1-b)+(1-\\beta(s))b$ is the total publishable rate, $\\alpha=0.05$ is the type-1 error, and $1-\\beta(s)$ is statistical power. The equilibrium sample size $ESS$ is the sample size that maximizes this profit. Because statistical power saturates as samples increase while cost grows linearly, the profit curve is concave, and anything that lowers the marginal publishing value of more samples—rarer true hypotheses, smaller effects, or lower income per publication—pushes the $ESS$ left. Simulating the model over distributions the paper considers plausible yields a bimodal distribution of achieved power and mean positive predictive values of 0.26 to 0.40 under low base rates, matching empirical reproducibility rates; with a uniform base rate near 0.5, the same machinery yields positive predictive value near 0.8. The paper additionally claims that conditional equivalence testing, which makes significant negative findings publishable, raises equilibrium power and positive predictive value above 90 percent for most tested distributions.","pith_inferences":["A natural extension is to fit the predicted two-component power distribution to a large corpus of published t-tests and compare fits against a single-peaked scientifically normative model; the relative fit would estimate how much of observed sample-size behavior is economic.","The same profit logic might apply to other effort margins, such as number of conditions, measurement depth, or replication attempts, since any costly input that saturates in publishable yield should be underprovided.","If conditional equivalence testing became standard, the model implies a new gaming margin: researchers could inflate equivalence bounds to secure income from negative findings, so pre-registered bounds would be needed to keep the incentive honest.","The bimodality prediction suggests that meta-analyses pooling power across fields may be averaging two distinct regimes; stratifying by novelty versus confirmatory orientation would sharpen the test."],"forward_implications":["Policies that raise mean grant income per publication should shift equilibrium sample sizes upward and improve statistical power.","Supporting more confirmatory research, which raises the base probability $b$, is predicted to be a direct lever on reproducibility.","The model predicts a bimodal distribution of achieved power, so the absence of a mode near 80 percent power is a predicted signature, not a puzzle.","Conditional equivalence testing should improve power and positive predictive value for most fields, often above 90 percent, by making negative results publishable.","Because high-novelty journals are assumed to have lower $b$, the model predicts they will contain smaller samples than confirmatory journals."],"supporting_citations":[{"why":"Supplies the pre-study probability and positive predictive value formalism that the profit calculation builds on.","marker":"[8]"},{"why":"Documents the empirically low power in neuroscience that motivates the model and provides the empirical baseline for its predictions.","marker":"[10]"},{"why":"Provides the empirically matched distribution of published effect sizes used as input for the power-distribution simulations.","marker":"[11]"},{"why":"Supplies the cultural-evolution and competitive-selection argument that lets the paper interpret profit maximization as an equilibrium rather than deliberate choice.","marker":"[13]"},{"why":"An earlier optimality model with similar conclusions; the paper positions itself as a simpler, more empirically tractable version.","marker":"[24]"},{"why":"A complementary optimality model exploring consequences of stricter statistical stringency; used as contrast and support for economic-pressure explanations.","marker":"[25]"},{"why":"Defines conditional equivalence testing, the procedure the paper models as a way to realign economic incentives.","marker":"[26]"}],"fun_headline_variants":["Why rational scientists choose tiny sample sizes","Profit-seeking scientists pick small samples rationally","Economically rational sample sizes drive irreproducibility","Small samples: the rational choice when only positives count"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The quantitative claim that reproducibility falls below 50 percent depends on assuming most hypotheses are unlikely to be true, with $b$ concentrated near 0.1; the paper chooses that distribution from plausibility arguments rather than measurement, and if $b$ were actually near 0.5 the same model gives reproducibility near 80 percent.","fun_headline_variants_meta":{"raw":{"variants":["Why rational scientists choose tiny sample sizes","Profit-seeking scientists pick small samples rationally","Economically rational sample sizes drive irreproducibility","Small samples: the rational choice when only positives count"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000791,"raw_usage":{"total_tokens":3539,"prompt_tokens":1049,"completion_tokens":2490,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":665,"completion_tokens_details":{"reasoning_tokens":2433}},"tokens_in":665,"tokens_out":2490,"duration_ms":18047,"temperature":1.0,"reasoning_tokens":2433,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:32:11.354821+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the actual distribution of $b$ across hypotheses in a field, for example by tracking preregistered predictions that are later confirmed, and measure the actual relationship between sample size, effect size, and per-publication funding. If the base-rate distribution were centered near 0.5, the model's own simulations predict a positive predictive value near 0.8 rather than below 0.5, so the headline low-reproducibility prediction fails. Alternatively, a large corrected dataset showing no positive correlation between sample size and true effect size, or no effect of funding per publication on sample size, would contradict the mechanism.","supporting_citations":[{"cited_title":"Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature","cited_arxiv_id":null,"evidence_quote":"Provides the empirically matched distribution of published effect sizes used as input for the power-distribution simulations."},{"cited_title":"Current Incentives for Scientists Lead to Underpowered Studies with Erroneous Conclusions","cited_arxiv_id":null,"evidence_quote":"An earlier optimality model with similar conclusions; the paper positions itself as a simpler, more empirically tractable version."},{"cited_title":"Conditional equivalence testing: An alternative remedy for publication bias","cited_arxiv_id":null,"evidence_quote":"Defines conditional equivalence testing, the procedure the paper models as a way to realign economic incentives."}],"review_version":1}