{"id":"02aa54d2-bfd1-464c-97cd-383a5c719bd4","arxiv_id":"2504.12860","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Randomizing split directions in forests helps when variance dominates bias, which occurs at low signal-to-noise ratio and with correlated covariates, but hurts when irrelevant covariates or fat-tailed features make bias dominant.","lead":"This paper uses simulation experiments to compare random forests against bagging, asking when randomizing split directions helps prediction. It finds that low signal-to-noise favors forests, while correlated covariates reduce prediction bias for both methods.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Bias-variance dominance mechanism is decomposition-dependent; the alternative decomposition (7), introduced in Section 2.1, is never checked.","rationale":"The reader's weakest assumption was representativeness of the three DGPs and default hyperparameters, which is a valid external-validity concern. I am raising a different, more internal concern: the paper's explanatory mechanism is tied to a specific bias-variance decomposition, and the authors themselves demonstrate that the alternative decomposition is equally valid and can give different bias/variance terms. This matters because the headline claim is not merely that forests outperform bagging at low SNR—that is an empirical MSE comparison—but that this happens 'because variance dominates bias for both methods.' The dominance statement is a claim about the decomposition, and it is not checked under decomposition (7). The reader's representativeness concern is secondary because the paper's 'when' claims are already explicitly conditional on the studied DGPs, whereas the 'why' claims are presented as a general mechanism. The paper deserves credit for being transparent about the non-uniqueness in Section 2.1 and for the invariant relative-difference result in Proposition 3, but the omitted robustness check leaves the central explanatory claim underdetermined. The verdict remains CONDITIONAL, with the additional condition that the authors show their dominance conclusions are invariant to the two decompositions they themselves identify.","tokens_in":22471,"tokens_out":11407,"duration_ms":117818,"concrete_test":"Recompute the squared-bias and variance entries for Tables 1-6 using decomposition (7): for each of the 500 training sets and each method, compute mu_i = (1/M) sum_{m=1}^{M} (f(x_m) - fhat_i(x_m)) and v_i = (1/M) sum_{m=1}^{M} (f(x_m) - fhat_i(x_m) - mu_i)^2 on the fixed 10,000-point test set, then average mu_i^2 and v_i over i. If the qualitative conclusions (variance > squared bias for both methods at low SNR; randomization increases bias and reduces variance) survive, the mechanism is robust. If the ordering flips, the explanation is decomposition-dependent and the central claim needs qualification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central explanation is that randomization improves MSE 'exactly when variance dominates squared bias' (e.g., at low SNR and with correlated covariates). This dominance is measured under decomposition (6), which conditions on X first. However, Section 2.1 introduces a second, equally valid decomposition (7) that conditions on the trained model first, and explicitly states that the two decompositions can disagree about which term is larger. All bias and variance tables (Tables 1-6 and 8-13) are computed only under decomposition (6), so the claims that 'variance dominates bias for both methods' at low SNR, and that randomization works because it reduces this dominant variance, are not established under decomposition (7). Since the authors themselves flag the non-uniqueness but do not test robustness, the explanatory mechanism—which is the main novelty of the paper—lacks support even within their own DGPs. The MSE comparisons themselves are invariant and reproducible in principle, but the 'because' in the abstract is conditional on an arbitrary choice of decomposition.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper asks under what data conditions randomizing split directions in random forests improves out-of-sample MSE relative to bagging, and uses bias–variance decompositions to explain the differences. It states three elementary propositions: an unconditional bias–variance decomposition conditioning on X (Eq. 6), an alternative decomposition conditioning on the fitted model (Eq. 7), a large-B variance formula in terms of tree correlation (Proposition 2), and invariance of relative MSE difference to normalization of the regression function (Proposition 3). The simulation study replicates three DGPs (N-LINEAR, U-MARS, U-HIDDEN) from Hastie et al. (2009), Mentch and Zhou (2020), and Liu and Mazumder (2024) at SNR 0.05, 1, and 6, then extends the setup to fat-tailed covariate distributions, irrelevant covariates, and correlated covariates at SNR=1, with 500 training sets and one 10,000-point test set. The main findings are that randomization decorrelates trees in all settings, that forests beat bagging at low SNR because variance dominates bias, and that with correlated covariates bias shrinks for both methods so that randomization is effective; with fat-tailed covariates and irrelevant covariates, bias increases and randomization tends to hurt.","tokens_in":22591,"tokens_out":5750,"duration_ms":59112,"significance":"If the empirical conclusions hold up, the paper makes a useful conceptual contribution by shifting the discussion from SNR alone to the bias–variance balance and by documenting that correlated covariates reduce bias in both bagging and forests, an observation that goes beyond the standard variance-reduction story. The paper's strengths are that its propositions are elementary, correctly proved, and clearly stated; the simulation protocol is careful and reproducible, with 500 training sets, a single large test set, explicit SNR levels, and the default randomForest implementation; and Proposition 3 provides a clean normalization-invariance result for the reported relative differences. The study is, however, limited to three benchmark DGPs, one set of default hyperparameters, and a single decomposition choice, which restricts how far the 'because' clauses in the abstract can be generalized.","major_comments":[{"comment":"The paper's central explanatory claims, such as 'forests outperform bagging when the SNR is low because variance dominates bias for both methods' and 'randomization tends to be effective [with correlated covariates] because variance dominates bias', are evaluated exclusively under decomposition (6). The authors introduce decomposition (7), conditioning on the fitted model, and explicitly state that the two decompositions can disagree about whether bias or variance dominates and about which term drives an MSE improvement. No table in the manuscript reports the bias and variance terms under (7), so the asserted mechanism is not robust to the decomposition choice. Since the non-uniqueness is acknowledged in Section 2.1, the manuscript should either report both decompositions for the key comparisons (at least Tables 1, 4, 5, and 6) or provide a substantive argument for why (6) is the relevant conditioning for the causal statements; without this, the 'because' claims are conditional on an arbitrary convention.","section":"Section 2.1, Eq. (7)"},{"comment":"The bias and variance components are reported as point estimates without Monte Carlo standard errors. The t-statistic of Eq. (16) is applied only to the difference in MSE, not to the bias and variance components. Several qualitative conclusions rest on small component differences, such as Table 6's N-HIDDEN bias terms (0.01 vs 0.02) and Table 4's LINEAR relative difference of 0.37%. Without standard errors or confidence intervals for the bias and variance terms, the claims that 'bias dominates variance' or that 'bias increases by more for forest' cannot be distinguished from estimation noise. Please add simulation standard errors or paired intervals for the bias and variance columns, at least for the key tables in Sections 3 and 4.","section":"Sections 3-4, Tables 1-13"},{"comment":"All extensions fix n=250, mtry=floor(p/3), node size 5, and 500 trees. Because the paper's title and abstract make general claims about when random forests work, the absence of any variation in mtry or tree depth leaves open whether the bias-variance balance findings are artifacts of these defaults. The mtry value directly controls the bias-variance tradeoff in Proposition 2, so the qualitative conclusions could change under other settings. A small sensitivity analysis over mtry (or at least a clearly bounded claims statement in the conclusion) would make the stated generality match the evidence actually presented.","section":"Section 4, first paragraph"}],"minor_comments":[{"comment":"The statement 'Why increasing rho can result in forests even outperforming bagging, as happens for rho=0.9 in this example, is something we cannot explain' is an honest limitation, but it leaves a headline-adjacent observation without any diagnostic follow-up; a brief speculation or a plot of the conditional bias and variance at rho=0.9 would help the reader understand the reversal.","section":"Section 4.3"},{"comment":"The table header lists the covariate distribution for three models in a single row, which is compact but hard to parse because the model names and distributions are not repeated in separate columns; consider restructuring the table or adding a clearer caption.","section":"Table 4"},{"comment":"The text describes these figures in detail but the figures themselves are not reproduced in the manuscript text provided; if they appear in the full submission, please ensure each panel has clear axis labels and a legend so the conditional MSE differences are interpretable.","section":"Figures 1 and 2"},{"comment":"The sentence 'What we've observed in prior literature is that the average prediction at a given value is used as a proxy for the true regression function' would benefit from a citation and a precise definition of the proxy, since the claim that the bias term is 'effectively eliminated' depends on whether the proxy is exactly the conditional mean.","section":"Section 2.1"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a careful simulation study, and the parts that are new are worth keeping. It should go to peer review, not be desk-rejected. The three propositions are elementary and correct; the simulation protocol (500 training sets, one large test set, explicit SNR levels, normalized invariance check) is described carefully enough to reimplement, even though no code is shipped. The replications of Hastie et al., Mentch and Zhou, and Liu and Mazumder are faithful, and the bias-variance tables add something those papers didn't show.\n\nThe genuinely new piece is Section 4: fat-tailed covariates, irrelevant covariates, and especially correlated covariates. The claim that correlated covariates reduce bias for both bagging and forests, and that randomization then tends to help because variance dominates, is the main contribution. It is empirical and limited to three DGPs and default hyperparameters, but the pattern is consistent and worth testing elsewhere.\n\nThe soft spots are real but not fatal. The biggest is one the authors themselves flag. They introduce a second valid decomposition (7) and explicitly say it can disagree with (6) about which term dominates, then compute every table only under (6). So 'variance dominates bias'—and the 'because' in the abstract—is true only under that choice. The MSE comparisons themselves are invariant; the explanation is not. They should either report the key tables under (7) or soften the causal language. This is a moderate fix, not a load-bearing flaw.\n\nSmaller: no standard errors on the bias/variance components, and no code. With 500 training sets the MSE differences have test statistics, but the components are read off as point estimates. The title is a bit broad for three DGPs with fixed mtry and node size. I would not over-penalize this; the paper is explicitly a simulation study.\n\nWho gets value: applied statisticians deciding between bagging and forests, and anyone working on why forests work. The correlated-covariate result is a useful empirical data point, possibly a stepping stone for theory.\n\nRecommendation: send to peer review. Ask for the alternative decomposition check, code, and standard errors on components; with those, the paper is publishable as an applied/ML-statistics contribution.","headline":"Worth a real refereeing; the correlated-covariate finding is new and plausible, but the paper's explanation leans entirely on one of two equally valid bias-variance decompositions and never checks the other.","tokens_in":23142,"tokens_out":2594,"would_cite":true,"duration_ms":27966,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62G08","68T05"],"pacs":[],"model":"deepseek-v4-flash","headline":"Randomization wins exactly when variance beats squared bias","keywords":["random forests","bagging","split randomization","bias-variance decomposition","signal-to-noise ratio","correlated covariates","decorrelation","regression trees"],"falsifier":"Run the same simulations at a fixed moderate SNR with a new regression function and correlated covariates: if forests beat bagging while the bias-variance decomposition shows squared bias still larger than variance, or if increasing correlation raises rather than lowers bias for both methods, the paper's mechanism fails.","tokens_in":22248,"feed_emoji":"🌲","tokens_out":5949,"duration_ms":60903,"temperature":0.7,"pith_summary":"The paper asks when the split-direction randomization that defines random forests improves on bagging, and answers in terms of which term of a bias-variance decomposition dominates. Across simulations on three standard regression settings—a linear model, the MARS benchmark, and a narrow-jump hidden pattern—randomization always decorrelates trees and lowers ensemble variance, but it usually raises squared bias. Forests improve on bagging exactly in the regimes where the variance reduction outweighs the bias increase: at low signal-to-noise ratio, and at moderate SNR when covariates are mutually correlated. A second finding is that correlated covariates reduce squared bias substantially for bagging and forests alike, so averaging helps through bias reduction and not only variance reduction.","feed_headline":"Randomization wins exactly when variance beats squared bias","feed_subtitle":"Correlated covariates cut bias for both bagging and forests, which explains real-world random forest gains.","key_machinery":"The argument rests on three formal pieces. Proposition 1 fixes an unconditional out-of-sample MSE and writes it as squared bias plus variance plus irreducible noise, conditioning on covariates first. Proposition 2 shows that, for a large ensemble, variance equals the pairwise correlation between two trees times the variance of a single tree, which isolates the decorrelation effect of randomization. Proposition 3 proves that the relative difference in MSE between bagging and forests is unchanged when the regression function is normalized to unit variance. Together they reduce the question \"does randomization help?\" to a comparison of two numbers: the variance saved by decorrelation against the squared bias added by restricted splits.","core_discovery":"The paper's central claim is that split randomization is effective precisely when variance dominates squared bias, and that this can happen for reasons unrelated to noise level. Using three benchmark data-generating processes, the authors show that randomization decorrelates trees in every scenario and always lowers ensemble variance, yet it increases squared bias in most scenarios; forests beat bagging in low-SNR settings because variance is the larger term, and lose in high-SNR settings because bias dominates. At fixed moderate SNR, covariate structure flips the balance: fat tails and irrelevant covariates push bias up, making randomization ineffective, while mutual correlation among covariates lowers bias for both methods and lets forests beat bagging. The paper further proves that the relative MSE gap between bagging and forests is invariant to rescaling the regression function, and that the bias-variance split depends on the order of conditioning, so comparisons must state which decomposition they use.","pith_inferences":["If correlated covariates reduce the harm of missing a good split, then the same mechanism should strengthen other randomized ensembles, such as extremely randomized trees; that is a testable extension the paper does not run.","The results suggest a practical diagnostic for a given dataset: estimate the squared-bias and variance terms on validation splits, and choose bagging versus forest by which term dominates, rather than by SNR alone.","The normalization-invariance result implies percentage MSE gaps reported under any scaling of the regression function are comparable, which could make cross-paper comparisons of forest-versus-bagging gains more meaningful.","A natural next simulation is to replace equicorrelated covariates with block or autocorrelated structures and check whether the bias-reduction effect persists when correlation is uneven, since real data rarely have constant pairwise correlations."],"forward_implications":["At low signal-to-noise ratio, forests consistently outperform bagging on all three benchmark DGPs, because variance dominates squared bias for both methods.","At high SNR, bagging tends to win by a larger margin than forests win at low SNR, so the cost of using randomization where it does not help is asymmetric.","Fat-tailed covariates make split randomization worse: the squared-bias increase appears mainly in the tails of the covariate distribution.","Adding irrelevant covariates raises bias for both methods, and randomization makes it worse, so bagging outperforms forests once enough irrelevant covariates are present.","Mutual correlation among covariates reduces squared bias for bagging and forests, so forests beat bagging at moderate SNR; with perfect correlation the two methods coincide."],"supporting_citations":[{"why":"Defines bagging, the data-sampling baseline that the paper compares forests against.","marker":"(Breiman, 1996)"},{"why":"Introduces random forests and the decorrelation argument that randomization helps by lowering tree correlation.","marker":"(Breiman, 2001)"},{"why":"Formalizes the ensemble variance as correlation times tree variance and supplies the N-LINEAR example and bias-variance tradeoff being revisited.","marker":"(Hastie et al., 2009)"},{"why":"Provides the regularization explanation and low-SNR comparison, plus the U-MARS DGP used for replication.","marker":"(Mentch and Zhou, 2020)"},{"why":"Shows forests can reduce bias via split randomization in a hidden-pattern DGP, the claim the paper goes beyond with correlated covariates.","marker":"(Liu and Mazumder, 2024)"}],"fun_headline_variants":["When variance beats bias, forests beat bagging","Correlated covariates cut bias, boosting forests","Randomization works when variance dominates","Bias-variance split depends on conditioning order","Rescaling doesn't change forest vs bagging gap"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusions are drawn from three simulated regression functions with default forest settings (mtry = floor(p/3), node size 5, 500 trees), so the qualitative claims assume these benchmarks represent behavior on other data-generating processes and hyperparameter choices.","fun_headline_variants_meta":{"raw":{"variants":["When variance beats bias, forests beat bagging","Correlated covariates cut bias, boosting forests","Randomization works when variance dominates","Bias-variance split depends on conditioning order","Rescaling doesn't change forest vs bagging gap"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000929,"raw_usage":{"total_tokens":4002,"prompt_tokens":989,"completion_tokens":3013,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":605,"completion_tokens_details":{"reasoning_tokens":2944}},"tokens_in":605,"tokens_out":3013,"duration_ms":22767,"temperature":1.0,"reasoning_tokens":2944,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:20:08.743561+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same simulations at a fixed moderate SNR with a new regression function and correlated covariates: if forests beat bagging while the bias-variance decomposition shows squared bias still larger than variance, or if increasing correlation raises rather than lowers bias for both methods, the paper's mechanism fails.","supporting_citations":[{"cited_title":"Randomization as regularization: A degrees of freedom explanation for random forest success","cited_arxiv_id":null,"evidence_quote":"Provides the regularization explanation and low-SNR comparison, plus the U-MARS DGP used for replication."},{"cited_title":"Randomization Can Reduce Both Bias and Variance: A Case Study in Random Forests","cited_arxiv_id":"2402.12668","evidence_quote":"Shows forests can reduce bias via split randomization in a hidden-pattern DGP, the claim the paper goes beyond with correlated covariates."}],"review_version":1}