{"id":"6bce0648-0c48-4d59-ba5b-6769f5e30270","arxiv_id":"2505.14312","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A regime-stratified benchmark of 196 tabular datasets shows that model rankings depend strongly on dataset characteristics such as sample size, feature correlation, and label imbalance.","lead":"The paper introduces MultiTab, a benchmark of 196 tabular datasets and 13 models that evaluates performance within data regimes defined by sample size, label imbalance, feature interaction, and other statistics. Regime-aware analysis reveals that no single model wins everywhere: models using sample-level similarity lead on large or highly correlated datasets, while feature-attention models lead on weakly correlated ones.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The regime rankings rest on a per-split min-max normalized error that can turn noise-level differences into 'excels' claims; split-level averaging over non-independent folds makes the reported confidence intervals unsound.","rationale":"Good-faith reading: the paper is a benchmark artifact paper. Its central claim is not that a single model is state-of-the-art, but that regime-conditioned rankings differ and are stable enough to guide model selection. The strongest evidence is the sub-category analysis in §4. For that evidence to carry the claim, the evaluation metric must preserve meaningful performance differences and the statistical comparisons must be valid across datasets. Both conditions are insecure. The min-max normalization is especially dangerous because it is applied per dataset-split: it removes dataset difficulty, but it also removes the magnitude of differences. A model can receive a large normalized penalty for being worst on a split where all 13 models are within 0.001 of each other; another can receive a perfect score for being best by 0.0001. Averaging these normalized scores across hundreds of splits makes rankings reflect consistency in relative position more than actual predictive quality. This directly threatens the specific examples in the abstract, which are claims about which model is best in a regime. The split-level aggregation also inflates the effective sample size: 196 datasets become roughly 620 small-dataset splits plus 204 large-dataset splits, and correlated splits from the same dataset are treated as independent. The confidence intervals are therefore too narrow and the 'within confidence interval' statements are unreliable. I do not think the paper is fraudulent or the findings definitely false; the authors provide public code and logs, a consistent 100-trial TPE protocol, and a large dataset collection, which are real contributions. But the central empirical claim is conditional on a metric that is not validated against raw error and on a statistical aggregation that is not dataset-level. The reader's weakest_assumption identified the same normalization issue; I agree and would keep the CONDITIONAL verdict, with the concrete reanalysis as a condition.","tokens_in":34904,"tokens_out":5040,"duration_ms":53965,"concrete_test":"Recompute the headline regime comparisons (Table 2, Figures 1–2) under two alternative analyses: (1) aggregate raw log loss/RMSE per dataset first (averaging folds), then rank models within each dataset and average ranks within each sub-category; (2) bootstrap at the dataset level (resample datasets, not folds) and test whether NN-Sample is actually better than NN-Feature/GBDTs on large-sample and high-correlation regimes and whether NN-Feature is better on weakly correlated regimes, using paired tests with multiple-comparison correction. If either NN-Sample's or NN-Feature's advantage disappears under both alternatives, the central claim is an artifact of per-split min-max normalization and split-level averaging.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central regime-level claims in §4.2 (NN-Sample excels on large-sample/high-correlation datasets; NN-Feature excels on weakly correlated datasets) are established exclusively through the normalized predictive error defined in §3.4 and Appendix B.3. For each dataset-split pair, raw error is rescaled as (e - e_min)/(e_max - e_min) using only the 13 models evaluated on that split, then averaged across all dataset-split pairs. This makes the score a relative rank-like quantity within each split: a model that is best by a tiny, statistically negligible margin receives 0, while a model that is worst by a tiny margin receives 1. Averaging these scores across heterogeneous datasets therefore amplifies systematic small margins and outlier collapses into large normalized differences, so the reported 'best-performing' labels and regime effects may not correspond to meaningful absolute improvements. The problem is compounded by the aggregation unit: each split is a data point, so small datasets contribute 10 folds and large datasets contribute 3, and fold-level scores from the same dataset are correlated. The 95% confidence intervals and 'statistically comparable' statements in Table 2 and Figure 1 are computed over these non-independent split-pairs, not over datasets, so they overstate significance. The paper's secondary metrics (Appendix E.1) are not used to validate the primary claims, leaving the regime rankings unvalidated against raw performance.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MultiTab, a benchmark suite and evaluation framework for tabular learning that moves beyond average-case rankings. It collects 196 publicly available datasets, evaluates 13 models spanning classical methods, GBDTs, and neural networks with different inductive biases (sample-level similarity, feature-level attention, both, or neither), and partitions datasets along seven axes (task type, sample size, feature heterogeneity, feature-to-sample ratio, label imbalance, function irregularity, and feature interaction) using heuristic thresholds. The primary metric is a per-dataset-split min-max normalized predictive error, averaged across dataset-split pairs. The central empirical claim is that model performance is regime-dependent: for example, NN-Sample models excel on large-sample or high-correlation datasets, while NN-Feature models excel when features are weakly correlated, and that these patterns are invisible under aggregate evaluation. The paper also reports Spearman correlations between dataset statistics and model error, a comparison with TabPFN on 42 eligible datasets, and extensive hyperparameter optimization logs.","tokens_in":1741,"tokens_out":2029,"duration_ms":56976,"significance":"If the results are robust, MultiTab would be a valuable community resource: it is large in scale (196 datasets, 13 models, 100 TPE trials per model-dataset pair with per-fold optimization), it is released with optimization logs, and it targets a real gap in the literature by proposing conditional, regime-aware evaluation instead of single-number averages. The paper also deserves credit for attempting to connect architectural inductive biases to dataset statistics in a falsifiable way, and for including regression tasks and multiple complementary metrics (entropy ratio, skewness, imbalance factor) for label imbalance. However, the central claims currently rest on a normalized-error metric whose validity is not demonstrated, and on split-level aggregation that makes the reported confidence intervals unreliable; these issues must be addressed before the regime-level conclusions can be accepted.","major_comments":[{"comment":"The primary metric is the per-dataset-split min-max normalized error, \\hat e_{m,d} = (e_{m,d} - e_min_d)/(e_max_d - e_min_d), computed over the 13 models evaluated on that split. This is a relative, rank-like score: a model that is best by a tiny margin receives 0 and a model that is worst by a tiny margin receives 1, so averaging these scores across heterogeneous datasets can amplify negligible raw-error differences into the \"excels\" claims in §4.2. Since the raw log loss/RMSE and average-rank results in Appendix E.1 are not used to validate the regime-level conclusions, the central findings currently rest on a metric that may not reflect meaningful absolute improvements. I request a sensitivity analysis: report the raw-error margins for the highlighted comparisons (e.g., ModernNCA vs XGBoost in Table 2), and re-run the key regime analyses with raw log loss/RMSE or average rank as the primary outcome.","section":"§3.4, Appendix B.3"},{"comment":"Aggregation is performed over dataset-split pairs, not datasets: each dataset contributes 10 folds (or 3 folds for large datasets), and fold-level scores from the same dataset are correlated because they share samples and validation strategies. The 95% confidence intervals and \"statistically comparable\" statements in Table 2 and Figure 1 are therefore computed over non-independent units and will overstate significance, while also weighting each small dataset more heavily than each large dataset. A dataset-level analysis (for example, averaging over folds first and then using paired tests or bootstrap over datasets) is needed to support the significance claims.","section":"§3.3–3.4, Table 2, Figure 1"},{"comment":"There are direct inconsistencies in the reported sub-category definitions. Table 1 states function-irregularity thresholds of 0.25 (Regular) and 0.95 (Irregular) with 38 and 43 datasets, while Appendix C.6 uses 0.3 and 0.7 with 58 and 43 datasets. For entropy ratio, Table 1 reports Balanced >0.7 with 43 datasets and Imbalanced <0.3 with 61, whereas Appendix C.5.1 reports 61 and 43, respectively. These differences change the dataset composition of the regimes and make it impossible to reproduce Table 2 and Figure 2 from the appendix alone. Please harmonize the thresholds and counts, and state which definitions generated the reported results.","section":"Table 1 vs Appendix C"},{"comment":"The preprocessing section excludes datasets with categorical features having more than 1,000 unique values \"due to frequent training failures across models.\" The number and identity of excluded datasets are not reported, and no sensitivity analysis is given. Because the excluded datasets are likely concentrated in exactly the high-cardinality categorical regime analyzed in §4.2.3, this exclusion could bias the very comparison it is used to support. Please report the excluded datasets and assess whether the categorical-cardinality conclusions change when they are handled through a different encoding or excluded with a documented justification.","section":"Appendix B.1, §3.1"}],"minor_comments":[{"comment":"The caption states \"Blue (negative) indicates worse-than-average performance; red (positive) indicates better-than-average,\" but a negative deviation from the overall mean error should indicate better-than-average performance. Please clarify the color/sign convention and make the text in §4.2.3 consistent with it.","section":"Figure 2"},{"comment":"There are typos in the hyperparameter tables: \"d_embedings\" and \"frequency_sclae\" in the ModernNCA table should be \"d_embeddings\" and \"frequency_scale.\" The text in §D.5.2 also says \"d_embedings\" in the paragraph preceding Table 12.","section":"Appendix D.5, Tables 12 and 13"},{"comment":"The sentence \"where as ModernNCA often achieves larger improvements\" should read \"whereas.\" Also, Appendix E.3 contains a reference to \"Figure ??\" that should be replaced with the actual figure number.","section":"Appendix E.4"},{"comment":"The statement that \"All datasets are distributed under the CC-BY license\" is likely too strong for OpenML datasets, which have heterogeneous licenses. Please verify the license of every dataset or soften the claim to describe the verified subset.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The paper is a well-scoped benchmark contribution and the public release of datasets, code, and optimization logs is a strength. The main risk is that the headline regime-level findings are driven by the normalization and aggregation choices rather than by robust raw-error differences; the authors should be asked to re-analyze the key claims at the dataset level with raw metrics. The threshold inconsistencies between Table 1 and Appendix C need to be fixed in the same revision. This is not a rejection because the issues are addressable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful thing here is the artifact: 196 datasets, 13 models, standardized training protocol per fold, public code and logs. That is real work and it gives the community something to build on. The multi-axis stratification (sample size, feature interaction, function irregularity, etc.) goes beyond average-case leaderboards, and the specific finding that NN-Sample does well on correlated features while NN-Feature does well on weakly correlated ones is a genuinely new empirical observation worth testing.\n\nThe soft spots are the ones the attached notes flag, and they are load-bearing if the goal is to justify the regime rankings. The min-max normalized error in Section 3.4 rescales per dataset-split pair, so a model that is best by a tiny margin gets 0 and the worst by a tiny margin gets 1. Averaging those scores across heterogeneous datasets can amplify noise-level differences into 'excels' claims. The confidence intervals are computed over split-pairs, not datasets, so the folds from the same dataset are correlated and the significance is overstated.\n\nThe internal inconsistencies are also real: Table 1 gives function irregularity thresholds 0.25/0.95 while Appendix C uses 0.3/0.7, and the entropy-ratio dataset counts are swapped (43 vs 61). That does not kill the paper, but it makes the sub-category construction look sloppy and needs fixing. The exclusion of datasets with training failures could bias regime comparisons, though Appendix F does at least acknowledge it as a limitation.\n\nTo the paper's credit, the authors ship secondary metrics (accuracy, rank) in Appendix E.1, and they say these 'largely align' — but they do not systematically validate the primary claims against raw performance. That is the missing piece. If the regime findings survive a check on raw error or average rank with proper dataset-level aggregation, they would be solid.\n\nWho is this for? Practitioners who want a map for model selection across data regimes, and researchers working on tabular architectures. The suite itself is worth having regardless of the normalization debate. A serious referee should see this, but the revision needs to either justify the normalization or re-run the core comparisons on raw metrics with dataset-level confidence intervals.","headline":"A genuinely reusable benchmark artifact whose headline regime findings are plausible but currently rest on a fragile relative-error metric and some internal inconsistencies.","tokens_in":35740,"tokens_out":1135,"would_cite":true,"duration_ms":13847,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Tabular model performance flips with data regime, and average benchmarks hide it.","keywords":["tabular data","benchmark","data regimes","normalized predictive error","inductive bias","sample-level similarity","feature interaction","model selection"],"falsifier":"Recompute the per-regime rankings using raw errors or average ranks instead of normalized scores, and check whether the reported patterns, NN-Sample best on large and correlated data and NN-Feature best on weakly correlated data, persist under both metrics and across splits with different dataset compositions per sub-category; if the ordering flips, the central claim is an artifact of normalization.","tokens_in":34686,"feed_emoji":"📊","tokens_out":4297,"duration_ms":41565,"temperature":0.7,"pith_summary":"The paper argues that how well a tabular model performs depends strongly on the statistical shape of the dataset, and that average-case benchmarks hide these conditional strengths. It builds MultiTab, a suite of 196 publicly available datasets evaluated with 13 models, and groups the datasets along seven data-characteristic axes such as sample size, label imbalance, and feature correlation. The central finding is a stable regime sensitivity: models built on sample-level similarity win on large or highly correlated datasets, feature-attention models win on weakly correlated features, and gradient-boosted trees remain strong on regression and low-imbalance tasks. If the finding is right, model selection and architecture design should be data-regime-aware, and benchmark reporting should include conditional results rather than a single global ranking.","feed_headline":"Tabular model rankings flip across data regimes","feed_subtitle":"On 196 datasets, sample-based models win on large correlated data; feature-attention models win when features are independent.","key_machinery":"MultiTab's sub-categorization framework partitions 196 datasets along seven axes, each measured by quantitative statistics: task type, sample size, feature heterogeneity (proportion and cardinality of categorical features), feature-to-sample ratio, label imbalance (entropy ratio, skewness, imbalance factor), function irregularity (a frequency-based high-frequency energy ratio), and feature interaction (Frobenius norm of correlation and minimum eigenvalue of the covariance matrix). Models are grouped into six inductive-bias families. The primary metric is normalized predictive error: within each dataset-split pair, each model's error is linearly rescaled between the best and worst errors to a value in [0,1], then averaged across splits, which is meant to make relative performance comparable across datasets of very different difficulty.","core_discovery":"The central claim is that model performance on tabular data is highly sensitive to data regimes, and that the regime-specific rankings are systematic rather than noise. Specifically, the paper reports that models relying on sample-level similarity excel on datasets with large sample sizes or high inter-feature correlation, while models encoding inter-feature dependencies perform best when features are weakly correlated; gradient-boosted trees dominate regression and low-imbalance settings. The paper also finds that inductive biases do not always behave as intended, citing cases where a model designed for both feature and sample dependencies underperforms specialized architectures, and where a tree-based model known for robustness degrades sharply under function irregularity. These patterns support the conclusion that average-case rankings are insufficient for tabular model evaluation.","pith_inferences":["The normalized-error protocol may reward models that are consistently close to the best on each dataset, so the regime patterns could partly reflect compression of error scales; checking the same claims with raw ranks or fold-level statistical tests would sharpen them.","The dataset axes are not independent, so the attribution of performance differences to a single characteristic such as sample size may be confounded with feature-to-sample ratio or correlation structure; synthetic datasets that vary one axis at a time could test the claimed mechanisms.","The reported TabPFN comparison suggests pretrained tabular models may occupy a niche in imbalanced small-scale classification, which is a testable prediction for tabular foundation models as they scale.","The 'inductive bias does not behave as intended' finding implies that reporting per-family averages can be misleading; future conditional evaluations should report per-model results alongside family-level summaries."],"forward_implications":["Practitioners can choose a model based on dataset statistics rather than relying on global benchmark rankings.","Benchmark suites should report performance across data regimes as a standard practice, not only aggregate averages.","The identified weak spots, such as neural models degrading under skewed regression targets and all models struggling on highly irregular functions, point to concrete gaps for architectural improvement.","Models that exploit sample-level similarity are a reasonable default for large, highly correlated datasets, whereas feature-attention models suit weakly correlated, high-dimensional data.","Inductive-bias labels alone do not predict success; implementation-level behavior within a model family can diverge sharply across regimes."],"supporting_citations":[{"why":"Supplies the prior finding that tree-based models outperform neural networks on typical tabular data, which MultiTab extends and qualifies by showing regime-dependent reversals.","marker":"[20]"},{"why":"Provides the large-scale rank-based benchmark and the 'no dominant model' conclusion that MultiTab reframes through sub-category analysis.","marker":"[21]"},{"why":"Contributes the training pipeline conventions and the ResNet and FT-Transformer implementations used in the benchmark.","marker":"[11]"},{"why":"Supplies the MLP-CN model with numerical embeddings and informs the search space for embedding-based neural networks.","marker":"[26]"},{"why":"Supplies the TabR retrieval-based architecture that defines the NN-Sample model class.","marker":"[39]"},{"why":"Supplies ModernNCA, the metric-learning model that the paper finds most robust across task types and many regimes.","marker":"[40]"},{"why":"Provides the frequency-based function irregularity metric used to define the regular versus irregular sub-categories.","marker":"[46]"},{"why":"Provides the imbalance factor metric and the long-tailed learning framework used for the label imbalance axis.","marker":"[25]"},{"why":"Supplies the pretrained transformer TabPFN that the paper compares against ModernNCA, delimiting the scope of pretrained tabular models.","marker":"[12]"}],"fun_headline_variants":["No single best tabular model: performance flips by data regime","Tabular model winners change with dataset size and correlation","One tabular model doesn't rule all: regime-specific winners emerge","Tabular benchmarks miss systematic flips in model ranking","On 196 datasets, best tabular model depends on data structure"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The main load-bearing assumption is that rescaling each model's error to a 0-to-1 range within every dataset-split and then averaging these relative scores makes errors comparable across datasets of very different difficulty, so the reported regime differences reflect true model strengths rather than artifacts of the normalization scale.","fun_headline_variants_meta":{"raw":{"variants":["No single best tabular model: performance flips by data regime","Tabular model winners change with dataset size and correlation","One tabular model doesn't rule all: regime-specific winners emerge","Tabular benchmarks miss systematic flips in model ranking","On 196 datasets, best tabular model depends on data structure"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000424,"raw_usage":{"total_tokens":2154,"prompt_tokens":901,"completion_tokens":1253,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":1168}},"tokens_in":517,"tokens_out":1253,"duration_ms":9973,"temperature":1.0,"reasoning_tokens":1168,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:35:57.460972+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the per-regime rankings using raw errors or average ranks instead of normalized scores, and check whether the reported patterns, NN-Sample best on large and correlated data and NN-Feature best on weakly correlated data, persist under both metrics and across splits with different dataset compositions per sub-category; if the ordering flips, the central claim is an artifact of normalization.","supporting_citations":[{"cited_title":"An inductive bias for tabular deep learning.Advances in Neural Information Processing Systems, 36, 2024","cited_arxiv_id":null,"evidence_quote":"Provides the frequency-based function irregularity metric used to define the regular versus irregular sub-categories."},{"cited_title":"Towards heterogeneous long-tailed learning: Benchmarking, metrics, and toolbox.Advances in Neural Information Processing Systems, 37:73098–73123, 2024","cited_arxiv_id":null,"evidence_quote":"Provides the imbalance factor metric and the long-tailed learning framework used for the label imbalance axis."}],"review_version":1}