{"id":"f70cf111-aee4-4ac0-b7dc-915297e1ba3f","arxiv_id":"2501.14717","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"By independently varying base models and training data across 12 models, this study shows that base model choice influences out-of-domain table task performance more than the instruction-tuning dataset does.","lead":"This paper trains 12 table-focused language models by combining three base models with four existing training datasets, then measures all of them on a shared set of table and general benchmarks. It finds that the choice of base model matters more than the training data, and that table fine-tuning does not necessarily hurt general capabilities.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim of base-model dominance lacks a quantitative effect-size analysis; the reported tables show large effects on both axes.","rationale":"The reader's weakest assumption about shared learning rate and epoch count is real, but the more load-bearing gap is that 'dominant role' is never operationalized. Even with perfect hyperparameter tuning, the paper's tables do not by themselves establish which factor dominates, because metrics are incomparable across benchmarks and no effect-size statistic is reported. The proposed variance-components test would settle the central claim directly. The paper deserves credit for the controlled 3x4 grid and the broad evaluation suite; the issue is the inference from raw tables to the headline. A conditional verdict is appropriate: the empirical contribution is strong enough to warrant revision and release of code/checkpoints, but the central claim should be quantitatively backed or softened.","tokens_in":20646,"tokens_out":10362,"duration_ms":96351,"concrete_test":"Re-analyze Table 10 (and Appendix C) with a two-way mixed-effects model on z-scored per-benchmark scores, excluding cells grayed for training-set overlap, and estimate variance components for base model vs training data with bootstrap confidence intervals. If the training-data variance component is not significantly smaller than the base-model component, the central claim of base-model dominance fails.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that base model choice plays a 'more dominant role' than training data requires a direct comparison of effect sizes, but the paper never performs one. Table 10 reports 16 benchmarks with different metrics, and visual inspection shows that both factors produce large, task-dependent swings. For example, on WikiTQ, training-data changes for Mistral span 23.8-37.4 while base-model changes on TableLLM data span 26.7-37.7; on TabFact, training-data changes for Mistral span 11.9-86.8, which is larger than most base-model effects. Without per-benchmark standardization and a variance decomposition or equivalent statistic, the abstract's 'quantitatively disentangle' and 'more dominant role' are not supported by the reported numbers. This concern is compounded by Section 3's single learning rate (5e-7) and three epochs applied to all 12 cells: if per-cell optima differ by base family, base-model differences are confounded with tuning quality. The controlled 3x4 grid is a valuable design, but the headline inference needs an explicit quantitative measure of dominance.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a controlled replication of four table instruction-tuned LLMs by fine-tuning three open 7B base models (Mistral v0.3, OLMo, and Phi-3 Small Instruct) on four public training recipes (TableLlama, TableLLM, TableBenchLLM, and Table-GPT), yielding 12 models. These models are evaluated on 16 table benchmarks spanning table QA, table fact verification, table-to-text generation, schema reasoning, and a miscellaneous task, plus five general benchmarks (MMLU, MMLUPro, AI2ARC, GPQA, IFEval). The headline finding is that base model choice plays a more dominant role than training data for out-of-domain table performance; secondary findings include strong untuned baselines, data effects that depend on the base model, transferability across table tasks, and limited degradation of general capabilities after table instruction tuning.","tokens_in":20826,"tokens_out":5803,"duration_ms":49869,"significance":"The controlled 3x4 grid is a genuine contribution: it is the first apples-to-apples replication of four existing table LLMs and provides reusable results, including a new SOTA on HiTab with only 5% of TableLlama's training data. If the dominance claim held, it would give practitioners actionable guidance to prioritize base-model selection over dataset choice. The paper also offers falsifiable observations, such as TableLLM's training data improving table QA and the untuned base models being competitive. However, the headline quantitative claim is not actually quantified, and several supporting statements are contradicted by the paper's own tables; these issues must be resolved before the paper can be accepted.","major_comments":[{"comment":"The central claim that base model choice plays a 'more dominant role' than training data is never quantified. Table 10 reports 16 benchmarks in heterogeneous metrics, and direct inspection shows large effects on both axes: for example, on WikiTQ, varying the training data for Mistral moves accuracy from 23.8 to 37.4, while varying the base model on TableLLM data moves it from 26.7 to 37.7; on TabFact, varying the training data for Mistral moves accuracy from 11.9 to 86.8. 'Dominant' requires a per-benchmark standardized effect-size analysis, such as a variance partition, ANOVA on ranks, or a regression with base model, training data, and interaction terms, together with a statement of uncertainty. The current tables provide neither, so the abstract's claim of 'quantitatively disentangl[ing]' overstates the evidence.","section":"Abstract; §5.3; Table 10"},{"comment":"The hyperparameter choice is a potential confound for the main comparison. The paper states that models were trained with learning rate 5e-7 for three epochs after a sweep over eight rates, but it does not say whether the sweep was performed separately for each of the 12 base-model/data combinations or only once. If 5e-7 is optimal for only some cells, the measured differences between base models and between datasets are partly tuning-quality differences, not pure base/data effects. Please add per-cell validation curves or an explicit sensitivity analysis (for example, re-running at neighboring learning rates on a subset of cells) to show that the conclusion is robust.","section":"§3 Experimental Setups"},{"comment":"The claim that TableLLM's training data 'consistently achieves the best ... on table QA tasks across all three base models' is not supported by the displayed results. In Table 7, on HiTab the best Mistral-based model is TableLlama (70.6), not TableLLM (44.1), and TableLlama's training data includes HiTab (gray cell), so this is an in-domain comparison rather than an out-of-domain one. In Table 10, the best WikiTQ result for Mistral is TableBench (37.4), not TableLLM (32.3), and the best TabFact result for OLMo is TableLlama (83.8), with TableLLM data at 40.6. Please either revise the claim to name the specific datasets where it holds and exclude overlapping gray cells, or present a formal aggregation that supports the stated conclusion.","section":"§5.2; Table 7 and Table 10"},{"comment":"The out-of-domain analysis includes gray cells whose training data overlaps with the evaluation benchmark, and these cells appear to contribute to the 'most number of top performance' markers used in the analysis. Because the paper's main claims are about generalization to unseen tasks, overlapping cells should be removed from the comparison or analyzed separately; otherwise the out-of-domain finding is partly circular. Please report the counts with and without gray cells and make clear which of the paper's claims each analysis supports.","section":"§5; Table 10 note"}],"minor_comments":[{"comment":"The abstract writes 'Hitab' where the rest of the paper uses 'HiTab'; please correct the spelling.","section":"Abstract"},{"comment":"The caption says 'Out-of-domain evaluation' but the same table marks train-data overlap in gray; please clarify in the caption how gray cells should be interpreted relative to the out-of-domain framing.","section":"Table 7 caption"},{"comment":"The phrase 'we rule out the table operation datasets and only maintain the training data' is awkward; 'we exclude the table operation datasets and keep only the training data' would be clearer.","section":"§4.1"},{"comment":"The suggestion that the AI2ARC improvement is 'likely due to' table tasks involving reasoning is not tested by any analysis; please soften the causal language or provide supporting evidence.","section":"§6.2"},{"comment":"For the multiple-choice benchmarks, the paper selects the completion with highest likelihood without length normalization; this is a defensible choice, but a sentence of justification or a citation would help readers interpret the results.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern raised in the reader's report is valid and lands on the central claim: the paper never quantifies 'dominance' with an effect-size analysis. I also agree with the secondary concern about the single learning rate, though it is fixable with a sensitivity analysis. The controlled design and the breadth of evaluation are strong, and the paper could be publishable after the authors add a proper quantitative decomposition, correct the contradicted claims about TableLLM data, and handle the gray overlapping cells explicitly. I would also ask in revision for details on code/checkpoint release, since the paper positions itself as providing open replications but gives no repository link."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know about this paper is that the abstract's central claim—that base model choice plays a more dominant role than training data—is not supported by the numbers it reports. The controlled 3x4 grid (three base models, four instruction-tuning datasets, sixteen benchmarks) is a genuinely useful contribution, and the HiTab SOTA is a nice side product, but the \"dominance\" inference needs a quantitative effect-size analysis that isn't there.\n\nWhat's actually new: no prior cited work trains the same three base models on the four existing table instruction datasets and evaluates all combinations on a common benchmark suite. That design lets the authors show, for example, that the same training data behaves very differently across base models, and that untuned base models are often strong baselines. These are useful empirical findings for practitioners. The observation that TableLLM's data helps QA while TableGPT's spreadsheet-style data transfers to fact verification is also informative. The citation pattern is standard, and the handling of the TableLLM metric discrepancy (CritiqueLLM in Chinese) is transparent.\n\nThe soft spots, in order of importance. First, the headline claim: Table 10 shows huge swings on both axes. For Mistral, TabFact accuracy ranges from 11.9 to 86.8 depending on training data; for TableLLM's data, TabFact ranges from 40.6 (OLMo) to 69.6 (Phi). Without per-benchmark standardization or a variance decomposition, \"more dominant\" is shorthand, not a result. Second, Section 5.2 contains a plain factual error: it says TableLLM's data achieves the best on HiTab, but Table 7 shows Mistral-TableLlama at 70.6 versus Mistral-TableLLM at 44.1. That misreading undermines confidence in the qualitative analysis. Third, the out-of-domain evaluation includes cells whose training data overlaps the benchmark (marked gray); those should be excluded from the \"out-of-domain\" framing or analyzed separately. Fourth, using one learning rate and three epochs for all twelve cells is an acknowledged simplification, but the confounding with tuning quality is real and should be discussed. Also, no code or checkpoints are released, which limits reproducibility.\n\nThis is a paper worth sending to reviewers—the controlled grid is a resource, and the over-claim is fixable with an effect-size analysis and sharper wording. But as submitted, the abstract overstates what the data show.\n\nRecommendation: send to peer review with a request for major revision focused on quantifying effect sizes and correcting the HiTab analysis.","headline":"Solid controlled grid, but the 'base-model dominance' claim is not backed by the paper's own tables.","tokens_in":21381,"tokens_out":3825,"would_cite":true,"duration_ms":32010,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Base model choice, not training data, dominates table LLM instruction-tuning performance.","keywords":["table instruction tuning","base model selection","out-of-domain evaluation","table question answering","instruction tuning","LLM evaluation","meta-evaluation","generalization"],"falsifier":"Train a known-weak base model on the training set the paper finds best and a known-strong base model on the training set it finds weakest, each with its own separately tuned learning rate and epoch count. If the weak-base/best-data model outperforms the strong-base/weak-data model on the out-of-domain suite, or if a variance decomposition over per-model hyperparameter sweeps shows a larger data component than model component, the claim that base model dominates training data would be undercut.","tokens_in":20452,"feed_emoji":"📊","tokens_out":10695,"duration_ms":85292,"temperature":0.7,"pith_summary":"This paper isolates two factors that earlier table-LLM work conflates: which base model you start from and which table instruction data you fine-tune on. The authors fine-tune three 7B instruction-tuned models on four public table datasets, producing twelve models, and evaluate them on sixteen table benchmarks plus general reasoning and instruction-following benchmarks. The central finding is that the base model matters more than the training data: switching base models changes out-of-domain performance more than switching training sets. The paper also reports that the untuned base models are already strong table learners, that a training set that works best for one base model is not necessarily best for another, and that fine-tuning for tables does not always compromise general abilities. If this finding holds, practitioners building table LLMs should rank base-model selection above dataset construction.","feed_headline":"Base model beats training data for table LLMs","feed_subtitle":"The same twelve fine-tunes on a 16-benchmark grid: swapping the base model moves scores more than swapping the dataset.","key_machinery":"The load-bearing mechanism is a crossed design: three 7B instruction-tuned base models (Mistral v0.3, OLMo, and Phi 3 Small) are each fine-tuned on the four public training sets from TableLlama, TableLLM, TableBench, and Table-GPT, forming a 3 × 4 grid of twelve models. Every model is evaluated on sixteen out-of-domain table benchmarks and on general reasoning and instruction-following benchmarks. The grid separates data effects (same base model, different data) from model effects (same data, different base model), and the out-of-domain evaluation prevents comparisons from reducing to memorized training distributions.","core_discovery":"The paper claims that in table instruction tuning, the choice of base model exerts a more dominant effect on downstream performance than the choice of training data. The claim is supported by a controlled grid in which the same three 7B instruction-tuned base models are each fine-tuned on four existing table instruction datasets, so every comparison differs in exactly one factor. Out-of-domain evaluation across table question answering, fact verification, table-to-text generation, and schema reasoning shows that untuned base models frequently match or beat their tuned counterparts, that the best training data differ across base models, and that the base model with the strongest pretrained abilities produces the best tuned models. The paper further shows that table fine-tuning can preserve or even improve scores on general benchmarks such as MMLU, ARC, GPQA, and IFEval for some base models, so specialization and general capability are not strictly opposed.","pith_inferences":["The paper uses one shared learning rate and epoch count for all twelve runs; a natural follow-up is to repeat the grid with per-combination hyperparameter search and compare the variance attributed to data versus models.","A cheap screening rule follows if the claim transfers: probe candidate base models on a few representative table tasks before fine-tuning any of them, because the untuned ranking largely predicts the tuned ranking.","The data-by-model interaction suggests future table instruction datasets should be designed against a target base model rather than assumed to transfer, and the same crossed design could test whether base-model dominance is a general property of instruction tuning in other structured-data domains such as text-to-SQL."],"forward_implications":["Practitioners should choose the strongest available base model before investing in table-specific training data, because the same data produces very different out-of-domain results across bases.","Untuned base models should become routine baselines in table-LLM work, since they already match or exceed tuned 7B models on several benchmarks.","Dataset quality rankings are not portable: a training set that is best for one base model can be below-average for another.","Strong performance on one benchmark within a task category, such as TabFact for fact verification, does not predict performance on a sibling dataset such as InfoTabs.","Table-specific fine-tuning does not necessarily degrade general reasoning or instruction-following ability, so specialization and general utility can coexist."],"supporting_citations":[{"why":"It supplies the TableLlama instruction data that the paper reuses to produce a replicated model and the new HiTab state of the art.","marker":"Zhang et al. (2024a)"},{"why":"It supplies the TableLLM instruction data, which the out-of-domain results identify as the most consistently effective training set for table QA.","marker":"Zhang et al. (2024b)"},{"why":"It supplies the TableBench training set and TableBeval evaluation used for the third replication arm.","marker":"Wu et al. (2024)"},{"why":"It supplies the Table-GPT spreadsheet instruction data and the schema-reasoning benchmarks used in the fourth replication arm.","marker":"Li et al. (2023)"},{"why":"It defines the Mistral 7B Instruct base model, one of the three model families whose ranking supports the base-model-dominance claim.","marker":"Jiang et al. (2023)"},{"why":"It defines the OLMo 7B Instruct base model, the weakest of the three bases and the clearest demonstration that data effects depend on the model.","marker":"Groeneveld et al. (2024)"},{"why":"It defines the Phi 3 Small Instruct base model, whose strong pretrained abilities drive the best tuned table performance.","marker":"Abdin et al. (2024)"}],"fun_headline_variants":["Base model, not data, drives table LLM performance","Table LLMs: base model matters more than training data","Swap base model, not dataset, to move table LLM scores","For table LLMs, base model choice beats data choice","Table tuning: base model dominates, data is secondary"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The analysis rests on the assumption that one shared learning rate (5e-7) and three training epochs give comparably good fits for all twelve model–data combinations, so performance differences are caused by base model and data rather than by uneven tuning quality.","fun_headline_variants_meta":{"raw":{"variants":["Base model, not data, drives table LLM performance","Table LLMs: base model matters more than training data","Swap base model, not dataset, to move table LLM scores","For table LLMs, base model choice beats data choice","Table tuning: base model dominates, data is secondary"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000177,"raw_usage":{"total_tokens":1243,"prompt_tokens":848,"completion_tokens":395,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":464,"completion_tokens_details":{"reasoning_tokens":313}},"tokens_in":464,"tokens_out":395,"duration_ms":3794,"temperature":1.0,"reasoning_tokens":313,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:51:30.963063+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a known-weak base model on the training set the paper finds best and a known-strong base model on the training set it finds weakest, each with its own separately tuned learning rate and epoch count. If the weak-base/best-data model outperforms the strong-base/weak-data model on the out-of-domain suite, or if a variance decomposition over per-model hyperparameter sweeps shows a larger data component than model component, the claim that base model dominates training data would be undercut.","supporting_citations":[],"review_version":1}