{"id":"be903ee2-86c7-48b5-9171-0d09a05862a8","arxiv_id":"2501.06863","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"End-to-end finetuning of DistilGPT2 is a competitive and cheap tabular classifier, but the headline superiority claim on small-feature datasets is contradicted by the reported numbers.","lead":"The paper fine-tunes a small language model (DistilGPT2) to classify tabular data and compares it with tree, neural, and LLM API baselines on ten datasets. It claims the LLM is best when datasets have fewer than ten features, but its own results do not support that claim.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2 contradicts the abstract's central claim: on the only two <10-feature datasets, end-to-end DistilGPT2 (0.738, 0.820) trails MLP (0.762) on blood-transfusion and GBT/FeatLLM (0.828/0.829) on diabetes.","rationale":"The reader's verdict rejects the paper on the grounds that Table 2 contradicts the headline claim, and I agree with that assessment. My stress-test pass identifies the same internal contradiction as the most load-bearing concern: the abstract promises superiority on '<10 features,' but the only two datasets satisfying that criterion show the proposed method losing to MLP or GBT/FeatLLM. This is not a matter of external consensus or missing baseline; it is a direct inconsistency between the stated conclusion and the reported numbers. The reader's formally listed weakest_assumption was the missing random-initialization ablation for the 'transfer learning' label. That is a real and important secondary concern, which is why I mark partial agreement, but the more decisive issue is the performance claim itself. If the performance claim were corrected to 'competitive on some small-feature datasets,' the transfer attribution would still need the random-init control, but the paper's central sales pitch would already be substantially weakened. A concrete paired statistical test on the two relevant datasets would settle whether the observed gaps are meaningful; the point estimates alone already place the burden on the authors. Therefore I do not propose changing the reader's REJECT verdict.","tokens_in":8758,"tokens_out":3306,"duration_ms":31322,"concrete_test":"Re-run the five-fold cross-validation on blood-transfusion and diabetes with the authors' exact protocol, but include (a) a randomly initialized DistilGPT2 of identical architecture and (b) paired bootstrap or Wilcoxon tests comparing End-to-end against GBT, MLP, and FeatLLM on per-fold AUC. If the random-init model matches End-to-end, transfer learning is not the cause; if the baselines win or tie after statistical testing, the 'outperforms SOTA on <10 features' claim is false.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in the abstract and §4.4 is that the proposed end-to-end finetuned LLM 'outperforms state-of-the-art machine and deep learning methods on tabular data with less than ten features.' The paper's own Table 2 directly undermines this. Taking the two datasets with fewer than ten features: blood-transfusion (5 features) has End-to-end AUC 0.738, below MLP's 0.762 (and only marginally above FeatLLM's 0.733); diabetes (9 features) has End-to-end 0.820, below GBT's 0.828 and FeatLLM's 0.829. Thus on the exact subset named in the claim, the proposed method is not the best method; in fact a standard MLP and GBT beat it. The only support for 'transfer learning via LLM can be the best solution' is a hedged reading of these two rows, and the point estimates contradict the abstract's unqualified 'outperforms.' Additionally, the attribution of any success to pretrained linguistic knowledge is untested: no ablation with a randomly initialized DistilGPT2 is reported, so the 'transfer learning' mechanism is not established. Because the headline conclusion rests on the small-feature subset, this internal inconsistency is the load-bearing weakness.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes end-to-end finetuning of DistilGPT2 with a classification head for tabular data classification. It describes a feature-to-text serialization and tokenization pipeline, compares frozen-weight and fully trainable finetuning against GBT, MLP, SCARF, and FeatLLM on ten benchmark datasets, reports five-fold cross-validated AUC and training times, and claims that the proposed method outperforms state-of-the-art machine and deep learning methods on tabular data with fewer than ten features. The paper also argues that text-to-tabular transfer learning is feasible and computationally cheaper than API-based in-context learning.","tokens_in":9040,"tokens_out":4777,"duration_ms":46020,"significance":"If substantiated, the central claim would be practically useful: a lightweight generative LLM finetuned end-to-end could serve as a strong classifier for small tabular datasets, where deep learning is often ineffective. The paper has some strengths: a transparent serialization procedure, a public lightweight model (DistilGPT2), a five-fold cross-validation setup, and direct runtime comparisons in Table 3. However, the headline assertion is contradicted by the paper's own results, and the attribution of observed performance to pretrained English text representations is untested. As reported, the evidence does not support the claimed advantage.","major_comments":[{"comment":"The central claim that the proposed end-to-end finetuned LLM 'outperforms state-of-the-art machine and deep learning methods on tabular data with less than ten features' is contradicted by Table 2. On the only two datasets with fewer than ten features, blood-transfusion (5 features) has End-to-end AUC 0.738, below MLP's 0.762, and diabetes (9 features) has End-to-end AUC 0.820, below GBT's 0.828 and FeatLLM's 0.829. The unqualified abstract claim and the §4.4 statement that 'transfer learning via LLM can be the best solution when the feature size is less than ten' are therefore not supported by the paper's own numbers.","section":"Abstract; §4.4; Table 2"},{"comment":"The 'transfer learning' attribution rests on the assumption that DistilGPT2's pretrained English representations, rather than the model architecture or the serialization setup, cause the observed performance. No ablation with a randomly initialized DistilGPT2 of identical architecture and training pipeline is reported, so the results do not establish text-to-tabular transfer. This is load-bearing because the novelty and the central framing of the paper depend on this attribution; the background section itself cites prior LLM finetuning approaches such as TabLLM and LIFT.","section":"§4.4; §3.1.3"},{"comment":"The 'fewer than ten features' regime is defined post hoc from only two datasets, and no statistical test, confidence-interval comparison, or multiple-comparison correction is provided. One of the two point estimates favors the proposed method over only some baselines, not over all of them, so the threshold cannot support a general claim about feature size. The paper should either narrow the conclusion to 'competitive on some small datasets' or provide a statistically grounded, preregistered analysis of a feature-size effect.","section":"§4.4; Table 2"},{"comment":"The overall ranking results are also in tension with the conclusions. End-to-end finetuning has average rank 3.2 across all datasets, whereas GBT has rank 1.9; the paper itself notes in §4.4 that GBT 'appears to be the best performing method.' The Conclusion's statement that 'transfer learning using LLM is superior to in-context LLM learning and deep learning methods proposed for tabular data in terms of performance and computational costs' is not supported by the reported AUC comparisons.","section":"Table 2; §4.4; §5"}],"minor_comments":[{"comment":"The paper says the authors 'updated the prompt generator of the FeatLLM method to include as many samples as possible under the constraint on token size,' but it does not report the exact prompt or how the update differs from the original FeatLLM protocol, which makes the in-context baseline difficult to reproduce.","section":"§3.2"},{"comment":"In the sentence 'text input prompts (0.733 (0.021)) are able to achieve a better performance than the best overall GBT model (0.711 (0.039))', the number 0.733 is not explicitly tied to FeatLLM in the text; the reader must infer it from Table 2.","section":"§4.4"},{"comment":"The easy/hard difficulty split is taken from reference [26], an unpublished manuscript, without a description of the criterion. The authors should either define the criterion in this paper or make the basis for the split available.","section":"Table 1; §4.1"},{"comment":"The discussion of overfitting after 30 epochs is based on training loss curves; since early stopping selects the model by validation loss, showing validation loss curves would be more informative.","section":"§4.3; Figure 4"},{"comment":"Several references are incomplete or inaccessible, including [3] and [26]; in particular [26] is used for the baseline setup and difficulty labels and should be described or made available for reproducibility.","section":"References"}],"recommendation":"reject","confidential_remarks":"The paper's main claim is internally inconsistent with its own Table 2, and the missing random-initialization ablation leaves the transfer-learning mechanism untested. While some of these issues could in principle be addressed by rebenchmarking and a substantially narrowed claim, the current manuscript does not meet the bar for publication as submitted. I recommend rejection, with the possibility of a future resubmission if the claims are revised and the missing ablation is added."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper's core claim falls apart on its own table. The abstract and Section 4.4 say end-to-end finetuning of DistilGPT2 outperforms state-of-the-art methods on tabular data with fewer than ten features. Table 2 shows the opposite: on blood-transfusion (5 features), MLP gets 0.762 AUC while End-to-end gets 0.738; on diabetes (9 features), GBT gets 0.828 and FeatLLM gets 0.829 while End-to-end gets 0.820. On the exact regime named in the claim, the method is not the best. The body hedges with 'can be the best solution,' but the abstract's 'outperforms' is simply false as stated.\n\nThe paper does some things well. It runs a clean ten-dataset comparison with standard baselines, reports computational costs, and is honest in the text that GBT has the best overall rank. Using DistilGPT2, a small open model, is a reasonable choice for a resource-constrained setting, and the cost table shows a real advantage over API-based FeatLLM and SCARF. The citation of prior work is fair: they cite TabLLM and LIFT, which already finetune LLMs on serialized tabular rows. That makes the novelty incremental—a different base model and a wider benchmark, not a new method.\n\nThe soft spots are serious. The 'transfer learning' label is unsupported because there is no ablation against a randomly initialized DistilGPT2. The pretrained English weights could be doing none of the work; the result could just be a small transformer trained from scratch. The easy/hard dataset split and baseline setup depend on the authors' own unpublished benchmark [26], which limits independent verification. The 'less than ten features' rule is defined post hoc after seeing which rows look favorable, and it still fails on those rows. These are not minor quibbles; the central argument does not hold.\n\nWho is this for? A reader interested in cheap LLM-based tabular classification might find the cost numbers and the competitive performance on some datasets useful as a data point. But the paper as written overclaims, and the missing control is a load-bearing gap. I would not cite it, and I do not think it deserves a full peer-review cycle. Desk reject, with a suggestion that the authors fix the abstract, add the random-init ablation, and reposition the paper as a benchmark study rather than a transfer-learning win.","headline":"The paper's own Table 2 contradicts its headline claim: on the two datasets with fewer than ten features, the proposed end-to-end finetuned LLM is beaten by MLP and GBT, and the 'transfer learning' mechanism is never tested.","tokens_in":9612,"tokens_out":2232,"would_cite":false,"duration_ms":21432,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"End-to-end finetuning of a small generative language model outperforms state-of-the-art machine and deep learning classifiers on tabular datasets with fewer than ten features, while using a fraction of the computational cost.","keywords":["tabular data","transfer learning","large language models","DistilGPT2","fine-tuning","in-context learning","classification"],"falsifier":"Train the exact same end-to-end pipeline on the blood-transfusion and diabetes datasets using a randomly initialized DistilGPT2 of the same architecture; if its five-fold AUC matches or exceeds the reported values (0.738 and 0.820), the claim that transfer from English text drives the result is falsified.","tokens_in":8531,"feed_emoji":"🤖","tokens_out":8163,"duration_ms":73210,"temperature":0.7,"pith_summary":"This paper tries to establish that transfer learning for tabular classification can be done by finetuning a small generative language model end to end, turning each data row into a short English sentence. The claim is that for the common case of tabular datasets with fewer than ten features, this finetuned model matches or beats gradient-boosted trees and other strong baselines, and outperforms a contrastive deep learning method on most of the benchmark sets. The reason this matters is that tabular data lacks the large pretrained foundation models that exist for text and images, so a text-to-tabular transfer route would give small scientific, medical, and business datasets access to deep learning. The paper also shows the finetuning route is far cheaper than API-based in-context prompting, because it runs locally on a small open-weight model.","feed_headline":"Finetuned language model beats tabular baselines on few-feature data","feed_subtitle":"Small text model finetuned end-to-end challenges gradient-boosted trees at a fraction of the compute.","key_machinery":"The load-bearing machinery is the combination of row serialization and full-weight finetuning. Each tabular row is converted to a prompt such as 'Age is 25. Sex is male.' with feature names and values, tokenized into sequences up to 1024 tokens, and passed through DistilGPT2, whose final 768-dimensional embedding is mapped to class logits through a classifier head trained with cross-entropy loss. With all transformer weights frozen the model underfits and ranks last; with all weights trainable, the pretrained English representations adapt to the tabular task, giving the reported gains. The same serialization format has no need for the elaborate prompt templates and token budgets that constrain API-based in-context learning.","core_discovery":"The paper's central discovery is that end-to-end finetuning of DistilGPT2—an 82-million-parameter distilled causal language model pretrained on English text—converts tabular classification into a text task and achieves competitive or superior accuracy. On the ten benchmark sets, the finetuned model with all weights trainable ranks ahead of the frozen-weight variant, the API-based in-context method FeatLLM, and the contrastive deep learning method SCARF, and its average ranking is close to gradient-boosted trees. The authors highlight the blood-transfusion and diabetes datasets, with five and nine features respectively, where the finetuned LLM matches or exceeds the best GBT baseline, and conclude that transfer learning via an LLM can be the best solution when the feature size is less than ten. They further report that the approach uses roughly 10 to 50 percent of the compute of the API or deep learning alternatives.","pith_inferences":["The 'fewer than ten features' boundary is supported by only two datasets (blood-transfusion and diabetes); an inference worth testing is whether the advantage generalizes across a broader suite of low-feature tabular benchmarks or is specific to these medical and transfusion domains.","If the pretrained English representations are doing the work, the finetuned model's strong results on medical datasets may partly reflect domain knowledge absorbed during pretraining; comparing against a randomly initialized model of the same architecture would separate transfer from architecture.","A natural extension the paper does not explore is using the finetuned LLM's penultimate-layer embeddings as features for a gradient-boosted tree, which could combine the LLM's semantic representation with the classical model's strong tabular inductive bias."],"forward_implications":["If the central claim holds, tabular datasets with fewer than ten features—very common in practice—can be handled by deep learning through a finetuned text model, closing a gap that standard deep networks have not closed.","The finetuned LLM would replace API-based in-context prompting for small tabular tasks, since local training is faster and cheaper and does not depend on external token limits.","Frozen-weight finetuning is not a viable transfer strategy for this setup; the model weights must be trained end to end, which clarifies where the transfer gain comes from.","The approach outperforms the contrastive deep learning baseline SCARF on six of the ten datasets, so LLM finetuning becomes a stronger default than self-supervised tabular pretraining on these benchmarks."],"supporting_citations":[{"why":"Supplies the SCARF contrastive learning method that serves as the state-of-the-art deep learning baseline.","marker":"[25]"},{"why":"Provides the FeatLLM API-based in-context learning baseline and the prompt template that the paper adapts for token-limited inputs.","marker":"[8]"},{"why":"Earlier work finetuning LLMs for few-shot tabular classification; the paper positions its end-to-end method relative to it.","marker":"[10]"},{"why":"Supplies the experimental setup, data-selection criterion, and baseline implementations used in the five-fold evaluation.","marker":"[26]"},{"why":"Introduces the DistilGPT2 model that the paper finetunes end to end.","marker":"[22]"},{"why":"Describes the pretraining corpus that gives DistilGPT2 its English text knowledge, the source of the claimed transfer.","marker":"[24]"},{"why":"Establishes the in-context learning paradigm that the API baseline represents and that the paper contrasts with transfer learning.","marker":"[20]"}],"fun_headline_variants":["Finetuned LLM tops tabular baselines on few-feature data","Small language model beats trees on small tabular data","LLM finetuning wins on tabular data under ten features","Text model finetuned for tables matches boosting at 10% compute","Cross-data transfer via LLM beats ML on small feature sets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that DistilGPT2's pretrained English knowledge, not the transformer architecture or the text serialization format, is what makes the finetuned model perform well; if a randomly initialized model of the same size did equally well, the transfer-learning explanation would collapse.","fun_headline_variants_meta":{"raw":{"variants":["Finetuned LLM tops tabular baselines on few-feature data","Small language model beats trees on small tabular data","LLM finetuning wins on tabular data under ten features","Text model finetuned for tables matches boosting at 10% compute","Cross-data transfer via LLM beats ML on small feature sets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000166,"raw_usage":{"total_tokens":1245,"prompt_tokens":931,"completion_tokens":314,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":224}},"tokens_in":547,"tokens_out":314,"duration_ms":3446,"temperature":1.0,"reasoning_tokens":224,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:49:10.341333+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the exact same end-to-end pipeline on the blood-transfusion and diabetes datasets using a randomly initialized DistilGPT2 of the same architecture; if its five-fold AUC matches or exceeds the reported values (0.738 and 0.820), the claim that transfer from English text drives the result is falsified.","supporting_citations":[{"cited_title":"SCARF: Self-supervised contrastive learning using random feature corruption","cited_arxiv_id":null,"evidence_quote":"Supplies the SCARF contrastive learning method that serves as the state-of-the-art deep learning baseline."},{"cited_title":"Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning","cited_arxiv_id":null,"evidence_quote":"Provides the FeatLLM API-based in-context learning baseline and the prompt template that the paper adapts for token-limited inputs."},{"cited_title":"Rabbani, Ivan V","cited_arxiv_id":null,"evidence_quote":"Supplies the experimental setup, data-selection criterion, and baseline implementations used in the five-fold evaluation."},{"cited_title":"Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter","cited_arxiv_id":null,"evidence_quote":"Introduces the DistilGPT2 model that the paper finetunes end to end."},{"cited_title":"Aligning books and movies: Towards story-like visual explanations by watching movies and reading books","cited_arxiv_id":null,"evidence_quote":"Describes the pretraining corpus that gives DistilGPT2 its English text knowledge, the source of the claimed transfer."}],"review_version":1}