{"id":"c6669068-fc94-4ee1-837d-ebbf249e369c","arxiv_id":"2508.21561","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"InsightTab builds rules from XGBoost-grouped few-shot rows, selects easy examples as demonstrations, and refines rules on misclassified hard examples, beating cited LLM and classical baselines on nine tabular datasets by 10 to 55 percent average F1.","lead":"InsightTab converts a small labeled table into LLM-generated rules and carefully chosen examples, then uses those insights to classify new rows. On nine tabular datasets it reports consistent average F1 gains over existing LLM-based and classical baselines in the few-shot setting.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed data-driven distillation is not isolated from LLM memorization of these standard TabLLM benchmarks; reported gains may reflect parametric prior knowledge rather than the distillation procedure.","rationale":"The reader's weakest assumption is that XGBoost's first tree, trained on as few as 16-128 rows, partitions data into within-leaf subsets whose similarity is meaningful for rule summarization. This is a legitimate algorithmic concern, and the ablation in Table 2 does show that removing grouping causes large average F1 drops (e.g., mistral-7b from 64.1 to 32.0). I do not dispute that grouping matters empirically; the ablation is consistent with the paper's mechanism. However, the grouping step alone cannot settle the more fundamental question of where the summarized rules come from. Because the rule summarizer is gpt-4-turbo and the benchmark suite is the standard TabLLM collection of widely studied datasets, the rules and predictions can be informed by memorized knowledge of those specific tables. The paper's own Limitations section acknowledges contamination and proposes as future work testing whether the method remains effective when column names lack semantic meaning; that is exactly the experiment needed to support the central claim. The position-bias analysis is not a substitute: shuffling column order changes serialization format but not semantic content, so it cannot identify leakage through feature names and value meanings. Five-fold cross-validation likewise does not prevent contamination because the same semantic column names and value semantics appear in every fold. For these reasons, the central claim 'data-driven insight distillation' is not yet established as data-driven. This is not a charge of dishonesty; the method is coherent, the ablation is supportive, and the cost analysis is useful. It means the empirical contribution is conditional on a contamination-controlled evaluation. The reader's CONDITIONAL verdict is therefore appropriate, and I recommend no change to the verdict, with the specific condition being a renamed/anonymized or synthetic benchmark test rather than only additional significance testing. Variance reporting would help, but it is secondary to the semantic-leakage question.","tokens_in":23876,"tokens_out":5079,"duration_ms":50724,"concrete_test":"Build five synthetic binary tabular tasks with 8-16 features generated from known ground-truth rules, serialize them with meaningless column names (f1, f2, ...) and abstract value codes, and run the full InsightTab pipeline (XGBoost first-tree grouping, gpt-4-turbo summarization, gpt-3.5-turbo/mistral-7b prediction) at 16/32/64/128 training samples. Compare against Original and TabLLM under identical prompts. If the average F1 advantage over TabLLM is not preserved (or falls below roughly half the margin reported in Table 1), the benchmark gains are substantially attributable to memorized dataset semantics rather than to the data-driven distillation pipeline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that few-shot training data are distilled into actionable insights, and that these insights drive InsightTab's improvements. The most load-bearing vulnerability is that the evaluation does not separate data-driven distillation from memorized dataset knowledge. All nine datasets come from the widely used TabLLM benchmark suite (Income, Diabetes, Heart, Bank, Creditg, Car, etc.), and the rule summarizer is gpt-4-turbo, whose pretraining almost certainly includes these famous tables and their labels. The method's 'insights' (e.g., 'significant capital gains in a year are a strong indicator of earning more than $50,000', Figure 4b) are exactly the kind of statements that could be retrieved from parametric memory rather than induced from 16-128 rows. The position-bias analysis only perturbs feature order, not semantic content; five-fold cross-validation does not prevent target memorization because the same column names and value meanings appear in every fold. The paper's Limitations section explicitly acknowledges this: 'future work should follow this direction and explore whether our model remains effective when column names lack semantic meaning.' If the benchmark gains are substantially explained by memorized priors, the central claim that the method is data-driven collapses, even if the algorithmic pipeline is internally coherent. The grouping concern raised by the reader is real, but secondary: even if XGBoost leaf partitions are meaningful, the resulting rules could still be contaminated by the summarizer's prior knowledge.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes InsightTab, a framework for few-shot tabular classification with LLMs that distills the few available labeled rows into natural-language insights. The method uses three principles: divide-and-conquer (training samples are grouped via the first tree of an XGBoost model, per-group rules are summarized by an LLM, and the rules are merged), easy-first (low-entropy training samples are selected as in-context demonstrations), and reflective learning (high-entropy samples are classified with the current rules, and the mistakes are summarized into additional rules). The final prompted prediction combines the merged rules, the easy demonstrations, and the reflection rules. The paper evaluates InsightTab on nine datasets from the TabLLM benchmark suite with mistral-7b and gpt-3.5-turbo as predictors, reporting average F1 gains over XGBoost, TabPFN, Original, Tablet, TabLLM, and SumBoost, and provides ablations, position-bias and class-bias analyses, a cost comparison, and open-source code.","tokens_in":24116,"tokens_out":4249,"duration_ms":41444,"significance":"If the reported gains hold up, InsightTab is a practically useful and conceptually clean contribution: it combines a cheap, traditional model (XGBoost) with LLM summarization to produce human-readable rules and few-shot demonstrations, and it consistently outperforms strong LLM baselines such as TabLLM and SumBoost in the paper's experiments. The method is simple, reproducible in principle, and the authors provide code and a data link. The framework of group-rank-summarize is clearly described and the ablations are consistent with the claimed importance of each component. However, the significance is conditional on separating the distillation procedure from possible LLM memorization of these widely used public benchmark datasets, which the paper itself acknowledges as an open issue.","major_comments":[{"comment":"The central claim of 'consistent improvement' is based on point estimates averaged over 5-fold cross-validation and multiple runs, but no variance, confidence intervals, or significance tests are reported anywhere in the main text or appendix. For example, the Creditg comparison between InsightTab (81.9) and TabLLM (80.9) with n=16 in Table 1 may not be meaningful without knowing run-to-run variability. Since the main message of the paper is a robust advantage over baselines, please report standard deviations or confidence intervals for the averaged F1 numbers, and ideally paired significance tests across folds or datasets.","section":"Table 1 and Table 2"},{"comment":"The evaluation does not isolate the data-driven distillation procedure from parametric knowledge in the LLMs. All nine datasets are standard public benchmarks (Income, Diabetes, Heart, etc.), the rule summarizer is gpt-4-turbo, and the position-bias analysis in Section 5.2 only shuffles feature order while preserving semantically meaningful column names and value labels. Five-fold cross-validation does not prevent target memorization because the same column names and label semantics appear in every fold. The paper itself acknowledges this at the end of the Limitations section: 'future work should follow this direction and explore whether our model remains effective when column names lack semantic meaning.' This is load-bearing for the claim that the method is data-driven. Please add an experiment with sanitized/randomized column names (or an evaluation on a private or novel dataset) and compare the quality of the mined rules and the resulting accuracy under that condition.","section":"Limitations and Section 5.2"},{"comment":"The grouping step uses the first tree of an XGBoost model trained on as few as 16 to 128 rows. With such few samples, leaf partitions can be dominated by a single class or by noise, and the resulting per-leaf summaries may not capture transferable classification rules. The alternative-strategy experiments in Appendix C.3 compare different sampling/grouping heuristics, but they do not evaluate the stability of the first-tree grouping, the distribution of leaf sizes, or per-leaf label purity, and they do not vary the number of trees used for grouping. Please add an analysis of group quality (e.g., leaf sizes and class entropy across folds) and a sensitivity check on the number of boosting rounds or tree depth used for grouping.","section":"Algorithm 1, lines 3-6"}],"minor_comments":[{"comment":"The sentence reporting 'average improvements of 21%, 32%, 55%, 37%, 10%, and 44%' should state whether these are relative or absolute F1 changes, since the base rates differ across the baselines.","section":"Section 4.2"},{"comment":"The variant label '- demostr.' contains a typo for '- demonstration'; please fix it in the table and in the text that refers to it.","section":"Table 2"},{"comment":"The symbol ⊕ used for rule enhancement is not defined in the text; please explain how the additional rules R_h are merged with R.","section":"Algorithm 1, line 13"},{"comment":"The paper reports a hyperparameter grid for XGBoost but not the selected hyperparameter values per dataset or fold; please state the selected values or note that the defaults in the code were used throughout.","section":"Appendix B.3"},{"comment":"No decoding temperature or sampling parameters are reported for the LLM calls (mistral-7b, gpt-3.5-turbo, gpt-4-turbo); please specify these settings to enable reproducibility.","section":"Section 4.1 and Appendix B.3"},{"comment":"The caption for the full-training-data results should explicitly state that TabPFN is omitted due to runtime errors, as this is mentioned only in the main text of Appendix C.2.","section":"Table 5, n = all"}],"recommendation":"major_revision","confidential_remarks":"This is a solid empirical paper with a clear framework, extensive ablations, and a code link; the main risk is the memorization/contamination concern, which the authors themselves acknowledge. In my view the paper is not rejectable because the pipeline is internally coherent and the ablations support the contribution of each component, but it needs a sanitized evaluation (e.g., anonymized column names or a private dataset) and uncertainty quantification before the central data-driven claim can be considered established. Please also ask the authors to address the grouping-stability point, as it bears on the validity of the divide-and-conquer principle."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a coherent, incremental method paper that improves few-shot tabular classification by having an LLM summarize rules from XGBoost-grouped training samples, pick easy demonstrations, and reflect on hard mistakes. The empirical gains relative to TabLLM and SumBoost look real and consistent, but the evaluation never separates what the summarizer learns from the data and what it already knows about these famous benchmark tables. Until that is addressed, \"data-driven\" in the title is doing more work than the evidence supports.\n\nWhat is new: the specific combination of first-tree XGBoost grouping, entropy-based easy/hard ranking, and reflective rule enhancement is not in the cited literature. The ablations are thorough, the pattern holds across nine datasets, and they report costs, include code/data links, and write an honestly hedged limitations section. That is good practice.\n\nThe soft spots are real but not disqualifying. First, there are no error bars or significance tests anywhere. The paper says results are averaged over 5x4=20 tests but gives no variance, so we cannot tell if the average gains are robust or driven by a few datasets. That is an easy fix.\n\nSecond, the contamination issue is the one that matters. All nine datasets come from the standard TabLLM suite, and the rule summarizer is gpt-4-turbo, which very likely has seen Income, Heart, Diabetes, and the rest. The rules in Figure 4b are exactly the kind of statements that could come from parametric memory. Five-fold cross-validation does not help because the column names and value semantics are identical in every fold; the position-bias analysis only shuffles feature order, not content. The authors acknowledge this in the Limitations and say future work should test with semantically meaningless column names, but that is precisely the experiment this paper needs to claim data-driven distillation. If the gains survive column anonymization, the method is solid. If they do not, the contribution is mostly prompt engineering on memorized priors.\n\nThird, the XGBoost first-tree grouping is a bit hand-wavy. With 16-128 rows, a single tree can easily split on noise. The ablation shows grouping helps, so empirically it works, but the mechanism is not well argued.\n\nAll in all, this deserves a serious referee. The method is coherent, the writing is clear, and the limitations are stated. I would send it to review with a request for per-seed variance and, more importantly, a contamination-controlled experiment with anonymized or renamed columns. If the authors can show the gains hold there, I would be happy to see it accepted.","headline":"A sensible, incremental method with a real contamination confound: the gains over baselines are consistent, but the paper never proves the rules come from the data rather than the summarizer's memory of these benchmark tables.","tokens_in":24686,"tokens_out":2874,"would_cite":false,"duration_ms":26858,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that distilling few-shot tabular rows into natural-language rules makes LLM classifiers substantially more accurate, and it demonstrates consistent gains over classical and LLM baselines across nine datasets.","keywords":["few-shot tabular classification","large language models","insight distillation","rule summarization","in-context learning","XGBoost","reflective learning","data serialization"],"falsifier":"Compare InsightTab against a control that replaces the XGBoost first-tree grouping with random partitions of the same size while keeping ranking, summarization, and reflection identical; if the F1 gap disappears or reverses on the nine datasets, the grouping step is not doing the work the paper claims.","tokens_in":23670,"feed_emoji":"📊","tokens_out":4955,"duration_ms":43241,"temperature":0.7,"pith_summary":"The paper tries to establish that an LLM can classify tabular data accurately from very few labeled examples if, before predicting, the training rows are distilled into compact natural-language rules. It proposes InsightTab, a pipeline that groups similar rows, ranks them by prediction difficulty, summarizes each group into rules, and then refines those rules on mistakes. The claim matters because few-shot tabular classification is common in real services where labels are scarce, and classical models like XGBoost need much more data to work. On nine datasets, InsightTab reports consistent gains over prior LLM prompting methods and over classical and transformer baselines, with the largest gains in the 16-to-128-sample regime.","feed_headline":"Few-shot tables: distilled rules lift LLM F1 by up to 55%","feed_subtitle":"InsightTab beats prior LLM methods on 19 of 20 comparisons across nine datasets using only 16–128 labeled rows.","key_machinery":"The load-bearing object is the multifaceted serialization prompt $\\mathrm{MSP}(x, T, D_s, R)$, which turns the test row, task description, few-shot demonstrations, and mined rules into one text prompt. It is carried by Algorithm 1: an XGBoost model's first tree partitions the training rows (group), class-probability entropy ranks rows from easy to hard (rank), an LLM summarizes and then merges group-wise rules (summarize), and incorrectly predicted hard rows produce extra rules (reflect). The first tree is the partition mechanism; entropy is the difficulty signal; rule merging is what keeps the prompt concise.","core_discovery":"InsightTab claims that the bottleneck in few-shot LLM tabular classification is not the LLM's reasoning but the absence of task-specific knowledge, and that this gap can be closed by distilling the few labeled rows into natural-language insights before asking the LLM to predict. The distillation follows three human-learning principles: divide-and-conquer (split training rows by the leaves of a single XGBoost tree and summarize each group's rules), easy-first (use the lowest-entropy rows as in-context demonstrations), and reflective learning (have the predictor try hard, high-entropy rows, keep the mistakes, and summarize additional rules from them). The final prompt combines the merged rules, the easy demonstrations, and any reflection-derived rules. On nine datasets with 16 to 128 training rows, InsightTab reports average F1 of 64.8 with mistral-7b and 62.8 with gpt-3.5-turbo, beating the same-base-LLM baselines in 19 of 20 comparisons and improving over XGBoost, TabPFN, Original, Tablet, TabLLM, and SumBoost by 21, 32, 55, 37, 10, and 44 percent on average.","pith_inferences":["An untested corollary of the two-model split is that the expensive summarizer runs once while the cheap predictor runs many times, so insight distillation could serve as a general cost-saving pattern for structured LLM inference.","Because grouping only needs a partition, any interpretable weak model could replace XGBoost's first tree; testing whether random partitions perform worse would isolate where the gain comes from.","The natural-language rules are auditable and reusable: they could be inspected by humans, transferred to datasets with similar schemas, or updated incrementally, implications the paper leaves unexplored.","The position-bias results suggest rule prompts may reduce LLM reliance on token-order memorization, which connects to the data-contamination risk the authors acknowledge."],"forward_implications":["If a task has only 16 to 128 labeled rows, InsightTab's rule-first prompting yields higher F1 than both classical tabular models and LLM baselines on the nine datasets tested.","The same base LLM with InsightTab beats other LLM methods in 19 of 20 comparisons, so the gain comes from the distillation process rather than from choosing a better model.","Rules distilled from grouped samples let the predictor use more in-context examples before performance degrades, reducing overfitting to individual examples.","On feature-order shuffling and class-imbalance tests, InsightTab degrades less than serialization-only baselines, indicating better generalization under distribution shifts.","With full training data, XGBoost is still generally stronger, but InsightTab is the only LLM method that matches or beats it on some datasets."],"supporting_citations":[{"why":"Supplies XGBoost, whose first tree defines the grouping used for rule summarization.","marker":"Chen and Guestrin, 2016"},{"why":"Contributes the TabLLM baseline, the Text Template serialization, and the nine datasets.","marker":"Hegselmann et al., 2022"},{"why":"Contributes TabPFN, the in-context tabular transformer baseline to beat.","marker":"Hollmann et al., 2022"},{"why":"Contributes SumBoost, the rule-summarization baseline and a direct predecessor.","marker":"Manikandan et al., 2023"},{"why":"Contributes Tablet, the instruction-based prompting baseline.","marker":"Slack and Singh, 2023"},{"why":"Motivates the contamination concern that the position-bias analysis addresses.","marker":"Bordt et al., 2024"}],"fun_headline_variants":["InsightTab: distilled insights lift LLM few-shot tabular F1","Summarize-exemplify-reflect: LLM tabular few-shot boost","Few-shot tabular: distilling rules into insights for LLMs","Turn few labeled rows into rules that power LLM classifiers","Three-step distillation improves LLM few-shot tabular accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire gain rests on the assumption that the first tree of an XGBoost model trained on as few as 16 to 128 rows partitions the data into groups whose shared patterns can be summarized into transferable rules; with so few samples, those splits may instead reflect noise or single-class leaves, producing misleading rules.","fun_headline_variants_meta":{"raw":{"variants":["InsightTab: distilled insights lift LLM few-shot tabular F1","Summarize-exemplify-reflect: LLM tabular few-shot boost","Few-shot tabular: distilling rules into insights for LLMs","Turn few labeled rows into rules that power LLM classifiers","Three-step distillation improves LLM few-shot tabular accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000238,"raw_usage":{"total_tokens":1523,"prompt_tokens":968,"completion_tokens":555,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":464}},"tokens_in":584,"tokens_out":555,"duration_ms":5182,"temperature":1.0,"reasoning_tokens":464,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:40:25.250526+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare InsightTab against a control that replaces the XGBoost first-tree grouping with random partitions of the same size while keeping ranking, summarization, and reflection identical; if the F1 gap disappears or reverses on the nine datasets, the grouping step is not doing the work the paper claims.","supporting_citations":[],"review_version":2}