{"id":"aed95c42-f42a-4f79-8351-9206e9481ca7","arxiv_id":"2505.13421","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An LLM prompted with nearest-neighbor labels and external model predictions performs instance-level ensembling for tabular data, reaching the best average rank on TinyBench2.","lead":"This paper uses a large language model to decide, for each data row, how to combine predictions from several standard tabular models. The LLM reads nearby rows and model votes and reasons step by step, which the authors say beats fixed averaging rules.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CoT2's local model selection (Eq. 5) is driven by in-sample neighbor predictions from models trained on the same training set D, so the claimed gains may reflect memorization rather than transferable local expertise.","rationale":"The reader's weakest_assumption is that external models' predictions on retrieved training-set neighbors are optimistically biased because they are in-sample, and that this undermines the LLM's local model selection. I agree: this is the single most load-bearing assumption because the method's entire mechanism -- instance-wise model selection from neighbor evidence -- depends on the transferability of neighbor performance to the target. The in-sample nature of the neighbor predictions is not a peripheral detail; it directly affects step c (Eq. 5) and step d (Eq. 6), and it can be corrected or measured with out-of-fold predictions. The proposed test is concrete, feasible, and would settle whether CoT2's reported edge comes from memorized training-set agreement or from genuine local generalization. I do not see a more serious threat: the paper contains no formal proofs to check, the empirical comparison is on an established benchmark, and the authors include ablations and a non-LLM baseline. The remaining concerns in the reader's verdict -- incomplete statistical documentation, a contradictory caption, and no released code -- are real but secondary. The reader's CONDITIONAL verdict remains appropriate: the claim is plausible but not fully established until the in-sample leakage is tested. Hence UNCHANGED.","tokens_in":28137,"tokens_out":6843,"duration_ms":64649,"concrete_test":"Re-run CoT2 on the 30 TinyBench2 classification datasets with identical prompts, LLMs, seeds, and hyperparameters, but replace each neighbor's model prediction with an out-of-fold prediction: for each neighbor j, use predictions from models trained on D without j (or from the corresponding cross-validation fold), while keeping target predictions and validation accuracies unchanged. Then recompute the average rank and per-dataset accuracy and compare against Table 4, Figure 3, and the Average Voting, Weighted Voting, and Best Model baselines. If CoT2's advantage shrinks or reverses, the in-sample neighbor predictions are load-bearing; if the advantage persists, the local-transfer concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that LLM instance-wise integration of external model predictions outperforms fixed ensemble rules. The mechanism's decisive step is step c (Eq. 5): the LLM selects the 'most suitable' models for the target by comparing each model's predictions on the K retrieved neighbors against those neighbors' true labels, and step d (Eq. 6) then relies on the selected models' predictions for the target. But the neighbors are retrieved from the same training set D used to fit all external models (Sections 3.1 and 3.2), so every neighbor prediction the LLM sees is an in-sample prediction. For any model that memorizes training data, especially KNN, in-sample neighbor predictions are optimistically near-perfect regardless of out-of-sample quality. The prompt in Figure 2 even reports KNN training accuracy 1.00 while showing KNN predictions that disagree with neighbor true labels, illustrating the confusion between memorization and generalization. Consequently, the premise needed for the central claim -- that local agreement on neighbors predicts accuracy on the target -- is not established; the LLM may simply be rewarded for selecting models that memorized the retrieved points. The reported improvement over fixed ensembles could then be an artifact of exposing training-set predictions to the LLM rather than evidence of robust instance-wise ensembling.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CoT^2 (Chain of Tabular Thoughts), a prompting strategy that uses a large language model (LLM) to perform instance-wise integration of predictions from an external pool of tabular models. For each test instance, the method builds a tabular context from the true labels and model predictions of its nearest neighbors in the training set, plus dataset-level statistics, and guides the LLM through four reasoning steps: well-performing model selection, outlier identification, suitable model selection, and final prediction. The method is evaluated on the TinyBench2 benchmark across 30 classification and 15 regression datasets, reporting the best average ranking among a broad set of baselines, including a non-LLM context-based baseline called MetaXGB.","tokens_in":28405,"tokens_out":5741,"duration_ms":56050,"significance":"If the reported results are trustworthy, the paper would introduce a meaningful new application of LLMs as interpretable ensemble meta-learners for tabular data, with a privacy-friendly context that avoids raw features and semantic descriptions. The idea of letting an LLM select and combine local model predictions through structured reasoning is novel and of broad interest. The manuscript also includes a useful non-LLM baseline and an ablation study. However, the strength of the empirical claim is weakened by a load-bearing methodological issue: the local model selection step uses in-sample neighbor predictions, which are optimistically biased, and the study design has a self-benchmarking overlap and hyperparameter tuning on the evaluation datasets. These issues preclude accepting the central claim without additional experiments.","major_comments":[{"comment":"The 'Suitable Model Selection' step selects the most suitable external models for the target instance by evaluating their predictions on the K nearest neighbors retrieved from the training set D. Since all external models are trained on D, these neighbor predictions are in-sample and optimistically biased; for KNN they can be near-perfect regardless of out-of-sample performance. The prompt in Figure 2 even shows KNN with training accuracy 1.00 while its neighbor predictions disagree with several true labels, illustrating exactly this memorization-vs-generalization confusion. The paper does not establish that in-sample agreement on training neighbors is a valid proxy for model quality on the target, so the claimed gains over fixed ensembles may be largely an artifact of the LLM selecting models that memorized the retrieved points rather than models that generalize locally. I request the authors to either retrieve neighbors from a held-out validation set (where model predictions are out-of-sample) or use cross-validated predictions on the training set, and rerun the main comparisons; they should also report how often the models selected in step c actually outperform the globally best model on the corresponding validation neighborhoods.","section":"Section 3.3 / Eq. (5)"},{"comment":"The hyperparameters (number of neighbors k=10, hard-sample threshold tau=3/4, temperature 0.2, and the choice of distance metric) are ablated and effectively tuned on the same five datasets (BAS, DIS, SYL, CRE, FOR) that are also used in the main comparison in Table 1 and in Figures 4-8. This creates a risk of selection bias: the reported performance on those datasets may be optimistic because the method is tuned on the evaluation set. The authors should use a separate dataset-level validation split for hyperparameter selection or justify why the chosen values are dataset-independent a priori; at minimum, the paper should acknowledge this overlap and report results with a fixed default configuration on all datasets.","section":"Section 4.3 / Appendix B"},{"comment":"The paper claims that CoT2 'significantly outperforms' MetaXGB, but the per-dataset results in Table 4 show that CoT2-Deepseek-v3 loses to MetaXGB on BAS (94.55 vs 95.52), WQW (63.59 vs 63.67), and JC2 (95.26 vs 98.57), and ties on VUL. The Wilcoxon-Holm critical-difference diagram is the only statistical evidence, but the exact p-values or the number of wins/losses across the 30 datasets are not reported in the text. I ask the authors to report the paired test statistics and to clarify whether the advantage over MetaXGB and over the simple Averaging baseline is significant at the 0.05 level after correction; this is important because the regression results in Table 5 show CoT2 with average rank 3.00 versus Simple Average rank 3.67, which is a much smaller margin than the classification diagram suggests.","section":"Section 4.2"}],"minor_comments":[{"comment":"The evaluation protocol repeats each experiment with five random seeds and reports mean ± std, but the LLM inference temperature is 0.2; the reported variance likely reflects both model training randomness and LLM sampling stochasticity. The authors should clarify whether the LLM responses are fixed across seeds or resampled, and whether the same LLM output is used for all five runs per dataset.","section":"Section 4.1"},{"comment":"The claim that 'no prior work has explored using LLMs as intelligent ensemble experts' is too strong; there are existing works on LLM-based ensembling of model predictions (e.g., LLM-Blender) and on LLMs for model selection. The authors should soften this novelty claim and cite relevant prior art.","section":"Section 2.4"},{"comment":"The regression prompt in Figure 10 contains a regex extraction error: the text in Section 3.3 says the regex is changed to '(-?d+.d+)' but the displayed code still uses 'I predict the label of the target instance as (\\d+)'. Also, the backslashes in the regex are missing in the text. This should be corrected for reproducibility.","section":"Appendix D"},{"comment":"The notation for the number of selected models is confusing: the paper writes 'M_s' for both the set of models and its cardinality. Please use a clearer notation, e.g., 'S' for the selected set and '|S|' for its size.","section":"Section 3.3"},{"comment":"Equation (2) includes '{y_j}_{j=1}^N' as an input to the context function, but the surrounding text inputs the entire dataset label set only as a label range/frequency summary. This is inconsistent; the authors should either remove that term or clarify that it denotes the label frequencies.","section":"Section 3.2"},{"comment":"The evaluation uses TinyBench2, which is introduced in reference [74] by the same research group that authors this paper, and the model implementations are built on TALENT ([49], also the authors' group). This does not invalidate the results, but the paper should explicitly acknowledge the potential conflict and provide a brief external sanity check, for example by reporting results on a few datasets from an independent benchmark.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The central methodological concern is the in-sample neighbor predictions used in the local model selection step, which could explain part of the reported gains. This is addressable with additional experiments, so major revision rather than rejection seems appropriate. I also note the heavy use of self-authored benchmarks (TinyBench2, TALENT) and the tuning of hyperparameters on the evaluation datasets; I would encourage the editor to have the authors address these overlap issues in the revision. The paper is otherwise clearly written and the idea is interesting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a solid new application: an LLM as an instance-wise ensemble aggregator for tabular data, using only model predictions and neighbor labels, no raw features or semantic names. The CoT2 prompting decomposition is a reasonable way to make the LLM reason about local model performance. The authors also include a sensible non-LLM baseline (MetaXGB) and a few honest ablations, including anonymizing model names. That part is good.\n\nThe soft spot is in the evaluation. The neighbors used to build the context are retrieved from the same training set the external models were fit on, so every neighbor prediction the LLM sees is an in-sample prediction. For KNN and high-capacity GBDTs, those predictions are near-perfect regardless of out-of-sample quality. The LLM's 'suitable model selection' step (Eq. 5) can therefore reward models that memorized the neighbors instead of models that generalize to the target. The prompt example even shows KNN train accuracy 1.00 while KNN disagrees with a neighbor's true label, which highlights the confusion. The right fix is straightforward: use out-of-fold predictions for the training set, or retrieve neighbors from the validation set. As it stands, the reported gains over standard averaging and MetaXGB likely include some in-sample bias, and per-dataset regression results show several losses to both. So the average rank is real but the margin is not trustworthy.\n\nOther issues are minor: the paper doesn't release code, the critical-difference figure is underdocumented, and one figure caption says 'right prediction' on an example that is actually wrong. The benchmark and the toolbox are from the same group, and hyperparameters are ablated on a subset of the same datasets used in the main comparison — mild circularity.\n\nNet: the idea is novel and worth engaging. The in-sample neighbor problem is a flaw but a fixable one. I would send this to a serious referee, and the referee should ask for out-of-fold evaluation or a validation-set retrieval experiment. Without that, treat the headline numbers as optimistic.","headline":"Novel LLM-as-ensemble-aggregator idea, but the main evaluation feeds in-sample neighbor predictions to the LLM, so the reported gains are probably inflated.","tokens_in":28921,"tokens_out":4264,"would_cite":true,"duration_ms":40501,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A chain of tabular thoughts lets an LLM act as an instance-wise ensemble expert, achieving the best average ranking across TinyBench2 classification and regression tasks.","keywords":["tabular data","ensemble learning","large language models","chain-of-thought prompting","instance-wise prediction","retrieval-augmented context","model selection","TinyBench2"],"falsifier":"Replace the in-sample neighbor predictions in the prompt with out-of-fold predictions (each neighbor excluded from the training of the models that predict it) and rerun the TinyBench2 comparison; if CoT2's best average ranking falls behind standard weighted voting, the gain came from in-sample optimism rather than genuine local model selection.","tokens_in":27935,"feed_emoji":"🧠","tokens_out":9344,"duration_ms":80596,"temperature":0.7,"pith_summary":"CoT2 is a method for tabular prediction that delegates model ensembling to a large language model. For every test row, it builds a prompt from the row's nearest neighbors, their true labels, the predictions of several trained external models, and dataset-level statistics such as class balance and per-model training and validation accuracy. A four-step chain of thought—select well-performing models, discard outlier neighbors, choose locally suitable models, then decide—guides the LLM to an instance-specific integration of the external votes. On the TinyBench2 benchmark the authors report the best average ranking among all compared methods on classification and regression tasks, ahead of standard voting rules, best-model selection, and a non-LLM meta-learner. A consensus gate that skips LLM calls on easy rows further reduces inference cost.","feed_headline":"Chain-of-thought ensembling tops tabular benchmarks","feed_subtitle":"A prompt of neighbor labels and model votes lets an LLM pick the best predictor for each row.","key_machinery":"The load-bearing mechanism is the tabular context, a per-instance prompt assembled without raw features: the $K$ nearest neighbors (found by Manhattan distance re-weighted with mutual information), their true labels, each external model's predictions on neighbors and target, the external models' training and validation accuracies, and label frequencies (classification) or label range (regression). Carrying the reasoning is the Chain of Tabular Thoughts (CoT2), a four-step prompting strategy that asks the LLM to (a) identify well-performing models, (b) flag outlier neighbors, (c) select the models most suited to the clean local neighborhood, and (d) make the final prediction by combining local labels with the chosen models' outputs. A consensus threshold (agreement of at least $\\tau = 3/4$ of external models) decides which instances skip LLM inference entirely.","core_discovery":"The paper's central discovery claim is that a large language model, given no raw features and no semantic descriptions, can act as a competition-style ensemble expert by reading a structured 'tabular context'. The context encodes the local evidence around each test instance: the true labels and external-model predictions of its nearest neighbors, the external models' predictions on the target itself, the models' training and validation accuracies, and the label distribution. CoT2 then instructs the LLM to reason in four explicit steps, first filtering models and neighbor outliers, then selecting models that fit the local neighborhood, and finally combining local neighbor labels with the chosen models' votes. The authors report that this instance-wise, LLM-driven integration achieves the best average ranking on TinyBench2 classification and regression tasks, improving on the same LLM prompted without the chain of thought and on all static ensemble baselines.","pith_inferences":["A testable extension the paper does not run: feed the LLM out-of-fold neighbor predictions (each neighbor predicted by models trained without it); the result would separate genuine local model competence from agreement that an in-sample fit can produce.","The tabular context could drive other LLM outputs besides a single label, such as a confidence score, an abstention decision, or a natural-language explanation of which models were trusted and why; the paper's framework already supplies the evidence for such outputs.","The consensus gate is set at a fixed threshold $\\tau = 3/4$; the paper's own ablation shows accuracy and inference cost both change with $\\tau$, so per-dataset threshold tuning or learning the threshold from validation data is a plausible way to push the cost-accuracy frontier.","Because the LLM never sees feature values, the same interface could be deployed as a model-votes-only API for privacy-sensitive tabular data, with external models running locally and only non-semantic predictions leaving the data owner's environment."],"forward_implications":["CoT2 achieves the best average ranking across 30 TinyBench2 classification datasets and the best average ranking across 15 regression datasets, surpassing all single models and all tested static ensemble rules.","Adding the four-step chain of thought improves accuracy over the same LLM using the bare tabular context; for example, gpt-3.5-turbo mean accuracy rises from 84.79 to 86.85 on five reported datasets.","The consensus gate skips LLM inference for the majority of easy test rows, so the method preserves most of its accuracy while substantially lowering inference cost.","Anonymizing external model names improves performance on four of five datasets, suggesting the LLM can reason from model behavior rather than model identity.","Increasing the external model pool size and including stronger models improves CoT2's accuracy, so the framework benefits from richer ensembles."],"supporting_citations":[{"why":"Provides TinyBench2, the benchmark whose average ranking is the paper's headline result, and supplies the evaluation protocol.","marker":"[74]"},{"why":"Chain-of-thought prompting, the template that CoT2 adapts to tabular contexts.","marker":"[69]"},{"why":"XGBoost, one of the external models whose predictions fill the tabular context and a baseline in the comparison.","marker":"[17]"},{"why":"LightGBM, another external model in the pool and a baseline.","marker":"[43]"},{"why":"CatBoost, another external model in the pool and a baseline.","marker":"[61]"},{"why":"Supplies the ResNet and FT-Transformer architectures used as external models and baselines.","marker":"[26]"},{"why":"TabR, a retrieval-augmented tabular baseline that frames the neighbor-context design and appears in the comparison.","marker":"[25]"},{"why":"TabM, a recent deep ensemble method, compared as a baseline in the leaderboard.","marker":"[23]"}],"fun_headline_variants":["LLM ensemble reasoning tops static tabular baselines","CoT2: LLM picks best model per row for tabular win","Instance-level ensembling via interpreted LLM reasoning","Chain of thought turns LLM into tabular ensemble expert","LLM reads neighbor votes to beat static tabular ensembles"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that how well external models predict on the training-set neighbors of a target tells the LLM how well those models will predict on the target itself, even though those neighbor predictions are in-sample and therefore optimistically biased.","fun_headline_variants_meta":{"raw":{"variants":["LLM ensemble reasoning tops static tabular baselines","CoT2: LLM picks best model per row for tabular win","Instance-level ensembling via interpreted LLM reasoning","Chain of thought turns LLM into tabular ensemble expert","LLM reads neighbor votes to beat static tabular ensembles"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000449,"raw_usage":{"total_tokens":2253,"prompt_tokens":924,"completion_tokens":1329,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":1246}},"tokens_in":540,"tokens_out":1329,"duration_ms":9420,"temperature":1.0,"reasoning_tokens":1246,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:13:39.347980+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the in-sample neighbor predictions in the prompt with out-of-fold predictions (each neighbor excluded from the training of the models that predict it) and rerun the TinyBench2 comparison; if CoT2's best average ranking falls behind standard weighted voting, the gain came from in-sample optimism rather than genuine local model selection.","supporting_citations":[{"cited_title":"Catboost: unbiased boosting with categorical features","cited_arxiv_id":null,"evidence_quote":"CatBoost, another external model in the pool and a baseline."}],"review_version":1}