{"id":"120fe5ca-bc45-4b09-b87b-54d1cc5f7b87","arxiv_id":"2501.16247","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A prompt-based algorithm that constructs CART-style decision trees from feature descriptions alone, using LLM probability estimates instead of data.","lead":"This paper builds decision trees by asking a large language model to propose splits and estimate class probabilities, with no labeled data. The results are mixed: the trees beat a zero-shot text baseline on some benchmarks but fail on a dataset released after the model's training cutoff.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Probability estimates that drive split selection are never validated against real frequencies, and the one post-cutoff dataset fails; the zero-shot generalization claim is unsupported.","rationale":"The reader's weakest assumption is exactly the load-bearing point: the whole pipeline is driven by LLM probability estimates, and the paper offers no direct validation of them. Section 3.4 describes prompt calibration to fix overconfidence, which means the final prompts were fitted to the observed behavior of the pipeline; Appendix 9.1.5 further biases the estimator by telling the model that splits improve purity. On the only post-cutoff dataset, Presidential Approval, the method lands near chance at every depth while TabLLM achieves 66%, so the mechanism does not transfer to genuinely unseen data. The other four datasets are standard tabular benchmarks likely present in pretraining, making contamination a plausible alternative explanation for the positive results. The paper does have a genuine contribution in framing tree construction as a sequence of LLM calls, and the resulting trees are interpretable, but that does not establish the abstract's comparative and applicability claims. Since the reader already recommended REJECT, my stress-test reinforces that recommendation without moving it; if the proposed calibration check on a post-cutoff dataset were to pass, a more modest claim limited to in-distribution domains might become viable. I agree with the reader's assessment and set verdict_should_be to UNCHANGED to indicate no adjustment is needed.","tokens_in":14614,"tokens_out":6039,"duration_ms":59744,"concrete_test":"On the Presidential Approval held-out test set, for every node reached by the constructed tree, compare the LLM probability estimate from Appendix 9.1.5 with the empirical class frequency in that branch, then recompute the harmonic-mean Gini split selection using empirical probabilities in place of LLM estimates. If mean absolute calibration error is large (e.g., > 0.15) or the selected feature at any node changes, the probability premise fails and the central claim should be rejected.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that these trees offer a general zero-shot alternative to data-driven construction rests on the LLM's conditional probability estimates, because both split selection (Gini, §3.5) and leaf stopping (probability > 0.9, §3.2 and Figure 1) are computed entirely from them. The paper never checks these estimates against empirical class frequencies. Section 3.4 and Appendix 9.1.5 show the prompts were engineered during development to prevent overconfidence and instruct the model that 'splits usually improve node purity,' so the probabilities are prompt artifacts rather than measured quantities. The decisive case is Presidential Approval (Table 2): the only dataset past GPT-4o's October 2023 cutoff, where accuracy is 39–42%, near chance and far below TabLLM's 66%. The four remaining datasets are standard benchmarks within the model's training distribution, so the favorable results cannot rule out memorized dataset associations. This is exactly the limitation Section 6.2 concedes: performance is bounded by pretraining knowledge. The strongest claim in the abstract is therefore not supported by the evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a zero-shot decision-tree construction method in which a large language model (GPT-4o Mini) proposes feature splits, estimates per-branch class probabilities, and selects splits by minimizing a harmonic mean of branch Gini impurities. The resulting trees are evaluated on five binary tabular datasets and compared against TabLLM zero-shot and supervised scikit-learn decision trees. The abstract claims the zero-shot trees can outperform zero-shot baselines and be competitive with supervised trees in low-data settings. The algorithm is clearly described and the prompts are included in an appendix, but the empirical evidence consists of single runs without uncertainty quantification, and the key probability estimates are never validated against empirical frequencies.","tokens_in":14790,"tokens_out":3577,"duration_ms":35621,"significance":"If the central claims were established, the method would be a novel and practically useful contribution: it would provide interpretable, knowledge-driven decision trees that could operate without labeled data, which is attractive for data-scarce domains and for interpretable LLM outputs. The paper also deserves credit for including a dataset beyond the model's training cutoff and for openly acknowledging in Section 6.2 that performance is bounded by the LLM's pretraining knowledge. However, as presented, the evidence is far too weak to support the advertised generality: results are single runs, the split-selection heuristic (harmonic mean Gini) is unvalidated, and the only out-of-distribution test fails near chance. The contribution therefore remains an interesting proposal rather than an established method.","major_comments":[{"comment":"All reported accuracy and macro-F1 values appear to come from single runs, with no standard deviations, confidence intervals, or repeated trials. Because LLM outputs are stochastic and the construction procedure samples splits and probabilities, the differences between methods (e.g., 0.74 vs. 0.51 for Diabetes) could easily arise from decoding or prompt-seed variance. Without repeated runs and error bars, the central empirical claim that zero-shot trees outperform TabLLM is not statistically supported.","section":"Section 5, Tables 1 and 2"},{"comment":"The paper replaces CART's instance-weighted mean of branch Gini impurities with the harmonic mean, justified only by a verbal statement that it 'penalizes extreme imbalances.' No experiment or analysis shows that this surrogate behaves like CART's criterion or improves downstream accuracy. Since split selection is the core of the algorithm, the authors should validate the aggregation rule, for example by comparing trees grown with harmonic versus weighted Gini on the same datasets using true labels, or by reporting an ablation. As written, the claim that the method is 'based on CART principles' is not established.","section":"Section 3.5, harmonic mean equation"},{"comment":"The probability estimates drive both split selection (through Gini) and leaf stopping (threshold 0.9), yet they are never checked against empirical class frequencies. The prompts explicitly instruct the model to avoid high probabilities and to assume that splits usually improve purity; the estimates are therefore prompt artifacts rather than calibrated quantities. The only dataset beyond the model's cutoff, Presidential Approval (Table 2), yields accuracy around 0.40, near chance and far below TabLLM's 0.66, indicating the probability-estimation premise fails on genuinely unseen data. Section 6.2 concedes exactly this limitation, but the abstract's generality claim remains unsupported.","section":"Section 3.4, Appendix 9.1.5"},{"comment":"The abstract claims that the proposed trees 'can outperform baseline zero-shot methods,' but the evidence is mixed at best. On Weather, the zero-shot tree achieves 0.50 accuracy versus TabLLM's 0.88; on Presidential Approval it achieves 0.41 versus TabLLM's 0.66. The text acknowledges these drops informally, but the abstract and conclusion still state a general superiority, which is contradicted by the paper's own tables. The claim should be qualified to specific datasets, or the paper should demonstrate a consistent advantage across a broader benchmark.","section":"Tables 1 and 2, Weather and Presidential Approval rows"}],"minor_comments":[{"comment":"The sentence 'One of the main advantages of our proposed method its the interpretability of decision tress mixed with zero-shot training' contains typos; it should read 'is the interpretability' and 'trees.'","section":"Section 5.1"},{"comment":"The Weather dataset was transformed by selecting only two opposing climates, which may make the task artificially simple; this subsampling should be described more precisely, including the number of classes and the split sizes.","section":"Section 4.2"},{"comment":"The algorithm description does not specify how the maximum depth interacts with the 0.9 probability threshold when both are reached simultaneously; Figure 1 suggests priority to the probability check, but this is not stated in the text.","section":"Section 3.2"},{"comment":"The prompt listings contain many spacing artifacts (e.g., 'a tt rib ut e', 'grpups'), which make them hard to read; they should be cleaned for reproducibility.","section":"Appendix 9"},{"comment":"The paper does not include a reproducibility statement or code availability, which is important for an empirical claim of this type; the appendix provides prompts, but details such as decoding temperature and number of repetitions are missing.","section":"General"}],"recommendation":"reject","confidential_remarks":"The idea is interesting and the paper is clearly written, but the empirical support is not sufficient for a journal publication: single-run results, an unvalidated core design choice, and a decisive failure on the only post-cutoff dataset. These are not merely cosmetic issues; they undermine the main claims. If the authors can add rigorous evaluation with error bars, validate the harmonic-mean Gini against CART on real data, and demonstrate that the probability estimates are calibrated or that the method scales to genuinely new domains, a resubmission would be worth considering."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this paper describes a new way to build a decision tree by asking an LLM to propose splits, estimate branch probabilities, and then pick the split with lowest harmonic-mean Gini impurity. No training data, no fitted probabilities. The pipeline is clearly described, the prompts are in an appendix, and the construction logic is simple enough to follow. The idea is genuinely new in the sense that the cited work (MetaTree, TabLLM) does not do this. Credit where due: the authors also included a dataset past GPT-4o's knowledge cutoff and explicitly discuss limitations like bias and dependence on pretraining knowledge.\n\nThe soft spot is the empirical support for the central claim. The experiments are single runs with no variance estimate, and no repeated trials or seeds. The probability estimates that drive split selection are never validated against actual class frequencies. Worse, the prompts and the harmonic-mean aggregation were refined during development to make the method work on these datasets—Section 3.1 and the appendix make that clear. That is fitting to the benchmark, not a clean zero-shot evaluation. And the decisive test fails: Presidential Approval, the only dataset after the model's cutoff, gives about 40% accuracy, essentially chance and far below TabLLM's 66%. The abstract says the method outperforms zero-shot baselines, but that is not true for Weather or Presidential. So the strong generalization claim is not supported.\n\nI would still send this to a serious referee. The idea is worth engaging: if the probability estimates could be made reliable, the tree construction is a legitimate low-data tool. But the current evidence is a preliminary demo, not a result. The authors need error bars, probability calibration checks, a post-cutoff test set larger than one dataset, and an ablation of the aggregation rule. This paper deserves peer review, but my verdict on the current version is reject—not because the method is worthless, but because the evaluation does not back the abstract.\n\nWho is this for? Anyone working on zero-shot tabular learning or interpretable LLM use. It is a useful cautionary data point, and the prompt design is a good starting point for follow-up work.\n\nRecommendation: send to review, request major revision, and ask for the evaluation work described above.","headline":"A genuinely novel prompt-based tree construction pipeline that fails to establish its main zero-shot claim because the probability estimates are unvalidated and the one post-cutoff dataset collapses to chance.","tokens_in":15341,"tokens_out":3229,"would_cite":false,"duration_ms":31542,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T05","68T07","62H30"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that decision trees can be built with zero labeled data by having an LLM supply CART's splits and probabilities, beating a zero-shot baseline on several tabular tasks.","keywords":["zero-shot learning","decision trees","large language models","tabular data","Gini impurity","CART","interpretability","low-data classification"],"falsifier":"A direct test would build trees from real feature names and from randomly permuted feature labels on the same datasets: if accuracy does not drop when names are scrambled, the method is not using knowledge at all. The paper's own Presidential Approval result, about 40% accuracy near chance on post-cutoff data, already shows the failure mode to look for.","tokens_in":14361,"feed_emoji":"🌳","tokens_out":6932,"duration_ms":54691,"temperature":0.7,"pith_summary":"The paper tries to establish that the recursive partitioning logic of CART can be run without any labeled examples, using a large language model as the source of every quantity a decision tree normally derives from data. The LLM proposes candidate thresholds and categorical groupings, estimates conditional class probabilities for each branch, and the algorithm selects splits by Gini impurity, growing a tree until a branch reaches a 0.9 probability or the depth limit. On five tabular datasets the resulting zero-shot trees beat the TabLLM zero-shot baseline on several tasks and roughly match supervised decision trees trained on only 4 to 32 examples. If true, this gives practitioners an interpretable, knowledge-driven model for domains where labeled data is scarce, expensive, or privacy-restricted.","feed_headline":"LLM-built decision trees beat zero-shot baselines without data","feed_subtitle":"An LLM's probability estimates grow interpretable trees that outperform zero-shot baselines, no training data needed.","key_machinery":"The load-bearing mechanism is the LLM acting as a conditional-probability oracle inside a CART-style loop. Each prompt supplies the feature description, the branch constraints on the current path, and the previous node's class probabilities, and asks for a rough estimate of class probabilities for each proposed split; those estimates feed the Gini formula $Gini = 1 - \\sum_i p_i^2$, and branch impurities are aggregated with the harmonic mean $2 G_1 G_2/(G_1+G_2)$ because no instance counts are available. The same probability estimates trigger leaf creation when a class exceeds $0.9$, and split proposals respect constraints accumulated down the tree, such as adjusted upper and lower bounds on numeric features and remaining categories for categorical features.","core_discovery":"The central claim is that a decision tree can be constructed as a pure knowledge-driven artifact: the LLM proposes split values and category groupings from feature descriptions alone, estimates the probability of each target class in each branch given the constraints accumulated along the tree path, and the algorithm chooses the split with the lowest Gini impurity, with the two branch impurities combined by harmonic mean instead of CART's instance-weighted average. The paper reports that this method outperforms TabLLM zero-shot on Diabetes, Credit, and Hepatic damage, reaches 83% accuracy on Hepatic damage against 93% for a supervised tree on full data, and is competitive with supervised trees in 4-shot to 32-shot regimes. The authors present this as establishing a new zero-shot baseline for interpretable tabular classification.","pith_inferences":["A natural extension the paper does not explore is replacing the fixed 0.9 leaf threshold with a per-dataset confidence threshold, since the reported sensitivity to depth suggests stopping rules materially change accuracy.","Because the method relies on semantic feature names, it likely degrades when features are anonymized or uninformative; comparing trees built from real feature names against trees built from randomly permuted names would isolate how much genuine knowledge the method uses.","The harmonic-mean aggregation is a design choice worth stress-testing: alternative data-free aggregations such as the geometric mean or the maximum branch impurity would change split selection and tree shape even when the LLM's probability estimates are fixed."],"forward_implications":["Zero-shot decision trees become a usable baseline whenever labeled data is unavailable, and deployment cost is just the small tree rather than per-example LLM inference.","The resulting trees expose the LLM's reasoning in an auditable form, with thresholds such as glucose ≤ 140, which direct text classification does not provide.","The method's performance is bounded by the LLM's knowledge: datasets inside its training distribution can be handled reasonably, while genuinely post-cutoff data shows near-chance accuracy in the paper's own Presidential Approval experiment.","Increasing tree depth generally helped on the datasets where it was tested, so leaf-stopping rules and depth limits are part of the method's behavior, not incidental details."],"supporting_citations":[{"why":"Supplies the CART algorithm whose recursive Gini-based partitioning the method reproduces without data.","marker":"[Breiman(2017)]"},{"why":"Supplies the classic decision-tree induction framing that the paper positions its zero-shot approach against.","marker":"[Quinlan(2014)]"},{"why":"Provides TabLLM, the zero-shot tabular classification baseline whose prompts and results are compared.","marker":"[Hegselmann et al.(2023)]"},{"why":"Grounds the assumption that LLMs can perform zero-shot reasoning from pre-trained knowledge.","marker":"[Brown et al.(2020)]"},{"why":"Provides the scikit-learn decision tree implementation used as the supervised baseline.","marker":"[Pedregosa et al.(2012)]"},{"why":"Documents the GPT-4o model and its October 2023 knowledge cutoff, supporting the claim that the Presidential Approval dataset is unseen.","marker":"[OpenAI(2023)]"},{"why":"Supplies MetaTree, a transformer-based decision-tree learner, as the closest related work on learned tree induction.","marker":"[Zhuang et al.(2024)]"}],"fun_headline_variants":["LLM-built trees rival supervised ones, no data needed","Zero-shot decision trees from LLM knowledge beat baselines","LLMs grow interpretable trees without any training data","LLM probability estimates craft competitive decision trees","Knowledge-driven trees from LLMs match supervised accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire tree rests on the language model's probability estimates being accurate enough proxies for real conditional class distributions, because every split choice and stopping decision is computed from those estimates alone.","fun_headline_variants_meta":{"raw":{"variants":["LLM-built trees rival supervised ones, no data needed","Zero-shot decision trees from LLM knowledge beat baselines","LLMs grow interpretable trees without any training data","LLM probability estimates craft competitive decision trees","Knowledge-driven trees from LLMs match supervised accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000146,"raw_usage":{"total_tokens":1143,"prompt_tokens":865,"completion_tokens":278,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":481,"completion_tokens_details":{"reasoning_tokens":203}},"tokens_in":481,"tokens_out":278,"duration_ms":2811,"temperature":1.0,"reasoning_tokens":203,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T13:36:09.192196+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test would build trees from real feature names and from randomly permuted feature labels on the same datasets: if accuracy does not drop when names are scrambled, the method is not using knowledge at all. The paper's own Presidential Approval result, about 40% accuracy near chance on post-cutoff data, already shows the failure mode to look for.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the scikit-learn decision tree implementation used as the supervised baseline."}],"review_version":1}