Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

GPTree: Towards Explainable Decision-Making via LLM-powered Decision Trees

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read GPTree claims an LLM-generated-question decision tree identifies unicorn startups at 7.8% precision, beating GPT-4o few-shot and the best human venture investors (3.1% to 5.6%).

desk verdict A novel LLM-decision-tree framework whose headline comparison to human VCs is undermined by an unjustified prevalence rescale and inconsistent arithmetic, though the underlying idea deserves a serious look. read the letter →

arxiv 2411.08257 v1 pith:BCNZ7SON submitted 2024-11-13 cs.LG cs.AIcs.CE

Sichao Xiong , Yigit Ihlamur , Fuat Alican , Aaron Ontoyin Yin This is my paper · ORCID
classification cs.LGcs.AIcs.CE
keywords explainableAIdecisiontreeslargelanguagemodelsGPTreeventurecapitalstartupsuccesspredictionexpert-in-the-loopGiniimpurity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that decision trees can handle complex, text-heavy data by letting a large language model propose the split questions, while keeping the tree's transparency. On a dataset of nearly 10,000 founders, GPTree reaches 37.3% precision at the dataset's 9.9% success rate, which the authors scale down by 5.5x to claim a 7.8% precision rate for identifying unicorn startups at inception. That scaled figure would exceed both GPT-4o with few-shot prompting and the best human venture investors. The framework needs only a task description, not feature engineering or prompt chains, and includes an expert-in-the-loop step that can rebuild or refine decision paths.

What carries the argument

The central object is GPTree, a decision tree whose internal nodes are natural-language questions answered by an LLM rather than threshold conditions on numeric features. The tree is grown greedily: a first LLM pass summarizes success patterns from batches of samples, a second pass generates zero to three candidate questions per feature, and the question yielding the lowest weighted Gini impurity is chosen at each node. An expert can later collapse nodes, rebuild subtrees, or advise question generation, and a sensitivity parameter converts leaf-node class proportions into binary labels by optimizing F0.5 on a validation set. This machinery supplies explainability, since every split is a readable question, while shifting pattern recognition from hand-crafted features to LLM judgment.

What would settle it

Apply the exact decision rule learned by GPTree, with its F0.5-optimal sensitivity, to a random sample from the industry population whose true unicorn rate is 1.9%, and count precision directly; if the resulting precision falls below 3.1% to 5.6%, the paper's headline comparison fails. Alternatively, derive precision from the confusion matrix at 9.9% prevalence and recompute it under 1.9% prevalence using the same score threshold; any answer that differs from 7.8% falsifies the linear-scaling assumption.

Watch

Extended reading notes

Core claim

The central claim is that LLM-generated questions can serve as decision-tree splits and yield better precision than both general-purpose LLM prompting and human experts in a high-stakes prediction task. Specifically, the paper reports that GPTree, using a GPT-4o-mini backend and greedy weighted-Gini split selection, reaches 37.3% precision and 27.1% recall on the founder-success test set, rising to 40.8% precision with expert refinement. For comparison, vanilla GPT-4o reaches 15.7% precision and few-shot GPT-4o reaches 16.2% precision on the same dataset. After the authors scale precision by the base-rate ratio between their dataset and the industry, they claim a 7.8% precision rate for identifying unicorn startups, surpassing the 3.1% to 5.6% range of human decision-makers.

Load-bearing premise

The claim that GPTree beats humans and GPT-4o rests on scaling measured precision linearly by the ratio of dataset success rate to industry success rate (9.9% versus 1.9%, a 5.5x adjustment), but precision does not generally scale linearly with class prevalence, so the 7.8% headline number is only as good as that assumption.

Editorial extensions

If this is right

  • If GPTree's precision claim survives proper base-rate adjustment, LLM-powered decision trees would give venture firms an explainable screening tool that outperforms both raw LLM prompting and human experts.
  • The framework removes typical prompt-engineering overhead: a user supplies only a task string, and question generation, splitting, and label calibration are automated.
  • Because each split is a human-readable question, the model produces audit trails for investment decisions, which matters in regulated or high-stakes settings.
  • The expert-in-the-loop mechanism lets domain knowledge correct or refine decision paths after training without retraining the whole tree, potentially combining human and machine judgment more tightly.
  • The same recipe could transfer to other text-rich domains such as medical triage or credit decisions, provided labeled data and base-rate-calibrated evaluation are used.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 5.5x linear scaling of precision is not a generally valid statistical adjustment: precision depends on the score threshold and the class-conditional distributions, so the true industry-prevalence precision could be higher or lower than 7.8%, and a fair comparison requires evaluating the same tree on a sample with 1.9% prevalence.
  • Because LLM answers are non-deterministic and the paper selects splits with a single Gini calculation, different runs or phrasings of the same question could produce different trees; the variance over repeated LLM calls is not quantified, so the reported margin over baselines may not be stable.
  • The paper's own limitations section acknowledges code-node unreliability and hallucination, meaning the claimed automation without feature engineering still depends on LLM correctness at every node.
  • A testable extension would be to report precision-recall curves or confusion matrices at multiple sensitivity thresholds, letting readers see whether the human-baseline superiority holds across operating points rather than only at the F0.5-optimal point.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces GPTree, a decision tree whose split questions are generated by an LLM from a task prompt and from summaries of positive samples, with splits selected greedily by weighted Gini impurity and optional expert-in-the-loop refinement. The authors evaluate GPTree on a founder-success dataset of 9,892 founders (9.9% positive) and report 37.3% precision and 27.1% recall. They then divide the precision by 5.5 to obtain a 7.8% figure, which they compare with industry benchmarks (1.9% indexing strategy, 3.2% Y Combinator, 5.6% Tier-1 VCs) and with gpt-4o baselines, claiming that GPTree surpasses both GPT-4o and the best human decision-makers.

Significance. If the empirical claims were valid, an explainable decision tree that identifies future unicorns more precisely than top VC firms would be a meaningful result. The paper has strengths: the pipeline is clearly described, it reports per-fold results in Table 5, and it explicitly lists limitations in Section 6.2. The idea of having an LLM propose interpretable split questions and then selecting splits by Gini impurity is a reasonable way to make LLM-based decisions more inspectable. However, the headline comparison to human decision-makers rests on an unjustified linear rescaling of precision, and the expert-in-the-loop mechanism does not improve the reported F0.5. The central empirical claim is therefore not supported as stated.

major comments (4)
  1. [Section 4.2, Abstract, Table 4] The headline 7.8% precision is obtained by dividing the measured 37.3% by 5.5 because the dataset prevalence (9.9%) is 5.5 times the industry outlier rate (1.9%). This linear rescaling is not a valid way to transport precision across prevalences. For a fixed decision rule, precision is p*TPR / (p*TPR + (1-p)*FPR), which is a nonlinear function of prevalence p. From Table 4, TPR = 0.271 and, using precision 0.373 and p = 0.099, the implied FPR is roughly 0.050; applying the same TPR and FPR at p = 0.019 gives a precision near 9.5%, not 6.8% (37.3/5.5). The paper provides no score-distribution or calibration analysis to justify the constant divisor, and it also assumes without evidence that TPR and FPR are population-invariant. Since every superiority claim over human benchmarks (3.1% to 5.6%) depends on this transformed number, the central empirical claim is not established.
  2. [Section 3.5, Tables 4 and 5] The abstract and Section 3.5 claim that the expert-in-the-loop feedback mechanism further enhances performance, but the averaged test results in Table 5 do not support this. The F0.5 score without expert refinement is 33.4, while with expert refinement it is 33.2, and recall drops from 27.1% to 23.4% while precision rises from 37.3% to 40.8%. On the stated primary metric (F0.5, justified in Section 4.1), expert refinement slightly hurts performance. The paper needs to either report a metric that improves with expert refinement or temper the claim.
  3. [Section 2.1, Table 4] The dataset contains founder profiles of companies founded between 2010 and 2016, but the paper does not state when the profile features were collected. If features such as current employment, funding history, or social-reach proxies were scraped after the companies became successful, the model may be using post-outcome information, making the task partially retrospective and inflating precision. The authors should state the timestamp of the feature collection or otherwise demonstrate that all features were available at inception time.
  4. [Table 5, Section 4.5] The averaged results are reported without confidence intervals or significance tests, and the per-fold test precision varies widely (e.g., from 19.9% to 50.6% across the rows shown). With this variance, the claim that GPTree 'significantly outperform[s]' baselines and human decision-makers is not statistically supported. In addition, the proprietary Vela model with 17.9% precision is not cross-validated and is insufficiently described for assessment.
minor comments (5)
  1. [Introduction] The text says 'gpt-1o preview'; this appears to be a typo for 'o1' or 'gpt-4o preview'.
  2. [Equations (1) and (2)] The displayed equations have formatting artifacts (e.g., 'kX', 'Pk', 'CX') in the rendered text; they should be typeset cleanly.
  3. [Table 5] The asterisks on folds 8 and 10 are unexplained; please add a footnote.
  4. [Section 5] The description of cross-validation says 'all ten possible combinations of three folds' and then reports '20 partitions'; the connection between the 10 combinations and the 20 partitions is not immediately clear and should be spelled out.
  5. [References] The paper cites the GPT-4 technical report for gpt-4o; a more specific reference or model card for gpt-4o would be helpful.

Circularity Check

0 steps flagged · score 0.0 of 10

No equation-level circularity: GPTree's training, validation tuning, and held-out test evaluation are self-contained, and the contested 5.5x precision rescaling is a statistical extrapolation concern, not a circular derivation.

full rationale

The derivation chain is not circular. GPTree generates candidate questions with an LLM, selects splits by minimizing the weighted Gini impurity (Eqs. 1-2), tunes the leaf sensitivity on a validation set by F0.5, and evaluates on held-out test folds; the target result is not used to define the algorithm or its inputs. The only external citations are Brown et al. (2020), OpenAI (2024), and Tang et al. (2023); none is a self-citation carrying a uniqueness or ansatz argument. The main weakness is Section 4.2's statement that the reported metrics 'should be scaled down 5.5x to match the 1.9% outlier rate in the industry,' from which the abstract's 7.8% is obtained. This is a doubtful statistical assumption because precision is not generally a constant linear function of class prevalence, and it is therefore a correctness/validity risk rather than a circular step: the rescaled number is not defined as the quantity it is supposed to predict, and no fitted parameter is renamed as an independent prediction. The proprietary Vela model's 17.9% figure and the expert refinement process are under-reported and not independently reproducible, which reduces evidential weight but does not make the derivation circular.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The ledger captures the main hand-chosen and fitted quantities: sensitivity thresholds, hyperparameters, and the 5.5x scaling factor. It also records the domain assumptions about data temporality, LLM reliability, and comparability of human benchmarks. No new physical or conceptual entities are introduced.

free parameters (5)
  • sensitivity threshold = 0.12 to 0.48 per fold (Table 5)
    Selected per fold on the validation set to maximize F0.5; directly determines leaf labels and reported precision and recall.
  • prevalence scaling factor = 5.5
    Applied to convert dataset precision (9.9% base rate) to industry precision (1.9% base rate); no justification that precision scales linearly, and all headline claims depend on it.
  • maximum tree depth = 18
    Hand-chosen hyperparameter to prevent overfitting; affects tree complexity and results.
  • minimum samples per leaf = 31 (0.5%)
    Hand-chosen hyperparameter; affects split stopping and generalization.
  • summarization batch size = 250
    Chosen for LLM context management; affects insight summaries and downstream question generation.
assumptions (4)
  • domain assumption Precision scales linearly with class prevalence
    Section 4.2 and Figure 1: the 7.8% headline assumes dividing by 5.5 preserves precision. This is not generally true and is not empirically checked.
  • domain assumption Founder profile features do not contain post-outcome information
    Section 2: profiles collected and enriched from current LinkedIn and Crunchbase sources for companies founded 2010-2016; no feature list or temporal cutoff is given.
  • domain assumption LLM inference nodes are sufficiently reliable and deterministic
    Sections 3.3 and 6.2: INFERENCE nodes use GPT-4o-mini answers to yes/no questions; the paper itself notes non-deterministic evaluations and hallucination.
  • domain assumption Human fund benchmarks are comparable to GPTree's founder-level evaluation
    Section 4.2: human rates come from 115K companies founded 2014-2016, while the model runs on 9,892 founders from companies founded 2010-2016; different units and selection criteria make the comparison indirect.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GPTree: Towards Explainable Decision-Making via LLM-powered Decision Trees." pith.science (2026). https://pith.science/paper/BCNZ7SON

@misc{pith2026241108257,
  author       = {Pith},
  title        = {Pith review of: GPTree: Towards Explainable Decision-Making via LLM-powered Decision Trees},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BCNZ7SON}},
  note         = {Machine review of arXiv:2411.08257}
}
read the original abstract

Traditional decision tree algorithms are explainable but struggle with non-linear, high-dimensional data, limiting its applicability in complex decision-making. Neural networks excel at capturing complex patterns but sacrifice explainability in the process. In this work, we present GPTree, a novel framework combining explainability of decision trees with the advanced reasoning capabilities of LLMs. GPTree eliminates the need for feature engineering and prompt chaining, requiring only a task-specific prompt and leveraging a tree-based structure to dynamically split samples. We also introduce an expert-in-the-loop feedback mechanism to further enhance performance by enabling human intervention to refine and rebuild decision paths, emphasizing the harmony between human expertise and machine intelligence. Our decision tree achieved a 7.8% precision rate for identifying "unicorn" startups at the inception stage of a startup, surpassing gpt-4o with few-shot learning as well as the best human decision-makers (3.1% to 5.6%).

Figures

Figures reproduced from arXiv: 2411.08257 by the authors.

Figure 1
Figure 1. Comparison of Different Models/Methods non-linear, high-dimensional datasets and are inherently un￾suitable for text-rich and multi-modal datasets. Thus, our work to extend decision trees is motivated by this fundamen￾tal question: how to incorporate LLMs? In recent years, Large Language Models (LLMs) have emerged as powerful tools capable of capturing the intrica￾cies of natural language: models such as gpt-4o and … view at source ↗
Figure 2
Figure 2. GPTree pipeline [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Decision Tree example summaries into a cohesive list of insights. This refined list is then passed onto the question candidate generation process. 3.3. Question Candidate Generation Unlike traditional decision trees, which have a finite number of input values and therefore a finite set of candidate condi￾tions to test, natural language inputs introduce an unbounded question space. To make the problem more tractable,… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Ensembling LLM-Induced Decision Trees for Explainable and Robust Error Detection

    cs.CL 2025-12 conditional novelty 6.0 of 10

    LLM-induced hybrid decision trees (rules + trained graph checks) ensembled via EM detect erroneous table cells with an average 16.1-point F1 gain over the best baseline.

  2. From Limited Data to Rare-event Prediction: LLM-powered Feature Engineering and Multi-model Learning in Venture Capital

    cs.LG 2025-09 conditional novelty 4.0 of 10

    An LLM-feature-driven ensemble predicts billion-dollar startup outcomes with 9.8X to 11.1X the precision of a random classifier, but the label and the model's intermediate target are both funding, so the result partly...

Reference graph

Works this paper leans on

4 extracted references · 2 linked inside Pith · cited by 2 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, ...

    arXiv 2020

  3. [3]

    Gpt-4 technical report, 2024

    OpenAI. Gpt-4 technical report, 2024. URL https://arxiv.org/abs/2303.08774

    arXiv 2024

  4. [4]

    Large language models can be lazy learners: Analyze shortcuts in in-context learning

    Tang, R., Kong, D., Huang, L., and Xue, H. Large language models can be lazy learners: Analyze shortcuts in in-context learning. In Findings of the Association for Computational Linguistics: ACL 2023. Association for Computational Linguistics, 2023. doi:10.18653/v1/2023.findings-acl.284. URL http://dx.doi.org/10.18653/v1/2023.findings-acl.284

    doi:10.18653/v1/2023.findings-acl.284 2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.