{"id":"417a6b21-0086-4b11-a835-0dfa345ef959","arxiv_id":"2411.08257","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"An LLM-powered decision tree is claimed to identify future unicorn startups at 7.8% precision, but the comparison rests on an unvalidated prevalence scaling and possible lookahead leakage.","lead":"GPTree builds a decision tree whose split questions are generated and answered by a large language model, then selects the best splits with Gini impurity. The authors report it finds future unicorn startups more precisely than human venture capitalists, but that headline depends on a prevalence-scaling assumption and on founder data collected after the outcome.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 7.8% headline precision is an unsupported linear rescaling by 5.5x of a 9.9%-prevalence measurement; precision does not scale as a constant ratio of base rates, so the comparison to human VCs is not established.","rationale":"The reader's weakest assumption correctly identifies the 5.5x rescaling as the central flaw. All headline superiority over human VCs rests on this transformation, and the paper provides no statistical derivation for it. The alternative Bayes-consistent adjustment produces a different number (about 8.4% vs 6.8%), showing the reported 7.8% is not robust even under the most charitable assumption of invariant error rates. Moreover, the dataset's 9.9% prevalence is itself unexplained relative to the 1.9% industry figure, so the two populations may differ in ways that affect TPR and FPR. This is a load-bearing concern because the abstract and Figure 1 explicitly compare the rescaled GPTree precision to human performance. Other issues (potential label leakage through founder profiles, no code/data release, expert refinement not improving F0.5) are real but secondary; the scaling issue alone invalidates the central claim as stated. I therefore agree with the reader's REJECT verdict and recommend no change.","tokens_in":7642,"tokens_out":7634,"duration_ms":69674,"concrete_test":"Recompute the three headline comparisons from the published confusion matrices without the 5.5x linear scaling. For each model in Tables 4 and 5, derive TPR and FPR from the reported precision, recall, and 9.9% dataset prevalence, then calculate precision at the industry 1.9% prevalence using the standard formula p*TPR / (p*TPR + (1-p)*FPR). If the adjusted precision falls below the 5.6% Tier-1 VC benchmark, the central claim fails. If it remains above, the paper must still replace the ad hoc rescaling with this calculation and, ideally, validate on a new sample drawn from the 115K-company universe with approximately 1.9% prevalence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 states that the model's precision 'should be scaled down 5.5x' because the evaluation set has a 9.9% success rate while the industry outlier rate is 1.9%. The abstract's 7.8% figure is this rescaled value, not a measured precision on an industry-representative sample. Precision is not a linear function of class prevalence. For a fixed decision rule with true-positive rate TPR and false-positive rate FPR, precision at prevalence p is p*TPR / (p*TPR + (1-p)*FPR). From the published Table 4 (precision 37.3%, recall 27.1%, accuracy 87.6%, prevalence 9.9%), the implied FPR is about 5.75%; applying the same TPR/FPR at p=1.9% gives a prevalence-adjusted precision of about 8.4%, not 6.8%. The 5.5x divisor has no theoretical justification and the paper offers no score-distribution analysis to support it. Because the dataset is an enriched 9.9%-positive sample, extrapolating to the 1.9% industry base rate also requires assuming TPR/FPR invariance across populations, which is not established. The headline claim of surpassing the best human decision-makers (3.1-5.6%) depends entirely on this invalid rescaling, so the central empirical claim is not supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces GPTree, a decision tree whose split questions are generated by an LLM from a task prompt and from summaries of positive samples, with splits selected greedily by weighted Gini impurity and optional expert-in-the-loop refinement. The authors evaluate GPTree on a founder-success dataset of 9,892 founders (9.9% positive) and report 37.3% precision and 27.1% recall. They then divide the precision by 5.5 to obtain a 7.8% figure, which they compare with industry benchmarks (1.9% indexing strategy, 3.2% Y Combinator, 5.6% Tier-1 VCs) and with gpt-4o baselines, claiming that GPTree surpasses both GPT-4o and the best human decision-makers.","tokens_in":7916,"tokens_out":5014,"duration_ms":48678,"significance":"If the empirical claims were valid, an explainable decision tree that identifies future unicorns more precisely than top VC firms would be a meaningful result. The paper has strengths: the pipeline is clearly described, it reports per-fold results in Table 5, and it explicitly lists limitations in Section 6.2. The idea of having an LLM propose interpretable split questions and then selecting splits by Gini impurity is a reasonable way to make LLM-based decisions more inspectable. However, the headline comparison to human decision-makers rests on an unjustified linear rescaling of precision, and the expert-in-the-loop mechanism does not improve the reported F0.5. The central empirical claim is therefore not supported as stated.","major_comments":[{"comment":"The headline 7.8% precision is obtained by dividing the measured 37.3% by 5.5 because the dataset prevalence (9.9%) is 5.5 times the industry outlier rate (1.9%). This linear rescaling is not a valid way to transport precision across prevalences. For a fixed decision rule, precision is p*TPR / (p*TPR + (1-p)*FPR), which is a nonlinear function of prevalence p. From Table 4, TPR = 0.271 and, using precision 0.373 and p = 0.099, the implied FPR is roughly 0.050; applying the same TPR and FPR at p = 0.019 gives a precision near 9.5%, not 6.8% (37.3/5.5). The paper provides no score-distribution or calibration analysis to justify the constant divisor, and it also assumes without evidence that TPR and FPR are population-invariant. Since every superiority claim over human benchmarks (3.1% to 5.6%) depends on this transformed number, the central empirical claim is not established.","section":"Section 4.2, Abstract, Table 4"},{"comment":"The abstract and Section 3.5 claim that the expert-in-the-loop feedback mechanism further enhances performance, but the averaged test results in Table 5 do not support this. The F0.5 score without expert refinement is 33.4, while with expert refinement it is 33.2, and recall drops from 27.1% to 23.4% while precision rises from 37.3% to 40.8%. On the stated primary metric (F0.5, justified in Section 4.1), expert refinement slightly hurts performance. The paper needs to either report a metric that improves with expert refinement or temper the claim.","section":"Section 3.5, Tables 4 and 5"},{"comment":"The dataset contains founder profiles of companies founded between 2010 and 2016, but the paper does not state when the profile features were collected. If features such as current employment, funding history, or social-reach proxies were scraped after the companies became successful, the model may be using post-outcome information, making the task partially retrospective and inflating precision. The authors should state the timestamp of the feature collection or otherwise demonstrate that all features were available at inception time.","section":"Section 2.1, Table 4"},{"comment":"The averaged results are reported without confidence intervals or significance tests, and the per-fold test precision varies widely (e.g., from 19.9% to 50.6% across the rows shown). With this variance, the claim that GPTree 'significantly outperform[s]' baselines and human decision-makers is not statistically supported. In addition, the proprietary Vela model with 17.9% precision is not cross-validated and is insufficiently described for assessment.","section":"Table 5, Section 4.5"}],"minor_comments":[{"comment":"The text says 'gpt-1o preview'; this appears to be a typo for 'o1' or 'gpt-4o preview'.","section":"Introduction"},{"comment":"The displayed equations have formatting artifacts (e.g., 'kX', 'Pk', 'CX') in the rendered text; they should be typeset cleanly.","section":"Equations (1) and (2)"},{"comment":"The asterisks on folds 8 and 10 are unexplained; please add a footnote.","section":"Table 5"},{"comment":"The description of cross-validation says 'all ten possible combinations of three folds' and then reports '20 partitions'; the connection between the 10 combinations and the 20 partitions is not immediately clear and should be spelled out.","section":"Section 5"},{"comment":"The paper cites the GPT-4 technical report for gpt-4o; a more specific reference or model card for gpt-4o would be helpful.","section":"References"}],"recommendation":"reject","confidential_remarks":"The prevalence-scaling issue is fundamental to the paper's headline result, not a local presentation problem. The within-dataset comparison between GPTree and GPT-4o on the same 9.9%-prevalence sample is more defensible, and a revised paper that confines its claims to that comparison and drops the human-benchmark claim could be reconsidered. The expert-in-the-loop result also needs to be reconciled with the reported F0.5 values."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know that GPTree is a genuinely novel idea, but its headline result about beating top VCs is not backed by the reported numbers. The framework — LLM-generated candidate questions, greedy Gini selection, expert refinement — is coherent and clearly explained. That much is real. The problem is the evaluation.\n\nThe paper takes a precision of 37.3% measured at 9.9% class prevalence and divides by 5.5 to claim about 6.8% (though the abstract says 7.8%, and Figure 1 says 7.2 and 7.8 for the two variants; none of those match 37.3/5.5=6.8 or 40.8/5.5=7.4). Beyond the arithmetic, precision does not scale linearly with prevalence. The stress-test note correctly shows that applying the implied TPR/FPR from Table 4 at 1.9% prevalence gives about 8.4%, not 6.8%. So the paper's rescaling is wrong in method and execution. Even the 'corrected' number still clears the 5.6% human benchmark, but only if you assume TPR/FPR transfer across populations — an assumption the paper neither states nor defends.\n\nOther soft spots: the expert-in-the-loop refinement lowers F0.5 (33.4 to 33.2), so that selling point isn't demonstrated. The founder profiles may contain post-outcome information, and no feature list is given. There are no confidence intervals, no code or data, and the fold-level test precision swings from about 20% to 53%. The paper also lacks standard ML baselines (e.g., a plain decision tree or logistic regression on the same features), so we can't tell what the LLM-generated splits actually buy you.\n\nThat said, the method itself is worth a serious look. The idea of using an LLM to propose splits and then selecting by Gini is a natural and potentially useful combination, and the limitations section is honest about the failure modes. I'd send this to peer review, but the authors need to redo the evaluation: justify or replace the prevalence adjustment, report confidence intervals, release the code and a feature list, and add standard baselines. As written, the central claim is unsupported, but the underlying framework has promise.","headline":"A novel LLM-decision-tree framework whose headline comparison to human VCs is undermined by an unjustified prevalence rescale and inconsistent arithmetic, though the underlying idea deserves a serious look.","tokens_in":8479,"tokens_out":5242,"would_cite":false,"duration_ms":46368,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GPTree claims an LLM-generated-question decision tree identifies unicorn startups at 7.8% precision, beating GPT-4o few-shot and the best human venture investors (3.1% to 5.6%).","keywords":["explainable AI","decision trees","large language models","GPTree","venture capital","startup success prediction","expert-in-the-loop","Gini impurity"],"falsifier":"Apply the exact decision rule learned by GPTree, with its F0.5-optimal sensitivity, to a random sample from the industry population whose true unicorn rate is 1.9%, and count precision directly; if the resulting precision falls below 3.1% to 5.6%, the paper's headline comparison fails. Alternatively, derive precision from the confusion matrix at 9.9% prevalence and recompute it under 1.9% prevalence using the same score threshold; any answer that differs from 7.8% falsifies the linear-scaling assumption.","tokens_in":7418,"feed_emoji":"🦄","tokens_out":7159,"duration_ms":65981,"temperature":0.7,"pith_summary":"This paper tries to show that decision trees can handle complex, text-heavy data by letting a large language model propose the split questions, while keeping the tree's transparency. On a dataset of nearly 10,000 founders, GPTree reaches 37.3% precision at the dataset's 9.9% success rate, which the authors scale down by 5.5x to claim a 7.8% precision rate for identifying unicorn startups at inception. That scaled figure would exceed both GPT-4o with few-shot prompting and the best human venture investors. The framework needs only a task description, not feature engineering or prompt chains, and includes an expert-in-the-loop step that can rebuild or refine decision paths.","feed_headline":"GPTree beats humans and GPT-4o at picking unicorn startups","feed_subtitle":"An LLM-driven decision tree hit 7.8% precision on unicorn founders, above GPT-4o and top seed funds.","key_machinery":"The central object is GPTree, a decision tree whose internal nodes are natural-language questions answered by an LLM rather than threshold conditions on numeric features. The tree is grown greedily: a first LLM pass summarizes success patterns from batches of samples, a second pass generates zero to three candidate questions per feature, and the question yielding the lowest weighted Gini impurity is chosen at each node. An expert can later collapse nodes, rebuild subtrees, or advise question generation, and a sensitivity parameter converts leaf-node class proportions into binary labels by optimizing F0.5 on a validation set. This machinery supplies explainability, since every split is a readable question, while shifting pattern recognition from hand-crafted features to LLM judgment.","core_discovery":"The central claim is that LLM-generated questions can serve as decision-tree splits and yield better precision than both general-purpose LLM prompting and human experts in a high-stakes prediction task. Specifically, the paper reports that GPTree, using a GPT-4o-mini backend and greedy weighted-Gini split selection, reaches 37.3% precision and 27.1% recall on the founder-success test set, rising to 40.8% precision with expert refinement. For comparison, vanilla GPT-4o reaches 15.7% precision and few-shot GPT-4o reaches 16.2% precision on the same dataset. After the authors scale precision by the base-rate ratio between their dataset and the industry, they claim a 7.8% precision rate for identifying unicorn startups, surpassing the 3.1% to 5.6% range of human decision-makers.","pith_inferences":["The 5.5x linear scaling of precision is not a generally valid statistical adjustment: precision depends on the score threshold and the class-conditional distributions, so the true industry-prevalence precision could be higher or lower than 7.8%, and a fair comparison requires evaluating the same tree on a sample with 1.9% prevalence.","Because LLM answers are non-deterministic and the paper selects splits with a single Gini calculation, different runs or phrasings of the same question could produce different trees; the variance over repeated LLM calls is not quantified, so the reported margin over baselines may not be stable.","The paper's own limitations section acknowledges code-node unreliability and hallucination, meaning the claimed automation without feature engineering still depends on LLM correctness at every node.","A testable extension would be to report precision-recall curves or confusion matrices at multiple sensitivity thresholds, letting readers see whether the human-baseline superiority holds across operating points rather than only at the F0.5-optimal point."],"forward_implications":["If GPTree's precision claim survives proper base-rate adjustment, LLM-powered decision trees would give venture firms an explainable screening tool that outperforms both raw LLM prompting and human experts.","The framework removes typical prompt-engineering overhead: a user supplies only a task string, and question generation, splitting, and label calibration are automated.","Because each split is a human-readable question, the model produces audit trails for investment decisions, which matters in regulated or high-stakes settings.","The expert-in-the-loop mechanism lets domain knowledge correct or refine decision paths after training without retraining the whole tree, potentially combining human and machine judgment more tightly.","The same recipe could transfer to other text-rich domains such as medical triage or credit decisions, provided labeled data and base-rate-calibrated evaluation are used."],"supporting_citations":[{"why":"Introduces the few-shot prompting paradigm used as a baseline comparison for GPTree.","marker":"Brown et al. (2020)"},{"why":"Documents the GPT-4 model family that powers GPTree's question generation and inference nodes.","marker":"OpenAI (2024)"},{"why":"Demonstrates shortcut-taking in LLMs, motivating GPTree's separate insight-generation stage before question generation.","marker":"Tang et al. (2023)"}],"fun_headline_variants":["GPTree finds unicorn startups better than GPT-4o or humans","Explainable AI beats GPT-4o and humans at spotting unicorns","LLM-driven decision tree tops humans and GPT-4o on unicorns","GPTree: a transparent AI that picks unicorn startups better than humans","GPTree outperforms GPT-4o and humans at predicting unicorn founders"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that GPTree beats humans and GPT-4o rests on scaling measured precision linearly by the ratio of dataset success rate to industry success rate (9.9% versus 1.9%, a 5.5x adjustment), but precision does not generally scale linearly with class prevalence, so the 7.8% headline number is only as good as that assumption.","fun_headline_variants_meta":{"raw":{"variants":["GPTree finds unicorn startups better than GPT-4o or humans","Explainable AI beats GPT-4o and humans at spotting unicorns","LLM-driven decision tree tops humans and GPT-4o on unicorns","GPTree: a transparent AI that picks unicorn startups better than humans","GPTree outperforms GPT-4o and humans at predicting unicorn founders"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000332,"raw_usage":{"total_tokens":1818,"prompt_tokens":891,"completion_tokens":927,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":507,"completion_tokens_details":{"reasoning_tokens":829}},"tokens_in":507,"tokens_out":927,"duration_ms":7482,"temperature":1.0,"reasoning_tokens":829,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:47:35.068228+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Apply the exact decision rule learned by GPTree, with its F0.5-optimal sensitivity, to a random sample from the industry population whose true unicorn rate is 1.9%, and count precision directly; if the resulting precision falls below 3.1% to 5.6%, the paper's headline comparison fails. Alternatively, derive precision from the confusion matrix at 9.9% prevalence and recompute it under 1.9% prevalence using the same score threshold; any answer that differs from 7.8% falsifies the linear-scaling assumption.","supporting_citations":[],"review_version":1}