{"id":"f45cd86d-20ef-4056-b367-323422900ad8","arxiv_id":"2509.08140","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"An LLM-feature-driven ensemble predicts billion-dollar startup outcomes with 9.8X to 11.1X the precision of a random classifier, but the label and the model's intermediate target are both funding, so the result partly restates what the label already encodes.","lead":"This paper builds a startup-success predictor that uses a large language model to turn founder biographies into 63 features, then stacks XGBoost, Random Forest, and regression models to predict whether a company will eventually be worth over half a billion dollars. The authors report about ten times the precision of a random classifier, but the evaluation design makes the headline number fragile.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No temporal split or data blackout is specified; the sub-4% funding MAPE strongly suggests outcome information reached the LLM-engineered features, making the headline precision multiples potentially tautological.","rationale":"The reader's weakest assumption—the absence of a temporal cut or blackout on funding outcomes—is the single most load-bearing fragility. The paper itself does not address it in the limitations, and the reported funding MAPE of 3–4% is a red flag: predicting early-stage startup funding to within 4% from founder profiles alone would be a remarkable result, and the paper offers no external validation or feature inspection to support it. The coupling between the intermediate funding target and the success label (§3.3, §3.1) further amplifies the risk: even if the LLM features were clean, the meta-model's funding estimate is directly derived from the same features that predict success, so the precision multiple partly restates the label. However, this is not an internal inconsistency—it is a missing control. The ablation in §4.4.1 (drop from 10.4X to 4.6X without LLM features) shows that the LLM features carry signal, but it does not distinguish genuine predictive signal from leakage. A direct audit and a temporal-split retraining would settle the matter. Given the high stakes of the claim and the absence of such controls, the CONDITIONAL verdict remains appropriate; no verdict change is needed beyond what the reader already assigned.","tokens_in":5794,"tokens_out":1755,"duration_ms":24767,"concrete_test":"Request the authors to release or audit the exact prompts and input records used for LLM feature engineering, and grep for fields such as 'total funding', 'amount raised', 'valuation', 'IPO', 'acquired', and 'exit'. Additionally, retrain the pipeline on a strict temporal split: train on startups founded before 2015 (or using only profiles collected before 2015) and test on those founded after, regenerating all 63 features from only the pre-2015 public information. If the funding MAPE rises above ~20% or the precision multiple drops below ~2X, the original results were leakage-contaminated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—9.8X–11.1X precision versus random baseline—depends on LLM-derived features being constructed without access to the outcome they are later used to predict. Nowhere in §3.1–§3.2 or the limitations (§5.1) does the paper state a temporal cutoff, a variable blackout, or a leakage test for the 10,825 founder profiles. The 63 LLM-generated features include text embeddings of company descriptions and semantic encodings like 'skill relevance' and 'domain expertise' (§3.2). If any of the underlying profiles, or the LLM prompts, included the amount raised, valuation, IPO status, or acquisition news, then the features encode the label itself. The reported funding MAPE below 4% (§4.1) is implausibly low for early-stage startup funding prediction unless funding outcomes reached the feature set—either through the source data or through LLM pretraining knowledge that the prompts could invoke. The paper's own Table 6 makes the coupling visible: predicted funding classes map almost monotonically to success probabilities, with 100% success in the $1B+ class. This is exactly the pattern one would expect if the model is reciting observed outcomes rather than predicting them. The reader's 'conditional' verdict is therefore appropriate: the architecture and ablations are interesting, but the causal/leakage control is the missing load-bearing pillar. Without an explicit statement of when the profile data were collected and a demonstration that the LLM was not given funding/exit information, the 10X precision claim is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a pipeline for rare-event prediction in venture capital: LLM-powered feature engineering converts unstructured founder profiles into 63 features, and a multi-model ensemble (XGBoost, Random Forest, and a Linear Regression meta-model) first predicts total funding, which is then thresholded by logistic regression to predict binary startup success. Evaluated on 10,825 founders (8.5% success rate, held-out 2,166 split into three 722-company subsets), the paper reports funding MAPE below 4% and precision of 9.8x–11.1x over the random-classifier baseline with recall above 30%. Ablation studies suggest that LLM-derived features, embeddings, and the ensemble structure contribute to performance. The central claim is that LLM-powered feature engineering plus this multi-model architecture yields precision roughly an order of magnitude above the baseline in held-out subsets.","tokens_in":5981,"tokens_out":5110,"duration_ms":57682,"significance":"If the claims are valid, the paper would demonstrate a valuable use of LLM feature engineering for rare-event prediction, with practical relevance for early-stage investing and a useful emphasis on interpretability via feature sensitivity. Strengths include explicit ablations (LLM vs. traditional features, embedding sources, component removal, feature-type removal), reporting precision relative to the random baseline, and attempting to show robustness across three subsets. However, the central empirical claim is currently not supported because the evaluation protocol lacks a leakage control and the classification threshold is selected on the test data. The framework itself is interesting, but the paper needs substantial methodological revision before its headline precision figure can be accepted.","major_comments":[{"comment":"The paper never states a temporal cutoff or variable blackout for the 10,825 founder profiles. The success label in Section 3.1 is defined by raising, valuation, or acquisition above $500M, and the LLM-derived features in Section 3.2 are built from textual profiles that may include current funding, valuation, IPO, or acquisition information. If any of this outcome information reaches the features or embeddings, the reported <4% funding MAPE and 9.8x–11.1x precision are near-tautological. Please provide: (i) the collection date of the publicly available profiles; (ii) a demonstration that the 63 features and text embeddings were constructed without access to funding amounts, valuations, or exit outcomes; and (iii) an out-of-time validation split, e.g., train on companies founded before a cutoff and test on companies founded after, plus a leakage test such as training with shuffled labels","section":"Section 3.1/3.2/5.1"},{"comment":"The classification threshold of 0.8 is selected by inspecting the precision curve in Figure 2, which is computed on the evaluation/test subsets. This constitutes test-set tuning: the precision values in Table 5 are measured at a threshold chosen on the same data, and are therefore optimistic. The statement that thresholds were tested at fixed intervals to reduce overfitting does not address the fact that the test set is used for model selection. Use a validation split or cross-validation for threshold selection, and report test performance for a pre-specified threshold or across the full threshold range. Also report absolute precision and the number of predicted positives for each subset, rather than only multiples of the baseline rate.","section":"Section 4.2, Figure 2, Table 5"},{"comment":"A funding MAPE below 4% for early-stage startups is implausibly low and, in the absence of any leakage control, is itself evidence that outcome information may have reached the feature set. The paper provides no analysis of which features correlate with funding outcomes, nor a validation on companies outside the training period. Because the funding prediction is the intermediate target for success classification, this point is load-bearing. Please add an explicit leakage audit and an out-of-time funding-prediction experiment. If the low MAPE is genuine, explain the mechanism by which the features can predict funding so precisely.","section":"Section 4.1, Table 4"},{"comment":"Table 6 reports probabilities of success by predicted funding class, but these probabilities are generated by the logistic regression applied to the model's own continuous funding predictions. This table therefore reflects calibration of the model's output, not external validation of predictive power. The monotonic increase in success probability across funding classes is expected by construction. Please label the table accordingly and avoid presenting it as independent evidence of model quality.","section":"Section 3.3, Table 6"}],"minor_comments":[{"comment":"The sentence 'Companies that raised between $100K and $4M are categorized as unsuccessful' is ambiguous: does the unsuccessful class include all companies that did not meet the success threshold, or only those in the $100K–$4M range? Please clarify the labeling of companies that raised <$100K or between $4M and $500M.","section":"Section 3.1"},{"comment":"Typo: 'T extual Data' should be 'Textual Data'.","section":"Section 3.2.2"},{"comment":"Section 5.1 says 'all 63 features are LLM-derived', but Section 3.2.1 includes deterministic mappings such as education level. Clarify which features are LLM-derived and which are standard structured features.","section":"Section 3.2 / Section 5.1"},{"comment":"In Section 3.2.1, the citation is given as 'Xiong and Ihlamur (2024) [2]', but the reference list shows [2] as 'Xiong, S., & Ihlamur, Y. (2023)'. Please correct the year.","section":"References"},{"comment":"Figure 2 is mentioned but not shown in the text. Ensure the figure includes the precision curve with axis labels and threshold values so that the 'stable interval' claim can be checked.","section":"Figure 2"},{"comment":"Table 9 reports only deltas in precision and recall (e.g., XGBoost -3.2X, -7%). Please provide the absolute precision and recall for the full model and for each ablation so that the reader can assess the impact without ambiguity.","section":"Section 4.4.3, Table 9"},{"comment":"The train/test split is described only by counts (8,659 / 2,166). Please state whether the split is random, stratified by success label, or temporal. This is especially important given the leakage concerns.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The paper's novelty is plausible, but the current evaluation cannot support the headline claim until the leakage question is resolved. I recommend asking the authors for a full temporal split, a variable blackout, and a threshold-selection protocol that does not use the test set. If the authors cannot provide these, the reported precision multiples should be regarded as unvalidated. The paper is not beyond repair, but the revision is substantial."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead this one because the 10X precision headline is the kind of thing that gets press, but the paper doesn't support it yet. The architecture is a plausible new combination: LLM-engineered features from founder profiles, XGBoost + RF, a linear meta-model predicting total funding, and a logistic threshold to binary success. The strongest piece is Section 4.4.1: removing the LLM features drops precision from 10.4X to 4.6X, which suggests these features carry real signal, though the signal may be leakage rather than predictive insight.\n\nThe problems are load-bearing. First, the 0.8 threshold is chosen after inspecting the precision curve on the same test subsets that produce the headline numbers. That is test-set tuning, not an honest evaluation. Second, the success label is a threshold on funding, and the model's intermediate target is funding itself. Without a temporal cutoff or variable blackout, the features may already contain the funding outcomes. The sub-4% MAPE is the tell: early-stage startup funding is noisy; predicting it to within 3-4% is only plausible if the model saw the answer. Third, the numbers rest on about 20-25 predicted positives per 722-company subset, so the multiples have no error bars.\n\nNone of this is fatal to the idea. A proper temporal split, cross-validated threshold selection, confidence intervals, and a code/data release could rescue a real effect, likely smaller. The ablation study is worth keeping; it is the cleanest experiment in the paper.\n\nI would send this to peer review, because the leakage question is exactly a referee's job, and the authors seem capable of addressing it. But I would not cite the 10X claim in its current form.","headline":"A promising pipeline and a clean ablation, but the 10X precision claim is not established because the threshold is tuned on the test set and no leakage control is described.","tokens_in":6698,"tokens_out":2276,"would_cite":false,"duration_ms":26408,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM-powered feature engineering plus a stacked ensemble predicts which startups reach a $500M valuation, with 9.8–11.1x random-baseline precision and recall above 30%.","keywords":["rare-event prediction","LLM feature engineering","venture capital","startup success prediction","precision-recall","funding prediction","feature sensitivity","multi-model ensemble"],"falsifier":"A decisive check: re-run the pipeline on a temporally split dataset in which founder profiles and company descriptions are truncated before the funding and exit events that define the labels, so the LLM cannot see outcome information during feature construction. If held-out precision drops from ~10x toward the ~4.6x ablation level, the central claim fails. A simpler observational check is to search the generated features and text embeddings for strings that memorize exact funding round sizes or exit values, which would indicate direct label leakage.","tokens_in":5512,"feed_emoji":"📈","tokens_out":6435,"duration_ms":71360,"temperature":0.7,"pith_summary":"This paper claims that a pipeline combining LLM-generated features with an ensemble of XGBoost, Random Forest, and Linear Regression predicts which startups become worth more than $500 million with precision 9.8 to 11.1 times the random-classifier baseline, while keeping recall above 30 percent. The target is rare—only 8.5 percent of the 10,825 founders in the dataset are successful—so high precision is the practical goal. The paper matters because it offers a route from sparse, unstructured founder data to a decision-grade signal: the LLM turns resumes, startup descriptions, and domain fit into 63 structured features, and the ensemble converts those into a continuous funding estimate, then a binary success call. It also claims interpretability: feature sensitivity analysis ranks the startup's category list first at 15.6 percent of predictive influence, followed by founder count.","feed_headline":"LLM features lift startup-success precision to 10x baseline","feed_subtitle":"Investors get a 10x precision lift on a rare 8.5% success class, with interpretable feature rankings.","key_machinery":"The load-bearing mechanism is LLM-powered feature engineering: a large language model extracts and synthesizes signals from unstructured founder profiles into 63 trainable features, organized as categorical (education level, domain expertise), textual (startup-description embeddings), continuous (standardized with Z-scores), and boolean types. These feed a layered ensemble—XGBoost and Random Forest in the first layer, a Linear Regression meta-model in the second—that produces a continuous total-funding estimate. That funding estimate is the intermediate object that carries information from features to labels; logistic regression then thresholds the estimate into a binary success prediction.","core_discovery":"On its own terms, the paper establishes that a two-layer multi-model architecture can turn large language model features into rare-event predictions with precision an order of magnitude above chance. Success is defined by a valuation above $500 million, an acquisition above $500 million, or funding above $500 million; companies that raised only $100K–$4M are labeled unsuccessful. The first layer combines XGBoost and Random Forest with text embeddings, and the second layer is a Linear Regression meta-model that outputs a continuous estimate of total funding. That funding prediction is mapped to binary success by logistic regression with a threshold of 0.8, chosen at fixed intervals to avoid o","pith_inferences":["If a temporal blackout validation succeeds, the same LLM-feature-plus-ensemble recipe could transfer to other rare-event domains where unstructured text leads, such as equipment failure from maintenance notes or disease onset from clinical narratives.","The reported 3–4% funding MAPE is low enough to warrant a targeted test: masking all funding amounts and exit news during LLM feature construction and checking whether precision collapses would settle whether the result is foresight or leakage.","The feature-importance ranking suggests a data-collection rule the paper leaves implicit: investors gain more from accurate category tagging and founder-count data than from richer educational detail, so those fields deserve the most cleaning effort.","The success-class definition itself biases toward cohorts with public funding and exit records; applying the pipeline to geographies or stages with thinner public data would likely compress the precision multiple."],"forward_implications":["If the 10X precision holds, early-stage investors can cut a large startup funnel to a shortlist where roughly one in ten flagged companies reaches a $500M+ outcome, with coverage above 30%.","The ablation shows LLM features are load-bearing: removing them drops precision from 10.4X to 4.6X, so the claimed capability is specifically about semantic feature engineering, not ensemble stacking alone.","Because the threshold is fixed at 0.8 across subsets, the result implies stable precision across different base success rates (6.8%–9.3%) rather than a single lucky split.","The funding-class calibration—1.27% success probability for the $100K–$1M predicted class, 100% for the $1B+ class—offers a graded risk scale, not just a binary flag, if the funding model generalizes."],"supporting_citations":[{"why":"Supplies the LLM-powered feature engineering and automated labeling approach that the paper scales up to 63 features.","marker":"[4]"},{"why":"Provides the perspective of encoding founder–idea fit through LLM reasoning, adapted here into structured predictive features.","marker":"[2]"},{"why":"Baseline system with 9.4x precision that the multi-model architecture is designed to extend.","marker":"[3]"},{"why":"Earlier LLM-heuristic ensemble with 5.4x precision used as a comparison point.","marker":"[1]"},{"why":"Supplies the varying-threshold logistic-regression method used to tune the binary success classifier.","marker":"[7]"},{"why":"Justifies Z-score standardization of continuous features for gradient-based models.","marker":"[5]"},{"why":"Basis for selecting XGBoost and Random Forest as complementary first-layer models.","marker":"[6]"}],"fun_headline_variants":["LLM features boost startup success precision to 10x baseline","Rare startup exits predicted at 10x precision with LLM features","LLM-driven features yield 10x precision on high-value startup wins","From limited data, LLM features achieve 10x precision on rare startup wins","Multi-model LLM framework predicts rare startup outcomes at 10x precision"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The model's 10x precision and under-4% funding error depend on the LLM having built its 63 features without ever seeing the funding amounts or exit outcomes that later define the labels; the paper provides no temporal cutoff or variable blackout to guarantee this.","fun_headline_variants_meta":{"raw":{"variants":["LLM features boost startup success precision to 10x baseline","Rare startup exits predicted at 10x precision with LLM features","LLM-driven features yield 10x precision on high-value startup wins","From limited data, LLM features achieve 10x precision on rare startup wins","Multi-model LLM framework predicts rare startup outcomes at 10x precision"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001234,"raw_usage":{"total_tokens":4895,"prompt_tokens":724,"completion_tokens":4171,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":468,"completion_tokens_details":{"reasoning_tokens":4085}},"tokens_in":468,"tokens_out":4171,"duration_ms":32277,"temperature":1.0,"reasoning_tokens":4085,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T21:12:25.675706+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive check: re-run the pipeline on a temporally split dataset in which founder profiles and company descriptions are truncated before the funding and exit events that define the labels, so the LLM cannot see outcome information during feature construction. If held-out precision drops from ~10x toward the ~4.6x ablation level, the central claim fails. A simpler observational check is to search the generated features and text embeddings for strings that memorize exact funding round sizes or exit values, which would indicate direct label leakage.","supporting_citations":[{"cited_title":"Automating Venture Capital: Founder assessment using LLM-powered segmentation, feature engineering and automated labeling techniques","cited_arxiv_id":"2407.04885","evidence_quote":"Supplies the LLM-powered feature engineering and automated labeling approach that the paper scales up to 63 features."},{"cited_title":"Founder-GPT: Self-play to evaluate the Founder-Idea fit","cited_arxiv_id":"2312.12037","evidence_quote":"Provides the perspective of encoding founder–idea fit through LLM reasoning, adapted here into structured predictive features."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the varying-threshold logistic-regression method used to tune the binary success classifier."},{"cited_title":"ZNorm: Z-Score Gradient Normalization Accelerating Skip-Connected Network Training without Architectural Modification","cited_arxiv_id":"2408.01215","evidence_quote":"Justifies Z-score standardization of continuous features for gradient-based models."},{"cited_title":"(2021, April 26)","cited_arxiv_id":null,"evidence_quote":"Basis for selecting XGBoost and Random Forest as complementary first-layer models."}],"review_version":1}