{"id":"d14d953e-37c4-4dd7-bbf1-c03b5a9269b5","arxiv_id":"2505.24622","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"An LLM-generated set of yes/no founder questions, combined by simple threshold voting, is reported to predict startup success with 13.1% precision versus a 1.9% base rate, though the supporting evidence has significant gaps.","lead":"The paper introduces Random Rule Forest, an ensemble where an LLM generates simple yes/no questions and a unit-weight vote predicts startup success, reporting higher precision than direct LLM prompting. The abstract also claims a clinical-trial benchmark result, but the full text contains no such experiment, and no code or data are released.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's Phase I clinical-trial benchmark claim is entirely absent from the body; the headline result therefore has no supporting experiment.","rationale":"The paper's core proposal—an interpretable ensemble of LLM-generated YES/NO questions—is coherent, and the startup experiments are described in enough detail to be taken seriously as a preliminary result. However, the strongest claim identified in the review includes an established Phase I clinical-trial benchmark on which RRF outperforms published baselines on PR-AUC and ROC-AUC. That experiment is nowhere in the body or appendices. A headline claim that cannot be checked against the submitted evidence is the most load-bearing weakness because it directly determines whether the paper delivers its stated contribution. The reader's rationale also notes this absence, and we agree on the overall REJECT verdict. The reader's stated weakest assumption, however, focuses on LLM answer reliability; that is a real concern, but it is secondary to the missing clinical-trial result, and it is the kind of issue that could be probed with additional experiments even if the clinical-trial claim were removed. The proposed concrete test is a direct way to settle whether the central claim is merely unsupported in this draft or actually reproducible.","tokens_in":13181,"tokens_out":3746,"duration_ms":50182,"concrete_test":"Search the full manuscript text, including appendices, for the strings 'Phase I', 'clinical trial', 'PR-AUC', and 'ROC-AUC' (case-insensitive). If none appear outside the abstract, the clinical-trial claim is unsupported. To resolve the scientific question, obtain the Phase I benchmark and the authors' code, run the RRF pipeline end-to-end, and compute PR-AUC and ROC-AUC with confidence intervals across at least 10 seeds, comparing against the published baselines cited in the claim. If the experiment cannot be reproduced, the abstract must be revised to remove or substantially qualify the clinical-trial claim.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim, as stated in the abstract, has two parts: (1) RRF improves startup-screening precision, and (2) on an established Phase I clinical-trial benchmark, RRF outperforms published baselines on PR-AUC and ROC-AUC. The full manuscript contains no Phase I clinical-trial dataset, no clinical benchmark description, no PR-AUC/ROC-AUC results, and no comparison to published clinical-trial baselines. Section 5 reports only the startup experiments, and Appendices A-K contain no clinical content. This is not a minor omission: the threshold-independent clinical-trial result is part of the paper's headline contribution and is explicitly invoked to support the general claim that LLMs can serve as auditable feature generators for high-stakes text-based decisions. If the manuscript under review is the body text supplied here, the abstract is inconsistent with it; if a version with the clinical-trial experiment exists, it is missing from this submission. Either way, the strongest claim is unsupported. The startup-only results are plausible but do not establish the clinical-trial claim, and they carry their own unresolved issues (e.g., LLM answer reliability and baseline tuning fairness).","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Random Rule Forest (RRF), an ensemble method in which an LLM generates candidate yes/no questions about unstructured founder profiles; questions are filtered for semantic and predictive redundancy, ranked, and combined by unit-weight threshold voting. The model is evaluated on a proprietary dataset of 9,892 U.S. founders (1.9% success rate), reporting held-out precision of 13.1% (6.9x base rate), F0.5 of about 0.124 vs 0.088 for the best direct-prompting baseline, and an expert-in-the-loop variant at 15.3% precision. The supplied abstract additionally claims RRF outperforms published baselines on an established Phase I clinical-trial benchmark on PR-AUC and ROC-AUC, but no such experiment appears in the body. The paper also includes ablations over similarity metrics, thresholds, ranking criteria, and a temporal split.","tokens_in":13416,"tokens_out":9261,"duration_ms":107069,"significance":"The startup-screening result, if it survives additional validation, is a useful contribution: RRF is genuinely interpretable, requires only two tunable hyperparameters, and the pipeline separates question generation/filtering from evaluation, which is a sound design choice in a low-base-rate setting. The ablations (Section 5.5, Appendices H and K) and the explicit cost table are welcome transparency. However, the contribution is currently overstated: the clinical-trial result named in the abstract is absent, and the core mechanism (LLM yes/no answers) is not tested for reliability. The paper's value therefore depends on additional experiments that are within scope.","major_comments":[{"comment":"The abstract supplied for this submission states that RRF 'outperforms published baselines on the threshold-independent metrics PR-AUC and ROC-AUC' on an established Phase I clinical-trial benchmark. The body text (Sections 1–7) and Appendices A–K contain no Phase I clinical-trial dataset, no PR-AUC or ROC-AUC results, and no comparison to published clinical-trial baselines. This is not a minor omission: the clinical-trial benchmark is presented as the second of two low-base-rate domains and supports the general claim in Section 6 that LLMs can serve as auditable feature generators for high-stakes decisions. The authors must either add the experiment or remove the claim from the abstract and discussion; the full-text abstract, which mentions only startup success, must be reconciled with the submitted abstract.","section":"Abstract"},{"comment":"The test-time prediction in §5.3, and the question scoring in §4.3, rely entirely on LLM yes/no answers to the generated questions, but no experiment assesses the reliability or stability of these answers. There is no inter-run consistency check (e.g., repeated sampling with different temperatures/seeds), no sensitivity to prompt wording or model version, and no human agreement study on a subsample. Because the ensemble's signal is mediated by this answering step, noisy or systematically biased answers could produce the reported precision gains even if the questions are informative. Please add such sensitivity analyses.","section":"§4.3, §5.3"},{"comment":"Table 1 reports each model's precision, recall, and F0.5 as single point estimates, while §5.5 reports that ablation results are averaged over 100 random seeds. The headline comparison (RRF F0.5 = 0.124 vs best baseline 0.088) is therefore presented without any uncertainty quantification. Please report confidence intervals or standard deviations across seeds/repeats, and, if possible, a statistical test of the difference, for the main results in Table 1 as well as the expert-in-the-loop comparison.","section":"Table 1"},{"comment":"The cost comparison in Appendix J (Table 3) shows that RRF's sequential evaluation takes 67–72 hours versus 0.9–4.1 hours for the prompting baselines, roughly a 20–70x gap. The paper acknowledges this in passing but still frames the comparison as a practical improvement. To support the claim that RRF is a practical method, please add a cost-controlled comparison (e.g., RRF restricted to a similar inference budget, or cost-normalized performance) or explicitly discuss the trade-off in the main text.","section":"Appendix J / §5.3"}],"minor_comments":[{"comment":"The supplied abstract mentions the Phase I clinical-trial benchmark and 'and Manageable' in the title, while the full-text title and abstract are startup-only; please unify the versions.","section":"Title/Abstract"},{"comment":"F0.5 values are inconsistent: the submitted abstract says 0.121, Table 1 reports 0.124, and Appendix H reports 0.1233; please reconcile the reported numbers.","section":"Abstract, Table 1, Appendix H"},{"comment":"Section 7 states that expert-in-the-loop refinement achieves 'an 9× improvement,' but 15.3% precision against a 1.9% base rate is 8.05×, and the full-text abstract says 8×; correct the arithmetic and the typo.","section":"Section 7"},{"comment":"Section 3.1 says the dataset 'covers nearly a third of this population'; 9,892/35,000 is 28%, so please either provide exact denominators or soften the claim.","section":"Section 3.1"},{"comment":"Appendix K reports F0.5 = 0.267 on the forward split with only 35 positive test cases; the text acknowledges instability, but the claim that the model 'generalises well across time' should be tempered or accompanied by bootstrap confidence intervals.","section":"Appendix K"},{"comment":"The filtering thresholds (semantic similarity 0.9, precision filter 10%, Hamming 0.15) are chosen without sensitivity analysis except for the Hamming threshold; please add a brief discussion or ablation for the fixed thresholds.","section":"Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"The gap between the submitted abstract and the body is the main integrity concern; please check which version is actually under consideration. The paper also cites several closely related works by the same group (Xiong et al., Mu et al., Preuveneers et al.); the discussion of incremental novelty over GPTree could be sharpened but this does not affect my recommendation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The first thing to know: the abstract claims RRF beats published baselines on an established Phase I clinical-trial benchmark with PR-AUC and ROC-AUC, but the body contains no clinical-trial experiment, no dataset, no metrics, and no baselines. This is not a small omission; it is the paper's headline contribution. As written, a reader can only conclude that the claim is either unsupported or refers to a version that was not submitted. That alone undermines the paper.\n\nWhat the paper does well is the core RRF idea. Using an LLM to generate a large pool of yes/no questions, then filtering for semantic and predictive redundancy and combining via threshold voting, is simple, interpretable, and genuinely distinct from GPTree, Tree Prompting, and RAISE. The expert-in-the-loop variant is a sensible extension. The dataset is split into generation, filtering, and evaluation sets, which shows more care than many similar papers, and the ablation studies on similarity metrics and ranking criteria are a fair attempt to probe robustness.\n\nThe soft spots go beyond the missing clinical experiment. The main performance table has no error bars—we see point estimates for precision, recall, and F0.5 but no variance across the 10x10 CV folds that are mentioned later. The compute comparison is lopsided: RRF takes 67 sequential hours and about $21, while the cheapest baseline runs in 1.3 hours for $2.90. That is roughly a 50x cost gap, and it is not discussed as a trade-off. The dataset is selectively labeled, excluding companies that raised between $4M and $500M, which inflates the separation between success and failure and makes the 6.9x precision gain look larger than it would be in a realistic screening pool. No code or data are released, so none of this is reproducible. Finally, the method leans entirely on the LLM's yes/no answers being stable and meaningful, but the paper reports no test of inter-run consistency, prompt sensitivity, or human agreement on those answers. If the answering step is noisy in a way that correlates with the outcome, the whole ensemble signal could be an artifact.\n\nThis is a clever method in search of better evidence. The startup results are plausible but not convincing as presented, and the clinical-trial claim is what would have made it a strong paper. I would send it to review, because the RRF idea deserves a serious look and the flaws are fixable, but I would expect the referee report to demand removal of the unsupported claim, proper uncertainty quantification, matched-compute baselines, and at least a reproducibility statement.","headline":"The abstract promises a clinical-trial result that does not exist in the body; the startup results are plausible but under-supported, so the paper as submitted falls short of its claims.","tokens_in":13958,"tokens_out":1800,"would_cite":false,"duration_ms":23678,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM-generated yes/no questions, voted together, beat direct prompting at startup screening.","keywords":["random rule forest","LLM-generated questions","ensemble learning","interpretable AI","startup success prediction","weak learners","low-base-rate classification"],"falsifier":"On a fixed set of founders and a fixed question pool, vary the model version, temperature, and prompt phrasing used for the answering step and measure per-question agreement; if the ensemble's top-N ranking or the reported F0.5 changes by more than the gap between RRF and the best baseline, the LLM-answer assumption is falsified.","tokens_in":13017,"feed_emoji":"🎯","tokens_out":7812,"duration_ms":79796,"temperature":0.7,"pith_summary":"Random Rule Forest (RRF) claims that a large language model can be used as a generator of simple YES/NO questions rather than as an end-to-end predictor, and that combining those questions with a plain unit-weight voting rule produces an interpretable predictor that beats direct prompting in low-base-rate tasks. The paper demonstrates this on early-stage startup screening: on 9,892 founder profiles, RRF reaches 13.1% precision versus a 1.9% base rate, a 6.9x improvement, and outperforms zero- and few-shot LLM baselines on F0.5. Adding a handful of expert-written questions raises precision to 15.3%. If right, this matters because high-stakes screening decisions, where errors are costly and outcomes must be auditable, can get transparent, editable scorecards without giving up predictive performance.","feed_headline":"Yes/no question votes lift startup screening precision 6.9x","feed_subtitle":"On 9,892 founder profiles, an auditable scorecard of LLM-generated questions outperforms zero- and few-shot prompts.","key_machinery":"Random Rule Forest (RRF) is an ensemble whose weak learners are natural-language YES/NO questions generated by an LLM rather than trained classifiers. The pipeline has three stages: (1) an LLM generates 250 candidate questions from balanced founder profiles; (2) a filtering stage uses sentence-embedding cosine similarity to remove semantically identical questions, discards questions with precision below chance (10%), removes predictively redundant questions via a Hamming-distance threshold, and ranks survivors by F0.5; (3) the top N questions vote, predicting success when at least T answer YES. The plain unit-weight vote is the load-bearing design choice: it keeps the model fully auditable, with all decisions traceable to plain-language green flags.","core_discovery":"The central discovery is that an LLM's ability to generate plain-language YES/NO questions can be turned into a strong, transparent predictor by treating each question as a weak learner and combining them with threshold voting. On the founder dataset, RRF filters hundreds of generated questions down to a ranked pool, keeps the top N questions, and predicts success when at least T of them answer YES, yielding a precision of 13.1% against a 1.9% base rate (6.9x) and an F0.5 of 0.124 that beats all zero- and few-shot prompting baselines tested across three LLM architectures. Replacing several LLM questions with expert-crafted ones raises precision to 15.3%. The paper also reports that on an established Phase I clinical-trial benchmark, RRF beats published baselines on PR-AUC and ROC-AUC, evidence that the mechanism transfers to another low-base-rate text-screening domain.","pith_inferences":["A direct test implied by the paper is to measure answer stability across LLM versions and prompt wordings; high variance would require voting-based calibration or answer aggregation.","The approach should transfer to other low-base-rate text screening tasks, such as grant review, hiring, or underwriting, where explanations are obligatory, though geographies and label definitions would need local data.","The forward-split result (F0.5 = 0.267 versus 0.124 in nested cross-validation) hints at temporal generalization, but the authors flag the small positive cohort; a larger temporal holdout would settle whether the gain is real.","Unit-weight voting may be more robust than learned weights when positives are scarce, but that claim needs an explicit comparison against a logistic-regression meta-learner on the same question outputs."],"forward_implications":["The voting threshold and ensemble size give operators direct control over the precision-recall trade-off, so the same question pool can be tuned for precision, F1, F2, or F0.5 without retraining.","Experts can audit, edit, or add individual rules, and a small set of expert-crafted questions raises precision from 13.1% to 15.3%, showing human-LLM collaboration works inside the ensemble.","Decision traces reduce to a short list of YES/NO answers, so every prediction can be justified in natural language to stakeholders.","The pipeline transfers to a second low-base-rate domain (Phase I clinical-trial screening), where the paper reports superior PR-AUC and ROC-AUC over published baselines.","The method recovers classical weak-learner theory with conceptual diversity replacing algorithmic diversity, because each question encodes a different hypothesis about success."],"supporting_citations":[{"why":"Supplies the theoretical claim that slightly-better-than-chance weak learners can be combined into a strong predictor.","marker":"Schapire, 1990"},{"why":"Provides the bagging ensemble baseline RRF contrasts with, motivating independence across base learners.","marker":"Breiman, 1996"},{"why":"Establishes ensemble diversity as the driver of accuracy gains.","marker":"Dietterich, 2000"},{"why":"GPTree is the main prior interpretable LLM-tree baseline that RRF compares against for precision.","marker":"Xiong et al., 2024"},{"why":"Tree Prompting is a competing transparent LLM pipeline that RRF benchmarks against.","marker":"Morris et al., 2023"},{"why":"SiliconCrowd's LLM-ensemble predictions serve as the closest fusion baseline RRF aims to simplify.","marker":"Schoenegger et al., 2024"},{"why":"Supplies the Y Combinator and tier-1 VC success rates used as industry benchmarks.","marker":"Mu et al., 2025"},{"why":"Supports the human-LLM collaboration benefit that expert-in-the-loop refinement relies on.","marker":"Yan et al., 2024"}],"fun_headline_variants":["LLM yes/no votes make auditable startup scorecard","Question votes beat prompts on startup screening","Interpretable LLM question ensemble lifts precision 6.9x","Voting on yes/no questions yields auditable predictions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that the LLM's YES/NO answers to each generated question are reliable and stable enough that a count of YES votes carries genuine predictive signal; if answer noise or prompt sensitivity dominates, the reported precision gains could be artifacts.","fun_headline_variants_meta":{"raw":{"variants":["LLM yes/no votes make auditable startup scorecard","Question votes beat prompts on startup screening","Interpretable LLM question ensemble lifts precision 6.9x","Voting on yes/no questions yields auditable predictions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000489,"raw_usage":{"total_tokens":2422,"prompt_tokens":972,"completion_tokens":1450,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":1386}},"tokens_in":588,"tokens_out":1450,"duration_ms":12037,"temperature":1.0,"reasoning_tokens":1386,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:17:24.976790+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a fixed set of founders and a fixed question pool, vary the model version, temperature, and prompt phrasing used for the answering step and measure per-question agreement; if the ensemble's top-N ranking or the reported F0.5 changes by more than the gap between RRF and the best baseline, the LLM-answer assumption is falsified.","supporting_citations":[],"review_version":1}