{"id":"8af270df-35f8-4988-a787-6d35e8137f0f","arxiv_id":"2504.18600","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"QuantBench introduces a unified, industry-aligned benchmark platform for evaluating AI methods across the full quantitative investment pipeline, with data, models, and empirical comparisons.","lead":"QuantBench is a proposed benchmark platform for AI-driven quantitative investing, combining market, fundamental, relational, and news data across four regions with many machine-learning models and evaluation metrics. It aims to standardize how quant AI methods are tested, matching industry practices and covering the full research-to-trading pipeline.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The backtest universe in Section 6 is not point-in-time and ignores transaction costs, so reported returns and model rankings are not yet trustworthy; the benchmark's standardization claim depends on fixing this.","rationale":"The reader and I identify the same weakest assumption: no point-in-time universe and no transaction costs. I choose this as the single load-bearing concern because the paper's empirical section is the evidence for the claimed research directions (continual learning, relational modeling, overfitting mitigation), and those directions are derived from backtest rankings such as Tables 2-5. A benchmark can have a good architecture and still fail its central purpose if the evaluation numbers are not trustworthy. I do not see an internal inconsistency in the platform design; the layered pipeline, task coverage, and breadth of models are genuine contributions, and the paper is transparent that code will be open-sourced. But the empirical protocol, as described, would not meet the standard of an industrial backtest: index membership must be as-of each date, delisted stocks must remain in the test set, and costs must be modeled. The paper's Appendix A even lists 'more realistic and efficient backtesting methods' as future work, which is an explicit admission that the current evaluation is not yet industry-grade. I therefore keep the reader's CONDITIONAL verdict rather than moving to ACCEPT or REJECT; the concern is serious but fixable. A single re-run of the Section 6.1 protocol with point-in-time membership and costs would settle whether the headline numbers survive.","tokens_in":17917,"tokens_out":4594,"duration_ms":46553,"concrete_test":"Reconstruct the exact backtest in Section 6.1 using point-in-time CSI constituent lists and delisted stocks (e.g., from CSMAR or Wind), and rerun the XGBoost-versus-LSTM comparison on Alpha101/Alpha158 with a realistic round-trip cost of roughly 20 bps plus Chinese stamp tax. If Table 2's 24.58% return or the XGBoost/LSTM ranking changes materially, or if Sharpe ratios drop substantially, the Section 6 empirical claims are contaminated by survivorship and omitted costs.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing condition for the paper's central claim—that QuantBench is an industrial-grade, standardized benchmark whose empirical studies reveal research directions—is that the Section 6 backtests are valid comparisons. The weakest point is the backtest protocol: Section 3.1/Table 6 define universes using index constituent lists (CSI 300, S&P 500, etc.) without any statement that membership is point-in-time, and Section 6.1 describes a top-300 ranking strategy with no mention of transaction costs, turnover constraints, or delisted-stock handling. If the universe is the current constituent list applied retrospectively, stocks not yet listed or since delisted are included in historical cross-sections, which inflates long-only returns and biases model rankings. The reported 24.58% annualized return (Table 2) and Sharpe ratios near 3.4 (Table 3) assume frictionless execution; for a strategy that rebalances a top-300 portfolio, ignoring costs can change both absolute performance and relative model ordering. The paper itself concedes in Appendix A that 'implementing more realistic and efficient backtesting methods' is future work, which undercuts the 'industrial-grade' and 'standardization' claims for the empirical component. The platform architecture may still be valuable, but these empirical findings cannot be assessed until the universe and cost assumptions are specified and corrected.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces QuantBench, a proposed benchmark platform for AI methods in quantitative investment. The platform is designed to cover the full quant pipeline—data preparation, factor mining, alpha modeling, portfolio optimization, and order execution—and to provide standardized datasets, model implementations, and evaluation metrics. The authors report empirical studies using the platform: comparisons of tree models versus deep networks (Section 6.1), a broad model comparison (Section 6.2), training-objective comparisons (Section 6.3), rolling-window retraining for alpha decay (Section 6.4), validation set selection for hyperparameter tuning (Section 6.5), and ensembling to mitigate overfitting (Section 6.6). From these experiments, the paper draws research directions including continual learning, relational modeling, and overfitting mitigation. The central claim is that QuantBench is an industrial-grade, standardized benchmark whose empirical studies reveal critical research directions for AI in quantitative investment.","tokens_in":18158,"tokens_out":4139,"duration_ms":45730,"significance":"If the platform and its empirical results are sound, QuantBench would be a useful community resource: it covers a broad range of models and data types, includes temporal relational data with explicit leakage-aware snapshots, and proposes task-agnostic metrics such as alpha decay and robustness. The breadth of model coverage and the attempt to unify the pipeline are genuinely valuable. However, the paper's central empirical claims currently rest on backtests whose data universe, cost assumptions, and release status are not adequately specified. The reported findings (for example, the advantage of adaptive graph models over homogeneous GNNs, or the superiority of frequent rolling retraining) cannot yet be considered reliable evidence for the stated research directions.","major_comments":[{"comment":"The backtest universe is not demonstrated to be point-in-time or free of survivorship bias. Table 6 lists universes by current index names (CSI 300, S&P 500, etc.) with no statement about historical constituent membership, delisted stocks, or point-in-time availability. Section 6.1 selects the top 300 stocks at each cross-section without specifying whether the candidate set is the historical index membership or today's constituents applied retrospectively. If current constituent lists are used for past dates, the results in Tables 2 and 3 will be inflated and model rankings biased. The authors must describe how membership is constructed over time, whether delisted stocks are included, and how the data provider handles corporate actions and old tickers.","section":"Section 3.1 / Section 6.1 / Table 6"},{"comment":"Transaction costs, slippage, and market impact are not modeled, yet the paper reports gross annualized returns such as 24.58% (Table 2) and Sharpe ratios near 3.4 (Table 3). The strategies rebalance a top-300 portfolio at each cross-section, which implies substantial turnover; ignoring costs can change both absolute performance and the relative ordering of models. The paper itself concedes in Appendix A that 'implementing more realistic and efficient backtesting methods' is future work. This directly conflicts with the 'industrial-grade' and 'standardization' claims for the empirical component. The backtests should either include realistic cost and turnover modeling or be explicitly labeled as cost-free and interpreted with that caveat.","section":"Section 6.1 / Section 6.2 / Appendix A"},{"comment":"The data universe used in the main model comparison is inconsistent with the supplementary figure. The text of Section 6.2 says the experiment used US stock data, while Appendix C.1 labels Figure 6 as 'Comparison of diffrent models on CSI300 dataset.' This discrepancy makes it unclear which tables correspond to which market and undermines the reproducibility of the claimed findings. The authors should state the exact market, date range, and universe for each table in Section 6 and reconcile the appendix labels.","section":"Section 6.2 / Appendix C.1"},{"comment":"The paper repeatedly refers to 'supplementary materials' for full descriptions of models, data levels, and implementation details, but no supplementary material or repository link is included with the manuscript. For a benchmark paper, the ability to verify implementations and data pipelines is load-bearing; the statement that the code 'will be open-sourced' is not sufficient. The authors should provide a working repository link, an explicit data schema, and versioned evaluation code as part of the submission.","section":"Sections 3.2/4, Appendix A, and code availability"},{"comment":"Several conclusions in Section 6.2 are drawn from comparisons with very high variance. For example, Hawkes-GRU has return 0.35% ± 10.81% and Sharpe -0.0195 ± 0.7246, while several Transformer and GNN models have standard errors that overlap with zero on some metrics. The claim that 'adaptive graph models outperformed others' rests on point estimates (THGNN IC 4.93 vs. GAT 3.90) without reporting the number of repeated runs, seeds, or significance tests. The paper should report the evaluation protocol (number of seeds, train/validation/test split, fixed random seeds) and avoid strong research-direction conclusions from differences that are within noise.","section":"Section 6.2 / Table 3"}],"minor_comments":[{"comment":"The text contains a typo: 'charaterstic-sorted portfolios' should be 'characteristic-sorted portfolios'.","section":"Section 2"},{"comment":"The meaning of the 'Diff' row is not defined. Clarify whether it is the relative difference (XGBoost minus LSTM divided by LSTM) and specify this in the caption.","section":"Table 2"},{"comment":"The model name 'STHCN' in the text is listed as 'STHGCN' in the table; please unify the naming.","section":"Section 4.1 / Table 3"},{"comment":"In Table 9, the 'VPFNW' row for XGBoost is reported as '-', but no explanation is given for the missing value. State why this condition is absent.","section":"Appendix C.2 / Table 9"},{"comment":"Figure 6(b) shows a correlation matrix but the caption does not define the color scale or the metric being correlated (e.g., predictions vs. returns). A brief definition would make the figure interpretable.","section":"Appendix C.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is appealing, but in its current form the empirical component resembles an extended position paper rather than a verifiable benchmark. The lack of point-in-time universe specification, cost modeling, and code/data release are the main blockers. I would be willing to look at a revised version that addresses these points; if the empirical section is replaced with properly documented backtests or reframed as a platform description without strong empirical claims, it might become acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"QuantBench is a genuine engineering effort: a multi-market, multi-frequency benchmark platform with a rich model zoo, feature sets, and an evaluation pipeline, including Wikidata relational snapshots with temporal tracking. That integration is new and, if open-sourced, could be a useful common testbed for AI4Finance. The empirical studies, however, rest on backtests that are not yet realistic enough to support the 'industrial-grade' claim or the specific model rankings.\n\nWhat the paper does well is the platform itself. It covers the full pipeline from data to execution, spans four markets and multiple universes, and provides both task-specific and task-agnostic metrics. The inclusion of Wikidata with temporal snapshots is a thoughtful touch, and the authors are honest enough to list limitations in Appendix A. The empirical observations—trees vs. deep networks, alpha decay with rolling retraining, ensemble benefits, validation-set choice—are interesting research directions, but they are conditional on the backtest protocol being sound.\n\nThat is where the soft spots are. Section 6 defines universes via current index constituent lists (Table 6) with no statement that membership is point-in-time or that delisted stocks are handled. The top-300 selection strategy is backtested with no transaction costs, turnover, or slippage. The reported 24.58% annualized return and Sharpe ratios above 3 are frictionless numbers; in a strategy that rebalances a top-300 portfolio, costs can change both absolute performance and relative ordering of models. Appendix A concedes that 'implementing more realistic and efficient backtesting methods' is future work, which directly undercuts the paper's standardization claim. Also, no code or data are currently released, so the reproducibility that a benchmark needs is promised, not delivered. Some comparisons, like Table 2, show LSTM with higher IC but XGBoost with higher return; without costs and a clean universe, we cannot know if that ranking holds.\n\nThese are fixable problems, not fatal ones. The platform architecture itself is valuable, and the empirical studies can be re-run once the protocol is corrected and the assets are released. I would send this to peer review, not desk reject, because it is a substantial benchmark paper with real engineering and the core idea matters to a growing subfield. But I would ask for major revision: release code and data, add point-in-time universes, include delisting and transaction costs, and specify the entire data pipeline. After those fixes, this could be the standard reference.","headline":"Solid benchmark engineering, but the headline empirical claims rest on backtests that ignore costs and point-in-time universes, so treat the results as provisional.","tokens_in":18701,"tokens_out":1900,"would_cite":false,"duration_ms":19363,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"QuantBench is an industrial-grade benchmark platform that standardizes data, models, and evaluation across the whole quantitative investment pipeline, and its comparisons point to continual learning, better relational modeling, and…","keywords":["quantitative investment","benchmark platform","AI for finance","stock prediction","backtesting","machine learning","alpha decay","portfolio optimization"],"falsifier":"Re-run the paper's empirical comparison on a point-in-time universe that includes delisted stocks and apply realistic transaction costs and turnover constraints to the top-300 backtests; if the reported returns shrink to near zero or the model ordering flips, the platform's industrial-grade claim would be refuted.","tokens_in":17742,"feed_emoji":"📈","tokens_out":7041,"duration_ms":64614,"temperature":0.7,"pith_summary":"The paper argues that AI for quantitative investing has advanced without a shared way to test and compare methods, and that this gap blocks academic results from reaching trading practice. It presents QuantBench, a benchmark platform that standardizes data, model implementations, and evaluation metrics across the full pipeline from raw data and factor mining through prediction, portfolio construction, and order execution. If the platform works as claimed, researchers and practitioners can compare stock-prediction models on equal terms, much as standardized benchmarks accelerated other fields. The paper's own comparisons on the platform then identify four concrete research gaps: models degrade quickly as markets shift, graph-based relational models do not consistently help, deep networks fit well but often lose to tree models on returns, and ensembling mitigates low-signal overfitting.","feed_headline":"One test track now spans the full AI quant pipeline","feed_subtitle":"Researchers can compare stock-prediction AI models on equal data, tasks, and backtests in one platform.","key_machinery":"The central object is the layered quant research pipeline: Data Preparation drives Factor Mining, which feeds Alpha Modeling, which flows into Portfolio Optimization and then Order Execution, with an upper layer defining learning objectives and an evaluation layer applying metrics. This pipeline is the mechanism that lets otherwise incompatible models be compared: every model receives the same standardized data format, is trained against the same objective, and is evaluated on the same backtest. Supporting machinery includes the unified dataset covering market, fundamental, relational, and news data; the model zoo spanning tree models, RNNs, transformers, graph networks, and hypergraph models; and metrics that separate task-specific signal, portfolio, and execution quality from robustness, correlation, and alpha decay.","core_discovery":"QuantBench claims to provide, in a single open platform, the layered structure of an industrial quant workflow: data preparation, factor mining, alpha modeling, portfolio optimization, and order execution, together with datasets spanning markets, frequencies, and information types, a model zoo of temporal, spatiotemporal, and graph-based learners, and task-specific plus task-agnostic evaluation metrics. Its empirical study, run on the platform, reports that frequent rolling model updates beat stale models; that adaptive graph models outperform fixed relational graphs; that deep models produce higher information coefficients but not consistently higher returns than XGBoost; and that averaging many seeded runs improves robustness. The paper frames these results as evidence that the benchmark can surface research directions worth pursuing, not as a final ranking of methods.","pith_inferences":["Inference: If transaction costs and turnover were added to the top-300 backtests, the gap between tree models and deep models could widen or narrow; cost-aware evaluation is a testable extension the paper does not carry out.","Inference: The finding that fixed relational graphs rarely help while adaptive graphs do suggests a concrete next experiment: vary the graph construction (industry taxonomy, knowledge-base edges, learned similarity) while holding the predictor fixed, to isolate where relational information actually enters.","Inference: The combination of low correlation across models and high variance within one model implies that reporting single-seed results in quant papers is probably misleading; a minimum of multiple seeds and an ensemble baseline could become a reporting standard.","Inference: Alpha decay results imply benchmark leaderboards need time-stamped validity; an interesting design would be to track how long each model's edge survives after publication."],"forward_implications":["Researchers get a common ground where a new model can be compared against a fixed set of baselines and datasets, so gains can be attributed to the method rather than to data preprocessing or evaluation choices.","Frequent retraining (three-month rolling) noticeably beats no rolling, which argues that continual and online learning methods are a priority for quant AI.","Fixed relational graphs from industry classifications or knowledge bases do not reliably help predictions, while adaptive graph models that learn relations from data do, pointing to latent-relation modeling as the promising direction.","Deep networks' higher information coefficient does not translate into better returns or Sharpe ratios than tree models on some feature sets, so training objectives and evaluation targets need to be aligned.","Averaging predictions across repeated runs of the same model reduces variance and improves backtest results, indicating that ensemble methods are a partial answer to low signal-to-noise overfitting."],"supporting_citations":[{"why":"Supplies the computer-vision benchmark precedent that motivates why standardization can accelerate a field.","marker":"[Deng et al., 2009]"},{"why":"Supplies the NLP benchmark precedent used to argue that shared evaluation platforms speed research progress.","marker":"[Wang et al., 2019a]"},{"why":"Existing AI-oriented quantitative investment platform that QuantBench extends and distinguishes itself from.","marker":"[Yang et al., 2020]"},{"why":"Provides XGBoost, the tree-model baseline in the empirical comparisons.","marker":"[Chen and Guestrin, 2016]"},{"why":"Provides LSTM, the recurrent deep-network baseline in the empirical comparisons.","marker":"[Hochreiter and Schmidhuber, 1997]"},{"why":"Provides the Alpha101 feature set used in the tree-versus-DNN experiment.","marker":"[Kakushadze, 2016]"},{"why":"Provides the Relational Stock Ranking model included in the model suite and the comparison of relational approaches.","marker":"[Feng et al., 2019]"},{"why":"Motivates the experiment on validation-set selection for hyperparameter tuning.","marker":"[de Prado, 2018]"},{"why":"Supports the observation that tree models can outperform deep learning on tabular data, which the paper cites to explain its empirical results.","marker":"[Grinsztajn et al., 2022]"},{"why":"Provides the spatiotemporal hypergraph attention model used as a representative hypergraph method in the model zoo.","marker":"[Sawhney et al., 2021b]"}],"fun_headline_variants":["QuantBench: Full-pipeline benchmark for AI quant","One platform to test AI across every quant stage","Industry-aligned benchmark for AI investing","QuantBench: From data mining to order execution"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole comparison rests on the assumption that the backtest data has no survivorship or lookahead bias and that the top-300 stock-selection results are meaningful without modeling transaction costs or turnover, so if stocks that later delisted or statements not available at trade time are absent, the reported returns and model rankings would not be trustworthy.","fun_headline_variants_meta":{"raw":{"variants":["QuantBench: Full-pipeline benchmark for AI quant","One platform to test AI across every quant stage","Industry-aligned benchmark for AI investing","QuantBench: From data mining to order execution"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000449,"raw_usage":{"total_tokens":2218,"prompt_tokens":851,"completion_tokens":1367,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":467,"completion_tokens_details":{"reasoning_tokens":1309}},"tokens_in":467,"tokens_out":1367,"duration_ms":13543,"temperature":1.0,"reasoning_tokens":1309,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:29:28.197771+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the paper's empirical comparison on a point-in-time universe that includes delisted stocks and apply realistic transaction costs and turnover constraints to the top-300 backtests; if the reported returns shrink to near zero or the model ordering flips, the platform's industrial-grade claim would be refuted.","supporting_citations":[{"cited_title":"Long short-term memory","cited_arxiv_id":null,"evidence_quote":"Provides LSTM, the recurrent deep-network baseline in the empirical comparisons."},{"cited_title":"Temporal Relational Ranking for Stock Prediction","cited_arxiv_id":"1809.09441","evidence_quote":"Provides the Relational Stock Ranking model included in the model suite and the comparison of relational approaches."},{"cited_title":"Advances in Financial Machine Learning","cited_arxiv_id":null,"evidence_quote":"Motivates the experiment on validation-set selection for hyperparameter tuning."},{"cited_title":"Why do tree-based models still outperform deep learning on typical tabular data? June 2022","cited_arxiv_id":null,"evidence_quote":"Supports the observation that tree models can outperform deep learning on tabular data, which the paper cites to explain its empirical results."}],"review_version":1}