{"id":"7d7eecfa-0c3f-4fe2-a093-2ee94a70d1b1","arxiv_id":"2412.03167","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A weighted-majority ensemble with utility-based reweighting outperforms the average member on Indian intraday data, but not consistently the best individual model.","lead":"This paper combines eight machine-learning stock predictors into a weighted-majority ensemble that reweights models every five minutes based on recent accuracy or a profit-proxy utility score. Tests on five days of Indian intraday data suggest the ensemble beats the average of its members, but the headline claim that it beats every individual model is contradicted by the paper's own tables.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Paper's own tables contradict the abstract: at (5,10) MOE MLP accuracy 26.98% exceeds WMA AccWts 25.76%, and at (5,5) LSTM Cat utility 1.983 exceeds WMA UtilWts 0.078; Table 2 vs. Table 3 are also mutually inconsistent.","rationale":"The reader's verdict of REJECT is justified. The most load-bearing concern is that the paper's own experimental tables contradict the central claim, and the tables are internally inconsistent. Table 1 shows a clear accuracy counterexample: at window (5,10), the MOE_MLP individual model achieves 26.98% accuracy while the best WMA ensemble (WMA AccWts) achieves 25.76%. Table 3 shows a utility counterexample at (5,5): LSTM Cat's average utility is 1.9830 versus 0.0780 for WMA UtilWts. These are not marginal differences, and they directly refute the abstract's wording 'over any of the individual models.' The reader's weakest_assumption about per-trade utility normalization is a real and serious flaw, and it may explain the implausibly high LSTM Cat utility of nearly 2.0 if that model trades very rarely. However, the accuracy counterexample is independent of the utility normalization and suffices on its own to falsify the headline claim. Additionally, the discrepancy between Table 2 and Table 3 for LSTM Cat utility at (5,5) means the reported results are not self-consistent, making the paper's conclusions irreproducible as written. Therefore, the verdict should remain REJECT, and no verdict adjustment is needed.","tokens_in":14191,"tokens_out":7221,"duration_ms":56543,"concrete_test":"Recompute all accuracy and utility entries from the raw per-round model predictions for the five test days (2, 3, 8, 10, and 14 May 2024), or at minimum reproduce Table 3 from the code and reconcile Table 2 against it. Specifically verify (i) MOE_MLP accuracy 26.98% versus WMA AccWts 25.76% at window (5,10), and (ii) LSTM Cat utility 1.983 versus WMA UtilWts 0.078 at window (5,5), including the masking procedure and per-model support counts. If those numbers reproduce, the abstract's 'any individual model' claim is false; if they do not, the paper contains reporting errors that invalidate its conclusions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract claims that weighted-majority ensembles show improved accuracy as well as utility over any individual model. Table 1 falsifies the accuracy half: at window (5,10), MOE_MLP accuracy is 26.98% while WMA AccWts is 25.76% and WMA UtilWts is 25.30%, so the ensemble is not the most accurate. Table 3 falsifies the utility half even internally: at (5,5), LSTM Cat average utility is 1.9830 versus WMA UtilWts 0.0780 and WMA AccWts 0.0265. The paper also contains an irreconcilable inconsistency: Table 2 reports LSTM Cat (5,5) utility as -0.0489, which matches Table 3's CNN Cat value, while Table 3's LSTM Cat row shows 1.9830; both cannot be correct. Section 4.2's normalization of utility by the per-model masked support, i.e. 'average utility per trade', further undermines cross-model comparisons when trade counts differ by orders of magnitude, as the reader notes. Yet even setting utility aside, the accuracy counterexample alone refutes the universal superiority claim. The central contribution is therefore unsupported by the paper's own reported evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Numin, a weighted-majority ensemble framework for intraday equity trading. Eight ML models predict five-class discretized ten-candle returns from five-minute OHLCV data; the ensemble combines their predictions via dynamic weights updated with an exponential moving average, using either accuracy or a hand-set utility payoff matrix as the scoring metric. Experiments are run on five trading days in May 2024, and the abstract claims that the weighted-majority ensembles show improved accuracy as well as utility over any individual model. The body does not support this claim: the paper's own tables contain direct counterexamples to both halves of the claim, and the evaluation protocol has comparability problems that make the utility comparison unreliable.","tokens_in":14508,"tokens_out":6172,"duration_ms":50239,"significance":"If the claimed results were valid, the paper would offer a practically relevant template for Numerai-style aggregation of intraday models, with the useful design choice of scoring models by a trading-utility proxy. The paper also has realistic elements: delayed ground-truth labels, a 10-round holding period, and a 10-round masking protocol that mimics an open position. However, the central empirical claim is contradicted by the reported numbers, and the utility metric is both hand-set and used for weighting and evaluation, so the current significance is mostly in posing the ensemble problem rather than in establishing an effective solution.","major_comments":[{"comment":"The abstract's accuracy claim is refuted by Table 1. At window size (5,10), MOE_MLP achieves 26.98% accuracy, while WMA AccWts achieves 25.76% and WMA UtilWts achieves 25.30%. Thus the weighted-majority ensemble is not more accurate than every individual model, and the unconditional claim in the abstract cannot stand.","section":"Abstract / Table 1"},{"comment":"The utility claim is also refuted by Table 3. At window size (5,5), LSTM Cat has average utility 1.9830, while WMA UtilWts has 0.0780 and WMA AccWts has 0.0265. Additionally, Table 2 reports LSTM Cat utility at (5,5) as -0.0170, which matches Table 3's LSTM Cat value for (5,20) rather than (5,5); the two tables are mutually inconsistent, so the reported utility numbers cannot all be correct.","section":"Abstract / Table 3"},{"comment":"The utility comparison is not interpretable as presented. The protocol masks the next 10 rounds for each model whenever that model predicts an extreme class, and then normalizes total utility by the per-model test support, yielding 'average utility per trade.' Because different models have different trade counts and supports, a model that trades rarely but happens to be correct can report a large average utility per trade, while the ensemble's average is diluted over a different support. The paper reports neither the trade counts nor the total utilities, so comparing WMA UtilWts 0.078 with LSTM Cat 1.983 is not a valid cross-model comparison.","section":"Section 4.2"},{"comment":"The utility payoff matrix is hand-set without external calibration or sensitivity analysis, and transaction costs are ignored. Since the same utility matrix is used both to update ensemble weights and to evaluate the final predictions, the 'improved utility' result is partly a statement about the metric's own optimization target rather than about realized trading profitability. The five-day test period and the absence of any significance test further weaken the claim that utility-weighted WMA reliably outperforms individual models.","section":"Sections 4.2 and 5.0.2"}],"minor_comments":[{"comment":"The phrase 'weighed majority vote' should be 'weighted majority vote.'","section":"Section 3.1"},{"comment":"Several standard-deviation entries in Table 2 are negative (e.g., -0.0057, -0.0136), which is impossible for a standard deviation; the table also appears to have misaligned columns.","section":"Table 2"},{"comment":"There are minor grammatical errors such as 'using it's pre-training knowledge' and 'the model shares the same architecture' where the verb agreement is incorrect.","section":"Sections 3.2.1 and 3.2.4"},{"comment":"The figure labels contain 'A VG_MODEL' with an extra space; this should be corrected to 'AVG_MODEL.'","section":"Figures 1 and 2"},{"comment":"The paper says the five bins were chosen to give a roughly equal class distribution, but it does not report the resulting class frequencies or the exact bin edges, which makes the discretization hard to reproduce.","section":"Section 4.1"}],"recommendation":"reject","confidential_remarks":"The paper appears to be a published ICAIF workshop-style contribution, but as a journal submission the central abstract claim is falsified by the paper's own tables and the tables are internally inconsistent. These are load-bearing issues, not presentation issues, so I recommend rejection rather than major revision. If the authors can correct the tables and replace the unconditional claim with the weaker, defensible claim about beating the average model, and can provide a fair trade-count-matched comparison plus a robustness check of the utility matrix, a resubmission could be considered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper's headline claim — that its weighted-majority ensembles beat every individual model on both accuracy and utility — is false on its own numbers. The underlying idea, using a utility proxy instead of raw accuracy for dynamic reweighting in an intraday ensemble, is reasonable and worth a look. But the reported evidence is internally inconsistent and the evaluation protocol has a normalization problem that undermines the comparisons.\n\nWhat's genuinely useful: the authors specify a concrete trading simulation with masking after extreme predictions, and they compare accuracy-weighted vs. utility-weighted WMA across window sizes on Indian 5-minute equity data with eight diverse models. The algorithmic skeleton is standard (Littlestone-Warmuth, Kolter-Maloof) but the application to utility-based reweighting in this setting is a sensible experiment, and the paper is readable enough to be reimplemented.\n\nThe soft spots are serious. First, the abstract says the ensembles show improved accuracy as well as utility over any individual model. Table 1 contradicts the accuracy claim: at window (5,10), MOE_MLP has accuracy 26.98% against WMA AccWts 25.76% and WMA UtilWts 25.30%. Table 3 contradicts the utility claim: at (5,5), LSTM Cat has average utility 1.9830 against WMA UtilWts 0.0780. So the universal superiority claim is simply not supported by the data in the paper.\n\nSecond, the tables don't agree with each other. Table 2 lists LSTM Cat's (5,5) utility as -0.0170, while Table 3 shows 1.9830 for the same cell. That is not a rounding issue; it is a data integrity problem that a referee would flag immediately.\n\nThird, the utility normalization is questionable. The authors compute 'average utility per trade' after masking 10 rounds following an extreme prediction. A model that trades very rarely but happens to be correct on those trades will report utility near +2 (indeed, LSTM Cat, Cont MLP, and AutoEncoder all show ~1.95-2.00 at (5,5) despite ~20-24% accuracy). Comparing that number directly to the ensemble's utility per trade, when trade counts differ by orders of magnitude, is not a fair test.\n\nFourth, the empirical base is thin: five test days, no transaction costs, post hoc selection of window size, and no code or data. These are individually common in short conference papers, but together with the above they sink the paper's central claim.\n\nWho is this for? Someone building a Numerai-style ensemble for intraday markets might find the framework worth considering, and the utility-reweighting idea is not silly. But the current paper needs a major reanalysis and a corrected abstract before it can be taken seriously. I would not send this to peer review as-is; I'd desk reject or ask for major revision. I wouldn't cite it in its current form.","headline":"The paper's central claim is contradicted by its own tables, and the utility normalization makes the comparisons unfair; the framework idea is fine but the evidence isn't.","tokens_in":15043,"tokens_out":4580,"would_cite":false,"duration_ms":37541,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A weighted-majority ensemble that re-weights models by recent trading utility is claimed to beat every individual model on intraday stock prediction.","keywords":["stock prediction","intraday trading","weighted-majority algorithm","ensemble learning","dynamic model weighting","utility metric","five-minute candles","model competition"],"falsifier":"A reader can settle the central claim by recomputing raw total utility, i.e., summing utility over all non-masked rounds for each model rather than dividing by each model's trade count, using the per-day data in Table 3; if the best individual model's raw total exceeds the ensemble's on a majority of days, the 'better than any individual model' claim fails.","tokens_in":13988,"feed_emoji":"📈","tokens_out":8929,"duration_ms":69161,"temperature":0.7,"pith_summary":"The paper claims that a weighted-majority ensemble can reliably beat every one of eight individual machine-learning models at short-horizon intraday stock prediction, provided the models are re-weighted by their recent performance rather than trusted equally. The setting is a Numerai-style competition in which independent data scientists submit five-minute-ahead predictions of a discretized ten-candle return, and the ensemble must decide in real time which submitters to trust. The authors argue that the weight-update rule matters as much as the models themselves: scoring each model by a utility proxy for profitability, over a short lookback window, produces the best ensemble accuracy and the only consistently positive average utility, even though most individual models lose money on average. If this works, it gives a practical way to harvest many noisy forecasting models into a single tradable signal and to reward model owners proportionally to their recent contribution.","feed_headline":"Utility-weighted ensemble beats all eight individual models","feed_subtitle":"Re-weighting by recent trading utility, not accuracy, lets a meta-model profit where most solo models lose.","key_machinery":"The central mechanism is a dynamic weighted-majority algorithm with exponential moving average weight updates. Each model's score is its accuracy or its utility over a sliding window of recent rounds (with minimum window size $\\mu$ and maximum $\\lambda$); scores are normalized across models, and weights are updated as $w_j^{(r)} = \\alpha \\tilde{s}_j^{(r)} + (1-\\alpha) w_j^{(r-1)}$ with $\\alpha = 2/(|\\text{win}|+1)$. The utility metric is defined by a $5\\times 5$ payoff matrix $U$ whose nonzero entries are concentrated on the extreme-class diagonal and the opposite-extreme off-diagonal. At test time, the ensemble votes by summing each model's weight onto the class it predicts, choosing the class with the largest total; trading is simulated by masking the ten rounds after each extreme prediction, so reported utilities are per-trade averages.","core_discovery":"The paper claims to show that a weighted-majority ensemble of eight machine-learning models—CNN classifiers and regressors, LSTMs, a continually retrained MLP, mixture-of-experts models, a test-time-adapting autoencoder, and a k-means MoE—can beat every one of its members on both accuracy and profitability when used to predict discretized ten-candle returns from five-minute equity candles. The profitability signal is a utility matrix in which only extreme predictions (class 0 or 4) open a long or short position held for ten rounds; correct extremes earn +2, opposite extremes lose 2, and non-extreme predictions earn nothing. The paper reports that using this utility, rather than plain accuracy, to re-weight models via an exponential moving average over a short window yields positive average utility per trade on the test days, and that the ensemble's accuracy also exceeds the average of the individual models. The intended upshot is a practical aggregation scheme for a Numerai-style intraday forecasting competition, where the same weights can reward model owners.","pith_inferences":["The masking procedure makes models incomparable: a model that almost never predicts extremes will have very few trades and can report per-trade utility near +2, so the ensemble's average utility should be compared against raw total utility or a minimum-trade-count requirement.","A sharper test of the superiority claim is per-day comparison against the best individual model; if the ensemble wins only against the average model, it is not 'better than any individual model' as the abstract states.","Adding realistic transaction costs to the utility matrix (even a small per-trade charge) would turn the zero-payoff middle cells negative and could erode the reported positive average utilities, making live profitability dependent on cost level."],"forward_implications":["A Numerai-style intraday competition can use the same weights both to form a tradable consensus and to reward model owners, aligning payouts with recent contribution.","Short re-weighting windows (e.g., 25 minutes) are reported to give the best utility, while longer windows (about two hours) give better accuracy, so a platform can tune which objective it wants.","An ensemble of many mediocre models can become profitable on average even when most individual models lose money, provided the weighting metric tracks recent skill rather than global accuracy.","The approach is model-agnostic: new predictors can be added or removed without retraining the ensemble, since only the per-model scores change."],"supporting_citations":[{"why":"Supplies the competitive forecasting scenario and the idea of rewarding model owners with weights.","marker":"[1]"},{"why":"Provides the dynamic weighted majority algorithm whose weight updates the paper adapts.","marker":"[6]"},{"why":"Origin of the weighted-majority voting rule the ensemble uses to combine predictions.","marker":"[8]"},{"why":"Source of the utility metric used as a profitability proxy for re-weighting models.","marker":"[10]"},{"why":"Recent stream-adaptation variant of dynamic weighting that motivates the short-window updates.","marker":"[9]"}],"fun_headline_variants":["Weighted-majority ensemble beats every single model in trading","Utility weighting turns model ensemble into profitable trader","Ensemble reweighted by recent utility beats all 8 solo models","Dynamic utility weights make ensemble outperform every member"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results stand on the assumption that average utility per trade, computed after skipping the ten rounds that follow each extreme prediction and ignoring transaction costs, is a fair yardstick for comparing models that may trade wildly different numbers of times.","fun_headline_variants_meta":{"raw":{"variants":["Weighted-majority ensemble beats every single model in trading","Utility weighting turns model ensemble into profitable trader","Ensemble reweighted by recent utility beats all 8 solo models","Dynamic utility weights make ensemble outperform every member"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000769,"raw_usage":{"total_tokens":3394,"prompt_tokens":921,"completion_tokens":2473,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":2408}},"tokens_in":537,"tokens_out":2473,"duration_ms":18541,"temperature":1.0,"reasoning_tokens":2408,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:41:49.295620+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader can settle the central claim by recomputing raw total utility, i.e., summing utility over all non-masked rounds for each model rather than dividing by each model's trade count, using the per-day data in Table 3; if the best individual model's raw total exceeds the ensemble's on a majority of days, the 'better than any individual model' claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the competitive forecasting scenario and the idea of rewarding model owners with weights."},{"cited_title":"Zico Kolter and Marcus A","cited_arxiv_id":null,"evidence_quote":"Provides the dynamic weighted majority algorithm whose weight updates the paper adapts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the utility metric used as a profitability proxy for re-weighting models."},{"cited_title":"On the Adaptation to Concept Drift for CTR Prediction","cited_arxiv_id":"2204.05101","evidence_quote":"Recent stream-adaptation variant of dynamic weighting that motivates the short-window updates."}],"review_version":1}