{"id":"2caa2787-caf8-497f-9267-9502b2b26f4e","arxiv_id":"2605.25527","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":3.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Policy-based RL with group-normalized PPO variants improves PnL, profitability, and drawdown over Q-learning on order-book data for three stocks in simplified backtests.","lead":"The paper applies policy-gradient reinforcement learning methods, including group-aware variants inspired by DeepSeekMath, to high-frequency directional trading using order-flow states from limit order books. It reports that these methods outperform a Q-learning baseline on backtests for AMZN, AAPL, and GOOG under spread-scaled rewards.","discovery_kind":"unclear","skeptic_critique":{"model":"grok-4.3","headline":"Simplified spread-scaled backtest may omit execution frictions and market impact that would erase reported PnL gains.","rationale":"The reader's weakest_assumption directly identifies the evaluation environment as the load-bearing assumption; the abstract and claim description give no indication that the full text supplies independent evidence (e.g., live-trading results or impact-robustness checks) that would remove this dependence. Therefore the reader's UNVERDICTED stance with low confidence is unchanged.","tokens_in":1625,"tokens_out":347,"duration_ms":15616,"concrete_test":"Re-run the exact GRPO/GSPO and Q-learning agents on the same order-flow states but replace the spread-scaled reward with a simulator that (a) matches limit orders against the historical LOB with realistic queue priority and (b) applies a linear temporary impact term proportional to trade size; if the net-PnL advantage disappears or reverses on any of the three assets, the headline claim does not survive more realistic execution.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the reported net PnL, profitability, and drawdown improvements on AMZN/AAPL/GOOG are not artifacts of the evaluation environment. The paper states that results come from a \"simplified backtesting setup based on spread-scaled rewards.\" For this to support the conclusion that group-aware PPO variants are preferable, the reward and execution model must capture at least partial fills, latency, adverse selection, and temporary/permanent impact; otherwise the ranking versus Q-learning can reverse under any realistic microstructure. No machine-checked proof or parameter-free derivation is offered to insulate the claim from this modeling choice.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper studies reinforcement learning for high-frequency directional trading on limit order books, pairing an order-flow-based state representation with policy-gradient methods. It deploys vanilla PPO along with DeepSeekMath-inspired group-aware variants (GRPO and GSPO) that incorporate group-normalized updates and downside-aware reward shaping. On backtests for AMZN, AAPL, and GOOG under a simplified spread-scaled reward setup, the new policies are reported to improve net average PnL, profitability, and drawdown relative to a tabular Q-Learning baseline. The authors conclude that order-flow signals suffice as states for policy RL and that group-aware PPO surrogates are preferable to value-based methods.","tokens_in":1754,"tokens_out":444,"duration_ms":31766,"significance":"If the reported ranking of policies survives more realistic execution modeling, the work would provide evidence that group-normalized policy optimization can outperform standard value-based RL in HFT settings. The adaptation of group-relative techniques from mathematical reasoning models to trading constitutes a modest methodological contribution.","major_comments":[{"comment":"The central empirical claim—that GRPO/GSPO variants improve net PnL, profitability, and drawdown over Q-Learning—rests entirely on results from the 'simplified backtesting setup based on spread-scaled rewards.' This modeling choice omits partial fills, latency, adverse selection, temporary/permanent market impact, and realistic order execution. Because the performance ordering versus the baseline can reverse once these frictions are included, the claim that group-aware PPO surrogates are preferable is not yet supported by the evaluation.","section":"Backtesting / Evaluation"}],"minor_comments":[{"comment":"The abstract asserts quantitative improvements without supplying any numerical values, confidence intervals, or statistical tests; this should be remedied for clarity.","section":"Abstract"},{"comment":"Provide explicit equations or pseudocode for the group-normalization step and downside-aware shaping used in GRPO and GSPO so that the variants can be reproduced.","section":"Methods"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the detailed feedback on our work. The primary concern centers on the simplified nature of our backtesting evaluation, which we address directly below.","responses":[{"response":"We agree that the evaluation relies on a simplified backtesting setup that abstracts away important execution frictions including partial fills, latency, adverse selection, and market impact. This design choice was made to focus on the relative performance of the order-flow state representation and the group-aware policy optimization methods in a controlled environment. We recognize that the observed ranking of GRPO/GSPO over Q-Learning could change under more realistic execution modeling. Accordingly, we will revise the manuscript to (i) more explicitly qualify all performance claims as holding under the spread-scaled reward backtest, (ii) strengthen the limitations section with a dedicated discussion of omitted frictions, and (iii) outline concrete next steps for incorporating realistic order execution. These changes will prevent overgeneralization while preserving the contribution of the controlled comparison.","revision_made":"yes","referee_comment":"[Backtesting / Evaluation] The central empirical claim—that GRPO/GSPO variants improve net PnL, profitability, and drawdown over Q-Learning—rests entirely on results from the 'simplified backtesting setup based on spread-scaled rewards.' This modeling choice omits partial fills, latency, adverse selection, temporary/permanent market impact, and realistic order execution. Because the performance ordering versus the baseline can reverse once these frictions are included, the claim that group-aware PPO surrogates are preferable is not yet supported by the evaluation."}],"tokens_in":1270,"tokens_out":344,"duration_ms":22110,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The paper applies policy-gradient methods with group normalization to order-book trading and reports that GRPO and GSPO beat a Q-learning baseline on net PnL, profitability, and drawdown for AMZN, AAPL, and GOOG. The main move is replacing value-based RL with these group-normalized policy updates on an order-flow state representation.\n\nWhat the work does is take the group-normalization trick from recent math-focused RL papers and plug it into a trading environment with downside-aware reward shaping. That produces a modest but concrete extension: the policies appear more stable under the spread-scaled reward they chose. The order-flow state itself is a reasonable choice and the paper shows it is sufficient to drive policy learning.\n\nThe soft spot is the evaluation. The backtests use a simplified spread-scaled reward that does not model partial fills, latency, adverse selection, or market impact. Under those conditions the reported ranking can easily reverse once realistic microstructure is added, and the abstract supplies no numbers, confidence intervals, or statistical tests to judge the size of the effect. Three stocks is also narrow for claiming broader preference for group-aware methods.\n\nThis paper is for people already running RL experiments on limit-order data who want to try group normalization as a drop-in stabilizer. A reader looking for production-ready trading agents will not get much usable signal here.\n\nThe work shows honest engagement with the RL-for-trading literature and the central idea is testable, so it deserves a serious referee who can check the experimental controls and ask for more assets and friction modeling. I would not cite it as is, but I would send it out for review rather than desk reject.","headline":"Group-aware PPO variants beat Q-learning on simplified backtests for three stocks using order-flow states, but the evaluation setup is too stripped down to support the claims.","tokens_in":2225,"tokens_out":413,"would_cite":false,"duration_ms":20930,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Group-aware policy optimization outperforms Q-learning in order-book trading backtests.","keywords":["reinforcement learning","high-frequency trading","limit order book","policy optimization","order flow","PPO","group normalization"],"falsifier":"A follow-up experiment that replaces the spread-scaled reward with a simulator including realistic slippage, latency, and market impact and finds that the reported PnL and drawdown gains disappear or reverse.","tokens_in":2531,"feed_emoji":"📈","tokens_out":625,"duration_ms":31662,"temperature":0.7,"pith_summary":"The paper examines reinforcement learning for high-frequency trading on limit order books by pairing an order-flow state model with policy-gradient methods. It deploys vanilla PPO along with group-normalized variants GRPO and GSPO that incorporate downside-aware shaping, rather than relying on tabular Q-learning. In backtests on AMZN, AAPL, and GOOG under a spread-scaled reward setup, the new policies produce higher net average PnL, improved profitability, and reduced drawdowns relative to the Q-learning baseline. The work concludes that order-flow signals provide an adequate state representation and that group-aware PPO surrogates are preferable to value-based methods.","feed_headline":"Group-aware RL raises PnL in order-book trading tests","feed_subtitle":"GRPO and GSPO variants beat Q-learning on AMZN, AAPL, GOOG with higher returns and smaller drawdowns","key_machinery":"Group-aware PPO surrogates (GRPO and GSPO) that apply group-normalized updates and downside-aware shaping to an order-flow state model for directional trading decisions.","core_discovery":"Pairing an order-flow-based state model with policy-gradient methods that use group-normalized updates and downside-aware shaping yields higher net average PnL, profitability, and lower drawdowns than tabular Q-learning on backtests of AMZN, AAPL, and GOOG under simplified spread-scaled rewards.","pith_inferences":["The group-normalization step may stabilize learning in other financial time-series control problems beyond single-asset directional trades.","Testing the same policies on additional assets or longer horizons would clarify whether the observed improvements generalize past the three stocks examined.","Replacing the simplified reward with a full transaction-cost model could expose whether the group-aware advantage survives more realistic execution conditions."],"forward_implications":["Order-flow signals serve as a sufficient state representation for effective policy reinforcement learning in limit order book environments.","Group-aware policy methods deliver measurable gains in net PnL and drawdown control over value-based baselines under the tested reward structure.","Policy-gradient approaches with group normalization become viable alternatives to Q-learning for high-frequency directional trading tasks."],"fun_headline_variants":["Group-aware RL boosts order book PnL","Policy RL beats Q-learning in order book tests","GRPO improves trading PnL on AMZN AAPL GOOG","Group normalized PPO cuts drawdowns in LOB trading","Order flow policy gradients raise net PnL"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The simplified backtesting setup based on spread-scaled rewards accurately captures real-world trading frictions, execution quality, and market impact for the tested assets.","fun_headline_variants_meta":{"raw":{"variants":["Group-aware RL boosts order book PnL","Policy RL beats Q-learning in order book tests","GRPO improves trading PnL on AMZN AAPL GOOG","Group normalized PPO cuts drawdowns in LOB trading","Order flow policy gradients raise net PnL"]},"model":"grok-4.3","cost_usd":0.008521,"raw_usage":{"total_tokens":3801,"prompt_tokens":570,"num_sources_used":0,"completion_tokens":72,"cost_in_usd_ticks":85212000,"prompt_tokens_details":{"text_tokens":570,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":3159,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":570,"tokens_out":72,"duration_ms":24991,"temperature":1.0,"reasoning_tokens":3159,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-29T22:32:52.923712+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A follow-up experiment that replaces the spread-scaled reward with a simulator including realistic slippage, latency, and market impact and finds that the reported PnL and drawdown gains disappear or reverse.","supporting_citations":[],"review_version":1}