{"id":"7dce0c2a-e5e8-4853-afd8-32e9e42d4f2a","arxiv_id":"2606.24994","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"ExTra adds embedding-based novelty rewards and entropy-guided prefix regeneration to GRPO, improving pass@1 by ~5 and pass@16 by ~7 on math benchmarks with Qwen3-1.7B.","lead":"The paper presents ExTra, a method that adds novelty-based diversity rewards and entropy-guided trajectory continuation to standard RL training for language models on reasoning tasks. Smart readers might examine it to see practical ways to boost exploration when standard reward signals are weak or absent.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Embedding-based novelty may reward superficial diversity or model-intrinsic patterns rather than useful exploration, undermining attribution of the reported gains.","rationale":"The reader's weakest assumption matches the load-bearing point exactly: whether the extracted signals are non-spurious. The abstract-only review already flagged low confidence; the mechanism description does not resolve it. No other internal inconsistency (e.g., normalization details or benchmark selection) rises to the same level of risk for the headline empirical claim.","tokens_in":1651,"tokens_out":325,"duration_ms":14791,"concrete_test":"Ablate the novelty term (set its coefficient to zero or replace with random bonuses of matched magnitude) while retaining entropy-guided prefix regeneration; re-run the six-benchmark evaluation on Qwen3-1.7B. If the advantage over GRPO vanishes, the novelty signal is load-bearing; if gains persist, the concern does not land.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim attributes +5 pass@1 / +7 pass@16 gains on six math benchmarks to ExTra's two signals. The novelty reward is added after GRPO normalization and is described as rewarding diverse correct solutions via embeddings. If the embeddings come from the policy model itself, the bonus may simply correlate with already-high-probability trajectories rather than true solution diversity; if external, it may capture lexical rather than semantic variety. No control is described that isolates whether a random or variance-increasing bonus produces comparable gains, leaving open whether the signal is causal or the entropy-guided prefix regeneration alone drives results.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces ExTra, a GRPO-compatible framework for RLVR in language models that addresses low-diversity rollouts on easy prompts and zero-reward groups on hard prompts. It adds two mechanisms: (i) a novelty reward that applies embedding-based diversity bonuses after GRPO normalization to reward diverse correct solutions, and (ii) entropy-guided prefix regeneration that scores partial trajectories and restarts exploration from promising prefixes. On six mathematical reasoning benchmarks, ExTra improves Qwen3-1.7B over GRPO by roughly +5 points pass@1 and +7 points pass@16.","tokens_in":1785,"tokens_out":593,"duration_ms":15819,"significance":"If the gains prove robust and causally attributable to the two signals, the work supplies a lightweight, rollout-internal method for improving both training signal and inference coverage in reasoning RL. The approach of extracting novelty and entropy signals directly from the policy's own trajectories is a practical strength that avoids external models or additional parameters.","major_comments":[{"comment":"Experimental section (results tables and §4): the central empirical claim of +5 pass@1 / +7 pass@16 gains is presented without reported run counts, standard deviations, number of random seeds, or statistical tests. This prevents verification that the improvements exceed baseline variance and is load-bearing for the attribution to ExTra.","section":"Experimental results / §4"},{"comment":"§3.1 (novelty reward definition): the embedding-based bonus is added after GRPO normalization and described as rewarding diverse correct solutions, yet no ablation or control (e.g., random bonus of matched magnitude, or lexical-only diversity) is reported. Without this, it is impossible to rule out that any variance-increasing additive term would produce comparable gains, undermining the claim that the embedding signal is the causal mechanism.","section":"§3.1"},{"comment":"§3.2 (entropy-guided prefix regeneration): the method scores partial trajectories with entropy and regenerates from promising prefixes, but the paper supplies no comparison isolating this component from the novelty reward alone, nor any analysis of how often regeneration is triggered or its effect on gradient variance.","section":"§3.2"}],"minor_comments":[{"comment":"Notation for the combined reward (Eq. in §3) should explicitly state whether the novelty term is normalized per group or globally, and whether it is applied only to correct trajectories.","section":"§3"},{"comment":"The six benchmarks are listed in the abstract but the main text should include a table with per-benchmark pass@1 and pass@16 numbers plus GRPO baselines for direct comparison.","section":"Results"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback on strengthening the empirical validation and providing necessary ablations. We address each major comment below and will revise the manuscript accordingly.","responses":[{"response":"We agree that the absence of run statistics limits verification of the gains. In the revised manuscript we will report results over multiple random seeds (with the exact count specified), include standard deviations, and add statistical significance tests to confirm the improvements exceed baseline variance.","revision_made":"yes","referee_comment":"[Experimental results / §4] Experimental section (results tables and §4): the central empirical claim of +5 pass@1 / +7 pass@16 gains is presented without reported run counts, standard deviations, number of random seeds, or statistical tests. This prevents verification that the improvements exceed baseline variance and is load-bearing for the attribution to ExTra."},{"response":"We acknowledge that controls are required to establish causality for the embedding signal. We will add ablations in the revision comparing the embedding novelty reward against a random bonus of matched magnitude and a lexical-only diversity baseline.","revision_made":"yes","referee_comment":"[§3.1] §3.1 (novelty reward definition): the embedding-based bonus is added after GRPO normalization and described as rewarding diverse correct solutions, yet no ablation or control (e.g., random bonus of matched magnitude, or lexical-only diversity) is reported. Without this, it is impossible to rule out that any variance-increasing additive term would produce comparable gains, undermining the claim that the embedding signal is the causal mechanism."},{"response":"We will add an ablation isolating the entropy-guided prefix regeneration from the novelty reward alone. We will also report regeneration trigger frequency and analyze its impact on gradient variance.","revision_made":"yes","referee_comment":"[§3.2] §3.2 (entropy-guided prefix regeneration): the method scores partial trajectories with entropy and regenerates from promising prefixes, but the paper supplies no comparison isolating this component from the novelty reward alone, nor any analysis of how often regeneration is triggered or its effect on gradient variance."}],"tokens_in":1417,"tokens_out":470,"duration_ms":16046,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"ExTra adds two mechanisms on top of GRPO: an embedding-based novelty reward applied after normalization to encourage diverse correct solutions, and entropy-guided regeneration that restarts from high-entropy promising prefixes. These target the sparsity problem where easy prompts yield uniform correct rollouts and hard ones yield none.\n\nThe paper does a clean job of spelling out the practical fixes and reports concrete gains of about +5 pass@1 and +7 pass@16 on six math benchmarks with Qwen3-1.7B. The entropy regeneration idea in particular looks like a straightforward way to keep exploring from intermediate states instead of discarding partial trajectories.\n\nThe soft spots are the missing controls. The abstract supplies no run counts, variance numbers, or ablations that turn the novelty bonus or the regeneration off separately. The stress-test concern lands: if the embeddings come from the policy model itself, the bonus may simply track already-likely trajectories rather than add independent diversity, and nothing in the description rules out that a random or variance-boosting term would produce similar numbers. Without those checks, attribution stays shaky.\n\nThis is for groups already running GRPO on reasoning tasks who want ready-to-try exploration tweaks. A reader looking for implementable ideas will find value, but anyone needing solid causal evidence will want the full experiments expanded. It deserves peer review because the problem is real and the mechanisms are clearly defined, even though the current results need more rigor to hold up.","headline":"ExTra layers embedding novelty bonuses and entropy prefix regen onto GRPO for better exploration in LLM math RL, but the gains rest on thin evidence without ablations or controls.","tokens_in":2271,"tokens_out":372,"would_cite":false,"duration_ms":18704,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"ExTra adds embedding novelty rewards and entropy prefix regeneration to GRPO, lifting math reasoning pass rates by five points.","keywords":["ExTra","Exploratory Trajectory Optimization","GRPO","reinforcement learning","language models","exploration","mathematical reasoning","verifiable rewards"],"falsifier":"An ablation that removes both the novelty reward and the entropy-guided regeneration, then measures no change or a drop in pass@1 and pass@16 on the six benchmarks, would falsify the claim that these signals drive the observed gains.","tokens_in":2574,"feed_emoji":"🚀","tokens_out":645,"duration_ms":13939,"temperature":0.7,"pith_summary":"The paper introduces ExTra to address two failure modes in reinforcement learning with verifiable rewards for language models. Easy prompts produce uniform correct rollouts that supply little gradient signal, while hard prompts produce uniform incorrect rollouts that supply no positive reward. ExTra extracts two exploration signals directly from the model's own rollouts: an embedding-based novelty bonus that rewards diverse correct answers after GRPO normalization, and entropy-guided regeneration that restarts from promising intermediate prefixes. These additions produce measurable gains on standard benchmarks. A reader would care because the method improves both single-answer accuracy and the variety of solutions found at inference time without requiring external data or changes to the base optimizer.","feed_headline":"ExTra lifts math reasoning by five points over GRPO","feed_subtitle":"Embedding novelty rewards and entropy prefix regeneration improve both pass@1 accuracy and solution coverage on six benchmarks.","key_machinery":"ExTra's novelty reward using embedding diversity after GRPO normalization and its entropy-guided prefix regeneration from high-entropy partial trajectories.","core_discovery":"ExTra is a GRPO-compatible framework that extracts exploration signals from the model's own rollouts. It combines a novelty reward that adds embedding-based diversity bonuses after GRPO normalization to reward diverse correct solutions, and entropy-guided prefix regeneration that scores partial trajectories by entropy and continues exploration from promising intermediate steps. Across six mathematical reasoning benchmarks this produces roughly five-point gains on pass@1 and seven-point gains on pass@16 for Qwen3-1.7B.","pith_inferences":["The same rollout-derived signals could be tested in reinforcement learning setups outside mathematical reasoning.","If the signals prove stable, they might reduce the performance gap between smaller and larger models on verifiable-reward tasks.","Measuring whether the novelty and entropy bonuses remain effective when the base model changes would test robustness."],"forward_implications":["Single-sample accuracy on mathematical reasoning tasks increases.","Inference-time coverage of distinct correct solutions increases.","The framework remains compatible with existing GRPO training pipelines.","Exploration bonuses are derived entirely from the model's rollouts without external data."],"fun_headline_variants":["ExTra gains five points on math reasoning over GRPO","ExTra novelty rewards produce five point math gains over GRPO","Entropy guided prefixes improve ExTra math by five points over GRPO","Trajectory level signals add five points to math in ExTra vs GRPO"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The embedding-based novelty signal and entropy scores from the model's own rollouts supply useful, non-spurious exploration bonuses that improve learning rather than merely increasing variance.","fun_headline_variants_meta":{"raw":{"variants":["ExTra gains five points on math reasoning over GRPO","ExTra novelty rewards produce five point math gains over GRPO","Entropy guided prefixes improve ExTra math by five points over GRPO","Trajectory level signals add five points to math in ExTra vs GRPO"]},"model":"grok-4.3","cost_usd":0.011302,"raw_usage":{"total_tokens":4941,"prompt_tokens":625,"num_sources_used":0,"completion_tokens":71,"cost_in_usd_ticks":113024500,"prompt_tokens_details":{"text_tokens":625,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":4245,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":625,"tokens_out":71,"duration_ms":34282,"temperature":1.0,"reasoning_tokens":4245,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-26T00:21:15.969276+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"An ablation that removes both the novelty reward and the entropy-guided regeneration, then measures no change or a drop in pass@1 and pass@16 on the six benchmarks, would falsify the claim that these signals drive the observed gains.","supporting_citations":[],"review_version":1}