{"id":"594c581a-2ea9-4e71-85af-fdf7172936b6","arxiv_id":"2604.08577","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"DRTO improves LLM consistency on reasoning tasks under distribution shifts by 4.4 points on MATH-500 and 2.7 points on LiveCodeBench via f-divergence ambiguity sets in token-level RLHF.","lead":"The paper proposes Distributionally Robust Token Optimization (DRTO) that applies distributionally robust optimization to token-level RLHF using f-divergence ambiguity sets over response spans. A smart generalist might read it to see a concrete way to make LLM reasoning more stable when prompts vary slightly in wording or format.","discovery_kind":"unclear","skeptic_critique":{"model":"grok-4.3","headline":"The reported gains may arise from token-level RLHF regularization rather than the f-divergence ambiguity sets on spans actually capturing prompt distribution shifts.","rationale":"The reader's weakest assumption directly identifies the missing link between the proposed construction and the claimed robustness. Because the supplied text is only the abstract, the technical details that would confirm or refute that link remain unverified; the concrete ablation above would settle the issue without requiring external data.","tokens_in":1628,"tokens_out":319,"duration_ms":32512,"concrete_test":"Reproduce the MATH-500 and LiveCodeBench numbers with the exact DRTO implementation; then rerun with the ambiguity-set radius set to zero (recovering standard RTO) while keeping all other hyperparameters fixed. If the deltas fall below 1 pp, the robustness mechanism is not load-bearing.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim requires that constructing f-divergence balls around span-level actor losses produces a policy robust to real prompt shifts (wording, format) on reasoning tasks. The abstract provides no definition of span boundaries, no closed-form or algorithmic description of the inner maximization over the ambiguity set, and no ablation separating the DRO term from the base token-level RL objective. Consequently the +4.4 pp and +2.7 pp deltas could be explained by any additional optimization pressure on difficult tokens rather than distributional robustness. If the span-level construction does not align with the actual failure modes induced by prompt shifts, the DRO component contributes nothing beyond standard RTO.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper proposes Distributionally Robust Token Optimization (DRTO), which augments token-level RLHF with DRO by constructing f-divergence ambiguity sets over span-level actor losses. The goal is to emphasize difficult response segments and improve policy robustness to prompt distribution shifts (wording, format) on reasoning tasks. Empirically, DRTO is reported to outperform standard RTO by +4.4 percentage points on MATH-500 and +2.7 percentage points on LiveCodeBench.","tokens_in":1771,"tokens_out":475,"duration_ms":33495,"significance":"If the gains can be shown to arise specifically from the f-divergence mechanism rather than generic token-level regularization, the work would offer a principled extension of RLHF that addresses a practically important failure mode in LLMs. The approach could influence robustness techniques in alignment research, provided the span-level construction aligns with real prompt-shift failure modes.","major_comments":[{"comment":"Abstract: the central claim that f-divergence ambiguity sets over span-level actor losses produce robustness to prompt shifts is load-bearing, yet no definition of span boundaries, no closed-form or algorithmic description of the inner maximization, and no ablation isolating the DRO term from the base RTO objective are supplied; without these the reported +4.4 pp and +2.7 pp deltas cannot be attributed to distributional robustness rather than additional optimization pressure on difficult tokens.","section":"Abstract"},{"comment":"Empirical evaluation: the abstract reports clear deltas on MATH-500 and LiveCodeBench but supplies neither error bars, ablation tables separating DRO from token-level RL, nor verification that ambiguity-set radii were not tuned post-hoc to the test shifts; this directly undermines the claim that the method enhances consistency under distribution shifts.","section":"Empirical evaluation"}],"minor_comments":[{"comment":"The phrase 'among different tasks' in the abstract is ambiguous; clarify which tasks and shifts are evaluated.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's soundness is limited by missing methodological detail and controls; the central attribution to DRO remains unverified. The work fits the journal's scope on robust learning but would benefit from code release for reproducibility."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback on our work. The comments highlight important areas for improving clarity and empirical rigor, and we will revise the manuscript to address them directly.","responses":[{"response":"We agree that the abstract is too concise and that these details are essential for attributing the gains. In the revision we will expand the abstract to define span boundaries explicitly (as contiguous token groups aligned with reasoning steps), provide the closed-form dual solution for the inner maximization over the f-divergence ball, and include a new ablation table that isolates the DRO regularizer from the base token-level RL objective. These additions will make clear that the reported improvements arise from the worst-case emphasis within the ambiguity set rather than generic token-level pressure.","revision_made":"yes","referee_comment":"[Abstract] Abstract: the central claim that f-divergence ambiguity sets over span-level actor losses produce robustness to prompt shifts is load-bearing, yet no definition of span boundaries, no closed-form or algorithmic description of the inner maximization, and no ablation isolating the DRO term from the base RTO objective are supplied; without these the reported +4.4 pp and +2.7 pp deltas cannot be attributed to distributional robustness rather than additional optimization pressure on difficult tokens."},{"response":"We accept that the current empirical section lacks sufficient controls. We will add error bars computed over multiple random seeds, a full ablation table that separates the DRO component from standard token-level RL, and a dedicated paragraph explaining radius selection: radii were chosen via grid search on a held-out validation split drawn from the original training distribution, with no access to or tuning on the test shifts. These changes will substantiate that the consistency gains are due to distributional robustness.","revision_made":"yes","referee_comment":"[Empirical evaluation] Empirical evaluation: the abstract reports clear deltas on MATH-500 and LiveCodeBench but supplies neither error bars, ablation tables separating DRO from token-level RL, nor verification that ambiguity-set radii were not tuned post-hoc to the test shifts; this directly undermines the claim that the method enhances consistency under distribution shifts."}],"tokens_in":1314,"tokens_out":459,"duration_ms":40731,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The headline takeaway is that this paper folds distributional robustness into token-level RLHF by building f-divergence balls around span-level actor losses, and it shows small but positive lifts on two reasoning benchmarks under distribution shifts. What stands out is the attempt to handle brittleness in LLMs on multi-step problems by emphasizing difficult segments through the worst-case distribution in the ambiguity set. This is a natural move if you accept that prompt variations cause localized failures in reasoning traces. The empirical deltas of 4.4 points on MATH-500 and 2.7 on LiveCodeBench suggest the method has some practical effect compared to standard RTO. The soft spots are more noticeable. The abstract supplies no error bars, no ablation that isolates the contribution of the DRO component, and no description of how spans are chosen or how the inner maximization over the ambiguity set is computed. Without those, it's plausible that any added regularization on low-probability tokens could produce similar gains. The assumption that these particular ambiguity sets capture real user prompt shifts is stated but not tested in the provided summary. This work is for groups already running token-level RLHF experiments on reasoning models. A reader who wants ideas for making fine-tuning more stable across prompt styles would find the high-level construction useful, though they would have to fill in the implementation details themselves. I would bring this to the next reading group as a maybe, mainly to talk through whether the span-level DRO actually adds robustness or just changes the loss landscape. I would not cite it in my own work until the full methods and ablations are available. It deserves peer review because the underlying problem is important for deployed systems and the proposed direction is coherent enough to warrant detailed referee comments on the experiments and theory.","headline":"DRTO folds f-divergence DRO into span-level token RLHF and reports modest gains on shifted reasoning benchmarks, but the abstract leaves the mechanism's specific contribution unverified.","tokens_in":2243,"tokens_out":426,"would_cite":false,"duration_ms":34122,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":{"model":"grok-4.3","evidence":[{"relation":"unclear","rs_module":"IndisputableMonolith/Cost/FunctionalEquation.lean","rs_theorem":"washburn_uniqueness_aczel","paper_passage":"DRTO constructs f-divergence ambiguity sets over span-level actor losses... KL-DRTO yields entropic objective... χ²-DRTO uses minibatch mean and standard deviation (Theorems 3.1–3.2)"},{"relation":"unclear","rs_module":"IndisputableMonolith/Foundation/ArithmeticFromLogic.lean","rs_theorem":"absolute_floor_iff_bare_distinguishability","paper_passage":"RTO uses token-wise shaping reward rRTO_h ... PPO clipped surrogate Li(θ)"}],"headline":"DRTO applies standard f-divergence DRO (KL/χ² balls, entropic risk, variance surrogates) to token-level PPO losses; no J-cost, φ-ladder, or distinction-forcing structure.","alignment":"orthogonal","rationale":"Paper machinery (Theorems 3.1–3.2, Algorithms 2–3, RCL family absent) is conventional DRO on RLHF actor losses; RS derives J(x)=½(x+x⁻¹)−1 and φ from one distinction (AbsoluteFloorClosure, Cost.FunctionalEquation, AlexanderDuality). No overlap in cost function, periodicity, or parameter-free constants.","tokens_in":52336,"confidence":"high","tokens_out":326,"duration_ms":11629,"cache_read_input_tokens":32896,"cache_creation_input_tokens":0},"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"DRTO builds f-divergence sets on span-level losses to make token RLHF consistent under prompt shifts.","keywords":["Distributionally Robust Optimization","RLHF","Token Optimization","Reasoning Benchmarks","Distribution Shifts","Robustness"],"falsifier":"No gain or a loss in accuracy on a held-out collection of reworded MATH-500 and LiveCodeBench prompts when DRTO is compared with standard token-level RLHF.","tokens_in":2535,"feed_emoji":"📈","tokens_out":377,"duration_ms":38159,"temperature":0.7,"pith_summary":"Large language models often fail on reasoning tasks after small changes in prompt wording or format. The paper proposes Distributionally Robust Token Optimization to combine token-level RLHF with distributionally robust optimization. It constructs f-divergence ambiguity sets around span-level actor losses to focus training effort on the hardest response segments. This produces measurable gains on shifted versions of reasoning benchmarks. The result is a policy that maintains accuracy when user inputs deviate from the training distribution.","feed_headline":"Robust token optimization lifts reasoning accuracy under shifts","feed_subtitle":"DRTO improves MATH-500 by 4.4 points by focusing RLHF on difficult response spans","key_machinery":"f-divergence ambiguity sets over span-level actor losses, which bound worst-case losses and steer optimization toward harder segments.","core_discovery":"DRTO constructs f-divergence ambiguity sets over span-level actor losses to emphasize difficult response segments during policy optimization, yielding greater consistency under distribution shifts on multi-step reasoning tasks.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["DRTO uses f-divergence sets over span RLHF actor losses","Token-level DRO targets difficult segments in RLHF optimization","DRTO constructs ambiguity sets for robust reasoning under shifts","DRTO combines distributionally robust optimization with RLHF tokens"],"cache_read_input_tokens":64,"weakest_assumption_plain":"That f-divergence ambiguity sets constructed over span-level actor losses will reliably capture and mitigate the distribution shifts that occur in real user prompts on reasoning problems.","fun_headline_variants_meta":{"raw":{"variants":["DRTO uses f-divergence sets over span RLHF actor losses","Token-level DRO targets difficult segments in RLHF optimization","DRTO constructs ambiguity sets for robust reasoning under shifts","DRTO combines distributionally robust optimization with RLHF tokens"]},"model":"grok-4.3","cost_usd":0.007562,"raw_usage":{"total_tokens":3322,"prompt_tokens":541,"num_sources_used":0,"completion_tokens":66,"cost_in_usd_ticks":75615500,"prompt_tokens_details":{"text_tokens":541,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2715,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":541,"tokens_out":66,"duration_ms":34584,"temperature":1.0,"reasoning_tokens":2715,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-14T23:15:48.729380+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"No gain or a loss in accuracy on a held-out collection of reworded MATH-500 and LiveCodeBench prompts when DRTO is compared with standard token-level RLHF.","supporting_citations":[],"review_version":1}