{"id":"7ce04d4a-339f-46b0-8414-dc0e0a602efc","arxiv_id":"2505.22866","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"SORL trains a shortcut-model policy with a one-stage actor-critic objective and uses the learned Q-function to select among sampled actions at test time.","lead":"This paper introduces SORL, an offline reinforcement learning algorithm built on shortcut generative models, so a policy can act in one step or many steps depending on available compute. It reports strong results across 40 standard robot tasks and shows that extra compute at decision time can often compensate for less training compute.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Self-consistency assumption and Lemma 5 use different time arguments (t vs t+h); as written, Assumption 1 does not imply Lemma 5's key bound, so Theorem 2 is unproven — likely fixable by aligning Assumption 1/Eq (11)-(13) with Algorithm 1.","rationale":"I read the paper as claiming two things: (i) SORL's training objective regularizes the policy to the behavior policy in W2 (Theorem 2), and (ii) the algorithm performs well empirically. For (i), the proof's key step is Lemma 5, which converts Assumption 1's self-consistency error into a bound on single-step error at larger step sizes. That step currently does not follow because the assumption and the lemma use different time arguments for the second shortcut step, so the 'by assumption' claim is unjustified. This is a concrete, verifiable gap in the mathematical argument, not a stylistic or 'outside consensus' issue. It is also where the reader's weakest assumption pointed, as the reader noted that Lemma 5's self-consistency condition is assumed rather than derived from Assumption 1. The gap is likely repairable: the algorithm pseudocode uses t+h, so the paper's own implementation appears to use the correct consistency target, and the error in Assumption 1 and Eq (11)-(13) may be typographical. However, until the assumption is corrected and the proof re-checked, the central regularization theorem is unproven as stated. For (ii), the empirical results are substantial — 40 tasks, 8 seeds, 10 baselines, runtime and ablations — and those claims are independent of the theorem, so they do not need to be rejected. A CONDITIONAL verdict with the proof-repair requirement is the right call, and it matches the reader's verdict.","tokens_in":28907,"tokens_out":6833,"duration_ms":60586,"concrete_test":"Inspect the released code's self-consistency target (Algorithm 1 vs Eq 11-13): does it compute the second shortcut as s_theta(a_{t+h}, t+h, h) or s_theta(a_{t+h}, t, h)? Independently re-derive Lemma 5's first bullet from Assumption 1 as written; for any shortcut model that depends on t (e.g., s(z,t,h)=z+t*h), the two quantities differ, so the implication should fail. If the code uses t+h, correct Assumption 1 and the equations, and re-verify the proof with the amended assumption; the theorem's conclusion should then follow. If the code uses t, measure the W2 error of the trained policy and check whether the reported scaling behavior persists under the t-based objective — the theory would need to be redone.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central theoretical claim is Theorem 2, which bounds W2(hat p(h), p*) by discretization, flow-matching, and self-consistency errors under Assumption 1. The proof reduces to Lemma 5, whose first bullet assumes E||F(2h')(z_t,t,t+2h') - F(h')(z_t,t,t+2h')||^2 <= 4h'^2 * eps_SC^2, where F(h') is the composition of two h' Euler steps, so the second step queries the shortcut model at time t+h'. Assumption 1's second bullet instead bounds E||s(z_t,t,h)/2 + s(z'_{t+h}, t, h)/2 - s(z_t,t,2h)||^2 <= eps_SC^2, with the second shortcut evaluation at time t. These quantities are not equal in general; the first involves s(.,t+h',h') after an intermediate step, while the second uses s(.,t,h). The paper's Eq (11)-(13) also write s_theta(a_{t+h}, t, h), although Algorithm 1 (the line st+h <- s_theta(a_{t+h}, t+h, h | x)) uses t+h. Because Lemma 5's proof says 'by assumption' for the first term, and that term is not the assumption as written, the bound in Lemma 5 does not follow from Assumption 1. Theorem 2 therefore does not hold under the stated assumptions; it requires either amending Assumption 1 to evaluate the second shortcut at t+h, or proving a new bound relating the two quantities. If the released code follows Algorithm 1 (t+h), the fix is minor; if it follows Eq (11)-(13) (t), the trained objective is not the two-step Euler consistency the theorem analyzes, making the regularization guarantee inapplicable to the actual algorithm.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SORL, an offline RL algorithm whose policy is a shortcut model trained jointly with a Q-function through a Q loss, a flow-matching loss, and a self-consistency loss. The claimed contributions are (i) a one-stage training procedure that supports variable numbers of inference steps, including one-step generation; (ii) a theoretical guarantee (Theorem 2) that if flow-matching and self-consistency losses are small then the learned sampling distribution is close in 2-Wasserstein distance to the offline data distribution for every step size h; and (iii) empirical results on 40 OGBench tasks showing best or competitive performance against 10 baselines, plus positive sequential and parallel inference-time scaling. The paper also releases code and includes ablations on backpropagation depth and network size.","tokens_in":29335,"tokens_out":7652,"duration_ms":78737,"significance":"If the theoretical guarantee is repaired, the paper makes a solid contribution: it unifies flow-matching regularization with a self-consistency objective so that one network can be queried at multiple discretization levels, and it provides a Wasserstein-style regularization interpretation analogous to recent flow-Q-learning results. The empirical evaluation is unusually thorough for this area: 8 seeds per task, 40 tasks, shared hyperparameters with baselines, and runtime comparisons. The authors are also appropriately cautious about the lack of a statistical guarantee for Q-function-based best-of-N verifiers, and they disclose that parallel scaling is an empirical rather than formal benefit. The main weakness is that the central theorem is not proved under the stated assumptions because of the two concrete gaps below; these are local and likely fixable rather than fatal to the approach.","major_comments":[{"comment":"The proof of Lemma 4 uses the bound ||v_t(z_{t'}) - v_{t'}(z_{t'})||_2 <= L_v(t - t') at Eq. (37), which requires v_t to be Lipschitz in t. However, Theorem 2 and Assumption 1 only assume v_t is L_v-Lipschitz in z for every t. The theorem's statement therefore needs an additional time-Lipschitz condition on v_t, or the discretization-error derivation must be replaced by one that does not use Eq. (37); as written, the discretization term in Theorem 2 is not justified.","section":"Appendix B.2, proof of Lemma 4; Eq. (37)"},{"comment":"The self-consistency assumption and the quantity bounded in Lemma 5 are not the same object. Assumption 1's second bullet bounds E||s(z_t,t,h)/2 + s(z'_{t+h}, t, h)/2 - s(z_t,t,2h)||^2, with both shortcut evaluations at time t, and Eq. (5) and Eq. (11)-(13) use the same convention. Lemma 5's first bullet, however, bounds the difference between F(2h')(z_t,t,t+2h') and F(h')(z_t,t,t+2h'), where F(h') is the composition of two Euler steps and the second shortcut evaluation occurs at time t+h'. Algorithm 1 indeed evaluates the second shortcut at time t+h (line 'st+h <- s_theta(a_{t+h}, t+h, h | x)'). Since Lemma 5's proof invokes the first term 'by assumption' but Assumption 1 does not control the quantity involving s(., t+h', h'), Theorem 2 does not follow from Assumption 1 as stated. The fix is to align Assumption 1 and Eqs. (5)/(11)-(13) with Algorithm 1, or to add a separate bound relating the two quantities; if the released code matches Algorithm 1, this is a local but necessary amendment.","section":"Assumption 1; Eq. (5); Eq. (11)-(13); Algorithm 1; Lemma 5, Appendix B.3"}],"minor_comments":[{"comment":"Equation (10) writes 'h ~ p(h,t)' although the flow-matching loss fixes h = 1/Mdisc; the notation should clarify that only t is sampled and h is held at the smallest step size.","section":"Section 3.1, Eq. (10)"},{"comment":"Equation (52) writes 'Le^{Lvh0}' where the Lipschitz constant should be L_v with a subscript, matching the rest of the proof.","section":"Appendix B.2, Eq. (52)"},{"comment":"The proof says 'Because s(., t, h') is L-Lipschitz (Assumption 1)', but this Lipschitz property is not part of Assumption 1; it is an additional condition stated in the lemma and in Theorem 2, so the citation should be corrected.","section":"Appendix B.3, proof of Lemma 5"},{"comment":"The environment list in Appendix E.1 names humanoidmaze-large and humanoidmaze-giant, while Table 1 reports humanoidmaze-medium and humanoidmaze-large; the lists should be aligned.","section":"Appendix E.1"},{"comment":"In Figure 1, antmaze-large performance decreases from 91 at one inference step to 88 at eight steps; the text says 'generally improves', which is accurate, but a brief note that the trend is not monotone in every environment would improve clarity.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The two proof gaps are fixable in a revision: add the missing time-Lipschitz assumption to Theorem 2, and align Assumption 1/Eqs. (5) and (11)-(13) with Algorithm 1's second shortcut evaluation at time t+h. I would ask the authors to verify which convention is implemented in the released code, since the theorem should match the algorithm actually run. If the alignment cannot be made, the theoretical contribution should be weakened to a statement about the idealized shortcut-model objective rather than the SORL training loss; the empirical contribution may still stand, but the claim that SORL regularizes to the behavior policy would need to be reworded."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The empirical work is the real contribution; the theory has a gap that is likely a typo but needs fixing before the theorem can be trusted.\n\nWhat is new: SORL is a one-stage actor-critic that trains a shortcut model policy with flow matching, self-consistency, and a Q loss. That combination is new for offline RL. The self-consistency objective is what lets the same network act with 1 to 8 inference steps and keeps backprop through time cheap. The experiments are the strongest part: 8 seeds, 40 OGBench tasks, best performance in 5 of 8 categories, and clear evidence of sequential and parallel scaling. The runtime comparison against FQL is a useful addition, and the paper is honest about the longer training time.\n\nThe theory is the soft spot. The bound in Theorem 2 does not follow from Assumption 1 as written. Lemma 4 needs the drift to be Lipschitz in time, but Theorem 2 only states Lipschitz in space; Eq. (37) silently uses the time-Lipschitz condition. More seriously, Lemma 5's first bullet assumes a bound on the difference between two-step Euler composition (second step at time t+h') and a single 2h' jump, while Assumption 1's second bullet bounds a quantity where the second shortcut is evaluated at time t. These are not the same, so the \"by assumption\" in Lemma 5's proof does not land. This is very likely a typo—Algorithm 1 correctly uses s(a_{t+h}, t+h, h)—but the authors need to align the assumption with the algorithm (or prove a new bound) before Theorem 2 can be accepted as proven.\n\nMinor issues: Eqs. (11)-(13) use t where Algorithm 1 uses t+h; the code URL is given but no artifact in the PDF. These are fixable and do not affect the empirical claims. The parallel-scaling caveat is appropriately self-reported: the Q-function is not a guaranteed verifier, and I agree with the authors there.\n\nCitation pattern looks fine—Frans et al. for shortcuts, Park et al. for baselines, plus the relevant RL and flow literature. Overall, this is a solid empirical paper with a fixable theory gap. It deserves a serious referee, and I would send it to review.","headline":"Empirical win, theory gap: SORL's one-stage shortcut-model actor-critic is worth reviewing, but Theorem 2 needs a fix before it is cited.","tokens_in":29894,"tokens_out":3248,"would_cite":true,"duration_ms":30126,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SORL's one-stage shortcut-model objective keeps offline RL policies near the behavior policy and lets them scale at test time.","keywords":["offline reinforcement learning","shortcut models","flow matching","self-consistency","Wasserstein regularization","inference-time scaling","best-of-N sampling","OGBench"],"falsifier":"Take a task with a known multimodal action distribution in the dataset, train SORL, and estimate the empirical $W_2$ distance between actions generated at $M_{inf} = 1, 2, 4, 8$ steps and held-out dataset actions. The theorem predicts a uniform bound that shrinks as the discretization grid refines and the losses decrease, so observing $W_2$ increase with more inference steps, or exceeding the predicted bound given measured $\\epsilon_{FM}$ and $\\epsilon_{SC}$, would falsify the regularization claim.","tokens_in":28708,"feed_emoji":"🤖","tokens_out":11007,"duration_ms":105038,"temperature":0.7,"pith_summary":"Offline reinforcement learning needs policies expressive enough to model multimodal behavior data without the slow iterative sampling that makes diffusion policies hard to optimize. This paper introduces Scalable Offline Reinforcement Learning (SORL), which trains a shortcut model as the policy in a single stage, combining a Q-loss, a flow-matching loss, and a self-consistency loss. The central claim is that this objective regularizes the learned policy to the behavior policy in $W_2$ distance, so the policy does not drift far from the offline data while it maximizes return. Empirically, SORL reports the best average performance on 5 of 8 evaluated environment categories (40 tasks) and shows that performance improves when more inference steps or best-of-$N$ sampling with the learned $Q$-function is used at test time. The reason to care: a practitioner gets one network that can act with one-step speed or multi-step precision, with a formal guarantee tying the training objective to staying near the data distribution.","feed_headline":"SORL tops 5 of 8 offline RL suites with one-stage training","feed_subtitle":"Self-consistency lets one network generate actions in 1 to 8 steps; extra inference compute improves success.","key_machinery":"The load-bearing object is the shortcut model $s_\\theta(z_t, t, h \\mid x)$: a flow model conditioned on step size $h$ that predicts where the flow will be after one jump of size $h$. Self-consistency is enforced by the target $s_{\\text{target}} = (s_\\theta(a_t,t,h)+s_\\theta(a_{t+h},t,h))/2$, so the network learns that one $2h$ jump equals two $h$ jumps. The training objective is $L_\\pi = L_{QL} + L_{FM} + L_{SC}$, where $L_{FM}$ is flow matching onto the offline data at the smallest step size and $L_{SC}$ is the self-consistency error; backpropagation through time uses at most $M_{BTT}$ steps, while inference can use any number. Theorem 2 assembles these pieces into a Wasserstein bound, showing that the two losses act as a behavior-regularizer: the discretization error decays as $1/M$ and the self-consistency error grows only logarithmically in $M$, so low training losses keep $\\hat{p}(h)$ close to $p^\\star$ uniformly over step sizes.","core_discovery":"SORL's policy is a shortcut model $s_\\theta(z_t, t, h \\mid x)$ that predicts the normalized direction from a noised action $z_t$ toward the next point $z_{t+h}$, trained so that a jump of size $2h$ matches two jumps of size $h$. The paper's main theoretical result, Theorem 2, states that if the flow-matching and self-consistency losses are small at every discretization point and the model and drift are Lipschitz, then for every step size $h$, $W_2(\\hat{p}(h), p^\\star)$ is bounded by a sum of a discretization error, the flow-matching error $\\epsilon_{FM}$, and the self-consistency error $\\epsilon_{SC}\\log_2 M$. This is a Wasserstein regularization guarantee: minimizing the training objective keeps the action distribution of the induced policy close to the behavior distribution rather than merely matching a per-action likelihood. On the experimental side, the paper shows SORL achieves the best performance on 5 of 8 evaluated environment categories, with positive sequential scaling in the number of inference steps and positive parallel scaling via best-of-$N$ selection using the learned $Q$-function as a verifier.","pith_inferences":["Editorial inference: the proof structure suggests any policy class with a self-consistency-trained shortcut representation could carry a Wasserstein behavior-regularization guarantee, so the result may transfer beyond the specific actor-critic instantiation.","Editorial inference: the empirical best-of-$N$ gain rests on the learned $Q$-function ranking actions well; a natural testable extension is to add uncertainty penalties or ensembles to the verifier and check whether parallel scaling survives when $Q$ is miscalibrated.","Editorial inference: on tasks where the behavior data itself is weak, the theorem bounds proximity to behavior policy but not task success, so test-time scaling is likely to help most when the behavior policy is already competent.","Editorial inference: adaptive per-state choice of inference steps, for example using $Q$-value gradients or uncertainty to decide when to spend more compute, is a direct next step the authors mention as future work and the sequential-scaling curves make plausible."],"forward_implications":["A single SORL policy can be deployed with one-step inference for latency-critical control or with more steps when precision matters, without retraining.","Training-time compute can be traded for test-time compute: reducing backpropagation-through-time steps from 8 to 1 or 2 can be partially compensated by more inference steps and best-of-$N$ sampling, up to a saturation point around 4 steps.","The Wasserstein bound gives a distribution-shift mitigation guarantee: minimizing the flow-matching and self-consistency losses keeps the policy's action distribution close to the behavior policy for every step size $h$.","SORL avoids the two-stage teacher-student distillation used by one-step flow policies; the same network is trained for all inference budgets in one run.","Because the model generalizes to inference steps beyond those used in backpropagation, test-time scaling does not require retraining the actor."],"supporting_citations":[{"why":"It introduces shortcut models and the self-consistency objective that SORL adapts as its policy class.","marker":"[Frans et al., 2024]"},{"why":"It provides the flow-matching regression formulation used for the policy's flow-matching loss.","marker":"[Lipman et al., 2022]"},{"why":"It supplies the ODE marginal-distribution property that justifies sampling actions by Euler integration.","marker":"[Liu et al., 2022]"},{"why":"It defines the benchmark environments, datasets, and evaluation protocol used in the experiments.","marker":"[Park et al., 2024a]"},{"why":"It provides the FQL baseline, the tuned baseline numbers, and the evaluation scheme SORL compares against.","marker":"[Park et al., 2025]"},{"why":"It gives the Wasserstein convergence analysis for flow models that the paper extends with self-consistency.","marker":"[Roy et al., 2024]"},{"why":"It underpins the paper's caveat that a learned Q-function verifier offers no statistical guarantee for best-of-N sampling.","marker":"[Swamy et al., 2025]"},{"why":"It supplies the repeated-sampling test-time scaling idea behind SORL's parallel scaling.","marker":"[Brown et al., 2024]"}],"fun_headline_variants":["SORL tops 5 of 8 offline RL suites with shortcut models","Shortcut-model SORL wins 5/8 offline RL suites","One-stage shortcut models let SORL top 5 of 8 suites","SORL scales offline RL via shortcut models, wins 5/8"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The bound depends on Assumption 1, that the flow-matching and self-consistency losses are small at every grid point, while training only minimizes them in expectation; if either loss stays large at some time steps, the Wasserstein guarantee does not follow. The proof of Lemma 4 additionally assumes the drift is Lipschitz in time, an assumption not stated in Theorem 2.","fun_headline_variants_meta":{"raw":{"variants":["SORL tops 5 of 8 offline RL suites with shortcut models","Shortcut-model SORL wins 5/8 offline RL suites","One-stage shortcut models let SORL top 5 of 8 suites","SORL scales offline RL via shortcut models, wins 5/8"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000355,"raw_usage":{"total_tokens":1936,"prompt_tokens":961,"completion_tokens":975,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":893}},"tokens_in":577,"tokens_out":975,"duration_ms":9334,"temperature":1.0,"reasoning_tokens":893,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:58:43.560561+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a task with a known multimodal action distribution in the dataset, train SORL, and estimate the empirical $W_2$ distance between actions generated at $M_{inf} = 1, 2, 4, 8$ steps and held-out dataset actions. The theorem predicts a uniform bound that shrinks as the discretization grid refines and the losses decrease, so observing $W_2$ increase with more inference steps, or exceeding the predicted bound given measured $\\epsilon_{FM}$ and $\\epsilon_{SC}$, would falsify the regularization claim.","supporting_citations":[],"review_version":1}