{"id":"c974cf40-6c5c-499d-950b-8ede02cb1977","arxiv_id":"2412.02931","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper proposes an off-policy adversarial IRL method on augmented delayed states and claims, via a Lipschitz bound and MuJoCo experiments, that this outperforms IRL on raw delayed observations.","lead":"This paper introduces IDRL, an inverse reinforcement learning framework that recovers rewards and policies from expert demonstrations corrupted by observation and action delays, using a state augmentation trick. It is worth reading because delayed feedback is common in robotics and teleoperation, and the paper claims both a theoretical and empirical advantage for augmented delayed observations over raw delayed ones.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proposition 5 is an absolute, symmetric upper bound on |V^{π∆} − V^{π}|, so it cannot establish the abstract's claim that augmented delayed observations outperform direct delayed observations.","rationale":"The reader's weakest assumption is exactly the gap I identify: the paper reads the symmetric upper bound of Proposition 5 as evidence that the augmented-state policy is closer to the expert's intent. I agree that this is the load-bearing weakness. The proposition is not a comparison theorem; it is a universal bound that any two bounded-reward policies satisfy (up to the Lipschitz reward term), and it is invariant under swapping the two policies. The theoretical superiority claim in the abstract therefore has no derivation. The experiments would not rescue the claim even if they were clean, because they change the policy optimizer along with the state representation; however, the primary defect is the missing logical link in the theory. I find no independent support such as machine-checked proofs or reproducible code that would offset this gap, so the reader's REJECT verdict is appropriate. My concrete test would settle the mathematical point by attempting the one-sided derivation; the symmetry of the bound indicates it cannot succeed without new assumptions.","tokens_in":18673,"tokens_out":7774,"duration_ms":78685,"concrete_test":"Attempt to derive a one-sided inequality V^{π∆}(x) ≥ V^{π}(x) from the assumptions of Section 4. Since Proposition 5 and Lemma 4 are proved entirely with absolute-value steps, an independent derivation will either fail or require an additional assumption (e.g., that R∆ is the true belief reward and π∆ is optimal in the belief MDP). As a sharpened check, compute both bounds with π and π∆ exchanged; if the resulting statements are identical, the theorem is symmetric and cannot order the two representations.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central theoretical assertion—that recovering expert policies from augmented delayed observations outperforms using direct delayed observations—is not supported by the stated mathematics. Section 4's Proposition 5 bounds the absolute value-function difference by (1−γ)^{-1}[Rmax + ΔL_R L_T]. This is an upper bound, not a comparison: it is invariant under exchanging π∆ and π, holds for any two policies regardless of which is better, and is proved by bounding a first term ‖V^{π∆} − E_{a∼π}Q^{π∆}‖ by Rmax/(1−γ). The reward bound in Lemma 4 is likewise a two-sided inequality on |E_b[R(s,a)] − R(s_{t−∆},a)| and contains no directional information. Consequently, the step from 'the difference is bounded' to 'augmented-state recovery is better' is a non sequitur. The empirical comparison in Section 6 is also confounded: IDRL differs from AIRL and DAC in both state representation and policy optimizer (Algorithm 2 vs PPO vs SAC), so the tabular gains do not isolate augmentation. The manuscript itself softens the conclusion to 'a theoretical insight' after Prop. 5, while the abstract claims a proof of outperformance. Separately, the off-policy derivation in Section 5 drops the importance-sampling ratio on empirical grounds, voiding the GAIL-style occupancy-match guarantee; even setting that aside, Prop. 5 does not deliver the advertised comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies inverse reinforcement learning from expert trajectories corrupted by observation/action delay. It proposes IDRL, which (i) augments delayed observations with the intervening action sequence to form an augmented state, (ii) trains an off-policy adversarial discriminator/reward model, and (iii) optimizes the policy with an auxiliary-delay policy optimizer adapted from Wu et al. (2024a). Section 4 claims a theoretical result that recovering policies from augmented delayed states outperforms recovering from direct delayed states; Section 6 reports MuJoCo experiments across delays 5/10/25 and 10/100/1000 demonstrations. The empirical tables show large gains for IDRL over BC, AIRL, and DAC.","tokens_in":18952,"tokens_out":8450,"duration_ms":75959,"significance":"The problem is genuinely under-studied: most delayed RL work assumes delay-free expert data, while this paper targets delays inside the expert trajectories, which is relevant for teleoperation and cyber-physical systems. The proposed pipeline is coherent and the experiments are extensive, covering five environments, three delay settings, three dataset sizes, and multiple baselines. The paper also reuses existing validated components (VDPO expert data, auxiliary-delay policy optimization), which is sensible engineering. If the central claim were proven and the comparisons cleanly controlled, this would be a useful contribution. As it stands, however, the advertised theory does not establish the superiority of augmented states, and the experiments change optimizer and representation simultaneously, so the incremental scientific claim is not yet supported.","major_comments":[{"comment":"The abstract states that the theoretical analysis shows that recovering expert policies from augmented delayed observations \"outperforms\" direct delayed observations, but Proposition 5 only upper-bounds the absolute difference, ∥V^{π∆}(x_t) − V^{π}(x_t)∥ ≤ (1−γ)^{-1}[Rmax + ΔL_R L_T]. This bound is invariant under exchanging the two policies and holds for any pair of policies whose values lie in the bounded range; it contains no directional information. The inference \"From Prop. 5 ... choose augmented state x instead of delayed observation state\" is therefore a non sequitur. A superiority claim would require a directional inequality (e.g., V^{π∆} ≥ V^π) or a lower bound on the gap, neither of which is derived.","section":"Section 4, Prop. 5; Abstract"},{"comment":"The proof does not align with the definitions in Eqs. (6)–(7). Equation (7) defines V^π using the policy π(·|s_{t−Δ}) and reward R(s_{t−Δ}, a_t), but the proof writes the middle term with a_t ∼ π(·|x_t) and compares E_{s_t∼b(·|x_t)}[R(s_t,a_t)] with an unindexed R(s_t,a_t); Lemma 4 bounds against R(s_{t−Δ},a_t). The first term V^{π∆}(x_t) − E_{a∼π(·|x_t)}[Q^{π∆}(x_t,a)] is an absolute policy-value gap that can be as large as 2Rmax/(1−γ), so the Rmax/(1−γ) coefficient also needs justification. As written, the proof does not establish the displayed bound.","section":"Appendix A, proof of Prop. 8"},{"comment":"The discriminator loss in Eq. (9) is presented as an off-policy version of the GAIL objective, but the importance-sampling ratio is explicitly dropped. The text says the occupancy-match proof \"should be similar\" to Ho and Ermon (2016), but no such proof is given, and the off-policy expectation over a replay buffer does not match the on-policy occupancy measure without the ratio. Since the paper's theoretical motivation for the learned reward rests on this occupancy match, the omission should be either rigorously justified or explicitly labeled as a heuristic; the current statement overclaims.","section":"Section 5, Eq. (9)"},{"comment":"The experimental comparison does not isolate the effect of the state representation. IDRL differs from AIRL and DAC in both the state representation (augmented x_t vs delayed s_{t−Δ}) and the policy optimizer (Algorithm 2 vs PPO vs SAC); additionally, in Hopper, HalfCheetah, Walker2d, and Ant, AIRL is trained for 10M steps while DAC and IDRL are trained for 1M. Therefore the large performance differences in Tables 1 and 2 could be due to the optimizer, the step budget, or the overall algorithm rather than to augmentation. Controlled ablations with the same optimizer and budget—e.g., AIRL/DAC on augmented states and IDRL on delayed-only states—are needed to support the stated conclusion.","section":"Section 6, Tables 1–2"}],"minor_comments":[{"comment":"The abstract's claim that the theory \"shows\" outperformance is stronger than Section 4's own wording \"From Prop. 5, we provide a theoretical insight\"; the two should be aligned, preferably by weakening the abstract.","section":"Abstract vs Section 4"},{"comment":"Definition 1 writes dR(R(s1,a1) − R(s2,a2)); since dR is a metric, the intended expression is dR(R(s1,a1), R(s2,a2)).","section":"Definition 1"},{"comment":"Lemma 3 states a \"LR-Time Lipschitz Continuous Dynamic\" but the relevant constant should be L_T; the same subscript confusion appears in the proof of Lemma 6.","section":"Lemma 3"},{"comment":"Algorithm 2 contains apparent typos: in line 3, Y2 samples from πψ(·|x_{t+n}) but subtracts log πφ(â|x_{t+1}); in line 8, the policy πψ is updated with critics Q^τ that were defined for the auxiliary policy πτ. These should be corrected or explicitly explained.","section":"Algorithm 2"},{"comment":"The Acknowledgments section contains the placeholder \"We thank a bunch of people.\"","section":"Acknowledgments"}],"recommendation":"reject","confidential_remarks":"The paper has a solid empirical scaffolding, but the main theorem advertised in the abstract is not proved, and the experiments are confounded. The stress-test concern is accurate and lands on the central claim. If the authors can supply a genuine directional bound (or withdraw the theoretical superiority claim) and run controlled ablations over representation and optimizer, a future submission may be competitive. Given the prominence of the theoretical claim in the abstract and conclusion, I do not see a minor revision fixing this within the current scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is the problem framing: IRL when expert demonstrations are subject to observation and action delays, with an augmented-state construction applied to both the expert and agent data. The MuJoCo benchmark is systematic (three delay levels, three demonstration counts, five environments), and the results are strikingly consistent—IDRL gets near-expert performance with 1000 trajectories while AIRL and DAC mostly collapse. That empirical pattern is worth having, and the paper is honest about leaning on VDPO for expert collection and policy optimization; citing your own prior work is fine when the work is doing real lifting.\n\nThe soft spot is the theoretical claim, and it is load-bearing. Proposition 5 bounds the absolute value-function difference by (1-γ)^{-1}[Rmax + Δ L_R L_T]. That bound is symmetric in the two policies and carries no directional information, so it cannot support the abstract's statement that recovering from augmented delayed observations \"outperforms\" direct delayed observations. Lemma 4 is the same story: a two-sided reward difference bound. The manuscript's own language softens to \"a theoretical insight\" after the proposition, but the abstract and conclusions claim a proof. That gap is not a minor presentational issue; it is the advertised contribution.\n\nThe experiments have a second confound: IDRL differs from AIRL and DAC in both the state representation and the policy optimizer (VDPO vs PPO vs SAC). The tabular gains could come from the optimizer alone. A clean ablation—same IRL wrapper and same policy optimizer on delayed versus augmented states—is missing, and without it the empirical case for the representation advantage is suggestive rather than demonstrated.\n\nThe off-policy derivation also drops the importance-sampling ratio on empirical grounds. That is a defensible practical choice, but it means the GAIL-style occupancy-match guarantee no longer holds, so the theory attached to the method is heuristic rather than proven.\n\nWho gets value from this paper? People working on delayed RL, imitation learning, and teleoperation control will find the benchmark useful and the problem statement valuable. It deserves peer review, but with a clear mandate: reframe the theory as motivation rather than proof, add an ablation that isolates the representation effect, and release code and hyperparameters. As it stands, the central claim is over-claimed, but the core idea is sensible and the empirical work is a solid starting point.","headline":"A useful empirical study of IRL under observation/action delay, but the advertised theoretical proof of augmented-state superiority is a symmetric bound that cannot deliver what the abstract claims.","tokens_in":19479,"tokens_out":2280,"would_cite":true,"duration_ms":25525,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that learning from augmented delayed states—delayed observations plus the actions taken during the delay—recovers expert intent better than learning from raw delayed observations, via an off-policy adversarial IRL…","keywords":["Inverse Reinforcement Learning","Imitation Learning","delayed observations","state augmentation","adversarial imitation learning","off-policy learning","cyber-physical systems","MuJoCo"],"falsifier":"On a small delayed MDP whose true expert reward is known, learn one policy on augmented states and one on delayed observations with the same optimizer; if a delay setting exists where the delayed-observation policy's return is closer to the expert's return than the augmented-state policy's, or where the Proposition 5 bound is tight in the direction favoring delayed observations, the superiority claim fails.","tokens_in":18462,"feed_emoji":"🤖","tokens_out":11991,"duration_ms":101308,"temperature":0.7,"pith_summary":"The paper tackles inverse reinforcement learning (IRL)—the problem of inferring a reward function from expert behavior—when the supplied demonstrations are corrupted by observation or action delay. Its central proposal is to learn rewards on an augmented state, namely the delayed observation together with the sequence of actions issued during the delay, rather than on the delayed observation alone, and it claims this recovers policies that match or approach expert return in delayed MuJoCo tasks. The framework is off-policy and adversarial: a discriminator trained over augmented state-action pairs supplies a reward signal, and an auxiliary delayed RL routine handles the policy optimization. The paper also gives a theoretical bound on the value-function gap between the augmented-state and delayed-observation representations, which it reads as evidence that augmented states are the right choice for reward shaping under delay.","feed_headline":"Augmented states beat delayed observations in inverse RL","feed_subtitle":"An off-policy adversarial IRL method recovers near-expert MuJoCo policies from delay-corrupted demonstrations.","key_machinery":"The load-bearing object is the augmented delayed state $x_t = (s_{t-\\Delta}, a_{t-\\Delta}, \\ldots, a_{t-1})$ together with the delayed MDP $M_\\Delta$ that it renders Markovian; stacking the stale observation with the actions issued during the delay turns a temporally misaligned expert trajectory into a well-posed imitation problem. On top of this representation, the method runs an adversarial reward learner with discriminator $D_\\theta(x,a) = \\frac{\\exp(R_\\theta(x,a))}{\\exp(R_\\theta(x,a)) + \\pi_\\Delta(a|x)}$, whose log-ratio gives the reward, and an off-policy auxiliary-delay policy optimizer. The theoretical support comes from Lipschitz continuity assumptions on the reward and the transition dynamics: with those, belief drift over the delay is bounded by $\\Delta L_T$, the reward misspecification by $\\Delta L_R L_T$, and the value-function gap between the two representations by $\\frac{1}{1-\\gamma}[R_{\\max} + \\Delta L_R L_T]$.","core_discovery":"The paper's central claim is that inverse reinforcement learning under observation/action delay should be performed on the augmented delayed state $x_t = (s_{t-\\Delta}, a_{t-\\Delta}, \\ldots, a_{t-1})$ rather than on the raw delayed observation $s_{t-\\Delta}$. Restoring the Markov property through this augmentation, the authors build an off-policy adversarial IRL method (IDRL) in which a discriminator $D_\\theta(x,a)$ distinguishes expert augmented state-action pairs from those of the learned policy, and the log-ratio of the discriminator supplies the reward. The same augmentation is used during policy optimization, helped by an auxiliary short-delay task. In MuJoCo benchmarks with delays of 5, 10, and 25 time steps and with 10 to 1000 expert trajectories, IDRL reports near-expert returns on Hopper, HalfCheetah, Walker2d, and Ant, while the AIRL, DAC, and behavior-cloning baselines largely fail to learn meaningful policies under delay. The paper's theoretical analysis bounds the difference between the value functions obtained from augmented versus delayed-observation reward shaping by $\\frac{1}{1-\\gamma}[R_{\\max} + \\Delta L_R L_T]$, and interprets this bound as justification for preferring augmented states.","pith_inferences":["The bound in Proposition 5 is symmetric in the two value functions, so it establishes only that the two representations are within a certain distance of each other; the conclusion that augmented states are closer to expert intent needs an additional directional argument that the paper does not give.","A clean ablation would hold the policy optimizer fixed and vary only the state representation, separating the effect of augmentation from the contribution of the auxiliary delayed policy routine.","The augmentation idea should extend to stochastic or time-varying delays by replacing the fixed action window with a belief over the true state, linking this work to belief-based delayed RL.","From a data-collection standpoint, the method implies a simple rule for practice: record the action sequence along with delayed observations, because that is what makes the augmented feature space available."],"forward_implications":["Inverse RL becomes usable on delay-corrupted expert data, so teleoperated robots and networked controllers can skip explicit time-alignment of their demonstrations.","With sufficient demonstrations and augmented-state reward shaping, near-expert policies are recovered at delays up to 25 steps, whereas delayed-observation baselines that use the same IRL machinery collapse.","The augmentation makes off-policy adversarial training practical, reducing the number of environment interactions needed for IRL under delay.","The theoretical bound implies the advantage of augmentation grows with delay and with the roughness of the reward and dynamics, since the gap scales as $\\Delta L_R L_T$."],"supporting_citations":[{"why":"introduces the augmented-state construction that converts a delayed control problem into a Markov decision process; the delayed MDP formulation borrows this representation.","marker":"(Altman and Nain, 1992)"},{"why":"formally defines MDPs with delays and asynchronous costs, grounding the transition kernel $T_\\Delta$ used in the delayed setting.","marker":"(Katsikopoulos and Engelbrecht, 2003a)"},{"why":"supplies the Lipschitz-continuous reward definition used as the main regularity assumption in Lemmas 3-4 and Proposition 5.","marker":"(Rachelson and Lagoudakis, 2010)"},{"why":"supplies the time-Lipschitz dynamic assumption used to bound belief drift over the delay by $\\Delta L_T$.","marker":"(Metelli et al., 2020)"},{"why":"provides the adversarial imitation formulation whose occupancy-matching proof the paper adapts to augmented states for reward learning.","marker":"(Ho and Ermon, 2016)"},{"why":"defines the adversarial IRL baseline (AIRL) that the paper extends and compares against.","marker":"(Fu et al., 2017)"},{"why":"introduces the off-policy discriminator-actor-critic baseline and the off-policy adversarial training scheme the method builds on.","marker":"(Kostrikov et al., 2018)"},{"why":"establishes the maximum-entropy IRL objective of Equation (2) that frames the reward inference problem.","marker":"(Ziebart et al., 2008, 2010)"},{"why":"provides the auxiliary short-delay policy optimization algorithm used to train the delayed policy efficiently in Algorithm 2.","marker":"(Wu et al., 2024a)"},{"why":"supplies the delayed expert trajectories (trained with variational delayed policy optimization) used as demonstrations in the MuJoCo experiments.","marker":"(Wu et al., 2024b)"}],"fun_headline_variants":["Augmented states beat delayed observations in inverse RL","IRL recovers expert policies from delay-corrupted data using augmented states","Inverse RL with augmented states: better than raw delayed observations","Delayed rewards? Augment state with actions to fix inverse RL","IDRL: Augmenting delayed states improves inverse reinforcement learning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument that augmented states are better rests on reading a symmetric upper bound on the absolute value-function gap as evidence that the augmented-state policy is closer to the expert's intent; the bound itself does not say which policy has higher value.","fun_headline_variants_meta":{"raw":{"variants":["Augmented states beat delayed observations in inverse RL","IRL recovers expert policies from delay-corrupted data using augmented states","Inverse RL with augmented states: better than raw delayed observations","Delayed rewards? Augment state with actions to fix inverse RL","IDRL: Augmenting delayed states improves inverse reinforcement learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000213,"raw_usage":{"total_tokens":1392,"prompt_tokens":889,"completion_tokens":503,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":427}},"tokens_in":505,"tokens_out":503,"duration_ms":5159,"temperature":1.0,"reasoning_tokens":427,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:56:26.852106+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a small delayed MDP whose true expert reward is known, learn one policy on augmented states and one on delayed observations with the same optimizer; if a delay setting exists where the delayed-observation policy's return is closer to the expert's return than the augmented-state policy's, or where the Proposition 5 bound is tight in the direction favoring delayed observations, the superiority claim fails.","supporting_citations":[],"review_version":1}