{"id":"6e7a6dec-8c56-4830-a0e8-323431464d41","arxiv_id":"2505.21182","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"ContraDICE learns policies that imitate expert behavior while repelling undesirable demonstrations via a difference-of-KL objective that is convex when the expert weight dominates.","lead":"This paper introduces ContraDICE, an offline imitation learning method that learns from both expert demonstrations and explicitly undesirable (bad) demonstrations. The method uses a difference-of-KL objective that stays convex under a stated condition, yielding a stable, non-adversarial training algorithm that outperforms several baselines on D4RL locomotion, Adroit, and FrankaKitchen benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's central claim depends on the unstated Lagrangian-duality step converting Eq. (2) to Eq. (3); the appendix never derives it, and a standard DICE dual of Eq. (2) has a different max/min order and exponential form. This step must be verified before the theory supports ContraDICE.","rationale":"The reader's weakest assumption is exactly the unstated Lagrangian-duality step, and that is the same concern I identify as most load-bearing. The empirical results are strong and the method may well work, but the paper's theoretical contribution is that Algorithm 1 provably optimizes the difference-of-KL objective in Eq. (1)/(2). That proof has a single missing link: the derivation of Eq. (3). Without it, the convexity result in Prop. 4.1 and the algebra in Prop. 4.2 only show that the visitation-distribution problem is convex in d for fixed pi; they do not by themselves show that the Q-learning loss in Eq. (3) or its lower bound in Eq. (6) solves the same problem. My reconstruction of the standard dual is not identical to Eq. (3) in form, so the gap is not merely expository: it may hide a real sign, normalization, or minimax-order error. There are other issues worth noting, such as the incorrect displayed expansion in the Prop. 4.1 proof (Eq. 9 drops the logarithms of dG and dB), but that typo is repairable and does not threaten convexity. The SafeDICE comparison is also not perfectly matched, but it is secondary to the theoretical claim. I therefore keep the reader's CONDITIONAL verdict: the paper should be accepted only if the duality derivation is supplied or verified, since the central claim currently rests on an unproven and non-obvious equivalence.","tokens_in":25121,"tokens_out":22907,"duration_ms":261431,"concrete_test":"Write out the Lagrangian for Eq. (2) with the Bellman-flow constraint, fix pi, compute inf_{d >= 0} [ (1-alpha) D_KL(d||dU) + E_d[Q - gamma P^pi Q - Psi] ], and compare the resulting dual with Eq. (3) (set beta = 0, alpha < 1). To settle it numerically: take a 2-state, 2-action MDP with gamma = 0.9, dU uniform over all (s,a), dG concentrated on the 'good' action, dB concentrated on the 'bad' action, and fixed p0; enumerate all occupancy measures satisfying the flow constraint, compute the exact optimum of Eq. (2), then solve the exact max_pi min_Q L(Q,pi) of Eq. (3) (with the softmax V omitted) by enumerating pi and solving the inner Q minimization. If the optimal policies differ (up to the sign convention that maps min to max), the duality step is invalid; if they coincide, the omitted derivation can be supplied and the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is the claimed equivalence between the constrained difference-of-KL program in Eq. (2) and the Q-learning objective in Eq. (3). The text says 'details of the derivation are given in the appendix', but no appendix section actually derives Eq. (3). Every downstream theoretical property -- convexity in Q, the lower bound in Prop. 4.3, the minimax swap in Prop. 4.4, and the QW-BC policy extraction in Prop. 5.1 -- is a property of Eq. (3)/(6), not of the original visitation-distribution objective. Reconstructing the standard DICE-style dual for fixed pi gives g_pi(Q) = -(1-gamma) E_{p0,pi}[Q] - (1-alpha) E_{dU}[ exp( (Psi - Q + gamma E_{T,pi}[Q])/(1-alpha) - 1 ) ], with the original value being min_pi max_Q g_pi(Q). Eq. (3) instead states max_pi min_Q L with L = (1-gamma)E[V] + (1-alpha)E_{dU}[ exp( (Psi - T^pi[Q])/(1-alpha) ) ]. These differ in max/min order, in the '-1' in the exponent, and in how the entropy term enters; they may coincide only after a non-obvious sign/normalization change that the paper does not supply. If this step is wrong, Algorithm 1 optimizes a heuristic surrogate rather than the stated objective, and the 'provably balances' claim collapses. The issue is addressable by supplying the full derivation or a counterexample, which is why the verdict stays CONDITIONAL, but it is the single most load-bearing gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ContraDICE, an offline imitation learning method that learns from both expert (good) and undesirable (bad) demonstrations. The central idea is to optimize a difference of KL divergences, f(d_pi) = D_KL(d_pi || d_G) - alpha D_KL(d_pi || d_B), over state-action occupancy measures. The authors claim that for alpha <= 1 this objective is convex in d_pi (Proposition 4.1), which enables a Lagrangian-duality reformulation as a Q-learning objective (Equation 3). A lower-bound surrogate is then introduced (Equation 6) that is non-adversarial and convex in Q, and a Q-weighted behavior-cloning policy extraction is proposed (Proposition 5.1). The method is evaluated on 18 MuJoCo, Adroit, and FrankaKitchen dataset combinations, reporting strong performance against baselines including SMODICE, ILID, ReCOIL, SafeDICE, and DWBC-GB, together with ablations on the number of bad trajectories, the trade-off parameter alpha, and the entropy parameter beta.","tokens_in":25502,"tokens_out":11750,"duration_ms":133598,"significance":"If the theoretical claims are fully substantiated, this is a useful contribution: it provides a principled, non-adversarial objective for offline imitation learning that explicitly repels undesirable behavior, while reducing to IQ-Learn as a special case when alpha = 0. The empirical study is extensive, covers diverse domains, includes ablations, and the authors provide source code. The convexity observation for the difference-of-KL objective is plausible and the Q-weighted BC extraction is a practical simplification. However, the manuscript's central theoretical grounding is currently incomplete: the key Lagrangian-duality step from the constrained program (2) to the Q-learning objective (3) is asserted but never derived, and the proof of Proposition 4.1 in the appendix contains a typographical error that drops logarithmic terms. These gaps must be fixed before the 'provably balances' claim is supported.","major_comments":[{"comment":"The paper states that 'details of the derivation are given in the appendix' for converting the constrained difference-of-KL program (2) into the Q-learning objective (3) via Lagrangian duality, but Appendix A does not contain this derivation; it only proves Propositions 4.1, 4.2, 4.3, 4.4, and 5.1. This step is load-bearing because all downstream claims—convexity in Q, the lower-bound surrogate in Proposition 4.3, the min-max swap in Proposition 4.4, and the QW-BC extraction in Proposition 5.1—concern the derived objective (3)/(6) rather than the original occupancy-measure objective (1). A standard DICE-style dual of (2) would have a different max/min order and a different exponential form, so the claimed equivalence is not obvious. Please provide the full derivation, including the treatment of the pi-dependent entropy term, any normalization constants, and the exact max-min order, or provide a counterexample showing the equivalence fails.","section":"Section 4.1, Eq. (3)"},{"comment":"The displayed expansion in the proof of Proposition 4.1 is incorrect as written: it writes d_pi(s,a)(alpha d_B(s,a) - d_G(s,a)) in the second term, dropping the logarithms from d_B and d_G. The correct expansion is (1-alpha) d_pi log d_pi + d_pi(alpha log d_B - log d_G). Although the corrected expansion still supports the convexity conclusion, the proof as printed is invalid and should be fixed.","section":"Appendix A, Eq. (9) (proof of Proposition 4.1)"}],"minor_comments":[{"comment":"The definition of the occupancy measure d_pi(s,a) writes the sum over t starting at t=1 and omits the initial state distribution p_0; the standard definition is (1-gamma) sum_{t=0}^infinity gamma^t P(s_t=s) pi(a|s). The constraint in Eq. (2) uses the correct form, so this is a presentation error.","section":"Section 3, Preliminaries"},{"comment":"Proposition 4.1 states convexity for alpha <= 1, but the objective in Eq. (3) is only valid for alpha < 1 because of the division by (1-alpha). The remark handles alpha = 1 separately, but the main text should state this restriction more prominently to avoid confusion.","section":"Section 4.1, Remark"},{"comment":"The claim that maximizing the lower bound eL 'promotes' the original objective L is informal; a pointwise lower bound does not guarantee that the maximizer of eL is close to a maximizer of L. If the paper intends a theoretical justification for the surrogate, a formal statement with a bound would be helpful; otherwise the wording should be softened.","section":"Section 4.2 / Algorithm 2"},{"comment":"The Q-update in Algorithm 2 minimizes eL(Q|V) plus a quadratic term (Q - gamma V)^2/2, but this regularizer is only described in the text as a chi^2 divergence regularizer; the connection would be clearer if the equation number for the regularized objective were given.","section":"Algorithm 2, line 17"}],"recommendation":"major_revision","confidential_remarks":"The missing Lagrangian-duality derivation is a fixable but essential gap: the paper's theoretical contribution is not verifiable until it is supplied. The proof typo in Proposition 4.1 is also easily corrected. I would not reject the paper, as the empirical results are strong and the core convexity observation is plausible, but the authors should be required to provide the full derivation or explicitly reposition the method as a heuristic surrogate. The report should also ask them to tighten the 'provably' language in the abstract and conclusion until the derivation is complete."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before you read it. First, the empirical story is strong: ContraDICE beats the stated baselines on 18 D4RL-style tasks, with ablations on bad-data size, good-data size, alpha, beta, and an honest comparison of its Q-weighted BC against advantage-weighted BC. Code is public. Second, the central theory as written does not close: the paper claims Eq. (2) is dualized into Eq. (3) via Lagrangian duality and points to an appendix, but the appendix never derives Eq. (3). I reconstructed the standard DICE-style dual and got a min-Q / max-pi objective with a different exponent and a '-1' inside; Eq. (3) is max-pi / min-Q with no '-1'. They might coincide after a change of variables, but the paper does not say what it is. That is a load-bearing gap, not cosmetic: every downstream property (convexity in Q, the lower-bound surrogate, the minimax swap, QW-BC) is a property of Eq. (3)/(6), not of the original visitation-distribution objective.\n\nThe convexity claim itself is plausible, but the appendix proof of Prop. 4.1 has a typo: Eq. (9) drops the logarithms and writes dpi(alpha dB - dG) where the correct expansion is dpi(alpha log dB - log dG). With the logs restored the argument is fine, so this is minor. Prop. 4.3's lower bound is just exp(t) >= t+1, and Prop. 5.1's QW-BC equivalence is a two-line calculation; both check out.\n\nI largely agree with the reader's take. The paper is honest about limitations: it states alpha<=1, assumes clean labels, and in Appendix D.2 admits that ILID scales better with more expert data. The SafeDICE comparison is slightly uncharitable because SafeDICE does not use the good dataset by design; not fatal. Hyperparameters alpha and beta are tuned per task, but the sensitivity plots show broad plateaus, so I would not call it curve fitting.\n\nWho is this for: anyone working on offline IL with negative or unsafe demonstrations. It is a useful empirical baseline even if the theory needs patching. I would bring it to a reading group precisely because the missing dual step is a good exercise. Recommendation: send it to serious peer review, but the referee should require either the full derivation of Eq. (3) from Eq. (2) or an explicit statement that Eq. (3) is a heuristic surrogate. As it stands, the paper should not be accepted without that clarification.","headline":"Useful empirical paper with a strong method and a real hole: the paper never shows the Lagrangian dual step that turns Eq. (2) into Eq. (3), and the standard DICE dual looks different.","tokens_in":26025,"tokens_out":3249,"would_cite":true,"duration_ms":38355,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proves that an imitation objective combining KL-matching to good data and KL-repulsion from bad data stays convex when the good term is weighted at least as heavily as the bad term, and it turns this into a practical…","keywords":["offline imitation learning","contrastive demonstrations","difference of KL divergences","convex optimization","occupancy measure","Q-learning","behavior cloning","safe imitation"],"falsifier":"Write out the Lagrangian dual of Equation (2) explicitly; alternatively, on a small tabular MDP compute the true minimizer of $f(d_\\pi)$ and compare it with the policy produced by minimizing the surrogate objective of Equation (6). If the two disagree on a case where $\\alpha \\le 1$, the claimed equivalence fails.","tokens_in":24940,"feed_emoji":"🤖","tokens_out":5567,"duration_ms":52531,"temperature":0.7,"pith_summary":"Offline imitation learning usually treats all demonstrations as positive examples, ignoring the signal in explicitly undesirable behavior. This paper proposes to learn from both expert and bad data by minimizing the KL divergence to the good policy while maximizing the KL divergence away from the bad policy, giving the objective $\\min_{d_\\pi} D_{\\mathrm{KL}}(d_\\pi \\| d_G) - \\alpha D_{\\mathrm{KL}}(d_\\pi \\| d_B)$. The central claim is that despite being a difference of convex functions, this objective is convex in the occupancy measure whenever $\\alpha \\le 1$, so it can be recast as a tractable, non-adversarial Q-learning problem. The resulting algorithm, ContraDICE, is reported to consistently outperform prior baselines across standard offline imitation benchmarks.","feed_headline":"Imitation objective stays convex when good demos outweigh bad demos","feed_subtitle":"A new offline algorithm, ContraDICE, beats prior imitation baselines across locomotion and manipulation benchmarks.","key_machinery":"The load-bearing object is the occupancy-measure objective $\\min_{d_\\pi} D_{\\mathrm{KL}}(d_\\pi \\| d_G) - \\alpha D_{\\mathrm{KL}}(d_\\pi \\| d_B)$, with $d_G$ and $d_B$ the visitation distributions of good and bad demonstrations. Its convexity for $\\alpha \\le 1$ allows the constrained problem to be dualized into a Q-learning form, and the identity $\\exp(t) \\ge t+1$ gives a linear lower bound that replaces the unstable exponential with a linear surrogate, keeping the final objective convex in $Q$ and concave in the policy $\\pi$.","core_discovery":"The paper's central discovery is that the natural 'imitate the good, avoid the bad' objective, written as a difference of KL divergences over state-action occupancy measures, remains convex as long as the good term is weighted at least as heavily as the bad term ($\\alpha \\le 1$). This convexity licenses a Lagrangian-duality reformulation into a soft Q-learning objective, and a linear lower bound on the exponential term turns that objective into a convex, non-adversarial surrogate in the Q-function space. The paper also introduces a Q-weighted behavior-cloning step, proven to recover the same optimal policy as advantage-weighted cloning, and reports that the full method outperforms existing offline imitation baselines on a wide range of benchmark dataset combinations.","pith_inferences":["The convexity result appears specific to KL divergence; the appendix shows the same difference-of-divergence objective fails to be convex for general $f$-divergences, so the framework may not transfer to other divergences without additional machinery.","If the duality derivation holds, the same recipe of pairing a convex KL-matching term with a weighted KL-avoidance term could be applied to other Q-learning-based imitation or offline reinforcement learning pipelines.","A natural testable extension is to replace the discriminator-based ratio estimates in the algorithm with more sample-efficient density-ratio estimators, which could matter more in low-data regimes.","The claimed 2–5 bad trajectories ≈ 1 good trajectory equivalence is empirical and task-dependent; checking it across more diverse environments would clarify when bad data is genuinely substitutable for expert data."],"forward_implications":["With $\\alpha \\le 1$, the training objective is convex, so no adversarial discriminator is needed and optimization is stable.","The surrogate lower bound preserves the original objective's optima in the idealized case where the soft Bellman residual vanishes, and it generalizes the IQ-Learn objective, which it recovers exactly when $\\alpha=0$ and $d_G=d_U$.","The proposed Q-weighted behavior cloning provably recovers the same optimal policy as advantage-weighted cloning while avoiding the bias introduced by a separately estimated value function.","Empirically, one good trajectory plus ten bad trajectories suffices for near-expert performance on the tested benchmarks, and the paper estimates that about 2–5 bad trajectories can substitute for one good trajectory.","The method is limited to $\\alpha \\le 1$; the authors explicitly note that relaxing this constraint remains future work and that noisy or weakly labeled demonstrations are not yet handled."],"supporting_citations":[{"why":"Provides the IQ-Learn objective that the surrogate generalizes; the paper shows its objective reduces to IQ-Learn when $\\alpha=0$ and $d_G=d_U$.","marker":"[8]"},{"why":"Introduces the DICE framework for policy optimization from arbitrary experience that the paper builds on for the occupancy-measure formulation.","marker":"[31]"},{"why":"Establishes off-policy distribution matching via stationary distribution correction, which motivates the dual KL framework used here.","marker":"[22]"},{"why":"Shows how convex KL objectives over occupancy measures yield tractable offline imitation learning from mixed-quality data.","marker":"[21]"},{"why":"SafeDICE is the key baseline designed to avoid undesirable demonstrations, which ContraDICE must outperform.","marker":"[17]"},{"why":"Supplies the Extreme-V log-sum-exp-free value update used in the practical algorithm.","marker":"[9]"},{"why":"Provides a state-of-the-art baseline and the dataset construction recipe used for the benchmarks.","marker":"[35]"}],"fun_headline_variants":["Convex imitation when good demos outweigh bad ones","ContraDICE: Imitate good, avoid bad, stay convex","Difference of KL divergences yields convex imitation","When good demos outweigh bad, imitation stays convex"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The Lagrangian-duality step that is supposed to convert the constrained convex program into the Q-learning objective is only referenced to the appendix, which does not actually derive it; if that step is invalid, the loss being optimized is not the stated difference-of-KL objective.","fun_headline_variants_meta":{"raw":{"variants":["Convex imitation when good demos outweigh bad ones","ContraDICE: Imitate good, avoid bad, stay convex","Difference of KL divergences yields convex imitation","When good demos outweigh bad, imitation stays convex"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000662,"raw_usage":{"total_tokens":2965,"prompt_tokens":828,"completion_tokens":2137,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":444,"completion_tokens_details":{"reasoning_tokens":2072}},"tokens_in":444,"tokens_out":2137,"duration_ms":17130,"temperature":1.0,"reasoning_tokens":2072,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:32:48.850073+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Write out the Lagrangian dual of Equation (2) explicitly; alternatively, on a small tabular MDP compute the true minimizer of $f(d_\\pi)$ and compare it with the policy produced by minimizing the surrogate objective of Equation (6). If the two disagree on a case where $\\alpha \\le 1$, the claimed equivalence fails.","supporting_citations":[{"cited_title":"Iq-learn: Inverse soft-q learning for imitation.Advances in Neural Information Processing Systems, 34:4028–4039, 2021","cited_arxiv_id":null,"evidence_quote":"Provides the IQ-Learn objective that the surrogate generalizes; the paper shows its objective reduces to IQ-Learn when $\\alpha=0$ and $d_G=d_U$."},{"cited_title":"Imitation learning via off-policy distribu- tion matching","cited_arxiv_id":null,"evidence_quote":"Establishes off-policy distribution matching via stationary distribution correction, which motivates the dual KL framework used here."},{"cited_title":"Demodice: Offline imitation learning with supplementary imperfect demonstrations","cited_arxiv_id":null,"evidence_quote":"Shows how convex KL objectives over occupancy measures yield tractable offline imitation learning from mixed-quality data."},{"cited_title":"Safedice: offline safe imitation learning with non-preferred demonstra- tions.Advances in Neural Information Processing Systems, 36, 2024","cited_arxiv_id":null,"evidence_quote":"SafeDICE is the key baseline designed to avoid undesirable demonstrations, which ContraDICE must outperform."},{"cited_title":"Dual rl: Unification and new methods for reinforcement and imitation learning","cited_arxiv_id":null,"evidence_quote":"Provides a state-of-the-art baseline and the dataset construction recipe used for the benchmarks."}],"review_version":1}