{"id":"20474684-0c83-4a45-8539-4ceb1db43c9b","arxiv_id":"2501.09327","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A variational autoencoder over skill sequences produces label-free trajectory embeddings that separate and imitate policies of different ability levels in MuJoCo control tasks.","lead":"This paper introduces a method that learns a compact numeric summary, called a trajectory embedding, of a sequence of states and actions without needing reward labels. The embedding can then be used to imitate different skill levels, classify demonstrations, and predict returns, with tests on simulated walking and running tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'without reward labels' claim is compromised: Appendix D.1 early-stops the LOVE skill extractor using ground-truth ability clusters derived from returns.","rationale":"The reader's weakest_assumption focuses on the generative model's assumption that a single latent e explains all trajectory variation. That is a valid concern about generalization, but it is less directly tied to the central claim than the label leakage in Appendix D.1. The abstract's 'without the need for reward labels' is a distinctive selling point; if the skill extractor is early-stopped using ability clusters defined by returns, the method at least partially relies on reward information. This is a concrete, checkable flaw, not a matter of distribution shift. The reader's rationale mentioned this issue, so I partially agree. The regression results are marginal but do not threaten the main claim; the classification results are 100%, but VTE-MLP also achieves 100% without the skill extractor, so the skill extractor is not necessary for classification—it is the imitation task where VTE shines. However, the label leakage affects the entire pipeline. I recommend keeping the CONDITIONAL verdict: the concern is addressable by retraining without label-based early stopping and reporting those results. If the authors can show that the method works without this supervision, the paper is acceptable; otherwise, the central claim needs to be revised. Thus UNCHANGED relative to the reader's CONDITIONAL.","tokens_in":18529,"tokens_out":8748,"duration_ms":128288,"concrete_test":"Retrain the full VTE pipeline with the LOVE skill extractor trained for a fixed 200 epochs (or early-stopped on a label-free held-out reconstruction loss), without any access to ground-truth ability/return labels for model selection. Then rerun the classification (Table 3) and conditional imitation (Table 2) evaluations. If classification accuracy drops below 90% or the imitation relative L2 error worsens substantially (e.g., doubling) on any environment, the 'no reward labels' claim is not supported; if performance is unchanged, the early stopping is benign and the claim stands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that VTE 'operates without the need for reward labels' (abstract), which is the primary differentiator from baselines like GCPC. However, Appendix D.1 states that the LOVE skill extractor is early-stopped 'once the clustering error falls below a sufficiently low threshold,' where the clustering error is computed via the Hungarian algorithm between 'the actual and predicted clusterings.' In the experimental protocol (Section 5.1), the 'actual' clustering is the ground-truth ability level (low/medium/expert), and ability level is defined by the average return of the policy that generated the trajectories. Thus the skill extractor SE-Logit, which provides the sole input to the VTE encoder, is selected using exactly the return information the method claims to avoid. This is not merely hyperparameter tuning; the model checkpoint is chosen on the basis of its ability to separate trajectories by return. Since the VAE encoder later processes the logits of this extractor, the unsupervised nature of the entire pipeline is unverified. The paper does not report results for a purely label-free version of the skill extractor, so we cannot tell whether the strong classification and imitation results would survive without this early stopping. This makes the headline claim vulnerable: if the early-stopping labels are necessary for good performance, the method is not label-free; if they are unnecessary, the paper should show that by removing them.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Variational Trajectory Encoding (VTE), a two-stage method for learning unsupervised trajectory embeddings. In the first stage, a LOVE-based skill extractor (SE-Logit) produces per-timestep skill and boundary logits; in the second stage, these logits are processed by a transformer encoder and a VAE-style decoder that reconstructs actions conditioned on the latent ability vector. The authors claim the method operates without reward or goal labels and demonstrate its use for imitation, classification, clustering, and regression on MuJoCo environments with three ability levels (low, medium, expert). The main empirical contributions are lower imitation error than GCPC and GCPC-NR baselines, 100% classification accuracy, and qualitative evidence of disentangled embedding dimensions.","tokens_in":18811,"tokens_out":3384,"duration_ms":30850,"significance":"If the label-free claim holds, the paper makes a useful contribution: a single unsupervised encoder that supports conditional imitation from mixed-expertise data and replaces reward-dependent trajectory encoders such as GCPC. The central VAE loss is a standard ELBO and does not itself include return or ability labels, so the core derivation is not circular. The paper also provides code, which supports reproducibility. However, the headline claim that the method operates 'without reward labels' is compromised by the early-stopping procedure for the skill extractor described in Appendix D.1, which uses ability labels derived from returns. This issue is load-bearing because the skill extractor provides the sole input to the VTE encoder. The paper's significance therefore depends on whether the method can be made genuinely label-free or whether the claim is appropriately revised.","major_comments":[{"comment":"The claim that VTE operates without reward labels is contradicted by the early-stopping criterion for the LOVE skill extractor. Appendix D.1 states that the model is early-stopped 'once the clustering error falls below a sufficiently low threshold,' where the error is computed via the Hungarian algorithm between 'the actual and predicted clusterings.' In Section 5.1, the 'actual' clustering is the ground-truth ability level, defined by the average return of the generating policy. Thus the skill extractor—whose logits are the sole input to the VTE encoder—is selected on the basis of return-derived labels. This directly conflicts with the abstract's 'without reward labels' and with Section 4.3.2, which states that ability-level and return labels are 'inaccessible to our trajectory encoding algorithm.' The authors must either re-run the experiments with a strictly label-free early-stopping rule (e.g., based on unsupervised clustering stability or reconstruction loss) and report whether the results hold, or explicitly acknowledge that the skill extractor is supervised by ability labels and reformulate the claim accordingly.","section":"Appendix D.1"},{"comment":"The generative model in Eq. (1) assumes every trajectory is produced by a single policy conditional on one unobserved ability variable e, and that e is the only source of trajectory-level variation. The experiments use exactly one checkpoint per ability level per environment (three checkpoints total per environment), so trajectories at the same ability level come from a single policy and contain no other sources of variation. The paper therefore does not test whether the embedding can handle multiple policies at the same ability level, stochastic skill switching, or other trajectory-level variability. The abstract and introduction claim the method captures 'diverse policies' and 'multi-modal behaviors,' which is stronger than what the experimental design can validate. I recommend either adding experiments with multiple checkpoints per ability level or tempering the claims about diversity.","section":"Section 5.1"},{"comment":"The regression results do not support the claim of strong representational power for regression. The relative regression errors are nearly identical across methods: e.g., Hopper 3.1 (VTE) vs 3.8 (GCPC-NR), Walker2D 0.7 vs 0.7, Half Cheetah 0.8 vs 0.9. With such small differences, the paper provides no evidence that VTE embeddings improve over the baseline in the regression task. Since regression is one of the four downstream tasks listed in the contributions, the authors should either design a more challenging regression evaluation or soften the claim to reflect the actual results.","section":"Section 5.4"}],"minor_comments":[{"comment":"The notation 'z_{1:T}, m_{1:T} = SE-Logit(x_{1:T}, a_{1:T})' is informal; it should be clarified that the output is a sequence of logit vectors, not sampled variables.","section":"Section 4.2"},{"comment":"The t-SNE plots are qualitative. Please report a quantitative clustering metric, such as adjusted Rand index or normalized mutual information, to support the claim that the embeddings are well-structured.","section":"Section 5.2"},{"comment":"There is a typo: 'init=ramdom' should be 'init=random'.","section":"Appendix D.2"},{"comment":"The Wasserstein distance calculation uses only 10 trajectory embeddings per policy to form an empirical distribution. Please justify this small sample size and report the variance of the distance estimates across multiple subsamples.","section":"Section 5.5"},{"comment":"The caption says 'relative L2 norm loss' but the precise formula (e.g., average over trajectories of ||predicted_return - target_return|| / target_return) is not given. Please add a definition.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a worthwhile problem, and the imitation results are strong if the label-free claim holds. The early-stopping issue in Appendix D.1 is the central obstacle: it means the method, as currently evaluated, is not fully label-free. This is fixable by re-running the skill extractor without the ability-labeled early stopping, or by honestly revising the claim. If the authors can show that the main results survive a label-free protocol, the paper would be a solid accept; if not, the claim would need substantial revision and the experimental comparison would need reinterpretation. I recommend asking for the label-free re-run as a condition for revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"One thing to know: the 'without reward labels' claim has a leak. Appendix D.1 says the LOVE skill extractor is early-stopped once the clustering error between 'actual and predicted clusterings' falls below a threshold, and in the experimental setup the actual clustering is the ground-truth ability level, which is defined by average return. So the model that feeds the VTE is selected using exactly the return information the paper says it avoids. That doesn't kill the whole paper, but it means the headline claim is not established as stated.\n\nWhat's actually new: the combination of LOVE skill logits with a transformer-VAE for trajectory-level embeddings is a reasonable new design, and it is distinct from GCPC (which needs rewards) and skill/option methods (which operate on sub-trajectories). The empirical story on MuJoCo is mostly solid: VTE matches or beats the reward-conditioned Known-Abl baseline on Hopper and Half-Cheetah imitation, beats GCPC across the board, and classification is at 100% on all three environments. The ablation against mean pooling in Table 5 shows the naive skill average collapses the ability signal, which makes the case for the transformer-VAE part. The VTE loss itself is a standard VAE objective with no label term, so the core derivation isn't circular.\n\nSoft spots: the early-stopping leakage is the real one, and it's proportionate to say it's load-bearing for the central claim. The paper should either remove the ability labels from early stopping or show that results survive without them. Also, the generative model in Eq. (1) assumes every trajectory is generated by a policy conditioned on a single unobserved ability variable, and the experiments only use three hand-picked checkpoints per environment; that's a thin test of the assumption. Regression results are marginal (VTE is not always the best), and the disentanglement evidence is qualitative. Those are minor.\n\nWho this is for: people working on trajectory representations for imitation and offline RL. It deserves a serious referee, but the referee should be asked to check the label-free claim carefully. My recommendation: engage with it, ask for a revised version that either fixes the early stopping or rephrases the claim.","headline":"Label-free claim leaks through early stopping: otherwise a solid new architecture for trajectory embeddings.","tokens_in":19322,"tokens_out":2404,"would_cite":true,"duration_ms":23187,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A two-stage variational encoder maps trajectories to latent ability vectors that, without reward labels, support imitation, classification, clustering, and regression.","keywords":["trajectory embeddings","unsupervised representation learning","imitation learning","variational autoencoder","skill extraction","offline reinforcement learning","sequential decision making","ability-conditioned policies"],"falsifier":"Train two policies in the same environment with equal expected returns but visibly different gaits, then train VTE on trajectories from both. If the two policy types collapse into overlapping embeddings and a classifier cannot separate them, while the conditional policy reproduces only one mode per embedding, the single-ability assumption is falsified. A quantitative check: compute the decoder's average per-step action log-likelihood on held-out trajectories from an intermediate checkpoint not included in training; if it degrades sharply relative to included checkpoints, the embedding overfits discrete ability levels rather than learning a continuous ability axis.","tokens_in":18345,"feed_emoji":"🤖","tokens_out":8220,"duration_ms":79988,"temperature":0.7,"pith_summary":"This paper proposes Variational Trajectory Encoding (VTE), a two-stage unsupervised method that maps a state-action trajectory to a low-dimensional 'ability' vector without reward or goal labels. The authors' central claim is that this vector captures the unobserved ability level of the policy that generated the trajectory, so conditioning a policy on the embedding recovers diverse behaviors from mixed-expertise datasets, while the same vector supports trajectory classification, return regression, and clustering. The motivation is that existing trajectory encoders for sequential decision making either require reward signals, depend on task-specific labels, or lose action information, which limits them in settings such as autonomous driving and healthcare. If the claim holds, a single reward-free representation would give imitation learning a mechanism for reproducing behavior at any observed ability level and would make trajectory embedding useful beyond expert-only benchmarks.","feed_headline":"Latent 'ability' vector reproduces policies with no reward labels","feed_subtitle":"One unsupervised encoder feeds imitation, classification and regression on mixed-expertise robot trajectories.","key_machinery":"The load-bearing object is the latent ability vector $e$ from the generative model, tied to trajectory reconstruction through the variational ELBO. The encoder side of the VAE is built from skill logits extracted by a hierarchical state-space model, the LOVE compression method, passed through MLPs and a shallow transformer so that the ordering of skills is respected; the decoder side is $p_\\theta(a_t|x_t,e)$, making trajectory reconstruction equivalent to behavioral cloning. Proposition 1 is the identity that carries the argument: maximizing $\\log p(\\tau|e)$ is the same as maximizing $\\sum_t \\log p(a_t|x_t,e)$, so the reconstruction term needs no reward model and no dynamics model, only an action-prediction objective.","core_discovery":"The paper's claim is that every trajectory can be encoded as a latent ability variable $e$, following the generative model $x_0 \\sim p_0$, $e \\sim p(e)$, $a_t \\sim \\pi(a_t|x_t,e)$, $x_{t+1} \\sim P(x_{t+1}|a_t,x_t)$, and that VTE learns the approximate posterior $q_\\phi(e|\\tau)$ through a VAE-style objective. The encoder first runs a skill extractor obtained from the LOVE compression method to get per-step skill logits $z_t$ and boundary logits $m_t$, maps these through MLPs, processes the time-indexed sequence with a shallow transformer, and outputs the mean and variance of a Gaussian posterior; the decoder is a neural policy $p_\\theta(a_t|x_t,e)$ trained with a conditional behavioral-cloning loss. The authors justify this by Proposition 1: maximizing the trajectory likelihood $\\log p_\\theta(\\tau|e)$ reduces to summing the per-step log-likelihoods $\\sum_t \\log p_\\theta(a_t|x_t,e)$. With this embedding, conditional IQ-Learn imitates policies at different ability levels, and simple MLP heads perform ability classification and return regression. Empirically on Hopper, Walker2D, and Half-Cheetah, VTE separates trajectories by ability in the latent space, reproduces low, medium, and expert returns more accurately than reward-based and reward-free baselines, achieves perfect classification accuracy, and shows that perturbing single embedding dimensions changes gait and posture in interpretable, opposite ways.","pith_inferences":["Going beyond the paper: if the single-ability generative model is correct, the embedding should also separate trajectories generated by two different policies that happen to have the same return; a test on same-return, different-style policies would confirm or refute whether the latent axis is truly 'ability' rather than 'behavior style'.","Going beyond the paper: the behavioral effect of perturbing embedding dimensions suggests vector arithmetic in the latent space, such as $e_{\\text{expert}} - e_{\\text{medium}} + e_{\\text{low}}$, might transfer or interpolate abilities; the paper does not test this, but its disentanglement results make it a natural next probe.","Going beyond the paper: because the decoder is only an action predictor, VTE should in principle transfer to control settings where rewards are unavailable but human demonstrations of varying competence exist, such as driving or clinical decision logs; that claim is untested outside MuJoCo.","Going beyond the paper: the current evaluation uses three hand-picked checkpoints per environment, so an extension sampling many intermediate checkpoints would show whether the embedding is a continuous interpolation axis or merely separates a few discrete ability clusters."],"forward_implications":["Conditioned on a VTE embedding, IQ-Learn can imitate trajectories from a mixture of low, medium, and expert-level policies, reaching returns close to the dataset values in the tested MuJoCo environments.","The same embedding is enough for downstream heads to classify a trajectory's ability level with perfect accuracy and to predict its return with relative error near one percent in the tested environments.","Individual dimensions of the embedding are behaviorally meaningful: increasing or decreasing one dimension produces opposite, interpretable changes in posture and gait, so the latent space can support coarse behavioral control without reward labels.","The ordering of skills within a trajectory matters: averaging skill logits over time collapses the ability distinctions, while the transformer-based VTE preserves them.","The embedding space separates policies of different ability levels by Wasserstein distance, while keeping trajectories from the same ability level close together."],"supporting_citations":[{"why":"Supplies the LOVE skill extractor that turns each trajectory into skill and boundary logits, the input representation for the VTE encoder.","marker":"[29]"},{"why":"Provides the VAE objective and reparameterization trick used to train the trajectory encoder and decoder.","marker":"[31]"},{"why":"Supplies IQ-Learn, which the paper adapts into a conditional version to imitate trajectories conditioned on the learned ability embedding.","marker":"[16]"},{"why":"The closest baseline, GCPC, whose reward-dependent trajectory encoding the paper adapts into a no-reward variant for comparison.","marker":"[51]"},{"why":"The soft actor-critic update used by the conditional IQ-Learn actor within the imitation pipeline.","marker":"[21]"},{"why":"The behavioral cloning insight behind Proposition 1, which reduces trajectory reconstruction to per-step action prediction.","marker":"[39]"},{"why":"Supplies the MuJoCo environments used to generate the low, medium, and expert ability trajectory datasets and to evaluate downstream tasks.","marker":"[44]"}],"fun_headline_variants":["Unsupervised trajectory embeddings master imitation, classification, regression","Reward-free latent abilities drive policies and classification","Single embedding space decodes skills, returns, decisions","VAE-style trajectory encoder separates ability without rewards","Latent ability vector: one encoder for imitation, classification, regression"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes the generative model of Eq. (1): each trajectory comes from a policy conditioned on a single unobserved ability variable $e$, and $e$ is the only source of trajectory-level behavioral variation; if real data contain other sources of variation, the embedding cannot faithfully recover the full behavior distribution.","fun_headline_variants_meta":{"raw":{"variants":["Unsupervised trajectory embeddings master imitation, classification, regression","Reward-free latent abilities drive policies and classification","Single embedding space decodes skills, returns, decisions","VAE-style trajectory encoder separates ability without rewards","Latent ability vector: one encoder for imitation, classification, regression"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001218,"raw_usage":{"total_tokens":5095,"prompt_tokens":1113,"completion_tokens":3982,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":729,"completion_tokens_details":{"reasoning_tokens":3904}},"tokens_in":729,"tokens_out":3982,"duration_ms":30383,"temperature":1.0,"reasoning_tokens":3904,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:06:15.442514+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train two policies in the same environment with equal expected returns but visibly different gaits, then train VTE on trajectories from both. If the two policy types collapse into overlapping embeddings and a classifier cannot separate them, while the conditional policy reproduces only one mode per embedding, the single-ability assumption is falsified. A quantitative check: compute the decoder's average per-step action log-likelihood on held-out trajectories from an intermediate checkpoint not included in training; if it degrades sharply relative to included checkpoints, the embedding overfits discrete ability levels rather than learning a continuous ability axis.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the LOVE skill extractor that turns each trajectory into skill and boundary logits, the input representation for the VTE encoder."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies IQ-Learn, which the paper adapts into a conditional version to imitate trajectories conditioned on the learned ability embedding."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The closest baseline, GCPC, whose reward-dependent trajectory encoding the paper adapts into a no-reward variant for comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The behavioral cloning insight behind Proposition 1, which reduces trajectory reconstruction to per-step action prediction."}],"review_version":1}