{"id":"f6ce305a-6f63-4596-9f02-8c591c33c9fc","arxiv_id":"2501.14622","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A policy model trained jointly with JEPA-style latent observation-sequence prediction matches ACT baselines in Meta-World and improves a proprioceptive reconstruction probe, though the task-success gain is within error bars.","lead":"ACT-JEPA trains a robot policy to predict both future actions and future abstract observations, combining imitation learning with self-supervised world-model learning. In small-scale Meta-World tests it matches a strong behavior-cloning baseline (91.6% vs 91.1% success), but the reported representation-probe gains do not support the abstract's larger claims.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"H1 probe is circular: it decodes the exact modality the observation loss was trained to predict, so it cannot establish world-model understanding or the claimed 40% improvement.","rationale":"The reader's weakest-assumption analysis identifies the same load-bearing concern: the H1 probe is circular because the probed modality (proprioceptive states) is exactly the target of the JEPA prediction loss. I agree with that assessment. The paper's central claim of an improved world model rests almost entirely on this probe; the task success results in Table 2 show only parity with the strongest baseline (91.6% vs 91.1%), and the abstract's 'up to 40% improvement in world model understanding' is not actually present in the reported aggregate RMSE/ATE improvements (12.4% and 11.8%). The proposed concrete test would settle the circularity by probing a modality that the observation loss did not target. If the advantage disappears, then the paper has not demonstrated world-model understanding beyond its training objective. This reinforces the reader's conditional verdict rather than overturning it: the architecture and task-level results are plausible and worth conditional acceptance, but the headline world-model claim requires stronger evidence. I therefore recommend no change to the reader's verdict.","tokens_in":14293,"tokens_out":4091,"duration_ms":41040,"concrete_test":"Retrain the Section 4.3.1 probe to reconstruct future image embeddings (from a frozen pretrained ResNet) instead of proprioceptive state sequences, using the same frozen context encoders for ACT-JEPA and ACT. If the ACT-JEPA RMSE/ATE advantage over ACT disappears or reverses, the original probe is explained by the observation loss optimizing for the probed modality, not by superior world-model understanding.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The sole evidence for H1 and for the abstract's '40% improvement in world model understanding' is the probing experiment in Section 4.3.1. There, a decoder is trained to reconstruct proprioceptive state sequences from the frozen context encoder's output. However, Section 3.3's observation loss explicitly trains that same context encoder to predict abstract proprioceptive sequences via the predictor and target encoder. The probe therefore measures whether the encoder retains information about its own training target, not whether it understands environment dynamics or generalizes beyond the training distribution. A model optimized to predict proprioceptive states will naturally score well on a proprioceptive-state reconstruction probe, so the RMSE/ATE improvements over ACT may simply reflect the difference in training objectives rather than a superior world model. Consequently, the paper's headline claim that predicting abstract observation sequences improves world-model understanding is not supported by the presented experiment.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ACT-JEPA, a policy-learning architecture that combines imitation learning (action chunking, as in ACT) with a Joint-Embedding Predictive Architecture (JEPA). The model is trained end-to-end with two losses: an action-reconstruction loss and an observation-prediction loss in latent space, where the latter is intended to build a world model. The authors formulate three hypotheses: (H1) predicting abstract observation sequences improves representation quality and environment-dynamics understanding, (H2) such representations transfer to action-sequence prediction, and (H3) ACT-JEPA performs comparably to established supervised baselines. Experiments on Meta-World (15 tasks) compare ACT-JEPA against a regression-based transformer (RBC) and ACT, reporting a 12.4% RMSE improvement and 11.8% ATE improvement on a probing task, a 91.6% vs. 91.1% success rate difference, and a decreasing action-reconstruction loss in a pretraining/fine-tuning protocol. The authors conclude that all three hypotheses are supported and that ACT-JEPA offers an efficient policy-representation learning method.","tokens_in":14488,"tokens_out":3720,"duration_ms":33706,"significance":"If the central claims were fully supported, ACT-JEPA would be a valuable contribution: it cleanly integrates a self-supervised JEPA-style objective into an imitation-learning policy, proposes a concrete pretraining/fine-tuning protocol for representation transfer, and is evaluated against a strong baseline (ACT) in a low-data regime. The paper is clearly structured, the architecture is described in enough detail to be reproducible, and the authors state that code and data will be released. However, the significance is substantially undercut by the gap between the abstract's headline claims and the reported numbers, and by the circularity of the H1 probing evaluation. The H2 experiment is a promising idea but is currently reported only through a proxy training loss. The core architectural idea is worth pursuing, but the present evidence does not yet support the strong conclusions the paper draws.","major_comments":[{"comment":"The abstract claims 'up to 40% improvement in world model understanding and up to 10% higher task success rate,' but the full text reports a 12.4% RMSE improvement and an 11.8% ATE improvement in Section 5.1 (Table 1) and a 0.5 percentage-point success-rate difference (91.6% vs. 91.1% in Table 2), which is within the reported error bars (91.1 ± 1.4 vs. 91.6 ± 1.7). These are numerically inconsistent, and the phrase 'up to 40%' does not correspond to any number reported in the experimental results. The abstract should be revised to match the actual reported magnitudes, and the success-rate claim should be phrased as 'comparable' rather than 'up to 10% higher.' This is load-bearing because the paper's framing and the scientific record of its claims depend on the accuracy of these numbers.","section":"Abstract, Section 5.1, Section 5.3"},{"comment":"The probing experiment used to support H1 is circular for the claimed conclusion. The probe trains a decoder to reconstruct proprioceptive state sequences from the frozen context encoder, but the observation loss in Section 3.3 (L_obs) explicitly trains the same context encoder (via the predictor and target encoder) to predict abstract proprioceptive sequences. Thus the probe measures how well the representation retains information about its own training target, not whether the model has learned general environment dynamics or can generalize beyond the training distribution. The improvement over ACT is unsurprising, since ACT is not trained to predict proprioceptive states at all. To support H1, the authors should probe a modality or a prediction task that is not used in training (e.g., image-frame reconstruction or action-conditioned future observation prediction), or demonstrate generalization to a held-out task with different dynamics. As written, the claim that 'ACT-JEPA significantly outperforms the baselines' on 'understanding environment dynamics' is not supported by this experiment.","section":"Section 4.3.1, Section 3.3, Section 5.1"},{"comment":"The evidence for H2 (that predicting abstract observation sequences generalizes to action prediction) is currently only a decreasing action-reconstruction loss during fine-tuning. This is a weak proxy for actual policy quality or generalization, and the protocol alternates pretraining with one-epoch fine-tuning using a newly initialized decoder, so the trend could partly reflect optimization dynamics rather than representation quality. The paper does not report task success rates or any held-out action-prediction metric after fine-tuning. The authors should either report downstream task performance after a full fine-tuning stage or present other evidence that the pretrained representations benefit action prediction, such as a control experiment that trains the same decoder from a randomly initialized or ACT-pretrained encoder.","section":"Section 4.3.2, Section 5.2, Figure 2"},{"comment":"The abstract states that ACT-JEPA is evaluated 'in different environments and across multiple tasks,' but the experimental section only describes and reports results for Meta-World (Section 4.1, Table 2). No results are shown for any other environment. This is a mismatch between the claimed scope and the presented evidence. If other environments were evaluated, their results should be reported; otherwise, the abstract and introduction should be revised to state that the evaluation is limited to a single environment suite.","section":"Abstract, Section 4.1, Table 2"}],"minor_comments":[{"comment":"The authors' affiliations contain typos: 'Faculy of Techical Sciences' should be 'Faculty of Technical Sciences.'","section":"Title page"},{"comment":"In the contributions list, 'different decision-masking tasks' should be 'different decision-making tasks.'","section":"Section 1"},{"comment":"The sentence 'Then, each modality is encoded with a different modality-specific function' is followed by 'It receives an image and the proprioceptive state, along with a task a task label,' which contains a duplicated article and is awkwardly phrased.","section":"Section 3.2.1"},{"comment":"The notation for the losses is inconsistent: the observation loss is written as L_obs (possibly with subscripts) and the action loss as L_act, but the symbols are not defined precisely in one place, and the subscript formatting is inconsistent (e.g., 'L_obs' vs. 'L_obs'). A single, clearly defined notation would improve readability.","section":"Section 3.3"},{"comment":"The caption of Table 1 states 'We observe that ACT-JEPA significantly outperforms the baselines,' but no statistical significance test is reported. Given the small number of seeds (3 training seeds), the word 'significantly' should either be supported by an appropriate test or replaced with 'consistently outperforms.'","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The discrepancy between the abstract's headline numbers and the full-text results is a serious presentation issue that should be corrected before publication. In addition, the H1 probe, as designed, cannot distinguish between representation quality that arises from the world-model objective and representation quality that simply reflects the training target being used as the probe. The authors should be encouraged to redesign the probe to test a held-out modality or a true dynamics-prediction task. The H2 experiment is a good idea but needs to be evaluated with a downstream task metric. The paper's current form overstates its conclusions relative to the evidence, but the underlying approach is plausible and worth pursuing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The actual new idea is narrow but real: ACT-JEPA applies ACT's action-chunking trick to latent observation prediction as well, using a JEPA-style EMA target encoder and a joint loss, so the model predicts a chunk of future abstract observations alongside the action chunk. DynaMo and DINO-WM predict a single future latent frame; nobody, as far as the related work shows, has done chunked latent observation prediction for policy learning. Second, the paper's tables undercut its headline: the metadata abstract claims up to 40% world-model improvement and up to 10% higher success rate, but the body reports a 12.4% RMSE probe gain and a 0.5 percentage point success difference (ACT 91.1±1.4 vs ACT-JEPA 91.6±1.7), inside the error bars. The body's own abstract says 'on par,' which contradicts the metadata version. That is sloppy and it is the first thing a referee should fix.\n\nWhat is good: the architecture is clearly described and the choices are sensible—cross-attention predictor, EMA target encoder, an action decoder symmetric to the predictor, and ACT as the closest baseline. The authors are candid about the saturated benchmark, the lack of SSL baselines (code/compute constraints), and the small homogeneous dataset. The limitation section is honest, and the citation coverage of the relevant JEPA/IL literature is solid.\n\nThe soft spots. The stress-test on H1 mostly lands. The probe reconstructs proprioceptive state sequences, the exact modality the observation loss trained the representation to predict, so the gain over ACT tells you the self-supervised objective worked, not that the model understands environment dynamics in an independent sense. It is not fully circular—the probe decodes raw states rather than target embeddings, and as a test of whether the representation retains more information about its training target than ACT's, it is valid. But 'world model understanding' is overreach. Either probe a different modality or a horizon beyond the training chunk, or drop the world-model wording. The H2 transfer experiment has no from-scratch control, so the decreasing action loss could partly be extra optimizer steps; add a control curve. Missing SSL baselines and unreleased code/data are minor at this stage but should be fixed for a camera-ready.\n\nVerdict: the architectural idea deserves referee time, and the paper is coherent on its own terms. The central argument survives in the weakened form: ACT-JEPA is comparable to strong BC baselines, and the SSL objective demonstrably shapes the representation. What does not survive is the abstract. I would send it out, with a gentle instruction to recalibrate claims and strengthen H1/H2. The reader's conditional verdict is about right.","headline":"A genuinely new combination of action chunking and JEPA-style latent observation prediction, cleanly described and honestly limited, but the world-model claim rests on a partly circular probe and the abstract oversells the numbers.","tokens_in":15000,"tokens_out":6609,"would_cite":true,"duration_ms":56187,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding a self-supervised objective that predicts future observations in latent space to an action-chunking policy improves the policy's world model and keeps task success at or above supervised baselines.","keywords":["imitation learning","self-supervised learning","joint-embedding predictive architecture","world model","action chunking","latent representation learning","policy learning","robotic manipulation"],"falsifier":"On a task suite where a held-out sensor stream, such as depth images, is recorded alongside the trained proprioceptive states, train ACT-JEPA and an action-only baseline with identical data, then probe both frozen encoders to reconstruct the held-out stream. If the JEPA-trained encoder does not reconstruct the held-out stream substantially better, the claim that latent observation prediction improves world-model understanding fails.","tokens_in":14094,"feed_emoji":"🤖","tokens_out":7303,"duration_ms":78385,"temperature":0.7,"pith_summary":"ACT-JEPA is a policy-learning architecture that trains one model to predict both a chunk of future actions and a chunk of future observations, with the observation prediction carried out in an abstract latent space rather than in raw pixels or joint coordinates. The paper's central claim is that this self-supervised world-model objective sharpens the shared representation of the environment, so the same context encoding that drives actions also anticipates how the state evolves. On a 15-task robotic manipulation benchmark with only 40 demonstrations per task, ACT-JEPA reaches a 91.6% average success rate against 91.1% for the strongest supervised baseline, and its frozen representations reconstruct proprioceptive trajectories with about 12% lower error. If the claim holds, imitation learning could gain an internal model of environment dynamics from unlabeled observation data without sacrificing task performance. The paper also reports a two-stage experiment in which representations learned by predicting abstract observations transfer to action-sequence prediction.","feed_headline":"Latent world-model loss lifts policy success to 91.6%","feed_subtitle":"Jointly predicting actions and abstract observations beats action-only cloning on 15 manipulation tasks","key_machinery":"The load-bearing component is the joint-embedding predictive head: a predictor that turns the context encoder's output plus learnable mask tokens into a sequence of abstract future-observation embeddings, matched against targets from an exponentially moving average target encoder. This is JEPA, a Joint-Embedding Predictive Architecture, which predicts in latent space to discard irrelevant detail. The second load-bearing idea is chunking: predicting a whole sequence of n future states and n future actions rather than one step, which is what lets the same representation support both the action policy and the world model. The cross-attention conditioning keeps the predictor's cost linear in context length, which the paper cites as an efficiency advantage over self-attention over all inputs.","core_discovery":"The central result is that a policy trained to predict both actions and future abstract observations develops a better internal representation of environment dynamics than a policy trained on actions alone. In ACT-JEPA, the context encoder takes the current image, proprioceptive state, and a task label and outputs a single context token; the target encoder embeds a sequence of future proprioceptive states; the predictor, conditioned on the context through cross-attention, predicts those future embeddings from learnable mask tokens; and the action decoder maps the same context plus mask tokens to a chunk of future actions. The model is trained end-to-end by summing an L1 action loss and an L1 latent-observation loss, with the target encoder updated by exponential moving average of the context encoder. The paper's headline measurements are that the frozen context encoder supports lower reconstruction error than the baseline's encoder, with RMSE 3.696 versus 4.219 and ATE 6.521 versus 7.393, and that the full policy reaches 91.6% success, slightly above the strongest baseline at 91.1%. This is presented as evidence that jointly predicting actions and abstract observation sequences improves policy representation and world-model understanding while remaining competitive on the task.","pith_inferences":["The probe's target modality is the same proprioceptive states used in the training objective, so the reported representation gain may partly reflect memorization of the prediction target; a held-out modality would test whether the world model generalizes.","The architecture's cross-attention predictor scales linearly in context length, so a natural extension is to test whether longer prediction horizons and longer observation histories continue to improve the representation.","The paper's closing speculation suggests that abstract state representations could transfer across robot embodiments with similar limb configurations; that is testable by pretraining on one embodiment and fine-tuning on another.","In the current benchmark most tasks saturate near 100% success, so the practical advantage of the world-model objective is more likely to appear on harder or more diverse tasks than the ones used here."],"forward_implications":["If the central claim is right, policies trained with a joint action-plus-latent-observation objective should transfer to downstream tasks better than action-only behavior cloning, as the probing results indicate.","The pretraining experiment suggests that large unlabeled observation collections can build the policy's context encoder, with expert action labels needed only in a short fine-tuning phase.","The parity in success rate implies that adding a world-model objective does not force a trade-off against task performance, at least in the low-data regime tested.","Because the model predicts whole observation sequences rather than a single future frame, its world model should be richer than single-step dynamics models, which should matter more in complex, long-horizon tasks."],"supporting_citations":[{"why":"Supplies the joint-embedding predictive architecture: predict targets in latent space and update the target encoder by exponential moving average.","marker":"[12]"},{"why":"Supplies action chunking and the supervised action-chunking baseline that ACT-JEPA extends; also the strongest baseline in the comparison.","marker":"[4]"},{"why":"Closest prior abstract-space dynamics pretraining for control; ACT-JEPA contrasts by predicting whole observation sequences instead of one future frame.","marker":"[15]"},{"why":"Shows JEPA-style prediction for video and the probing methodology used to evaluate learned representations.","marker":"[13]"},{"why":"Supports the choice of a transformer action decoder when action data are unimodal, as in the test environments.","marker":"[42]"},{"why":"Provides the 15-task manipulation environment and the low-data regime in which all hypotheses are tested.","marker":"[45]"}],"fun_headline_variants":["Predicting latent futures boosts policy success to 91.6%","Joint action and latent observation prediction lifts task success","ACT-JEPA: shared latent world model boosts imitation learning","Latent world model prediction beats action-only cloning","40% better world model, 10% higher success via joint prediction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's case for an improved world model rests on a probe that reconstructs the same proprioceptive states the model was trained to predict, so the probe may reward memorizing the training target rather than genuine understanding of environment dynamics.","fun_headline_variants_meta":{"raw":{"variants":["Predicting latent futures boosts policy success to 91.6%","Joint action and latent observation prediction lifts task success","ACT-JEPA: shared latent world model boosts imitation learning","Latent world model prediction beats action-only cloning","40% better world model, 10% higher success via joint prediction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000358,"raw_usage":{"total_tokens":1976,"prompt_tokens":1020,"completion_tokens":956,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":636,"completion_tokens_details":{"reasoning_tokens":874}},"tokens_in":636,"tokens_out":956,"duration_ms":8984,"temperature":1.0,"reasoning_tokens":874,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:56:58.559223+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a task suite where a held-out sensor stream, such as depth images, is recorded alongside the trained proprioceptive states, train ACT-JEPA and an action-only baseline with identical data, then probe both frozen encoders to reconstruct the held-out stream. If the JEPA-trained encoder does not reconstruct the held-out stream substantially better, the claim that latent observation prediction improves world-model understanding fails.","supporting_citations":[{"cited_title":"_0 : A Vision-Language-Action Flow Model for General Robot Control , url =","cited_arxiv_id":null,"evidence_quote":"Supplies the joint-embedding predictive architecture: predict targets in latent space and update the target encoder by exponential moving average."},{"cited_title":"Abbeel and A","cited_arxiv_id":null,"evidence_quote":"Supplies action chunking and the supervised action-chunking baseline that ACT-JEPA extends; also the strongest baseline in the comparison."},{"cited_title":"DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Control , url =","cited_arxiv_id":null,"evidence_quote":"Shows JEPA-style prediction for video and the probing methodology used to evaluate learned representations."}],"review_version":1}