{"id":"3a6545ba-a060-4b74-9dcc-0797e164934b","arxiv_id":"2508.06335","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"ViPro-2 learns to estimate object positions from video observations without state supervision, matching supervised performance on Orbits-2D and introducing a 3D variant with depth-based initialization.","lead":"ViPro-2 improves a video prediction model so it can estimate object states from observed frames instead of relying on ground-truth initial states, achieving better predictions on a 2D dynamic dataset. It also extends the benchmark to 3D using depth maps, though 3D state estimation remains noisy.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Robustness to noisy initial states—the paper's stated motivation—is never tested; experiments use clean screen coordinates, not noisy state inputs, so the shortcut-fix claim is unverified.","rationale":"The reader correctly identified that the 'unsupervised' claim is conditioned on privileged initialization (screen coordinates and known dynamics). My concern builds on this: even within that conditioned setup, the paper does not test the robustness scenario it motivates. The abstract states that ViPro fails 'if previous states are noisy,' and the introduction highlights 'inaccuracies in the provided initial state.' Yet the experiments replace the full ground truth state with clean screen coordinates, which are not noisy—they are simply a less complete but still accurate representation. This is a meaningful distinction: ViPro might fail with screen coordinates because they require an extra transformation, not because they are noisy. ViPro-2 might succeed because it learns that transformation, not because it corrects errors. Without injecting noise, the claim that the model 'estimates states' rather than 'passes through clean initial conditions' is not proven. This is exactly the kind of untested assumption that a conditional verdict should flag. The proposed experiment is concrete and would directly test whether the model lives up to its stated motivation. If noise robustness fails, the paper's central claim is significantly overstated; if it passes, the claim is supported. Thus the reader's conditional verdict remains appropriate, and I do not recommend changing it.","tokens_in":10744,"tokens_out":7561,"duration_ms":84701,"concrete_test":"Run the trained ViPro-2 model on Orbits-2D with Gaussian noise added to the initial screen coordinates (e.g., σ = 0, 1, 2, 5 pixels) and measure LPIPS and position MAE over the 6-frame burn-in plus 24-frame rollout. If MAE degrades sharply (e.g., >0.5) at σ=2, the model is not robust to noisy initial states, and the central claim that it corrects inaccurate states is unsupported. Also run the same noise test on the 'ViPro-2 Supervised' variant for comparison; if both degrade similarly, the robustness claim is not about the unsupervised state estimation but about the architecture.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and introduction claim ViPro fails when 'previous states are noisy' and that ViPro-2 enables the model to 'correctly infer states from observations' despite this. However, every experiment initializes from exact screen coordinates (and a depth map in 3D), which are clean, ground-truth-derived object positions—not noisy states. No noise is injected into the initialization or during burn-in. The dramatic improvement over ViPro (LPIPS 1.72 vs 31.8 with screen coords; vs 3.4 with GT init) could therefore be due to switching from a full GT state to a clean alternative representation, rather than to the model actively correcting noisy state estimates. The shortcut ViPro learned was propagating the clean GT state through F; ViPro-2's gain predictor and observation loss might similarly propagate a clean initial condition if the screen coordinates are accurate and the observations are consistent. Without noise perturbation, one cannot distinguish genuine state estimation from faithful propagation of a clean initialization. This is load-bearing because the paper's own motivation is tolerance to noisy/inaccurate states, yet that scenario is absent from the experiments. The 'unsupervised' claim is also weakened by privileged inputs (screen coordinates, known dynamics), as noted in the reader's verdict, but the missing noise test is a more specific and falsifiable gap.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ViPro-2, an extension of the ViPro video-prediction model that aims to estimate latent object states from observations without being given the full ground-truth initial state. The main architectural changes are restructuring the procedural knowledge module, adding an observation-alignment loss Lobs, separating the latent state per variable, and introducing a recurrent gain predictor G that fuses observed and predicted states. On the Orbits-2D benchmark, ViPro-2 initialized from screen coordinates achieves LPIPS 1.72 and position MAE 0.042, outperforming ViPro initialized from full ground-truth state (LPIPS 3.4, MAE 0.18). The paper also introduces Orbits-3D, where a depth sample from the first frame is used to initialize world positions; here the unsupervised ViPro-2 reaches LPIPS 6.9 and MAE 15.7, substantially worse than its supervised counterpart (LPIPS 2.5, MAE 0.08 for the Groundtruth Z baseline). The authors acknowledge in Section VI that the model is not fully unconditioned and depends on known dynamics equations.","tokens_in":11063,"tokens_out":4702,"duration_ms":56555,"significance":"The 2D result is a meaningful step: it demonstrates that an object-centric video predictor can be trained to recover a useful symbolic state from observations without direct state supervision, and the ablation in Table II clearly attributes the gains to specific design choices (observation alignment, latent separation, gain predictor). The release of code and data is a strength, and the Orbits-3D extension is a useful benchmark addition. However, the central motivation---robustness to noisy or inaccurate states---is never tested, and the 3D unsupervised results are far from the supervised upper bound. The 'unsupervised' label is also used in a narrow sense that depends on privileged conditioning (screen coordinates, known dynamics, and, in 3D, a depth map). If the missing experiments are added and claims are scoped accordingly, the contribution would be solid.","major_comments":[{"comment":"The paper's stated motivation is that ViPro fails when 'previous states are noisy' and that ViPro-2 'correctly infer[s] states from observations' despite noisy inputs. However, every experiment initializes from exact screen coordinates (and, in 3D, a depth sample), which are clean ground-truth-derived quantities, and no noise is injected into the initialization or into the burn-in observations. The gain predictor (Eq. 4-5) is explicitly motivated by Kalman-filter-style weighting between prediction and observation, but its behavior is never evaluated under the noise regime that motivates it. The improvement from ViPro (LPIPS 31.8, MAE 9.54) to ViPro-2 (LPIPS 1.72, MAE 0.042) could therefore be due to the model learning a better mapping from clean observations rather than to genuine correction of noisy state estimates. Please add experiments with corrupted initialization (e.g., Gaussian pe","section":"Section I and IV-D, Tables I-II"},{"comment":"The abstract states that state inference is possible 'in an unsupervised manner,' but the 3D experiments do not support this at the same standard as the 2D experiments. The unsupervised ViPro-2 reaches LPIPS 6.9 and position MAE 15.7, while the supervised variant reaches LPIPS 2.5 and the Groundtruth Z baseline reaches MAE 0.08 (Table VI); the authors note that Z contributes about 90% of the error. Moreover, the method uses a depth map provided with the first frame and known screen coordinates (Section V), and Section VI explicitly acknowledges that the model is not 'fully unconditioned' and that the availability of dynamics equations is a foundational assumption. These are not fatal flaws, but the title and abstract overstate the general claim. Please either add a 3D experiment that treats depth as an estimated variable rather than provided input, or clearly scope the claim to 'unsuperv","section":"Section V, Tables V-VII"}],"minor_comments":[{"comment":"The column header 'MAEP os' appears to be a typo for 'Position MAE' or 'MAE Pos'.","section":"Table II"},{"comment":"The name 'V on Rueden' should be 'von Rueden'; please check the spelling in the text and in reference [7].","section":"Section II"},{"comment":"The grouping of architectural changes into a single ablation stage is currently justified only by 'we have found' statements. Since this grouping is the main empirical evidence, please document the individual failed combinations in an appendix or supplementary material.","section":"Section IV-C"},{"comment":"The diagram uses symbols such as zapred, zaobs, and the '∼' node that are not all defined in the caption. Please define them in the caption or in the text near the figure.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ViPro-2 is a genuine improvement over its predecessor: the diagnosis of ViPro's shortcut is convincing, the ablation is clean, and on Orbits-2D the model actually beats ViPro with ground-truth initialization. But the paper's headline claim—robustness to noisy initial states—is never tested, and the 'unsupervised' label leans on more privilege than the abstract admits.\n\nThe shortcut story is the best part. ViPro with screen-coordinate initialization collapses to LPIPS 31.8 and position MAE 9.54, versus 3.4 and 0.18 with the full ground-truth state. That is strong evidence the original model was just propagating the initial symbolic state through F rather than reading it from the frames. The observation alignment loss plus the recurrent gain predictor fix that, and the staged ablation shows each piece earns its keep. Final 2D numbers—LPIPS 1.72, MAE 0.042—surpass the supervised baseline, which supports the claim that the state is genuinely being extracted from observations. The background-slot separation and disentangled latent variables are sensible engineering. The new Orbits-3D dataset is a reasonable step, and the honest limitations section is a plus.\n\nThe soft spots are real but addressable. First, nowhere is noise added to the initial state or the burn-in sequence. The motivation says 'if previous states are noisy,' but every experiment starts from clean screen coordinates (and a clean depth sample in 3D). So the specific robustness claim is unverified. The stress-test is right: the 2D improvement could partly reflect a more learnable conditioning signal rather than active correction of noisy estimates. A simple noise-injection experiment would settle this and should be added before publication. Second, 'unsupervised' overstates the case: the model gets first-frame screen coordinates per object, a depth map in 3D, and the exact dynamics equations. The paper admits this in Section VI, but the abstract doesn't. Third, the 3D results are much weaker—unsupervised LPIPS 6.9 vs supervised 2.5, position MAE 15.7 with Z dominating—so the 3D part reads as a benchmark and a work-in-progress, not a solved problem. Minor issue: the LPIPS values are clearly scaled (1.72, 3.4, 31.8) but the scaling is never stated, which makes comparisons with standard LPIPS confusing.\n\nWho is this for? People working on object-centric video prediction or informed machine learning. It deserves a serious referee. The core 2D result is solid and the ablation is reproducible evidence, but the noise claim needs an experiment and the wording needs to be tightened. I'd send it to review with those requests.","headline":"A solid fix for ViPro's shortcut, but the noise-robustness claim is untested and 'unsupervised' reaches further than the experiments.","tokens_in":11470,"tokens_out":3385,"would_cite":false,"duration_ms":35557,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ViPro-2 shows that a video-prediction model can estimate the symbolic state of objects from observed frames alone, without ground-truth initial states, by fusing predicted and observed states through a learned gain.","keywords":["video prediction","state estimation","procedural knowledge","informed machine learning","object-centric representation","dynamics equations","unsupervised learning","slot attention"],"falsifier":"Run ViPro-2 on Orbits-2D with the dynamics equations deliberately perturbed (for example, the gravity constant changed by 10%) or with the first-frame screen coordinates shifted by several pixels. If image and position errors stay at the reported levels, the model is still ignoring the integrated knowledge; if they degrade sharply, the unsupervised-state-estimation claim depends on the privileged initialization. A more direct test is removing the screen-coordinate initialization entirely and seeing whether object states emerge from raw video alone.","tokens_in":10677,"feed_emoji":"🎥","tokens_out":8671,"duration_ms":80425,"temperature":0.7,"pith_summary":"This paper tries to fix a hidden failure in dynamics-guided video prediction: the predecessor ViPro, when handed the true initial object state, learned to carry that state through the dynamics equations without ever connecting it to the pixels in the observed frames. The proposed ViPro-2 restructures the task so the model must read the symbolic state from the image, aligning the dynamics prediction with the observed state and fusing them through a learned gain. With only the first frame's screen coordinates as initialization, ViPro-2 predicts future frames and object positions on the Orbits-2D benchmark more accurately than the fully ground-truth-initialized predecessor (LPIPS 1.72 vs 3.4; position MAE 0.042 vs 0.18). The paper also introduces an Orbits-3D variant and shows that a single depth sample from an RGB-D first frame is enough to get depth estimation started. If correct, this makes procedural-knowledge video prediction usable where full state annotations are unavailable.","feed_headline":"Video model infers states from pixels, beats ground-truth baseline","feed_subtitle":"Pixel-only starts reach LPIPS 1.72 and position MAE 0.042, beating fully initialized ViPro.","key_machinery":"The procedural knowledge module integrates known dynamics equations $F$ into the object-centric prediction loop built on Slot Attention and SlotFormer-style burn-in/rollout. The load-bearing components are: (1) the observation-alignment loss $L_{\\text{obs}}=(s_{\\text{pred}}-s_{\\text{obs}})^2$, which prevents the shortcut by forcing the predicted state to agree with the state read from the current frame; (2) a recurrent gain predictor $G$, an MLP-plus-GRU-plus-sigmoid module, that computes a per-frame gain $K=\\sigma(G(z_a^{\\text{obs}}, z_a^{\\text{pred}}))$ to fuse observed and predicted states as in a Kalman filter. Supporting changes include per-variable MLPs in $F_{\\text{in}}$ and $F_{\\text","core_discovery":"The paper's central discovery is that ViPro had learned a shortcut: given a perfect initial symbolic state and a perfect dynamics equation, it propagated that state forward and decoded it, ignoring the video frames. Evidence is that switching initialization from world positions to screen coordinates made position MAE jump from 0.18 to 9.54. ViPro-2 changes the training target: the latent used for decoding is built from the observed state $s_{\\text{obs}}$, while the dynamics-predicted state $s_{\\text{pred}}$ is used as a regularizing target through $L_{\\text{obs}}=(s_{\\text{pred}}-s_{\\text{obs}})^2$, and a recurrent gain predictor $G$ produces a Kalman-style gain $K$ so that the final state i","pith_inferences":["The learned gain $G$ can be read as an empirical Kalman gain; a natural testable extension is to check whether $G$ converges to the optimal Kalman gain when process and observation noise levels are known, linking the architecture to classical recursive state estimation.","Because the Z-coordinate error accounts for about 90% of the position error in Orbits-3D, improving depth initialization—for example, aggregating depth samples across an object's mask instead of a single pixel—should yield large unsupervised gains.","The diagnosis of ViPro's shortcut suggests a general failure mode: whenever a perfect forward model is provided, a network can satisfy the training loss by copying the given state instead of perceiving it. The observation-alignment loss is a template for preventing this in other model-based prediction settings.","If the approach transfers to real video, an object-centric predictor with integrated dynamics could double as a self-supervised tracker, since consistent state estimates are required to reconstruct and predict frames."],"forward_implications":["Video prediction with physics knowledge no longer requires ground-truth object states for every scene; first-frame screen coordinates (plus a depth map for 3D) suffice.","State estimation becomes a byproduct of prediction, so the same model outputs object positions while generating future frames, with position MAE below 0.05 in the 2D benchmark.","Because the observation must be used, the model can correct imperfect initial states during burn-in rather than carrying them forward blindly, making it tolerant of noisy conditioning.","The 3D variant demonstrates that depth-aware dynamics can be handled with an RGB-D first frame, extending the approach toward real-world scenes where full 3D state labels are not available."],"supporting_citations":[{"why":"The ViPro model this work diagnoses and extends; supplies the Orbits dataset, the procedural knowledge module, and the ground-truth-initialized baseline.","marker":"[2]"},{"why":"Slot Attention: provides the object-centric slot mechanism used to refine latent object states from frames.","marker":"[16]"},{"why":"SAVi: supplies the slot-attention-for-video training scheme the paper adapts for burn-in processing.","marker":"[17]"},{"why":"SlotFormer: provides the autoregressive rollout scheme for future-frame prediction and is a comparison baseline.","marker":"[19]"},{"why":"LPIPS: the learned perceptual metric used to score image reconstruction quality.","marker":"[20]"},{"why":"GRU: the recurrent cell used inside the gain predictor G for temporal fusion of observations and predictions.","marker":"[21]"},{"why":"SlotDiffusion: supplies the MAE-plus-LPIPS reconstruction loss used in the Orbits-3D setting and is a comparison baseline.","marker":"[25]"}],"fun_headline_variants":["ViPro-2 exposes shortcut, learns states from pixels alone","Unsupervised state estimation from pixels beats fully initialized","State inference from pixels: no ground truth crutch","ViPro-2: Unsupervised states from video, beating ViPro"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"Everything rests on being given the exact dynamics equations $F$ and a starting state derived from screen coordinates (plus a depth map for 3D); if those external inputs are wrong or missing, the paper's unsupervised-state-estimation claim does not apply.","fun_headline_variants_meta":{"raw":{"variants":["ViPro-2 exposes shortcut, learns states from pixels alone","Unsupervised state estimation from pixels beats fully initialized","State inference from pixels: no ground truth crutch","ViPro-2: Unsupervised states from video, beating ViPro"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000505,"raw_usage":{"total_tokens":2268,"prompt_tokens":680,"completion_tokens":1588,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":424,"completion_tokens_details":{"reasoning_tokens":1518}},"tokens_in":424,"tokens_out":1588,"duration_ms":13859,"temperature":1.0,"reasoning_tokens":1518,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T22:46:14.840903+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ViPro-2 on Orbits-2D with the dynamics equations deliberately perturbed (for example, the gravity constant changed by 10%) or with the first-frame screen coordinates shifted by several pixels. If image and position errors stay at the reported levels, the model is still ignoring the integrated knowledge; if they degrade sharply, the unsupervised-state-estimation claim depends on the privileged initialization. A more direct test is removing the screen-coordinate initialization entirely and seeing whether object states emerge from raw video alone.","supporting_citations":[{"cited_title":"SlotFormer: Un- supervised Visual Dynamics Simulation with Object-Centric Models,","cited_arxiv_id":null,"evidence_quote":"SlotFormer: provides the autoregressive rollout scheme for future-frame prediction and is a comparison baseline."},{"cited_title":"ViPro: Enabling and Controlling Video Prediction for Complex Dynamical Scenarios Using Procedural Knowledge,","cited_arxiv_id":null,"evidence_quote":"The ViPro model this work diagnoses and extends; supplies the Orbits dataset, the procedural knowledge module, and the ground-truth-initialized baseline."},{"cited_title":"Object-Centric Learning with Slot Attention,","cited_arxiv_id":null,"evidence_quote":"Slot Attention: provides the object-centric slot mechanism used to refine latent object states from frames."},{"cited_title":"Conditional Object- Centric Learning from Video,","cited_arxiv_id":null,"evidence_quote":"SAVi: supplies the slot-attention-for-video training scheme the paper adapts for burn-in processing."},{"cited_title":"The Unreasonable Effectiveness of Deep Features as a Perceptual Metric,","cited_arxiv_id":null,"evidence_quote":"LPIPS: the learned perceptual metric used to score image reconstruction quality."},{"cited_title":"Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation,","cited_arxiv_id":null,"evidence_quote":"GRU: the recurrent cell used inside the gain predictor G for temporal fusion of observations and predictions."},{"cited_title":"SlotDiffusion: Object-Centric Generative Modeling with Diffusion Models,","cited_arxiv_id":null,"evidence_quote":"SlotDiffusion: supplies the MAE-plus-LPIPS reconstruction loss used in the Orbits-3D setting and is a comparison baseline."}],"review_version":1}