{"id":"cd45a525-3d18-4066-ae8d-62dac3dab11b","arxiv_id":"2412.01429","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CPA adds camera pose control to OpenSora-style diffusion video models by encoding a sparse Plücker motion field into a latent and injecting it into temporal attention layers.","lead":"This paper presents CPA, a plug-in that lets a diffusion-transformer video generator follow specific camera moves such as zoom, pan, and rotate. In tests on 72-frame clips it reports better camera-trajectory consistency than prior methods, though it evaluates on only 20 videos.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim rests on a 20-video, resolution-mismatched comparison; the 0.002 CamMC edge over VD3D is within plausible noise, so the headline conclusion is not yet established.","rationale":"The reader's weakest assumption identifies the right load-bearing issue: the long-video SOTA claim is not supported by a rigorous comparison. My independent reading confirms that the decisive evidence gap is the evaluation protocol, not the internal logic of the modules. The paper does present a reasonable plug-in design, with a VAE for a sparse Plücker-derived motion field, a temporal-attention injection module, and ablations showing internal design choices matter. Those are real contributions. However, the external comparison is the load-bearing part of the headline claim, and it has four concrete problems: resolution mismatch between LDM and DiT baselines, a 20-video test set without error bars, an unfair chaining protocol for LDM long-video generation, and test-set-based selection of the sparse-sampling ratio. The closest DiT baseline, VD3D, trails CPA by only 0.002 CamMC and 13 FVD, which is exactly the scale of difference that can flip under any of these confounds. I also note that Section 4.6 admits object consistency is relatively weak, which conflicts with the abstract's claim of optimal object consistency, but the camera-control SOTA claim is the primary target and is already undercut by the evaluation issues. Because the reader's verdict is already conditional, my read does not move the verdict; it reinforces the need for a larger, matched-resolution, properly randomized evaluation with confidence intervals. The concrete test above would settle whether the reported SOTA margin survives a statistically sound comparison.","tokens_in":10808,"tokens_out":7259,"duration_ms":66461,"concrete_test":"Re-run the Table 1 long-video comparison on a fixed 100-video subset of the RealEstate10K test set, evaluating all methods at a common resolution (e.g., 640x360 for DiT and CPA; 384x256 upsampled for LDM baselines), with three random seeds and per-video bootstrap 95% confidence intervals for CamMC and FVD. If CPA's CamMC remains below VD3D by more than the confidence-interval width, the SOTA claim stands; otherwise, the reported gap is not statistically meaningful. As a secondary check, verify that the sparse-sampling ratio is selected on a validation split rather than the same 20-video test set.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of SOTA in long-video camera control rests on Table 1, where CPA's long-video CamMC is 0.042 versus VD3D's 0.044 and FVD is 1387 versus 1400. These gaps are too small to interpret without error bars, and the evaluation setup introduces confounds. First, baselines are not evaluated at matched resolutions: LDM-based methods use 256x256 or 384x256, while DiT-based methods and CPA use 640x360. FVD, FID, CLIPSIM, and CamMC are all resolution-sensitive, so a resolution advantage can masquerade as a method advantage. Second, only 20 RealEstate10K videos are used, with no confidence intervals, no multiple seeds, and no per-video breakdown; a 0.002 CamMC difference is plausibly sampling noise. Third, long-video evaluation is not apples-to-apples: LDM baselines are converted to 72 frames by chaining segments with the previous final frame as reference, a procedure those models were not designed for, while CPA natively generates 72 frames. Fourth, the 40x sparse-sampling ratio in Table 2 is selected using the same test set, so the reported Table 1 numbers may partially reflect test-set tuning. If any of these confounds favors CPA, the 'SOTA' conclusion is not supported by the presented evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CPA, a plug-in for DiT-based text-to-video generation (built on OpenSora-v1.2) that adds camera-pose control. It introduces a Sparse Motion Encoding (SME) module that converts camera rotation/translation sequences into a sparse Plücker-coordinate motion field, a VAE that compresses this field into a pose latent, and a Temporal Attention Injection (TAI) module that fuses the pose latent into the temporal attention layers of the ST-DiT blocks. The backbone is kept largely frozen; only temporal-attention and LoRA parameters are updated. Experiments on 20 RealEstate10K test videos compare CPA with LDM- and DiT-based baselines for both 16-frame and 72-frame generation, reporting the best long-video CamMC, FVD, and CLIPSIM in Table 1. The paper's own discussion (Sec. 4.6) acknowledges that object consistency is relatively weak and that pose support is currently limited to 16 frames.","tokens_in":11088,"tokens_out":5838,"duration_ms":50817,"significance":"The core idea is timely and potentially useful: a camera-pose conditioner that works with DiT video generators, preserves the base model's parameters, and uses a compact VAE-compressed pose latent. If the evaluation were properly controlled, the method could be a meaningful step toward practical camera control in long-form video generation. The paper's claim to state-of-the-art long-video camera motion control is not yet established, however, because the comparison in Table 1 is confounded by resolution mismatch, a very small test set, no error bars, and a hyperparameter selected on the same test set. The manuscript would also benefit from releasing the pose VAE and checkpoints, since the contribution is largely architectural.","major_comments":[{"comment":"The long-video SOTA claim is not supported by the presented evaluation because the comparison is not controlled. LDM-based baselines are evaluated at 256×256 or 384×256 resolution and are extended to 72 frames by chaining 16-frame segments with the previous final frame as reference, while CPA natively generates 72 frames at 640×360. All four metrics (FID, FVD, CLIPSIM, CamMC) are resolution- and protocol-sensitive. The reported long-video CamMC gaps (0.042 vs 0.044 for VD3D, 0.042 vs 0.068 for EasyAnimate) are small enough that they may reflect the evaluation setup rather than a genuine method advantage. Please report baselines at matched resolution with the same chaining or native-generation protocol, and include confidence intervals or multiple seeds.","section":"§4.4, Table 1"},{"comment":"The evaluation uses only 20 test videos, with no error bars, no multiple seeds, and no per-video breakdown. Without these, a CamMC difference of 0.002 (0.042 vs 0.044) or an FVD difference of 13 (1387 vs 1400) is within plausible sampling noise. Additionally, Table 1 reports no CamMC for OpenSora, the base model of CPA; without this number, the camera-control improvement contributed by the proposed modules over the backbone is not quantified. Please evaluate OpenSora on the same videos and report the CamMC, along with a bootstrap or per-video analysis.","section":"§4.2, §4.4"},{"comment":"The sparse-motion sampling ratio (40×) is selected using the same 20-video test set that produces the headline results of Table 1. This is test-set tuning and can inflate the reported performance. The ratio should be selected on a validation split, or the main results should be re-reported with the chosen ratio applied to a held-out test set. The same concern applies to the injection-module ablation in Table 3.","section":"§4.5, Table 2"},{"comment":"The Plücker-coordinate derivation is not mathematically complete. Eq. (3) defines x = PX = [R|t]X, but Eq. (4) then introduces the intrinsic matrix K, so the notation is inconsistent. Eq. (5) back-projects a pixel to a camera-coordinate point at an implicit depth of 1, and Eq. (6) forms an inner product of homogeneous optical-center coordinates with that point, which is not the standard Plücker-coordinate representation of a line. Please provide a complete derivation with clearly defined R, t, K, o_c, depth conventions, and the exact construction of the sparse RGB motion field shown in Fig. 5, so that the SME module is reproducible.","section":"§3.2, Eqs. (3)–(6)"},{"comment":"Several training details are underspecified or inconsistent. The text says 'freeze all layers except temporal attention layers', but then says 'introduce LoRA during the update of the self-attention'; it is not clear which parameters are actually updated. The VAE training is described only as reconstruction loss plus KL loss on the sparse motion field, with no architecture, frame count, optimizer, or reconstruction-quality numbers. Since the pose latent is a core input to the whole pipeline, please report these details and a reconstruction metric for the sparse-motion VAE.","section":"§3.3, §4.1"}],"minor_comments":[{"comment":"The Abstract and Conclusion claim 'optimal performance in trajectory consistency and object consistency', but Sec. 4.6 states that object consistency is relatively weak and that large-scale object motion is limited. Please reconcile this wording with the stated limitations.","section":"Abstract and §4.6"},{"comment":"The TAI equations do not specify the dimensions of z^(k), z_p^(k), or the MLP output. Please include the exact tensor shapes, and state how the spatial patch count of the pose latent is aligned with the temporal-attention patch count when the two are inconsistent.","section":"§3.2, Eqs. (8)–(10)"},{"comment":"The selection of the 20 test videos is described only as 'randomly select', with no seed or list. For reproducibility, please provide the specific video identifiers or a documented sampling procedure.","section":"§4.2"},{"comment":"The 'Concat' baseline is described as 'channel-dimension concatation ... used in MotionCtrl', but it is not clear that this faithfully reproduces MotionCtrl's conditioning mechanism. Please clarify or cite the exact MotionCtrl configuration used.","section":"§4.5, Table 3"},{"comment":"The paper does not state whether code or checkpoints will be released. Given the plug-in design and the custom pose VAE, releasing these artifacts would substantially aid reproducibility and fair comparison.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper's headline SOTA claim rests on a small, resolution-mismatched, and possibly test-set-tuned evaluation; the methodological concerns raised in the major comments are, in my view, load-bearing and need to be addressed before the results can be taken at face value. The core architectural idea is plausible, and the paper is not beyond repair, so I recommend major revision rather than rejection. I also note that the authors' own Sec. 4.6 limitations are not fully reflected in the Abstract and Conclusion; this should be corrected during revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The method is worth a look: CPA is a clean plug-in for OpenSora-style DiT that encodes camera pose as a sparse Plücker motion field, VAE-compresses it, and injects the latent into temporal attention via LN+MLP. That specific configuration is new, and the design choices are sensible—freezing the backbone, LoRA for self-attention, and ablating injection methods are all reasonable. The paper is honest enough to include a limitations section that admits weak object consistency, which is more than many papers do.\n\nBut the headline claim is not established. The long-video SOTA rests on Table 1: CamMC 0.042 vs VD3D's 0.044, FVD 1387 vs 1400, all on 20 RealEstate10K videos with no error bars, no seeds, and no per-video breakdown. That CamMC gap is within plausible noise. Worse, the comparison is not apples-to-apples: LDM baselines run at 256x256 or 384x256 while DiT methods and CPA run at 640x360, and the LDM long-video numbers are produced by chaining 16-frame segments, a procedure those models weren't designed for. FVD, FID, CLIPSIM, and CamMC are all resolution-sensitive, so a resolution advantage can masquerade as a method advantage. The 40x sampling ratio is also selected on the same test set used for Table 1, which risks test-set tuning.\n\nThere's also an internal contradiction: the abstract claims \"optimal performance in trajectory consistency and object consistency,\" while Section 4.6 says \"the performance of the object consistency is relatively weak.\" That's not a minor wording issue; it undercuts the abstract's strongest phrase.\n\nThe Plücker derivation in Eqs 5-6 is sketched rather than derived, but that's a minor flaw—the motion-field construction is understandable and the sparse sampling is clear. The citation pattern is fine, and the method isn't circular: camera pose is an external condition, not a fitted target.\n\nWho is this for? People working on controllable video generation will find the injection module a useful reference point, even if they don't trust the numbers. The paper deserves a serious referee, but the referee should insist on a larger evaluation, matched resolutions, error bars, and a claim that matches the evidence.\n\nMy verdict: conditional accept at best, with major revision. The core idea is promising; the evidence for SOTA is not there yet.","headline":"A sensible plug-in for camera-aware DiT video generation, but the SOTA claim rides on a 20-video, resolution-mismatched comparison that doesn't support it.","tokens_in":11645,"tokens_out":1830,"would_cite":false,"duration_ms":16957,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a plug-in camera-pose module can give a diffusion-transformer video generator precise, long-horizon camera control without retraining the backbone.","keywords":["camera pose control","video generation","diffusion transformer","text-to-video","Plücker coordinates","temporal attention injection","sparse motion encoding","long video generation"],"falsifier":"Run the same 72-frame benchmark at one matched resolution for all methods (for instance everything at 640x360) with a larger test set of 100 or more clips and uncertainty estimates on CamMC and FVD; if the camera-motion error no longer beats the leading compared DiT baseline, or if the latent-diffusion baselines improve substantially with matched resolution, the state-of-the-art claim is falsified.","tokens_in":10575,"feed_emoji":"🎥","tokens_out":7252,"duration_ms":61314,"temperature":0.7,"pith_summary":"The paper presents CPA, a camera-pose-awareness method for diffusion-transformer (DiT) text-to-video generation. It claims that camera trajectories can be converted into a sparse motion field, compressed into a compact pose latent, and injected directly into the temporal attention layers of an existing DiT model, so the model follows the requested camera motion over long clips while keeping its original text-to-video quality. On 72-frame clips the method reports a camera-motion error of 0.042, lower than the previous best 0.044 among the compared methods, together with better video-consistency scores. If the result holds, camera pose becomes a practical first-class condition for long video generation, useful for pre-visualization, virtual cinematography, and controllable world-model applications.","feed_headline":"Camera-aware video generator claims best long-clip pose control","feed_subtitle":"A plug-in motion module feeds camera trajectories into a diffusion transformer for steadier long-video generation.","key_machinery":"Two plug-in modules carry the argument. The Sparse Motion Encoding (SME) module samples a camera-derived Plücker-coordinate motion field at regular grid intervals in the image, building a sparse field $F_s \\in \\mathbb{R}^{L\\times M\\times N}$ that is then encoded by a VAE into a pose latent $z_p \\in \\mathbb{R}^{l\\times m\\times n\\times 4}$. The Temporal Attention Injection (TAI) module aligns this pose latent with the temporal attention features of each spatial-temporal DiT block by normalizing both, concatenating them, and applying a learned shift-and-scale through an MLP; the authors argue this unifies the pose and temporal latents into a similar distribution, which is what makes the injection effective.","core_discovery":"The central claim is that camera-pose control for DiT-based video generators can be achieved by encoding the inter-frame camera extrinsics as a sparse pixel-wise motion field based on Plücker coordinates, where each sampled pixel's back-projected ray is moved by the rotation and translation between frames, then training a small VAE to compress that field into a pose latent, and injecting the latent into temporal attention through layer normalization and an MLP. The author argues that this preserves the pretrained DiT's capabilities because only the temporal attention layers are fine-tuned, with LoRA used in self-attention, while all other parameters stay frozen. On the long-video (72-frame) evaluation, CPA reports CamMC, a camera-motion error metric, of 0.042 and FVD 1387, improving on the 0.044 CamMC and 1400 FVD of the leading compared DiT camera-control baseline, while also scoring highest on CLIPSIM. The claim is therefore that CPA achieves state-of-the-art camera-motion consistency for long video generation.","pith_inferences":["We suspect the same SME-plus-TAI recipe could transfer to other DiT video backbones, since it does not modify the backbone's weights and only touches temporal attention, so a head-to-head test on a second base model would show whether the plug-in generalizes.","The sparse Plücker field could in principle also encode object trajectories as per-point displacements, which would let object motion and camera motion share one conditioning space; the paper does not claim this, but its representation invites the extension.","The reported numbers leave a resolution confound: the LDM baselines are evaluated at 256x256 or 384x256 while the DiT methods run at 640x360, so a matched-resolution evaluation would be a sharper test of the state-of-the-art claim."],"forward_implications":["If the claims hold, camera-pose conditioning can be layered onto existing DiT video generators as a plug-in, without retraining the full generative model.","Per-frame pose injection makes long clips follow a prescribed trajectory in one pass, avoiding the drift that comes from chaining short segments.","A sampling density around 40x, about 16 by 9 control points at 640 by 360, appears to be a practical sweet spot; the paper reports that both denser and sparser sampling degrade consistency.","Because the backbone is mostly frozen, adapting the plug-in to a new DiT base may be inexpensive: the reported fine-tune takes about 100k steps on four GPUs.","The paper's own limitation discussion notes the method is conservative about object motion, so large object movement remains an open problem."],"supporting_citations":[{"why":"Supplies the pretrained DiT backbone and base framework that CPA is built on and fine-tunes.","marker":"[41]"},{"why":"Supplies the camera-pose-annotated video dataset used to train the pose VAE and to evaluate long-video consistency.","marker":"[44]"},{"why":"Defines the CamMC camera-consistency metric and serves as the main LDM baseline whose long-video numbers CPA must beat.","marker":"[31]"},{"why":"Provides an earlier camera-control baseline and the comparison point for trajectory-following behavior.","marker":"[8]"},{"why":"Represents the leading DiT-based camera-control baseline that also uses Plücker coordinates and is the direct rival on long-video consistency.","marker":"[1]"},{"why":"Provides the VAE architecture chosen for the sparse motion encoding so the pose latent matches the temporal dimensions of the DiT model.","marker":"[36]"},{"why":"Gives the Plücker-coordinate formulation used to turn rotation and translation matrices into per-pixel motion vectors.","marker":"[20]"}],"fun_headline_variants":["Camera-pose control for video transformers with sparse motion injection","New DiT module steers camera paths without full retraining","Sparse motion encoding improves camera consistency in long videos","Plug-in camera-pose module achieves state-of-the-art consistency","Camera-pose-aware DiT injects motion into temporal attention"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The long-video state-of-the-art claim assumes that the metrics are comparable across the different resolutions used for different methods (256x256 or 384x256 for the latent-diffusion baselines, 640x360 for the transformer-based ones) and that the 20-video test set, reported without error bars, is representative.","fun_headline_variants_meta":{"raw":{"variants":["Camera-pose control for video transformers with sparse motion injection","New DiT module steers camera paths without full retraining","Sparse motion encoding improves camera consistency in long videos","Plug-in camera-pose module achieves state-of-the-art consistency","Camera-pose-aware DiT injects motion into temporal attention"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000853,"raw_usage":{"total_tokens":3692,"prompt_tokens":918,"completion_tokens":2774,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":534,"completion_tokens_details":{"reasoning_tokens":2691}},"tokens_in":534,"tokens_out":2774,"duration_ms":18470,"temperature":1.0,"reasoning_tokens":2691,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:22:41.513721+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 72-frame benchmark at one matched resolution for all methods (for instance everything at 640x360) with a larger test set of 100 or more clips and uncertainty estimates on CamMC and FVD; if the camera-motion error no longer beats the leading compared DiT baseline, or if the latent-diffusion baselines improve substantially with matched resolution, the state-of-the-art claim is falsified.","supporting_citations":[{"cited_title":"Open-sora: Democratizing efficient video production for all, March 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained DiT backbone and base framework that CPA is built on and fine-tunes."},{"cited_title":"Motionctrl: A unified and flexible motion controller for video generation","cited_arxiv_id":null,"evidence_quote":"Defines the CamMC camera-consistency metric and serves as the main LDM baseline whose long-video numbers CPA must beat."},{"cited_title":"Camera Pose Estimation from Lines using Pl\\\"ucker Coordinates","cited_arxiv_id":"1608.02824","evidence_quote":"Gives the Plücker-coordinate formulation used to turn rotation and translation matrices into per-pixel motion vectors."}],"review_version":1}