{"id":"ad035901-e81f-4d0d-a22f-a6d4475cb44b","arxiv_id":"2501.01409","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Jointly training a video diffusion model with a 3D point map reconstruction head improves the 3D consistency of generated videos and yields usable camera pose estimates on static scenes.","lead":"JOG3R is a single neural network that generates short videos and estimates camera motion for those videos at the same time. It shows that forcing a video generator to also do 3D reconstruction makes its frames more geometrically consistent with each other.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MEt3R consistency advantage may be an artifact of evaluating with JOG3R's own point maps trained on ZoeDepth pseudo-GT; re-evaluate with a fixed independent estimator.","rationale":"The reader identified the pseudo-GT point-map supervision and the self-referential MEt3R evaluation as the weakest assumptions, and my analysis agrees. This is the single most load-bearing concern because the abstract's headline claim of 3D consistency is quantitatively supported only by Table 1's MEt3R numbers. If the MEt3R advantage disappears under an independent estimator, then the paper's central claim is unsubstantiated, even though the V2C pose results and the unified architecture remain contributions. I keep the reader's CONDITIONAL verdict because the issue is concrete and fixable: re-running the metric with a fixed external point-map estimator, and optionally validating ZoeDepth pseudo-GT against MVS depths, would settle whether the consistency gain is genuine. An honest non-finding was not appropriate because this circularity is a real methodological risk, not a manufactured concern. The recommended verdict therefore remains UNCHANGED rather than moving to accept or reject.","tokens_in":17244,"tokens_out":3588,"duration_ms":35143,"concrete_test":"Recompute Table 1 MEt3R for all methods using a fixed, independent point-map estimator (e.g., DUSt3R or a DUSt3R variant fine-tuned on the same RealEstate10K data) rather than JOG3R's own point maps. If JOG3R no longer achieves the lowest MEt3R, the 3D-consistency advantage reported in Table 1 is an artifact of estimator choice.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (abstract) is that JOG3R is the first 3D-consistent video generator. Table 1 supports this with MEt3R (0.0736 vs 0.0819 for pre-trained OS). However, the metric is computed (Section 4.2) by warping DINO features with point maps estimated by JOG3R itself for every method. Since JOG3R's point-map head and video generator are jointly trained with Lrec on ZoeDepth-unprojected pseudo-GT (Section 4.1), a low MEt3R may reflect that the generator produces videos that are geometrically self-consistent under its own (possibly biased) depth prior, rather than globally 3D-consistent. The paper provides no validation of ZoeDepth accuracy on RealEstate10K interiors, and no cross-check with an independent point-map estimator. The V2C pose results are less affected because they are measured against ground-truth cameras, but they do not directly validate 3D consistency of generated videos. Thus Table 1, the sole quantitative evidence for the headline claim, is potentially circular.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether video diffusion transformers can be jointly trained for video generation and 3D reconstruction. The authors stitch the OpenSora DiT backbone to a DUSt3R-style point-map decoder and head, and fine-tune the combined model with a generation loss and a reconstruction loss whose pseudo-ground-truth point maps come from ZoeDepth metric depth unprojected with RealEstate10K camera parameters. They report three usage modes: text-to-video (T2V), video-to-camera (V2C), and joint text-to-video-plus-camera (T2V+C). The central empirical claims are that the unified model improves the 3D consistency of generated videos relative to pre-trained OpenSora, as measured by MEt3R, and that it produces competitive camera pose estimates on real videos, as measured against ground-truth poses on RealEstate10K and DL3DV10K.","tokens_in":17570,"tokens_out":5159,"duration_ms":49953,"significance":"If the central claim holds, the paper makes a useful contribution: it shows that a video generator's features can be repurposed for a 3D task and that joint training with a geometric loss improves the geometric consistency of generated videos without sacrificing generation quality. The architecture is clean and the experimental setup is mostly standard, with external ground-truth camera poses used for the V2C evaluation. The main fragility is that the headline 3D-consistency result in Table 1 is evaluated with JOG3R's own point-map estimator for all methods, which makes the comparison potentially self-referential. The paper also does not validate the ZoeDepth pseudo-ground-truth point maps used to train the reconstruction head, and the reported improvements lack error bars. These issues are fixable and do not invalidate the overall approach, but they must be addressed before the headline claim is fully supported.","major_comments":[{"comment":"The MEt3R metric is computed with JOG3R's own point map estimator for every method. Because JOG3R's point-map head was trained on RealEstate10K ZoeDepth pseudo-GT, it may systematically favor JOG3R-generated videos over videos from pre-trained or fine-tuned OpenSora, so the reported improvement (0.0736 vs 0.0819) could reflect estimator specialization rather than a true difference in geometric consistency. Please recompute MEt3R with a fixed, independent point-map estimator (for example, pre-trained DUSt3R or MASt3R) applied to all methods, and also report per-method point-map quality or use each method's own estimator as a cross-check.","section":"Section 4.2, Table 1"},{"comment":"The reconstruction loss Lrec is supervised by point maps obtained by unprojecting ZoeDepth metric depth with RealEstate10K camera parameters, but the paper provides no validation of ZoeDepth accuracy on real-estate interior imagery. If ZoeDepth has systematic depth biases, those biases are baked into JOG3R's point-map head and could explain both the MEt3R improvement in Table 1 and the moderate V2C results in Table 2. Please quantify the pseudo-GT quality (e.g., compare ZoeDepth against COLMAP/SfM or DUSt3R point clouds on a held-out split) and, if feasible, ablate the training signal with an alternative pseudo-depth source.","section":"Section 4.1, Eq. (2)"},{"comment":"No error bars, confidence intervals, or multiple-seed results are reported. The MEt3R gap in Table 1 is only about 10% relative, and with 180 generated videos and a fixed evaluation set the difference may not be statistically significant. Without variance estimates, the central claim that JOG3R is more 3D-consistent than pre-trained OpenSora is not yet demonstrated. Please report standard errors or bootstrap confidence intervals and, ideally, results over multiple fine-tuning seeds.","section":"Tables 1 and 2"},{"comment":"The selected feature block b26 is outperformed by b20 in the paper's own ablation on both datasets (e.g., RealEstate10K mAA@30 47.25% vs 53.35%; DL3DV10K 34.22% vs 36.62%). This contradicts the text's claim that 'all later blocks should lead to similar results' and makes the choice of b26 appear unsupported. Since the feature block is a free parameter of the method, please either adopt the best-performing block, provide evidence that b26 is preferable on other criteria such as generation quality or MEt3R, or explicitly rescope the design claim.","section":"Appendix B, Table 3"}],"minor_comments":[{"comment":"The confidence term is written as C^{f->1}_i in the surrounding text but C^{f,1}_i in the equation; please use a single notation throughout and define the set of valid pixels i.","section":"Eq. (2)"},{"comment":"References [28] and [51] are duplicated entries, and references [34] and [35] are the same paper (the Platonic representation hypothesis); these should be consolidated.","section":"Related Work"},{"comment":"The 'Self consistency of T2V->V2C and T2V+C' paragraph reports a 0.45-degree average rotation difference and a 19.20-degree translation difference between two pipelines. This is an internal consistency check, not a validation of absolute pose accuracy; please state this explicitly in the main text.","section":"Section 4.3"},{"comment":"The caption says 'bold is best in each sub-table; underlined is the second place,' but the rows do not make clear whether the bolding is across all methods or only within the JOG3R/OpenSora variants; please clarify the grouping.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Core take: this is a genuine first — a single model that generates videos and predicts dense 3D point maps and camera poses — and the central finding holds up. Fine-tuning a video DiT with a DUSt3R-style reconstruction loss measurably improves 3D consistency without wrecking generation quality, and the negative result that frozen OpenSora features give mediocre poses supports the joint-training argument. What's new: stitching OpenSora's DiT to DUSt3R's decoder is clean and sensible, and the baselines are fair, including DUSt3R trained from scratch and fine-tuned on the same data. The supplementary ablation on which DiT block to route is good practice, and the limitations section is honest about static scenes and short videos. Soft spots, in order. First, the headline MEt3R gain (0.0736 vs 0.0819) is weaker than it looks because the metric is computed with JOG3R's own point maps for every method, as stated in Section 4.2. That means the metric rewards videos that agree with JOG3R's own learned depth prior. The stress-test note is right; the fix is straightforward: recompute MEt3R with a fixed independent estimator like pre-trained DUSt3R or MASt3R. Second, the point-map supervision is trained against ZoeDepth-unprojected pseudo-GT, and ZoeDepth's accuracy on RealEstate10K is never validated. It's standard practice, but a sanity check against the available ground-truth cameras would remove the concern. Third, no error bars anywhere; with 180 generated videos the MEt3R difference is plausible but not demonstrated. Fourth, the chosen block b26 is not the best in their own ablation — b20 gives better mAA on both datasets (53.35 vs 47.25 on RE10K) — yet the appendix dismisses this as \"on par.\" The differences are small on rotation/translation but the mAA gap is not negligible. The pose-competitiveness claim is weaker than the framing suggests: JOG3R beats pre-trained DUSt3R and from-scratch DUSt3R, but fine-tuned DUSt3R* at 75.50 mAA leaves a large gap. The paper says this honestly, though the abstract's \"competitive\" is doing some work. Who it's for: people working on camera control, video-to-3D, or geometry-aware video generation. It's not the final word, but the joint-training template is worth taking seriously. I'd send it to peer review with the MEt3R circularity flagged as a required fix, plus error bars and a ZoeDepth sanity check.","headline":"Genuine first: joint video generation + dense 3D point map/camera prediction in one trained model, with the headline 3D-consistency gain real but partly self-referential because MEt3R is computed with the model's own point maps.","tokens_in":764,"tokens_out":2473,"would_cite":true,"duration_ms":44089,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Joint training with a 3D point-map head makes video generators 3D-consistent, the paper claims.","keywords":["video generation","3D consistency","camera pose estimation","point map estimation","diffusion transformer","model stitching","DUSt3R","structure from motion"],"falsifier":"Retrain JOG3R with point-map targets produced by a different geometry source, such as multi-view stereo (COLMAP-style) or DUSt3R predictions refined by bundle adjustment, and measure MEt3R and pose accuracy on RealEstate10K-test; if the MEt3R advantage over pre-trained OpenSora and the mAA@30 gain over pre-trained DUSt3R vanish, the result depends on the ZoeDepth pseudo-labels. Also recompute the MEt3R comparisons using point maps from an independent estimator rather than JOG3R's own point maps for every method.","tokens_in":17038,"feed_emoji":"🎥","tokens_out":5621,"duration_ms":48733,"temperature":0.7,"pith_summary":"JOG3R asks whether video generators can be made genuinely 3D-consistent rather than merely temporally smooth. The authors find that pre-trained OpenSora features predict camera poses only weakly, and that generated frames fail a 3D-consistency check (MEt3R). Their solution is a single network that routes OpenSora's intermediate diffusion-transformer features into DUSt3R-style point-map decoders and trains the whole thing with generation and reconstruction losses together. The result is a video generator with better measured 3D consistency than the pre-trained model, without hurting visual quality, plus camera-pose estimates competitive with a dedicated pose estimator on the same data. The paper presents this as the first unified model that can generate video and regress 3D point maps and camera poses at once.","feed_headline":"Video generators can be made 3D-consistent by joint training","feed_subtitle":"JOG3R stitches video diffusion features to a 3D point-map head, improving pose estimates and consistency.","key_machinery":"The central mechanism is model stitching of a DiT video backbone to a DUSt3R point-map head: the output of STDiT block 26 is fed into DUSt3R's cross-view decoders and linear heads, which regress per-pixel 3D point maps; relative camera poses come from Procrustes alignment plus PnP-RANSAC and a global registration refined by temporal smoothness terms. The joint objective is $L_{\\text{total}} = L_{\\text{gen}} + \\lambda L_{\\text{rec}}$ with $\\lambda = 1$, where $L_{\\text{gen}}$ is the standard noise-prediction loss and $L_{\\text{rec}}$ is a confidence-weighted Euclidean error between normalized predicted and pseudo-ground-truth point maps. Only the temporal-attention weights of blocks 5 through 28 are updated; the first four blocks stay frozen, and the reconstruction decoder and heads are trained from scratch.","core_discovery":"The paper claims that a video generator and a 3D reconstructor can share one network: the DiT blocks of OpenSora are stitched to DUSt3R's decoder and point-map heads, and the combined model is trained with both the diffusion noise-prediction loss and a confidence-weighted point-map regression loss. The central discovery is that pre-trained video-generator features are only weakly 3D-aware, but joint fine-tuning makes the two tasks mutually supportive. On RealEstate10K-test, JOG3R reaches the lowest MEt3R error among the compared generators (0.0736 vs 0.0819 for pre-trained OpenSora) while keeping FID and FVD competitive, and it estimates camera poses better than pre-trained DUSt3R (mAA@30 47.25% vs 40.20%), though below DUSt3R fine-tuned on identical data (75.50%).","pith_inferences":["If the joint-training effect generalizes, diffusion video backbones could become reusable feature extractors for other geometric tasks such as depth, optical flow, and point tracking, since the stitching already yields pixels and geometry in one forward pass.","A stronger test the authors leave open is training the video DiT from scratch with both losses; their own results suggest features may become even more 3D-consistent than fine-tuning an already-trained generator.","Dynamic scenes are the natural next probe: the current evaluation is limited to static scenes, so the claim that generation and reconstruction are compatible has not yet been tested under object motion, where camera pose and scene flow are entangled."],"forward_implications":["JOG3R achieves the lowest MEt3R error (0.0736) among the compared video generators, indicating the generated videos are more 3D-consistent than pre-trained OpenSora (0.0819).","Adding the 3D point-map loss while keeping the generation loss does not degrade video quality on this benchmark: FID improves from 115.36 to 79.94 and FVD from 1872.41 to 1742.73 relative to pre-trained OpenSora.","On real RealEstate10K videos, JOG3R's camera poses beat a pre-trained DUSt3R (mAA@30 47.25% vs 40.20%), though a DUSt3R fine-tuned on the same training set remains ahead (75.50%).","The joint text-to-video-plus-camera pipeline gives camera trajectories consistent with the cascaded generate-then-estimate pipeline (average rotation difference 0.45 degrees), so the same network can output both modalities in one pass."],"supporting_citations":[{"why":"Supplies the DUSt3R architecture, including pair-wise decoders, point-map heads, and Procrustes/PnP registration, that is stitched to the video DiT.","marker":"[79]"},{"why":"OpenSora provides the DiT-based video generator whose intermediate features are routed and whose temporal-attention weights are fine-tuned.","marker":"[97]"},{"why":"MEt3R defines the warped-feature similarity score used to measure the 3D consistency of generated frames.","marker":"[2]"},{"why":"ZoeDepth generates the metric depth maps that, unprojected with RealEstate10K cameras, form the pseudo-ground-truth point maps for the reconstruction loss.","marker":"[6]"},{"why":"RealEstate10K supplies the static-scene training videos together with camera annotations and the train/test split.","marker":"[99]"},{"why":"Provides the RealEstate10K captions used as text conditioning for the generation loss.","marker":"[26]"},{"why":"DL3DV10K serves as the withheld out-of-distribution test set for camera pose generalization.","marker":"[46]"}],"fun_headline_variants":["JOG3R: unified model for 3D-consistent video generation","Joint training unlocks 3D consistency in video models","One network, two tasks: video and 3D pose estimation","Video generators go 3D with DUSt3R heads","From pixels to points: JOG3R unifies video and 3D"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The point-map supervision is built from metric depth estimated by ZoeDepth and unprojected with RealEstate10K camera parameters; if those pseudo-ground-truth 3D maps carry systematic depth errors for real-estate interiors, the reconstruction head learns those biases and the reported 3D-consistency gain may partly reflect fitting to them.","fun_headline_variants_meta":{"raw":{"variants":["JOG3R: unified model for 3D-consistent video generation","Joint training unlocks 3D consistency in video models","One network, two tasks: video and 3D pose estimation","Video generators go 3D with DUSt3R heads","From pixels to points: JOG3R unifies video and 3D"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000275,"raw_usage":{"total_tokens":1643,"prompt_tokens":948,"completion_tokens":695,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":603}},"tokens_in":564,"tokens_out":695,"duration_ms":7773,"temperature":1.0,"reasoning_tokens":603,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:28:14.950274+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain JOG3R with point-map targets produced by a different geometry source, such as multi-view stereo (COLMAP-style) or DUSt3R predictions refined by bundle adjustment, and measure MEt3R and pose accuracy on RealEstate10K-test; if the MEt3R advantage over pre-trained OpenSora and the mAA@30 gain over pre-trained DUSt3R vanish, the result depends on the ZoeDepth pseudo-labels. Also recompute the MEt3R comparisons using point maps from an independent estimator rather than JOG3R's own point maps for every method.","supporting_citations":[{"cited_title":"Dust3r: Geometric 3d vi- sion made easy","cited_arxiv_id":null,"evidence_quote":"Supplies the DUSt3R architecture, including pair-wise decoders, point-map heads, and Procrustes/PnP registration, that is stitched to the video DiT."},{"cited_title":"Open-sora: Democratizing efficient video production for all, 2024","cited_arxiv_id":null,"evidence_quote":"OpenSora provides the DiT-based video generator whose intermediate features are routed and whose temporal-attention weights are fine-tuned."},{"cited_title":"DL3DV-10K: A large-scale scene dataset for deep learning-based 3d vision","cited_arxiv_id":null,"evidence_quote":"DL3DV10K serves as the withheld out-of-distribution test set for camera pose generalization."}],"review_version":1}