{"id":"b5e4c521-b5ea-45d6-b9ea-6cad618327db","arxiv_id":"2411.19189","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A single-image latent diffusion model extended with cross-frame attention and global scale-shift alignment produces state-of-the-art video depth without a video diffusion model.","lead":"RollingDepth turns a single-image depth model into a video depth estimator by processing short frame triplets and stitching the results together with a global alignment step. On five benchmarks it reports more accurate and temporally consistent depth than video-diffusion methods, and it handles videos with hundreds of frames.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Per-snippet affine consistency is assumed by Eq 3 but never directly measured; if the snippet LDM has non-affine intra-snippet errors, co-alignment cannot correct them and the reported gain from co-alignment is fragile.","rationale":"The paper is a strong empirical contribution with a clear ablation structure: Table 3 isolates co-alignment as the dominant component, Table 2 shows the benefit of multi-scale dilation, and the method is evaluated on several diverse datasets. The central claim, however, depends on the co-alignment being able to register snippets with only per-snippet scale and shift. The reader's weakest_assumption correctly targets this dependency, but the specific mechanism offered, per-snippet inverse-depth normalization, is not the real source of risk: that normalization guarantees the training targets for different snippets are related by affine transforms, so a perfect network would satisfy the per-snippet affine model exactly. The actual risk is that the network's errors are not affine-invariant within a snippet, and the paper does not measure this directly. That risk is load-bearing because co-alignment is the main source of the reported improvement, and if the affine model is misspecified, the fused video inherits non-correctable residuals. The proposed test, comparing per-snippet vs per-frame alignment, would directly quantify whether such residuals exist. This does not overturn the reader's conditional recommendation: the method is plausible and well ablated, but the missing code, filtered test subsets, and lack of variance reporting already justify requesting changes. My concern reinforces the conditionality rather than demanding rejection, so the verdict remains conditional.","tokens_in":17516,"tokens_out":9634,"duration_ms":93159,"concrete_test":"Run the fusion on the same PointOdyssey (10-seq) and ScanNet (20-seq) subsets used in Tab. 3, replacing per-snippet (s_k,t_k) in Eq 3 with per-frame (s_i,t_i) parameters while keeping all other inference settings fixed, and report AbsRel/delta1 after the standard per-video affine fit to GT. Guard against overfitting by evaluating the per-frame fit on held-out snippets or with a small regularizer, and report both fits. If per-frame alignment improves AbsRel by more than ~1 point, the per-snippet affine assumption is measurably violated and the co-alignment is leaving accuracy on the table; if it does not improve, the assumption is supported and the central fusion mechanism is sound.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing structural assumption is in Sec 3.3: the co-alignment (Eq 3) assigns exactly one scale and one shift per snippet, shared by all frames in the snippet. The optimization can therefore correct only errors that are common to an entire snippet. If the multi-frame LDM's predictions contain intra-snippet inconsistencies that are not representable as a single affine map, e.g., errors that vary nonlinearly with depth or that depend on which other frames are in the snippet, those residuals survive the global co-alignment and are averaged into the final depth video. This is consequential because Table 3 attributes most of the accuracy improvement to co-alignment (PointOdyssey AbsRel 13.0 without vs 10.2 with), so the validity of the per-snippet affine model is the main reason the method beats single-frame baselines. The per-snippet inverse-depth normalization in Sec 3.4 ensures that the training targets of different snippets are related by affine transforms, but it does not ensure that the network's errors are affine; the reader's specific causal example is therefore imprecise. The genuine, untested risk is non-affine intra-snippet error, and no experiment in the paper isolates it.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RollingDepth, a video depth estimation method that extends the single-image latent diffusion model Marigold to short video snippets by retraining it with cross-frame self-attention and per-snippet inverse-depth normalization. At inference, snippets of three frames are sampled with a dilated rolling kernel at multiple dilation rates, processed by the snippet LDM, and merged into a consistent depth video by a global optimization that estimates one scale and shift per snippet, followed by averaging; an optional diffusion-based refinement step improves detail. The method is trained on TartanAir and Hypersim and evaluated zero-shot on PointOdyssey, ScanNet, Bonn, DyDToF, and DDAD, where it reports lower AbsRel and higher δ1 than single-frame and video-based baselines on most benchmarks, with additional temporal smoothness metrics in the supplementary.","tokens_in":17709,"tokens_out":6395,"duration_ms":51660,"significance":"If the central claims hold, the paper makes a valuable practical contribution: it shows that a single-image depth diffusion model can be turned into a state-of-the-art video depth estimator with modest additional training, avoiding the high cost of video diffusion models. The multi-scale snippet sampling and robust global co-alignment are clean, well-ablated ideas, and the evaluations cover several standard benchmarks with standard protocols. The paper also acknowledges several limitations (sky, glass, occasional failures), which strengthens credibility. However, the significance is tempered by the overbroad headline claim and the untested affine-consistency assumption that underlies the co-alignment step, both of which require attention before the paper can be accepted.","major_comments":[{"comment":"The global co-alignment assumes each snippet's depth output is related to a common scale by a single affine transform per snippet. This assumption is load-bearing because Table 3 attributes most of the accuracy gain to co-alignment (PointOdyssey AbsRel 13.0 without vs 10.2 with; ScanNet 12.4 vs 9.9). The paper never directly measures intra-snippet consistency; if the multi-frame LDM produces non-affine errors (e.g., errors that vary nonlinearly with depth or depend on the temporal context), the co-alignment cannot correct them and the fused video inherits them. I recommend adding an experiment that fits a single scale/shift per snippet to ground truth and reports the residual intra-snippet error, or that compares predictions of overlapping snippets before and after alignment to quantify the fraction of inconsistency that is affine-correctable.","section":"Sec. 3.3, Eq. (3)"},{"comment":"The claim that RollingDepth 'delivers more accurate depth videos than both dedicated video depth estimators and high-performing single-frame models' is contradicted by the Bonn row of Table 1: DepthCrafter achieves AbsRel 6.6 and δ1 96.7 versus RollingDepth's 7.9 and 93.9. The conclusion's 'best-in-class performance across multiple datasets' is likewise too strong. The paper should qualify the claim, e.g., 'on average across the evaluated benchmarks' or 'on all but the Bonn benchmark,' and discuss why the method underperforms there.","section":"Abstract, Sec. 4.3, Table 1"},{"comment":"The evaluations report point estimates without error bars or significance tests, and the PointOdyssey test set is filtered by removing toy scenes, smoke, and window pixels; the exact subset is promised only as 'will be provided with the code.' Since the PointOdyssey results are among the strongest claims, the paper should provide the selected sequence list, report per-sequence variance (e.g., interquartile ranges), and verify that the baselines are evaluated on the same filtered subset. Without this, a reader cannot assess whether the gaps are within noise or a consequence of the filter.","section":"Sec. 4.2, Table 1, Supplementary A.3"}],"minor_comments":[{"comment":"The OPW table marks RollingDepth with an asterisk for catastrophic failures on PointOdyssey, but the text only discusses failures for ChronoDepth and DepthCrafter. Please clarify whether RollingDepth also has such failures and how they were handled.","section":"Supplementary Tab. S1"},{"comment":"The main text states the fast setting takes 81 s, while Table S4 reports 105 s for the standard configuration and 81 s for the batched variant; please reconcile the numbers and specify which configuration corresponds to the 'fast' setting in Table 1.","section":"Sec. 4.4 vs Table S4"},{"comment":"The ablation tables use a subset of 10 PointOdyssey and 20 ScanNet sequences, but it is not stated whether these are the same subsets as in the main evaluation, so the relation between the ablation magnitudes and the final numbers is unclear.","section":"Sec. 4.2, Sec. 4.4"},{"comment":"The paper claims the method supports videos of 1000 frames or more, but the longest evaluated sequences are 250 frames; a scalability experiment on longer videos would substantiate this claim.","section":"Sec. 2.2 and Sec. 4.3"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"RollingDepth is a solid, careful empirical paper that convincingly shows an image-based LDM, extended to short snippets and stitched with global co-alignment, can beat video-diffusion depth models on several zero-shot benchmarks. What is actually new is the combination rather than any single ingredient: cross-frame self-attention in a Marigold-style LDM, dilated rolling snippets at multiple frame rates, and a robust optimization-based co-alignment that registers all snippets with per-snippet scale/shift. The ablations are well done and support the design decisions; the co-alignment clearly does the heavy lifting, and the optional refinement adds only marginal metric gains, which the authors honestly report. The paper also acknowledges its failure cases (sky, glass) in the supplement. That is good practice.\n\nThe soft spots are mostly about evidence quality rather than the core idea. No code or exact frame lists are released, so the PointOdyssey filtering (toy/smoke scenes excluded) is not independently reproducible yet; the authors promise the frame list with code. All tables report point estimates without variance, which matters because some comparisons (e.g., DepthCrafter on PointOdyssey) show catastrophic failures that are manually excluded in the OPW table. The concurrent DepthAnyVideo baseline is mentioned in related work but absent from the benchmark tables, which is a real gap for a SOTA claim.\n\nThe stress-test concern about per-snippet affine consistency is worth taking seriously but it is not a demonstrated flaw. The co-alignment model assumes errors within a snippet are representable as one scale and shift; if the snippet LDM produced non-affine intra-snippet errors, those would survive and degrade the final video. The paper does not directly measure intra-snippet consistency, so this is an untested structural assumption. That said, the evaluation against independent ground truth would have caught large non-affine errors, and the gains are consistent across datasets. A direct experiment comparing snippet-internal residuals against per-frame alignment would settle it and would be a good addition, but the absence does not undermine the central empirical claim.\n\nMy take: this is a strong, honest empirical contribution. It deserves a serious referee. I would ask for code and exact test splits, per-sequence variance, and the DepthAnyVideo comparison before accepting; the intra-snippet consistency test is a bonus rather than a blocker.","headline":"A strong empirical paper that shows an image LDM with snippet stitching beats video diffusion models for video depth; the main caveats are missing code, missing variance, and an untested affine-consistency assumption.","tokens_in":18316,"tokens_out":2362,"would_cite":true,"duration_ms":20834,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RollingDepth shows that a single-image latent diffusion depth model, applied to short overlapping snippets and globally co-aligned, produces video depth more accurate than dedicated video depth estimators and high-performing single-frame…","keywords":["video depth estimation","latent diffusion model","monocular depth","temporal consistency","global co-alignment","cross-frame self-attention","inverse depth","zero-shot generalization"],"falsifier":"Take a fixed frame and feed it to the snippet model in two different contexts—as the middle frame of an adjacent triplet (frames 99, 100, 101) and of a dilated triplet (frames 90, 100, 110). After optimally aligning each snippet's output with its own scale and shift, compare the two depth maps of frame 100; because per-snippet normalization renormalizes the same frame differently by context, a residual difference larger than the method's reported error would show that the single-scale-and-shift assumption fails and the global alignment cannot repair it.","tokens_in":17272,"feed_emoji":"🎥","tokens_out":11986,"duration_ms":95051,"temperature":0.7,"pith_summary":"The paper sets out to show that a single-image latent diffusion depth model can become a strong video depth estimator without training a video diffusion model. Its recipe is to process a video as many overlapping three-frame snippets, sampled at different frame spacings, with the depth model's attention mechanism operating jointly across each snippet's frames; each snippet yields its own inverse-depth prediction. A robust optimization then fits one scale and one shift per snippet so that all overlapping predictions agree, and this global co-alignment is what mainly produces the accuracy gain. The resulting system, RollingDepth, is reported to be more accurate than both dedicated video depth estimators and strong single-frame models on PointOdyssey, ScanNet, Bonn, DyDToF, and DDAD, while handling videos of hundreds of frames. If the claim holds, accurate video depth no longer requires the expensive training and stitching routines of video foundation models.","feed_headline":"A still-image depth model can beat video depth models","feed_subtitle":"RollingDepth processes frame triplets from one diffusion depth model and aligns them into consistent long videos.","key_machinery":"The central machinery is the pair formed by the dilated rolling snippet kernel and the per-snippet scale-and-shift co-alignment. The kernel selects frame triplets $(x_{i-g}, x_i, x_{i+g})$ for multiple dilation rates $g$, so every frame participates in many snippets with different temporal contexts; cross-frame self-attention inside the latent diffusion model lets depth evidence flow across the three frames of a snippet. The co-alignment step then assembles all snippets into one video by minimizing a robust L1 objective over the mean depth and mean inverse depth per frame, fitting one pair $(s_k, t_k)$ per snippet with soft constraints and stability weights (Eq. 3). Because inverse depth is normalized jointly within each snippet during training (Sec. 3.4), the same frame is represented comparably even when it appears in different contexts, which is what allows a single global scale-and-shift per snippet to register everything.","core_discovery":"The central discovery is that temporal consistency in video depth can be obtained from a single-image latent diffusion model by splitting the video into short snippets and solving a global alignment problem, rather than by learning a video prior. The multi-frame snippet model is fine-tuned from a single-image depth LDM by letting self-attention attend across the frames of a snippet; per-snippet inverse-depth normalization keeps depth ranges comparable within a snippet and lets the model handle sudden changes in depth range. Snippets are drawn with dilation rates $g \\in \\{1,10,25\\}$, so each frame appears with near and far temporal neighbors. The co-alignment then optimizes one scale $s_k$ and shift $t_k$ per snippet, minimizing an L1 discrepancy over all overlapping frames under soft constraints (Eq. 3). Ablations show that the co-alignment step is the largest contributor to the accuracy gain, while the optional diffusion refinement mostly improves visible detail.","pith_inferences":["A testable consequence of this design is that the global co-alignment could be reused with any per-snippet dense predictor that outputs affine-invariant maps, not only diffusion-based depth; if alignment is doing most of the work, a cheaper frame-wise predictor might close much of the gap on static scenes.","The method reframes video depth as largely a registration problem for per-snippet depth ranges, rather than a motion-generation problem; comparing RollingDepth against a flow-based or optimization-based registration baseline would isolate how much the cross-frame attention itself contributes.","The optional refinement, which adds moderate noise and denoises with decreasing frame spacings, acts as a learned detail enhancer; measuring its effect on long-range flicker (not just per-frame metrics) would show whether it also improves temporal consistency.","Because the same frame is normalized differently depending on the snippet it appears in, the approach may be sensitive to snippet sampling; stress-testing with extreme frame spacings or rapid scene cuts could reveal whether intra-snippet consistency, rather than global registration, becomes the bottleneck."],"forward_implications":["A 250-frame video can be processed in roughly 81 to 105 seconds on one GPU, with a constant memory footprint, because the network only ever sees short snippets; this makes the method applicable to long videos of hundreds of frames.","Dilation matters: adding a large frame spacing (25) to the base spacing (1) reduces the relative depth error by more than six percentage points on PointOdyssey and more than two on ScanNet, and an intermediate spacing (10) adds a further small gain.","Co-alignment is the decisive component: ablations show that merging snippets without it gives about 13.0 relative error on PointOdyssey, versus 10.2 with it, while the optional refinement changes the metric only marginally.","Across all reported benchmarks, RollingDepth achieves lower relative depth error and higher accuracy ($\\delta_1$) than the compared dedicated video depth estimators and single-frame models, including on the long-sequence subsets."],"supporting_citations":[{"why":"Supplies the single-image latent diffusion depth model whose self-attention is extended to snippets and which is also a comparison baseline.","marker":"[25]"},{"why":"Provides the latent diffusion architecture from which the depth LDM is derived, giving the generative prior.","marker":"[54]"},{"why":"Defines the affine-invariant evaluation protocol and the inverse-depth parametrization that RollingDepth adopts.","marker":"[52]"},{"why":"Provides the synthetic video dataset used to fine-tune the snippet LDM.","marker":"[62]"},{"why":"Provides the photorealistic single-image dataset mixed into training to improve domain diversity.","marker":"[53]"},{"why":"DepthCrafter is the strongest video-diffusion baseline; RollingDepth compares against it on long videos and on DDAD.","marker":"[24]"},{"why":"ChronoDepth is a video-diffusion baseline whose failure mode on sudden depth-range changes motivates RollingDepth's design.","marker":"[58]"},{"why":"DepthAnythingV2 is a high-performing single-frame baseline that the central accuracy claim must beat.","marker":"[72]"},{"why":"PointOdyssey is the primary dynamic-scene benchmark where the largest gains are reported.","marker":"[87]"},{"why":"ScanNet is the static indoor benchmark used for zero-shot evaluation.","marker":"[8]"}],"fun_headline_variants":["Snippet alignment gives single-image diffusion video depth","Depth videos from still-image diffusion, no video training","Optimize depth snippets, skip video priors","Turning still-image depth diffusion into video depth via alignment"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes that within any three-frame snippet the depth estimates are already mutually consistent up to one shared multiplication and addition, so the alignment needs only one scale and shift per snippet rather than per frame.","fun_headline_variants_meta":{"raw":{"variants":["Snippet alignment gives single-image diffusion video depth","Depth videos from still-image diffusion, no video training","Optimize depth snippets, skip video priors","Turning still-image depth diffusion into video depth via alignment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000675,"raw_usage":{"total_tokens":3090,"prompt_tokens":985,"completion_tokens":2105,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":2053}},"tokens_in":601,"tokens_out":2105,"duration_ms":15789,"temperature":1.0,"reasoning_tokens":2053,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:25:42.777906+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fixed frame and feed it to the snippet model in two different contexts—as the middle frame of an adjacent triplet (frames 99, 100, 101) and of a dilated triplet (frames 90, 100, 110). After optimally aligning each snippet's output with its own scale and shift, compare the two depth maps of frame 100; because per-snippet normalization renormalizes the same frame differently by context, a residual difference larger than the method's reported error would show that the single-scale-and-shift assumption fails and the global alignment cannot repair it.","supporting_citations":[{"cited_title":"Repurpos- ing diffusion-based image generators for monocular depth estimation","cited_arxiv_id":null,"evidence_quote":"Supplies the single-image latent diffusion depth model whose self-attention is extended to snippets and which is also a comparison baseline."},{"cited_title":"Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer","cited_arxiv_id":null,"evidence_quote":"Defines the affine-invariant evaluation protocol and the inverse-depth parametrization that RollingDepth adopts."},{"cited_title":"TartanAir: A dataset to push the limits of visual SLAM","cited_arxiv_id":null,"evidence_quote":"Provides the synthetic video dataset used to fine-tune the snippet LDM."},{"cited_title":"Susskind","cited_arxiv_id":null,"evidence_quote":"Provides the photorealistic single-image dataset mixed into training to improve domain diversity."},{"cited_title":"PointOdyssey: A large-scale synthetic dataset for long-term point tracking","cited_arxiv_id":null,"evidence_quote":"PointOdyssey is the primary dynamic-scene benchmark where the largest gains are reported."}],"review_version":1}