{"id":"8a0c3ec2-9778-40cc-9056-2523d8294a16","arxiv_id":"2412.08975","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A decoupled video inpainting framework that combines Stable Diffusion reference generation with one-shot flow-traced pixel pulling outperforms prior methods and scales to 2K video.","lead":"This paper introduces RGVI, a video inpainting system that removes objects by generating missing content with Stable Diffusion and then propagating that content across frames with a one-shot pixel pulling method. It also contributes HQVI, a high-resolution benchmark for realistic object removal and text-guided generation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Propagation verification threshold is too permissive to catch flow-completion errors; RGVI's superiority may depend on unvalidated optical-flow accuracy.","rationale":"The reader's weakest assumption—that completed optical flows are accurate enough for one-shot pixel pulling—is precisely the load-bearing point. The paper's core mechanism is a decoupled propagation that warps flows rather than colors, and its verification step is the only guard against flow errors. However, the threshold of 1 on normalized per-pixel L1 difference is extremely permissive: it only rejects pulls where the two directions differ by more than 0.33 per channel on average. In a textured or high-frequency region, a misalignment of even a few pixels will often produce a color difference below this threshold, meaning the verification fails to catch exactly the silent misalignments the paper claims it detects. This is not merely a statistical issue; it is a structural weakness in the robustness argument. The controlled ablation in Table 3 is informative but only shows that one-shot warping beats recurrent warping given the same flow inputs—it says nothing about whether those flow inputs are trustworthy. Since HQVI is constructed by compositing foregrounds onto known background videos, the ground-truth background is available, making it straightforward to compute ideal background flows and compare. If the performance gap between using GT flows and ProPainter-completed flows is large, then the method's state-of-the-art claim is inherited from the flow completion module rather than from the proposed propagation algorithm, and the verification threshold would need to be substantially tightened to make the method robust. I agree with the reader's CONDITIONAL verdict; the concern does not warrant rejection, but it does require empirical settling before the central claim can be taken at face value. I chose UNCHANGED because the reader already identified flow dependence as the key assumption, and my analysis sharpens that point without altering the verdict.","tokens_in":11166,"tokens_out":6976,"duration_ms":71383,"concrete_test":"On the HQVI dataset, compute RAFT optical flows from the ground-truth background videos and use them as the completed flows in RGVI's propagation (with the same masks and per-frame completion), keeping all other settings identical. Rerun the full pipeline and compare PSNR/SSIM/LPIPS against the paper's reported RGVI (w/o Ref.) numbers at 240p and 480p. If PSNR improves by more than 1 dB or LPIPS by more than 0.01, then flow-completion errors are the dominant error source and the verification's permissive threshold fails to catch them, making the central claim conditional on flow accuracy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that RGVI significantly outperforms all existing methods rests on the one-shot pixel pulling and its verification. Section 3.2 completes masked flows with ProPainter's recurrent protocol, but the paper does not evaluate the accuracy of these completed flows. The verification step compares forward and backward pulls using an L1 color threshold of 1 on normalized [0,1] values; this invalidates only pixels where average per-channel disagreement exceeds 1/3, so most silent misalignments from imperfect flows will pass as 'reliable.' The ablation in Table 3 compares one-shot vs recurrent warping under the same flows, but cannot reveal whether the flows themselves are good enough. Since HQVI provides ground-truth background, this dependency is testable. If RGVI's output with ground-truth flows is substantially better than with its completed flows, then flow completion, not propagation, is the bottleneck, and the claim of state-of-the-art propagation is contingent on an external component that is not adequately validated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"RGVI is a flow-guided video inpainting system that decouples content propagation from content generation. It completes masked optical flows with ProPainter's recurrent protocol, propagates known pixels by chaining flows and pulling colors once per target pixel, verifies forward/backward pulls with an L1 color threshold, generates a key-frame reference with Stable Diffusion under user text prompts, propagates the reference, and finishes with per-frame image inpainting. The paper introduces HQVI, a 1200x2160 benchmark built by alpha-matte compositing, and reports improved PSNR/SSIM/LPIPS/VFID over prior methods on HQVI plus DAVIS and YouTube-VOS, together with a 10-participant user study.","tokens_in":11312,"tokens_out":8545,"duration_ms":85352,"significance":"The one-shot pulling idea is the most distinctive contribution: warping flow maps rather than color values is a simple way to avoid re-sampling artifacts while retaining sub-pixel accuracy, and the controlled visual comparison in Figure 6 supports this benefit. The decoupled use of Stable Diffusion on a single key frame, with text-prompt controllability, is practical and scales to 2K resolution, where several baselines fail. The HQVI benchmark addresses a real gap in realistic, high-resolution object-removal evaluation. The main limitations are that the quantitative evidence rests heavily on a self-constructed benchmark without statistical testing, and that flow-completion accuracy and the verification threshold are not directly validated; these issues weaken but do not invalidate the central claim.","major_comments":[{"comment":"The verification rule accepts a forward/backward pull when the L1 distance of the three normalized color channels is below 1, i.e., an average per-channel disagreement of up to 1/3 is treated as reliable. The paper states that this threshold shows 'minimal observed variation' but provides no ablation or statistics on the fraction of invalidated pixels, so the threshold's robustness is not demonstrated. Since the verified one-shot pulling is the core propagation contribution and Table 3 compares it against recurrent warping as a whole, the benefit of one-shot pulling is not separated from the benefit of the threshold rule.","section":"Section 3.2, verification threshold"},{"comment":"The completed optical flows are never evaluated. The paper adopts ProPainter's recurrent flow completion and then relies on these flows for one-shot pulling, but the ablation in Table 3 holds the flows fixed and varies only the propagation protocol. Because HQVI contains ground-truth background frames, an experiment replacing completed flows with flows estimated from the ground-truth background would directly quantify how much of RGVI's performance depends on flow-completion accuracy. Without such an experiment, the claimed superiority of the propagation module is contingent on an external component whose error behavior is unmeasured.","section":"Section 3.2 and Table 3"},{"comment":"No error bars, confidence intervals, significance tests, or per-sequence statistics are reported for any quantitative comparison, yet the abstract and Section 5.1 claim RGVI 'significantly outperforms' all existing methods. Some reported gaps are small (e.g., 240p PSNR 30.66 vs. 30.62 for RGVI vs. ProPainter, and LPIPS 0.0390 vs. 0.0388 for RGVI without reference vs. ProPainter), and the user study in Table 5 is based on average ranks from 10 participants without any statistical test. The evidence is therefore weaker than the language of the claims.","section":"Section 5.1, Tables 1-2 and Section 5.3, Table 5"},{"comment":"The quantitative conclusions for object removal are drawn primarily from HQVI, a dataset constructed by the authors, but the paper does not state whether HQVI will be released and provides no details on dataset size, sequence lengths, or the number of occluding-object sequences. The abstract and GitHub link mention code and models only. If HQVI is not made available, the main benchmark result is not reproducible; if it is released, the paper should document the composition statistics and any selection criteria used when choosing Pexels background videos.","section":"Section 4, HQVI dataset"}],"minor_comments":[{"comment":"The symbol f_i->j is used both for the completed adjacent-frame flows and for the chained long-range flows, and the warping operation w(A,B) is not defined precisely; please clarify which argument is the displacement field and which is the quantity being warped, and state the direction convention.","section":"Equation (1)"},{"comment":"The text writes 'DA VIS 2016' but the dataset is normally called DAVIS 2016; please standardize the spelling and define the 'DA VI' and 'YTVI' abbreviations at first use.","section":"Section 5.1"},{"comment":"For the recurrent warping baseline in Table 3, the sentence 'the initial frame is used as the key frame since connection counts cannot be calculated' should be expanded to explain why the proposed key-frame selection is inapplicable to recurrent warping.","section":"Section 5.3"},{"comment":"The pipeline diagram is very small and the four components are hard to read; please enlarge the figure or restructure it so that the flow completion, internal propagation, reference generation, and per-frame completion stages are visually distinct.","section":"Figure 2"},{"comment":"The positive mask is user-provided; the paper should state how this mask is obtained in practice and confirm that it was not used in the main comparisons of Tables 1 and 2, so that the reported improvements are not dependent on an additional input that baselines do not receive.","section":"Section 3.5"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid systems paper with a simple, well-motivated propagation trick and a useful high-resolution benchmark. My main concern for the editor is that the headline claim of 'significantly outperforms' is made on a self-constructed benchmark without statistical support, and the two load-bearing components (flow completion and verification threshold) are under-validated. I would not reject; the method is plausibly state-of-the-art, and the requested experiments are feasible within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real news here is the propagation mechanism: instead of recurrently warping color values, they warp completed flow maps and pull colors from source frames in one shot. That is a clean idea, and the controlled ablation in Table 3 supports it—one-shot pulling beats recurrent warping under the same flows on every metric. The Stable Diffusion reference generation is a sensible decoupled addition, and the HQVI benchmark with alpha-matte compositing, 2K resolution, and occlusion-specific masks is a genuine contribution. The paper also ships code and models, and the citation pattern looks appropriate. This is a well-executed systems paper.\n\nThe soft spots are mostly about evidence strength rather than correctness. There are no error bars or significance tests anywhere, so the phrase \"significantly outperforms\" is doing work the data do not formally support. The user study is 10 participants with no variance reported. HQVI is self-constructed; that is not fatal, since DAVIS and YouTube-VOS are also used, but the primary benchmark being the authors' own makes the lack of statistical rigor more noticeable.\n\nThe stress-test concern about flow completion is real but not disqualifying. The paper never measures how accurate the completed flows are, and the verification step only catches forward/backward disagreement, not silent misalignment. Because HQVI has ground-truth background, the authors could easily compare their pipeline with completed flows versus ground-truth flows. That experiment would separate propagation quality from flow-completion quality and would settle the matter. The Limitations section does acknowledge reliance on optical flows, so this is a known gap, but it is testable and should be done.\n\nOne nuance: in Table 1, RGVI without reference sometimes has higher PSNR/SSIM than RGVI with reference at lower resolutions. The authors attribute this to blurry textures scoring well on PSNR, which is plausible. It is not a flaw, but it means the \"with reference\" advantage is mostly perceptual, so the generation component should be judged accordingly.\n\nOverall, this paper deserves a serious referee. It is a strong candidate for acceptance after adding variance estimates, a larger user study, the GT-flow experiment, and ideally a public release of the HQVI dataset. I would take it to our reading group and would likely cite the propagation method and benchmark in my own work.","headline":"A solid, well-engineered flow-based video inpainting system with a genuinely new one-shot pulling mechanism and a useful benchmark; the evaluation is good but overclaims significance and leaves the flow-completion dependency untested.","tokens_in":11845,"tokens_out":2645,"would_cite":true,"duration_ms":29205,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that video inpainting improves by decoupling known-pixel propagation from generative filling, and reports the best object-removal scores on a new 2K benchmark.","keywords":["video inpainting","flow-guided inpainting","one-shot pixel pulling","reference generation","Stable Diffusion","object removal","HQVI benchmark","optical flow propagation"],"falsifier":"Take a video in which a slowly moving target is occluded by a foreground object whose motion matches the background, so that flow completion plausibly aligns to the wrong texture; run RGVI with and without the positive mask and check whether the pulled pixels match the visible background at sub-pixel accuracy, and whether verification falsely accepts the wrong alignments. A cleaner quantitative version is to compare RGVI's propagation output on frames with known ground-truth optical flows, isolating how much error comes from flow completion rather than from the propagation or generation stages.","tokens_in":10969,"feed_emoji":"🎬","tokens_out":7735,"duration_ms":71913,"temperature":0.7,"pith_summary":"RGVI claims that video inpainting is best done by decoupling content propagation from content generation: first move known pixels between frames, then let a large generative model fill only what remains. The paper introduces one-shot pixel pulling, which chains completed optical flows across frames and warps flow maps rather than color values, so each target pixel is sampled exactly once and keeps sub-pixel precision without re-sampling artifacts. A Stable Diffusion reference generator fills the remaining holes in a single key frame chosen by connection count, and the generated pixels are propagated to every other frame through the same flow-warping operation. On the new HQVI benchmark, built by alpha-matte compositing at 2K resolution, RGVI reports the best object-removal scores among all compared methods, best perceptual quality with reference generation, and practical memory and runtime on 2K video.","feed_headline":"RGVI outscores prior methods on realistic video inpainting","feed_subtitle":"Warping flows instead of colors avoids artifacts and lets text prompts drive the final fill.","key_machinery":"The load-bearing object is the one-shot pixel pulling protocol built on chained optical flows. Flows are computed with RAFT, erased inside the mask, completed with ProPainter's recurrent flow completion, and chained across frames by grid-warping the flow maps themselves: $f_{i\\to j}=f_{i\\to j-1}+w(f_{j-1\\to j},\\,f_{i\\to j-1})$. The resulting correspondence map lets each target pixel pull its color once from the nearest source frames, avoiding the repeated sampling that blurs recurrent warping. A bi-directional forward/backward pass fills each target frame, and an L1 color-difference check between the two directions flags unreliable pixels for later stages. Reference generation then selects the key frame with the highest connection count $C_i=\\sum_j\\sum_p [w(\\hat{M}_j,f_{i\\to j})\\odot \\hat{M}_i]$, fills it with Stable Diffusion under a text prompt, propagates the generated content through the same flow-warping operation, and a lightweight convolutional network completes the residual holes.","core_discovery":"The paper's central claim is that a decoupled flow-guided pipeline composed of one-shot pixel pulling plus generative reference filling is the most effective current recipe for video inpainting. In the object-removal setting on HQVI, RGVI without reference achieves the highest PSNR and SSIM among all compared methods at 240p and 480p, RGVI with reference achieves the best LPIPS and VFID, and at 1200x2160 the reference-augmented version wins on all four metrics. On the DAVIS and YouTube-VOS restoration benchmarks, RGVI without reference matches or exceeds the previous best flow-guided method. The paper attributes these results to warping optical flows instead of colors, which preserves sub-pixel accuracy while avoiding the error accumulation and texture blur of recurrent pixel warping, and to using a strong latent-diffusion generator only for the residual holes.","pith_inferences":["My inference: because reference generation is a separate module, a stronger or more specialized image inpainting model could replace Stable Diffusion without retraining the propagation stage, so future image-inpainting advances should carry over to video inpainting almost directly.","My inference: the same flow-chaining and one-shot pulling scheme could benefit other video correspondence tasks such as frame interpolation, video object segmentation, or long-range matching, where sub-pixel accuracy and temporal consistency matter.","My inference: the verification step could be upgraded from color agreement to a learned confidence map, which might detect structural misalignment even when forward and backward flows agree on a wrong location.","My inference: HQVI's positive/negative mask annotations make it possible to benchmark occlusion handling separately from generation, which current datasets do not isolate."],"forward_implications":["On HQVI, RGVI outperforms all compared methods in the object-removal setting, with reference generation giving the best perceptual scores at 240p/480p and the best PSNR at 2K resolution.","One-shot pulling of colors via warped flow maps removes the re-sampling artifacts of recurrent pixel warping, so flow-guided inpainting can preserve fine texture over long propagation distances.","Decoupling generation from propagation makes video inpainting text-controllable: the same framework removes objects in the default mode and synthesizes new content from user prompts in generation mode.","The additional positive-mask trick reduces color-bleeding artifacts in occlusion scenarios by keeping the occluding object's original pixels intact.","HQVI provides a 2K, alpha-matte-composited benchmark with negative/positive masks and generation-required cases, which the paper presents as a more realistic evaluation setup than random masks."],"supporting_citations":[{"why":"Supplies RAFT optical flows that the pipeline erases inside the mask and then completes.","marker":"Teed and Deng 2020"},{"why":"Provides the recurrent flow completion protocol and the ProPainter baseline that RGVI compares against.","marker":"Zhou et al. 2023"},{"why":"Provides the Stable Diffusion latent diffusion model used for reference generation in both removal and generation modes.","marker":"Rombach et al. 2022"},{"why":"Documents the sub-pixel accuracy loss of per-pixel flow tracing and serves as the FGVC baseline.","marker":"Gao et al. 2020"},{"why":"Implements the recurrent pixel warping approach with re-sampling artifacts that one-shot pulling is designed to replace.","marker":"Li et al. 2022"},{"why":"Early flow-guided video inpainting baseline whose recurrent warping strategy is compared in the propagation analysis.","marker":"Xu et al. 2019"},{"why":"Supplies the DAVIS dataset used for the video restoration benchmark and qualitative object-removal comparisons.","marker":"Perazzi et al. 2016"},{"why":"Supplies YouTube-VOS images for training the per-frame completion network and the YTVI restoration benchmark.","marker":"Xu et al. 2018"},{"why":"Supplies the alpha mattes used to compose foreground objects onto background videos for the HQVI benchmark.","marker":"Lin et al. 2021"},{"why":"Provides the mask-dilation protocol and the 29 sampled videos used for the user study.","marker":"Huang et al. 2016"}],"fun_headline_variants":["RGVI beats prior methods in realistic video inpainting","Flow-guided reference generation wins video inpainting benchmarks","Warp flows, not colors, for error-free video inpainting","Text-prompted fill boosts video inpainting performance","One-shot flow pulling avoids error accumulation in video inpainting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the completed optical flows point to the correct source pixels: if a finished flow is silently wrong, one-shot pulling copies the wrong color, and the verification step only catches disagreement between the forward and backward passes, not misalignment on which both directions agree.","fun_headline_variants_meta":{"raw":{"variants":["RGVI beats prior methods in realistic video inpainting","Flow-guided reference generation wins video inpainting benchmarks","Warp flows, not colors, for error-free video inpainting","Text-prompted fill boosts video inpainting performance","One-shot flow pulling avoids error accumulation in video inpainting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00059,"raw_usage":{"total_tokens":2743,"prompt_tokens":894,"completion_tokens":1849,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":1765}},"tokens_in":510,"tokens_out":1849,"duration_ms":16210,"temperature":1.0,"reasoning_tokens":1765,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:20:47.405073+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a video in which a slowly moving target is occluded by a foreground object whose motion matches the background, so that flow completion plausibly aligns to the wrong texture; run RGVI with and without the positive mask and check whether the pulled pixels match the visible background at sub-pixel accuracy, and whether verification falsely accepts the wrong alignments. A cleaner quantitative version is to compare RGVI's propagation output on frames with known ground-truth optical flows, isolating how much error comes from flow completion rather than from the propagation or generation stages.","supporting_citations":[{"cited_title":"C.; and Loy, C","cited_arxiv_id":null,"evidence_quote":"Provides the recurrent flow completion protocol and the ProPainter baseline that RGVI compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the sub-pixel accuracy loss of per-pixel flow tracing and serves as the FGVC baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the DAVIS dataset used for the video restoration benchmark and qualitative object-removal comparisons."},{"cited_title":"B.; Ahuja, N.; and Kopf, J","cited_arxiv_id":null,"evidence_quote":"Provides the mask-dilation protocol and the 29 sampled videos used for the user study."}],"review_version":1}