Pith. sign in

REVIEW 3 major objections 6 minor 64 references

Grounding Degradations in Natural Language for All-In-One Video Restoration

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that per-frame natural-language degradation descriptions can be distilled into a lightweight prompt generator, so an all-in-one video restoration network can be deployed with no multimodal language model or text encoder…

desk verdict Solid all-in-one video restoration with practical offline distillation and useful new benchmarks, but the causal role of the language grounding is not isolated by the ablations. read the letter →

arxiv 2507.14851 v1 pith:YBVY2DY4 submitted 2025-07-20 cs.CV cs.AIcs.LGeess.IV

Muhammad Kamran Janjua , Amirhosein Ghasemabadi , Kunlin Zhang , Mohammad Salameh , Chao Gao , Di Niu This is my paper · ORCID
classification cs.CVcs.AIcs.LGeess.IV
keywords all-in-onevideorestorationlanguage-groundedpromptsmultimodallargelanguagemodelsknowledgedistillationtime-varyingdegradationsSnowyScenesbenchmarkbenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the degradation information needed to restore a video can be captured in natural language during training and then distilled into the restoration network so thoroughly that the language model is no longer needed at inference. The proposed method, RONIN, queries a multimodal large language model (MLLM) to describe the degradations in each frame, converts those descriptions into frozen text embeddings, and trains a small prompt generator to approximate them while learning to restore the frame. A sympathetic reading of the results is that this language grounding gives the network interpretable per-frame conditioning, lets it handle composite and time-varying degradations without knowing what they are, and outperforms prior all-in-one image and video restoration methods on the benchmarks tested. The paper also standardizes all-in-one video restoration evaluation with 3D and 4D benchmarks plus a new SnowyScenes benchmark with time-varying snow.

What carries the argument

The load-bearing device is the prompt approximation objective (Eq. 3), which combines a restoration loss with a small L1 loss forcing the generated prompt $P_t$ to match the frozen text embedding $e_t(C_t)$ from the grounded description. The prompt itself is produced from latent frame features by a two-layer projection after global average pooling, with cross-attention from the first encoder stage to reintroduce degradation information, and it is injected into the last two decoder stages as a sigmoid-gated per-channel soft mask. This setup separates training support from the deployed network: the language pipeline is only needed offline, while inference runs on the prompt generator and U-Net-style backbone alone.

What would settle it

Train RONIN identically but replace each frozen grounded text embedding with a random or permuted embedding from the same text encoder; if PSNR/SSIM on 3D and 4D stay at RONIN levels, then the language grounding itself is not carrying the reported gains. A complementary check is to feed deliberately wrong descriptions (for example, noise captions for deblurring frames) and see whether performance drops roughly as much as the paper's prompt-perturbation experiment suggests.

Watch

Extended reading notes

Core claim

RONIN grounds each degraded frame in natural language: a multimodal large language model (Q-Instruct) produces a per-frame description of image quality and degradations, a text encoder converts it into an embedding, and during training the restoration network must generate a prompt that is close to that embedding while also restoring the frame. At inference, the MLLM and text encoder are removed, and the prompt generator alone conditions the decoders through a per-channel soft mask. The paper's central claim is that this distilled language grounding is what allows one network to handle blur, noise, rain, snow, and time-varying mixtures of them, with no degradation assumption at train or test time and no extra inference cost, and that it achieves state-of-the-art PSNR/SSIM on 3D, 4D, TUD, and SnowyScenes benchmarks.

Load-bearing premise

The largest assumption is that the multimodal language model's per-frame degradation descriptions are accurate enough that their frozen text embeddings are a trustworthy training target; the paper itself counts misclassifications (noise and blur appearing in descriptions for desnowing and deblurring clips), and if those descriptions are systematically wrong, the prompt approximation loss may train prompts toward misleading language and the interpretability benefit would weaken.

Editorial extensions

If this is right

  • All-in-one video restoration can be deployed without an MLLM or text encoder at inference, removing per-frame language querying and its cost.
  • Because conditioning is per frame and text is generated automatically, the method can handle composite degradations and time-varying degradation without any label of which degradation is present.
  • The 3D, 4D, TUD, and SnowyScenes results imply that a single network can restore blur, noise, rain, and snow with better or comparable PSNR/SSIM than prior image and video restoration baselines.
  • The proposed SnowyScenes benchmark extends time-varying degradation evaluation to weather conditions, giving a common testbed for future methods.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if the improvement persists when the grounded text embeddings are replaced by random embeddings from the same encoder, that would suggest the gains come from the input-conditioned prompt structure rather than the language grounding itself; the paper's own misclassification table makes this a live distinction.
  • If language grounding is what carries the gain, the same training-time distillation could transfer to other conditioning signals such as region-level quality descriptions or expert-written degradation templates.
  • A testable extension would be to corrupt the MLLM descriptions (for example, feeding noise descriptions to deblurring frames) and measure the drop in PSNR; the paper predicts a substantial drop, whereas a pure blackbox-prompt account predicts little change.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper presents RONIN, an all-in-one video restoration framework. At training time, the multimodal LLM Q-Instruct generates per-frame degradation descriptions, a frozen text encoder (BGE-Micro-v2) converts them into embeddings, and a lightweight prompt generator (Eq. 1) is trained with an L1 loss (Eq. 3) to approximate those embeddings from the latent features of a Turtle-style U-Net. The resulting prompt is injected into the last two decoder stages through a learned soft mask. At inference, only the restoration network and prompt generator remain, with no MLLM or text encoder. The authors assemble and evaluate on four benchmarks—3D (deblur/denoise/derain), 4D (adds desnow), TUD (time-varying noise/blur/compression), and a new SnowyScenes benchmark with time-varying snow—and report PSNR/SSIM improvements over several image/video restoration baselines, together with qualitative results, prompt-perturbation tests, t-SNE, and cosine-similarity analyses.

Significance. If the central claim is established, this is a useful and timely contribution: the design decouples MLLM/text-encoder use from inference while preserving interpretable, language-style conditioning, and the proposed SnowyScenes benchmark addresses an underexplored time-varying weather setting. The paper is also transparent about annotation failures (Tab. 10) and limitations (Sec. 11), and the benchmark-standardization effort is valuable. However, the causal role of the language-grounding term is not isolated by the current experiments, and the 'no degradation knowledge' framing is in tension with the fixed candidate-degradation prompt procedure in Algorithm 1.

major comments (3)
  1. [§3.3, Eq. (3), Tab. 5] The paper does not isolate the language-grounding term. The prompt P_t is generated from latent features plus first-encoder cross-attention (Eq. 1, Fig. 1), so it could carry degradation information even if the text-embedding target e_t(C_t) were removed. The 'No Prompt' row in Tab. 5 only disables decoder injection; it keeps the prompt-generation path, and therefore cannot distinguish a learned blackbox-style prompt from a language-grounded one. The cosine-similarity and perturbed-prompt analyses in Appendix 7 show the prompt is informative and necessary, but any input-conditioned prompt trained only by restoration loss would also be informative and necessary. Given that Tab. 10 shows noisy targets (e.g., 13,471 noise mentions and 10,549 blur mentions in desnow frames), and λ2=0.01 is small, the text term may even be near-neutral. Please add (i) a λ2=0 baseline, (ii) a shuffled or mismatched text-target control, and (iii) a non-text control target (e.g., fixed random embeddings) to demonstrate that the language-aligned target, rather than the prompt mechanism, causes the reported gains.
  2. [§10, Algorithm 1; §3.1] The abstract and Tab. 1 claim RONIN assumes no degradation knowledge at train or test time, but the annotation procedure in Algorithm 1 explicitly queries Q-Instruct with a fixed set of candidate degradations and thresholded Yes/No intensity questions. This means the benchmark's degradation vocabulary is known a priori and hard-coded into the training-data pipeline; a degradation outside the candidate set would not be queried and might not appear in the open-ended initial description. The claim should be qualified to 'no per-sample degradation labels at inference,' or the authors should provide evidence that the open-ended initial description alone recovers all relevant degradations, including those not in the candidate list.
  3. [Tab. 2, §4.1] AverNet appears in the 3D block of Tab. 2 but is absent from the 4D block without any explanation. Since AverNet is one of the two video-specific baselines and the 4D benchmark is a central contribution, this omission makes the 'state-of-the-art on all benchmarks' claim incomplete. Please add the missing results, or explicitly state and justify why they are not reported, and keep the method list consistent across benchmark blocks.
minor comments (6)
  1. [§3.2, Eq. (1)] Eq. (1) has an unmatched parenthesis and the dimension 'Rb×d' should be written as R^{b×d}; please fix the notation.
  2. [Fig. 6] The Fig. 6 caption says 'all tasks of the 4D benchmark,' but the figure shows SnowyScenes results; correct the caption.
  3. [Tab. 2] In Tab. 2, the 3D block appears to include four task columns, including Desnow, although Sec. 4.1 defines 3D as three tasks; some rows also lack an Average entry. Please align the columns with the benchmark definitions and report averages consistently.
  4. [§10.1, Tab. 10] Tab. 10 is labeled 'Robustness Analysis' but reports only counts of degradation mentions in the MLLM text; rename it to something like 'Degradation Mention Statistics' and move the actual robustness evidence to an experiment that varies target noise or compares cleaned versus uncleaned targets.
  5. [Tab. 7] Tab. 7 uses the same asterisk symbol for two different exclusions (optical flow and text encoder); use distinct markers and clarify the note.
  6. [§4.2] In Sec. 4.2, the +0.23 dB improvement over AverNet holds for Set8; on the DAVIS test set RONIN is below AverNet on PSNR, so the abstract's unqualified 'state-of-the-art on all benchmarks' should be qualified by metric and test split.

Circularity Check

1 steps flagged · score 2.0 of 10

RONIN's training is a standard frozen-text-embedding distillation; the only circular-by-construction step is using the Eq. 3 training target as evidence that learned prompts are meaningful.

  1. self definitional [Supplementary Sec. 7.1, 'Are Learned Prompts Meaningful?'; main text Eq. (3)]
    "We find that in the former case, trained prompts align closely with raw text embeddings (similarity scores in range of 0.9852–0.9914), while random prompts do not (similarity scores in range of -0.0393–0.0370)."

    The reported cosine similarity is exactly the quantity minimized by the prompt-approximation loss λ2(1/N)Σ∥et(Ct)−Pt∥ in Eq. (3): after L1 training, Pt is driven toward et(Ct), so high cosine similarity (0.985–0.991) is an expected by-construction outcome, not independent confirmation that the prompts are language-grounded. The paper uses this alignment, together with tSNE separation, to argue that learned prompts are meaningful and necessary; however, because Pt is generated from input features via Eq. (1), the tSNE separation could also arise from the input-conditioned prompt mechanism alone, independently of the text target. The causal claim that the text-embedding target (rather than the prompt module itself) drives the reported gains is never isolated — Tab.

full rationale

The overall derivation is not circular: RONIN's training target et(Ct) is a frozen BGE-Micro-v2 embedding of Q-Instruct's per-frame descriptions, computed externally and stored offline; Eq. (3) minimizes an L1 distance between this target and a learned prompt, which is standard distillation. The restored video is supervised against VGT, the ground-truth frames, independent of any prompt target. State-of-the-art claims are tested on external benchmarks (3D, 4D, TUD, SnowyScenes) against re-run baselines, so the performance result is not read off from the model's own assumptions. The paper does cite its own prior Turtle backbone, but only as an architectural starting point, not as a premise that forces the conclusion; no uniqueness theorem or unverified self-citation is load-bearing. The one genuinely self-referential element is the supplementary 'prompts are meaningful' validation, which reports cosine similarity to the very et(Ct) vectors that Eq. (3) forced Pt to match; that is circular as evidence for language grounding, though it is not what produces the benchmark numbers. The paper's own limitation (Sec. 11 and Tab. 10) that Q-Instruct descriptions contain misclassified noise/blur raises a correctness risk for the language-grounding premise, but a noisy external target is not a circular reduction.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central method rests on the MLLM and text encoder as external components and on a candidate degradation list. No new physical entities are introduced. The only fitted scalar is the prompt loss weight, which is manually set.

free parameters (1)
  • prompt_approximation_weight_lambda2 = 0.01
    Balancing coefficient for the prompt approximation loss in Eq. 3. Chosen by hand with no sensitivity analysis reported; lambda1 is fixed to 1.0.
assumptions (3)
  • domain assumption Q-Instruct MLLM provides accurate per-frame degradation descriptions.
    Invoked in Sec. 3.1 and Algorithm 1; the language grounding pipeline relies on the MLLM correctly identifying degradations and intensities. The authors acknowledge misclassifications in Tab. 10, so this is an assumption, not a guarantee.
  • domain assumption The frozen text embedding et(Ct) is a learnable target that captures degradation information useful for restoration.
    The prompt approximation loss in Eq. 3 assumes that matching BGE-Micro-v2 embeddings of the MLLM descriptions is a useful objective. This is standard distillation but unproven as the optimal representation for the downstream task.
  • ad hoc to paper A fixed candidate degradation set is queried in Algorithm 1.
    Algorithm 1 loops over a fixed set of degradation types, which requires prior knowledge of possible degradations and contradicts the paper's 'no degradation knowledge' claim. This design choice is not derived from first principles.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Grounding Degradations in Natural Language for All-In-One Video Restoration." pith.science (2026). https://pith.science/paper/YBVY2DY4

@misc{pith2026250714851,
  author       = {Pith},
  title        = {Pith review of: Grounding Degradations in Natural Language for All-In-One Video Restoration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YBVY2DY4}},
  note         = {Machine review of arXiv:2507.14851}
}
read the original abstract

In this work, we propose an all-in-one video restoration framework that grounds degradation-aware semantic context of video frames in natural language via foundation models, offering interpretable and flexible guidance. Unlike prior art, our method assumes no degradation knowledge in train or test time and learns an approximation to the grounded knowledge such that the foundation model can be safely disentangled during inference adding no extra cost. Further, we call for standardization of benchmarks in all-in-one video restoration, and propose two benchmarks in multi-degradation setting, three-task (3D) and four-task (4D), and two time-varying composite degradation benchmarks; one of the latter being our proposed dataset with varying snow intensity, simulating how weather degradations affect videos naturally. We compare our method with prior works and report state-of-the-art performance on all benchmarks.

Figures

Figures reproduced from arXiv: 2507.14851 by the authors.

Figure 1
Figure 1. Bird’s-Eye View of RONIN and Assorted Examples. We visualize RONIN’s architecture in (a) along with a few grounded degradation examples in (b) and restoration results in (c). The grounded degradations in (b) are highlighted to emphasize the text that describes the degradations and quality of the image/frame. The restoration frames in (c) are taken from video deraining, video deblurring, and video desnowing tasks, re… view at source ↗
Figure 2
Figure 2. Word Cloud of Different Benchmarks. We visual￾ize word cloud of per-frame language descriptions generated from Q-Instruct [49] for three benchmarks, i.e., 3D, TUD [55] and our proposed SnowyScenes. We also plot the prompt approxima￾tion loss during training (bottom right) to verify that the optimiza￾tion procedure converges. This procedure is similar in spirit to several prompt injec￾tion modules whose goal is to co… view at source ↗
Figure 3
Figure 3. Visual Results on 3D Benchmark. We qualitatively compare three prior methods with RONIN on all tasks of the 3D benchmark. The first row contains frame crops from denoising video, while the second and third row contain frames crops from deblurring and deraining videos, respectively. Notice how RONIN’s outputs are visually pleasing e.g., the person in the back on the horse and the folded leg of the brown horse in the … view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Visual Results on 4D Benchmark. We qualitatively compare three prior methods with RONIN on all tasks of the 4D benchmark. The first row contains frame crops from denoising video, while the second, third, and fourth row contain frames crops from deblurring, deraining, a…
Figure 5
Figure 5. Figure 5: Samples from the SnowyScenes Benchmark. We present three frames from three different videos in the proposed SnowyScenes dataset. The first column includes frames sam￾pled from early in the video, while the second and third columns include frames from the middle and end…
Figure 6
Figure 6. Figure 6: Visual Results on SnowyScenes Benchmark. We qualitatively compare three prior methods with RONIN on all tasks of the 4D benchmark. The first row contains frame crops from t = 6 video, while the second and third rows contain frame crops from t = 12 and t = 24 videos, re…
Figure 7
Figure 7. Figure 7: Differences in Grounding Degradations and Instructions. We sample two frames (at different timesteps) from two different videos of SnowyScenes benchmark and compare RONIN’s language grounded descriptions with InstructIR [6]’s human-aligned instruc￾tions. Since Instruct…
Figure 8
Figure 8. Figure 8: tSNE Plot. Visualization of learned and untrained prompts taken from the latent space of RONIN on 4D benchmark. Methods Denoise Deblur Derain MACs (G) Params InstructIR [6] 0.1799 0.1444 0.0623 133.73∗ 73.9 ∗M PromptIR [36] 0.1793 0.1293 0.0578 158.49 35.6M ViWSNet [52…
Figure 9
Figure 9. Figure 9: Samples of Degradations Descriptions. A few samples of frames and their respective grounded degradation prompts taken from different benchmarks. In the first column, from top to bottom, the frames are taken from SnowyScenes (moderate snow), SnowyScenes (severe snow), 3…
Figure 10
Figure 10. Figure 10: Illustration of Limitation in Grounded Degradations. Two samples of language descriptions where extraneous degradations are present. The first frame is taken from a desnowing task video, but the prompt describes noise and blur. Although the frame has slight blur and a…
Figure 11
Figure 11. Figure 11: TUD Benchmark Visual Results. Qualitative results of RONIN on the TUD benchmark on three different settings. The first row contains frames from t = 6 test set, while second and third row contains frames from t = 12 and t = 24 test sets, respectively. RONIN’s outputs a…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

64 extracted references · 45 canonical work pages

  1. [1]

    Basicvsr++: Improving video super- resolution with enhanced propagation and alignment

    Kelvin CK Chan, Shangchen Zhou, Xiangyu Xu, and Chen Change Loy. Basicvsr++: Improving video super- resolution with enhanced propagation and alignment. InPro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5972–5981, 2022. 6

    work page 2022

  2. [2]

    Snow removal in video: A new dataset and a novel method

    Haoyu Chen, Jingjing Ren, Jinjin Gu, Hongtao Wu, Xuequan Lu, Haoming Cai, and Lei Zhu. Snow removal in video: A new dataset and a novel method. in 2023 ieee. In CVF International Conference on Computer Vision (ICCV), pages 13165–13176. 5, 4, 6

    work page 2023

  3. [3]

    Multiframe-to-multiframe network for video denoising

    Huaian Chen, Yi Jin, Kai Xu, Yuxuan Chen, and Changan Zhu. Multiframe-to-multiframe network for video denoising. IEEE Transactions on Multimedia, 24:2164–2178, 2021. 2

    work page 2021

  4. [4]

    Simple baselines for image restoration

    Liangyu Chen, Xiaojie Chu, Xiangyu Zhang, and Jian Sun. Simple baselines for image restoration. In European confer- ence on computer vision, pages 17–33. Springer, 2022. 2

    work page 2022

  5. [5]

    Cross-Consistent Deep Unfolding Network for Adaptive All-In-One Video Restoration

    Yuanshuo Cheng, Mingwen Shao, Yecong Wan, Lixu Zhang, Wangmeng Zuo, and Deyu Meng. Cross-consistent deep unfolding network for adaptive all-in-one video restoration. arXiv preprint arXiv:2309.01627, 2023. 1, 2, 3

    work page Pith review arXiv 2023

  6. [6]

    High- quality image restoration following human instructions

    Marcos V Conde, Gregor Geigle, and Radu Timofte. High- quality image restoration following human instructions. arXiv preprint arXiv:2401.16468, 2024. 1, 2, 3, 4, 5, 7, 8

    arXiv 2024

  7. [7]

    Adair: Adaptive all-in-one image restoration via frequency mining and mod- ulation

    Yuning Cui, Syed Waqas Zamir, Salman Khan, Alois Knoll, Mubarak Shah, and Fahad Shahbaz Khan. Adair: Adaptive all-in-one image restoration via frequency mining and mod- ulation. arXiv preprint arXiv:2403.14614, 2024. 1

    arXiv 2024

  8. [8]

    Learning truncated causal history model for video restoration

    Amirhosein Ghasemabadi, Muhammad Kamran Janjua, Mo- hammad Salameh, and Di Niu. Learning truncated causal history model for video restoration. In The Thirty-eighth An- nual Conference on Neural Information Processing Systems. 1, 2, 3, 4, 5, 8

    work page

Show all 64 references
  1. [9]

    Cascadedgaze: Efficiency in global context extraction for image restoration

    Amirhosein Ghasemabadi, Muhammad Kamran Janjua, Mo- hammad Salameh, CHUNHUA ZHOU, Fengyu Sun, and Di Niu. Cascadedgaze: Efficiency in global context extraction for image restoration. Transactions on Machine Learning Research, 2024. 2

    2024

  2. [10]

    Transformer in transformer

    Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. Advances in neural information processing systems, 34:15908–15919,

  3. [11]

    Recurrent back-projection network for video super- resolution

    Muhammad Haris, Gregory Shakhnarovich, and Norimichi Ukita. Recurrent back-projection network for video super- resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3897–3906,

  4. [12]

    Modulating image restoration with continual levels via adaptive feature modifi- cation layers

    Jingwen He, Chao Dong, and Yu Qiao. Modulating image restoration with continual levels via adaptive feature modifi- cation layers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11056– 11064, 2019. 4

    2019

  5. [13]

    Gaussian error linear units (gelus)

    Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016. 4

    2016 arXiv

  6. [14]

    Neu- ral compression-based feature learning for video restoration

    Cong Huang, Jiahao Li, Bin Li, Dong Liu, and Yan Lu. Neu- ral compression-based feature learning for video restoration. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 5872–5881, 2022. 2

    2022

  7. [15]

    Vjt: A video transformer on joint tasks of deblurring, low-light enhancement and de- noising

    Yuxiang Hui, Yang Liu, Yaofang Liu, Fan Jia, Jinshan Pan, Raymond Chan, and Tieyong Zeng. Vjt: A video transformer on joint tasks of deblurring, low-light enhancement and de- noising. arXiv preprint arXiv:2401.14754, 2024. 1, 2, 3

    2024 arXiv

  8. [16]

    Llmra: Multi-modal large language model based restoration assis- tant

    Xiaoyu Jin, Yuan Shi, Bin Xia, and Wenming Yang. Llmra: Multi-modal large language model based restoration assis- tant. arXiv preprint arXiv:2401.11401, 2024. 1, 2, 3

    2024 arXiv

  9. [17]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 ,

    arXiv

  10. [18]

    All-in-one image restoration for unknown corruption

    Boyun Li, Xiao Liu, Peng Hu, Zhongqin Wu, Jiancheng Lv, and Xi Peng. All-in-one image restoration for unknown corruption. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17452– 17462, 2022. 1, 2, 5, 6, 8, 3

    2022

  11. [19]

    A simple baseline for video restoration with grouped spatial- temporal shift

    Dasong Li, Xiaoyu Shi, Yi Zhang, Ka Chun Cheung, Simon See, Xiaogang Wang, Hongwei Qin, and Hongsheng Li. A simple baseline for video restoration with grouped spatial- temporal shift. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages ...

    2023

  12. [20]

    Mucan: Multi-correspondence aggregation net- work for video super-resolution

    Wenbo Li, Xin Tao, Taian Guo, Lu Qi, Jiangbo Lu, and Jiaya Jia. Mucan: Multi-correspondence aggregation net- work for video super-resolution. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23– 28, 2020, Proceedings, Part X 16, pages 335–351. Springer,

    2020

  13. [21]

    Prompt-in-prompt learning for universal image restoration

    Zilong Li, Yiming Lei, Chenglong Ma, Junping Zhang, and Hongming Shan. Prompt-in-prompt learning for universal image restoration. arXiv preprint arXiv:2312.05038, 2023. 1, 2, 4, 3

    2023 arXiv

  14. [22]

    Swinir: Image restoration us- ing swin transformer

    Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration us- ing swin transformer. InProceedings of the IEEE/CVF inter- national conference on computer vision , pages 1833–1844,

  15. [23]

    Recurrent video restoration trans- former with guided deformable attention

    Jingyun Liang, Yuchen Fan, Xiaoyu Xiang, Rakesh Ranjan, Eddy Ilg, Simon Green, Jiezhang Cao, Kai Zhang, Radu 9 Timofte, and Luc V Gool. Recurrent video restoration trans- former with guided deformable attention. Advances in Neu- ral Information Processing Systems, 35:378–393, ...

    2022

  16. [24]

    Vrt: A video restoration transformer

    Jingyun Liang, Jiezhang Cao, Yuchen Fan, Kai Zhang, Rakesh Ranjan, Yawei Li, Radu Timofte, and Luc Van Gool. Vrt: A video restoration transformer. IEEE Transactions on Image Processing, 2024. 1, 2

    2024

  17. [25]

    Kitti-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d

    Yiyi Liao, Jun Xie, and Andreas Geiger. Kitti-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):3292–3310, 2022. 3

    2022

  18. [26]

    Improving image restora- tion through removing degradations in textual representa- tions

    Jingbo Lin, Zhilu Zhang, Yuxiang Wei, Dongwei Ren, Dong- sheng Jiang, and Wangmeng Zuo. Improving image restora- tion through removing degradations in textual representa- tions. arXiv preprint arXiv:2312.17334, 2023. 1

    2023 arXiv

  19. [27]

    Video super-resolution based on deep learning: a compre- hensive survey

    Hongying Liu, Zhubo Ruan, Peng Zhao, Chao Dong, Fan- hua Shang, Yuanyuan Liu, Linlin Yang, and Radu Timofte. Video super-resolution based on deep learning: a compre- hensive survey. Artificial Intelligence Review, 55(8):5981– 6035, 2022. 1

    2022

  20. [28]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023. 3

    2023

  21. [29]

    Sgdr: Stochas- tic gradient descent with warm restarts

    Ilya Loshchilov and Frank Hutter. Sgdr: Stochas- tic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016. 5

    2016 arXiv

  22. [30]

    Controlling vision-language models for universal image restoration

    Ziwei Luo, Fredrik K Gustafsson, Zheng Zhao, Jens Sj¨olund, and Thomas B Sch¨on. Controlling vision-language models for universal image restoration. arXiv preprint arXiv:2310.01018, 2023. 1, 4

    2023 arXiv

  23. [31]

    Prores: Exploring degradation-aware visual prompt for universal image restora- tion

    Jiaqi Ma, Tianheng Cheng, Guoli Wang, Qian Zhang, Xinggang Wang, and Lefei Zhang. Prores: Exploring degradation-aware visual prompt for universal image restora- tion. arXiv preprint arXiv:2306.13653, 2023. 1

    2023 arXiv

  24. [32]

    Deep multi-scale convolutional neural network for dynamic scene deblurring

    Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring. In CVPR, 2017. 5, 7, 8, 1, 2, 4, 6

    2017

  25. [33]

    Ntire 2019 challenge on video deblurring and super- resolution: Dataset and study

    Seungjun Nah, Sungyong Baik, Seokil Hong, Gyeongsik Moon, Sanghyun Son, Radu Timofte, and Kyoung Mu Lee. Ntire 2019 challenge on video deblurring and super- resolution: Dataset and study. In CVPR Workshops, 2019. 7, 4, 5, 6

    2019

  26. [34]

    Restoring vision in adverse weather conditions with patch-based denoising dif- fusion models

    Ozan ¨Ozdenizci and Robert Legenstein. Restoring vision in adverse weather conditions with patch-based denoising dif- fusion models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(8):10346–10357, 2023. 6

    2023

  27. [35]

    The 2017 davis challenge on video object segmentation

    Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Ar- bel´aez, Alexander Sorkine-Hornung, and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv:1704.00675, 2017. 5, 6, 8, 1, 2, 3

    2017 arXiv

  28. [36]

    Promptir: Prompting for all-in-one image restoration

    Vaishnav Potlapalli, Syed Waqas Zamir, Salman Khan, and Fahad Khan. Promptir: Prompting for all-in-one image restoration. In Thirty-seventh Conference on Neural Infor- mation Processing Systems, 2023. 1, 2, 4, 5, 6, 7, 8, 3

    2023

  29. [37]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  30. [38]

    U- net: Convolutional networks for biomedical image segmen- tation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U- net: Convolutional networks for biomedical image segmen- tation. In Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, pa...

    2015

  31. [39]

    Video restoration based on deep learning: a comprehensive survey

    Claudio Rota, Marco Buzzelli, Simone Bianco, and Rai- mondo Schettini. Video restoration based on deep learning: a comprehensive survey. Artificial Intelligence Review , 56 (6):5317–5364, 2023. 1

    2023

  32. [40]

    Dual-camera joint deblurring-denoising

    Shayan Shekarforoush, Amanpreet Walia, Marcus A Brubaker, Konstantinos G Derpanis, and Alex Levinshtein. Dual-camera joint deblurring-denoising. arXiv preprint arXiv:2309.08826, 2023. 1, 3

    2023 arXiv

  33. [41]

    Many task learning with task routing

    Gjorgji Strezoski, Nanne van Noord, and Marcel Worring. Many task learning with task routing. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1375–1384, 2019. 4

    2019

  34. [42]

    Dvdnet: A fast network for deep video denoising

    Matias Tassano, Julie Delon, and Thomas Veit. Dvdnet: A fast network for deep video denoising. In 2019 IEEE In- ternational Conference on Image Processing (ICIP) , pages 1805–1809. IEEE, 2019. 6

    2019

  35. [43]

    Fastdvd- net: Towards real-time deep video denoising without flow estimation

    Matias Tassano, Julie Delon, and Thomas Veit. Fastdvd- net: Towards real-time deep video denoising without flow estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1354–1363,

  36. [44]

    Patch craft: Video denoising by deep modeling and patch matching

    Gregory Vaksman, Michael Elad, and Peyman Milanfar. Patch craft: Video denoising by deep modeling and patch matching. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2157–2166, 2021. 2

    2021

  37. [45]

    Transweather: Transformer-based restoration of im- ages degraded by adverse weather conditions

    Jeya Maria Jose Valanarasu, Rajeev Yasarla, and Vishal M Patel. Transweather: Transformer-based restoration of im- ages degraded by adverse weather conditions. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2353–2363, 2022. 1, 2, 6

    2022

  38. [46]

    Edvr: Video restoration with enhanced deformable convolutional networks

    Xintao Wang, Kelvin CK Chan, Ke Yu, Chao Dong, and Chen Change Loy. Edvr: Video restoration with enhanced deformable convolutional networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 0–0, 2019. 6

    2019

  39. [47]

    Image quality assessment: from error visibility to structural similarity

    Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Si- moncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600–612, 2004. 5

    2004

  40. [48]

    Mask-guided progressive network for joint raindrop and rain streak removal in videos

    Hongtao Wu, Yijun Yang, Haoyu Chen, Jingjing Ren, and Lei Zhu. Mask-guided progressive network for joint raindrop and rain streak removal in videos. In Proceedings of the 31st ACM International Conference on Multimedia, pages 7216– 7225, 2023. 5, 8, 1, 2, 4

    2023

  41. [49]

    Q-instruct: Improving low-level visual abilities for multi-modality foundation models

    Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Kaixin Xu, Chunyi Li, Jingwen Hou, Guangtao Zhai, et al. Q-instruct: Improving low-level visual abilities for multi-modality foundation models. arXiv preprint arXiv:2311.06783, 2023. 3, 4, 5, 6 10

    2023 arXiv

  42. [50]

    Textual prompt guided image restora- tion

    Qiuhai Yan, Aiwen Jiang, Kang Chen, Long Peng, Qiaosi Yi, and Chunjie Zhang. Textual prompt guided image restora- tion. arXiv preprint arXiv:2312.06162, 2023. 1, 2, 3

    2023 arXiv

  43. [51]

    Language- driven all-in-one adverse weather removal

    Hao Yang, Liyuan Pan, Yan Yang, and Wei Liang. Language- driven all-in-one adverse weather removal. arXiv preprint arXiv:2312.01381, 2023. 1, 2, 3

    2023 arXiv

  44. [52]

    Video adverse-weather- component suppression network via weather messenger and adversarial backpropagation

    Yijun Yang, Angelica I Aviles-Rivero, Huazhu Fu, Ye Liu, Weiming Wang, and Lei Zhu. Video adverse-weather- component suppression network via weather messenger and adversarial backpropagation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 13...

    2023

  45. [53]

    Restormer: Efficient transformer for high-resolution image restoration

    Syed Waqas Zamir, Aditya Arora, Salman Khan, Mu- nawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5728–5739,

  46. [54]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shecht- man, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 586–595, 2018. 2

    2018

  47. [55]

    Avernet: All-in-one video restoration for time-varying unknown degradations

    Haiyu Zhao, Lei Tian, Xinyan Xiao, Peng Hu, Yuanbiao Gou, and Xi Peng. Avernet: All-in-one video restoration for time-varying unknown degradations. In The Thirty-eighth Annual Conference on Neural Information Processing Sys- tems. 1, 2, 4, 5, 6, 7, 8, 3

  48. [56]

    Recursive fusion and deformable spatiotemporal attention for video compres- sion artifact reduction

    Minyi Zhao, Yi Xu, and Shuigeng Zhou. Recursive fusion and deformable spatiotemporal attention for video compres- sion artifact reduction. InProceedings of the 29th ACM inter- national conference on multimedia, pages 5646–5654, 2021. 2

    2021

  49. [57]

    Efficient spatio-temporal recurrent neural network for video deblurring

    Zhihang Zhong, Ye Gao, Yinqiang Zheng, and Bo Zheng. Efficient spatio-temporal recurrent neural network for video deblurring. In Computer Vision–ECCV 2020: 16th Euro- pean Conference, Glasgow, UK, August 23–28, 2020, Pro- ceedings, Part VI 16, pages 191–207. Springer, 2020. 2

    2020

  50. [58]

    Exploring motion ambiguity and alignment for high-quality video frame interpolation

    Kun Zhou, Wenbo Li, Xiaoguang Han, and Jiangbo Lu. Exploring motion ambiguity and alignment for high-quality video frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22169–22179, 2023. 2

    2023

  51. [59]

    Deep recurrent neural net- work with multi-scale bi-directional propagation for video deblurring

    Chao Zhu, Hang Dong, Jinshan Pan, Boyang Liang, Yuhao Huang, Lean Fu, and Fei Wang. Deep recurrent neural net- work with multi-scale bi-directional propagation for video deblurring. In Proceedings of the AAAI conference on artifi- cial intelligence, pages 3598–3607, 2022. 2 11...

    2022

  52. [60]

    Additional Ablation Studies We discuss the motivation behind grounding degradations, and present additional ablation studies to further understand different components of R ONIN and the design choices made. 7.1. Motivation: Grounding Degradations We posit that grounding the de...

  53. [61]

    Prompt Importance

    Additional Related Work Video restoration, in literature, is studied from several facets, mostly distributed in terms of how the motion is es- Prompt Style Deblur (GoPro [32]) Denoise (DA VIS [35]) Derain (VRDS [48]) PSNR SSIM PSNR SSIM PSNR SSIM Perturbed Prompts 15.93 0 .56 ...

  54. [62]

    Dataset Details All of the benchmarks considered in this work are created through standard datasets in video restoration literature and are available open-source for academic research purposes, except our proposed SnowyScenes benchmark, which will be open-sourced and released ...

  55. [63]

    Think step by step

    Details of Prompting Recall that the basic prompt to query Q-Instruct [49] to as- sess the degradation in the image is ‘Rate the quality of the image. Think step by step. ’. While this works in most cases where the degradation matches the synthetic degradations Deg. ‘Snow’ ‘No...

  56. [64]

    Limitations, Future Work, and Impact The descriptions may occasionally include more degrada- tions than are present in the video, such as the mention of noise in a frame which is a part of a video in the deblurring task. Although this rarely happens, as Q-Instruct [49] when pr...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.