Pith. sign in

REVIEW 3 major objections 5 minor 63 references

Efficient Video Face Enhancement with Enhanced Spatial-Temporal Consistency

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Blind face video restoration and de-flickering can be done in a single 3D-VQGAN pass with spatial-temporal codebooks, running about 2.995 seconds per 24-frame clip.

desk verdict A fast 3D-VQGAN with spatial/temporal codebooks that genuinely speeds up face video restoration, but the 'surpasses SOTA' claim overstates what its own tables show. read the letter →

arxiv 2411.16468 v1 pith:LPOMIWFW submitted 2024-11-25 cs.CV

Yutong Wang , Jiajie Teng , Jiajiong Cao , Yuming Li , Chenguang Ma , Hongteng Xu , Dixin Luo This is my paper · ORCID
classification cs.CV
keywords blindfacevideorestorationde-flickering3D-VQGANspatial-temporalcodebookscodebookcollapselookuptransformertemporalconsistencyenhancement
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that blind face video restoration and de-flickering can be solved in one efficient video-level pass rather than by chaining face detection, image restoration, and separate temporal smoothing. It builds a 3D-VQGAN whose spatial codebook stores high-quality facial features and whose temporal codebook stores motion residuals between frames, then trains two lookup transformers to predict the discrete code indices from a degraded input. On the VFHQ-Test benchmark the method runs at about 2.995 seconds per 24-frame clip while matching or beating prior blind face video restoration and de-flickering methods on fidelity and temporal consistency. The practical upshot is that restoring compressed face videos and removing flicker from AI-generated videos could become fast enough for interactive use without sacrificing consistency.

What carries the argument

The load-bearing object is a pair of spatial-temporal codebooks inside a 3D-VQGAN: a spatial codebook $\mathcal{C}_S$ that stores discrete high-quality portrait feature vectors, and a temporal codebook $\mathcal{C}_T$ that stores motion residuals between latents separated by a time window, fused with inter-frame temporal attention. The lookup transformers $T_S$ and $T_T$ convert degraded-video latents into predicted code indices, and the shared discrete codebook then decodes those indices into a clean, temporally consistent video. The marginal prior regularizer, which computes a soft usage distribution from similarity scores instead of hard counts, prevents codebook collapse and keeps both codebooks active during Stage I training.

What would settle it

Take real compressed talking-head videos with frame-varying noise, abrupt brightness changes, and camera cuts, restore them with this method, and inspect a temporal profile made by stacking one pixel column across frames; if the restored clip shows blur or brightness jumps comparable to the input under these conditions, or if FVD and Flow-Score on a per-frame randomly degraded test set are not clearly better than the input's, the claim of general spatial-temporal consistency for real-world videos is refuted.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that a single 3D-VQGAN forward pass, guided by spatial-temporal codebooks and two code-lookup transformers, can restore high-quality face videos from compressed low-quality inputs and remove both brightness flicker and AI-generated pixel flicker, while being substantially faster than existing blind face video restoration and de-flickering pipelines. Stage I learns an HQ encoder $E_h$, decoder $D_h$, and codebooks $\mathcal{C}_S$ (portrait features) and $\mathcal{C}_T$ (residual-based temporal information) by reconstructing clean videos, using a pre-trained feature network plus multi-scale discriminator heads and a marginal-prior regularizer that counts code usage by accumulated similarity scores rather than one-hot hits. Stage II freezes the codebooks and decoder, trains a low-quality encoder $E_l$, and trains two lookup transformers $T_S$ and $T_T$ to predict the spatial and temporal code sequences from degraded video; the training target is the ground-truth code sequence the HQ encoder produces on clean video degraded by a synthetic, clip-consistent model. On the VFHQ-Test benchmark the paper reports that this surpasses current blind face video restoration and de-flickering baselines on both efficiency and effectiveness.

Load-bearing premise

Stage II training assumes that the ground-truth code sequences produced by the HQ encoder on clean videos, degraded with clip-consistent synthetic blur, noise, and compression, are the right targets for restoring real-world degraded inputs; if a real video's degradations or motion fall outside that synthetic distribution, the temporal code lookup can fail, as the paper itself shows for fast motion and camera switching.

Editorial extensions

If this is right

  • Blind face video restoration no longer needs face detection, background separation, or an extra restoration model: the whole clip is enhanced by one encoder-codebook-decoder pass.
  • De-flickering requires only the degraded video itself, with no reference video and no per-video optimization, so brightness flicker in old footage and pixel flicker in AI-generated video are removed at roughly the cost of a single restoration pass.
  • Temporal consistency is enforced by shared discrete code indices rather than by propagating restored frames, which avoids the flicker that frame-by-frame image restoration produces.
  • At the reported speed of about 2.995 seconds per 24-frame clip, the method is substantially faster than the blind face video restoration and de-flickering baselines it is compared with, including diffusion- and atlas-based pipelines.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the discrete indices from the spatial-temporal codebooks really are temporally consistent, they could serve as a reusable, editable video representation for other consistency-sensitive tasks such as video colorization, deblotching, or style transfer.
  • The paper's own failure mode, blur under fast motion and camera switching, points to a concrete extension: replacing the global temporal attention with local temporal attention or motion-compensated residuals should recover performance exactly where the current model breaks.
  • A stronger test would evaluate on real-world videos whose degradations vary frame to frame; the paper's random-degradation ablation shows this is the regime where the model is weakest, so real-world generalization is not yet demonstrated.
  • Nothing in the codebook or lookup design is face-specific except the training data, so porting the same two-stage scheme to general video restoration is a direct testable extension.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a two-stage blind face video enhancement framework. Stage I trains a 3D-VQGAN with spatial and temporal codebooks, a DINOv2-based multi-scale discriminator, and a marginal-prior regularization to mitigate codebook collapse. Stage II trains a low-quality encoder and two lookup transformers to predict spatial and temporal code indices from degraded videos, with the Stage I codebooks and decoder frozen. The method is evaluated on VFHQ-Test for blind face video restoration and for brightness and pixel de-flickering, reporting a runtime of about 2.995 seconds per 24-frame clip and claiming to surpass state-of-the-art methods on both efficiency and effectiveness.

Significance. The efficiency result is genuinely valuable: processing a 24-frame clip in roughly 3 seconds with a single 3D-VQGAN forward pass, without separate face/background processing chains, is a credible practical improvement over the 7–85 s runtimes reported for prior methods. The two-stage design is coherent, the spatial-temporal codebook idea is well motivated, and the ablations cover useful design choices (discriminator feature network, fusion operator, codebook regularization, training resolution, and data curation). The paper also releases code. If the effectiveness claims were properly scoped, this would be a solid engineering contribution. However, the headline claim of surpassing state-of-the-art effectiveness is not supported by the paper’s own tables, and the generalization evidence relies on a favorable consistent-degradation training setup.

major comments (3)
  1. [Abstract and §4.2, Table 1] The claim that the method "surpasses the current state-of-the-art blind face video restoration and de-flickering methods on both efficiency and effectiveness" is not supported by Table 1: PGTFormer achieves higher PSNR (28.68 vs 27.47), lower LPIPS (0.1752 vs 0.1829), and lower AKD (3.519 vs 3.858), while the proposed method wins on SSIM, Face-Cons, IDS, FVD, and Flow-Score. The body text in §4.2 says the results are "comparable to those of state-of-the-art methods," which contradicts the abstract. Please revise the headline claim to reflect the metric-by-metric outcome, or provide a significance test and a pre-specified primary metric if a dominance claim is intended.
  2. [Supplementary §7.1, Table 6; §4.1 Eq. (9)] The generalization claim for blind face video restoration is undercut by the paper's own stochastic-degradation experiment. Stage II trains and tests with degradation parameters (sigma, r, delta) fixed per clip (Eq. (9), §4.1), and Supplementary Table 6 shows that switching to per-frame stochastic degradation degrades FVD from 105.1 to 156.2, PSNR from 27.47 to 26.92, and IDS from 0.9312 to 0.9120. Real-world videos contain per-frame varying blur, noise, and compression; without experiments on such data, or a strong argument for why consistent-degradation training transfers, the effectiveness claim in the abstract is not established. Section 9's admission that fast motion or camera switching causes blur further indicates the temporal consistency mechanism is fragile outside the training distribution.
  3. [§4.2, Tables 2 and 3] For the de-flickering task, the claim of surpassing prior methods on effectiveness is contradicted by the FVD metric: DVP achieves FVD 14.53 vs 100.7 for brightness de-flickering and 15.09 vs 86.88 for pixel de-flickering, while the proposed method has better Flow-Score and runtime. The text correctly notes that DVP uses an unprocessed reference video at test time, but the comparison should then be framed as "among methods that use only the flickering video," not as an unconditional effectiveness win. Please qualify the claim and state which metrics define the intended notion of effectiveness.
minor comments (5)
  1. [§2.3] The phrase "is not toilless" should read "is not effortless."
  2. [§4.1 and Tables 1–3] The hardware and settings used for the runtime measurements are not stated; please report the GPU model, batch size, and whether all methods were timed on the same platform so the efficiency comparison is reproducible.
  3. [Tables 2 and 3] The column header "GT FVD" is confusing; clarify whether "GT" is part of the metric name or a separate column, and define what the reported FVD values are computed against.
  4. [Eq. (4)] In Eq. (4), "arg min" is used both for returning the index I and for returning the quantized vector z_q; please separate the two definitions or use different notation for the index and the retrieved code vector.
  5. [Table 1] No error bars or significance tests are reported for the VFHQ-Test metrics; several differences are small (e.g., FVD 105.1 vs 107.6), so it would be helpful to report variance across multiple evaluation runs or a paired test.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the two-stage codebook-lookup pipeline is an empirical supervised system evaluated on held-out data, with no prediction that reduces to its inputs by construction.

full rationale

The paper makes no first-principles derivation whose conclusion is equivalent to its premises. Stage I (Eq. 1-7) trains a 3D-VQGAN and spatial/temporal codebooks on HQ videos using reconstruction, perceptual, adversarial, and code-level losses; Stage II (Eq. 8) trains an LQ encoder and two lookup transformers to predict the HQ encoder's code indices from degraded inputs, with targets generated by the Stage I encoder. This is standard supervised training, not a circular reduction: the targets are derived from clean videos, and the test outputs are evaluated against ground-truth HQ frames on the held-out VFHQ-Test set. No parameter is fit to the test data and then reported as a prediction. The reference list contains no papers by the present authors, so no load-bearing self-citation or imported uniqueness theorem exists. The design choices borrowed from prior work (e.g., DINOv2 discriminator, the Open-Sora-Plan sampling design) are external and are not used to force the reported result. The Supplementary's stochastic-degradation comparison (Table 6) and the Section 9 failure case show that the method is sensitive to per-frame degradation and fast motion; these are generalization/robustness limitations, not circularity. The abstract's 'surpasses current state-of-the-art' wording is optimistic given Table 1 (PGTFormer is better on PSNR, LPIPS, AKD), but overstatement is not a circularity defect. Overall, the derivation chain is self-contained with respect to the benchmark and no claimed prediction is equivalent to an input by construction.

Assumptions & free parameters 6 free parameters · 5 assumptions · 2 invented entities

The central claim is empirical and rests on a standard VQGAN-style training loop plus hand-chosen hyperparameters, data filtering rules, a private training set, and an assumed transfer from clean-video codes to real-world degraded videos. The most load-bearing free parameters are codebook size, latent dimension, unreported loss weights, and the degradation model, since changing them changes all benchmark numbers. The invented entities are the codebooks themselves, whose effectiveness has no independent evidence outside the paper's own evaluations.

free parameters (6)
  • Codebook sizes NS, NT = 1024
    Ablation in Table 4(e) shows 1024/1024 improves SSIM and FVD over 512/512 and 2048/2048; the reported performance depends on this choice.
  • Latent dimension D = 256
    Table 4(e) shows D=256 outperforms D=64; the capacity of the latent space is load-bearing for reconstruction quality.
  • Loss weights beta, lambda_adv, lambda_CE = not reported
    Eqs. 7 and 8 define code-level loss with beta and lookup loss with lambda_CE, and adversarial loss uses lambda_adv, but no values are given, so the exact training objective cannot be reproduced.
  • Degradation ranges sigma, r, delta = sigma in [2,5], r in [2,4], delta in [0,5]
    The synthetic degradation model in Eq. 9 defines the BFVR task and all benchmark comparisons; these hand-chosen ranges shape the difficulty of the test.
  • Side-face threshold alpha = alpha < 0.4 or alpha > 2.5
    Supplementary 6.1 states a manually defined threshold filters training videos by face orientation, changing the training distribution and downstream metrics.
  • Frame flicker probability = 30%
    For de-flickering training, each frame has a 30% probability of being degraded; this synthetic task definition drives the de-flickering benchmark numbers.
assumptions (5)
  • domain assumption Nearest-neighbor lookup with straight-through gradient estimation and code-level loss trains usable codebooks.
    Relied on throughout Section 3.2; codebook collapse is mitigated by the proposed regularizer, but no convergence guarantee is provided.
  • domain assumption A frozen DINOv2 feature extractor provides an adversarial space that stabilizes video VQGAN training and improves reconstruction.
    Section 3.2 discriminator design; validated only by an ablation on VFHQ-Test, not by theory or external benchmarks.
  • ad hoc to paper Ground-truth code indices from the clean HQ encoder are the correct training targets for restoring degraded videos.
    Stage II labels are produced by Eh on HQ frames (Eq. 8); this assumes the HQ code distribution covers all degradations the LQ encoder will see.
  • domain assumption A consistent per-clip degradation model is representative of real-world compressed videos.
    Supplementary 7.1 shows consistent degradation outperforms stochastic degradation in their benchmarks; real videos may mix degradation types over time.
  • domain assumption The selected metrics on 50 test clips are sufficient proxies for perceptual quality and temporal consistency.
    Section 4.1 defines metrics such as SSIM, FVD, Flow-Score, Face-Cons, and IDS; no human study or statistical significance testing is included.
invented entities (2)
  • Spatial codebook CS (1024 learned vectors)
    purpose: Records high-quality portrait features for quantization of spatial latents (Eq. 4).
    Only evaluated through internal reconstruction and restoration metrics; no external benchmark or theoretical justification shows that 1024 discrete vectors capture face appearance.
  • Temporal codebook CT (1024 learned vectors)
    purpose: Records residual-based temporal motion information between frames (Eq. 3).
    The temporal residual representation is a design choice; its superiority is shown only on the paper's own VFHQ-Test reconstruction and restoration tables.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient Video Face Enhancement with Enhanced Spatial-Temporal Consistency." pith.science (2026). https://pith.science/paper/LPOMIWFW

@misc{pith2026241116468,
  author       = {Pith},
  title        = {Pith review of: Efficient Video Face Enhancement with Enhanced Spatial-Temporal Consistency},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LPOMIWFW}},
  note         = {Machine review of arXiv:2411.16468}
}
abstract

As a very common type of video, face videos often appear in movies, talk shows, live broadcasts, and other scenes. Real-world online videos are often plagued by degradations such as blurring and quantization noise, due to the high compression ratio caused by high communication costs and limited transmission bandwidth. These degradations have a particularly serious impact on face videos because the human visual system is highly sensitive to facial details. Despite the significant advancement in video face enhancement, current methods still suffer from $i)$ long processing time and $ii)$ inconsistent spatial-temporal visual effects (e.g., flickering). This study proposes a novel and efficient blind video face enhancement method to overcome the above two challenges, restoring high-quality videos from their compressed low-quality versions with an effective de-flickering mechanism. In particular, the proposed method develops upon a 3D-VQGAN backbone associated with spatial-temporal codebooks recording high-quality portrait features and residual-based temporal information. We develop a two-stage learning framework for the model. In Stage \Rmnum{1}, we learn the model with a regularizer mitigating the codebook collapse problem. In Stage \Rmnum{2}, we learn two transformers to lookup code from the codebooks and further update the encoder of low-quality videos. Experiments conducted on the VFHQ-Test dataset demonstrate that our method surpasses the current state-of-the-art blind face video restoration and de-flickering methods on both efficiency and effectiveness. Code is available at \url{https://github.com/Dixin-Lab/BFVR-STC}.

Figures

Figures reproduced from arXiv: 2411.16468 by the authors.

Figure 1
Figure 1. Network architecture of Stage I. Stage I uses HQ face videos to train HQ 3D-VQGAN (Eh and Dh) and spatial and temporal codebooks (CS and CT ). (a) illustrates the quantization operation STLookUp through two codebooks in our proposed framework. (b) and (c) display the computation process of temporal attention and motion residual, respectively. (d) We leverage a pre-trained feature network DINOv2 and trainable multi-s… view at source ↗
Figure 2
Figure 2. Network architecture of Stage II. Stage II uses HQ-LQ face video pairs to train LQ encoder El and LookUp Transformers (TS and TT ). The weights of Dh are pre-trained in Stage I and fixed in Stage II. Transformer-based code lookup modules designed to pre￾dict the code sequence indices of the compressed spatial la￾tents zl,S and temporal latents zl,T within the codebooks, respectively. In stage II, we fix the HQ decod… view at source ↗
Figure 3
Figure 3. Qualitative comparison on the VFHQ-Test for BFVR task. Our method has better fidelity and fewer hallucination cases compared to other methods, such as wrinkles (1st row), eye orientation (1st row), nose shape (2nd row), and hairstyle (3rd row). ods typically employ a face detection module to separate faces for separate processing, resulting in inefficient infer￾ence. We improve the quality of the training data by cr… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Comparison of temporal profile for brightness and pixel de-flickering. We select a column to observe the changes across time. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Comparison of spatial and temporal codebooks’ uti￾lization when applying different regualizations. latents along the channel dimension and using 3D convolu￾tion to reduce it to normal size. 3DFFT represents the fusion of latents by processing two variables separately b…
Figure 7
Figure 7. Figure 7: Visualization of motion intensity of (a) VFHQ and (b) [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 6
Figure 6. Figure 6: Visualization of the training data processing, including [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 9
Figure 9. Figure 9: Visualization of restoration results when training with [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 8
Figure 8. Figure 8: Visualization of restoration results when applying differ [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 10
Figure 10. Figure 10: Comparison of temporal profile on the VFHQ-Test for BFVR task. We select a column to observe the changes across time. [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Qualitative comparison on the VFHQ-Test for BFVR task. Our method achieves higher fidelity and demonstrates fewer [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 12
Figure 12. Figure 12: Visualization of the changes in codebook utilization [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

63 extracted references · 40 canonical work pages

  1. [1]

    vq- wav2vec: Self-supervised learning of discrete speech repre- sentations

    Alexei Baevski, Steffen Schneider, and Michael Auli. vq- wav2vec: Self-supervised learning of discrete speech repre- sentations. arXiv preprint arXiv:1910.05453, 2019. 2

    arXiv 1910

  2. [2]

    Blind video temporal consistency

    Nicolas Bonneel, James Tompkin, Kalyan Sunkavalli, De- qing Sun, Sylvain Paris, and Hanspeter Pfister. Blind video temporal consistency. ACM Transactions on Graph- ics (TOG), 34(6):1–9, 2015. 2

    work page 2015

  3. [3]

    Glean: Generative latent bank for large-factor image super-resolution

    Kelvin CK Chan, Xintao Wang, Xiangyu Xu, Jinwei Gu, and Chen Change Loy. Glean: Generative latent bank for large-factor image super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14245–14254, 2021. 1, 2

    work page 2021

  4. [4]

    Basicvsr++: Improving video super- resolution with enhanced propagation and alignment

    Kelvin CK Chan, Shangchen Zhou, Xiangyu Xu, and Chen Change Loy. Basicvsr++: Improving video super- resolution with enhanced propagation and alignment. InPro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5972–5981, 2022. 1, 6, 7

    work page 2022

  5. [5]

    Investigating tradeoffs in real-world video super-resolution

    Kelvin CK Chan, Shangchen Zhou, Xiangyu Xu, and Chen Change Loy. Investigating tradeoffs in real-world video super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5962–5971, 2022. 6, 7

    work page 2022

  6. [6]

    Progressive semantic- aware style transformation for blind face restoration

    Chaofeng Chen, Xiaoming Li, Lingbo Yang, Xianhui Lin, Lei Zhang, and Kwan-Yee K Wong. Progressive semantic- aware style transformation for blind face restoration. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11896–11905, 2021. 1, 2

    work page 2021

  7. [7]

    Fsrnet: End-to-end learning face super-resolution with facial priors

    Yu Chen, Ying Tai, Xiaoming Liu, Chunhua Shen, and Jian Yang. Fsrnet: End-to-end learning face super-resolution with facial priors. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2492–2501,

    work page

  8. [8]

    Stabilization of flicker- like effects in image sequences through local contrast cor- rection

    Julie Delon and Agnes Desolneux. Stabilization of flicker- like effects in image sequences through local contrast cor- rection. SIAM Journal on Imaging Sciences , 3(4):703–734,

    work page

Show all 63 references
  1. [9]

    Fastblend: a powerful model- free toolkit making video stylization easier

    Zhongjie Duan, Chengyu Wang, Cen Chen, Weining Qian, Jun Huang, and Mingyi Jin. Fastblend: a powerful model- free toolkit making video stylization easier. arXiv preprint arXiv:2311.09265, 2023. 2, 7

    2023 arXiv

  2. [10]

    Diffutoon: High-resolution ed- itable toon shading via diffusion models

    Zhongjie Duan, Chengyu Wang, Cen Chen, Weining Qian, and Jun Huang. Diffutoon: High-resolution ed- itable toon shading via diffusion models. arXiv preprint arXiv:2401.16224, 2024. 2

    2024 arXiv

  3. [11]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021. 2, 4

    2021

  4. [12]

    Kalman-inspired feature propagation for video face super- resolution

    Ruicheng Feng, Chongyi Li, and Chen Change Loy. Kalman-inspired feature propagation for video face super- resolution. arXiv preprint arXiv:2408.05205, 2024. 1, 2, 5, 6, 7

    2024 arXiv

  5. [13]

    Unsuper- vised model-based learning for simultaneous video deflicker- ing and deblotching

    Anuj Fulari, Satish Mulleti, and Ajit Rajwade. Unsuper- vised model-based learning for simultaneous video deflicker- ing and deblotching. In Proceedings of the IEEE/CVF Win- 9 ter Conference on Applications of Computer Vision , pages 4117–4125, 2024. 1

    2024

  6. [14]

    Vqfr: Blind face restoration with vector-quantized dictionary and parallel de- coder

    Yuchao Gu, Xintao Wang, Liangbin Xie, Chao Dong, Gen Li, Ying Shan, and Ming-Ming Cheng. Vqfr: Blind face restoration with vector-quantized dictionary and parallel de- coder. In European Conference on Computer Vision, pages 126–143. Springer, 2022. 1, 2, 5, 6, 7

    2022

  7. [15]

    Face super-resolution guided by 3d facial priors

    Xiaobin Hu, Wenqi Ren, John LaMaster, Xiaochun Cao, Xi- aoming Li, Zechao Li, Bjoern Menze, and Wei Liu. Face super-resolution guided by 3d facial priors. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IV 16 , pages 7...

    2020

  8. [16]

    Straightening out the straight-through estimator: Over- coming optimization challenges in vector quantized net- works

    Minyoung Huh, Brian Cheung, Pulkit Agrawal, and Phillip Isola. Straightening out the straight-through estimator: Over- coming optimization challenges in vector quantized net- works. In International Conference on Machine Learning , pages 14096–14113. PMLR, 2023. 2

    2023

  9. [17]

    Percep- tual losses for real-time style transfer and super-resolution

    Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Percep- tual losses for real-time style transfer and super-resolution. In Computer Vision–ECCV 2016: 14th European Confer- ence, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 694–711. Springer, 2016. 4

    2016

  10. [18]

    Flicker removal and superpixel-based motion tracking for high speed videos

    Ali Kanj, Hugues Talbot, and Raoul Rodriguez Luparello. Flicker removal and superpixel-based motion tracking for high speed videos. In 2017 IEEE international conference on image processing (ICIP), pages 245–249. IEEE, 2017. 2

    2017

  11. [19]

    Video super-resolution with convolutional neu- ral networks

    Armin Kappeler, Seunghwan Yoo, Qiqin Dai, and Aggelos K Katsaggelos. Video super-resolution with convolutional neu- ral networks. IEEE transactions on computational imaging, 2(2):109–122, 2016. 1

    2016

  12. [20]

    A style-based generator architecture for generative adversarial networks

    Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 4401–4410, 2019. 1, 2

    2019

  13. [21]

    Analyzing and improv- ing the image quality of stylegan

    Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improv- ing the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8110–8119, 2020. 1, 2

    2020

  14. [22]

    Adam: A method for stochastic opti- mization

    Diederik P Kingma. Adam: A method for stochastic opti- mization. arXiv preprint arXiv:1412.6980, 2014. 7

    2014 arXiv

  15. [23]

    Open-sora-plan, 2024

    PKU-Yuan Lab and Tuzhan AI etc. Open-sora-plan, 2024. 3

    2024

  16. [24]

    Robust training of vector quantized bottleneck models

    Adrian Ła ´ncucki, Jan Chorowski, Guillaume Sanchez, Ri- card Marxer, Nanxin Chen, Hans JGA Dolfing, Sameer Khu- rana, Tanel Alumäe, and Antoine Laurent. Robust training of vector quantized bottleneck models. In 2020 International Joint Conference on Neural Networks (IJCNN) , ...

    2020

  17. [25]

    Blind video temporal consistency via deep video prior.Advances in Neu- ral Information Processing Systems, 33:1083–1093, 2020

    Chenyang Lei, Yazhou Xing, and Qifeng Chen. Blind video temporal consistency via deep video prior.Advances in Neu- ral Information Processing Systems, 33:1083–1093, 2020. 7

    2020

  18. [26]

    Blind video deflickering by neural filtering with a flawed atlas

    Chenyang Lei, Xuanchi Ren, Zhaoxiang Zhang, and Qifeng Chen. Blind video deflickering by neural filtering with a flawed atlas. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10439– 10448, 2023. 1, 2, 7

    2023

  19. [27]

    Learning warped guidance for blind face restoration

    Xiaoming Li, Ming Liu, Yuting Ye, Wangmeng Zuo, Liang Lin, and Ruigang Yang. Learning warped guidance for blind face restoration. In Proceedings of the European conference on computer vision (ECCV), pages 272–289, 2018. 1

    2018

  20. [28]

    Vrt: A video restoration transformer

    Jingyun Liang, Jiezhang Cao, Yuchen Fan, Kai Zhang, Rakesh Ranjan, Yawei Li, Radu Timofte, and Luc Van Gool. Vrt: A video restoration transformer. IEEE Transactions on Image Processing, 2024. 1

    2024

  21. [29]

    Diff- bir: Towards blind image restoration with generative diffu- sion prior

    Xinqi Lin, Jingwen He, Ziyan Chen, Zhaoyang Lyu, Bo Dai, Fanghua Yu, Wanli Ouyang, Yu Qiao, and Chao Dong. Diff- bir: Towards blind image restoration with generative diffu- sion prior. arXiv preprint arXiv:2308.15070, 2023. 2

    2023 arXiv

  22. [30]

    Evalcrafter: Benchmarking and eval- uating large video generation models

    Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond Chan, and Ying Shan. Evalcrafter: Benchmarking and eval- uating large video generation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern R...

    2024

  23. [31]

    Freelong: Training-free long video generation with spectralblend tem- poral attention

    Yu Lu, Yuanzhi Liang, Linchao Zhu, and Yi Yang. Freelong: Training-free long video generation with spectralblend tem- poral attention. arXiv preprint arXiv:2407.19918, 2024. 7, 8

    2024 arXiv

  24. [32]

    Sora: Creating video from text., 2024

    OpenAI. Sora: Creating video from text., 2024. https: //openai.com/sora/. 1

    2024

  25. [33]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 3

    2023 arXiv

  26. [34]

    Gener- ating diverse high-fidelity images with vq-vae-2

    Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Gener- ating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019. 2

    2019

  27. [35]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 2

    2022

  28. [36]

    Adversarial diffusion distillation

    Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. arXiv preprint arXiv:2311.17042, 2023. 3, 7

    2023 arXiv

  29. [37]

    Fast high- resolution image synthesis with latent adversarial diffusion distillation

    Axel Sauer, Frederic Boesel, Tim Dockhorn, Andreas Blattmann, Patrick Esser, and Robin Rombach. Fast high- resolution image synthesis with latent adversarial diffusion distillation. arXiv preprint arXiv:2403.12015, 2024. 3

    2024 arXiv

  30. [38]

    Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network

    Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference on compute...

    2016

  31. [39]

    Very deep convolutional networks for large-scale image recognition

    K Simonyan and A Zisserman. Very deep convolutional networks for large-scale image recognition. In 3rd In- ternational Conference on Learning Representations (ICLR 2015). Computational and Biological Learning Society,

    2015

  32. [40]

    Sq-vae: Variational bayes on discrete representation with self-annealed stochastic quantization

    Yuhta Takida, Takashi Shibuya, WeiHsiang Liao, Chieh- Hsin Lai, Junki Ohmura, Toshimitsu Uesaka, Naoki Murata, 10 Shusuke Takahashi, Toshiyuki Kumakura, and Yuki Mitsu- fuji. Sq-vae: Variational bayes on discrete representation with self-annealed stochastic quantization. arXiv...

    2022 arXiv

  33. [41]

    Blind face video restoration with temporal consis- tent generative prior and degradation-aware prompt

    Jingfan Tan, Hyunhee Park, Ying Zhang, Tao Wang, Kaihao Zhang, Xiangyu Kong, Pengwen Dai, Zikun Liu, and Wen- han Luo. Blind face video restoration with temporal consis- tent generative prior and degradation-aware prompt. InACM Multimedia 2024, 2024. 1, 2, 5

    2024

  34. [42]

    Raft: Recurrent all-pairs field transforms for optical flow

    Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23– 28, 2020, Proceedings, Part II 16, pages 402–419. Springer,

    2020

  35. [43]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information pro- cessing systems, 30, 2017. 2, 4

    2017

  36. [44]

    Edvr: Video restoration with enhanced deformable convolutional networks

    Xintao Wang, Kelvin CK Chan, Ke Yu, Chao Dong, and Chen Change Loy. Edvr: Video restoration with enhanced deformable convolutional networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 0–0, 2019. 1

    2019

  37. [45]

    To- wards real-world blind face restoration with generative fa- cial prior

    Xintao Wang, Yu Li, Honglun Zhang, and Ying Shan. To- wards real-world blind face restoration with generative fa- cial prior. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9168–9178,

  38. [46]

    Real-esrgan: Training real-world blind super-resolution with pure synthetic data

    Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan. Real-esrgan: Training real-world blind super-resolution with pure synthetic data. In Proceedings of the IEEE/CVF inter- national conference on computer vision , pages 1905–1914,

    1905

  39. [47]

    Restoreformer: High-quality blind face restoration from undegraded key-value pairs

    Zhouxia Wang, Jiawei Zhang, Runjian Chen, Wenping Wang, and Ping Luo. Restoreformer: High-quality blind face restoration from undegraded key-value pairs. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17512–17521, 2022. 1, 2

    2022

  40. [48]

    Vfhq: A high-quality dataset and bench- mark for video face super-resolution

    Liangbin Xie, Xintao Wang, Honglun Zhang, Chao Dong, and Ying Shan. Vfhq: A high-quality dataset and bench- mark for video face super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 657–666, 2022. 5

    2022

  41. [49]

    Beyond alignment: Blind video face restoration via parsing-guided temporal-coherent transformer

    Kepeng Xu, Li Xu, Gang He, Wenxin Yu, and Yun- song Li. Beyond alignment: Blind video face restoration via parsing-guided temporal-coherent transformer. arXiv preprint arXiv:2404.13640, 2024. 1, 2, 5, 6, 7

    2024 arXiv

  42. [50]

    Gan prior embedded network for blind face restoration in the wild

    Tao Yang, Peiran Ren, Xuansong Xie, and Lei Zhang. Gan prior embedded network for blind face restoration in the wild. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition , pages 672–681, 2021. 1, 2

    2021

  43. [51]

    Fast image and video colorization using chrominance blending

    Liron Yatziv and Guillermo Sapiro. Fast image and video colorization using chrominance blending. IEEE transactions on image processing, 15(5):1120–1129, 2006. 1

    2006

  44. [52]

    Soundstream: An end- to-end neural audio codec

    Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end- to-end neural audio codec. IEEE/ACM Transactions on Au- dio, Speech, and Language Processing , 30:495–507, 2021. 2

    2021

  45. [53]

    Regularized vector quantization for tokenized im- age synthesis

    Jiahui Zhang, Fangneng Zhan, Christian Theobalt, and Shi- jian Lu. Regularized vector quantization for tokenized im- age synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18467– 18476, 2023. 2, 8

    2023

  46. [54]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shecht- man, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 586–595, 2018. 4, 6

    2018

  47. [55]

    Svcnet: Scribble-based video colorization network with temporal aggregation

    Yuzhi Zhao, Lai-Man Po, Kangcheng Liu, Xuehui Wang, Wing-Yin Yu, Pengfei Xian, Yujia Zhang, and Mengyang Liu. Svcnet: Scribble-based video colorization network with temporal aggregation. IEEE Transactions on Image Process- ing, 2023. 1

    2023

  48. [56]

    Online clustered code- book

    Chuanxia Zheng and Andrea Vedaldi. Online clustered code- book. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 22798–22807, 2023. 2, 8

    2023

  49. [57]

    Movq: Modulating quantized vectors for high- fidelity image generation

    Chuanxia Zheng, Tung-Long Vuong, Jianfei Cai, and Dinh Phung. Movq: Modulating quantized vectors for high- fidelity image generation. Advances in Neural Information Processing Systems, 35:23412–23425, 2022. 2

    2022

  50. [58]

    Towards robust blind face restora- tion with codebook lookup transformer

    Shangchen Zhou, Kelvin Chan, Chongyi Li, and Chen Change Loy. Towards robust blind face restora- tion with codebook lookup transformer. Advances in Neural Information Processing Systems, 35:30599–30611, 2022. 1, 2, 5, 6, 7 11 Efficient Video Face Enhancement with Enhanced Spat...

    2022

  51. [59]

    In step A (second row in Figure 6), we crop the raw videos to increase the face proportion

    Dataset Analysis As illustrated in Figure 6, we preprocess the training videos in three aspects successively, including face proportion (A), face orientation (B), and the presence of text (C). In step A (second row in Figure 6), we crop the raw videos to increase the face prop...

  52. [60]

    Degradation: Consistent v.s

    Quantitative Analysis 7.1. Degradation: Consistent v.s. Stochastic In Stage II, our method applies a consistent degradation ap- proach, where all frames in a training video exhibit con- 1 Degradation PSNR↑ SSIM↑ FVD↓ IDS↑ Rand 26.92 0.8424 156.2 0.9120 Consistent 27.47 0.8641 ...

  53. [62]

    Qualitative Analysis 8.1. Analysis of codebook usage Figure 12 illustrates the variation in codebook utiliza- tion across training iterations, demonstrating that MPR en- hances the convergence speed of codebooks. The conver- gence curves for the spatial and temporal codebooks ...

  54. [63]

    This occurs when the character’s HQOurs Figure 13

    Limitations and Future Work Figure 13 illustrates a failure case in which our method may introduce video blurring. This occurs when the character’s HQOurs Figure 13. Visualization of a failure case generated by our method. motion is excessively dynamic or when camera switching...

  55. [2562]

    Although using a lower reso- lution enhances training efficiency, it adversely affects the quality and granularity of the input, and limits the expres- siveness of the output

    Due to the presence of learnable position embed- dings, the output video resolution must match that of the training videos in Stage II. Although using a lower reso- lution enhances training efficiency, it adversely affects the quality and granularity of the input, and limits t...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.