{"id":"4530f478-59ed-41c2-853c-6f08682652b4","arxiv_id":"2504.14600","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The NTIRE 2025 face restoration challenge ranked ten restoration systems using a weighted combination of no-reference image quality scores and an identity similarity check, with a three-stage StyleGAN, diffusion, and VAE pipeline taking first place.","lead":"This paper reports the results of the NTIRE 2025 competition on restoring old or damaged face photos. It describes the ten teams' methods and the scoring system used to rank them.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The ranking's central claim rests on an uncalibrated, participant-optimizable IQA score, and the top-two margin is smaller than any single metric's rounding error.","rationale":"The reader's weakest assumption identifies the same load-bearing point: the weighted Score formula is an unvalidated proxy for perceptual quality and identity preservation, and participants explicitly optimize metrics used in that formula. My read agrees with that assessment and adds two supporting observations from the manuscript text. First, Section 3.3 explicitly declares the use of no-reference IQA and simulated degradation pipelines during training and testing to be fair practice, so metric optimization is not an accidental loophole but a sanctioned part of the competition. Second, Section 4.3, 4.4, and 4.5 document concrete instances of teams using CLIPIQA, MUSIQ, and CLIP-IQA at inference or in the loss, meaning the top rankings are partly artifacts of optimizing the evaluation metric itself. Additionally, the top-two total scores differ by only 0.0077, a margin smaller than the rounding precision of the reported component scores, so the headline ranking is fragile. The paper is valuable as a record of the competition and of the methods submitted, and the code release is a real contribution. But the abstract's claim that 'this collaborative effort advances the performance of real-world face restoration' requires a human perceptual baseline or a holdout evaluation to distinguish genuine progress from metric overfitting. Since the reader already recommended CONDITIONAL, my verdict is UNCHANGED: the paper should be accepted as a benchmark report, not as proof of perceptual superiority.","tokens_in":15283,"tokens_out":2784,"duration_ms":27578,"concrete_test":"Re-rank the 10 valid submissions using a holdout perceptual criterion not used in the challenge: a crowdsourced pairwise preference study on a stratified sample of ~100 of the 450 test images, or a set of post-challenge NR-IQA models (e.g., TOPIQ, AHIQ, DOVER) with reported confidence intervals. If AllForFace no longer ranks first, or if the top two swap within the error bars, the challenge ranking should be reported as a metric-optimization benchmark rather than evidence of perceptual state of the art.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the challenge ranking identifies the current state of the art in real-world face restoration. That claim depends entirely on the weighted Score formula in Section 2.2.2, yet the formula is never calibrated against human perceptual judgments for face restoration. Section 3.3 explicitly permits participants to apply no-reference IQA during training and testing, and Sections 4.2-4.4 show that several teams did exactly that: PISA-MAP performs MAP estimation in diffusion latents guided by CLIPIQA and MUSIQ, MiPortrait uses CLIPIQA/MUSIQ thresholds to activate GFPGAN, and AIIA trains with a CLIP-IQA loss. Thus the ranking rewards optimization of the same metrics used to compute the score, and the absence of any human baseline means metric overfitting cannot be distinguished from genuine perceptual improvement. This concern is compounded by the margin between the top two teams: AllForFace scores 4.3638 and IIL scores 4.3561, a difference of 0.0077, which is smaller than the contribution of any single metric's rounding or a plausible run-to-run variation. The reported 'advancement of state of the art' is therefore not established as a perceptual fact; at minimum it is a statement about optimization of this particular composite metric.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports the NTIRE 2025 challenge on real-world face restoration. It describes the challenge setup (450 test images from five face datasets), the evaluation protocol, the weighted image quality score used for ranking, and the submitted solutions, with detailed descriptions of the top five teams and brief notes on the remaining teams. The central claim is that the resulting ranking identifies the current state of the art in real-world face restoration and that the submitted techniques, particularly diffusion-based realism, transformer-based identity preservation, and multi-model fusion, advance the field.","tokens_in":15519,"tokens_out":5230,"duration_ms":45473,"significance":"If the ranking is valid, this is a useful archival record for the community: the paper ships evaluation scripts, source code links, pretrained models, and a structured comparison of thirteen methods on a common real-world benchmark. The qualitative trends reported, such as diffusion for realism and transformers for identity, are plausible and consistent with the wider literature. The main scientific limitation is that the headline conclusions rest entirely on a composite no-reference IQA score whose validity for face restoration is not established, and whose components several participants optimized either during training or at inference time.","major_comments":[{"comment":"The ranking is computed from a weighted sum of CLIPIQA, MANIQA, MUSIQ, Q-Align, NIQE, and FID, yet Section 3.3 explicitly declares no-reference IQA during training and testing to be fair practice. Sections 4.2-4.4 show that three of the top five teams used these exact metrics: PISA-MAP performs MAP estimation in diffusion latents guided by CLIPIQA and MUSIQ, MiPortrait uses CLIPIQA/MUSIQ thresholds to activate GFPGAN, and AIIA trains with a CLIP-IQA loss. The reported ranking therefore partially measures optimization of the scoring metric itself, and the abstract's claim that the challenge 'advances the performance of real-world face restoration' is not substantiated without a human study or at least a cross-check on a held-out metric that no participant was allowed to optimize.","section":"Section 2.2.2 / Section 3.3 / Sections 4.2-4.4"},{"comment":"The Score formula is an ad-hoc weighted combination with no calibration or sensitivity analysis: Q-Align is divided by 5, NIQE and FID are truncated at zero after linear rescaling, while CLIPIQA and MANIQA enter directly. No justification is given for these coefficients or for the truncation points, and no analysis shows that the ranking is stable under plausible alternative weights. Since every conclusion about which method wins and which trends matter is derived from this formula, the paper should either calibrate the weights against human perceptual judgments or report a sensitivity analysis over the weights and thresholds.","section":"Section 2.2.2, Score formula"},{"comment":"The AdaFace identity gate uses per-dataset cosine thresholds of 0.3, 0.5, and 0.6, plus an allowance of up to 10 low-similarity failures, but the manuscript gives no rationale or calibration for these values. Because three teams were declared invalid and excluded from the ranking, the thresholds directly determine the final leaderboard. Without validation of these thresholds against human identity judgments, the paper's claim that the submissions 'maintain identity consistency' is not supported.","section":"Section 2.2.2, Step 1"},{"comment":"The top-two margin is 0.0077 Score points (AllForFace 4.3638 vs. IIL 4.3561), but the paper reports no variance, no confidence intervals, and no human baseline for either the restored images or the original low-quality inputs. FID in particular is computed from only 450 test images against the FFHQ distribution and is known to be high-variance at this sample size. In the absence of repeated runs or subjective ratings, the top-two ordering is not statistically meaningful, and the statement that these methods 'set a new standard for the state-of-the-art' is too strong.","section":"Table 1 and Section 3.4"}],"minor_comments":[{"comment":"The text states that 'This year's winning team, SRC-B' adopted a three-stage pipeline, but Table 1 lists AllForFace as the winner and no team named SRC-B appears anywhere in the paper; this appears to be an internal inconsistency that must be corrected.","section":"Section 3.1, item 3"},{"comment":"The ID loss is written as 1 - <R(IOut) - R(I)>, which is not a valid cosine-similarity expression; the angle brackets are undefined, and the argument should be the cosine similarity between the two embeddings.","section":"Equation (3)"},{"comment":"The table header appears to concatenate several column names ('Adaface Score Failed images ID Validation Total Score') and is hard to parse; please split the columns and clarify what 'ID Validation' means.","section":"Table 1"},{"comment":"References [51] and [54] appear to be the same paper by Sun et al. (PiSA-SR) with different venues and years; please cite the published version consistently and avoid duplication.","section":"References [51] and [54]"}],"recommendation":"major_revision","confidential_remarks":"This is a typical workshop challenge report, and its reproducibility assets are a genuine strength. The main editorial question is whether the journal is willing to publish a headline ranking based on an unvalidated, participant-optimizable composite metric. I would be satisfied if the authors add a clear limitation statement, a sensitivity analysis of the score weights, and preferably a small human evaluation or a hold-out metric cross-check; otherwise the central claim about advancing the state of the art is not supported by the evidence in the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is a competition report, not a controlled scientific study. It documents the NTIRE 2025 real-world face restoration challenge, releases evaluation scripts and code, and gives per-team method descriptions. That is genuinely useful for applied researchers who want a snapshot of what works in practice. But the ranking should not be read as a robust statement about perceptual state of the art. The Score formula in Section 2.2.2 is an uncalibrated composite of NR-IQA metrics, and Section 3.3 explicitly permits participants to use those same metrics during training and inference. Several teams did exactly that: PISA-MAP guides latent optimization with CLIPIQA and MUSIQ, MiPortrait uses those metrics to trigger GFPGAN, AIIA trains a CLIP-IQA loss. So the ranking partly measures who optimized the scoring metric best, not who produces the most perceptually faithful restorations.\n\nThe paper does several things well. The evaluation scripts are public, so the reported numbers are reproducible in principle. The fairness rules are stated plainly, with no pretense that the test set is a clean blind benchmark. The method summaries are clear enough to serve as a mini survey, and the observed trends - diffusion for realism, transformer for identity, multi-model fusion - are consistent with the broader literature. The AdaFace identity gate is a reasonable way to prevent trivial identity drift, even if its thresholds are arbitrary.\n\nThe soft spots are real but need to be sized correctly. The lack of a human baseline is the biggest one: without it, you cannot tell metric overfitting from genuine perceptual improvement. The top-two margin is smaller than the rounding error in the reported components: AllForFace's 4.3638 versus IIL's 4.3561 is 0.0077, while individual terms in the score differ by more than that and cancel out. That suggests the difference between ranks 1 and 2 is noise. The weights in the Score formula are chosen without independent calibration, and the test set is only 450 images. These are limitations inherent to the challenge format, but they directly undermine the abstract's claim that the collaborative effort 'advances the performance' of face restoration. As a benchmark record, the paper is sound; as evidence for a new perceptual SOTA, it is not.\n\nWho gets value from this? Practitioners looking for concrete recipes, and anyone who wants a citable reference for the challenge itself. A serious referee should not desk-reject it; the documentation quality and released code justify archival publication. I would recommend acceptance with revisions that soften the SOTA claims and add an explicit caveat about metric optimization and the uncertainty in the ranking. I would bring it to a reading group focused on restoration methods, and I would probably cite it if I needed to reference the challenge or its benchmark.","headline":"A transparent NTIRE challenge report useful as a recipe survey and benchmark record, but its ranking is a metric-optimization artifact, not a proven perceptual advance.","tokens_in":852,"tokens_out":1038,"would_cite":true,"duration_ms":37654,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The challenge claims the best face restoration splits the task into identity fidelity, texture realism, and naturalness, and ranks entries by a weighted perceptual-quality score.","keywords":["real-world face restoration","blind face restoration","perceptual image quality assessment","identity preservation","diffusion-based restoration","Transformer priors","face restoration challenge","no-reference IQA"],"falsifier":"Recompute the Table 1 rankings without the CLIPIQA term: the second-place team's score becomes $4.3561 - 0.9337 = 3.4224$, which beats the winner's $4.3638 - 0.9517 = 3.4121$, so the reported state of the art changes with the removal of a single metric; a human side-by-side preference test on the 450 test images would show which ranking, if either, matches perception.","tokens_in":15088,"feed_emoji":"🧑","tokens_out":11866,"duration_ms":95755,"temperature":0.7,"pith_summary":"This paper reports the 2025 real-world face restoration challenge, in which 141 people registered, 13 teams submitted working models, and 10 teams passed the identity gate and received final scores. Its central claim is that the ranking produced by the organizers' weighted perceptual-quality score identifies the current best practice for restoring degraded faces, with identity preservation enforced as a pass/fail condition rather than part of the score. The report's main conclusions are that diffusion-based generative priors are the primary source of realistic texture, Transformer-based priors are the main safeguard of identity, and the winning approach combines several generative priors in a serial pipeline rather than relying on one model. If the ranking is valid, these three trends describe the field's current best practice.","feed_headline":"Three-stage fusion wins face restoration challenge","feed_subtitle":"In a 13-team contest, the winner separates identity, texture, and naturalness; ranking uses a weighted quality score.","key_machinery":"The load-bearing object is the challenge's evaluation formula, a single perceptual score built from six quality measures (CLIPIQA, MANIQA, MUSIQ, Q-Align, NIQE, and FID against the FFHQ distribution), gated by an identity check from a pre-trained face-recognition model (AdaFace) with per-dataset cosine thresholds: 0.3 for WIDER-Test and WebPhoto-Test, 0.6 for LFW-Test and CelebChild-Test, and 0.5 for CelebA. This score converts the subjective goal of natural, identity-preserving restoration into one number, and every conclusion in the report about which methods win and which trends matter is derived from it. The identity gate does not enter the score; it only decides whether a submission is ranked at all.","core_discovery":"The central discovery, in the organizers' telling, is that the winning formula for real-world face restoration is not a single architecture but a staged division of labour: first enforce faithfulness to the input identity with a StyleGAN-based generative prior, then add realistic texture with a diffusion module adapted from DiffBIR (a diffusion-based blind image restoration prior), then re-cast the result as an image-reconstruction problem and steer a VAE latent space with vision foundation models to make the texture-rich output look like a natural photograph. The challenge's evaluation procedure supports this ranking by first requiring every submission to pass an identity-similarity check (AdaFace cosine similarity, with dataset-specific thresholds and at most 10 failures), and then ranking the survivors by the weighted score $$\\mathrm{Score} = \\mathrm{CLIPIQA} + \\mathrm{MANIQA} + \\frac{\\mathrm{MUSIQ}}{100} + \\frac{\\mathrm{QALIGN}}{5} + \\max\\left(0, \\frac{10-\\mathrm{NIQE}}{10}\\right) + \\max\\left(0, \\frac{100-\\mathrm{FID}}{100}\\right).$$ The report argues that diffusion-based methods now dominate for realism, Transformer-based codebook priors dominate for identity, and multi-model progressive fusion is the emerging best practice.","pith_inferences":["The fairness rules declare that applying no-reference IQA metrics and simulated degradation pipelines during training and testing is fair practice, so the contest is partly a competition in metric optimization; an independent human-preference study would be needed to know whether the top-ranked outputs are also the ones people prefer.","The top two teams in Table 1 are separated by only 0.0077 score points, and removing the CLIPIQA term alone would flip their order, so the reported 'leading trend' is sensitive to the hand-chosen weights of the formula.","The modular divide-and-conquer recipe of the winner—fidelity, then texture, then naturalness—could transfer to other object-class restoration tasks where faithful identity and photorealistic detail pull against each other, such as text, animals, or documents.","Because the identity gate allows up to 10 failures per submission, a system could deliberately spend that slack on hard images to chase perceptual score, a strategy the ranking would reward."],"forward_implications":["The winning three-stage recipe—identity first via a GAN prior, texture second via a diffusion prior, naturalness third via a VAE guided by a vision foundation model—defines the current template for high-scoring face restoration.","Diffusion-based restoration modules, especially one-step variants distilled from models like SDXL-Turbo, are now fast enough (about 0.21 seconds per image on an A6000 in one entry) to serve as practical components rather than research curiosities.","Because no-reference IQA metrics such as CLIPIQA and MUSIQ are part of both the score and several teams' inference-time optimization, future challenge entries will likely treat these metrics as training and inference objectives directly.","Transformer-based codebook priors remain the standard way to keep identity stable when input faces are heavily degraded, even when the final texture is produced by a diffusion model.","If the claimed ranking is correct, the top ten entries collectively set a new benchmark for real-world face restoration on the five test datasets (CelebChild-Test, LFW-Test, WIDER-Test, CelebA, WebPhoto-Test)."],"supporting_citations":[{"why":"Supplies the AdaFace identity embeddings and per-dataset similarity thresholds used as the pass/fail gate.","marker":"[25]"},{"why":"Provides CLIPIQA, one of the six terms in the ranking score and a metric several teams optimize at inference.","marker":"[60]"},{"why":"Provides MUSIQ, a score term and an inference-time optimization target for top-ranked pipelines.","marker":"[23]"},{"why":"Provides MANIQA, a score term in the ranking formula.","marker":"[74]"},{"why":"Provides Q-Align, a score term in the ranking formula.","marker":"[68]"},{"why":"Provides NIQE, a score term in the ranking formula.","marker":"[81]"},{"why":"Provides the FFHQ distribution used as the FID reference in the score and as the recommended training set.","marker":"[22]"},{"why":"DiffBIR is the diffusion prior that the winning entry adapts for texture realism and that several top teams use as a backbone.","marker":"[35]"},{"why":"CodeFormer is the transformer codebook prior cited as the standard way to preserve identity under heavy degradation.","marker":"[84]"},{"why":"GFPGAN provides the GAN prior and the benchmark test sets (CelebChild-Test, LFW-Test, WebPhoto-Test) used in the evaluation.","marker":"[63]"}],"fun_headline_variants":["Staged fusion wins face restoration: identity, texture, naturalness","Three-stage face restoration wins NTIRE 2025","Winner splits face restoration into identity, texture, realism","Staged approach wins real-world face restoration","How to win face restoration: three-stage fusion"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole ranking stands on the assumption that the hand-composed weighted score—CLIPIQA, MANIQA, MUSIQ, Q-Align, NIQE, and FID terms, with per-dataset AdaFace identity thresholds—faithfully captures what people mean by a natural, identity-preserving face restoration.","fun_headline_variants_meta":{"raw":{"variants":["Staged fusion wins face restoration: identity, texture, naturalness","Three-stage face restoration wins NTIRE 2025","Winner splits face restoration into identity, texture, realism","Staged approach wins real-world face restoration","How to win face restoration: three-stage fusion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000435,"raw_usage":{"total_tokens":2208,"prompt_tokens":935,"completion_tokens":1273,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":1196}},"tokens_in":551,"tokens_out":1273,"duration_ms":9565,"temperature":1.0,"reasoning_tokens":1196,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:44:36.426696+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the Table 1 rankings without the CLIPIQA term: the second-place team's score becomes $4.3561 - 0.9337 = 3.4224$, which beats the winner's $4.3638 - 0.9517 = 3.4121$, so the reported state of the art changes with the removal of a single metric; a human side-by-side preference test on the 450 test images would show which ranking, if either, matches perception.","supporting_citations":[{"cited_title":"Adaface: Quality adaptive margin for face recognition","cited_arxiv_id":null,"evidence_quote":"Supplies the AdaFace identity embeddings and per-dataset similarity thresholds used as the pass/fail gate."},{"cited_title":"Ex- ploring clip for assessing the look and feel of images","cited_arxiv_id":null,"evidence_quote":"Provides CLIPIQA, one of the six terms in the ranking score and a metric several teams optimize at inference."},{"cited_title":"MUSIQ: Multi-scale Image Quality Trans- former","cited_arxiv_id":null,"evidence_quote":"Provides MUSIQ, a score term and an inference-time optimization target for top-ranked pipelines."},{"cited_title":"MANIQA: Multi-dimension attention network for no- reference image quality assessment","cited_arxiv_id":null,"evidence_quote":"Provides MANIQA, a score term in the ranking formula."},{"cited_title":"Q-align: Teaching lmms for visual scoring via discrete text-defined levels","cited_arxiv_id":null,"evidence_quote":"Provides Q-Align, a score term in the ranking formula."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides NIQE, a score term in the ranking formula."},{"cited_title":"A style-based generator architecture for generative adversarial networks","cited_arxiv_id":null,"evidence_quote":"Provides the FFHQ distribution used as the FID reference in the score and as the recommended training set."},{"cited_title":"Diff- BIR: Towards blind image restoration with generative diffu- sion prior","cited_arxiv_id":null,"evidence_quote":"DiffBIR is the diffusion prior that the winning entry adapts for texture realism and that several top teams use as a backbone."},{"cited_title":"Chan, Chongyi Li, and Chen Change Loy","cited_arxiv_id":null,"evidence_quote":"CodeFormer is the transformer codebook prior cited as the standard way to preserve identity under heavy degradation."},{"cited_title":"To- wards real-world blind face restoration with generative facial prior","cited_arxiv_id":null,"evidence_quote":"GFPGAN provides the GAN prior and the benchmark test sets (CelebChild-Test, LFW-Test, WebPhoto-Test) used in the evaluation."}],"review_version":1}