{"id":"3bdf7b3c-4396-4f1f-a71a-fda608c7c163","arxiv_id":"2504.14582","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"The NTIRE 2025 x4 super-resolution challenge results show a new top PSNR of 33.46 dB for restoration and a perceptual winner with composite score 4.3472 across 25 valid submissions.","lead":"The NTIRE 2025 challenge report announces new winners in 4x image super-resolution: a Samsung team leads in pixel accuracy with 33.46 dB PSNR, and a Seoul National University team leads in perceptual realism with a composite score of 4.3472. The report is a useful benchmark of current super-resolution methods and training strategies.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA rankings in Sec. 3 rest on unverified Codalab scores and a code-validation tolerance that is not quantified; a released-code reproduction would settle whether 33.46 dB and the 'two teams surpass 31.94 dB' claim are actually established.","rationale":"The central claim is not a mathematical theorem; it is an empirical benchmark result. Section 2.2 says results are primarily based on Codalab submissions and code is checked for reproduction with 'small discrepancies in precision' accepted, but the preprint does not report any validation outcomes, the size of the tolerance, or an external audit of the hidden-test protocol. I therefore agree with the reader that integrity and code validation are the weakest load-bearing assumptions. I do not escalate to rejection: the large 33.46 dB score is plausible given the winners' 2M-image custom dataset and roughly 480 GPU-hours of training, and the paper's released code and pretrained models are real evidence that can be checked directly. The sharpest concrete sensitivity is the 0.03 dB margin separating BBox from the previous best, which sits inside the protocol's own 'small discrepancies' allowance, so the specific 'two teams surpass' statement is not robust to evaluation precision. A reproducibility audit of Table 1 would confirm or refute the headline claim; until then the CONDITIONAL verdict stands unchanged.","tokens_in":14780,"tokens_out":9370,"duration_ms":86644,"concrete_test":"Have an independent party rerun the official metric script on the submitted outputs against the DIV2K test HR and compare every Table 1 row; then separately load the released SamsungAICamera pretrained model from github.com/zhengchen1999/NTIRE2025_ImageSR_x4, generate x4 outputs on the 100 DIV2K LR test images, and compute PSNR/SSIM with the same 4-pixel-border Y-channel protocol. If Samsung's PSNR reproduces within 0.01 dB and BBox stays above 31.94 under the same script, the restoration SOTA claim is supported; if not, the reported rankings are not reproducible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is an empirical benchmark assertion: SamsungAICamera's 33.46 dB top PSNR, SNUCV's 4.3472 perceptual score, and the statement that two teams surpass last year's 31.94 dB (Sec. 3). Every number in Table 1 comes from the organizers' Codalab evaluation of submitted outputs. The only verification evidence given is Sec. 2.2: 'Code submitted by participants is used for reproduction and verification, with small discrepancies in precision being considered acceptable.' No per-team validation results, no definition of the accepted precision, and no independent check of the Sec. 3.3 rule against using DIV2K test HR images are reported. This is load-bearing because if evaluation or validation shifted by the tolerance the protocol itself permits, the exact ranking could change. In particular, BBox's 31.97 dB is only 0.03 dB above the previous best of 31.94, so the 'two teams surpass' headline is sensitive to sub-0.03 dB differences. None of this makes the results false; it makes them unverified as presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports the organization and outcomes of the NTIRE 2025 image super-resolution (x4) challenge. The challenge follows the classical bicubic degradation setting and contains two tracks: a restoration track ranked by PSNR on the hidden DIV2K test set, and a perceptual track ranked by a composite score of seven IQA metrics defined in Eq. (1). The paper describes the datasets, evaluation protocol (4-pixel border exclusion, Y channel, Codalab submissions with code validation), fairness rules, and the results table (Table 1). It then details the top five methods (SamsungAICamera, SNUCV, BBox, MicroSR, XiaomiMM) and states that the remaining team descriptions are in the supplementary material. The central empirical claims are that SamsungAICamera achieves the top restoration PSNR of 33.46 dB, that SNUCV has the highest perceptual score of 4.3472, and that two teams surpass last year's best PSNR of 31.94 dB.","tokens_in":15044,"tokens_out":4601,"duration_ms":42357,"significance":"If the reported results are trustworthy, the challenge establishes a new state of the art for classical bicubic x4 SR on DIV2K, with a substantial improvement over the NTIRE 2024 winner. The paper has clear strengths: the evaluation protocol is explicit (Sec. 2.2), the test HR images were hidden, the evaluation script is publicly available, and the main results table is internally consistent. As a community benchmark report, it also documents a range of practical techniques (hybrid Transformer-CNN designs, Mamba integration, multi-stage training, CLIP-based filtering, diffusion-based perceptual enhancement) that are useful to practitioners. The main caveats are the absence of quantified code-validation tolerance, the arbitrary perceptual composite score, and the missing supplementary material that is repeatedly cited as containing most of the method descriptions.","major_comments":[{"comment":"The paper states that 'Code submitted by participants is used for reproduction and verification, with small discrepancies in precision being considered acceptable,' but it does not report the accepted tolerance, the per-team reproduction results, or any independent audit. This is load-bearing because the headline claim that 'two teams surpass last year's best PSNR score (31.94 dB)' rests on BBox at 31.97 dB, a margin of only 0.03 dB that lies within the unquantified 'small discrepancies' the protocol permits. As presented, the state-of-the-art improvement claim is unverified; please provide a table of reproduced scores, the exact acceptable deviation, and an analysis of whether the rankings are stable under that tolerance.","section":"Sec. 2.2 and Sec. 3.1"},{"comment":"The perceptual score is an ad hoc sum of seven metrics, with arbitrary scaling choices such as MUSIQ/100 and max(0,(10-NIQE)/10), and no validation against human opinion scores or sensitivity analysis. Since the Track 2 ranking, including the claim that SNUCV 'ranks first with the highest perceptual score (4.3472),' is determined entirely by this formula, the perceptual ranking is only as meaningful as the unstated rationale for these weights and scales. Please justify the formula or demonstrate that the top rankings are robust to reasonable variations of the weights.","section":"Eq. (1), Sec. 2.2 and Sec. 3"},{"comment":"Section 5 and the challenge-results discussion state that the remaining teams' methods and implementation details are in Sec. A of the supplementary materials, but no supplementary material is included in this submission. The abstract promises summaries of the methods of each team, yet only five teams are described in the main text. This gap prevents the paper from delivering its stated coverage; please include the supplementary or move essential method descriptions into the main text.","section":"Sec. 5 and Abstract"}],"minor_comments":[{"comment":"The sentence 'To ensure fairness, participants do not have access to the high-resolution (HR) images from the DIV2K validation set except during the testing phase' is confusing because the testing-phase HR images are said to remain hidden; clarify whether validation HR are available for validation and only test HR are hidden.","section":"Sec. 2.1"},{"comment":"The paper says that 25 teams submitted valid entries, but Table 1 lists 26 rows, including Aimanga and IPCVTeam with N/A ranks; the comparison 'from 20 to 24' against last year also does not match the table; please reconcile the participant counts.","section":"Table 1 and Sec. 3.2"},{"comment":"There are typographical issues such as 'Trainning strategy' and the duplication of the caption 'Team SamsungAICamera' for both Fig. 1 and Fig. 2; figure captions should be distinct and descriptive.","section":"Sec. 4.1"},{"comment":"Metric names are inconsistently capitalized (ManIQA vs. MANIQA), and the ranges needed to interpret terms such as MUSIQ/100 and max(0,(10-NIQE)/10) are not stated; a sentence explaining the metric ranges would help reproducibility.","section":"Eq. (1)"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the unquantified code-validation tolerance. The 0.03 dB margin behind the 'two teams surpass 31.94 dB' claim makes this year's case especially sensitive, so I would require a reproduction/validation appendix before acceptance. The missing supplementary is also a straightforward fix; the paper is otherwise a standard and useful challenge report."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a standard NTIRE challenge report, and the main thing you should know is that the restoration-track numbers are probably trustworthy, but the headline \"two teams surpass last year's best\" is built on a very thin margin and the perceptual track ranking is only as meaningful as its arbitrary formula.\n\nWhat's new: the 2025 rankings, the winning team's recipe (HAT + NAFNet with a 2M-image custom dataset), and the introduction of a perceptual track. The evaluation protocol is explicitly described: hidden DIV2K test HR, 4-pixel border exclusion, Y-channel metrics, and the organizers release evaluation code and pretrained models. That is genuine, citable benchmark data.\n\nSoft spots, in order of severity:\n\n1. The code-validation tolerance is unspecified. Sec. 2.2 says \"small discrepancies in precision being considered acceptable,\" but no per-team validation results or tolerance thresholds are reported. The stress-test note is right that the 0.03 dB gap between BBox (31.97) and last year's best (31.94) is vulnerable to small evaluation shifts. However, the official scores come from Codalab submissions, not from code reproduction, so this mainly threatens the \"reproduced faithfully\" claim, not the submitted-score table itself. Still, the organizers should say how many teams' code was successfully reproduced and at what tolerance.\n\n2. The perceptual score, Eq. (1), is a hand-weighted sum of seven IQA metrics with no validation or justification. The SNUCV team wins Track 2 based on that formula, and it is not clear the formula would correlate with human preference any better than, say, LPIPS alone. The paper should at least acknowledge this as a design choice and discuss its limitations.\n\n3. The promised supplementary materials are missing from the arXiv version, so the remaining teams' methods and affiliations are not actually available. For a report whose stated purpose is to summarize all methods, this is a real gap.\n\n4. Minor: the text says \"25 teams submitting valid entries\" in the abstract and Sec. 3.2, then says \"increase in the number of valid submissions (from 20 to 24)\". Table 1 lists 24 ranked teams plus two unranked. The numbers need reconciling.\n\nI would not call the central SOTA claim false, but it is presented without error bars or significance discussion. For deterministic PSNR on a fixed test set, a 1.5 dB jump is likely real; the 0.03 dB margin is the part to treat cautiously.\n\nWho this is for: anyone working on classical SR who wants the current DIV2K x4 leaderboard and a quick look at what the top teams did. It is a legitimate extension of the NTIRE benchmark series, not a research breakthrough. The methods sections are mostly summaries of known architectures (HAT, NAFNet, Mamba, DRCT), which is fine for a challenge report.\n\nRecommendation: send it to peer review as a workshop/challenge report, but require the authors to (a) quantify the code-validation tolerance and report per-team reproduction status, (b) justify or soften the perceptual-score formula, and (c) either include the supplementary in the arXiv version or remove references to it. With those fixes, it's a solid archival benchmark paper.","headline":"A useful but conventional NTIRE challenge report: the restoration numbers are likely solid, yet the headline SOTA claim hinges on a 0.03 dB margin and the perceptual ranking rests on an unvalidated composite metric.","tokens_in":16095,"tokens_out":2698,"would_cite":true,"duration_ms":24620,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The 2025 edition of the classical ×4 super-resolution benchmark sets a new state of the art, with 33.46 dB PSNR for restoration and 4.3472 for perception.","keywords":["image super-resolution","bicubic downsampling","DIV2K benchmark","PSNR ranking","perceptual quality metrics","Transformer-CNN hybrid","Mamba state-space models","one-step diffusion prior"],"falsifier":"Re-running the submitted code and released models on the public validation split and comparing PSNR and perceptual scores to the recorded numbers would settle reproducibility; inspecting the top teams' training data manifests and logs for the hidden test high-resolution images would settle whether the no-test-data rule held.","tokens_in":14625,"feed_emoji":"🖼️","tokens_out":9155,"duration_ms":75041,"temperature":0.7,"pith_summary":"This paper reports the 2025 edition of a long-running annual image super-resolution competition, run in the classical setting where low-resolution inputs are produced by bicubic downsampling at a factor of 4. The central claim is that the benchmark has moved forward sharply: the best restoration entry reaches 33.46 dB PSNR on the 100-image hidden test set, more than 1.5 dB above the previous year's winning score, and ten entries exceed 31 dB. A newly added perceptual track, ranked by a composite score over six quality metrics, is won by an entry scoring 4.3472. The report attributes these gains to specific practices: hybrid Transformer/convolution designs, Mamba state-space models, progressive patch-size training, frequency-domain losses, and diffusion priors combined with CLIP-based text-prompt losses. If the reported scores are reproduced by the submitted code, these results define the new state of the art for classical super-resolution.","feed_headline":"Top ×4 super-resolution entry hits 33.46 dB PSNR","feed_subtitle":"Perceptual winner scores 4.35; both top results beat last year's state of the art.","key_machinery":"The machinery that carries the results has two parts. Evaluation: restoration rankings rest on PSNR computed on the Y channel after cropping a 4-pixel border, while perceptual rankings rest on the composite score $\\mathrm{Score} = (1-\\mathrm{LPIPS}) + (1-\\mathrm{DISTS}) + \\mathrm{CLIPIQA} + (\\mathrm{MANIQA}+\\mathrm{MUSIQ})/100 + \\max(0,(10-\\mathrm{NIQE})/10)$, with hidden test high-resolution images and organizer code validation intended to keep the comparison fair. Method: for the restoration winner, the load-bearing mechanism is the dynamic fusion of a transformer branch (HAT) and a convolutional branch (NAFNet) trained with alternating L1/L2/wavelet losses and progressive patch sizes; for the perceptual winner, it is a one-step diffusion model acting as a fixed generative prior on top of a Mamba upsampler, steered by an L1 plus LPIPS plus CLIP text-prompt loss toward a \"Good photo\" embedding and away from a \"Bad photo\" embedding.","core_discovery":"On the paper's own terms, the central discovery is that the classical ×4 super-resolution benchmark is not saturated. The top restoration solution dynamically fuses a transformer attention network (HAT) with a convolutional network (NAFNet), trains alternately with L1, L2, and stationary-wavelet-transform losses over progressively larger patches, and reaches 33.46 dB PSNR, beating last year's best of 31.94 dB. The top perceptual solution instead uses a fine-tuned Mamba-based upsampler followed by a frozen one-step diffusion model, trained with a perceptual loss and a CLIP text-prompt loss, and reaches 4.3472 on the composite perceptual score. The paper reads these results as evidence that hybrid architectures and generative priors now set the standard, and that the field's two objectives—pixel fidelity and visual realism—require different designs.","pith_inferences":["Editorial inference: the PSNR gap between the restoration winner (33.46 dB) and the perceptual winner (22.53 dB) suggests the field has split into two near-incompatible optima; combining fidelity and texture realism in one model would require a different reward than either track currently uses.","Editorial inference: because the composite perceptual score includes a no-reference NIQE term, future submissions may be able to inflate their rank by targeting that term; checking the score against human preference judgments would tell whether the ranking measures what it claims.","Editorial inference: the winning restoration entry's custom two-million-image training set makes the architecture contribution hard to isolate; an ablation that fixes the architecture and varies only the data scale would quantify how much of the 1.5 dB gain is data versus design.","Editorial inference: the same dynamic fusion of global-attention and local-convolution branches, plus progressive patch training, is a plausible recipe for other ill-posed restoration tasks such as denoising and face restoration, where both long-range context and fine detail matter."],"forward_implications":["The classical ×4 benchmark still has measurable headroom: a jump of more than 1.5 dB over the previous year's winner means that hybrid architectures and larger training corpora translate directly into fidelity gains.","The new composite perceptual score gives future super-resolution work a single reproducible target that mixes reference-based and no-reference metrics, allowing optimization without a human study at every step.","The leading solutions share transferable training techniques—progressive patch enlargement, wavelet-domain loss, dynamic model fusion—that are not tied to one architecture.","With ten teams above 31 dB, the previous year's state-of-the-art score is now a routine baseline, shifting the practical question from whether a method can reach it to how efficiently it can exceed it."],"supporting_citations":[{"why":"Supplies the NAFNet convolutional branch that the restoration winner fuses with a Transformer branch.","marker":"[2]"},{"why":"Supplies the HAT Transformer, the backbone of the restoration winner and of several other top entries.","marker":"[3]"},{"why":"Provides the previous year's winning PSNR (31.94 dB) and the fusion strategy reused this year.","marker":"[5]"},{"why":"Supplies the one-step diffusion architecture used as a frozen generative prior by the perceptual winner.","marker":"[12]"},{"why":"Supplies the MambaIRv2 upsampler that the perceptual winner fine-tunes before diffusion enhancement.","marker":"[17]"},{"why":"Introduces the wavelet (SWT) loss that the restoration winner alternates with L1 and L2 losses.","marker":"[26]"},{"why":"Provides the large LSDIR training set used by most leading teams.","marker":"[33]"},{"why":"Gives the CLIP text-prompt loss that steers the perceptual winner toward 'Good photo' and away from 'Bad photo'.","marker":"[43]"},{"why":"Defines the DIV2K benchmark and hidden test set on which all scores are measured.","marker":"[50]"},{"why":"Defines LPIPS, used both as a training loss by the perceptual winner and as a component of the perceptual score.","marker":"[65]"}],"fun_headline_variants":["Hybrid nets push ×4 SR to 33.46 dB PSNR","NTIRE 2025: SR hits 33.46 dB, perceptual 4.35","×4 SR challenge: hybrid wins with 33.46 dB","SR benchmark not saturated: 33.46 dB best"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported rankings depend on the integrity of the competition: if any top team trained on the hidden high-resolution test images, or if submitted code does not reproduce the submitted scores, the state-of-the-art claims collapse, and the paper reports only an organizers' code check, not an independent audit.","fun_headline_variants_meta":{"raw":{"variants":["Hybrid nets push ×4 SR to 33.46 dB PSNR","NTIRE 2025: SR hits 33.46 dB, perceptual 4.35","×4 SR challenge: hybrid wins with 33.46 dB","SR benchmark not saturated: 33.46 dB best"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000947,"raw_usage":{"total_tokens":4026,"prompt_tokens":911,"completion_tokens":3115,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":3033}},"tokens_in":527,"tokens_out":3115,"duration_ms":20301,"temperature":1.0,"reasoning_tokens":3033,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:45:21.597979+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-running the submitted code and released models on the public validation split and comparing PSNR and perceptual scores to the recorded numbers would settle reproducibility; inspecting the top teams' training data manifests and logs for the hidden test high-resolution images would settle whether the no-test-data rule held.","supporting_citations":[{"cited_title":"Simple baselines for image restoration","cited_arxiv_id":null,"evidence_quote":"Supplies the NAFNet convolutional branch that the restoration winner fuses with a Transformer branch."},{"cited_title":"Activating more pixels in image super- resolution transformer","cited_arxiv_id":null,"evidence_quote":"Supplies the HAT Transformer, the backbone of the restoration winner and of several other top entries."},{"cited_title":"NTIRE 2024 challenge on image super-resolution (×4): Methods and results","cited_arxiv_id":null,"evidence_quote":"Provides the previous year's winning PSNR (31.94 dB) and the fusion strategy reused this year."},{"cited_title":"Training transformer models by wavelet losses improves quantitative and visual performance in single image super-resolution","cited_arxiv_id":null,"evidence_quote":"Introduces the wavelet (SWT) loss that the restoration winner alternates with L1 and L2 losses."},{"cited_title":"Learn- ing transferable visual models from natural language super- vision","cited_arxiv_id":null,"evidence_quote":"Gives the CLIP text-prompt loss that steers the perceptual winner toward 'Good photo' and away from 'Bad photo'."},{"cited_title":"Ntire 2017 chal- lenge on single image super-resolution: Methods and results","cited_arxiv_id":null,"evidence_quote":"Defines the DIV2K benchmark and hidden test set on which all scores are measured."}],"review_version":1}