{"id":"4be4a6fe-8bc2-4af0-b827-5519ed954535","arxiv_id":"2505.03007","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A public benchmark and ranking of seven user-generated video enhancement methods, based on 150 real-world videos and over 8,000 crowdsourced side-by-side preference votes.","lead":"This paper reports the results of a competition to improve the visual quality of messy, real-world phone and short-form videos, using 150 videos and crowdsourced side-by-side voting by over 8,000 people. The winning methods, the enhanced videos, and all the comparison votes are now public, giving video platforms and researchers a shared benchmark for measuring enhancement quality.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1 ordering depends on unspecified treatment of 'can't choose' votes and an unreported random-balancing step; the released votes may not uniquely reproduce the leaderboard.","rationale":"The reader's weakest assumption identifies both the two-question screening and the unstated handling of 'can't choose' responses. I agree that the tie/indifference handling is the more technically load-bearing gap: the paper's likelihood is explicitly binary, while the data-collection protocol explicitly allowed a third response. The random balancing step is a further source of under-specification because it can change which votes enter the likelihood, and its randomness is not quantified. I am less convinced that the two-question screen is the decisive weakness, since failing it disqualifies all of a worker's votes and the validation pairs are deliberately easy; the remaining risk is diffuse noise rather than a specific, testable bias. The central measurement is probably sound, but the paper should state the tie treatment and release the preprocessing script. Since the reader already returned a CONDITIONAL verdict, my analysis does not move that verdict; it sharpens the condition that must be met.","tokens_in":14882,"tokens_out":6746,"duration_ms":79969,"concrete_test":"Download the released vote set from the GitHub repository and re-estimate the Bradley-Terry scores under three 'can't choose' treatments: (a) discard ties, (b) count each tie as half a win for each side, and (c) fit a Davidson tie-inclusive model. Independently re-run the 'random balancing' step with 100 different seeds on the same raw filtered votes. Report the rank order of Table 1 under each variant. If ranks 4 and 5, or any other adjacent pair, swap under a plausible treatment, the leaderboard is not uniquely determined by the released data; if no swap occurs, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the Table 1 ranking is a fair, statistically grounded evaluation. Section 2.2 gives assessors three options — left, right, or 'can't choose' — but the Bradley-Terry model written down has only two outcomes: P(i≻j)=e^{s_i}/(e^{s_i}+e^{s_j}). The paper never states whether 'can't choose' responses were discarded before fitting, counted as ties, or assigned half-weight. Since the instruction explicitly tells workers to use 'can't choose' when 'the quality is almost the same', the probability of that response carries information about |s_i−s_j|. If ties were dropped, the fit conditions on a preference being expressed; if ties were modeled (e.g., via a Davidson model), point estimates of s_i can shift. The same paragraph says the vote matrix was 'randomly balanced so that each pair had exactly 10 votes' without reporting the random seed, the number of discarded responses, or the number of balancing draws. For near-adjacent systems, another balancing draw could alter the estimated scores. This is load-bearing because ranks 4 (1.140±0.057) and 5 (1.100±0.057) are separated by only about one standard error; the middle of the leaderboard is exactly where the unspecified tie and balancing choices could change the ordering. Unless the exact preprocessing code is released, the reported scores are not uniquely determined by the released vote set.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This NTIRE 2025 challenge report describes a UGC video enhancement benchmark built from 150 real-world videos without ground truth, evaluated by crowdsourced pairwise subjective comparisons. Seven finalist teams passed source-code verification. The organizers fit a Bradley-Terry model to pairwise votes, report per-team scores with 95% confidence intervals relative to the original video in Table 1, and also give public/private score splits and runtime measurements. The paper includes method descriptions for each team and states that all processed videos, votes, and scores will be released publicly. The central claim is that the Table 1 ranking is a fair, statistically grounded evaluation of all submissions.","tokens_in":15169,"tokens_out":3969,"duration_ms":43411,"significance":"If the evaluation pipeline is fully reproducible, this is a valuable benchmark artifact: a public UGC video set, raw crowd votes, and a standard Bradley-Terry aggregation would give the community a reusable subjective benchmark. The paper's strengths include the public release commitment, source-code verification for finalists, reporting of public/private consistency, and the use of a standard statistical model with asymptotically justified confidence intervals. The central derivation is not circular: the ranking is a direct measurement from crowd votes, and no author-written normalization forces the scores. However, the evaluation's reproducibility and statistical completeness are currently compromised by unspecified treatment of the 'can't choose' response and by an unreported random balancing step, both of which can affect the leaderboard.","major_comments":[{"comment":"The paper defines P(i≻j)=e^{s_i}/(e^{s_i}+e^{s_j}) as the only outcome probability, but assessors were explicitly given three options: 'left', 'right', or 'can't choose'. The manuscript never states whether 'can't choose' votes were discarded before fitting, counted as ties, or assigned fractional weight. This is load-bearing because the instruction tells assessors to use 'can't choose' when quality is 'almost the same', so the tie rate carries information about |s_i−s_j|; discarding ties conditions the fit on a preference being expressed, while modeling them (e.g., with a Davidson-type term) can shift the point estimates. Please state the likelihood actually used, report the number of 'can't choose' responses per pair, and provide a sensitivity analysis (e.g., ties dropped vs. modeled as half-votes) to show the Table 1 ordering is robust.","section":"Section 2.2, Eq. (1)"},{"comment":"The same paragraph states that 'the matrix of pairwise votes was randomly balanced so that each pair had exactly 10 votes', but it does not report the random seed, the number of votes discarded, or the number of balancing draws. Since the balancing is random and unreported, the released vote set may not uniquely determine the published scores; a different balancing draw could change the estimates. This is not merely a cosmetic issue: in Table 1, ranks 4 and 5 differ by only 0.040 (1.140 ± 0.057 vs. 1.100 ± 0.057), so the middle of the leaderboard is exactly where this choice could alter the ordering. Please release the exact preprocessing code or, failing that, report the seed and demonstrate that scores are stable across multiple balancing draws.","section":"Section 2.2, balancing step"},{"comment":"The only quality control described is that each assessor answered 20 pairs, 2 of which were validation questions with predefined answers, and only assessors who passed both were kept. The paper does not report the validation pass rate, the number of assessors excluded, or any inter-rater reliability measure. With only two screening questions, noisy or inattentive workers can pass and add unstructured noise to the vote matrix. Please report the screening statistics and, ideally, compute scores with and without low-agreement assessors or with assessor-level random effects to confirm that the ranking is stable.","section":"Section 2.2, assessor screening and reliability"},{"comment":"The confidence-interval formula is given for the difference s_i − s_j, but Table 1 reports intervals only relative to the original video. As a result, the paper does not tell readers whether adjacent systems (e.g., ranks 3/4 or 4/5) are statistically distinguishable from each other. Consistent with the claim of a 'fair and statistically grounded evaluation', please add pairwise confidence intervals for adjacent ranks or an equivalence/significance test for the differences between consecutive leaderboard entries.","section":"Section 2.2, Table 1"}],"minor_comments":[{"comment":"'Zero scores corresponds to the original' should be 'Zero scores correspond to the original'.","section":"Table 1 caption"},{"comment":"The sentence 'Challenge dataset was split into training, validation, and testing sets with sizes of 40, 20, 20, 20, and 90 videos' is confusing because it lists five numbers for four named split types; please rewrite to clarify the three validation sets and the test set.","section":"Section 2.1"},{"comment":"'participants has access only to subjective assessments results' should be 'participants had access only to the subjective assessment results'.","section":"Section 2.2"},{"comment":"'aFusion Network' should be 'a Fusion Network'.","section":"Section 4.7.1"},{"comment":"The phrase 'two optimization strategies to improve objective evaluation metrics' should say 'perceptual' or 'subjective' metrics, since the challenge evaluation is subjective.","section":"Section 4.1.3"},{"comment":"The manuscript text contains a long run of '/uni' codepoint-corruption tokens after Figure 1; this appears to be a PDF-extraction artifact and should be removed from the submitted version.","section":"After Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The reported scores are a direct measurement from crowd votes, so I see no circularity problem. The main risk is reproducibility: the unspecified handling of 'can't choose' votes and the unseeded random balancing could change the leaderboard, especially near the middle of Table 1. This is fixable within the manuscript's scope by reporting the exact preprocessing, tie counts, and sensitivity analyses. I would also ask the editor to ensure the submitted source does not contain the apparent '/uni' corruption artifact before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper is best read as a measurement resource, not a methods breakthrough. What's actually new is a 150-video UGC enhancement benchmark with real-world degradations, processed outputs from seven finalist systems, and—crucially—the raw crowdsourced pairwise comparison votes plus the code to reproduce the processing. That public release is the real contribution, and it's a good one. The evaluation protocol (side-by-side voting, Bradley-Terry aggregation, confidence intervals relative to the original) is standard for the NTIRE series, and the leaderboard itself is a useful snapshot of what works in practice. The winning ShannonLab entry is a sensible four-stage cascade, and the team descriptions give a fair picture of the field's current toolbox.\n\nThe soft spots are mostly in the statistical reporting, and the stress-test concern lands. The Bradley-Terry model written down has only two outcomes, but the assessors were explicitly allowed to answer \"can't choose\" when quality was almost the same. The paper never states whether those votes were discarded, treated as ties, or modeled with an extra outcome. This is not a purely pedantic point: ranks 4 and 5 are separated by 0.04, about one standard error of the score difference. A different handling of the \"can't choose\" responses could plausibly swap them. The same paragraph mentions that the vote matrix was \"randomly balanced\" to exactly 10 votes per pair, but gives no seed, no number of discarded responses, and no analysis code. So the released vote file, as described, does not uniquely determine the reported scores. That's a real gap, but it's fixable by releasing the preprocessing script and stating the tie policy. The confidence intervals in Table 1 are also all relative to the original video, which tells you each system beats the baseline but not whether adjacent ranks are distinguishable; a sentence noting that pairwise CIs overlap in the middle of the table would have been much more honest. Minor issues: there's a corrupted block of \"uni...\" strings in the Introduction that should never have made it to print, and the paper doesn't report inter-rater reliability—though that's a smaller concern given that the raw votes are public.\n\nOverall, the central measurement is probably sound: the top spot is separated from the rest by a wide margin, and the dataset itself is valuable regardless of the exact leaderboard order. Who is this for? Anyone building or benchmarking UGC video enhancement systems, and anyone who wants a real-world subjective preference dataset with per-pair votes. It deserves a serious referee: the missing statistical details are a revision, not a rejection. I'd encourage engaging with it, and I'd likely cite the dataset in my own work.\n\nRecommendation: send to peer review, but ask the authors to clarify tie handling, release the balancing code, and clean up the text corruption.","headline":"A solid NTIRE challenge report that contributes a genuinely useful public UGC video enhancement benchmark with raw pairwise votes; the leaderboard is probably right, but the paper under-specifies how 'can't choose' votes and random balancing were handled, so the exact ordering is provisional until those details are released.","tokens_in":15827,"tokens_out":1890,"would_cite":true,"duration_ms":22720,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pairwise crowd preferences, aggregated by the Bradley-Terry model, rank UGC video enhancement without reference ground truth, putting a four-stage pipeline first.","keywords":["UGC video enhancement","subjective quality assessment","crowdsourced pairwise comparison","Bradley-Terry model","no-reference benchmark","video restoration","NTIRE challenge","compression-aware enhancement"],"falsifier":"Download the released pairwise votes and refit the Bradley-Terry model while counting 'can't choose' responses as ties rather than discarding them; if the top two or adjacent teams swap order, the reported leaderboard depends on an undocumented tie-handling choice.","tokens_in":14699,"feed_emoji":"🎬","tokens_out":11201,"duration_ms":104492,"temperature":0.7,"pith_summary":"This paper reports a challenge that ranks seven UGC video enhancement methods without any reference ground truth. The organizers assembled 150 real user-generated videos suffering from noise, blur, faded colors, and compression artifacts, and had more than 8,000 crowdsourced assessors compare processed clips side by side. The paper claims that aggregating those pairwise votes with the Bradley-Terry model yields a fair, statistically grounded leaderboard, with ShannonLab's four-stage progressive restoration pipeline scoring highest at 1.848 ± 0.060 and every submitted method beating no enhancement. Because all outputs were recompressed at 3000 kbps to mimic short-form video platforms, the ranking is meant to reflect practical deployment. The released videos and pairwise votes let others reuse the benchmark.","feed_headline":"8,000 crowd votes pick the best UGC video enhancer","feed_subtitle":"Pairwise side-by-side votes, aggregated by Bradley-Terry, rank real-world video enhancers with no reference video.","key_machinery":"The load-bearing mechanism is pairwise subjective comparison aggregated by the Bradley-Terry model, a probability model for paired preferences that assigns each video a scalar score $s_i$ such that the probability of preferring $i$ over $j$ is $P(i \\succ j)=e^{s_i}/(e^{s_i}+e^{s_j})$. The organizers gave each pair exactly 10 votes, screened workers with two embedded validation questions, and computed 95% confidence intervals from the inverse Fisher information matrix of the maximum-likelihood estimate. The second mechanism is the winner's four-stage pipeline: CLUT color enhancement, a lightweight U-Net denoiser, BasicVSR++ for temporal stabilization, and a U-Net-like SwinIR refinement stage, linked by residual connections and inference-time feature interpolation.","core_discovery":"The paper's central claim is that subjective side-by-side preference, aggregated through the Bradley-Terry model, can serve as a reproducible no-reference benchmark for UGC video enhancement. On the 150-video final set, ShannonLab's cascaded pipeline of CLUT color enhancement, U-Net denoising, BasicVSR++ temporal restoration, and U-Net-like SwinIR refinement reaches a score of 1.848 relative to the original's zero, with a 95% confidence interval of ±0.060. The order is the same on the 120 public and 30 private videos, which the paper presents as evidence that the leaderboard is consistent rather than an artifact of the visible test portion.","pith_inferences":["Editorial inference: the paper does not state how 'can't choose' votes enter the Bradley-Terry fit, so re-fitting the released votes with ties modeled explicitly would test whether the gaps between adjacent teams are robust.","Editorial inference: the winner's 1.2x color residual amplification and feature interpolation across 30-frame segments are manual inference-time choices; ablating them would separate architectural gains from tuning gains.","Editorial inference: with only 10 votes per pair, the precision of the leaderboard is limited, and the released vote counts could be used to determine how many votes per pair are needed for a stable ranking in future challenges."],"forward_implications":["Under the challenge protocol, every finalist's enhancement is preferred to the original video, since all final scores are positive relative to the zero baseline.","The ranking order is identical on the public and private subsets, suggesting the leaderboard is not driven by overfitting to the visible test videos.","The released data, including processed videos and pairwise votes, make the benchmark reusable for future no-reference UGC enhancement comparisons.","The winning staged architecture, trained with recompression in mind, offers a concrete design pattern for enhancement systems deployed on short-form video platforms."],"supporting_citations":[{"why":"Supplies the Bradley-Terry paired-comparison model and maximum-likelihood estimation that turn crowd votes into leaderboard scores and confidence intervals.","marker":"[3]"},{"why":"BasicVSR++, the temporal restoration stage of the winning ShannonLab pipeline, stabilizes results across frames before recompression.","marker":"[5]"},{"why":"SwinIR, used as the fourth-stage image restoration network in the winning pipeline, is modified into a U-Net-like structure for faster inference.","marker":"[29]"},{"why":"Real-ESRGAN, with its second-order degradation pipeline, is the shared restoration backbone for two other final teams.","marker":"[51]"},{"why":"The LDV3 dataset supplies paired training videos used by the winning and DeepView systems for degradation simulation.","marker":"[63]"},{"why":"CLUT-Net, the color-enhancement stage of the winning pipeline, performs adaptive 3D LUT prediction for fast and robust color correction.","marker":"[66]"}],"fun_headline_variants":["Crowd votes pick winning UGC enhancer without references","8,000 assessors judge video enhancers on real UGC clips","No-reference benchmark: 150 UGC videos, 8,000 decisions","ShannonLab tops UGC enhancement leaderboard via crowdsourcing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire leaderboard rests on the assumption that crowd workers who pass a two-question attention check give reliable preferences, and that the handling of 'can't choose' answers does not change the ranking.","fun_headline_variants_meta":{"raw":{"variants":["Crowd votes pick winning UGC enhancer without references","8,000 assessors judge video enhancers on real UGC clips","No-reference benchmark: 150 UGC videos, 8,000 decisions","ShannonLab tops UGC enhancement leaderboard via crowdsourcing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000752,"raw_usage":{"total_tokens":3313,"prompt_tokens":879,"completion_tokens":2434,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":495,"completion_tokens_details":{"reasoning_tokens":2370}},"tokens_in":495,"tokens_out":2434,"duration_ms":15345,"temperature":1.0,"reasoning_tokens":2370,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:02:33.408274+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Download the released pairwise votes and refit the Bradley-Terry model while counting 'can't choose' responses as ties rather than discarding them; if the top two or adjacent teams swap order, the reported leaderboard depends on an undocumented tie-handling choice.","supporting_citations":[{"cited_title":"Basicvsr++: Improving video super- resolution with enhanced propagation and alignment","cited_arxiv_id":null,"evidence_quote":"BasicVSR++, the temporal restoration stage of the winning ShannonLab pipeline, stabilizes results across frames before recompression."},{"cited_title":"Real-esrgan: Training real-world blind super-resolution with pure synthetic data","cited_arxiv_id":null,"evidence_quote":"Real-ESRGAN, with its second-order degradation pipeline, is the shared restoration backbone for two other final teams."},{"cited_title":"AIM 2022 challenge on super-resolution of compressed image and video: Dataset, methods and results","cited_arxiv_id":null,"evidence_quote":"The LDV3 dataset supplies paired training videos used by the winning and DeepView systems for degradation simulation."},{"cited_title":"Clut-net: Learning adaptively compressed representations of 3dluts for lightweight image enhancement","cited_arxiv_id":null,"evidence_quote":"CLUT-Net, the color-enhancement stage of the winning pipeline, performs adaptive 3D LUT prediction for fast and robust color correction."}],"review_version":1}