{"id":"81ebdb24-9dc9-463c-ba4f-89fb56706c38","arxiv_id":"2412.08939","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Dynamic contrastive knowledge distillation with EMA-generated negatives and VQGAN codebook distribution alignment improves compact image restoration students.","lead":"This paper presents a new way to train small image-restoration models by learning from a larger teacher model, using a dynamically updated contrastive loss and pixel-level distribution alignment. The method reports improved image quality over existing distillation methods on super-resolution, deblurring, and deraining benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The dynamic-EMA mechanism is never isolated: without a fixed-history control, the reported gains cannot be attributed to tracking the student's learning state rather than to the contrastive degradation augmentation.","rationale":"The reader's weakest assumption identifies essentially the same gap, and I agree with it. My concern sharpens it: the headlining novelty is the word 'dynamic.' The paper's evidence for dynamism is only the update-step ablation in Fig. 8(b), which shows that the schedule matters, but not that a tracked lower bound is the cause. A static history model would still yield a contrastive loss, possibly with similar effect. Therefore, the single most load-bearing uncertainty is whether the EMA mechanism contributes anything beyond static contrastive regularization plus degradation-based augmentation. The proposed test is a standard control that directly settles this. Since the reader already conditioned on this gap, my verdict remains CONDITIONAL, i.e., UNCHANGED. I do not see an internally inconsistent argument or a demonstrably false claim; the gap is missing evidence, not a contradicted mechanism.","tokens_in":16580,"tokens_out":4791,"duration_ms":49325,"concrete_test":"Run the exact DCKD training on RCAN x2 SR (same hyperparameters) with three variants: (i) F_his updated by EMA as in Eq. 6; (ii) F_his frozen at the student's initialization for the whole run; (iii) F_his replaced by the frozen teacher (an upper-bound control). Keep DMM, lambda_dcl, lambda_ce, and all training settings identical. If variant (ii) achieves within ~0.05 dB PSNR on Urban100 of variant (i), the 'dynamic' tracking is not responsible for the gain; if (i) is substantially better, the concern is resolved. Also report the distance ||f_anc - f_neg|| over training for variant (i) to show the lower bound actually moves with the student.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Eq. 4 generates negatives by degrading the input and passing through the EMA history model F_his. Eq. 5 then forms a ratio that pulls student features toward the teacher while pushing them away from these negatives. The central claim is that because F_his is an EMA of the student, the negatives track the student's evolving lower bound and thus 'dynamically adjust' the solution space. But nothing in the experiments measures this lower bound or isolates the EMA mechanism. The ablations (Tab. 5, Fig. 8) vary the initial update step and the number of negatives, yet never compare against a static history model, e.g., F_his frozen at initialization or replaced by a fixed pretrained network. Under a static history model, the contrastive loss would still penalize directions where the student deviates from its own older outputs on degraded inputs, acting as a regularizer or data-augmentation term unrelated to dynamic state. Fig. 8(b) shows a peak at update step 1000, but that only shows the schedule matters, not that it implements a tracked lower bound. Because the paper's novelty rests on the word 'dynamic,' the absence of a static-history control is a serious gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DCKD, a knowledge distillation framework for image restoration that combines dynamic contrastive regularization (DCR) with a distribution mapping module (DMM). DCR generates negative samples by applying random degradations to the low-quality input and passing them through an EMA-updated history model of the student, then uses a contrastive loss (Eq. 5) to pull student features toward the teacher output while pushing them away from these negatives. DMM extracts pixel-level category distributions from teacher and student outputs using a pretrained VQGAN encoder and codebook, and aligns them with cross-entropy loss (Eq. 9). The total loss (Eq. 10) adds these terms to reconstruction and vanilla KD losses. Experiments on super-resolution (SwinIR, RCAN), deblurring (NAFNet, Restormer), and deraining (Restormer) report PSNR/SSIM improvements over the compared methods, with ablations on components, degradation types, balancing weights, number of negatives, and update step.","tokens_in":16888,"tokens_out":3786,"duration_ms":33817,"significance":"If the reported results are reproducible, DCKD provides a potentially useful plug-in for image-restoration knowledge distillation, with a concrete mechanism for using learned lower bounds and distribution-level alignment. The paper's strengths include evaluation across three tasks and multiple backbone architectures, ablations of the main components, and a promised public code release. However, the central claim of 'dynamic' adjustment is not directly evidenced: the EMA mechanism is never compared against a static-history counterpart, and the closest contrastive baselines (CSD, Luo et al.) are absent from the main tables. The improvements over the strongest baselines are often modest (about 0.1 dB) and no variance information is given, so the significance at this stage is conditional on filling these gaps.","major_comments":[{"comment":"The manuscript's central claim is that the EMA history model in Eq. (6) tracks the student's learning state and thereby dynamically adjusts the lower bound of the solution space. However, no experiment isolates this dynamic property. In the ablations (Tab. 5, Fig. 8), the authors vary the initial update step and the number of negatives, but they never compare against a control with a static history model (e.g., F_his frozen at initialization or replaced by a fixed pretrained network). Under such a static-history control, the contrastive loss would still penalize deviation of the student from older outputs on degraded inputs, so any gain in that control would indicate that the improvement comes from the degradation-based contrastive augmentation rather than from tracking the student's state. Because the paper's novelty and title rest on the word 'dynamic,' this omitted control is load-bearing. I request adding this ablation, and ideally a plot of the distance between negatives and the student anchor over training to substantiate the 'perceive the student's learning state' claim.","section":"Methodology, Eq. (4)-(6); Experiments, Fig. 8"},{"comment":"The related work identifies CSD (Wang et al. 2021d) and Luo et al. (2023) as the closest contrastive KD baselines with fixed lower bounds, yet neither appears in the main comparison tables (Tabs. 2-4). Table 8 compares only MCLIR on super-resolution; the claim of 'significantly outperforms the state-of-the-art KD methods' cannot be evaluated without these direct competitors on the same tasks and backbones. In addition, DCKD* in Tab. 2 is presented as the combination of DCKD with DUKD, but no DUKD-only result is reported; without that baseline, the gain of DCKD* over DCKD cannot be attributed to the upper-bound combination. Please add these experiments or temper the claims accordingly.","section":"Experiments, Tabs. 2-4 and Table 8"},{"comment":"The reported improvements over the strongest baseline are small (e.g., about 0.1 dB on Urban100 in Tab. 2, 0.17 dB in Tab. 3, and 0.55 dB on Rain100L in Tab. 4), and no error bars, repeated-run statistics, or significance tests are provided. Moreover, the hyperparameters λdcl, λce, N, and the initial update step are selected on the validation sets and then fixed across all tasks (Tabs. 7, Fig. 8), which raises a selection-overfitting concern. For the main claims of consistent improvement, at least three repeated runs with mean and standard deviation (or paired tests) should be reported for the key comparisons.","section":"Experimental Settings, Implementation Details"}],"minor_comments":[{"comment":"The Figure 8 caption contains a corrupted placeholder string ('/uni00000016/...') and the figure labels are unreadable; the figure and caption need to be regenerated.","section":"Figure 8"},{"comment":"Several entries in Table 4 have inconsistent spacing (e.g., '30.54 /0.8969' and '38.02 /0.9762'); please format uniformly.","section":"Table 4"},{"comment":"In the NAFNet and Restormer training details, 'weight deacy' should read 'weight decay'.","section":"Appendix, Implementation Details"},{"comment":"The denominator in Eq. (5) sums over N negative samples, but the scaling behavior of the loss with N is not discussed; a brief explanation of whether and how the loss is normalized would improve clarity.","section":"Eq. (5)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript shares authors with MiPKD (Li et al. 2024) and DUKD (Zhang et al. 2023), and DCKD* directly builds on DUKD; this is not a problem per se, but the absence of DUKD-only results makes the combination claim harder to assess. The code is promised but was not available at the time of review; I encourage the editor to verify reproducibility after release. The paper is within the scope of the journal, but the central 'dynamic' claim needs the static-history control described in my major comments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. What's new: the EMA history model that generates negatives by degrading the input and reconstructing through a moving average of the student, plus the distribution mapping module that aligns pixel-level category distributions via a VQGAN codebook. Both are reasonable extensions of existing contrastive KD ideas, and the paper tests them across SR, deblurring, and deraining with multiple backbones. The ablations show both modules contribute, and the gains are consistent (roughly 0.1–0.25 dB on SR, up to 0.55 dB on Rain100L). That's a useful plug-in result for edge deployment.\n\nThe soft spots are real but not fatal. The big one, which the stress-test note nails, is that the \"dynamic\" claim is never isolated. The EMA mechanism is only compared against no-DCR, not against a fixed history model. Without a static-history control, the observed gains could come from the degradation augmentation and the contrastive ratio itself, not from tracking the student's learning state. The paper even shows in Fig. 8(b) that the initial update step matters, which suggests staleness plays a role, but it doesn't prove that a slowly updated EMA is better than a frozen one. This is a fixable gap, but it should be addressed before the paper's main narrative is accepted.\n\nSecond, the paper omits the closest contrastive baselines, CSD and Luo et al., from the main comparisons. That makes the \"state-of-the-art\" claim weaker than it could be. MCLIR appears only in the appendix for one setting. This is a straightforward omission to correct.\n\nThird, no error bars or repeated runs. Given the modest margins over Logits and MiPKD, some of the differences could be within run-to-run noise. This is common in the field, but it matters more here because the gains are small.\n\nThe citation pattern is mostly fine; the self-citation overlap with MiPKD and DUKD is transparent but warrants independent confirmation. All in all, the work is honest and the method is plausible. It's an incremental contribution, not a breakthrough, but for someone working on low-level vision model compression, it's worth engaging with. I'd send it to a serious referee, with the static-history control and additional baselines as required revisions.","headline":"A broadly tested but incremental KD method whose central \"dynamic\" claim is undermined by a missing control experiment.","tokens_in":17420,"tokens_out":2610,"would_cite":false,"duration_ms":26133,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes DCKD, a knowledge-distillation framework that makes the lower bound of the solution space dynamic via EMA-generated negatives and aligns teacher–student pixel category distributions with a codebook, reporting…","keywords":["image restoration","knowledge distillation","contrastive learning","dynamic lower bound","distribution mapping","super-resolution","deblurring","deraining"],"falsifier":"Train the same DCKD pipeline but replace the EMA history model with a frozen random network, the teacher network, or the current student itself; if PSNR/SSIM do not drop materially for the frozen or teacher replacements, the dynamic tracking of the student's state is not the source of the gain. A second check is to record the feature distances in Eq. 5 over training and confirm the negatives stay below the anchor, since the argument requires the anchor to move away from a retreating lower bound.","tokens_in":16357,"feed_emoji":"🖼️","tokens_out":6746,"duration_ms":56892,"temperature":0.7,"pith_summary":"This paper argues that knowledge distillation for image restoration is held back by fixed solution spaces, where the lower-bound constraint weakens as the student improves. It proposes DCKD, which generates negative examples from an exponential-moving-average copy of the student applied to randomly degraded inputs, and uses contrastive learning to keep the student close to the teacher and ground truth while repelling it from these dynamic negatives. A second module, DMM, aligns pixel-level category distributions between teacher and student outputs through a pretrained VQGAN codebook and cross-entropy loss. Across super-resolution, deblurring, and deraining, the paper reports consistent PSNR/SSIM gains over existing KD methods on CNN and Transformer backbones.","feed_headline":"Dynamic lower bound sharpens image-restoration distillation","feed_subtitle":"EMA-generated negatives and codebook distribution alignment lift student PSNR on SR, deblurring, and deraining.","key_machinery":"The carrier of the argument is the Dynamic Negative Sample Generator (DNSG): a degradation module paired with an exponential-moving-average history model of the student. Given a low-quality input, it produces $N$ negative images that define a moving lower bound in feature space, and the dynamic contrastive loss $L_{\\mathrm{dcl}}$ in Eq. 5 enforces a ratio between the distance to the positive and the summed distances to the negatives. The second carrier is the Distribution Mapping Module (DMM): a pretrained VQGAN encoder and codebook convert both teacher and student outputs into per-pixel soft category distributions aligned by cross-entropy. Together these replace a static solution space with one that tightens as the student learns.","core_discovery":"The central claim is that making the lower bound of the distillation solution space dynamic, rather than fixed, is what unlocks better student performance. Concretely, the student anchor is contrasted against negatives produced by applying random degradations to the low-quality input and reconstructing with a history model that tracks the student by exponential moving average; the dynamic contrastive loss pulls the student toward the teacher/ground-truth positive and pushes it away from these negatives. The paper also introduces distribution mapping, which turns teacher and student outputs into pixel-level category distributions via a VQGAN codebook and aligns them with cross-entropy. The combined loss is $L = L_{\\mathrm{rec}} + L_{\\mathrm{kd}} + \\lambda_{\\mathrm{dcl}} L_{\\mathrm{dcl}} + \\lambda_{\\mathrm{ce}} L_{\\mathrm{ce}}$, and the paper shows this recipe improves student PSNR/SSIM over fixed lower-bound methods and remains compatible with upper-bound-improving methods such as DUKD.","pith_inferences":["The EMA history model behaves like a soft negative curriculum; an untested extension is to anneal the degradation severity or the EMA update rate over training to control how far the negatives trail the anchor.","Because the negatives are derived from the input image itself rather than from a fixed database, the same mechanism should transfer to blind restoration settings where a degradation model is available.","The reliance on a fixed pretrained VQGAN codebook raises the question of whether finer or task-specific codebooks would change the outcome; that is a testable variant the paper does not run.","The repulsion term in Eq. 5 is a sum of distances; whether a max-margin or triplet form of the same dynamic lower bound behaves differently is left open."],"forward_implications":["Students trained with DCKD outperform students trained with logits-only or fixed-lower-bound contrastive KD on super-resolution, deblurring, and deraining benchmarks.","Because DCKD is structure-agnostic, its losses can be attached to both CNN and Transformer restoration backbones without architectural changes.","DCKD can be composed with upper-bound-optimizing distillation methods, and the combination yields larger gains than either component alone.","The DMM's pixel-level category alignment gives low-level vision a distribution-level distillation signal that global distribution matching does not provide.","The main gains come from the dynamic lower bound, since DCKD without any upper-bound enhancement still beats the fixed-lower-bound baseline."],"supporting_citations":[{"why":"Supplies the logits baseline and the overall distillation-loss template that DCKD extends.","marker":"Hinton, Vinyals, and Dean 2015"},{"why":"CSD is the fixed-lower-bound contrastive distillation method that DCKD's dynamic negatives are designed to replace.","marker":"Wang et al. 2021d"},{"why":"Provides the pretrained VQGAN encoder and codebook used by both the contrastive feature extractor and the DMM.","marker":"Esser, Rombach, and Ommer 2021"},{"why":"Provides the degradation recipe (blur, noise, resize, mix) used by the Dynamic Negative Sample Generator.","marker":"Wang et al. 2021c"},{"why":"MiPKD is a main upper-bound baseline and the teacher–student configuration that DCKD is compared against and combined with.","marker":"Li et al. 2024"},{"why":"DUKD is the upper-bound-optimizing method combined with DCKD in the DCKD* variant.","marker":"Zhang et al. 2023"}],"fun_headline_variants":["Dynamic contrastive KD adapts to student for image restoration","Student-state-aware distillation sharpens restoration models","Contrastive distillation with dynamic lower bound boosts SR","EMA negatives and codebook alignment improve KD for restoration","Dynamic distillation space lifts student PSNR in restoration"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that randomly degrading the input and running the EMA copy of the student produces negatives that actually represent the lower boundary of the student's current output space at every training stage; the paper does not measure this directly, and its ablation shows the result is sensitive to the history-model update schedule.","fun_headline_variants_meta":{"raw":{"variants":["Dynamic contrastive KD adapts to student for image restoration","Student-state-aware distillation sharpens restoration models","Contrastive distillation with dynamic lower bound boosts SR","EMA negatives and codebook alignment improve KD for restoration","Dynamic distillation space lifts student PSNR in restoration"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000191,"raw_usage":{"total_tokens":1334,"prompt_tokens":927,"completion_tokens":407,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":543,"completion_tokens_details":{"reasoning_tokens":332}},"tokens_in":543,"tokens_out":407,"duration_ms":4674,"temperature":1.0,"reasoning_tokens":332,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:21:13.335755+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same DCKD pipeline but replace the EMA history model with a frozen random network, the teacher network, or the current student itself; if PSNR/SSIM do not drop materially for the frozen or teacher replacements, the dynamic tracking of the student's state is not the source of the gain. A second check is to record the feature distances in Eq. 5 over training and confirm the negatives stay below the anchor, since the argument requires the anchor to move away from a retreating lower bound.","supporting_citations":[],"review_version":1}