{"id":"a4d2ef97-1d83-4192-8136-3e107336a6b1","arxiv_id":"2412.00124","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A loss computed in auto-encoded space, instead of raw pixel space, improves the perception-distortion trade-off in GAN-based super-resolution by penalizing fidelity bias while preserving perceptual variance.","lead":"This paper proposes replacing the usual pixel-level reconstruction loss in GAN-based super-resolution with a loss computed in the output space of a pretrained auto-encoder, aiming to penalize only the 'fidelity bias' and preserve realistic texture variance. The authors report consistent gains in perceptual and fidelity metrics across multiple SR backbones and benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (8)'s identification of LAESOP with SE conflates the true posterior mean ψAE(ISR) with the SR predictive mean µ_ŷ; for degradation-consistent outputs LAESOP ≈ 0 even when SE > 0.","rationale":"The reader's weakest assumption focuses on AE faithfulness: whether ψAE is a good approximation of ψ and preserves regressable HF. My concern is more fundamental: even a perfect ψAE does not give the SE of Eq. (2), because ψAE(ISR) estimates E[y|LR(ISR)] (true posterior mean given the SR image's LR), whereas SE uses µ_ŷ, the mean of the SR network's predictive distribution. These coincide only at the optimum. During training, an SR output that is degradation-consistent maps both ψAE(IHR) and ψAE(ISR) to approximately the same true conditional mean, driving LAESOP toward zero regardless of systematic predictive bias. Thus the loss behaves as a learned cycle/consistency loss on the LR channel, not as an SE isolator; the reported improvements may be genuine but the paper's central 'SE/VE separation' narrative is not established. This does not overturn the empirical findings, but it does elevate the required revision: the authors should either prove the identity under training conditions or reframe the contribution as a degradation-consistency loss. Hence I keep the reader's CONDITIONAL verdict.","tokens_in":29431,"tokens_out":14220,"duration_ms":128455,"concrete_test":"Re-derive Eq. (8) from Eqs. (1)-(2) without assuming the SR predictive distribution equals the true posterior; if the step ψAE(ISR) ≈ µ_ŷ is unjustified, the identification fails. Experimentally: with the pretrained AE, construct δ in the nullspace of the learned encoder (ψenc(IHR+δ) ≈ ψenc(IHR)) and set ISR = IHR + δ. If LAESOP(IHR, ISR) ≈ 0 while the point-mass predictive centroid ISR differs from E[y|x] (SE > 0 in Eq. 2), then LAESOP does not estimate SE; verify the gradient is also zero for such δ.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Eq. (8)'s identification LAESOP ≈ SE conflates two different centroids. In Sec. 3, SE is defined (Eq. 2) using µ_ŷ = argmin_µ E_ŷ[L(ŷ, µ)], the mean of the SR network's predictive distribution. In Sec. 4.1, the AE is pretrained to approximate ψ(·) = argmin_µ E_{y∼p(y|x)}[L(y,µ)] with x = φ(·,s), so ψAE(ISR) ≈ E[y|LR(ISR)], the true posterior mean given the SR image's own LR — not the predictive mean µ_ŷ. For an SR network conditioned on LR x whose output is approximately degradation-consistent (LR(ISR) ≈ x), both ψAE(ISR) and ψAE(IHR) are near E[y|x], making LAESOP ≈ 0 regardless of any bias in the predictive distribution. Thus LAESOP behaves as a learned degradation-consistency (cycle) loss, not as an SE estimator. The nonzero gradients that drive the reported improvements come from the encoder's imperfect LR estimation on out-of-distribution SR outputs, not from a principled SE/VE separation. This is a quantitative collapse, not merely an approximation error: even a perfect conditional-mean AE yields a near-zero loss for consistent-but-biased SR outputs. The paper's own 'bottleneck collapse' discussion (Sec. 4.1) acknowledges the degeneracy but asserts joint encoder-decoder training resolves it; joint training does not change the fact that ψAE depends on the input mostly through its LR estimate.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Auto-Encoded Supervision for Optimal Penalization (LAESOP), which replaces the pixel-level reconstruction loss Lpix in GAN-based perceptual super-resolution with an Lp loss computed on the outputs of a frozen auto-encoder (AE) pretrained with Lpix on HR/LR pairs. The central claim is that Lpix jointly minimizes a systematic-effect (SE) term and a variance-effect (VE) term, where VE is the component that causes blurring and the PD trade-off; the AE, pretrained to map an HR image to its LR and back, is said to act as a differentiable approximation of the conditional-mean operator, so that the AE output space contains only the fidelity-bias (SE) component. The authors therefore argue that LAESOP provides strong reconstruction guidance on the fidelity bias without penalizing perceptual variance. The manuscript reports consistent improvements over LDL, ESRGAN, SPSR, and CALGAN baselines on RRDB, SwinIR, and DRCT backbones, on bicubic and real-world ×4 SR, with ablations, PD trade-off curves, spectral analyses, and the AE-PSNR/LR-PSNR metrics.","tokens_in":29648,"tokens_out":7184,"duration_ms":71978,"significance":"If the central claim were correct, the paper would offer a simple and principled replacement for a loss term that is widely used in perceptual SR, with clear practical value: stronger fidelity guidance without the usual blurring side effect. The empirical work is a genuine strength: the method is evaluated across multiple backbones (RRDB, SwinIR, DRCT), multiple benchmarks, real-world settings, and with ablations and PD curves, and the reported improvements are consistent. The paper also honestly discloses cases where the method does not help (e.g., Manga109). However, the conceptual contribution is not established: the identification of the AE output space with the fidelity-bias-only space, and the equation LAESOP ≈ SE, are asserted rather than derived, and the analysis below shows the identification conflates two different conditional-mean operators. The empirical gains may be real and useful, but the theoretical framing as an exact SE/VE separation is likely incorrect and needs substantial revision before the paper can be accepted.","major_comments":[{"comment":"The claimed identification LAESOP ≈ SE conflates two different centroids. In Eq. (2), SE is defined through µ_ŷ = argmin_µ E_ŷ[L(ŷ, µ)], the mean of the SR network's predictive distribution for the given LR input. The pretrained AE, however, is trained with Eqs. (4)-(5) to map an input image to its own LR estimate and then reconstruct the HR from that LR; hence ψAE(I) approximates E[y | LR(I)], the true posterior mean conditioned on the LR of I. For the HR image, LR(IHR) = x, so ψAE(IHR) ≈ E[y|x]. For an SR output with LR(ISR) ≈ x — which is exactly what the adversarial and artifact losses encourage — ψAE(ISR) is also close to E[y|x], making LAESOP ≈ 0 regardless of how far the SR predictive mean µ_ŷ is from µ_y. Thus the loss behaves as a learned degradation-consistency (cycle) loss, not as an SE estimator. The 'bottleneck collapse' discussion in Sec. 4.1 acknowledges the degeneracy for exact downscaling, but joint encoder/decoder pretraining does not change the functional form: after pretraining, ψAE(I) = f(φ(I,s)) for a fixed deterministic mapping f, so the loss still vanishes for any degradation-consistent SR output even when SE is large. The nonzero gradients that drive the reported improvements come from imperfect LR estimation on out-of-distribution SR outputs, not from a principled SE/VE separation.","section":"Sec. 4.1-4.2, Eq. (8)"},{"comment":"The AE-PSNR and LR-PSNR metrics cannot validate the SE-reduction claim. AE-PSNR uses the very AE that defines the training loss, making it circular, a point the authors acknowledge. LR-PSNR is independent of the AE, but it measures the PSNR between the downscaled SR and the original LR, i.e., the degradation consistency φ(ISR,s) ≈ x. Under the analysis in the previous comment, this is precisely the quantity that LAESOP directly optimizes, so the improvements in Table 4 (and Appendix Table 8) are expected for a cycle-consistency loss and do not establish that the fidelity bias µ_ŷ is closer to µ_y. An independent measure of the SR predictive mean, or a controlled experiment that varies the LR consistency of SR outputs while holding their fidelity bias fixed, is needed to support the SE interpretation.","section":"Sec. 5.2, Table 4"},{"comment":"The paper asserts that the pretrained AE 'will act as a differentiable approximation of ψ' and that its output space contains only fidelity-bias factors, but no formal statement or quantitative bound connects the pretraining objective in Eqs. (4)-(5) to the operator ψ(·) := argmin_µ E[L(·,µ)] used in Eq. (1). The supporting evidence is limited to the qualitative spectral visualization in Fig. 6 and the ablation in Table 3. Because the entire conceptual contribution rests on this equivalence, the authors should provide at least a synthetic or semi-synthetic experiment with known µ_y and µ_ŷ (e.g., a Gaussian mixture posterior) showing that LAESOP tracks SE rather than degradation consistency. Without such a test, the central claim that LAESOP 'only minimizes SE' is not supported.","section":"Sec. 4.1 AE pretraining"}],"minor_comments":[{"comment":"The quantities µ_y and µ_ŷ are used in Eq. (1) but defined only after the equation; please state their definitions before Eq. (1) to avoid ambiguity.","section":"Sec. 3, Eq. (1)"},{"comment":"The superscripts † and * in Table 1 are not defined in the table caption; the text defines them, but the table should be self-contained.","section":"Sec. 5.1, Table 1"},{"comment":"The column 'DIV2K100' in Table 5 is not defined and is inconsistent with 'DIV2K-val' used in Table 1; please clarify which split is used.","section":"Sec. 5.2, Table 5"},{"comment":"It would be helpful to state explicitly which baseline numbers are taken from official checkpoints and which are re-trained by the authors, since Table 1 reports both 'LDL*' and 'LDL' columns without a clear explanation of the difference.","section":"Sec. 5.1"},{"comment":"Writing Lpix = ||IHR − ISR||p is an abuse of notation: Lpix is an expected loss over the data distribution, and the equality is exact only for L2 after omitting the irreducible variance term in Eq. (1). Please add a clarifying sentence.","section":"Eq. (7)"}],"recommendation":"major_revision","confidential_remarks":"The empirical method appears to be a useful practical contribution, and the experiments are broad and honest. The main obstacle is the theoretical narrative: the paper's central claim that LAESOP equals SE is very likely incorrect, and the reported metrics (LR-PSNR in particular) actually support a degradation-consistency interpretation. I would be open to a revised version that reframes the contribution as a learned cycle-consistency/reconstruction loss with strong empirical benefits, removes or substantially weakens the SE/VE claims, and adds the suggested control experiments. In its current form, the paper risks misleading readers about what the loss actually optimizes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper's practical claim is credible, but its central theoretical identification is not. The proposed loss LAESOP, which replaces pixel-level Lpix with a distance in the output space of an auto-encoder pretrained with Lpix, consistently improves both distortion and perceptual metrics across RRDB, SwinIR, and DRCT backbones, including real-world SR benchmarks. That's a genuinely useful drop-in modification for GAN-based SR. The authors deserve credit for the breadth of experiments, the PD trade-off curves, and the honest appendix where they concede LAESOP is not a distortion measure and cannot escape the PD trade-off.\n\nThe soft spot is load-bearing. The paper claims that LAESOP ≈ SE, the systematic-error component of Lpix, because the AE approximates the conditional-mean operator ψ(·) = argmin_μ E[L(·, μ)]. But as the stress-test note points out, the AE is pretrained to map an HR image to its LR and back, so ψAE(ISR) is approximately the conditional mean given the LR of the SR image, not the mean of the SR network's predictive distribution. For a degradation-consistent SR output, both ψAE(IHR) and ψAE(ISR) are near E[y|x], so LAESOP ≈ 0 even when the SR prediction's centroid is far from the HR centroid. In short, LAESOP behaves as a learned cycle-consistency loss, not an SE estimator. The paper neither proves the identification nor provides a bound; Sec. 4.1 asserts it. The bottleneck-collapse discussion actually acknowledges the degeneracy but doesn't resolve the centroid conflation.\n\nI don't think this kills the paper. The empirical gains are consistent and the loss is novel. But the narrative overclaims the mechanism. The authors would need to either derive the SE/VE separation formally (perhaps by showing the AE's output space is exactly the set of conditional means, which seems unlikely for a finite-capacity network) or reframe LAESOP as a learned degradation-consistency regularizer that helps GAN training stability. As it stands, a referee should request that before acceptance. Code and checkpoints would also help: currently none are released, which makes the empirical claims harder to verify independently.\n\nBottom line: worth a serious referee, but the theory as written won't survive scrutiny.","headline":"The loss works empirically, but the paper's claim that it isolates systematic error is not established and likely false as stated.","tokens_in":30326,"tokens_out":3539,"would_cite":true,"duration_ms":61531,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Replacing pixel loss with an auto-encoder loss sharpens GAN super-resolution.","keywords":["image super-resolution","perceptual quality","fidelity bias","perceptual variance","auto-encoder","reconstruction loss","perception-distortion trade-off","GAN-based super-resolution"],"falsifier":"On a fixed dataset, take HR images that share the same LR downsampling but differ in high-frequency random texture and measure whether $\\|\\psi_{\\text{AE}}(I_{\\text{HR},1}) - \\psi_{\\text{AE}}(I_{\\text{HR},2})\\|_1$ is zero: any nonzero value shows the AE retains perceptual variance and $\\mathcal{L}_{\\text{AESOP}}$ is not purely SE. Conversely, a pair of HR images that differ only by a sharp, regressable edge but produce identical AE outputs would show the AE dropped regressable structure.","tokens_in":29101,"feed_emoji":"🖼️","tokens_out":7187,"duration_ms":64109,"temperature":0.7,"pith_summary":"The paper asks what exactly makes the pixel-level $\\mathcal{L}_p$ reconstruction loss blur the outputs of GAN-based super-resolution, and claims the culprit is only one of two components. It decomposes $\\mathcal{L}_{\\text{pix}}$ into a systematic-effect term (SE), the distance between the centroids of the HR and SR solution distributions, and a variance-effect term (VE), the penalty for realistic stochastic texture. Minimizing SE improves fidelity without hurting perception, while minimizing VE is what collapses the prediction space onto a blurry average. The paper proposes replacing $\\mathcal{L}_{\\text{pix}}$ with $\\mathcal{L}_{\\text{AESOP}} = \\|\\psi_{\\text{AE}}(I_{\\text{HR}}) - \\psi_{\\text{AE}}(I_{\\text{SR}})\\|_p$, where $\\psi_{\\text{AE}}$ is a pixel-loss-pretrained auto-encoder that, the paper argues, removes the perceptual-variance factor and leaves only the fidelity-bias factor. If correct, this gives GAN-based SR a reconstruction loss that can be weighted at full strength without blurring, improving the perception-distortion trade-off compared with small-coefficient pixel loss or low-pass filtering.","feed_headline":"Auto-encoded loss beats pixel loss in GAN super-resolution","feed_subtitle":"It keeps sharp edges and detail while removing the variance term that makes pixel loss blur textures.","key_machinery":"The load-bearing object is the auto-encoder $\\psi_{\\text{AE}} = \\psi_{\\text{dec}} \\circ \\psi_{\\text{enc}}$, whose encoder maps HR images to LR space (a bottleneck of LR dimensionality) and whose decoder maps back to HR, trained with the LR-reconstruction loss $\\|\\psi_{\\text{enc}}(I_{\\text{HR}}) - I_{\\text{LR}}\\|_p$ and the HR-reconstruction loss $\\|\\psi_{\\text{AE}}(I_{\\text{HR}}) - I_{\\text{HR}}\\|_p$. The AE is then frozen, and $\\mathcal{L}_{\\text{AESOP}}$ is computed in the 'AE space', meaning the output of the decoder rather than the bottleneck. This object functions as the claimed differentiable stand-in for the conditional-mean operator, so the pixel-space SE+VE sum is replaced by an approximation of SE alone.","core_discovery":"The central claim is that the blurring associated with $\\mathcal{L}_{\\text{pix}}$ in perceptual super-resolution is caused specifically by its variance-effect component, not by reconstruction guidance as such. For a symmetric loss and $y \\sim p(y|x)$, the paper rewrites the expected training objective as $\\text{SE}(y,\\hat y) + \\text{VE}(y,\\hat y)$: SE is the fidelity-bias-induced error, the squared distance between the conditional means $\\mu_{\\hat y}$ and $\\mu_y$, while VE is the variance of the estimator around its own mean, which penalizes the realistic, non-regressable high-frequency texture that perceptual SR should preserve. An auto-encoder $\\psi_{\\text{AE}}$ pretrained with $\\mathcal{L}_{\\text{pix}}$ on HR/LR pairs, with a bottleneck of LR dimension, is proposed as a differentiable approximation of the conditional-mean operator $\\psi(\\cdot) = \\arg\\min_\\mu E[\\mathcal{L}(\\cdot,\\mu)]$; therefore $\\mathcal{L}_{\\text{AESOP}} = \\|\\psi_{\\text{AE}}(I_{\\text{HR}}) - \\psi_{\\text{AE}}(I_{\\text{SR}})\\|_p$ penalizes almost only SE. Because fidelity bias includes regressable high-frequency edges, unlike low-pass-filtered loss, and because the loss can be applied with coefficient 1 while the AE is kept frozen to avoid collapse, the paper claims GAN-based SR networks reach a better perception-distortion trade-off and fewer artifacts than networks trained with scaled $\\mathcal{L}_{\\text{pix}}$ or LPF variants.","pith_inferences":["Beyond the paper: the SE/VE decomposition does not depend on super-resolution, so the auto-encoded supervision recipe may transfer to other ill-posed inverse problems such as deblurring, denoising, or compression-artifact removal, where pixel loss likewise blurs stochastic detail; that transfer is not tested here.","Beyond the paper: the paper shows an SRResNet-based AE loses little performance, so the essential ingredient is the LR-dimensional bottleneck plus pixel-loss pretraining rather than a specific SR backbone, suggesting that a cheaper or distilled AE could sustain the method.","Beyond the paper: since $\\mathcal{L}_{\\text{AESOP}}$ can be zero for distinct images sharing one fidelity bias, it is not a distortion metric in the sense of the perception-distortion theorem, so in principle one could push perception without paying a per-pixel fidelity cost; the paper notes this does not make the trade-off disappear entirely."],"forward_implications":["GAN-based SR training can use reconstruction guidance at full strength ($\\lambda_{\\text{AESOP}}=1$) instead of the customary 0.01, because the loss no longer suppresses perceptual variance and therefore no longer conflicts with perceptual-quality losses.","Regressable high-frequency features such as object edges remain supervised, unlike with low-pass filtering, so the method reaches a better point on the perception-distortion curve than LPF-based alternatives.","Stronger reconstruction guidance also reduces GAN artifacts, because the adversarial head is still held aligned by a strong fidelity-bias signal.","The benefit grows with model capacity: the reported gains are larger on SwinIR and DRCT backbones than on RRDB.","The same substitution improves real-world SR benchmarks, where the conflict between pixel loss and perceptual objectives is more severe."],"supporting_citations":[{"why":"Supplies the notion of the optimal fidelity point as the minimum-expected-error average over plausible SR solutions, used to define fidelity bias.","marker":"[31]"},{"why":"Provides the bias-variance decomposition of expected loss that grounds the systematic-effect/variance-effect split in Eq. (1).","marker":"[21]"},{"why":"Shows maximum-likelihood-style training jointly minimizes bias and variance terms, supporting the claim that pixel loss penalizes VE.","marker":"[32]"},{"why":"Establishes the perception-distortion trade-off that the paper uses to argue SE can be reduced without harming perception.","marker":"[2]"},{"why":"Defines the SRGAN framework and the common practice of multiplying pixel loss by a small coefficient, the practice AESOP replaces.","marker":"[30]"},{"why":"LDL is the baseline framework whose loss terms and coefficients are adopted, with only the pixel-loss term modified.","marker":"[37]"},{"why":"Provides the RRDB backbone, the adversarial loss, and the pretrained fidelity-oriented weights used to initialize the AE decoder.","marker":"[54]"},{"why":"Represents the low-pass-filter circumvention that the paper argues removes regressable high-frequency structure, contrasted with the AE.","marker":"[11]"},{"why":"Another low-pass-filter-based approach showing the same limitation that motivates the AE-based separation.","marker":"[69]"}],"fun_headline_variants":["Perceptual SR: auto-encoder space loss keeps edges, avoids blur","AESOP: loss in auto-encoder space improves super-resolution","Replace pixel loss with auto-encoder space for sharper SR","Fidelity without blur: auto-encoder loss for perceptual SR","Auto-encoded supervision: better GAN super-resolution"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that an auto-encoder trained with pixel loss on HR/LR pairs really is a faithful differentiable approximation of the conditional-mean operator, so its output space keeps every regressable edge and drops exactly the stochastic textures; if it drops any regressable structure the loss under-constrains it, and if it keeps any perceptual variance the blurring it claims to avoid is partly reintroduced.","fun_headline_variants_meta":{"raw":{"variants":["Perceptual SR: auto-encoder space loss keeps edges, avoids blur","AESOP: loss in auto-encoder space improves super-resolution","Replace pixel loss with auto-encoder space for sharper SR","Fidelity without blur: auto-encoder loss for perceptual SR","Auto-encoded supervision: better GAN super-resolution"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0004,"raw_usage":{"total_tokens":2190,"prompt_tokens":1146,"completion_tokens":1044,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":762,"completion_tokens_details":{"reasoning_tokens":958}},"tokens_in":762,"tokens_out":1044,"duration_ms":10182,"temperature":1.0,"reasoning_tokens":958,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:34:22.801205+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a fixed dataset, take HR images that share the same LR downsampling but differ in high-frequency random texture and measure whether $\\|\\psi_{\\text{AE}}(I_{\\text{HR},1}) - \\psi_{\\text{AE}}(I_{\\text{HR},2})\\|_1$ is zero: any nonzero value shows the AE retains perceptual variance and $\\mathcal{L}_{\\text{AESOP}}$ is not purely SE. Conversely, a pair of HR images that differ only by a sharp, regressable edge but produce identical AE outputs would show the AE dropped regressable structure.","supporting_citations":[{"cited_title":"Noise-free optimization in early training steps for image super-resolution","cited_arxiv_id":null,"evidence_quote":"Supplies the notion of the optimal fidelity point as the minimum-expected-error average over plausible SR solutions, used to define fidelity bias."},{"cited_title":"Variance and bias for general loss func- tions","cited_arxiv_id":null,"evidence_quote":"Provides the bias-variance decomposition of expected loss that grounds the systematic-effect/variance-effect split in Eq. (1)."},{"cited_title":"The perception-distortion tradeoff","cited_arxiv_id":null,"evidence_quote":"Establishes the perception-distortion trade-off that the paper uses to argue SE can be reduced without harming perception."},{"cited_title":"Details or artifacts: A locally discriminative learning approach to realistic im- age super-resolution","cited_arxiv_id":null,"evidence_quote":"LDL is the baseline framework whose loss terms and coefficients are adopted, with only the pixel-loss term modified."},{"cited_title":"Esrgan: En- hanced super-resolution generative adversarial networks","cited_arxiv_id":null,"evidence_quote":"Provides the RRDB backbone, the adversarial loss, and the pretrained fidelity-oriented weights used to initialize the AE decoder."},{"cited_title":"Wavelet domain style transfer for an effective perception- distortion tradeoff in single image super-resolution","cited_arxiv_id":null,"evidence_quote":"Represents the low-pass-filter circumvention that the paper argues removes regressable high-frequency structure, contrasted with the AE."},{"cited_title":"Perception- distortion balanced admm optimization for single-image super-resolution","cited_arxiv_id":null,"evidence_quote":"Another low-pass-filter-based approach showing the same limitation that motivates the AE-based separation."}],"review_version":1}