Pith. sign in

REVIEW 2 major objections 28 references

A Projection-Based Surrogate Gradient Interpretation for Neural Codec Wrappers

T0 review · 2 major / 0 minor · reviewed 2026-06-27 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The SCALED surrogate gradient for neural codec wrappers is a first-order local approximation of the codec.

desk verdict The paper reinterprets SCALED as a first-order local approximation of the codec and extends it to full pre/post neural wrappers, with reported BD-rate gains but the exact match to the Taylor expansion needs checking in the math. read the letter →

arxiv 2606.20671 v1 pith:UKSPAMPY submitted 2026-06-12 cs.CV cs.AIeess.SP

Esteban Pesnel , Julien Le Tanou , Michael Ropert , Aline Roumy (COMPACT) , Thomas Maugey (COMPACT) This is my paper · ORCID
classification cs.CVcs.AIeess.SP
keywords neuralwrapperssurrogategradientsvideocodecsnon-differentiableoptimizationSCALEDgradientpre-andpost-processingBD-Rate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper establishes that the SCALED surrogate gradient, previously introduced only as a reparameterization trick, admits an interpretation as a first-order local approximation to the non-differentiable video codec. This view explains its effectiveness for end-to-end training of neural pre- and post-processors without needing an auxiliary mimic network. The same gradient supports both simple downscaling tasks and the harder case of full neural wrapping. Experiments show consistent gains across codecs, quality settings, and downscaling ratios, with BD-Rate reductions reaching -23.59 percent on x264.

What carries the argument

The SCALED gradient, which replaces the non-differentiable codec with a scaled version of the true compression error.

What would settle it

A direct comparison, on identical training runs, between the SCALED gradient and the true partial derivative of a differentiable proxy codec; if the performance gap widens substantially for full wrapping, the local-approximation claim does not hold.

Watch

Extended reading notes

Core claim

The SCALED surrogate gradient can be reinterpreted as a first-order local approximation of the video codec. This reinterpretation supplies insight into why the gradient enables effective training of neural wrappers and remains valid when the wrapper consists of both pre- and post-processing networks.

Load-bearing premise

The scaled gradient introduced as a reparameterization trick continues to function as a mathematically valid first-order approximation even when the wrapper includes both pre- and post-processing networks.

Editorial extensions

If this is right

  • Training becomes possible for complete neural wrappers rather than only downscalers.
  • Performance improvements generalize across different video codecs and quantization parameters.
  • Multiple downscaling ratios can be handled by the same gradient construction.
  • BD-Rate gains of up to 23.59 percent on x264 are obtained relative to standard resampling.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The local-approximation view may extend to other non-differentiable signal-processing blocks whose error surface is locally smooth.
  • One could test whether replacing the true error with a cheaper estimate still preserves the first-order property.
  • The same gradient might be applied to audio or image codecs whose internal decisions are also discrete.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 0 minor

Summary. The paper claims that the SCALED surrogate gradient, originally introduced as a reparameterization trick, admits a projection-based reinterpretation as a first-order local approximation to the video codec. It extends the approach from downscaling to full neural codec wrappers (pre- and post-processing networks) and reports generalization across codecs (x264, VVenC), quality factors, and downscaling ratios, with BD-Rate (PSNR) gains up to -23.59% and -20.07% relative to standard resampling baselines.

Significance. If the local-approximation interpretation is shown to hold exactly (or under clearly stated conditions), the work would supply a useful analytic lens for surrogate gradients in hybrid neural-conventional codecs and justify extending SCALED to more complex wrapper architectures. The reported BD-Rate numbers are concrete and the generalization claim is testable; both would be strengthened by the promised mathematical grounding.

major comments (2)
  1. [Abstract / §3 (interpretation)] The central claim equates the SCALED reparameterization to the gradient of a first-order Taylor expansion of the codec. The abstract states the interpretation but supplies neither the explicit local model nor the derivation showing that the surrogate matches the Jacobian of that model; without this step the reinterpretation remains heuristic rather than identity-level.
  2. [§4 (full neural wrapping)] When extending to full pre- and post-processing networks, the manuscript must verify that the codec's local linear model (and therefore the surrogate gradient) remains independent of the wrapper parameters; otherwise the unbiasedness or variance properties inherited from the original reparameterization may no longer hold. No such verification or regularity condition is referenced in the provided abstract.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments, which help strengthen the mathematical grounding of our claims. We address each major comment below and indicate the revisions we will make.

read point-by-point responses

  1. Referee: [Abstract / §3 (interpretation)] The central claim equates the SCALED reparameterization to the gradient of a first-order Taylor expansion of the codec. The abstract states the interpretation but supplies neither the explicit local model nor the derivation showing that the surrogate matches the Jacobian of that model; without this step the reinterpretation remains heuristic rather than identity-level.

    Authors: The explicit local linear model (first-order Taylor expansion of the codec around the input point) and the derivation showing that the SCALED surrogate equals the gradient of this approximation (via the projection interpretation) are provided in full in Section 3. The abstract summarizes the result at a high level. We will revise the abstract to include one additional sentence referencing the local model and directing readers to the derivation in §3, thereby making the identity-level nature of the claim under the stated conditions explicit. revision: yes

  2. Referee: [§4 (full neural wrapping)] When extending to full pre- and post-processing networks, the manuscript must verify that the codec's local linear model (and therefore the surrogate gradient) remains independent of the wrapper parameters; otherwise the unbiasedness or variance properties inherited from the original reparameterization may no longer hold. No such verification or regularity condition is referenced in the provided abstract.

    Authors: The local linear model is constructed exclusively from the fixed, non-differentiable codec function and is therefore independent of the neural wrapper parameters by definition. The surrogate gradient continues to use the true compression error, preserving the unbiasedness and variance properties of the original reparameterization. We will add an explicit verification paragraph in §4 stating this independence and the regularity condition that the codec remains unchanged. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: interpretation presented as analysis of existing reparameterization without reduction to fitted inputs or self-citation chains.

full rationale

The abstract and provided context present the SCALED surrogate as an existing reparameterization trick whose reinterpretation as a first-order local approximation is offered as an independent analytical insight. No equations are shown that define the approximation in terms of the surrogate (or vice versa) by construction, nor is the central claim load-bearing on self-citations or fitted parameters renamed as predictions. The extension to full neural wrapping is demonstrated empirically across codecs and tasks, remaining self-contained against external benchmarks. This matches the default expectation of non-circularity for papers whose core contribution is reinterpretation rather than a closed derivation loop.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

The central claim rests on the domain assumption that the codec admits a usable first-order local approximation via the compression error; no free parameters or invented entities are indicated in the abstract.

assumptions (1)
  • domain assumption Video codecs admit a first-order local approximation via the compression error as a projection.
    This underpins the reinterpretation of the surrogate gradient as providing insight into effectiveness.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Projection-Based Surrogate Gradient Interpretation for Neural Codec Wrappers." pith.science (2026). https://pith.science/paper/UKSPAMPY

@misc{pith2026260620671,
  author       = {Pith},
  title        = {Pith review of: A Projection-Based Surrogate Gradient Interpretation for Neural Codec Wrappers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UKSPAMPY}},
  note         = {Machine review of arXiv:2606.20671}
}
read the original abstract

Neural wrappers are learned pre-and postprocessing networks designed to enhance the performance of conventional video codecs. Although these approaches can significantly improve compression efficiency, training them remains challenging due to the non-differentiability of video codecs, which arises from the multiple discrete decisions involved in the encoding process. Surrogate gradients have recently emerged as an effective solution for enabling end-to-end learning with conventional codecs. They offer two main advantages: they avoid training an additional network to mimic the codec, and they can improve compression performance. In particular, the recently proposed SCALED method, which leverages the true compression error, has shown strong results for training neural pre-processors such as downscalers. However, this SCALED gradient was originally introduced as a reparameterization trick, which limits its interpretability. In this paper, we show that this surrogate gradient can be interpreted as a first-order local approximation of the video codec, providing insight into its effectiveness. We further demonstrate that it is effective not only for learning downscaling operations, but also for the more challenging task of full neural wrapping with pre-and post-processing networks. Finally, we show that the approach generalizes well across different video codecs, quality factors, and tasks, including multiple downscaling ratios, yielding BD-Rate (PSNR) reductions of up to -23.59% on x264 and -20.07% on VVenC relative to standard resampling baselines.

Figures

Figures reproduced from arXiv: 2606.20671 by the authors.

Figure 1
Figure 1. Overview of the proposed neural wrapper training pipeline. Blue arrows denote the forward pass ( [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Codec idempotence: ϕ 2 vs ϕ (x264, medium preset, average over 30 sequences of Google dataset [15]). Near-unity slope confirms ϕ 2 ≈ ϕ. σ is the residual std from the fitted line y = ax [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Orthogonality: ϵ T yˆ vs ¯ϵ T yˆ (x264, medium preset, 100 sequences). Near-unity slope confirms (ϵ − ¯ϵ) T yˆ ≈ 0. Non-centered mean: points fall on the diagonal ¯ϵ T yˆ ≈ ϵ T yˆ ̸= 0 rather than on the axis, so ¯ϵ ̸= 0 and ϵ is not centered. σ is the residual std from the fitted line y = ax. D. Comparison with learned proxies From this projection perspective, we can derive a set of properties explaining the superi… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 5 canonical work pages

  1. [1]

    ITU-T Recommendation H.274 (ISO/IEC 23002-7): Versatile Supple- mental Enhancement Information for Coded Video Bitstreams , Interna- tional Telecommunication Union Std., 2023

    2023

  2. [2]

    Neural video compression with feature modulation,

    J. Li, B. Li, and Y . Lu, “Neural video compression with feature modulation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024

    2024

  3. [3]

    Dvc: An end-to-end deep video compression framework,

    G. Lu, W. Ouyang, D. Xu, X. Zhang, C. Cai, and Z. Gao, “Dvc: An end-to-end deep video compression framework,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019

    2019

  4. [4]

    Enhanced standard compatible image compression framework based on auxiliary codec networks,

    H. Son, T. Kim, H. Lee, and S. Lee, “Enhanced standard compatible image compression framework based on auxiliary codec networks,” IEEE Transactions on Image Processing , vol. 31, 2021

    2021

  5. [5]

    Video compression with arbitrary rescaling network,

    M. Guo, S. Zhao, H. Jiang, J. Li, and L. Zhang, “Video compression with arbitrary rescaling network,” arXiv preprint arXiv:2306.04202 , 2023

    work page arXiv 2023

  6. [6]

    Self- conditioned probabilistic learning of video rescaling,

    Y . Tian, G. Lu, X. Min, Z. Che, G. Zhai, G. Guo, and Z. Gao, “Self- conditioned probabilistic learning of video rescaling,” in Proceedings of the IEEE/CVF international conference on computer vision , 2021

    2021

  7. [7]

    Optimizing an image coding framework with deep learning-based pre-and post-processing,

    P. Eus ´ebio, J. Ascenso, and F. Pereira, “Optimizing an image coding framework with deep learning-based pre-and post-processing,” in 28th European Signal Processing Conference (EUSIPCO) . IEEE, 2021

    2021

  8. [8]

    An enhanced video compression framework based on rescaling networks,

    Z. Chen and L. Chen, “An enhanced video compression framework based on rescaling networks,” in 2023 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB) . IEEE, 2023

    2023

Show all 28 references
  1. [9]

    Learning a virtual codec based on deep convolutional neural network to compress image,

    L. Zhao, H. Bai, A. Wang, and Y . Zhao, “Learning a virtual codec based on deep convolutional neural network to compress image,” Journal of Visual Communication and Image Representation , vol. 63, 2019

    2019

  2. [10]

    Codec-simulation network for joint optimiza- tion of video coding with pre-and post-processing,

    K. Qiu, L. Yu, and D. Li, “Codec-simulation network for joint optimiza- tion of video coding with pre-and post-processing,” IEEE Open Journal of Circuits and Systems , vol. 2, 2021

    2021

  3. [11]

    Scale-arbitrary invertible image downscaling,

    J. Xing, W. Hu, M. Xia, and T.-T. Wong, “Scale-arbitrary invertible image downscaling,” IEEE Transactions on Image Processing , vol. 32, 2023

    2023

  4. [12]

    Self-asymmetric invert- ible network for compression-aware image rescaling,

    J. Yang, M. Guo, S. Zhao, J. Li, and L. Zhang, “Self-asymmetric invert- ible network for compression-aware image rescaling,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 37, no. 3, 2023

    2023

  5. [13]

    Learned image downscaling for upscaling using content adaptive resampler,

    W. Sun and Z. Chen, “Learned image downscaling for upscaling using content adaptive resampler,” IEEE Transactions on Image Processing , vol. 29, 2020

    2020

  6. [14]

    Learning a convolutional neural network for image compact-resolution,

    Y . Li, D. Liu, H. Li, L. Li, Z. Li, and F. Wu, “Learning a convolutional neural network for image compact-resolution,” IEEE Transactions on Image Processing, vol. 28, no. 3, 2019

    2019

  7. [15]

    Sandwiched compression: Repurposing standard codecs with neural network wrappers,

    O. G. Guleryuz, P. A. Chou, B. Isik, H. Hoppe, D. Tang, R. Du, J. Taylor, P. Davidson, and S. Fanello, “Sandwiched compression: Repurposing standard codecs with neural network wrappers,” arXiv preprint arXiv:2402.05887, 2024

    2024

  8. [16]

    SCALED: Surrogate-gradient for codec-aware learning of downsam- pling in ABR streaming,

    E. Pesnel, J. Le Tanou, M. Ropert, T. Maugey, and A. Roumy, “SCALED: Surrogate-gradient for codec-aware learning of downsam- pling in ABR streaming,” in PCS 2025-Picture Coding Symposium , 2025

    2025

  9. [17]

    Efficient evaluation of quantization-effects in neural codecs,

    W. Mack, A. Mustafa, R. Laganowski, and S. Hijazy, “Efficient evaluation of quantization-effects in neural codecs,” arXiv preprint arXiv:2502.04770, 2025

    2025

  10. [18]

    T. M. Cover and J. A. Thomas, Elements of information theory (wiley se- ries in telecommunications and signal processing) . Wiley-interscience, 2006

    2006

  11. [19]

    The shannon lower bound is asymptotically tight,

    T. Koch, “The shannon lower bound is asymptotically tight,” IEEE Transactions on Information Theory , vol. 62, no. 11, 2016

    2016

  12. [20]

    On the asymptotic tightness of the shannon lower bound,

    T. Linder and R. Zamir, “On the asymptotic tightness of the shannon lower bound,” IEEE Transactions on Information Theory, vol. 40, no. 6, 1994

    1994

  13. [21]

    U-net: Convolutional networks for biomedical image segmentation,

    O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international con- ference, Munich, Germany, October 5-9, 2015, proceedings, part III 18 ...

    2015

  14. [22]

    x264: a free software library for encoding video streams into the h.264/mpeg-4 avc format,

    J.-B. Kempf and VideoLAN, “x264: a free software library for encoding video streams into the h.264/mpeg-4 avc format,” https://www.videolan. org/developers/x264.html, 2024

    2024

  15. [23]

    VVenC: An open and optimized VVC encoder implementation,

    A. Wieckowski, J. Brandenburg, T. Hinz, C. Bartnik, V . George, G. Hege, C. Helmrich, A. Henkel, C. Lehmann, C. Stoffers, I. Zupancic, B. Bross, and D. Marpe, “VVenC: An open and optimized VVC encoder implementation,” inProc. IEEE International Conference on Multimedia Expo Wo...

    2021

  16. [24]

    Estimating or propagat- ing gradients through stochastic neurons for conditional computation,

    Y . Bengio, N. L ´eonard, and A. Courville, “Estimating or propagat- ing gradients through stochastic neurons for conditional computation,” CoRR, vol. abs/1308.3432, 2013

    2013 arXiv

  17. [25]

    What matters in unsupervised optical flow,

    R. Jonschkowski, A. Stone, J. T. Barron, A. Gordon, K. Konolige, and A. Angelova, “What matters in unsupervised optical flow,” in European conference on computer vision . Springer, 2020

    2020

  18. [26]

    Adam: A method for stochastic optimization,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 , 2014

    2014 arXiv

  19. [27]

    Uvg dataset: 50/120fps 4k sequences for video codec analysis and development,

    A. Mercat, M. Viitanen, and J. Vanne, “Uvg dataset: 50/120fps 4k sequences for video codec analysis and development,” in Proceedings of the 2020 ACM Multimedia Systems Conference (MMSys ’20) , 2020

    2020

  20. [28]

    Perceptual video compression with neural wrapping,

    M. U. K. Khan, A. Chadha, M. A. Anam, and Y . Andreopoulos, “Perceptual video compression with neural wrapping,” in Proceedings of the Computer Vision and Pattern Recognition Conference , 2025

    2025

Pith tools

Reviewed June 27, 2026 · model on record in the stance chip above.