REVIEW 4 major objections 5 minor 15 references
Data-driven Video Codec with Implicit Neural Representations
T0 review · 4 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash
Pith's one-line read This paper claims that a video and its audio track can be stored as the weights of a single sine-activated MLP, and that the resulting representation can be compressed from 9.05 MiB to 2.33 MiB with only modest quality loss.
desk verdict Genuinely new INR architecture combination, but the shared-stack claim is under-tested because there is no matched-capacity separate-network baseline. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is a unified SIREN (sinusoidal representation network): a coordinate MLP with sine activations, separate initialization layers for audio and video, five shared hidden layers in the teacher (three in the student), one linear video output branch, and two linear Siamese audio output branches. Its load-bearing role is to let a single set of weights serve both modalities; the difference between the two audio branches gives the noise estimate that drives the spectral denoiser, while the separate first-layer weight ranges let the network fit audio's high frequencies without corrupting video colors.
What would settle it
Train two separate SIRENs — one audio-only, one video-only — whose combined parameter count equals the unified student's 1.30M, with the same training data, epochs, and loss; if their per-modality PSNR on Video 3 surpasses the unified student's audio PSNR of 24.18 dB by more than a small margin, the shared-representation claim is unsupported; if they do not, the unified design is justified.
Extended reading notes
Core claim
The central discovery is that one shared-hidden-layer MLP can jointly fit pixels and audio amplitudes from coordinates, provided audio and video get separate first-layer weight ranges (audio (-25,25), video (-2/3,2/3)), and that two Siamese audio branches whose outputs differ can estimate reconstruction noise for spectral denoising at decode time. The representation is then treated as a code: response-based knowledge distillation cuts the teacher from 2.37M to a 1.30M-parameter student, 16-bit symmetric quantization halves size again, and LZMA2 provides a small further lossless gain. On the paper's headline test video, the quantized student reaches 28.72 dB video PSNR / 0.75 SSIM and 24.18 d
Load-bearing premise
The load-bearing premise is that sharing hidden layers between audio and video does not degrade either modality relative to separate networks of the same total capacity — a premise the paper never isolates, since no two-network baseline is run.
Editorial extensions
If this is right
- The stored model size is independent of video length and resolution, so the codec becomes relatively more efficient as content grows; conventional bitstreams scale linearly with content.
- At the aggressive end of conventional codec settings (CRF 51), the quantized student can beat H.264/HEVC on video PSNR (28.72 dB versus 22.17–24.16 dB on Video 3), while its audio lands near MP3 at 64 kbps; at higher quality settings conventional codecs win by one to two orders of magnitude in compression ratio.
- Reconstruction quality saturates at 16-bit quantization: below 10 bits quality collapses and above 16 bits additional precision buys nothing visible, fixing the operating point for this pipeline.
- Distillation and quantization are nearly lossless at 16 bits (SQNR about 39 dB), so model compression, not signal coding, is what determines the bitstream.
- Encoding currently means training a network per video for thousands of epochs, which rules out real-time use; lowering the fixed size floor and handling motion are named as the next steps.
Reading between the lines
- Editorial: if the constant-size property holds beyond short clips, then a single model could encode arbitrarily long footage at fixed storage, turning codec design from bit-rate allocation into model-capacity allocation; this is testable by training on hour-long video and checking whether stored size and quality stay flat.
- Editorial: the distillation results on Video 3 — audio PSNR falling from 62.5 dB to 24.2 dB while video PSNR rises — suggest a capacity-allocation trade-off between modalities inside the shared MLP; a per-modality distillation temperature or a loss schedule that protects audio could recover audio quality without hurting video, which is a direct, testable extension.
- Editorial: the Siamese-branch noise-estimate trick is not specific to codecs; it could serve as a self-supervised denoiser in other INR-based generation tasks, since the branch disagreement is obtained without paired noisy/clean data.
- Editorial: because the model maps continuous coordinates to values, the same compressed weights could support frame interpolation or resolution upsampling by evaluating at unobserved coordinates; the paper measures only reconstruction at training coordinates, so testing at interpolated positions would probe this.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a video codec that stores an entire video-plus-audio clip as the weights of a single sinusoidal MLP (SIREN). The network has separate audio and video initialization layers, a shared hidden stack, one video branch, and two Siamese audio branches used for noise estimation. The overfitted teacher is compressed by knowledge distillation into a smaller student, followed by 16-bit symmetric quantization and LZMA2 encoding. The authors report per-modality quality metrics, a bit-width study, comparisons against H.264/HEVC and MP3, and a browser-based prototype. The central claim is that this is the first sample-wise, MLP-only representation in which one set of shared hidden layers serves both pixel and audio functions, and that the pipeline compresses the representation from 9.05 MiB to 2.33 MiB at an overall ratio of 2.61.
Significance. If the shared-hidden-layer architecture were properly validated, the paper would be a compelling proof-of-concept for resolution- and length-independent joint storage of audio and video in one implicit neural representation. The authors are unusually candid: they report negative comparisons with conventional codecs, disclose the fixed model-size floor, and explicitly acknowledge that distillation trades audio quality for video quality (Sec. VI). The parameter table (Table I), the distillation/quantization tables, and the bit-width sweep provide useful reproducibility anchors. However, the central contribution is currently under-supported because the experiments never isolate the effect of the shared stack from the per-modality components, and the headline compression ratio is mislabeled. These issues are fixable within the scope of a revision.
major comments (4)
- [§III-B, §V-D, §VI] The paper's central novelty is that a single shared hidden stack can jointly represent audio and video in one plain MLP. The Introduction asks 'what it costs' (Sec. I), but no experiment compares this architecture against two modality-specific SIRENs with matched total parameter count, the same distillation, and the same quantization. Without such a control, the shared stack's contribution is not isolated; the separate initialization layers and output branches could be doing most of the work. Table VII strengthens this worry: after distillation Video 3 audio PSNR drops from 62.5 dB to 24.2 dB while video PSNR rises, and Sec. VI admits this trade is unresolved. Please add a matched-capacity two-network baseline (or an ablation that reallocates shared layers) and report per-modality PSNR, SSIM, and LSD.
- [Abstract; §V-F, Table VIII] The abstract states that the pipeline 'shrinks the representation from 9.05 MiB to 2.33 MiB, an overall compression ratio of 2.61.' But 9.05/2.33 is approximately 3.88; the value 2.61 is 6.08/2.33, i.e., the original AVI/WAV container size divided by the stored model size. The phrase 'overall compression ratio' conflates two different bases. This is a central quantitative claim and should be corrected by explicitly labeling the numerator (original container vs. teacher representation) and reporting both ratios if both are intended.
- [§V-E, Fig. 9] The 16-bit operating point is chosen from a bit-width sweep on a single video (Video 1) using five reference frames, with no error bars or repeated runs. Since 16-bit quantization is used for the whole pipeline (Sec. III-E) and the abstract claims quality 'saturates at 16 bits,' this is insufficient support for a global claim. Please provide sweeps for at least two other videos, ideally with substantial motion (Videos 3 and 5), or report the 16-bit quantization cost for all five videos and quantify run-to-run variance.
- [§VII vs. §V-D, Table VII] The conclusion says that distillation, quantization, and LZMA2 together leave reconstruction quality 'essentially unchanged.' The authors' own Table VII shows Video 3 audio PSNR dropping from 62.5 dB (teacher) to 24.2 dB (student) during distillation, a 38 dB loss, while video PSNR improves from 21.9 dB to 28.9 dB. The statement is accurate only for the quantization and encoding stages, not for the full pipeline. Rephrase to say that quantization and LZMA2 are nearly lossless, and explicitly attribute the audio/video trade to distillation.
minor comments (5)
- [Tables IV, VI, VII, VIII] The 'Compression ratio' row is computed with different numerators across tables (source video size vs. teacher model size). Define the convention in each caption or once in Sec. III-C, and distinguish 'container compression ratio' from 'representation compression ratio.'
- [§III-C.3] The Siamese difference is treated as the noise estimate and used to drive spectral denoising, but there is no ablation comparing the denoised audio against either a single raw branch or an oracle-noise version. Since the denoiser is part of the decode pipeline, a one-line ablation would substantiate this design choice.
- [§IV-B] Training continues past loss flattening because perceptual metrics keep improving, yet the checkpoint kept is 'the checkpoint with the lowest loss.' Clarify whether the reported metrics come from the lowest-loss checkpoint or a checkpoint selected by perceptual metrics; the two criteria may select different epochs.
- [Table VII] For direct comparison, include a teacher row in Table VII (or in its caption) showing teacher audio PSNR/LSD for Video 3, since the abstract's audio figure (24.18 dB) is for the student and the teacher's 62.5 dB appears only in Table V.
- [§II, Related Work] The description of NeRV-A is brief; specify the architectural difference (frame-wise vs. sample-wise, convolutional vs. MLP-only) more concretely so the claimed gap is clear.
Circularity Check
No circular derivation: all reported quantities are measured from an implemented pipeline, with no self-citation chain and no fit renamed as prediction.
full rationale
The paper is an empirical systems paper rather than a derivation. Its central outputs—teacher/student PSNR, SSIM, LPIPS, LSD, ViSQOL, file sizes, and compression ratios—are all direct measurements after explicitly specified training, distillation, quantization, and encoding procedures. There is no parameter-free law from which a result is derived, and no fitted parameter is relabeled as a prediction. The only definition-like construct is Eq. (2), noise = f1 - f2; however, the paper uses this as an operational heuristic for spectral gating, not as a derivation of the audio-quality claims. Those claims are scored against ground-truth audio, so the denoiser's benefit or harm is empirically falsifiable; indeed the paper reports that the denoiser removes quiet speech in Videos 2 and 5 (Section V-C). No load-bearing argument relies on a self-citation: all cited prior works are external (Sitzmann et al. [3], Chen et al. [4], Gomes et al. [5], Lanzendörfer and Wattenhofer [7], Choudhury et al. [8], etc.), and no uniqueness theorem is imported from the authors' own prior work. The paper openly flags the unisolated audio/video capacity tradeoff in distillation ('we have not isolated the cause', Section V-D) and lists it as a limitation in Section VI; this is a missing control, not a circular step. The absence of a matched-capacity separate-network baseline is an experimental gap that bears on the strength of the 'unified representation' claim, but it does not make any reported result equivalent to its inputs by construction. Likewise, choosing 16 bits from the bit-width saturation curve is a design decision, not a prediction validated on the same data. Hence no significant circularity.
Assumptions & free parameters
free parameters (6)
- First-layer weight ranges =
audio (-25, 25); video (-2/3, 2/3)
- Loss weights lambda_audio, lambda_video =
not specified
- KD mixture weight alpha =
not specified
- Denoiser threshold k in mu + k sigma =
not specified
- Frequency factor omega_0 =
30
- Quantization bit-width =
16 bits
assumptions (5)
- domain assumption SIREN sine activations with standard initialization can fit high-frequency video and audio functions.
- ad hoc to paper A single shared hidden stack can jointly represent audio and video without per-modality collapse.
- domain assumption The difference between two Siamese branch outputs estimates reconstruction noise.
- domain assumption 16-bit per-layer symmetric quantization is near-lossless for these SIREN weights.
- domain assumption Standard distortion metrics (PSNR, SSIM, LPIPS, LSD, ViSQOL) capture codec-relevant quality.
Cite this review
Pith. "Pith review of Data-driven Video Codec with Implicit Neural Representations." pith.science (2026). https://pith.science/paper/5DEIOOSU
@misc{pith2026260715298,
author = {Pith},
title = {Pith review of: Data-driven Video Codec with Implicit Neural Representations},
year = {2026},
howpublished = {\url{https://pith.science/paper/5DEIOOSU}},
note = {Machine review of arXiv:2607.15298}
}
read the original abstract
A conventional codec stores a video as compressed pixel data. We instead store the video, together with its audio track, as the weights of a single sinusoidal representation network (SIREN) that maps space-time coordinates to RGB values and audio amplitudes. The network uses separate audio and video initialization layers, a stack of shared fully connected hidden layers, and three output branches: one for video and two Siamese audio branches whose disagreement is used to estimate and subtract residual noise. The overfitted teacher network is then compressed by response-based knowledge distillation into a smaller student, followed by 16-bit symmetric weight quantization and lossless LZMA2 (xz) encoding. On a 6.08 MiB test video, the quantized student reaches a video PSNR of 28.72 dB with SSIM of 0.75, and an audio PSNR of 24.18 dB with a log spectral distance of 10.69 dB, while the pipeline shrinks the representation from 9.05 MiB to 2.33 MiB, an overall compression ratio of 2.61. A bit-width sweep from 1-bit to 32-bit quantization shows that reconstruction quality saturates at 16 bits. We compare against H.264, HEVC, and MP3, report where the approach falls short of them, and describe a browser-based prototype that trains, transfers, and decodes these models over WebRTC.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
The H.264 video coding standard,
H. Kalva, “The H.264 video coding standard,”IEEE MultiMedia, vol. 13, no. 4, pp. 86–90, 2006
2006
-
[2]
HEVC: The new gold standard for video compression: How does HEVC compare with H.264/A VC?
M. T. Pourazad, C. Doutre, M. Azimi, and P. Nasiopoulos, “HEVC: The new gold standard for video compression: How does HEVC compare with H.264/A VC?”IEEE Consumer Electronics Magazine, vol. 1, no. 3, pp. 36–46, 2012
2012
-
[3]
Implicit neural representations with periodic activation functions,
V . Sitzmann, J. N. P. Martel, A. W. Bergman, D. B. Lindell, and G. Wet- zstein, “Implicit neural representations with periodic activation functions,” inAdvances in Neural Information Processing Systems (NeurIPS), 2020. 8
2020
-
[4]
NeRV: Neural representations for videos,
H. Chen, B. He, H. Wang, Y . Ren, S.-N. Lim, and A. Shrivastava, “NeRV: Neural representations for videos,” inAdvances in Neural Information Processing Systems (NeurIPS), 2021
2021
-
[5]
Video compression with entropy-constrained neural representations,
C. Gomes, R. Azevedo, and C. Schroers, “Video compression with entropy-constrained neural representations,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 18 497–18 506
2023
-
[6]
Boosting neural representations for videos with a conditional decoder,
X. Zhang, R. Yang, D. He, X. Ge, T. Xu, Y . Wang, H. Qin, and J. Zhang, “Boosting neural representations for videos with a conditional decoder,” 2024, arXiv:2402.18152
arXiv 2024
-
[7]
Siamese SIREN: Audio com- pression with implicit neural representations,
L. A. Lanzendörfer and R. Wattenhofer, “Siamese SIREN: Audio com- pression with implicit neural representations,” 2023, arXiv:2306.12957
arXiv 2023
-
[8]
NeRV A: Joint implicit neu- ral representations for videos and audios,
A. Choudhury, P. Singh, and G.-M. Su, “NeRV A: Joint implicit neu- ral representations for videos and audios,” in2024 IEEE International Conference on Multimedia and Expo (ICME), 2024, pp. 1–6
2024
Show all 15 references
-
[9]
Guide to MPEG-1 audio standard,
S. Shlien, “Guide to MPEG-1 audio standard,”IEEE Transactions on Broadcasting, vol. 40, no. 4, pp. 206–218, 1994
1994
-
[10]
Deep contextual video compression,
J. Li, B. Li, and Y . Lu, “Deep contextual video compression,” inAdvances in Neural Information Processing Systems (NeurIPS), 2021
2021
-
[11]
Neural video compression with diverse contexts,
J. Li, B. Li, and Y . Lu, “Neural video compression with diverse contexts,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023
2023
-
[12]
Deep learning in latent space for video prediction and compression,
B. Liu, Y . Chen, S. Liu, and H.-S. Kim, “Deep learning in latent space for video prediction and compression,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 701–710
2021
-
[13]
Effec- tive and efficient video compression by the deep learning techniques,
K. Panneerselvam, K. Mahesh, V . Josephine, and R. Anandan, “Effec- tive and efficient video compression by the deep learning techniques,” Computer Systems Science and Engineering, vol. 45, pp. 1047–1061, 2022
2022
-
[14]
KD-INR: Time-varying volumetric data compression via knowledge distillation-based implicit neural represen- tation,
J. Han, H. Zheng, and C. Bi, “KD-INR: Time-varying volumetric data compression via knowledge distillation-based implicit neural represen- tation,”IEEE Transactions on Visualization and Computer Graphics, vol. 30, no. 10, pp. 6826–6838, 2024
2024
-
[15]
Quantization — PyTorch documentation,
PyTorch, “Quantization — PyTorch documentation,”https://pytorch. org/docs/stable/quantization.html, accessed 2024-10-15. 9
2024
Reviewed August 2, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.