Pith. sign in

REVIEW 4 major objections 5 minor 39 references

Unlocking Temporal Flexibility: Neural Speech Codec with Variable Frame Rate

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A neural speech codec that varies frame rate by information density beats constant-rate coding at the same bitrate.

desk verdict VFR for acoustic codecs is a real idea, but the key comparison has an arithmetic slip and the eval reporting is too thin to accept as-is. read the letter →

arxiv 2505.16845 v1 pith:AAVNASNT submitted 2025-05-22 eess.AS cs.AIcs.SD

Hanglei Zhang , Yiwei Guo , Zhihan Li , Xiang Hao , Xie Chen , Kai Yu This is my paper · ORCID
classification eess.AScs.AIcs.SD
keywords neuralspeechcodecvariableframeratetemporalentropybitratecontrolcompressionRVQlanguagemodelsequencelengthreduction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a neural speech codec can operate at a variable frame rate (VFR), rather than the standard constant frame rate (CFR), and that this flexibility improves reconstruction quality rather than merely saving bits. The proposed Temporally Flexible Coding (TFC) estimates the information density of each short speech segment with a simple non-parametric entropy calculation, then routes quiet or predictable segments to coarse, low-frame-rate codes and busy segments to fine, high-frame-rate codes. Built on the DAC backbone for 24 kHz audio, TFC can be set at inference time to any average frame rate between 18.75 Hz and 75 Hz by choosing granularity ratios. The paper reports that at a fixed average bitrate and average frame rate, TFC beats constant-rate DAC on most quality metrics, and that even at lower frame rates than the 75 Hz baseline it matches or improves perceptual quality. If true, this would let speech codecs and the generative models built on them run with shorter token sequences and tunable latency without sacrificing audio quality.

What carries the argument

The load-bearing mechanism is the entropy-routed granularity mask. TFC computes a per-window Shannon entropy $H(T)$ from Gaussian affinities of raw waveform amplitudes to a set of bins, then derives binary masks $b_c\, b_m\, b_f$ at coarse, medium, and fine resolutions by comparing those entropy sequences to quantiles specified by the granularity ratios $r_c, r_m, r_f$. These masks decide which temporal windows are coded at which resolution; the chosen quantized vectors are repeated and merged by element-wise mask multiplication to form a single latent, and a conditional hierarchical decoder progressively refines coarse features into fine ones. The masks make frame-rate allocation a deterministic, user-controllable function of the signal's measured information density.

What would settle it

Take a set of evaluation utterances and replace TFC's entropy-based mask with an oracle mask that, at each average frame rate, assigns fine frames to the windows where the fine-versus-coarse reconstruction error difference is largest, then compare the resulting quality scores; if the oracle mask is noticeably better, waveform entropy is not tracking coding difficulty.

Watch

Extended reading notes

Core claim

The paper's central claim is that entropy-guided variable frame rate allocation yields better audio reconstruction than constant frame rate operation at the same average bitrate and average frame rate, and it also shortens the token sequence. The method, TFC, creates three temporal resolutions from the encoder's latent: fine frames at 75 Hz, medium at 37.5 Hz, and coarse at 18.75 Hz. For each window, a non-parametric Shannon entropy computed on raw waveform amplitude determines which resolution is used, via quantile thresholds set by user-chosen granularity ratios. Quantized vectors from the chosen resolution are repeated and fused into a single sequence aligned with the finest grid, and a conditional hierarchical decoder refines the signal from coarse to fine. In the reported tables and figures, DAC+TFC outperforms the DAC baseline across most metrics, and mixing frame rates at the same average rate yields consistent gains over pure 37.5 Hz or the 75 Hz constant-rate baseline, supporting the claim that time-varying frame rates are beneficial.

Load-bearing premise

The load-bearing premise is that the entropy of raw waveform samples in a short window measures how many code frames that window needs for good reconstruction, so routing low-entropy windows to coarse frames will not hurt quality.

Editorial extensions

If this is right

  • A single trained TFC codec can span average frame rates from 18.75 Hz to 75 Hz by changing the granularity ratios at inference, replacing the need for multiple codecs or for codebook dropout as the only bitrate knob.
  • At a fixed average bitrate and average frame rate, VFR allocation improves Mel distance, STFT distance, UTMOS, STOI, and WER relative to constant frame rate operation, as shown in the paper's Figure 2a and Table 1.
  • Lowering the frame rate to 56.25 Hz or below can still match or exceed the 75 Hz constant-rate baseline on perceptual metrics, which implies the token sequence can be shortened without hurting quality.
  • Shorter token sequences at a given bitrate mean faster autoregressive generation for speech language models, since generation speed depends on frame count, not codebook count.
  • TFC is designed to be orthogonal to other low-frame-rate codec techniques and can be stacked with them.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper trains only with granularity ratios $(0.4, 0.3, 0.3)$; a natural test is whether retraining on the actual inference ratio distribution changes the gap to CFR, which would reveal how much of the gain comes from the routing rule versus the added capacity of the multi-resolution decoder.
  • The entropy signal is computed on raw amplitude; using the codec's own reconstruction error or a learned importance map as the routing signal is a direct alternative that could be compared against the entropy mask.
  • Because TFC changes frame rate, it should also cut the computational cost of decoding and the memory footprint of autoregressive decoding (KV cache) beyond the token-count reduction, though the paper does not measure this.
  • VFR allocation could be applied to semantic tokens from self-supervised models, where the entropy signal may be even stronger, potentially yielding very low average frame rates for speech language models.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Temporally Flexible Coding (TFC), a plug-and-play module that introduces variable frame rate (VFR) into neural speech codecs built on a DAC backbone. TFC computes a non-parametric temporal entropy from raw waveform amplitudes and uses quantile-based masks to assign each temporal segment one of three granularities (75, 37.5, or 18.75 Hz), with the average frame rate controlled by user-supplied granularity ratios. The decoder fuses the three resolutions through a conditional hierarchical design. Experiments compare DAC+TFC with the DAC baseline in constant frame rate (CFR) mode at several bitrates, and compare VFR against CFR at a nominal average of 3 kbps / 37.5 Hz. The paper reports improved reconstruction metrics for TFC in both comparisons and claims that a single model can span average frame rates from 18.75 to 75 Hz while preserving quality.

Significance. If the VFR-versus-CFR comparison is properly controlled, the paper addresses a real gap: existing neural codecs vary bitrate through codebook dropout but still emit a fixed number of frames per second, which is suboptimal for downstream autoregressive models. The TFC design is simple, loss-agnostic, and orthogonal to other low-frame-rate codec efforts, so the core idea has clear practical value. Credit is due for the explicit CFR/VFR comparison, the use of multiple objective and perceptual metrics, and the attempt to connect bitrate and sequence-length reduction. The main significance hinges on whether the reported VFR gains are measured at a truly fixed average bitrate and average frame rate; the current manuscript does not yet demonstrate this.

major comments (4)
  1. [Section 4.3 / Section 3.2] The illustrative granularity ratios r_f=0.2, r_m=0.7, r_c=0.1 do not correspond to the stated average 3 kbps / 37.5 Hz setting. With the F=75 Hz backbone, the average number of transmitted frames per second is 75*0.2 + 37.5*0.7 + 18.75*0.1 = 43.125 Hz, and with N_q=8 10-bit codebooks this gives 43.125*80 = 3450 bps, not 3000 bps. Thus the example VFR configuration is not rate-matched to the 37.5 Hz / 3 kbps CFR baseline. The paper must report the exact granularity ratios and the realized average bitrate and frame rate for every point in Figure 2a, and either correct the example or confirm that the plotted points use ratio sets that genuinely average to 37.5 Hz. Without this, the 'consistent improvement' in Figure 2a could be explained by a higher average bitrate.
  2. [Section 4.3 / Figure 2 / Table 1] No evaluation corpus, utterance count, or confidence intervals are reported, and the WER numbers are internally anomalous. In Table 1, DAC at 6 kbps has WER 4.125, which is worse than DAC at 3 kbps (2.443) and worse than DAC+TFC at 3 kbps (2.193); this is surprising and unexplained. Since WER is one of the two metrics used in Figure 2b, a single ASR run without error bars cannot support the claim that TFC configurations 'consistently outperform' the 75 Hz baseline. Please specify the test set, report error bars or significance tests, and discuss the WER anomalies.
  3. [Section 3.1 / Eq. (2)-(4)] The entropy-based router is not validated. The Gaussian affinity entropy depends on the unstated sigma and bin count N, and the paper provides no evidence that raw-waveform amplitude entropy tracks codec reconstruction difficulty. The VFR-versus-CFR comparison shows that mixing granularities helps under the current training recipe, but it does not establish that entropy-based routing, rather than the mixture of granularities itself, is responsible. An ablation comparing entropy routing with uniform or random routing at the same granularity mixtures, or a correlation analysis between H(T) and per-segment reconstruction error, is needed to support the causal interpretation in the abstract and Section 3.1.
  4. [Section 4.1 / Section 4.2 / Section 4.3] The model is trained only with granularity ratios r_f=0.4, r_m=0.3, r_c=0.3, but inference advertises arbitrary ratios (e.g., 0.2/0.7/0.1 and the varying ratios in Figure 2b). The paper does not demonstrate that the model generalizes across the full claimed ratio range; if part of the VFR gain comes from matching the training distribution of ratios, the flexibility claim is overstated. Please provide a ratio-wise breakdown of results or otherwise show that performance is stable across the 18.75-75 Hz average-frame-rate range.
minor comments (5)
  1. [Section 3.1 / Section 4.1] The values of sigma, N, and epsilon in Eqs. (2)-(4) are never reported, and no sensitivity analysis is given; please state the hyperparameters used in the reported experiments.
  2. [Abstract / Section 1 / Section 2.2] The paper claims to introduce VFR to neural speech codecs 'for the first time' but Section 2.2 discusses VFR in semantic tokens from self-supervised models; please qualify the claim as first in acoustic neural codecs.
  3. [Section 4.3 / Figure 2] The figure axes and the meaning of '75Hz granularity' on the x-axis are not defined in the text; please clarify how each plotted point is generated and how the average frame rate and bitrate are held fixed.
  4. [Table 1] The 6 kbps DAC row appears to have a formatting issue where the Nq value is merged with the Mel distance; please fix the table layout.
  5. [Section 4.3] The sentence 'Figure 2a obviously shows...' is informal; please replace 'obviously' with a quantitative statement of the margin and its variability.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: TFC's quality claim rests on an external empirical comparison, not on a self-referential derivation.

full rationale

The paper's central claim is that entropy-guided variable frame rate allocation improves reconstruction quality relative to constant frame rate at matched average bitrate. This claim is supported by an empirical comparison using external metrics (Mel/STFT distance, UTMOS, STOI, WER). The entropy estimate (Eqs. 2-4) is computed non-parametrically from raw waveform amplitudes with fixed bins and an unreported sigma; it is not derived from the codec's reconstruction error, and no ablation tunes it to the evaluation metrics. The granularity masks (Eq. 5) are deterministic quantile functions of these entropy sequences, and the resulting frame counts and bitrates follow arithmetically from the user-specified ratios. That arithmetic is a definition of the method, not a prediction derived from something else. The quality improvement over CFR is not forced by construction: the metrics are independent of the entropy values and masks. There are two self-citations ([10], [20]), but they appear only as contextual examples of prior efficiency efforts and are not load-bearing for the main result. The reviewer's concern about Figure 2a is a potential experimental-control flaw: the example ratios r_f=0.2, r_m=0.7, r_c=0.1 yield 43.125 Hz and 3.45 kbps, not the stated 37.5 Hz and 3 kbps, so the VFR-vs-CFR comparison may not be rate-matched. However, this is an empirical validity issue, not circularity: no equation in the paper reduces the quality outcome to its input definitionally or via a fitted parameter. Therefore no circular step is identified, and the score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central method rests on the entropy proxy assumption and the fixed-ratio training schedule; the key hyperparameters sigma and N are unreported. No new physical or model entities are introduced.

free parameters (3)
  • sigma (Gaussian affinity sharpness)
    Sigma in Eq. (2) controls the sharpness of the affinity distribution and therefore the entropy estimate; no value is reported in the paper.
  • N (number of amplitude bins)
    N bins span [-1, 1] in Eq. (2)-(4); the value is never stated, yet it directly affects entropy values.
  • training granularity ratios (r_f, r_m, r_c) = 0.4, 0.3, 0.3
    These ratios are chosen by hand for training (Section 4.1); the model is evaluated at other ratios, so this choice is a free parameter of the method.
assumptions (3)
  • domain assumption Raw waveform amplitude entropy is a valid proxy for the information density relevant to neural codec reconstruction quality.
    Section 3.2 uses entropy sequences h_f, h_m, h_c computed on x[t] to allocate frame rates; no evidence links this to codec reconstruction difficulty.
  • domain assumption The model trained with fixed granularity ratios generalizes to arbitrary ratios at inference.
    Training uses r=(0.4, 0.3, 0.3) (Section 4.1) while inference uses various ratios (Section 4.2, 4.3); no experiment verifies this generalization.
  • standard math Shannon entropy with Gaussian affinity (Eq. 2-4) is a valid non-parametric density estimate.
    Background technique adapted from [29, 17]; standard but relies on the parameters above.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Unlocking Temporal Flexibility: Neural Speech Codec with Variable Frame Rate." pith.science (2026). https://pith.science/paper/AAVNASNT

@misc{pith2026250516845,
  author       = {Pith},
  title        = {Pith review of: Unlocking Temporal Flexibility: Neural Speech Codec with Variable Frame Rate},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AAVNASNT}},
  note         = {Machine review of arXiv:2505.16845}
}
read the original abstract

Most neural speech codecs achieve bitrate adjustment through intra-frame mechanisms, such as codebook dropout, at a Constant Frame Rate (CFR). However, speech segments inherently have time-varying information density (e.g., silent intervals versus voiced regions). This property makes CFR not optimal in terms of bitrate and token sequence length, hindering efficiency in real-time applications. In this work, we propose a Temporally Flexible Coding (TFC) technique, introducing variable frame rate (VFR) into neural speech codecs for the first time. TFC enables seamlessly tunable average frame rates and dynamically allocates frame rates based on temporal entropy. Experimental results show that a codec with TFC achieves optimal reconstruction quality with high flexibility, and maintains competitive performance even at lower frame rates. Our approach is promising for the integration with other efforts to develop low-frame-rate neural speech codecs for more efficient downstream tasks.

Figures

Figures reproduced from arXiv: 2505.16845 by the authors.

Figure 1
Figure 1. (a) Proposed framework of Temporally Flexible Coding. (b) VFR allocation examples under different granularity ratios. For each resolution, we compute entropy values based on Sec￾tion 3.1, where the size of T is set to the corresponding recep￾tive field, and temporal segments are slid by the corresponding stride. Denote hf , hm, hc as the scalar entropy sequences for fine, medium and coarse resolution, respectively, … view at source ↗
Figure 2
Figure 2. DAC+TFC (solid lines) is evaluated in VFR mode compared to CFR performances (dashed lines). segments for 1000k iterations. Reconstruction quality is measured via Mel and STFT dis￾tances (configurations following [3]), with the latter better cap￾turing high-frequency fidelity. Additionally, perceptual qual￾ity is assessed using UTMOS [32], a MOS prediction system that correlates strongly with human ratings, as well a… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 23 canonical work pages

  1. [1]

    Introduction Recent audio/speech codecs have integrated neural networks to achieve high-fidelity audio reconstruction [1, 2, 3]. These mod- els are typically trained end-to-end and consist of three main components: an encoder that compresses the input signal into compact representations, a quantization module that discretizes these representations, and a ...

    work page

  2. [2]

    Background This section situates our work within the broader context of de- velopment in feature compression, providing a comprehensive discussion of terms CBR, VBR, CFR, and VFR, as well as the necessary notations for further understanding. 2.1. Constant versus Variable Bitrate in Neural Codecs The concepts ofConstant Bitrate (CBR)andVariable Bitrate (VB...

    arXiv 2025

  3. [3]

    diffusing

    Temporally Flexible Coding In this section, we will describe our proposed TFC strategy. TFC is a plug-and-play module that can be integrated to various codec backbones like DAC [3], without introducing additional losses for optimization. 3.1. Measuring Information Density by Temporal Entropy To measure the information density of a speech segment for granu...

    work page

  4. [4]

    Architecture and Setup We implement TFC upon the DAC [3] backbone, which achieves high-fidelity reconstruction performance among vari- ous existing codecs [30]

    Experiments 4.1. Architecture and Setup We implement TFC upon the DAC [3] backbone, which achieves high-fidelity reconstruction performance among vari- ous existing codecs [30]. We use its official configuration for 24kHz audio that produces RVQ codes at 75Hz frame rate. Hence in our DAC+TFC framework, the finest granularity is F=75Hz. Our hierarchical de...

    work page

  5. [5]

    TFC effectively balances bitrate and audio quality while reducing sequence length

    Conclusion We present Temporally Flexible Coding (TFC), a method that introduces Variable Frame Rate (VFR) into neural speech codecs to dynamically adjust temporal resolution based on in- formation density. TFC effectively balances bitrate and audio quality while reducing sequence length. Due to resource con- straints, our implementation is limited to one...

    work page

  6. [6]

    92370206), the Shanghai Municipal Science and Technol- ogy Major Project (2021SHZDZX0102) and the Key Re- search and Development Program of Jiangsu Province, China (No.BE2022059)

    Acknowledgements This work was supported by the China NSFC Project (No. 92370206), the Shanghai Municipal Science and Technol- ogy Major Project (2021SHZDZX0102) and the Key Re- search and Development Program of Jiangsu Province, China (No.BE2022059)

  7. [7]

    SoundStream: An End- to-End Neural Audio Codec,

    N. Zeghidour, A. Luebs, A. Omranet al., “SoundStream: An End- to-End Neural Audio Codec,”IEEE/ACM Trans. ASLP ., vol. 30, pp. 495–507, 2021

    work page 2021

  8. [8]

    High Fidelity Neu- ral Audio Compression,

    A. D ´efossez, J. Copet, G. Synnaeveet al., “High Fidelity Neu- ral Audio Compression,”Transactions on Machine Learning Re- search, 2023

    2023

Show all 39 references
  1. [9]

    High-Fidelity Audio Compression with Improved RVQGAN,

    R. Kumar, P. Seetharaman, A. Luebset al., “High-Fidelity Audio Compression with Improved RVQGAN,”Proc. NeurIPS, vol. 36, 2024

    2024

  2. [10]

    AudioLM: A Language Modeling Approach to Audio Generation,

    Z. Borsos, R. Marinier, D. Vincentet al., “AudioLM: A Language Modeling Approach to Audio Generation,”IEEE/ACM Trans. ASLP ., vol. 31, pp. 2523–2533, 2023

    2023

  3. [11]

    Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers,

    S. Chen, C. Wang, Y . Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y . Liu, H. Wang, J. Li, L. He, S. Zhao, and F. Wei, “Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers,” IEEE/ACM Trans. ASLP ., pp. 1–15, 2025

    2025

  4. [12]

    Neural machine transla- tion of rare words with subword units,

    R. Sennrich, B. Haddow, and A. Birch, “Neural machine transla- tion of rare words with subword units,” inProceedings of the 54th Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers). Association for Computational Lin- guistics, Aug. 2016, pp...

    2016

  5. [13]

    Why Do Speech Lan- guage Models Fail to Generate Semantically Coherent Out- puts? A Modality Evolving Perspective,

    H. Wang, H. Wang, Y . Guoet al., “Why Do Speech Lan- guage Models Fail to Generate Semantically Coherent Out- puts? A Modality Evolving Perspective,”arXiv preprint arXiv:2412.17048, 2024

    2024

  6. [14]

    V ALL-E 2: Neural codec language models are hu- man parity zero-shot text to speech synthesizers,

    S. Chen, S. Liu, L. Zhou, Y . Liu, X. Tan, J. Li, S. Zhao, Y . Qian, and F. Wei, “V ALL-E 2: Neural codec language models are hu- man parity zero-shot text to speech synthesizers,”arXiv preprint arXiv:2406.05370, 2024

    2024 arXiv

  7. [15]

    TacoLM: Gated attention equipped codec language model are efficient zero-shot text to speech synthesizers,

    Y . Song, Z. Chen, X. Wang, Z. Ma, G. Yang, and X. Chen, “TacoLM: Gated attention equipped codec language model are efficient zero-shot text to speech synthesizers,” inProc. ISCA In- terspeech, 2024, pp. 4433–4437

    2024

  8. [16]

    Fast and high- quality auto-regressive speech synthesis via speculative decod- ing,

    B. Li, H. Wang, S. Zhang, Y . Guo, and K. Yu, “Fast and high- quality auto-regressive speech synthesis via speculative decod- ing,” inProc. IEEE ICASSP, 2025

    2025

  9. [17]

    Low Frame-Rate Speech Codec: a Codec Designed for Fast High-quality Speech LLM Training and Inference,

    E. Casanova, R. Langman, P. Neekharaet al., “Low Frame-Rate Speech Codec: a Codec Designed for Fast High-quality Speech LLM Training and Inference,”arXiv preprint arXiv:2409.12117, 2024

    2024 arXiv

  10. [18]

    Moshi: A Speech- Text Foundation Model for Real-Time Dialogue,

    A. D ´efossez, L. Mazar ´e, M. Orsiniet al., “Moshi: A Speech- Text Foundation Model for Real-Time Dialogue,”arXiv preprint arXiv:2410.00037, 2024

    2024 arXiv

  11. [19]

    SemantiCodec: An ultra low bitrate semantic audio codec for general sound,

    H. Liu, X. Xu, Y . Yuan, M. Wu, W. Wang, and M. D. Plumbley, “SemantiCodec: An ultra low bitrate semantic audio codec for general sound,”IEEE Journal of Selected Topics in Signal Pro- cessing, vol. 18, no. 8, pp. 1448–1461, 2024

    2024

  12. [20]

    WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Model- ing,

    S. Ji, Z. Jiang, W. Wanget al., “WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Model- ing,” inProc. ICLR, 2025

    2025

  13. [21]

    Scaling transformers for low-bitrate high-quality speech coding,

    J. D. Parker, A. Smirnov, J. Pons, C. Carr, Z. Zukowski, Z. Evans, and X. Liu, “Scaling transformers for low-bitrate high-quality speech coding,” inProc. ICLR, 2025

    2025

  14. [22]

    Variable Bitrate Resid- ual Vector Quantization for Audio Coding,

    Y . Chae, W. Choi, Y . Takidaet al., “Variable Bitrate Resid- ual Vector Quantization for Audio Coding,”arXiv preprint arXiv:2410.06016, 2024

    2024 arXiv

  15. [23]

    Once-for-all: Controllable generative image compression with dynamic granu- larity adaption,

    A. Li, F. Li, Y . Liu, R. Cong, Y . Zhao, and H. Bai, “Once-for-all: Controllable generative image compression with dynamic granu- larity adaption,” inProc. ICLR, 2025

    2025

  16. [24]

    Entropy-based variable frame rate analysis of speech signals and its application to ASR,

    H. You, Q. Zhu, and A. Alwan, “Entropy-based variable frame rate analysis of speech signals and its application to ASR,” in Proc. IEEE ICASSP, vol. 1, 2004, pp. I–549

    2004

  17. [25]

    Single-Codec: Single-Codebook Speech Codec towards High-Performance Speech Generation,

    H. Li, L. Xue, H. Guoet al., “Single-Codec: Single-Codebook Speech Codec towards High-Performance Speech Generation,” in Proc. ISCA Interspeech, 2024, pp. 3390–3394

    2024

  18. [26]

    LSCodec: Low-Bitrate and Speaker-Decoupled Discrete Speech Codec,

    Y . Guo, Z. Li, C. Duet al., “LSCodec: Low-Bitrate and Speaker-Decoupled Discrete Speech Codec,”arXiv preprint arXiv:2410.15764, 2024

    2024 arXiv

  19. [27]

    SNAC: Multi- Scale Neural Audio Codec,

    H. Siuzdak, F. Gr ¨otschla, and L. A. Lanzend¨orfer, “SNAC: Multi- Scale Neural Audio Codec,”arXiv preprint arXiv:2410.14411, 2024

    2024 arXiv

  20. [28]

    Speaking from coarse to fine: Improving neural codec language model via multi-scale speech coding and generation,

    H. Guo, F. Xie, D. Yang, X. Wu, and H. Meng, “Speaking from coarse to fine: Improving neural codec language model via multi-scale speech coding and generation,”arXiv preprint arXiv:2409.11630, 2024

    2024 arXiv

  21. [29]

    TS3-Codec: Transformer-based simple streaming single codec,

    H. Wu, N. Kanda, S. E. Eskimez, and J. Li, “TS3-Codec: Transformer-based simple streaming single codec,”arXiv preprint arXiv:2411.18803, 2024

    2024 arXiv

  22. [30]

    Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming,

    Z. Xie and C. Wu, “Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming,”arXiv preprint arXiv:2408.16725, 2024

    2024 arXiv

  23. [31]

    Self-supervised speech representation learning: A review,

    A. Mohamed, H.-y. Lee, L. Borgholt, J. D. Havtorn, J. Edin, C. Igel, K. Kirchhoff, S.-W. Li, K. Livescu, L. Maaløeet al., “Self-supervised speech representation learning: A review,”IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1179–1210, 2022

    2022

  24. [32]

    Sylber: Syllabic Embedding Representation of Speech from Raw Audio,

    C. J. Cho, N. Lee, A. Guptaet al., “Sylber: Syllabic Embedding Representation of Speech from Raw Audio,” inProc. ICLR, 2025

    2025

  25. [33]

    SyllableLM: Learning Coarse Semantic Units for Speech Language Models,

    A. Baade, P. Peng, and D. Harwath, “SyllableLM: Learning Coarse Semantic Units for Speech Language Models,” inProc. ICLR, 2025

    2025

  26. [34]

    Towards accurate image coding: Improved autoregressive image generation with dynamic vector quantization,

    M. Huang, Z. Mao, Z. Chen, and Y . Zhang, “Towards accurate image coding: Improved autoregressive image generation with dynamic vector quantization,” inProc. IEEE/CVF CVPR, 2023, pp. 22 596–22 605

    2023

  27. [35]

    Spatial entropy-based global and local image contrast enhancement,

    T. Celik, “Spatial entropy-based global and local image contrast enhancement,”IEEE Transactions on Image Processing, vol. 23, no. 12, pp. 5298–5308, 2014

    2014

  28. [36]

    Codec-SUPERB: An in-depth analysis of sound codec models,

    H. Wu, H.-L. Chung, Y .-C. Lin, Y .-K. Wu, X. Chen, Y .-C. Pai, H.- H. Wang, K.-W. Chang, A. Liu, and H.-y. Lee, “Codec-SUPERB: An in-depth analysis of sound codec models,” inFindings of the Association for Computational Linguistics: ACL 2024. Asso- ciation for Computational L...

    2024

  29. [37]

    LibriTTS: A corpus derived from librispeech for text- to-speech,

    H. Zen, V . Dang, R. Clark, Y . Zhang, R. J. Weiss, Y . Jia, Z. Chen, and Y . Wu, “LibriTTS: A corpus derived from librispeech for text- to-speech,” inProc. ISCA Interspeech, 2019, pp. 1526–1530

    2019

  30. [38]

    UTMOS: UTokyo-SaruLab System for V oice- MOS Challenge 2022,

    T. Saeki, D. Xin, W. Nakata, T. Koriyama, S. Takamichi, and H. Saruwatari, “UTMOS: UTokyo-SaruLab System for V oice- MOS Challenge 2022,” inProc. ISCA Interspeech, 2022, pp. 4521–4525

    2022

  31. [39]

    V oiceCraft: Zero-shot speech editing and text-to-speech in the wild,

    P. Peng, P.-Y . Huang, S.-W. Li, A. Mohamed, and D. Harwath, “V oiceCraft: Zero-shot speech editing and text-to-speech in the wild,” inProceedings of the 62nd Annual Meeting of the Associa- tion for Computational Linguistics (V olume 1: Long Papers). As- sociation for Computat...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.