Pith. sign in

REVIEW 3 major objections 7 minor 36 references

Discrete Audio Representations for Automated Audio Captioning

T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that discrete audio tokens trained with sound-event labels can match continuous audio features in automated audio captioning, with the supervised RVQ tokenizer scoring 0.294 SPIDEr on Clotho versus 0.299 for continuous…

desk verdict Useful systematic comparison of discrete audio tokens for AAC; the supervised tokenizer's claimed advantage is within noise and needs statistical support before it carries weight. read the letter →

arxiv 2505.14989 v1 pith:NCIA6DL6 submitted 2025-05-21 cs.SD eess.AS

Jingguang Tian , Haoqin Sun , Xinhui Hu , Xinkang Xu This is my paper
classification cs.SDeess.AS
keywords automatedaudiocaptioningdiscreterepresentationstokenssupervisedtokenizervectorquantizationtaggingClothodatasetBEATs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper investigates whether discrete audio tokens can replace continuous audio features in automated audio captioning, a task where discrete representations have been little explored. It compares five tokenization methods in two captioning architectures and finds that standard semantic and acoustic tokens degrade performance relative to continuous BEATs features. To close the gap, it proposes a supervised tokenizer that splits pre-trained BEATs and inserts a vector-quantization module trained to predict AudioSet sound-event labels. The resulting tokens outperform all other tokenizers on out-of-domain data and nearly match continuous features, which would make discrete audio a viable input for captioning systems.

What carries the argument

The central object is the supervised audio tokenizer: a pre-trained BEATs model split after layer 9 into Encoder1 and Encoder2, with a RepCodec-style vector-quantization module (encoder, vector quantizer, decoder) inserted between the two parts. The BEATs parameters are frozen, the inserted module is trained with a binary cross-entropy audio tagging loss on AudioSet, and codebook embeddings are updated by exponential moving average. After training, Encoder1 plus the VQ encoder and codebook convert a waveform into discrete code vectors, with the two codebook streams summed for the RVQ variant. This machinery is what lets the model inject explicit audio-event knowledge into the discrete representation, and it is the component that the paper credits for the performance gain.

What would settle it

Train the supervised tokenizer on AudioSet with the tag labels randomly shuffled and evaluate on Clotho with the BART model; if SPIDEr stays near 0.294, the improvement is not caused by semantic supervision, while a large drop would confirm the tagging objective is doing the work.

Watch

Extended reading notes

Core claim

The paper's central claim is that a supervised audio tokenizer trained with a multi-label audio tagging objective can produce discrete tokens that match continuous audio features for automated audio captioning. On the Clotho evaluation set, the supervised RVQ tokenizer reaches SPIDEr 0.294 with the BART-based model, against 0.299 for continuous BEATs features, and 0.277 with GPT-2 XL, against 0.270 for continuous BEATs. It also beats the out-of-domain RepCodec-RVQ baseline (0.294 vs 0.292 on BART). The authors interpret this as evidence that explicit audio-event supervision compensates for the information loss inherent in discretization, while unsupervised tokenizers lack that semantic grounding.

Load-bearing premise

The method's gain rests on the untested premise that inserting a vector-quantization module at layer 9 of BEATs and training it with AudioSet tag labels is the right way to inject semantic knowledge; if another split point or architecture captured the same supervision, the claimed parity with continuous features could disappear.

Editorial extensions

If this is right

  • Semantic tokens from k-means, RepCodec, and the supervised tokenizer clearly outperform acoustic tokens from EnCodec and DAC for captioning.
  • Discretizing BEATs features still loses information: continuous BEATs features achieve the highest SPIDEr on the BART model.
  • The supervised RVQ tokenizer trained on out-of-domain AudioSet nearly matches continuous BEATs on BART and surpasses it on GPT-2 XL.
  • Adding a second codebook (RVQ over VQ) improves both RepCodec and the supervised tokenizer.
  • For unsupervised tokenizers, in-domain training data works better than out-of-domain data, whereas the supervised tokenizer does not need in-domain labels.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension would test the same tokenizer on other captioning benchmarks or other audio tagging frontends; if the gain persists, the mechanism is the tagging supervision rather than a property of BEATs.
  • A control experiment with randomly shuffled AudioSet labels would separate the effect of the tagging objective from the effect of simply training the inserted VQ module.
  • Closing the gap to continuous features suggests discrete inputs could be fed to language models directly, without a separate audio encoder, once vocabulary and codebook length are handled.
  • The paper's claim that higher split layers work better is stated without an ablation for the supervised tokenizer, so the split-point choice remains part of the hypothesis.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. This paper asks whether discrete audio representations can substitute for continuous features in automated audio captioning (AAC). The authors build two captioners, a BART-base encoder-decoder and a GPT-2 XL prefix-based decoder, and feed them either continuous features (FBANK, BEATs layer-9) or tokens from five tokenizers: two acoustic codecs (EnCodec, DAC) and three semantic tokenization routes over BEATs features (k-means, RepCodec with VQ/RVQ, and a proposed "supervised audio tokenizer"). The supervised tokenizer inserts a RepCodec-style vector-quantization module between the first nine and last three layers of frozen pre-trained BEATs and trains it with an AudioSet multi-label audio-tagging objective (binary cross-entropy); the resulting summed code vectors are used as AAC inputs. On Clotho v2, the paper reports three main findings: semantic tokens vastly outperform acoustic tokens; discretization degrades performance relative to continuous BEATs; and the proposed supervised RVQ tokenizer achieves the best results among tokenizers trained on out-of-domain data (AudioSet), nearly matching, and in the GPT-2 XL configuration exceeding, continuous BEATs features. The paper also contributes a k-means-based ablation of BEATs layer choice and cluster count (Figure 2).

Significance. If the results are taken at face value, this is a competently executed systematic comparison on a standard public benchmark: five tokenizers times two captioning backbones, with tokenizers trained on AudioSet and evaluated on Clotho, so there is no training/evaluation circularity. The large-margin findings (acoustic tokens are poor for AAC; semantic tokens retain much of the BEATs information; a second residual codebook consistently helps across all rows of Table 1) are robust and useful for practitioners. The genuinely interesting, falsifiable data point is the GPT-2 XL result, where a codebook-bottlenecked supervised representation numerically exceeds continuous BEATs (0.277 vs 0.270 SPIDEr). The main limitations are statistical: the headline superiority over conventional tokens rests on SPIDEr gaps of 0.002 to 0.009, and the causal attribution to supervision is not isolated. The contribution is incremental rather than foundational, since the tokenizer stacks known components (BEATs, a RepCodec-style VQ, and an AudioSet tagging loss), but it is a useful reference result for the discrete-audio literature.

major comments (3)
  1. [Table 1; Section 3.4] The paper's central claim, that the supervised tokenizer "achieves the best performance among tokenization methods on out-of-domain training data, nearly matching the performance of continuous audio representations," rests on single-run SPIDEr gaps of 0.002 (BART: Supervised-RVQ 0.294 vs RepCodec-RVQ 0.292) and 0.009 (GPT-2 XL: 0.277 vs 0.268), with analogous FENSE gaps of 0.003 and 0.007. No seeds, error bars, or significance tests are reported for any configuration in Table 1, yet the evaluation set contains only 1,045 Clotho clips and run-to-run SPIDEr variability for AAC systems is typically of the same order as these gaps. Because the abstract's "outperform conventional audio tokens" and Section 3.4's "best performance" claims are exactly these comparisons, the authors should report mean and standard deviation over at least three seeds, or a paired significance test, for the critical cells of Table 1; if the deltas do not survive that test, the headline must be re-scoped to a qualitative finding.
  2. [Abstract; Section 3.4; Section 4] The unqualified claims in the abstract and conclusion ("the proposed audio tokens outperform conventional audio tokens"; "surpasses existing methods") are not supported once in-domain baselines are included in the comparison. Supervised-RVQ ties RepCodec-RVQ-ID on BART SPIDEr (0.294 vs 0.294) and is numerically worse in three of the other four metric-by-architecture comparisons (BART FENSE 0.490 vs 0.496; GPT-2 XL SPIDEr 0.277 vs 0.279; GPT-2 XL FENSE 0.484 vs 0.485). The only claim Table 1 supports is that Supervised-RVQ is best among the out-of-domain tokenizers. The abstract and conclusion should be re-scoped accordingly, and Section 3.4 should state the in-domain versus out-of-domain asymmetry explicitly instead of leaving it implicit in the table.
  3. [Section 2.2.4; Section 3.5; Figure 2] The causal claim that the audio tagging objective is what improves the discrete representations is not isolated. Section 2.2.4 justifies the layer-9 split with "Empirical evidence shows that selecting higher layers for splitting audio tagging models yields better results," but the only layer ablation in the paper (Figure 2) is for k-means clustering of BEATs features, not for the supervised tokenizer, so the split point of the proposed method is untested. In addition, Supervised-RVQ differs from the RepCodec-RVQ baseline in the training loss (tagging BCE vs feature reconstruction), in the input type (waveform through a frozen BEATs split vs BEATs features), and in the codebook update scheme (EMA, fully frozen backbone), and the VQ module is described only as "mirrors that of RepCodec" with no parameter counts. A control experiment training the same split-plus-VQ architecture with a reconstruction loss instead of the tagging loss, together with an ablation varying the supervised tokenizer's split layer, is needed before the gain can be attributed to supervision; without it, the conclusion in Section 3.4 that supervised learning "enables the tokenizer to capture knowledge of audio events, thereby mitigating information loss during discretization" is a hypothesis rather than a demonstrated result.
minor comments (7)
  1. [Table 1] In the typeset Table 1 the ENCLAP SPIDEr value (0.294, marked with an asterisk as sourced from [25]) appears as an orphaned cell above the K-means row, and the three-way column grouping (ID/OD across SPIDEr, FENSE, and #Words) is visually ambiguous; please reformat so that each row's values align clearly with their metric and domain columns.
  2. [Section 3.2.1; Abstract] The paper discloses that the models consume summed code vectors rather than quantized code indices, but the Abstract and Introduction repeatedly invoke "discrete audio representations" and "tokens"; please state explicitly that discreteness enters through the codebook bottleneck only, since the captioning models themselves receive continuous vectors.
  3. [Section 3.4] The "nearly matching" characterization is architecture-dependent: on BART the supervised tokenizer trails continuous BEATs by 0.005 SPIDEr, while on GPT-2 XL it leads by 0.007; the text should report this reversal explicitly, because the abstract's statement that "audio tokenization leads to performance degradation" holds only for the BART system and is contradicted on GPT-2 XL SPIDEr.
  4. [Section 3.4] The conjecture that the GPT-2 XL model "is more prone to overfitting in low-resource settings" is presented without supporting evidence such as training or validation curves; please either add supporting analysis or mark the sentence explicitly as a hypothesis.
  5. [Section 3.5] The conclusion that "increasing the number of codebooks in the quantizer enhances performance" is tested with exactly one versus two codebooks (Table 1, VQ vs RVQ); please rephrase to "using a second residual codebook improves performance" to avoid implying a tested trend.
  6. [Section 3.2.1] Several configuration details needed for reproducibility are missing: the embedding dimension of the code vectors (the codebook size of 1,024 is given, but not the per-code vector dimension), the EMA decay and any commitment-loss treatment for the supervised tokenizer's codebook updates, and the encoder/decoder layer configuration of the VQ module; a configuration table would resolve this.
  7. [Section 3.4] The statement that the BART discrete model "performs comparably to the ENCLAP model" (SPIDEr 0.294 vs 0.294) compares across different backbones, decoders, and training recipes; since the ENCLAP value is imported from [25] (as flagged in the table), please add an explicit caveat that this is an informal reference point rather than a controlled comparison.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the supervised tokenizer is trained on AudioSet tags and evaluated on Clotho captions, an external benchmark.

full rationale

The paper's central derivation is not circular. The proposed supervised audio tokenizer is trained on the AudioSet dataset with a multi-label audio tagging binary cross-entropy loss (Eq. 1), while the AAC models are trained on the Clotho development set (Sections 3.1 and 3.2) and evaluated on the Clotho evaluation set using SPIDEr and FENSE (Table 1). No AAC caption metric or Clotho label is used as a training signal for the tokenizer, so the central comparison (supervised tokens versus unsupervised tokens versus continuous BEATs features) is an external, out-of-domain evaluation rather than a restatement of a fitted parameter. Hyperparameters such as codebook size, codebook count, and prefix length are tuned on the held-out validation set, which is standard practice and does not constitute fitting the evaluation metric. The reference [27] may share an author with the present paper and motivates transfer of audio-tagging knowledge, but that citation is used only as background motivation; the actual tokenizer is implemented and evaluated here, and the conclusion does not depend on an unverified theorem or a self-citation chain. The unsupported assertion in Section 2.2.4 that higher split layers perform better, and the lack of significance testing on the small SPIDEr gaps, are correctness or robustness concerns, not circularity, because the results are measured against an independent benchmark. Therefore no circular step can be exhibited and the score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on standard domain assumptions: that BEATs features are a good continuous baseline, that AudioSet tags carry the semantics needed for captioning, and that Clotho metrics reflect quality. The main free parameters (cluster count, layer split, codebook count, prefix length) are tuned choices that affect the reported numbers. No new entities are postulated.

free parameters (4)
  • Number of k-means clusters (K) = 1024
    Selected for k-means and used as codebook size for VQ/RVQ; Fig. 2 shows performance varies with K, so the choice matters for the reported scores.
  • Layer split point in BEATs = 9 (Encoder1) / 3 (Encoder2)
    The split determines what the VQ module sees; the paper says empirical evidence favors higher layers but shows no experiment for the supervised tokenizer (Sec 2.2.4).
  • Number of RVQ codebooks = 2
    RVQ with two codebooks outperforms VQ in their experiments (Table 1), and the paper notes more codebooks help but does not test beyond two.
  • Prefix length for GPT-2 XL = 50
    Fixed number of learnable embeddings used as prefix; k=50 is stated in Sec 2.3.2 without justification.
assumptions (4)
  • domain assumption BEATs layer 9 output is a strong continuous representation for AAC
    Used as both the continuous baseline and the input to every tokenizer (Sec 2.2.1); if this representation were weak, the whole comparison would shift.
  • domain assumption Audio tagging supervision transfers semantic knowledge to captioning
    The core hypothesis of Sec 2.2.4; the paper assumes BCE on AudioSet tags teaches the tokenizer the event information that captions need, but it is not independently verified beyond the final AAC score.
  • domain assumption SPIDEr and FENSE are valid evaluation metrics for AAC
    The paper adopts these metrics without questioning whether they capture caption quality; higher values are taken at face value as better performance (Sec 3.3).
  • domain assumption Clotho v2 evaluation set size is sufficient to rank methods
    All conclusions rest on a single evaluation set of 1,045 clips with no reported variance (Sec 3.1); if the set is noisy, small differences between tokenizers are not reliable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Discrete Audio Representations for Automated Audio Captioning." pith.science (2026). https://pith.science/paper/NCIA6DL6

@misc{pith2026250514989,
  author       = {Pith},
  title        = {Pith review of: Discrete Audio Representations for Automated Audio Captioning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NCIA6DL6}},
  note         = {Machine review of arXiv:2505.14989}
}
read the original abstract

Discrete audio representations, termed audio tokens, are broadly categorized into semantic and acoustic tokens, typically generated through unsupervised tokenization of continuous audio representations. However, their applicability to automated audio captioning (AAC) remains underexplored. This paper systematically investigates the viability of audio token-driven models for AAC through comparative analyses of various tokenization methods. Our findings reveal that audio tokenization leads to performance degradation in AAC models compared to those that directly utilize continuous audio representations. To address this issue, we introduce a supervised audio tokenizer trained with an audio tagging objective. Unlike unsupervised tokenizers, which lack explicit semantic understanding, the proposed tokenizer effectively captures audio event information. Experiments conducted on the Clotho dataset demonstrate that the proposed audio tokens outperform conventional audio tokens in the AAC task.

Figures

Figures reproduced from arXiv: 2505.14989 by the authors.

Figure 1
Figure 1. Schematic diagram of the proposed automated audio captioning models utilizing audio tokens. 2. Method This section describes the AAC models that utilize audio to￾kens, as illustrated in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Performance of k-means applied to the BART-based AAC model across different layers and cluster numbers. 3.5. Ablation studies Employing k-means clustering as a case study, we investigate how layer selection within the BEATs model and codebook size (number of clusters) affect semantic token performance, as de￾picted in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

36 extracted references · 19 canonical work pages

  1. [1]

    Introduction Automated audio captioning (AAC) is a cross-modal translation task aimed at generating natural language descriptions for audio clips [1]. The primary approach employs an encoder-decoder- based deep learning framework, where the audio encoder ex- tracts acoustic features, and the text decoder generates captions based on these features [2, 3, 4...

    work page Pith review arXiv 2025

  2. [2]

    We construct BART-based and GPT-2 XL-based AAC systems, both utilizing semantic and acoustic tokens

    Method This section describes the AAC models that utilize audio to- kens, as illustrated in Figure 1. We construct BART-based and GPT-2 XL-based AAC systems, both utilizing semantic and acoustic tokens. 2.1. Acoustic tokens 2.1.1. EnCodec EnCodec is an audio codec based on a convolutional encoder- decoder architecture trained in an end-to-end manner. We e...

    work page

  3. [3]

    Experiment This section provides a detailed description of the datasets, ex- perimental setup, and evaluation metrics used in our study. 3.1. Datasets AudioSet [29] is an audio dataset comprising 527 sound classes and approximately 2 million 10-second audio clips. The dataset is divided into three subsets: an unbalanced subset containing 2,042,985 clips, ...

    work page

  4. [4]

    Our findings indicate that semantic tokens significantly outperform acoustic tokens in this context

    Conclusion This paper presents the first systematic study of discrete audio representations for AAC. Our findings indicate that semantic tokens significantly outperform acoustic tokens in this context. Moreover, the discretization of continuous audio representa- tions leads to information loss, diminishing AAC performance. To mitigate this, we develop a s...

    work page

  5. [5]

    Automated audio captioning: An overview of recent progress and new challenges,

    X. Mei, X. Liu, M. D. Plumbley, and W. Wang, “Automated audio captioning: An overview of recent progress and new challenges,” EURASIP journal on audio, speech, and music processing , vol. 2022, no. 1, p. 26, 2022

    work page 2022

  6. [6]

    Audio captioning based on transformer and pre-trained cnn,

    K. Chen, Y . Wu, Z. Wang, X. Zhang, F. Nian, S. Li, and X. Shao, “Audio captioning based on transformer and pre-trained cnn,” in Proceedings of the Detection and Classification of Acoustic Scenes and Events 2020 Workshop (DCASE2020), 2020, pp. 21– 25

    work page 2020

  7. [7]

    Automated audio caption- ing by fine-tuning bart with audioset tags,

    F. Gontier, R. Serizel, and C. Cerisara, “Automated audio caption- ing by fine-tuning bart with audioset tags,” in DCASE 2021-6th Workshop on Detection and Classification of Acoustic Scenes and Events, 2021

    work page 2021

  8. [8]

    Leveraging pre-trained bert for audio captioning,

    X. Liu, X. Mei, Q. Huang, J. Sun, J. Zhao, H. Liu, M. D. Plumb- ley, V . Kilic, and W. Wang, “Leveraging pre-trained bert for audio captioning,” in 2022 30th European Signal Processing Confer- ence (EUSIPCO). IEEE, 2022, pp. 1145–1149

    work page 2022

Show all 36 references
  1. [9]

    Conette: An efficient au- dio captioning system leveraging multiple datasets with task em- bedding,

    E. Labb, T. Pellegrini, J. Pinquier et al., “Conette: An efficient au- dio captioning system leveraging multiple datasets with task em- bedding,” IEEE/ACM Transactions on Audio, Speech, and Lan- guage Processing, 2024

    2024

  2. [10]

    Improving audio captioning mod- els with fine-grained audio features, text embedding supervision, and llm mix-up augmentation,

    S.-L. Wu, X. Chang, G. Wichern, J.-w. Jung, F. Germain, J. Le Roux, and S. Watanabe, “Improving audio captioning mod- els with fine-grained audio features, text embedding supervision, and llm mix-up augmentation,” inICASSP 2024-2024 IEEE Inter- national Conference on Acoustics...

    2024

  3. [11]

    Panns: Large-scale pretrained audio neural networks for audio pattern recognition,

    Q. Kong, Y . Cao, T. Iqbal, Y . Wang, W. Wang, and M. D. Plumb- ley, “Panns: Large-scale pretrained audio neural networks for audio pattern recognition,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 28, pp. 2880–2894, 2020

    2020

  4. [12]

    Adapting a convnext model to audio classification on au- dioset,

    T. Pellegrini, I. Khalfaoui-Hassani, E. Labb ´e, and T. Masque- lier, “Adapting a convnext model to audio classification on au- dioset,” in 24th INTERSPEECH Conference (INTERSPEECH 2023). ISCA, 2023, pp. 4169–4173

    2023

  5. [13]

    Beats: audio pre-training with acoustic tok- enizers,

    S. Chen, Y . Wu, C. Wang, S. Liu, D. Tompkins, Z. Chen, W. Che, X. Yu, and F. Wei, “Beats: audio pre-training with acoustic tok- enizers,” in Proceedings of the 40th International Conference on Machine Learning, 2023, pp. 5178–5193

    2023

  6. [14]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805 , 2018

    2018 arXiv

  7. [15]

    Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,

    M. Lewis, “Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,” arXiv preprint arXiv:1910.13461, 2019

    1910 arXiv

  8. [16]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al. , “Language models are unsupervised multitask learners,” OpenAI blog, vol. 1, no. 8, p. 9, 2019

    2019

  9. [17]

    Stab: Speech tokenizer assessment benchmark,

    S. Vashishth, H. Singh, S. Bharadwaj, S. Ganapathy, C. Asawaro- engchai, K. Audhkhasi, A. Rosenberg, A. Bapna, and B. Ramab- hadran, “Stab: Speech tokenizer assessment benchmark,” arXiv preprint arXiv:2409.02384, 2024

    2024 arXiv

  10. [18]

    Soundstream: An end-to-end neural audio codec,

    N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, “Soundstream: An end-to-end neural audio codec,” IEEE/ACM Transactions on Audio, Speech, and Lan- guage Processing, vol. 30, pp. 495–507, 2021

    2021

  11. [19]

    High fidelity neural audio compression,

    A. D ´efossez, J. Copet, G. Synnaeve, and Y . Adi, “High fidelity neural audio compression,” arXiv preprint arXiv:2210.13438 , 2022

    2022 arXiv

  12. [20]

    High-fidelity audio compression with improved rvqgan,

    R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar, “High-fidelity audio compression with improved rvqgan,” Ad- vances in Neural Information Processing Systems, vol. 36, 2024

    2024

  13. [21]

    Hubert: Self-supervised speech represen- tation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdi- nov, and A. Mohamed, “Hubert: Self-supervised speech represen- tation learning by masked prediction of hidden units,”IEEE/ACM transactions on audio, speech, and language processing, vol. 29, pp. 3451–3460, 2021

    2021

  14. [22]

    Wavlm: Large-scale self- supervised pre-training for full stack speech processing,

    S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiaoet al., “Wavlm: Large-scale self- supervised pre-training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing , vol. 16, no. 6, pp. 1505–1518, 2022

    2022

  15. [23]

    RepCodec: A speech represen- tation codec for speech tokenization,

    Z. Huang, C. Meng, and T. Ko, “RepCodec: A speech represen- tation codec for speech tokenization,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Bangkok, Thailand: Association for Computational Linguistics, ...

    2024

  16. [24]

    Neural codec language mod- els are zero-shot text to speech synthesizers,

    C. Wang, S. Chen, Y . Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y . Liu, H. Wang, J. Li et al. , “Neural codec language mod- els are zero-shot text to speech synthesizers,” arXiv preprint arXiv:2301.02111, 2023

    2023 arXiv

  17. [25]

    Base tts: Lessons from building a billion-parameter text-to-speech model on 100k hours of data,

    M. Łajszczak, G. C ´ambara, Y . Li, F. Beyhan, A. van Korlaar, F. Yang, A. Joly, ´A. Mart ´ın-Cortinas, A. Abbas, A. Michal- ski et al., “Base tts: Lessons from building a billion-parameter text-to-speech model on 100k hours of data,” arXiv preprint arXiv:2402.08093, 2024

    2024 arXiv

  18. [26]

    Exploration of efficient end-to-end asr using discretized input from self-supervised learning,

    X. Chang, B. Yan, Y . Fujita, T. Maekaku, and S. Watanabe, “Exploration of efficient end-to-end asr using discretized input from self-supervised learning,” in INTERSPEECH 2023 , 2023, pp. 1399–1403

    2023

  19. [27]

    How should we extract discrete audio tokens from self-supervised models?

    P. Mousavi, J. Duret, S. Zaiem, L. Della Libera, A. Ploujnikov, C. Subakan, and M. Ravanelli, “How should we extract discrete audio tokens from self-supervised models?” in Interspeech 2024, 2024, pp. 2554–2558

    2024

  20. [28]

    Discrete audio representation as an alternative to mel-spectrograms for speaker and speech recognition,

    K. C. Puvvada, N. R. Koluguri, K. Dhawan, J. Balam, and B. Ginsburg, “Discrete audio representation as an alternative to mel-spectrograms for speaker and speech recognition,” inICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) ....

    2024

  21. [29]

    Enclap: Combining neu- ral audio codec and audio-text joint embedding for automated au- dio captioning,

    J. Kim, J. Jung, J. Lee, and S. H. Woo, “Enclap: Combining neu- ral audio codec and audio-text joint embedding for automated au- dio captioning,” in ICASSP 2024-2024 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2024, pp. 6735–6739

    2024

  22. [30]

    Neural discrete represen- tation learning,

    A. Van Den Oord, O. Vinyals et al., “Neural discrete represen- tation learning,” Advances in neural information processing sys- tems, vol. 30, 2017

    2017

  23. [31]

    Investigating lo- cal and global information for automated audio captioning with transfer learning,

    X. Xu, H. Dinkel, M. Wu, Z. Xie, and K. Yu, “Investigating lo- cal and global information for automated audio captioning with transfer learning,” in ICASSP 2021-2021 IEEE international con- ference on acoustics, speech and signal processing (ICASSP) . IEEE, 2021, pp. 905–909

    2021

  24. [32]

    Prefix tuning for auto- mated audio captioning,

    M. Kim, K. Sung-Bin, and T.-H. Oh, “Prefix tuning for auto- mated audio captioning,” in ICASSP 2023-2023 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5

    2023

  25. [33]

    Audio set: An ontology and human-labeled dataset for audio events,

    J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in 2017 IEEE inter- national conference on acoustics, speech and signal processing (ICASSP). IEEE, 2017,...

    2017

  26. [34]

    Clotho: An audio cap- tioning dataset,

    K. Drossos, S. Lipping, and T. Virtanen, “Clotho: An audio cap- tioning dataset,” in ICASSP 2020-2020 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2020, pp. 736–740

    2020

  27. [35]

    Improved image captioning via policy gradient optimization of spider,

    S. Liu, Z. Zhu, N. Ye, S. Guadarrama, and K. Murphy, “Improved image captioning via policy gradient optimization of spider,” in Proceedings of the IEEE international conference on computer vision, 2017, pp. 873–881

    2017

  28. [36]

    Can audio captions be evaluated with image caption metrics?

    Z. Zhou, Z. Zhang, X. Xu, Z. Xie, M. Wu, and K. Q. Zhu, “Can audio captions be evaluated with image caption metrics?” in ICASSP 2022-2022 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP) . IEEE, 2022, pp. 981–985

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.