Pith. sign in

REVIEW 4 major objections 7 minor 1 cited by

SongEditor: Adapting Zero-Shot Song Generation Language Model as a Multi-Task Editor

T0 review · 4 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SongEditor claims to be the first song-editing paradigm built into a language-model song generator, letting users rewrite a chosen segment or an entire vocal or accompaniment track while keeping the rest coherent.

desk verdict Reasonable framework paper for LM-based song editing; segment-wise effects hold up, but track-wise independence is not demonstrated because separation leakage is never measured. read the letter →

arxiv 2412.13786 v2 pith:4KNNGIWX submitted 2024-12-18 eess.AS cs.SD

Chenyu Yang , Shuai Wang , Hangting Chen , Jianwei Yu , Wei Tan , Rongzhi Gu , Yaoxun Xu , Yizhi Zhou
show 2 more authors
Haina Zhu Haizhou Li
This is my paper · ORCID
classification eess.AScs.SD
keywords songeditinglanguagemodelsemantictokenssourceseparationmusicinfillingzero-shotgenerationtrack-wisesegment-wise
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SongEditor claims to be the first song-editing system built on a language-model song generator. It can rewrite a chosen span of an existing song (segment-wise editing) or regenerate an entire vocal or accompaniment track from the other track (track-wise editing), while also generating full songs from scratch. The value would be in targeted studio fixes: correcting one bad chorus or swapping a vocal without regenerating the whole piece. The authors support the claim with objective metrics and listening tests, and with a design that keeps generation quality comparable to the base generator.

What carries the argument

The key mechanism is token-sequence rearrangement with a delayed pattern. The segment being edited is removed from the middle of the token sequence and appended after a separator token, making infilling a continuation problem; the delayed pattern stacks multiple residual-quantizer codebooks so the model predicts all codebooks coherently. Force-smoothing training copies a few frames from the following context as extra prediction targets, and a score-based candidate selection chooses among regenerated endings by the likelihood with which they lead into the real following audio. Track-wise editing adds a gated multi-source encoder that conditions the decoder via cross-attention, with 40dB white noise added to the separated-vocal condition during training to mask residual leakage from the separator.

What would settle it

Feed a song through SongEditor's track-wise mode, then run a standard music source-separation model on the output and compare the leaked residual to the 40dB white-noise floor. If the regenerated 'vocal-only' track contains recognizable accompaniment content, or if the separated output's signal-to-distortion ratio is substantially worse than the training-time masking level, the claimed independence of track-wise editing fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that editing can be made an intrinsic capability of an autoregressive song language model, not a separate task added after generation. The editing segment is cut out of the semantic-token sequence and moved to the end, so the model learns to fill the gap by continuing from the surrounding audio context; only the target lyrics are supplied for the edited region, which the paper calls a context-free strategy. For track-wise editing, vocals and accompaniment are separated with BS-RNN, the given track is tokenized and injected through a multi-source encoder with cross-attention, and the model completes the missing track. The paper reports that this single framework handles generation, segment-wise editing, track-wise editing, and iterative long-song story-mode generation, and that the removed lyric context reduces word errors at edit boundaries.

Load-bearing premise

The weakest load-bearing premise is that the BS-RNN source separator leaves no significant cross-track leakage, because the track-wise editor is trained on vocals and accompaniments separated by this model, and only a fixed 40dB white-noise masking is used to hide leakage.

Editorial extensions

If this is right

  • A targeted segment of a song can be regenerated with new lyrics while the rest of the song stays untouched, reducing the cost of fixing one bad verse or chorus.
  • Full vocal or accompaniment tracks can be resynthesized from the other track, enabling vocal swaps or instrumental re-records without retraining the model.
  • Segment-wise and track-wise editing can be combined, so a short portion of the vocal can be edited independently of the accompaniment.
  • Because the lyric context is dropped, the model can generate songs longer than its training length by iterating over sections without manual prefix annotation (Story Mode).
  • The same system also synthesizes complete songs from lyrics and a 10-second prompt, so generation and editing share one architecture.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the tokenizer and diffusion generator are not intrinsically tied to songs, the same rearrangement-plus-source-conditioning recipe should transfer to long-form speech editing, where context preservation and speaker consistency are the same problems.
  • The fixed 40dB white-noise masking implies that track-wise editing quality is bounded by source-separation quality, so any future improvement in separation should carry over to editing without retraining the language model.
  • A testable extension would be to evaluate track-wise edits with an objective separation metric on the output, such as signal-to-distortion ratio against the original separated tracks, which would expose whether the generated track contains leakage from the condition track.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes SongEditor, an extension of a zero-shot song generation language model (SongLM) that adds segment-wise editing (regenerating a specific lyric sentence or section) and track-wise editing (regenerating the vocal or accompaniment track given the other). The method rearranges semantic token sequences in the style of VoiceCraft, introduces a lyric context-free editing strategy, a force-smoothing training objective, a score-based candidate selection step at inference, and a multi-source encoder plus BS-RNN source separation for track-wise editing. The authors train on roughly 700K songs (50K hours) and evaluate with PER, FAD, and MOS listening tests, reporting improvements over a SongLM baseline and over VoiceCraft on several metrics. The paper claims to be the first song editing paradigm that introduces editing capabilities into language-modeling song generation approaches.

Significance. If the evaluation were fully convincing, this would be a useful contribution to the relatively underexplored area of song editing, and the combination of context-free editing, force-smoothing, and candidate selection is a sensible and well-motivated design. The authors also provide a demo page with audio samples. However, the current empirical support is not sufficient for the strong claims in the abstract: the objective results are mixed, the subjective evaluation is small and lacks statistical grounding, and the track-wise editing capability depends on an unmeasured source-separation quality. The central architecture and training ideas are plausible and worth publishing after the load-bearing evaluation gaps are addressed.

major comments (4)
  1. [Multi-Source Track-Wise Editing, 'Source Separation' paragraph] Track-wise editing is trained and evaluated entirely on BS-RNN-separated stems, and the only explicit leakage safeguard is adding white noise with sigma=0.01 (approximately 40 dB SNR). White noise does not model structured leakage such as residual vocals in the accompaniment stem or drum/pad bleed in the vocal stem, which is correlated with the target track. No separation quality metric (e.g., SDR, SI-SNR) is reported for BS-RNN on the test set. Because the same separator is used in both training and evaluation, Table 3 cannot rule out that the model is partly reconstructing the target from leaked cues rather than generating it independently from the conditioning track. Please report separation metrics on the evaluation set and, if possible, include an analysis with oracle stems or with separators of varying quality to demonstrate that the editing is robust to separation error.
  2. [Results and Analysis, 'Segment-Wise Editing' (Table 2)] The smoothness MOS of SongEditor (3.53) is numerically below VoiceCraft (3.63), and the force-smoothing plus candidate-selection variant (3.43) is lower still. This contradicts the paper's assertion that force-smoothing and candidate selection improve transition naturalness. The text attributes the decline to 'potential inaccuracies in the annotation of temporal boundaries,' but no evidence is provided for that explanation. Please either present a controlled analysis of boundary-alignment errors or revise the claim that the proposed smoothing techniques improve smoothness.
  3. [Evaluation Metrics, subjective evaluation] The subjective evaluation uses only 15 samples and 30 listeners, with no confidence intervals, inter-rater agreement, or significance tests reported. Several differences that are used to support the main claims are small (e.g., quality 3.39 vs. 3.17, smoothness 3.53 vs. 3.63), so the MOS results do not support the abstract's 'exceptional performance' claim. Please provide per-item confidence intervals or significance tests, and temper the wording of the claims to match the statistical strength of the evidence.
  4. [Results and Analysis, 'Song Generation' (Table 1)] The text states that incorporating segment-wise editing capability 'does not degrade the performance' of SongLM, but Table 1 shows FAD increases from 1.99 (SongLM) to 2.24 (SongEditor). This is an objective degradation on an automatically computed metric. Please correct the statement, report statistical significance, or offer a concrete explanation for why this FAD increase is acceptable.
minor comments (7)
  1. [Base Model: SongLM, Eq. (2)] The summation in Eq. (2) is typeset incorrectly as 'KX'; it should be a sum over k=1,...,K.
  2. [Experimental Settings, 'Subjective evaluation'] The text says 'which are then assessed by expert musicians' but the sentence structure is unclear; also 'expert musicians' should be 'expert musicians'. More importantly, the allocation of the 30 listeners across the 15 samples is not described (e.g., how many ratings per sample).
  3. [Results and Analysis, Table 3] The rows of Table 3 use '-' in the V and A columns, but the reader must infer which rows correspond to 'neither source', 'vocal only', 'accompaniment only', and 'both'. Please add explicit row labels or a legend.
  4. [SongEditor, 'Score-Based Candidate Selection'] The number of candidates N is not specified in the main text. Please state the value used in the experiments.
  5. [Appendix B, 'Configuration Details'] The sentence 'Each model contains approximately 800M parameters' is ambiguous because SongLM/SongEditor have 16 decoder layers while SongEditor+ has 12 layers plus a multi-source encoder; the parameter counts must differ. Please report parameter counts separately.
  6. [Figure 4 caption] The caption reads 'the below corresponds to the vocal'; this should be 'the lower one corresponds to the vocal' for grammatical correctness.
  7. [Introduction, 'Related Work'] The related work cites MusicMagus and VampNet for music editing, but the paper's claim of being the first song editing paradigm for language-modeling song generation could be better positioned by explicitly discussing why these prior editing methods do not address song-level vocals-plus-accompaniment editing in a language-modeling setup.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: held-out evaluation and independent component citations support the editing claims; BS-RNN overlap is tooling, not a reduced derivation.

full rationale

The paper contains no circular step under the definitions used here. The central claims—segment-wise editing, track-wise editing, and long-form generation—are supported by held-out objective (PER, FAD) and subjective (MOS) evaluations on 200 test songs and 15 expert-rated samples, so the results are not encoded in the training objective by construction. Equations (1)-(6) define the tokenizer, autoregressive loss, candidate-selection score, and source-conditioning embeddings; none of these reduces a predicted quantity to a fitted input. The FAD metric uses MERT-95M while the tokenizer uses MERT-330M/HuBERT, so the evaluation embedding is not the same model whose features are being predicted, and no optimization is performed against FAD. BS-RNN is used as a fixed, externally published source-separation tool in both data preparation and PER extraction; although the authors of the present paper overlap with BS-RNN's authors, the separator is an independent, benchmarked component rather than an unverified self-citation, and the white-noise augmentation (σ=0.01) explicitly targets leakage. The potential concern that residual structured leakage in BS-RNN outputs could let the model exploit cross-track cues is an evaluation-validity limitation, acknowledged in the paper's own limitation sentence ('there is no explicit decoupling of tracks at the semantic level'), but it is not a circularity: the generated track is not, by any equation, forced to equal the separator's estimate. The AutoPrep preprocessing citation is likewise incidental tooling. No uniqueness theorem, ansatz, or renamed known result is imported from the authors' prior work to force the conclusion.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim is an empirical system capability, so the ledger lists hand-chosen hyperparameters that affect the method and the main domain assumptions about pretrained encoders, source separation, and the diffusion generator. No new physical or conceptual entities are introduced.

free parameters (6)
  • noise_sigma = 0.01
    White-noise standard deviation added to separated vocals during track-wise training to mask source-separation leakage; chosen by hand, not tuned.
  • force_smoothing_span_lambda = not specified
    Number of extra frames the model is forced to predict during force-smoothing training; value is not reported, making the method unreproducible.
  • candidate_count_N = not specified
    Number of candidates generated for score-based selection; not reported.
  • overlap_seconds = 1
    Overlap duration for context extraction during inference to avoid edge effects.
  • top_k_sampling = 250
    Top-k value used in autoregressive decoding during inference.
  • classifier_free_guidance = 1.5
    Guidance coefficient for classifier-free diffusion sampling.
assumptions (5)
  • domain assumption Pretrained MERT and HuBERT encoders provide sufficient semantic representations of music and vocals.
    The semantic tokenizer relies on these frozen encoders; the paper does not justify the choice beyond citing prior use.
  • domain assumption The diffusion generator can reconstruct high-quality audio from discrete semantic tokens.
    The final waveform is generated by an LDM with DiT and HifiGAN; this is a standard but unproved assumption in this framework.
  • domain assumption BS-RNN source separation yields sufficiently clean vocal and accompaniment tracks for training.
    Track-wise editing uses separated tracks as ground truth; separation errors propagate to the model.
  • ad hoc to paper Lyric sentences are sufficiently independent for a context-free editing strategy.
    The context-free strategy discards surrounding lyrics during editing; the authors gave two observational reasons but no controlled test.
  • domain assumption The structure detector (Kim and Nam 2023) reliably identifies song sections.
    Structure tokens are inserted into the lyric sequence based on this detector; errors would misguide generation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SongEditor: Adapting Zero-Shot Song Generation Language Model as a Multi-Task Editor." pith.science (2026). https://pith.science/paper/4KNNGIWX

@misc{pith2026241213786,
  author       = {Pith},
  title        = {Pith review of: SongEditor: Adapting Zero-Shot Song Generation Language Model as a Multi-Task Editor},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4KNNGIWX}},
  note         = {Machine review of arXiv:2412.13786}
}
read the original abstract

The emergence of novel generative modeling paradigms, particularly audio language models, has significantly advanced the field of song generation. Although state-of-the-art models are capable of synthesizing both vocals and accompaniment tracks up to several minutes long concurrently, research about partial adjustments or editing of existing songs is still underexplored, which allows for more flexible and effective production. In this paper, we present SongEditor, the first song editing paradigm that introduces the editing capabilities into language-modeling song generation approaches, facilitating both segment-wise and track-wise modifications. SongEditor offers the flexibility to adjust lyrics, vocals, and accompaniments, as well as synthesizing songs from scratch. The core components of SongEditor include a music tokenizer, an autoregressive language model, and a diffusion generator, enabling generating an entire section, masked lyrics, or even separated vocals and background music. Extensive experiments demonstrate that the proposed SongEditor achieves exceptional performance in end-to-end song editing, as evidenced by both objective and subjective metrics. Audio samples are available in https://cypress-yang.github.io/SongEditor_demo/.

Figures

Figures reproduced from arXiv: 2412.13786 by the authors.

Figure 1
Figure 1. SongEditor supports various song generation [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The architecture of the proposed SongEditor framework. We train the DiT and RVQ jointly first and then the semantic [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Mel spectrograms of transitions. The center of the [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: An example of track-wise editing. The Mel spectrogram above corresponds to the separated accompaniment, while [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Text token representation for the structure-guided [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MusiChat: Vibe Composing for Music Creation

    cs.AI 2026-07 conditional novelty 5.0 of 10

    MusiChat enables iterative, structure-preserving music editing through natural-language conversation by layering an LLM-based interface over a deterministic symbolic music engine.

Reference graph

Works this paper leans on

63 extracted references · 34 canonical work pages · cited by 1 Pith paper

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    I.; Borsos, Z.; Engel, J.; Verzetti, M.; Caillon, A.; Huang, Q.; Jansen, A.; Roberts, A.; Tagliasacchi, M.; et al

    Agostinelli, A.; Denk, T. I.; Borsos, Z.; Engel, J.; Verzetti, M.; Caillon, A.; Huang, Q.; Jansen, A.; Roberts, A.; Tagliasacchi, M.; et al. 2023. Musiclm: Generating music from text. arXiv preprint arXiv:2301.11325

    arXiv 2023

  4. [4]

    Bain, M.; Huh, J.; Han, T.; and Zisserman, A. 2023. WhisperX: Time-Accurate Speech Transcription of Long-Form Audio. INTERSPEECH

    work page 2023

  5. [5]

    Borsos, Z.; Marinier, R.; Vincent, D.; Kharitonov, E.; Pietquin, O.; Sharifi, M.; Roblek, D.; Teboul, O.; Grangier, D.; Tagliasacchi, M.; et al. 2023. Audiolm: a language modeling approach to audio generation. IEEE/ACM transactions on audio, speech, and language processing, 31: 2523--2533

    2023

  6. [6]

    Bredin, H. 2023. pyannote.audio 2.1 speaker diarization pipeline: principle, benchmark, and recipe . In Interspeech, 1983--1987

    work page 2023

  7. [7]

    Chen, K.; Wu, Y.; Liu, H.; Nezhurina, M.; Berg-Kirkpatrick, T.; and Dubnov, S. 2024 a . Musicldm: Enhancing novelty in text-to-music generation using beat-synchronous mixup strategies. In ICASSP, 1206--1210

    work page 2024

  8. [8]

    Chen, Y.; Niu, Z.; Ma, Z.; Deng, K.; Wang, C.; Zhao, J.; Yu, K.; and Chen, X. 2024 b . F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching. arXiv preprint arXiv:2410.06885

    arXiv 2024

Show all 63 references
  1. [9]

    W.; Hou, L.; Longpre, S.; Zoph, B.; Tay, Y.; Fedus, W.; Li, Y.; Wang, X.; Dehghani, M.; Brahma, S.; et al

    Chung, H. W.; Hou, L.; Longpre, S.; Zoph, B.; Tay, Y.; Fedus, W.; Li, Y.; Wang, X.; Dehghani, M.; Brahma, S.; et al. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70): 1--53

    2024

  2. [10]

    Copet, J.; Kreuk, F.; Gat, I.; Remez, T.; Kant, D.; Synnaeve, G.; Adi, Y.; and Defossez, A. 2023. Simple and Controllable Music Generation. In NeurIPS, volume 36, 47704--47720

    2023

  3. [11]

    D \'e fossez, A.; Copet, J.; Synnaeve, G.; and Adi, Y. 2022. High fidelity neural audio compression. arXiv preprint arXiv:2210.13438

    2022 arXiv

  4. [12]

    Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

    2018 arXiv

  5. [13]

    W.; Radford, A.; and Sutskever, I

    Dhariwal, P.; Jun, H.; Payne, C.; Kim, J. W.; Radford, A.; and Sutskever, I. 2020. Jukebox: A generative model for music. arXiv preprint arXiv:2005.00341

    2020 arXiv

  6. [14]

    Ding, S.; Liu, Z.; Dong, X.; Zhang, P.; Qian, R.; He, C.; Lin, D.; and Wang, J. 2024. Songcomposer: A large language model for lyric and melody composition in song generation. arXiv preprint arXiv:2402.17645

    2024 arXiv

  7. [15]

    Donahue, C.; Caillon, A.; Roberts, A.; Manilow, E.; Esling, P.; Agostinelli, A.; Verzetti, M.; Simon, I.; Pietquin, O.; Zeghidour, N.; et al. 2023. Singsong: Generating musical accompaniments from singing. arXiv preprint arXiv:2301.12662

    2023 arXiv

  8. [16]

    Du, C.; Guo, Y.; Shen, F.; Liu, Z.; Liang, Z.; Chen, X.; Wang, S.; Zhang, H.; and Yu, K. 2024 a . UniCATS: A unified context-aware text-to-speech framework with contextual vq-diffusion and vocoding. In AAAI, volume 38, 17924--17932

    2024

  9. [17]

    Du, C.; Guo, Y.; Wang, H.; Yang, Y.; Niu, Z.; Wang, S.; Zhang, H.; Chen, X.; and Yu, K. 2024 b . VALL-T: Decoder-Only Generative Transducer for Robust and Decoding-Controllable Text-to-Speech. arXiv preprint arXiv:2401.14321

    2024 arXiv

  10. [18]

    Elizalde, B.; Deshmukh, S.; Al Ismail, M.; and Wang, H. 2023. Clap learning audio concepts from natural language supervision. In ICASSP, 1--5

    2023

  11. [19]

    D.; Carr, C.; Zukowski, Z.; Taylor, J.; and Pons, J

    Evans, Z.; Parker, J. D.; Carr, C.; Zukowski, Z.; Taylor, J.; and Pons, J. 2024. Long-form music generation with latent diffusion. arXiv preprint arXiv:2404.10301

    2024 arXiv

  12. [20]

    F.; Seetharaman, P.; Kumar, R.; and Pardo, B

    Garcia, H. F.; Seetharaman, P.; Kumar, R.; and Pardo, B. 2023. Vampnet: Music generation via masked acoustic token modeling. arXiv preprint arXiv:2307.04686

    2023 arXiv

  13. [21]

    He, J.; Liu, J.; Ye, Z.; Huang, R.; Cui, C.; Liu, H.; and Zhao, Z. 2023. RMSSinger: Realistic-Music-Score based Singing Voice Synthesis. In Findings of ACL, 236--248

    2023

  14. [22]

    Ho, J.; Jain, A.; and Abbeel, P. 2020. Denoising Diffusion Probabilistic Models. In NeurIPS, volume 33, 6840--6851

    2020

  15. [23]

    Ho, J.; and Salimans, T. 2022. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598

    2022 arXiv

  16. [24]

    H.; Lakhotia, K.; Salakhutdinov, R.; and Mohamed, A

    Hsu, W.-N.; Bolte, B.; Tsai, Y.-H. H.; Lakhotia, K.; Salakhutdinov, R.; and Mohamed, A. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language processing, 29: 3451--3460

    2021

  17. [25]

    Y.; and Ellis, D

    Huang, Q.; Jansen, A.; Lee, J.; Ganti, R.; Li, J. Y.; and Ellis, D. P. 2022. Mulan: A joint embedding of music audio and natural language. ISMIR

    2022

  18. [26]

    Hwang, J.-S.; Lee, S.-H.; and Lee, S.-W. 2025. Hiddensinger: High-quality singing voice synthesis via neural audio codec and latent diffusion models. Neural Networks, 181: 106762

    2025

  19. [27]

    Jiang, Z.; Ren, Y.; Ye, Z.; Liu, J.; Zhang, C.; Yang, Q.; Ji, S.; Huang, R.; Wang, C.; Yin, X.; et al. 2023. Mega-tts: Zero-shot text-to-speech at scale with intrinsic inductive bias. arXiv preprint arXiv:2306.03509

    2023 arXiv

  20. [28]

    Kilgour, K.; Zuluaga, M.; Roblek, D.; and Sharifi, M. 2019. Fréchet Audio Distance: A Reference-Free Metric for Evaluating Music Enhancement Algorithms. In Interspeech, 2350--2354

    2019

  21. [29]

    Kim, J.; Kong, J.; and Son, J. 2021. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In ICML, 5530--5540

    2021

  22. [30]

    Kim, T.; and Nam, J. 2023. All-In-One Metrical And Functional Structure Analysis With Neighborhood Attentions on Demixed Audio. In IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)

    2023

  23. [31]

    P.; and Welling, M

    Kingma, D. P.; and Welling, M. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114

    2013 arXiv

  24. [32]

    Kong, J.; Kim, J.; and Bae, J. 2020. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. In Advances in neural information processing systems, volume 33, 17022--17033

    2020

  25. [33]

    Kumar, R.; Seetharaman, P.; Luebs, A.; Kumar, I.; and Kumar, K. 2023. High-Fidelity Audio Compression with Improved RVQGAN. In NeurIPS, volume 36, 27980--27993

    2023

  26. [34]

    Lam, M. W. Y.; Tian, Q.; Li, T.; Yin, Z.; Feng, S.; Tu, M.; Ji, Y.; Xia, R.; Ma, M.; Song, X.; Chen, J.; Yuping, W.; and Wang, Y. 2023. Efficient Neural Music Generation. In NeurIPS, volume 36, 17450--17463

    2023

  27. [35]

    Li, Y.; Yuan, R.; Zhang, G.; Ma, Y.; Chen, X.; Yin, H.; Xiao, C.; Lin, C.; Ragni, A.; Benetos, E.; et al. 2024. MERT: Acoustic music understanding model with large-scale self-supervised training. ICLR

    2024

  28. [36]

    Lin, L.; Xia, G.; Jiang, J.; and Zhang, Y. 2023. Content-based controls for music large language modeling. arXiv preprint arXiv:2310.17162

    2023 arXiv

  29. [37]

    Liu, J.; Li, C.; Ren, Y.; Chen, F.; and Zhao, Z. 2022. Diffsinger: Singing voice synthesis via shallow diffusion mechanism. In AAAI, volume 36, 11020--11028

    2022

  30. [38]

    Loshchilov, I.; and Hutter, F. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101

    2017 arXiv

  31. [39]

    Luo, Y.; and Yu, J. 2023. Music source separation with band-split RNN. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31: 1893--1901

    2023

  32. [40]

    Melechovsky, J.; Guo, Z.; Ghosal, D.; Majumder, N.; Herremans, D.; and Poria, S. 2024. Mustango: Toward controllable text-to-music generation. In NAACL, 8293--8316

    2024

  33. [41]

    Peebles, W.; and Xie, S. 2023. Scalable diffusion models with transformers. In CVPR, 4195--4205

    2023

  34. [42]

    Peng, P.; Huang, P.-Y.; Li, D.; Mohamed, A.; and Harwath, D. 2024. VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild. In ACL, 12442--12462

    2024

  35. [43]

    Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21: 1--67

    2020

  36. [44]

    Rajbhandari, S.; Rasley, J.; Ruwase, O.; and He, Y. 2020. Zero: Memory optimizations toward training trillion parameter models. In International Conference for High Performance Computing, Networking, Storage and Analysis, 1--16

    2020

  37. [45]

    Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; and Ommer, B. 2022. High-resolution image synthesis with latent diffusion models. In CVPR, 10684--10695

    2022

  38. [46]

    Ronneberger, O.; Fischer, P.; and Brox, T. 2015. U-net: Convolutional networks for biomedical image segmentation. In MICCAI 2015, 234--241

    2015

  39. [47]

    Shen, K.; Ju, Z.; Tan, X.; Liu, Y.; Leng, Y.; He, L.; Qin, T.; Zhao, S.; and Bian, J. 2023. Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers. arXiv preprint arXiv:2304.09116

    2023 arXiv

  40. [48]

    Su, J.; Ahmed, M.; Lu, Y.; Pan, S.; Bo, W.; and Liu, Y. 2024. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 127063

    2024

  41. [49]

    Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozi \`e re, B.; Goyal, N.; Hambro, E.; Azhar, F.; et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  42. [50]

    van den Oord, A.; Vinyals, O.; and kavukcuoglu, k. 2017. Neural Discrete Representation Learning. In NeurIPS, volume 30

    2017

  43. [51]

    Vyas, A.; Shi, B.; Le, M.; Tjandra, A.; Wu, Y.-C.; Guo, B.; Zhang, J.; Zhang, X.; Adkins, R.; Ngan, W.; et al. 2023. Audiobox: Unified audio generation with natural language prompts. arXiv preprint arXiv:2312.15821

    2023 arXiv

  44. [52]

    Wang, C.; Chen, S.; Wu, Y.; Zhang, Z.; Zhou, L.; Liu, S.; Chen, Z.; Liu, Y.; Wang, H.; Li, J.; et al. 2023 a . Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111

    2023 arXiv

  45. [53]

    Wang, Y.; Ju, Z.; Tan, X.; He, L.; Wu, Z.; Bian, J.; et al. 2023 b . Audit: Audio editing by following instructions with latent diffusion models. In NeurIPS, volume 36, 71340--71357

    2023

  46. [54]

    Wu, S.-L.; Donahue, C.; Watanabe, S.; and Bryan, N. J. 2024 a . Music controlnet: Multiple time-varying controls for music generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32: 2692--2703

    2024

  47. [55]

    Wu, Y.; Chen, K.; Zhang, T.; Hui, Y.; Berg-Kirkpatrick, T.; and Dubnov, S. 2023. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP, 1--5

    2023

  48. [56]

    Wu, Y.; Shi, J.; Tang, Y.; Yang, S.; Jin, Q.; et al. 2024 b . TokSing: Singing Voice Synthesis based on Discrete Tokens. In Interspeech, 2549--2553

    2024

  49. [57]

    Yang, D.; Wang, D.; Guo, H.; Chen, X.; Wu, X.; and Meng, H. 2024. SimpleSpeech: Towards Simple and Efficient Text-to-Speech with Scalar Latent Transformer Diffusion Models. In Interspeech

    2024

  50. [58]

    Yu, J.; Chen, H.; Bian, Y.; Li, X.; Luo, Y.; Tian, J.; Liu, M.; Jiang, J.; and Wang, S. 2024. AutoPrep: An Automatic Preprocessing Framework for In-The-Wild Speech Data. In ICASSP, 1136--1140

    2024

  51. [59]

    Yu, J.; Chen, H.; Luo, Y.; Gu, R.; and Weng, C. 2023. High Fidelity Speech Enhancement with Band-split RNN. In Interspeech, 2483--2487

    2023

  52. [60]

    Zeghidour, N.; Luebs, A.; Omran, A.; Skoglund, J.; and Tagliasacchi, M. 2021. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30: 495--507

    2021

  53. [61]

    Zhang, Y.; Cong, J.; Xue, H.; Xie, L.; Zhu, P.; and Bi, M. 2022 a . Visinger: Variational inference with adversarial learning for end-to-end singing voice synthesis. In ICASSP, 7237--7241

    2022

  54. [62]

    A.; Liao, W.-H.; Mitsufuji, Y.; and Dixon, S

    Zhang, Y.; Ikemiya, Y.; Xia, G.; Murata, N.; Martínez-Ramírez, M. A.; Liao, W.-H.; Mitsufuji, Y.; and Dixon, S. 2024. MusicMagus: Zero-Shot Text-to-Music Editing via Diffusion Models. In IJCAI, 7805--7813

    2024

  55. [63]

    Zhang, Y.; Xue, H.; Li, H.; Xie, L.; Guo, T.; Zhang, R.; and Gong, C. 2022 b . Visinger 2: High-fidelity end-to-end singing voice synthesis enhanced by digital signal processing synthesizer. arXiv preprint arXiv:2211.02903

    2022 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.