Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pair Prediction

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper introduces NTPP, a decoder-only generative model that learns the joint distribution of both speakers' speech-token streams in a dual-channel recording, and reports that it improves turn-taking prediction, response coherence, and…

desk verdict A clean, genuinely new architectural idea for full-duplex SLMs, with an empirical package that mostly supports it; the VAD contradiction and missing code are the main things to fix. read the letter →

arxiv 2506.00975 v4 pith:O4YLKYQB submitted 2025-06-01 cs.CL cs.AIcs.SDeess.AS

Qichao Wang , Ziqiao Meng , Wenqian Cui , Yifei Zhang , Pengcheng Wu , Bingzhe Wu , Irwin King , Liang Chen
show 1 more author
Peilin Zhao
This is my paper
classification cs.CLcs.AIcs.SDeess.AS
keywords speechlanguagemodeldual-channelnext-token-pairpredictionturn-takingspokendialoguedecoder-onlytransformerresidualvectorquantizationfull-duplexinteraction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Real-time spoken dialogue with an AI requires a model that can listen and speak simultaneously, but most speech language models treat conversation as one channel conditioned on another. This paper proposes next-token-pair prediction (NTPP), which trains a decoder-only transformer to predict the next pair of speech tokens, one from each speaker, at every aligned time step, thereby modeling the joint distribution of both channels. The authors claim this yields better turn-taking statistics, more coherent and natural responses in human evaluation, and lower inference latency than prior dual-channel dialogue systems, because the model keeps a single KV cache and needs no voice-activity detection or extra encoder. The method is demonstrated with both VQ and RVQ speech tokenizers and evaluated on the Fisher corpus plus the out-of-distribution CANDOR set.

What carries the argument

The load-bearing object is the token pair $(s^a_t, s^b_t)$ together with the pair-wise causal masking matrix. The two speakers' token streams are interleaved so that aligned tokens share a positional embedding but receive distinct one-hot channel embeddings; the $2\times2$ block-diagonal mask keeps the two tokens within a step from attending to each other, enforcing the conditional-independence factorization of Eq. (7). For RVQ tokenizers, a cyclic depth embedding derived from $\sin$/ $\cos$ of position modulo depth tells the model which residual quantization level each token belongs to, and the mask is expanded to $2D\times2D$ blocks. This mechanism is what lets a standard decoder-only LLM learn a joint, speaker-independent distribution over both channels while keeping a single KV cache for streaming inference.

What would settle it

Compute the conditional mutual information $I(s^a_t; s^b_t \mid s^a_{<t}, s^b_{<t})$ on the Fisher test set at aligned time steps in overlap and backchannel regions. If it is appreciably nonzero, NTPP's factorized objective cannot assign more probability to those events than the product of its marginals, and a variant of the same transformer with intra-step cross-channel attention should score higher on turn-taking metrics and human naturalness judgments.

Watch

Extended reading notes

Core claim

The central claim is that dual-channel speech recordings can be modeled directly by an autoregressive transformer as a joint distribution over aligned token pairs rather than as a conditional distribution of one speaker given the other. NTPP factorizes $p(S_a,S_b)=\prod_t p(s^a_t|s^a_{<t},s^b_{<t})p(s^b_t|s^a_{<t},s^b_{<t})$, predicting both speakers' next tokens at each time step while a pair-wise causal mask prevents the two current tokens from attending to each other. This makes the learned distribution speaker-independent (permutation invariant over channel order), removes the need for VAD modules and external encoders, and extends naturally to RVQ tokenizers via a cyclic depth embedding. In experiments, NTPP reports closer alignment with human-annotated pause and interruption judgments, higher MOS naturalness and meaningfulness than baselines including Moshi and SyncLLM, near-zero turn-taking metric drift under speaker-channel swapping, and latency below the perceptual threshold even across ten turn-taking rounds.

Load-bearing premise

The model assumes that, given everything both speakers have already said, what each speaker says at the next moment can be predicted separately without needing to know what the other speaker is saying at that very same moment.

Editorial extensions

If this is right

  • Speech language models can be trained for full-duplex dialogue without an extra encoder, a VAD module, or speaker-order conditioning, so the same decoder-only recipe used for text LLMs transfers to dual-channel speech.
  • Turn-taking statistics such as overlap, gap, pause, and interruption rates move measurably closer to ground-truth human conversation on the Fisher benchmark, and naturalness and meaningfulness scores generalize to the out-of-distribution CANDOR corpus.
  • Latency grows more slowly with conversation length than in two-transformer designs like Moshi, because NTPP maintains a single KV cache; the paper reports response times below the 220 ms perceptual threshold across ten turns.
  • Speaker independence holds under channel swapping: turn-taking metric variation between original and swapped speaker order is near zero on training and low on test, making the model robust to which microphone is labeled speaker A.
  • Textless training without ASR transcripts converges faster and reaches lower perplexity than training with textual supervision, suggesting that pure acoustic self-supervision alone is sufficient for dual-channel dialogue modeling.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the conditional-independence assumption in Eq. (7) is the binding constraint, NTPP's gains should concentrate on longer-range conversational dynamics; same-timestep reactions such as a backchannel triggered by the other speaker's just-uttered syllable would need cross-channel attention within a step, which the block-diagonal mask forbids.
  • The pair-prediction scheme is a general recipe for aligned multi-stream generation: any streams that share a time axis and have distinguishable identities (instrument stems, music and vocals, speech and gesture) could be interleaved the same way, provided per-step cross-stream dependence is weak.
  • A direct falsifier would be to measure the conditional mutual information $I(s^a_t; s^b_t \mid \text{history})$ on dual-channel corpora; where it is substantively nonzero in overlap and backchannel regions, NTPP's factorized objective cannot represent that dependence, and a model with intra-step cross-attention should outperform it on those events.
  • Since the paper uses only 2,200 hours of dual-channel data, the practical ceiling may be data rather than architecture; synthetic dual-channel data generation, which the authors flag as future work, is the most direct test of whether the modeling paradigm scales.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces Next-Token-Pair Prediction (NTPP), a decoder-only autoregressive modeling paradigm for dual-channel speech dialogue. Instead of modeling one speaker conditional on the other, NTPP factorizes the joint distribution p(Sa,Sb) as a product over time steps of p(sa_t,sb_t | past) and further assumes conditional independence of the two speakers' current tokens given the past (Eq. 7). This is implemented with interleaved token pairs, shared positional embeddings, channel embeddings, and a pair-wise causal mask (Section 4.2), with an RVQ extension using a cyclic depth embedding (Section 4.3). The model is trained in two stages: first on roughly 140,000 hours of single-channel audio, then on the Fisher dual-channel corpus. Experiments cover turn-taking event statistics (Table 2), interruption behavior (Figure 6), human MOS evaluation (Table 3), inference latency (Figure 7), speaker independence (Table 4), and ablations (Figure 8). The authors claim that NTPP improves turn-taking prediction, response coherence, and naturalness while achieving lower latency, and they position the method as VAD-free, speaker-independent, and encoder-free.

Significance. If validated, NTPP is a clean and appealing contribution: it adapts a standard decoder-only transformer to full-duplex dual-channel speech with a single KV cache, avoids a separate context encoder, and explicitly models a joint rather than conditional distribution. The probability model is stated precisely, the masking scheme is internally consistent, and the paper includes a useful RVQ extension. The availability of demo and code is a concrete strength. However, the paper's headline claims currently exceed the evidence: the conditional-independence assumption removes same-step cross-speaker dependencies that are relevant to overlap and interruption behavior, the VAD-free claim is contradicted within the text, and the main comparative evaluations lack significance testing. These issues are fixable but need to be addressed before the claims can be accepted at face value.

major comments (5)
  1. [Section 5.3] The conditional independence assumption p(sa_t,sb_t|past)=p(sa_t|past)p(sb_t|past) is load-bearing and structurally prevents the model from representing any same-step dependence between the two speakers' current tokens. The pair-wise causal mask in Section 4.2 explicitly blocks sa_t and sb_t from attending to each other at time t, so the model cannot capture interruptions, backchannels, or overlaps whose content is contingent on the other speaker's just-produced syllable. The experiments in Table 2 measure aggregate counts and durations, which can be matched by one-step-lagged statistics; Table 3 reports holistic MOS. Neither isolates same-step content dependence. Please either add an experiment that tests content-level same-step contingency (for example, whether an interruption completes or reacts to the current word being produced) or explicitly temper the claims about improved turn-taking naturalness to one-step-lagged dynamics.
  2. [Section 5.4] The turn-taking statistics in Table 2 are evaluated on the Fisher corpus, which is also the training corpus for stage two. No held-out split, cross-validation, or out-of-distribution turn-taking evaluation is reported, so the low absolute deviations from Fisher ground truth may partly reflect memorization rather than generalization. Please report the turn-taking metrics on a held-out Fisher split or on an external dual-channel dataset such as CANDOR, whose audio was only used for human evaluation here.
  3. [Table 3 and Table 2] Table 1 lists 'VAD-Free' as an advantage of NTPP, but Section 5.4 states that 'NTPP employs Voice Activity Detection (VAD) and considers a silence state to have been reached after 200ms.' Appendix C.2 also describes using VAD tools to compute turn-taking metrics, though that is an evaluation detail rather than a model component. The contradiction is material because the VAD-free claim is a headline advantage. Please clarify whether VAD is used during inference, at evaluation, or both, and revise the claims accordingly.
  4. [Section 5.6] The paper uses the word 'significantly' in the abstract and throughout, but no significance tests, confidence intervals, or error bars are reported for the turn-taking statistics or the MOS comparisons. In Table 3, for example, the overall meaningfulness scores are 3.95 (0.04) for NTPP vs. 3.90 (0.07) for Moshi, and the Fisher naturalness scores are 4.42 (0.06) vs. 3.90 (0.08); these differences are small relative to the reported standard errors and could be within noise. Please provide paired significance tests or, at minimum, explicit confidence intervals for all claimed improvements, or soften the language to 'numerically higher' where significance is not established.
  5. [Section 5.4] The inference latency comparison is made only against Moshi, despite LSLM and dGSLM being included in earlier experiments. The abstract and introduction claim 'substantially lower inference latency than existing methods,' but the evidence only supports a comparison with one baseline on this metric. Please add latency measurements for at least LSLM and dGSLM, or restrict the claim to the Moshi comparison.
minor comments (6)
  1. [Table 2 heading] There is a typo in 'hard tunr-taking interval threshold'—it should be 'turn-taking.'
  2. [Section 5.4] The heading 'Cumulated duration' should be 'Cumulative duration' for consistency with standard usage.
  3. [Figure 5] The phrase 'For instance, informing its response strategy' is grammatically awkward and should be rephrased for clarity.
  4. [Section 4.2] The sentence 'Perplexity curves highlights that...' has a subject-verb agreement error; it should be 'Perplexity curves highlight that...'.
  5. [Section 5.3] The notation [za_t, zb_t] is ambiguous: it is not clear whether this denotes concatenation along a new axis or a block-diagonal construction, especially since the subsequent pair-wise causal mask operates on a 2T×2T matrix. Please clarify the tensor shapes in the embedding operation.
  6. [Section 5.8] The evaluation of turn-taking statistics should state explicitly how many generated dialogues were used, the lengths of those dialogues, and whether the same random seeds were used across baselines, to support reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: NTPP's joint-distribution factorization and masking are structural modeling choices, and the reported turn-taking, MOS, and latency results are measured external outcomes rather than re-statements of the training objective.

full rationale

The paper's derivation chain is self-contained in the relevant sense. Equation (6) defines the joint distribution over the two speaker channels as a product of next-pair conditionals, and Equation (7) explicitly imposes conditional independence between speakers within each time step. Section 4.2 then implements this factorization through token-pair embeddings, shared positional and channel embeddings, and a pair-wise causal mask whose 2x2 diagonal blocks allow only self-attention, so that sa_t and sb_t cannot attend to each other at the same time step. Section 4.3 extends the same masking logic to RVQ with a 2D x 2D block structure. These are architectural choices, not quantities fitted to the evaluation metrics. The evaluation sections report turn-taking statistics of generated dialogues against Fisher ground-truth statistics, human MOS ratings, interruption-alignment judgments, speaker-swap robustness, and inference latency; none of these numbers are derived by construction from the training loss or from the parameters used to define the model. Using the Fisher corpus both for training and for grounding the target turn-taking statistics is standard target-distribution evaluation, following the dGSLM protocol, and does not reduce the predictions to the inputs. There is no load-bearing self-citation chain and no uniqueness theorem imported from the authors' prior work. The skeptical concern about Equation (7) is a genuine modeling limitation: same-step cross-speaker dependencies such as content-level interruption or immediate backchannel contingency are structurally excluded. But that is a limitation of the proposed factorization, not a circularity, since the paper does not claim to derive that capability from the same assumption. Overall, the derivation and the empirical claims are independent in the sense required by the circularity analysis, so no circular step is identified.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims rest on one structural probability assumption and several hand-set engineering constants. No parameter is fitted to the evaluation targets, so the methodological risk is reproducibility rather than circularity.

free parameters (4)
  • lambda (streaming chunk size)
    Hand-set hyperparameter controlling how many user tokens arrive before the model starts generating; directly affects latency and the streaming inference procedure in Appendix B.
  • D (RVQ depth levels)
    Number of residual quantization levels per time step; not stated in the paper, but it determines sequence length and is required to reproduce the RVQ variant.
  • VAD silence threshold = 200 ms
    Used in Section 5.4 to decide when the model speaks; a hand-set constant, and its interplay with the 'VAD-free' claim is unclear.
  • temperature = 0.1, 0.5, 0.9
    Sampling temperatures used in Table 2; evaluation hyperparameters, not learned.
assumptions (4)
  • ad hoc to paper Conditional independence of the two speakers' current tokens given the past (Eq. 7).
    The central modeling assumption; it defines the joint distribution that NTPP optimizes and is implemented by the block-wise causal mask.
  • domain assumption Fisher dataset provides clean separate-channel recordings of two speakers.
    The method and evaluation rely on the channels being speaker-separated without crosstalk; used throughout Sections 5.1 and 5.3.
  • domain assumption RVQ tokens at 40 frames/sec preserve turn-taking-relevant acoustic events.
    Relies on the SoundStream-style tokenizer and multi-speaker HiFi-GAN; if short overlap events are lost, the model cannot learn them.
  • standard math Autoregressive next-token prediction over discrete speech tokens is a valid generative model for speech.
    Standard SLM assumption inherited from prior work (Eqs. 2-3), not defended in this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pair Prediction." pith.science (2026). https://pith.science/paper/O4YLKYQB

@misc{pith2026250600975,
  author       = {Pith},
  title        = {Pith review of: NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pair Prediction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O4YLKYQB}},
  note         = {Machine review of arXiv:2506.00975}
}
read the original abstract

Inspired by the impressive capabilities of GPT-4o, there is growing interest in enabling speech language models (SLMs) to engage in natural, fluid spoken interactions with humans. Recent advancements have led to the development of several SLMs that demonstrate promising results in this area. However, current approaches have yet to fully exploit dual-channel speech data, which inherently captures the structure and dynamics of human conversation. In this work, we systematically explore the use of dual-channel speech data in the context of modern large language models, and introduce a novel generative modeling paradigm, Next-Token-Pair Prediction (NTPP), to enable speaker-independent dual-channel spoken dialogue learning using decoder-only architectures for the first time. We evaluate our approach on standard benchmarks, and empirical results show that our proposed method, NTPP, significantly improves the conversational abilities of SLMs in terms of turn-taking prediction, response coherence, and naturalness. Moreover, compared to existing methods, NTPP achieves substantially lower inference latency, highlighting its practical efficiency for real-time applications.

Figures

Figures reproduced from arXiv: 2506.00975 by the authors.

Figure 1
Figure 1. The dual-channel speech encapsulates various conversa￾tional turn-taking events, including: (a) Overlap, (b) Backchannel, (c) Pause, and (d) Interruption. These events are intermingled within the single-channel audio stream but could be explicitly ob￾served in the dual-channel audio stream. OpenAI, 2023; 2024), has significantly revolutionized the realm of artificial intelligence. These potent language mod￾els (LMs)… view at source ↗
Figure 2
Figure 2. An illustration of three different generative models for spoken dialogue is shown: (a) Turn-based dialogue modeling, as formulated in Eq. 4, which is commonly used in cascading and multi-modal approaches; (b) The NTP paradigm with a context encoder architecture, adopted by models such as LSLM, Moshi, and similar variants; (c) Our NTPP paradigm, which employs a decoder-only transformer. Replacing the decoder-only arc… view at source ↗
Figure 3
Figure 3. The illustration of the autoregressive transformer for learning the dual-channel speech sequence, with the token pair embedding operation (left), the overall architecture (middle) and the pair-wise causal masking mechanism (right). Existing popular approaches model dialogue in a sequential generation manner, p(S b |S a ), treating one speaker sequence (assume a) as a given condition: p(S b |S a ) = Y T t=1 p(s b t |… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: The illustration of two modifed components of RVQ-based dual-channel tranformer: (a) Token pair embedding operation (including cyclic depth embedding) and (b) RVQ causal masking mechanism. ((s a t , sb t ),(s a t−1 , sb t−1 ), ...,(s a 1 , sb 1 )). At each time step t,…
Figure 5
Figure 5. Figure 5: Comparison of training loss curves across different mod￾els. Solid lines show the training progress for different foundation models. Dashed lines represent an ablation study comparing a model trained with textual data (w Text) versus without (w/o Text). it with establi…
Figure 7
Figure 7. Figure 7: Our method (blue) demonstrates lower latency growth compared to Moshi’s linear degradation(red), maintaining response times below perceptual thresholds (220 ms) across all rounds. formance remains stable when the roles of the two speakers are exchanged—an essential ind…
Figure 8
Figure 8. Figure 8: Comparative performance analysis. (a) Ablation study of NTPP model training stages, illustrating perplexity versus training steps for the full two-stage model and variants lacking either stage one or stage two. For better visualization, we trim the stage one training s…
Figure 9
Figure 9. Figure 9: The figure illustrates the chunk-wise streaming inference process. Within each chunk, (s a 1, sa 2, sa 3, sa 4, sa 5) represents the pro￾vided speaker sequence. Their corresponding keys and values are stored in the KV-cache. NTPP sequentially predicts tokens (ˆs b 1, s…
Figure 10
Figure 10. Figure 10: Illustration of turn-taking events: IPU (Interpausal Unit), Turn (for speaker A and Speaker B, resp), P.(within-speaker Pause), Gap and Overlap. Our model generates two audio channels at the same time, allowing us to use basic Voice Activity Detection (VAD) tools on t…
Figure 11
Figure 11. Figure 11: Case study of NTPP interrupt human speaking correctly and timely. To intuitively understand the differences in responses from our models, we provide an example in [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TurnNat: Automatic Evaluation of Turn-Taking Naturalness in Dyadic Spoken Dialogue

    cs.CL 2026-07 unverdicted novelty 6.0 of 10

    TurnNat introduces a likelihood-based automatic evaluation method for turn-taking naturalness in dyadic spoken dialogues using a causal prediction model and a human-validated perturbation benchmark.

Reference graph

Works this paper leans on

85 extracted references · 54 canonical work pages · cited by 1 Pith paper

  1. [1]

    URL https://chattts.com/

    Chattts, 2024. URL https://chattts.com/

    2024

  2. [2]

    L., Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., Binkowski, M., Barreira, R., Vinyals, O., Zisserman, A., and Simonyan, K

    Alayrac, J., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J. L., Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., Binkowski, M., Barreira, R., Vinyals, O., Zisserman, A., and Simonyan, K. Flaming...

    2022

  3. [3]

    Seed-tts: A family of high-quality versatile speech generation models

    Anastassiou, P., Chen, J., Chen, J., Chen, Y., Chen, Z., Chen, Z., Cong, J., Deng, L., Ding, C., Gao, L., Gong, M., Huang, P., Huang, Q., Huang, Z., Huo, Y., Jia, D., Li, C., Li, F., Li, H., Li, J., Li, X., Li, X., Liu, L., Liu, S., Liu, S., Liu, X., Liu, Y., Liu, Z., Lu, L., Pan, J., Wang, X., Wang, Y., Wang, Y., Wei, Z., Wu, J., Yao, C., Yang, Y., Yi, Y...

    arXiv 2024

  4. [4]

    Speecht5: Unified-modal encoder-decoder pre-training for spoken language processing

    Ao, J., Wang, R., Zhou, L., Wang, C., Ren, S., Wu, Y., Liu, S., Ko, T., Li, Q., Zhang, Y., Wei, Z., Qian, Y., Li, J., and Wei, F. Speecht5: Unified-modal encoder-decoder pre-training for spoken language processing. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland,...

    2022

  5. [5]

    M., and Weber, G

    Ardila, R., Branson, M., Davis, K., Henretty, M., Kohler, M., Meyer, J., Morais, R., Saunders, L., Tyers, F. M., and Weber, G. Common voice: A massively-multilingual speech corpus. arXiv preprint arXiv:1912.06670, 2019

    arXiv 1912

  6. [6]

    UniverSLU: Universal Spoken Language Understanding for Diverse Tasks with Natural Language Instructions

    Arora, S., Futami, H., Jung, J., Peng, Y., Sharma, R. S., Kashiwagi, Y., Tsunoo, E., and Watanabe, S. Universlu: Universal spoken language understanding for diverse classification and sequence generation tasks with a single network. CoRR, abs/2310.02973, 2023

    work page Pith review arXiv

  7. [7]

    Wav2vec 2.0: Learning the structure of speech from raw audio

    Baevski, A., Auli, M., and Conneau, A. Wav2vec 2.0: Learning the structure of speech from raw audio. Meta AI, 24, 2020

    2020

  8. [8]

    Audiolm: A language modeling approach to audio generation

    Borsos, Z., Marinier, R., Vincent, D., Kharitonov, E., Pietquin, O., Sharifi, M., Roblek, D., Teboul, O., Grangier, D., Tagliasacchi, M., and Zeghidour, N. Audiolm: A language modeling approach to audio generation. IEEE ACM Trans. Audio Speech Lang. Process. , 31: 0 2523--2533, 2023

    2023

Show all 85 references
  1. [9]

    D., J \' u nior, A

    Casanova, E., Weber, J., Shulby, C. D., J \' u nior, A. C., G \" o lge, E., and Ponti, M. A. Yourtts: Towards zero-shot multi-speaker TTS and zero-shot voice conversion for everyone. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryla...

    2022

  2. [10]

    Chang, H., Zhang, H., Jiang, L., Liu, C., and Freeman, W. T. Maskgit: Masked generative image transformer. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pp.\ 11305--11315. IEEE , 2022

    2022

  3. [11]

    VALL-E 2: Neural codec language models are human parity zero-shot text to speech synthesizers

    Chen, S., Liu, S., Zhou, L., Liu, Y., Tan, X., Li, J., Zhao, S., Qian, Y., and Wei, F. VALL-E 2: Neural codec language models are human parity zero-shot text to speech synthesizers. CoRR, abs/2406.05370, 2024

    2024 arXiv

  4. [12]

    Qwen2-audio technical report

    Chu, Y., Xu, J., Yang, Q., Wei, H., Wei, X., Guo, Z., Leng, Y., Lv, Y., He, J., Lin, J., Zhou, C., and Zhou, J. Qwen2-audio technical report. CoRR, abs/2407.10759, 2024

    2024 arXiv

  5. [13]

    Fisher english training speech part 1 transcripts

    Cieri, C., Graff, D., Kimball, O., Miller, D., and Walker, K. Fisher english training speech part 1 transcripts. Philadelphia: Linguistic Data Consortium, 2004

    2004

  6. [14]

    Simple and controllable music generation

    Copet, J., Kreuk, F., Gat, I., Remez, T., Kant, D., Synnaeve, G., Adi, Y., and D \' e fossez, A. Simple and controllable music generation. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, Ne...

    2023

  7. [15]

    Y., Ermon, S., Rudra, A., and R \' e , C

    Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and R \' e , C. Flashattention: Fast and memory-efficient exact attention with io-awareness. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Or...

    2022

  8. [16]

    Moshi: a speech-text foundation model for real-time dialogue

    D\'efossez, A., Mazar\'e, L., Orsini, M., Royer, A., P\'erez, P., J\'egou, H., Grave, E., and Zeghidour, N. Moshi: a speech-text foundation model for real-time dialogue. Technical report, Kyutai, September 2024. URL http://kyutai.org/Moshi.pdf

    2024

  9. [17]

    Pengi: An audio language model for audio tasks

    Deshmukh, S., Elizalde, B., Singh, R., and Wang, H. Pengi: An audio language model for audio tasks. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2...

    2023

  10. [18]

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al - Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Roz...

    2024 arXiv

  11. [19]

    A., and Wang, H

    Elizalde, B., Deshmukh, S., Ismail, M. A., and Wang, H. CLAP learning audio concepts from natural language supervision. In IEEE International Conference on Acoustics, Speech and Signal Processing ICASSP 2023, Rhodes Island, Greece, June 4-10, 2023 , pp.\ 1--5. IEEE , 2023

    2023

  12. [20]

    Taming transformers for high-resolution image synthesis

    Esser, P., Rombach, R., and Ommer, B. Taming transformers for high-resolution image synthesis. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021 , pp.\ 12873--12883. Computer Vision Foundation / IEEE , 2021

    2021

  13. [21]

    Llama-omni: Seamless speech interaction with large language models

    Fang, Q., Guo, S., Zhou, Y., Ma, Z., Zhang, S., and Feng, Y. Llama-omni: Seamless speech interaction with large language models. arXiv preprint arXiv:2409.06666, 2024

    2024 arXiv

  14. [22]

    Audiochatllama: Towards general-purpose speech abilities for llms

    Fathullah, Y., Wu, C., Lakomkin, E., Li, K., Jia, J., Shangguan, Y., Mahadeokar, J., Kalinli, O., Fuegen, C., and Seltzer, M. Audiochatllama: Towards general-purpose speech abilities for llms. In Proceedings of the 2024 Conference of the North American Chapter of the Associati...

    2024

  15. [23]

    Vita: Towards open-source interactive omni multimodal llm, 2024

    Fu, C., Lin, H., Long, Z., Shen, Y., Zhao, M., Zhang, Y., Dong, S., Wang, X., Yin, D., Ma, L., Zheng, X., He, R., Ji, R., Wu, Y., Shan, C., and Sun, X. Vita: Towards open-source interactive omni multimodal llm, 2024

    2024

  16. [24]

    Funasr: A fundamental end-to-end speech recognition toolkit

    Gao, Z., Li, Z., Wang, J., Luo, H., Shi, X., Chen, M., Li, Y., Zuo, L., Du, Z., and Zhang, S. Funasr: A fundamental end-to-end speech recognition toolkit. In 24th Annual Conference of the International Speech Communication Association, Interspeech 2023, Dublin, Ireland, August...

    2023

  17. [25]

    A., Gat, I., Conneau, A., Kreuk, F., Copet, J., D \' e fossez, A., Synnaeve, G., Dupoux, E., Schwartz, R., and Adi, Y

    Hassid, M., Remez, T., Nguyen, T. A., Gat, I., Conneau, A., Kreuk, F., Copet, J., D \' e fossez, A., Synnaeve, G., Dupoux, E., Schwartz, R., and Adi, Y. Textually pretrained speech language models. In Advances in Neural Information Processing Systems 36: Annual Conference on N...

    2023

  18. [26]

    Kenlm: Faster and smaller language model queries

    Heafield, K. Kenlm: Faster and smaller language model queries. In Proceedings of the sixth workshop on statistical machine translation, pp.\ 187--197, 2011

    2011

  19. [27]

    Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models

    Huang, R., Huang, J., Yang, D., Ren, Y., Liu, L., Li, M., Ye, Z., Liu, J., Yin, X., and Zhao, Z. Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , ...

    2023

  20. [28]

    Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023 a

    2023 arXiv

  21. [29]

    Mega-tts: Zero-shot text-to-speech at scale with intrinsic inductive bias

    Jiang, Z., Ren, Y., Ye, Z., Liu, J., Zhang, C., Yang, Q., Ji, S., Huang, R., Wang, C., Yin, X., Ma, Z., and Zhao, Z. Mega-tts: Zero-shot text-to-speech at scale with intrinsic inductive bias. CoRR, abs/2306.03509, 2023 b

    2023 arXiv

  22. [30]

    Speak, read and prompt: High-fidelity text-to-speech with minimal supervision

    Kharitonov, E., Vincent, D., Borsos, Z., Marinier, R., Girgin, S., Pietquin, O., Sharifi, M., Tagliasacchi, M., and Zeghidour, N. Speak, read and prompt: High-fidelity text-to-speech with minimal supervision. Trans. Assoc. Comput. Linguistics, 11: 0 1703--1718, 2023

    2023

  23. [31]

    Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis

    Kong, J., Kim, J., and Bae, J. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp.\ 17022--170...

    2020

  24. [32]

    Diffwave: A versatile diffusion model for audio synthesis

    Kong, Z., Ping, W., Huang, J., Zhao, K., and Catanzaro, B. Diffwave: A versatile diffusion model for audio synthesis. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021

    2021

  25. [33]

    Audiogen: Textually guided audio generation

    Kreuk, F., Synnaeve, G., Polyak, A., Singer, U., D \' e fossez, A., Copet, J., Parikh, D., Taigman, Y., and Adi, Y. Audiogen: Textually guided audio generation. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . Ope...

    2023

  26. [34]

    Voicebox: Text-guided multilingual universal speech generation at scale

    Le, M., Vyas, A., Shi, B., Karrer, B., Sari, L., Moritz, R., Williamson, M., Manohar, V., Adi, Y., Mahadeokar, J., and Hsu, W. Voicebox: Text-guided multilingual universal speech generation at scale. In Advances in Neural Information Processing Systems 36: Annual Conference on...

    2023

  27. [35]

    Autoregressive image generation using residual quantization

    Lee, D., Kim, C., Kim, S., Cho, M., and Han, W. Autoregressive image generation using residual quantization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pp.\ 11513--11522. IEEE , 2022

    2022

  28. [36]

    Li, J., Li, D., Xiong, C., and Hoi, S. C. H. BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedi...

    2022

  29. [37]

    Li, J., Li, D., Savarese, S., and Hoi, S. C. H. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 202 of Proceeding...

    2023

  30. [38]

    Autoregressive image generation without vector quantization

    Li, T., Tian, Y., Li, H., Deng, M., and He, K. Autoregressive image generation without vector quantization. CoRR, abs/2406.11838, 2024

    2024 arXiv

  31. [39]

    Evolutionary-scale prediction of atomic level protein structure with a language model

    Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., dos Santos Costa, A., Fazel-Zarandi, M., Sercu, T., Candido, S., and Rives, A. Evolutionary-scale prediction of atomic level protein structure with a language model. bioRx...

    2022 doi

  32. [40]

    P., Wang, W., and Plumbley, M

    Liu, H., Chen, Z., Yuan, Y., Mei, X., Liu, X., Mandic, D. P., Wang, W., and Plumbley, M. D. Audioldm: Text-to-audio generation with latent diffusion models. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 202 of Proce...

    2023

  33. [41]

    Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023 b

    2023

  34. [42]

    Language model can listen while speaking

    Ma, Z., Song, Y., Du, C., Cong, J., Chen, Z., Wang, Y., Wang, Y., and Chen, X. Language model can listen while speaking. CoRR, abs/2408.02622, 2024

    2024 arXiv

  35. [43]

    R., Subramanian, S., Mohr, B

    Madani, A., Krause, B., Greene, E. R., Subramanian, S., Mohr, B. P., Holton, J. M., Olmos, J. L., Xiong, C., Sun, Z. Z., Socher, R., Fraser, J. S., and Naik, N. V. Large language models generate functional protein sequences across diverse families. Nature Biotechnology, pp.\ 1...

    2023

  36. [44]

    J., and Ramanovich, M

    Nachmani, E., Levkovitch, A., Salazar, J., Asawaroengchai, C., Mariooryad, S., Skerry - Ryan, R. J., and Ramanovich, M. T. Lms with a voice: Spoken language modeling beyond speech tokens. CoRR, abs/2305.15255, 2023

    2023 arXiv

  37. [45]

    J., and Ramanovich, M

    Nachmani, E., Levkovitch, A., Hirsch, R., Salazar, J., Asawaroengchai, C., Mariooryad, S., Rivlin, E., Skerry - Ryan, R. J., and Ramanovich, M. T. Spoken question answering and speech continuation using spectrogram-powered LLM . In The Twelfth International Conference on Learn...

    2024

  38. [46]

    A., Kharitonov, E., Copet, J., Adi, Y., Hsu, W., Elkahky, A., Tomasello, P., Algayres, R., Sagot, B., Mohamed, A., and Dupoux, E

    Nguyen, T. A., Kharitonov, E., Copet, J., Adi, Y., Hsu, W., Elkahky, A., Tomasello, P., Algayres, R., Sagot, B., Mohamed, A., and Dupoux, E. Generative spoken dialogue language modeling. Trans. Assoc. Comput. Linguistics, 11: 0 250--266, 2023

    2023

  39. [47]

    A., Muller, B., Yu, B., Costa - juss \` a , M

    Nguyen, T. A., Muller, B., Yu, B., Costa - juss \` a , M. R., Elbayad, M., Popuri, S., Duquenne, P., Algayres, R., Mavlyutov, R., Gat, I., Synnaeve, G., Pino, J., Sagot, B., and Dupoux, E. Spirit-lm: Interleaved spoken and written language model. CoRR, abs/2402.05755, 2024

    2024 arXiv

  40. [48]

    GPT-4 technical report

    OpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023

    2023 arXiv

  41. [49]

    OpenAI. 2024. URL https://openai.com/index/hello-gpt-4o/

    2024

  42. [50]

    S., Constant, N., Raffel, C., and Callison - Burch, C

    Patel, A., Li, B., Rasooli, M. S., Constant, N., Raffel, C., and Callison - Burch, C. Bidirectional language models are also few-shot learners. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023

    2023

  43. [51]

    Efficiently scaling transformer inference

    Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury, J., Heek, J., Xiao, K., Agrawal, S., and Dean, J. Efficiently scaling transformer inference. In Proceedings of the Sixth Conference on Machine Learning and Systems, MLSys 2023, Miami, FL, USA, June 4-8, 2023. mlsys.or...

    2023

  44. [52]

    Efficiently scaling transformer inference

    Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury, J., Heek, J., Xiao, K., Agrawal, S., and Dean, J. Efficiently scaling transformer inference. In Proceedings of the Sixth Conference on Machine Learning and Systems, MLSys 2023, Miami, FL, USA, June 4-8, 2023. mlsys.or...

    2023

  45. [53]

    W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I

    Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on ...

    2021

  46. [54]

    W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I

    Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 202 of Proceedings of Machine ...

    2023

  47. [55]

    The candor corpus: Insights from a large multimodal dataset of naturalistic conversation

    Reece, A., Cooney, G., Bull, P., Chung, C., Dawson, B., Fitzpatrick, C., Glazer, T., Knox, D., Liebscher, A., and Marin, S. The candor corpus: Insights from a large multimodal dataset of naturalistic conversation. Science Advances, 9 0 (13): 0 eadf3197, 2023

    2023

  48. [56]

    K., Asawaroengchai, C., Nguyen, D

    Rubenstein, P. K., Asawaroengchai, C., Nguyen, D. D., Bapna, A., Borsos, Z., de Chaumont Quitry, F., Chen, P., Badawy, D. E., Han, W., Kharitonov, E., Muckenhirn, H., Padfield, D., Qin, J., Rozenberg, D., Sainath, T. N., Schalkwyk, J., Sharifi, M., Ramanovich, M. T., Tagliasac...

    2023 arXiv

  49. [57]

    A., Bekas, C., and Lee, A

    Schwaller, P., Laino, T., Gaudin, T., Bolgar, P., Hunter, C. A., Bekas, C., and Lee, A. A. Molecular transformer: A model for uncertainty-calibrated chemical reaction prediction. ACS Central Science, 5 0 (9): 0 1572--1583, 2019

    2019

  50. [58]

    Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers

    Shen, K., Ju, Z., Tan, X., Liu, E., Leng, Y., He, L., Qin, T., Zhao, S., and Bian, J. Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Aus...

    2024

  51. [59]

    Graphaf: a flow-based autoregressive model for molecular graph generation

    Shi, C., Xu, M., Zhu, Z., Zhang, W., Zhang, M., and Tang, J. Graphaf: a flow-based autoregressive model for molecular graph generation. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020

    2020

  52. [60]

    Vocos: Closing the gap between time-domain and fourier-based neural vocoders for high-quality audio synthesis

    Siuzdak, H. Vocos: Closing the gap between time-domain and fourier-based neural vocoders for high-quality audio synthesis. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024

    2024

  53. [61]

    Su, J., Ahmed, M. H. M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024. doi:10.1016/J.NEUCOM.2023.127063. URL https://doi.org/10.1016/j.neucom.2023.127063

    2024

  54. [62]

    Pandagpt: One model to instruction-follow them all

    Su, Y., Lan, T., Li, H., Xu, J., Wang, Y., and Cai, D. Pandagpt: One model to instruction-follow them all. CoRR, abs/2305.16355, 2023

    2023 arXiv

  55. [63]

    Sutskever, I., Vinyals, O., and Le, Q. V. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pp.\ 3104--3112, 2014

    2014

  56. [64]

    SALMONN: towards generic hearing abilities for large language models

    Tang, C., Yu, W., Sun, G., Chen, X., Tan, T., Li, W., Lu, L., Ma, Z., and Zhang, C. SALMONN: towards generic hearing abilities for large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenRevie...

    2024

  57. [65]

    C., Ture, F., and Lin, J

    Tang, R., Kumar, K., Yang, G., Pandey, A., Mao, Y., Belyaev, V., Emmadi, M., Murray, G. C., Ture, F., and Lin, J. Speechnet: Weakly supervised, end-to-end speech recognition at industrial scale. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Pro...

    2022

  58. [66]

    Chameleon: Mixed-modal early-fusion foundation models

    Team, C. Chameleon: Mixed-modal early-fusion foundation models. CoRR, abs/2405.09818, 2024

    2024 arXiv

  59. [67]

    S., Love, J., et al

    Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivi \`e re, M., Kale, M. S., Love, J., et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024

    2024 arXiv

  60. [68]

    Visual autoregressive modeling: Scalable image generation via next-scale prediction

    Tian, K., Jiang, Y., Yuan, Z., Peng, B., and Wang, L. Visual autoregressive modeling: Scalable image generation via next-scale prediction. CoRR, abs/2404.02905, 2024

    2024 arXiv

  61. [69]

    Neural discrete representation learning

    van den Oord, A., Vinyals, O., and kavukcuoglu, k. Neural discrete representation learning. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associa...

    2017

  62. [70]

    N., Yu, B., Gong, H., and Gollakota, S

    Veluri, B., Peloquin, B. N., Yu, B., Gong, H., and Gollakota, S. Beyond turn-based interfaces: Synchronous llms as full-duplex dialogue agents. arXiv preprint arXiv:2409.15594, 2024

    2024 arXiv

  63. [71]

    Viola: Unified codec language models for speech recognition, synthesis, and translation

    Wang, T., Zhou, L., Zhang, Z., Wu, Y., Liu, S., Gaur, Y., Chen, Z., Li, J., and Wei, F. Viola: Unified codec language models for speech recognition, synthesis, and translation. CoRR, abs/2305.16107, 2023

    2023 arXiv

  64. [72]

    a ckstr \

    Weissenborn, D., T \" a ckstr \" o m, O., and Uszkoreit, J. Scaling autoregressive video models. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020

    2020

  65. [73]

    Speechgen: Unlocking the generative power of speech language models with prompts

    Wu, H., Chang, K., Wu, Y., and Lee, H. Speechgen: Unlocking the generative power of speech language models with prompts. CoRR, abs/2306.02207, 2023. doi:10.48550/ARXIV.2306.02207. URL https://doi.org/10.48550/arXiv.2306.02207

    2023 doi arXiv

  66. [74]

    Next-gpt: Any-to-any multimodal LLM

    Wu, S., Fei, H., Qu, L., Ji, W., and Chua, T. Next-gpt: Any-to-any multimodal LLM . In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024

    2024

  67. [75]

    J., Wang, W., Lin, K

    Xie, J., Mao, W., Bai, Z., Zhang, D. J., Wang, W., Lin, K. Q., Gu, Y., Chen, Z., Yang, Z., and Shou, M. Z. Show-o: One single transformer to unify multimodal understanding and generation, 2024

    2024

  68. [76]

    and Wu, C

    Xie, Z. and Wu, C. Mini-omni: Language models can hear, talk while thinking in streaming, 2024. URL https://arxiv.org/abs/2408.16725

    2024 arXiv

  69. [77]

    Diffsound: Discrete diffusion model for text-to-sound generation

    Yang, D., Yu, J., Wang, H., Wang, W., Weng, C., Zou, Y., and Yu, D. Diffsound: Discrete diffusion model for text-to-sound generation. IEEE ACM Trans. Audio Speech Lang. Process. , 31: 0 1720--1733, 2023

    2023

  70. [78]

    Instructtts: Modelling expressive TTS in discrete latent space with natural language style prompt

    Yang, D., Liu, S., Huang, R., Weng, C., and Meng, H. Instructtts: Modelling expressive TTS in discrete latent space with natural language style prompt. IEEE ACM Trans. Audio Speech Lang. Process. , 32: 0 2913--2925, 2024

    2024

  71. [79]

    L., and Leskovec, J

    You, J., Ying, R., Ren, X., Hamilton, W. L., and Leskovec, J. Graphrnn: Generating realistic graphs with deep auto-regressive models. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsm \" a ssan, Stockholm, Sweden, July 10-15, 2018 ...

    2018

  72. [80]

    Soundstream: An end-to-end neural audio codec

    Zeghidour, N., Luebs, A., Omran, A., Skoglund, J., and Tagliasacchi, M. Soundstream: An end-to-end neural audio codec. IEEE ACM Trans. Audio Speech Lang. Process. , 30: 0 495--507, 2022

    2022

  73. [81]

    Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities

    Zhang, D., Li, S., Zhang, X., Zhan, J., Wang, P., Zhou, Y., and Qiu, X. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023 , p...

    2023

  74. [82]

    S., Haghani, P., Riesa, J., Perng, G., Soltau, H., Strohman, T., Ramabhadran, B., Sainath, T

    Zhang, Y., Han, W., Qin, J., Wang, Y., Bapna, A., Chen, Z., Chen, N., Li, B., Axelrod, V., Wang, G., Meng, Z., Hu, K., Rosenberg, A., Prabhavalkar, R., Park, D. S., Haghani, P., Riesa, J., Perng, G., Soltau, H., Strohman, T., Ramabhadran, B., Sainath, T. N., Moreno, P. J., Chi...

    2023 arXiv

  75. [83]

    Transfusion: Predict the next token and diffuse images with one multi-modal model, 2024

    Zhou, C., Yu, L., Babu, A., Tirumala, K., Yasunaga, M., Shamis, L., Kahn, J., Ma, X., Zettlemoyer, L., and Levy, O. Transfusion: Predict the next token and diffuse images with one multi-modal model, 2024

    2024

  76. [84]

    Mmspeech: Multi-modal multi-task encoder-decoder pre-training for speech recognition

    Zhou, X., Wang, J., Cui, Z., Zhang, S., Yan, Z., Zhou, J., and Zhou, C. Mmspeech: Multi-modal multi-task encoder-decoder pre-training for speech recognition. In 24th Annual Conference of the International Speech Communication Association, Interspeech 2023, Dublin, Ireland, Aug...

    2023

  77. [85]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.