Pith. sign in

REVIEW 3 major objections 6 minor 52 references

Unlocking Speech Instruction Data Potential with Query Rewriting

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Rewriting text instructions with multiple LLMs before text-to-speech raises usable speech instruction data from 72.19% to 93.07% and lifts a speech-language model's ROUGE-L from 52.26 to 63.04.

desk verdict A practical speech-data construction pipeline whose headline usability gain is partly self-referential, but the downstream instruction-tuning improvement is real enough to take seriously. read the letter →

arxiv 2507.08603 v1 pith:2WHMRR5Y submitted 2025-07-11 cs.AI cs.SDeess.AS

Yonghua Hei , Yibo Yan , Shuliang Liu , Huiyu Zhou , Linfeng Zhang , Xuming Hu This is my paper
classification cs.AIcs.SDeess.AS
keywords speechinstructiondataqueryrewritingtext-to-speechlargelanguagemodelsmulti-LLMknowledgefusionembeddingsimilarityusabilityspeech-textalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the main obstacle to building speech instruction datasets by text-to-speech synthesis is not voice quality but the gap between written instructions and what a TTS model can pronounce reliably. It proposes closing that gap by rewriting each instruction with several large language models, synthesizing each candidate, and keeping the rendering whose recognized speech best matches the original text in embedding space. A third stage trains a fused rewriter on successful rewrites to fix the failures. The headline results are a rise in usable data from 72.19% to 93.07% on a multi-speaker TTS setup and a rise in average ROUGE-L of a speech-language model from 52.26 to 63.04 when trained on the rewritten data. If these numbers hold, high-quality speech instruction data can be produced automatically, without human recording or annotation.

What carries the argument

The load-bearing object is the SIM score, defined as the maximum over three ASR models of the average cosine similarity, computed with three embedding models, between the original text and each recognized transcription (Equations 3-4). The paper uses SIM in three places: to choose the best rewritten candidate among the original text and the three LLM rewrites, to define the Pass rate as the fraction of samples scoring above a threshold of 0.9, and to filter the training data for the speech-language model. Around SIM the method builds a multi-agent annotation system (three architecturally different ASR models to recognize, three embedding models to compare) and a knowledge-fusion loop in which a LoRA-tuned Llama-3-8B-Instruct learns from successful rewrites to correct the failures.

What would settle it

Have human listeners transcribe or rate the meaning of every rewritten query that passes the 0.9 threshold and compare the human-match rate against the reported 93% Pass rate; a large shortfall would show the SIM metric is inflating usability.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that zero-shot query rewriting by off-the-shelf language models moves text instructions into a distribution that TTS models can actually speak, and that the best of several rewritten candidates, chosen by a multi-agent similarity check, consistently beats both the original text and standard text normalization. The paper further claims that fusing the rewriting abilities of Llama-3, Phi-3, and Qwen2, by LoRA-training one model on successful rewrites and redoing the failures, handles complex rewrites that no single model manages on its own. It then shows that training a Qwen2-Audio-7B model on this high-quality synthesized speech, aligned to human golden answers instead of LLM-continued answers, is what improves spoken instruction following. The authors present this as a route to constructing large-scale speech instruction datasets without human annotation.

Load-bearing premise

The whole pipeline trusts that the numerical similarity between the original text and what speech recognizers hear is a reliable stand-in for 'the speech really says what the text means', and that trust is checked only on text that has not been rewritten, not on the rewritten text the method actually produces.

Editorial extensions

If this is right

  • On ParlerTTS-Large with multi-speaker synthesis, the framework raises usable speech instruction data from 72.19% to 93.07% and average embedding similarity from 93.06% to 97.98%.
  • A Qwen2-Audio-7B model trained on the rewritten, golden-aligned speech scores 63.04 average ROUGE-L on four benchmarks, versus 52.26 when trained on original-text speech and 39.77 with no training; a filtering threshold of 0.9 beats both lower and higher thresholds.
  • The method generalizes across TTS models: with MeloTTS and MMS-TTS the usability gap to ParlerTTS-Large narrows from 13.95% to 0.9%.
  • Ablations show multi-LLM rewriting outperforms any single LLM, and multi-ASR annotation lowers average WER from about 10% to 8.36%.
  • Human evaluation on hard samples (original SIM below 50) finds far more rewritten utterances judged linguistically consistent with the original text than text-normalized or original versions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same auto-curation loop could be pointed at other TTS models, domains, or languages, but the paper only demonstrates it on seven short-question QA datasets and three English TTS models, so transfer is an open question.
  • Because the filter keeps the rewrite with the highest recognizer similarity, the final dataset is tilted toward utterances that ASR models find easy; a reader should ask whether that ease is correlated with answerability of the downstream question, not just with pronounceability.
  • The knowledge-fusion step is effectively distillation of rewriting preferences from three teacher LLMs into one small tuned model; a natural test is whether the advantage survives when the teachers are swapped for stronger or weaker models.
  • The cost comparison implies the pipeline can build speech instruction data for under a tenth of the labor cost of human collection, but that conclusion depends on the SIM metric being a valid quality gate, which the paper checks only indirectly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes a fully automatic pipeline for constructing speech instruction data: multiple LLMs (Llama-3, Phi-3, Qwen2) rewrite a text instruction, several TTS systems synthesize speech for the original and rewritten texts, three ASR models transcribe each utterance, and an embedding-based similarity score q (Eqs. 3-5) selects the best candidate. A knowledge-fusion stage trains an LLM on successfully rewritten pairs to repair failed rewrites. The authors report that the data usability Pass rate (fraction of samples with q > 0.9) rises from 72.19% to 93.07% for multi-speaker ParlerTTS-Large (Table 2), and that fine-tuning Qwen2-Audio-7B on rewritten, golden-aligned speech improves average ROUGE-L on DROP, Quoref, ROPES, and NarrativeQA from 52.26 to 63.04 (Table 3). The paper also includes a small human evaluation (Table 10) and a cost analysis (Table 8).

Significance. If the results hold, the paper offers a practical, low-cost route to large-scale speech instruction data without human annotation, which is a relevant problem for end-to-end spoken language models. The strongest part of the evidence is the downstream LSLM fine-tuning experiment: the ROUGE-L evaluation in Table 3 is independent of the SIM metric, and the improvement over both the no-training baseline and the original-text baseline is meaningful. The multi-ASR/multi-embedding ensemble and the explicit knowledge-fusion component are reasonable design choices, and the paper is transparent about experimental settings. However, the headline usability claim depends entirely on the SIM metric, which is used both as the selection oracle and as the evaluation metric; that circularity is the main load-bearing weakness and needs to be addressed before the central claim is fully convincing.

major comments (3)
  1. [§4.2, §5.2, Table 2] The headline Pass-rate improvement is self-referential. Equation (5) defines q(A,E|c,co) for each candidate text c, and Section 4.2 selects the optimal candidate by maximizing this same function; the Pass rate in Section 5.2 is then the proportion of chosen candidates with q > α. Even if rewriting had no true effect on speech quality, selecting the best of four candidates by q would by construction raise the fraction of samples above a fixed threshold. The validation in Table 1 does not close this gap: it tests SIM only for non-rewritten text, uses a weak consistency definition (the SIM-selected ASR transcript has the lowest WER, not that SIM scores are calibrated), and the 'Satisfying Equation 1 (WER=0)' subset is a narrow, favorable selection. The human evaluation in Table 10 samples only utterances with SIM < 50 under the original method, so it does not test the α = 0.9 boundary used for Pass and for LSLM data filtering. To make the 72%→93% claim load-bearing, the authors should validate q on rewritten speech (for example, human annotation of a random sample straddling the threshold, or an independent WER/meaning-equivalence pass rate) and should report a selection-corrected baseline, such as the best-of-k Pass rate when no rewriting is used.
  2. [Table 3] The LSLM fine-tuning results are reported as single numbers with no error bars, significance tests, or number of training runs. The key comparisons—Original 52.26 vs. Ours 60.50 vs. threshold variants 59.55/63.04/60.44—are differences that could be within run-to-run noise in a single-seed fine-tuning experiment. Since the paper uses these results to claim both that rewritten data helps and that t=0.90 is the optimal threshold, the authors should report mean and standard deviation over at least three seeds or provide a significance test. Without this, the downstream evidence, though independent of SIM, is not yet quantitatively decisive.
  3. [Appendix B.2, Table 9] The downstream validation of the SIM metric in Appendix B.2 uses the best ASR transcript as a text query to Llama-3-8B-Instruct; it does not evaluate end-to-end speech instruction following through an LSLM. This is a useful diagnostic, but it is not equivalent to the Table 3 scenario, and the claim of 'consistent improvement on downstream tasks' is not supported by the numbers for Quoref (73.56→74.64) and ROPES (16.34→16.68), which are very small. The authors should either temper this claim or report significance/effect sizes for these text-proxy results.
minor comments (6)
  1. [Abstract and §1] The abstract and conclusion state the usability increase as 'from 72% to 93%', while Section 1 says 'from 71% to 93%'; please reconcile the numbers.
  2. [Table 2] In the MMS-TTS rows, the PASS values for SQUAD2.0 (96.07 for Ours w/o KF and 96.58 for Ours) are far above the corresponding values for all other datasets and above the row averages; please verify that these entries are not typographical errors.
  3. [§4.1-4.2] Notation is used inconsistently: Eq. (2) writes q = F(¯c,c), while Eq. (5) writes q(A,E|c,co) with a different argument structure; please clarify the relationship between F and q and define the candidate-selection rule explicitly.
  4. [Appendix A.2] The training objective for LSLMs includes a text context d, but the main text does not specify how d is obtained or used during evaluation; please clarify whether d is always the associated document and how it is handled for datasets without context.
  5. [Table 13] In the third example, 'Where dies Ella’s wealthy neighbor’s wife return from?' appears to contain a typo ('dies' for 'does'); if this is from the original dataset, please mark it as such, as it affects the interpretation of the ASR/SIM inconsistency example.
  6. [Appendix D] The paper provides licenses for third-party assets but does not give a repository URL for the authors' own code and prompts; please include a link or state that code will be released.

Circularity Check

2 steps flagged · score 6.0 of 10

The headline 72%-to-93% data-usability gain is measured by the same q function used to select the optimal rewritten candidate, so a substantial part of the reported Pass improvement is built into the selection rule.

  1. self definitional [Section 4.2, Eq. (5); Section 5.2 'Pass' definition; Table 2]
    "Following the evaluation and validation methods mentioned in Section 4.1, we can obtain the quality of every synthesized speech through q(A, E|c, co) = max_j(F(co, ¯cj)), c∈ C. ... Then we can obtain the optimal synthesized speechs and the text ˆc ∈ C which is the optimal input into TTS. ... Pass: The proportion of speech in the dataset with a quality higher than α = 0.9, calculated according to Equation (5)."

    Equation (5) is used both as the candidate-selection oracle and as the definition of the headline Pass metric. Because the candidate set C always contains the original text, the selected candidate's q is at least as large as the original candidate's q for every sample; the max over candidates therefore guarantees a non-negative Pass and SIM shift in the direction reported. The 72.19-to-93.07 Pass increase in Table 2 is consequently not an independent measurement of 'usability' after rewriting; it is the value of the objective function after optimizing that same objective. The downstream ROUGE-L evaluation in Table 3 and the human evaluation in Table 10 provide separate evidence, but the headline Pass claim itself is self-referential.

  2. fitted input called prediction [Section 4.3 'Knowledge Fusion'; Section 5.3 'Synthetic data quality']
    "we consider the sample pairs < ci, ˆci >, i∈ [0, n− 1] that satisfy q(A, E|ˆci, ci) > αand q(A, E|ci, ci) < α as successfully rewritten samples for knowledge fusion training, and the sample pairs < c i, ˆci >, i ∈ [0, n − 1] that satisfy q(A, E|ˆci, ci) < α as the samples with failed rewrites, α is the hyperparameters used to control data quality, where q(A, E|ci, ci) fellow equation (5). In this paper, we set α = 0.9."

    The knowledge-fusion model is trained on samples labeled 'successful' by the same q(A,E) threshold that later defines Pass and the LSLM data-filtering threshold. Its outputs are then evaluated and curated with that same q, so the final 'Ours' Pass number in Table 2 is produced by applying the same oracle that created the training labels and selected the candidates. The KF contribution to the headline gain is therefore not externally validated; the only independent checks are the ROUGE-L downstream results and the limited human evaluation, neither of which uses q as the outcome metric.

full rationale

The core circularity is that q(A,E) from Eq. (5) plays three roles: it selects the best rewritten candidate, it defines the Pass rate, and it labels knowledge-fusion training successes. Since the selected q is a maximum over candidates that includes the original text, the reported improvement in Table 2 is partly enforced by the selection rule rather than discovered by evaluation. The paper does provide independent grounding: Table 3 evaluates trained LSLMs with ROUGE-L on external QA benchmarks, and Table 10 includes human annotation of rewritten speech. Those results support the method's value even if the exact Pass gain is inflated by the selection/evaluation overlap. The Table 1 WER-consistency check is not itself circular, but it validates SIM only for non-rewritten speech, and Table 10 samples only known low-SIM failures, so the α=0.9 threshold is not independently calibrated for rewritten candidates; these are validity gaps rather than additional reduction steps. No load-bearing self-citation chain or imported uniqueness theorem is present, so the score is driven by the metric-conflation issue, not by citation practices.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper's central claim depends on domain assumptions about ASR reliability, embedding-similarity validity, and meaning-preserving rewrites, plus two hand-chosen quality thresholds (alpha=0.9, t=0.90). No new physical or mathematical entities are introduced. The free parameters are not fitted to a target result in a hidden way, but they do control the reported Pass numbers.

free parameters (3)
  • alpha (Pass threshold) = 0.9
    Defined in Section 4.3 as the data quality threshold: samples with q(A,E) > 0.9 count as Pass and are used for knowledge fusion training. The value is chosen by hand, not derived.
  • training threshold t = 0.90
    Section 5.1 sets t as the minimum SIM quality for samples used in LSLM fine-tuning; Table 3 shows t=0.90 gives the best ROUGE-L, so it is selected empirically.
  • speaker description count = 192
    Section 4.2 generates 192 GPT-4 speaker descriptions for style diversity; the count is arbitrary and affects the style distribution but not the core rewriting claim.
assumptions (4)
  • domain assumption The best-of-three ASR transcription is a reliable proxy for the linguistic content of synthesized speech.
    Section 4.1, equation (4) scores speech quality by the maximum similarity over three ASR models; the filtering and Pass metrics depend on this.
  • domain assumption Embedding-space cosine similarity between original text and transcription measures linguistic equivalence well enough to guide data selection and evaluation.
    Section 4.1, equation (3) defines SIM; Section 5.2 validates it against WER only for non-rewrite scenarios (Table 1), but the main pipeline uses it on rewritten text.
  • domain assumption LLM rewriting preserves the meaning of the original instruction closely enough that downstream QA performance is not harmed.
    Section 4.2 and Figure 4 assume rewriting is meaning-preserving; the downstream LSLM fine-tuning (Table 3) tests this indirectly, but there is no direct semantic equivalence check on the rewritten text.
  • domain assumption Short QA questions, under 100 words each, are representative of the speech instruction distribution relevant to LSLMs.
    Section 5.1 chooses short-question QA datasets; the Limitations section explicitly restricts the claim to texts under 100 words.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Unlocking Speech Instruction Data Potential with Query Rewriting." pith.science (2026). https://pith.science/paper/2WHMRR5Y

@misc{pith2026250708603,
  author       = {Pith},
  title        = {Pith review of: Unlocking Speech Instruction Data Potential with Query Rewriting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2WHMRR5Y}},
  note         = {Machine review of arXiv:2507.08603}
}
read the original abstract

End-to-end Large Speech Language Models~(\textbf{LSLMs}) demonstrate strong potential in response latency and speech comprehension capabilities, showcasing general intelligence across speech understanding tasks. However, the ability to follow speech instructions has not been fully realized due to the lack of datasets and heavily biased training tasks. Leveraging the rich ASR datasets, previous approaches have used Large Language Models~(\textbf{LLMs}) to continue the linguistic information of speech to construct speech instruction datasets. Yet, due to the gap between LLM-generated results and real human responses, the continuation methods further amplify these shortcomings. Given the high costs of collecting and annotating speech instruction datasets by humans, using speech synthesis to construct large-scale speech instruction datasets has become a balanced and robust alternative. Although modern Text-To-Speech~(\textbf{TTS}) models have achieved near-human-level synthesis quality, it is challenging to appropriately convert out-of-distribution text instruction to speech due to the limitations of the training data distribution in TTS models. To address this issue, we propose a query rewriting framework with multi-LLM knowledge fusion, employing multiple agents to annotate and validate the synthesized speech, making it possible to construct high-quality speech instruction datasets without relying on human annotation. Experiments show that this method can transform text instructions into distributions more suitable for TTS models for speech synthesis through zero-shot rewriting, increasing data usability from 72\% to 93\%. It also demonstrates unique advantages in rewriting tasks that require complex knowledge and context-related abilities.

Figures

Figures reproduced from arXiv: 2507.08603 by the authors.

Figure 1
Figure 1. Some recognition errors in ASR models. Sentence Pattern Error indicates that the ASR model failed to provide appropriate punctuation based on the user’s intent. In the given examples,the original sentence expresses a question, while the ASR-recognized sentence, lacking proper punctuation, conveys a tone of disdain or assertion. Parler-TTS (Lacombe et al., 2024) to synthesize speech. Speech-text alignment training in… view at source ↗
Figure 2
Figure 2. The structure of Question Rewriting with Multi-LLM. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The structure of Knowledge Fusion. where c¯j is the speech recognition result of s ∈ S by ASR model Aj ∈ A (4.1). Then we can obtain the optimal synthesized speech s and the text cˆ ∈ C which is the optimal input into TTS. 4.3 Knowledge Fusion for Challenging Query Rewriting The performance of the models on challenging zero-shot tasks shows a degree of orthogonality due to differences in their knowledge. However, th… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Examples of successfully rewritten queries. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

52 extracted references · 12 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  4. [4]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  5. [5]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  6. [6]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    work page

  7. [7]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matt...

    arXiv 2024

  8. [8]

    Ardila, M

    R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers, and G. Weber. Common voice: A massively-multilingual speech corpus. In Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pp.\ 4211--4215, 2020

    work page 2020

Show all 52 references
  1. [9]

    Unveiling the flaws: Exploring imperfections in synthetic data and mitigation strategies for large language models, 2024

    Jie Chen, Yupeng Zhang, Bingning Wang, Wayne Xin Zhao, Ji-Rong Wen, and Weipeng Chen. Unveiling the flaws: Exploring imperfections in synthetic data and mitigation strategies for large language models, 2024. URL https://arxiv.org/abs/2406.12397

    2024 arXiv

  2. [10]

    Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models, 2023

    Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models, 2023. URL https://arxiv.org/abs/2311.07919

    2023 arXiv

  3. [11]

    Qwen2-audio technical report, 2024

    Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen2-audio technical report, 2024. URL https://arxiv.org/abs/2407.10759

    2024 arXiv

  4. [12]

    Speechverse: A large-scale generalizable audio language model, 2024

    Nilaksh Das, Saket Dingliwal, Srikanth Ronanki, Rohit Paturi, Zhaocheng Huang, Prashant Mathur, Jie Yuan, Dhanush Bekal, Xing Niu, Sai Muralidhar Jayanthi, Xilai Li, Karel Mundnich, Monica Sunkara, Sundararajan Srinivasan, Kyu J Han, and Katrin Kirchhoff. Speechverse: A large-...

    2024 arXiv

  5. [13]

    Liu, Ana Marasovi \'c , Noah A

    Pradeep Dasigi, Nelson F. Liu, Ana Marasovi \'c , Noah A. Smith, and Matt Gardner. Q uoref: A reading comprehension dataset with questions requiring coreferential reasoning. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on ...

    2019

  6. [14]

    Smith, Hannaneh Hajishirzi, Ross Girshick, Ali Farhadi, and Aniruddha Kembhavi

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch,...

    2024 arXiv

  7. [15]

    Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs, 2019

    Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs, 2019. URL https://arxiv.org/abs/1903.00161

    2019 arXiv

  8. [16]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zha...

    2024 arXiv

  9. [17]

    Llama-omni: Seamless speech interaction with large language models, 2024

    Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. Llama-omni: Seamless speech interaction with large language models, 2024. URL https://arxiv.org/abs/2409.06666

    2024 arXiv

  10. [18]

    Audiochatllama: Towards general-purpose speech abilities for llms, 2024

    Yassir Fathullah, Chunyang Wu, Egor Lakomkin, Ke Li, Junteng Jia, Yuan Shangguan, Jay Mahadeokar, Ozlem Kalinli, Christian Fuegen, and Mike Seltzer. Audiochatllama: Towards general-purpose speech abilities for llms, 2024. URL https://arxiv.org/abs/2311.06753

    2024 arXiv

  11. [19]

    Liu, Hongyin Luo, Leonid Karlinsky, and James Glass

    Yuan Gong, Alexander H. Liu, Hongyin Luo, Leonid Karlinsky, and James Glass. Joint audio and speech understanding, 2023. URL https://arxiv.org/abs/2309.14405

    2023 arXiv

  12. [20]

    Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio

    Guanbo Wang Jiayu Du Wei-Qiang Zhang Chao Weng Dan Su Daniel Povey Jan Trmal Junbo Zhang Mingjie Jin Sanjeev Khudanpur Shinji Watanabe Shuaijiang Zhao Wei Zou Xiangang Li Xuchen Yao Yongqing Wang Yujun Wang Zhao You Zhiyong Yan Guoguo Chen, Shuzhou Chai. Gigaspeech: An evolvin...

    2021

  13. [21]

    Text normalization challenge - english language

    Addison Howard, RichardSproat, wellformedness, and Will Cukierski. Text normalization challenge - english language. https://kaggle.com/competitions/text-normalization-challenge-english-language, 2017. Kaggle

    2017

  14. [22]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https://arxiv.org/abs/2106.09685

    2021 arXiv

  15. [23]

    The narrativeqa reading comprehension challenge, 2017

    Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The narrativeqa reading comprehension challenge, 2017. URL https://arxiv.org/abs/1712.07040

    2017 arXiv

  16. [24]

    Parler-tts

    Yoach Lacombe, Vaibhav Srivastav, and Sanchit Gandhi. Parler-tts. https://github.com/huggingface/parler-tts, 2024

    2024

  17. [25]

    Building and better understanding vision-language models: insights and future directions., 2024

    Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon. Building and better understanding vision-language models: insights and future directions., 2024

    2024

  18. [26]

    Voicebox: Text-guided multilingual universal speech generation at scale, 2023

    Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, and Wei-Ning Hsu. Voicebox: Text-guided multilingual universal speech generation at scale, 2023. URL https://arxiv.org/abs/2306.15687

    2023 arXiv

  19. [27]

    ROUGE : A package for automatic evaluation of summaries

    Chin-Yew Lin. ROUGE : A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp.\ 74--81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https://aclanthology.org/W04-1013

    2004

  20. [28]

    Reasoning over paragraph effects in situations, 2019

    Kevin Lin, Oyvind Tafjord, Peter Clark, and Matt Gardner. Reasoning over paragraph effects in situations, 2019. URL https://arxiv.org/abs/1908.05852

    2019 arXiv

  21. [29]

    Lawrence Zitnick, and Piotr Dollár

    Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015. URL https://arxiv.org/abs/1405.0312

    2015 arXiv

  22. [30]

    Towards emotional support dialog systems, 2021

    Siyang Liu, Chujie Zheng, Orianna Demasi, Sahand Sabour, Yu Li, Zhou Yu, Yong Jiang, and Minlie Huang. Towards emotional support dialog systems, 2021. URL https://arxiv.org/abs/2106.01144

    2021 arXiv

  23. [31]

    Natural language guidance of high-fidelity text-to-speech with synthetic annotations, 2024

    Dan Lyth and Simon King. Natural language guidance of high-fidelity text-to-speech with synthetic annotations, 2024. URL https://arxiv.org/abs/2402.01912

    2024 arXiv

  24. [32]

    Mteb: Massive text embedding benchmark, 2023

    Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. Mteb: Massive text embedding benchmark, 2023. URL https://arxiv.org/abs/2210.07316

    2023 arXiv

  25. [33]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

    2024 arXiv

  26. [34]

    Text normalization

    piAI. Text normalization. https://www.kaggle.com/code/econdata/text-normalization, 2017. Accessed: 2024-11-27

    2017

  27. [35]

    Mls: A large-scale multilingual dataset for speech research

    Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. Mls: A large-scale multilingual dataset for speech research. ArXiv, abs/2012.03411, 2020

    2012 arXiv

  28. [36]

    Scaling speech technology to 1,000+ languages

    Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, Alexei Baevski, Yossi Adi, Xiaohui Zhang, Wei-Ning Hsu, Alexis Conneau, and Michael Auli. Scaling speech technology to 1,000+ langua...

    2023

  29. [37]

    Robust speech recognition via large-scale weak supervision, 2022

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision, 2022. URL https://arxiv.org/abs/2212.04356

    2022 arXiv

  30. [38]

    Squad: 100,000+ questions for machine comprehension of text, 2016

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text, 2016. URL https://arxiv.org/abs/1606.05250

    2016 arXiv

  31. [39]

    Know what you don't know: Unanswerable questions for squad, 2018

    Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad, 2018. URL https://arxiv.org/abs/1806.03822

    2018 arXiv

  32. [40]

    How bad is training on synthetic data? a statistical analysis of language model collapse, 2024

    Mohamed El Amine Seddik, Suei-Wen Chen, Soufiane Hayou, Pierre Youssef, and Merouane Debbah. How bad is training on synthetic data? a statistical analysis of language model collapse, 2024. URL https://arxiv.org/abs/2404.05090

    2024 arXiv

  33. [41]

    Llasm: Large language and speech model, 2023

    Yu Shu, Siwei Dong, Guangyao Chen, Wenhao Huang, Ruihua Zhang, Daochen Shi, Qiqi Xiang, and Yemin Shi. Llasm: Large language and speech model, 2023. URL https://arxiv.org/abs/2308.15930

    2023 arXiv

  34. [42]

    Open automatic speech recognition leaderboard

    Vaibhav Srivastav, Somshubra Majumdar, Nithin Koluguri, Adel Moumen, Sanchit Gandhi, et al. Open automatic speech recognition leaderboard. https://huggingface.co/spaces/hf-audio/open_asr_leaderboard, 2023

    2023

  35. [43]

    Salmonn: Towards generic hearing abilities for large language models, 2024

    Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Salmonn: Towards generic hearing abilities for large language models, 2024. URL https://arxiv.org/abs/2310.13289

    2024 arXiv

  36. [44]

    Fusechat: Knowledge fusion of chat models, 2024

    Fanqi Wan, Longguang Zhong, Ziyi Yang, Ruijun Chen, and Xiaojun Quan. Fusechat: Knowledge fusion of chat models, 2024. URL https://arxiv.org/abs/2408.07990

    2024 arXiv

  37. [45]

    Neural codec language models are zero-shot text to speech synthesizers, 2023

    Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. Neural codec language models are zero-shot text to speech synthesizers, 2023. URL https://arxiv.org/abs/2301.02111

    2023 arXiv

  38. [46]

    Globe: A high-quality english corpus with global accents for zero-shot speaker adaptive text-to-speech, 2024

    Wenbin Wang, Yang Song, and Sanjay Jha. Globe: A high-quality english corpus with global accents for zero-shot speaker adaptive text-to-speech, 2024

    2024

  39. [47]

    Qwen2 technical report, 2024 a

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze...

    2024 arXiv

  40. [48]

    Air-bench: Benchmarking large audio-language models via generative comprehension, 2024 b

    Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, and Jingren Zhou. Air-bench: Benchmarking large audio-language models via generative comprehension, 2024 b . URL https://arxiv.org/abs/2402.07729

    2024 arXiv

  41. [49]

    Boosting conversational question answering with fine-grained retrieval-augmentation and self-check

    Linhao Ye, Zhikai Lei, Jia-Peng Yin, Qin Chen, Jie Zhou, and Liang He. Boosting conversational question answering with fine-grained retrieval-augmentation and self-check. ArXiv, abs/2403.18243, 2024. URL https://api.semanticscholar.org/CorpusID:268724200

    2024 arXiv

  42. [50]

    Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities, 2023

    Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities, 2023. URL https://arxiv.org/abs/2305.11000

    2023 arXiv

  43. [51]

    Melotts: High-quality multi-lingual multi-accent text-to-speech, 2023

    Wenliang Zhao, Xumin Yu, and Zengyi Qin. Melotts: High-quality multi-lingual multi-accent text-to-speech, 2023. URL https://github.com/myshell-ai/MeloTTS

    2023

  44. [52]

    Tat-qa: A question answering benchmark on a hybrid of tabular and textual content in finance, 2021

    Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. Tat-qa: A question answering benchmark on a hybrid of tabular and textual content in finance, 2021. URL https://arxiv.org/abs/2105.07624

    2021 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.