Pith. sign in

REVIEW 5 major objections 6 minor 42 references

Leveraging LLM and Self-Supervised Training Models for Speech Recognition in Chinese Dialects: A Comparative Analysis

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A self-supervised speech encoder paired with a 0.5-billion-parameter LLM achieves the best reported error rate on the Kespeech Chinese dialect benchmark.

desk verdict Useful ablation recipe for dialect ASR, but the SOTA claim rests on an unreported Kespeech split and the CTC-finetuning analysis contradicts its own table. read the letter →

arxiv 2505.21138 v2 pith:QFZJ57A4 submitted 2025-05-27 cs.CL cs.SDeess.AS

Tianyi Xu , Hongjie Chen , Wang Qing , Lv Hang , Jian Kang , Li Jie , Zhennan Lin , Yongxiang Li
show 1 more author
Xie Lei
This is my paper · ORCID
classification cs.CLcs.SDeess.AS
keywords Chinesedialectspeechrecognitionself-supervisedpre-trainingData2Vec2largelanguagemodelASRdecodermulti-stagefine-tuningLoRAKespeechbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a self-supervised speech encoder paired with a large language model (LLM) decoder can outperform existing systems on Chinese dialects, where labeled data is scarce. Its recipe pre-trains a Data2Vec2 encoder on 300,000 hours of unlabeled dialect and accented speech, aligns it on 40,000 supervised hours, and fine-tunes with a four-stage procedure. On its evaluation sets, the method reaches a Kespeech character error rate of 6.48%, compared with 7.52% for the previous best system, while using a relatively small LLM and a low input frame rate. If the numbers hold, this is a practical recipe for high-accuracy dialect ASR without a large decoder.

What carries the argument

The central mechanism is a four-stage fine-tuning schedule applied to an audio-text cross-modal model: first train only the projector, then train the speech encoder, then freeze both and apply LoRA to the LLM, and finally unfreeze everything for joint optimization. The encoder is a Data2Vec2 model pre-trained on 300,000 hours of unlabeled dialect speech and operating at 25 Hz, and the projector downsamples its output by another factor of four, giving a 6.25 Hz frame rate at the LLM input; this low rate is what keeps inference affordable while preserving accuracy.

What would settle it

Run the released model on the exact Kespeech test split used by the previous best system and recompute the CER under identical decoding settings; if the result is 7.52% or higher, the headline SOTA claim fails.

Watch

Extended reading notes

Core claim

The paper claims that a Data2Vec2 speech encoder pre-trained on large amounts of unlabeled dialect data, coupled with a compact LLM decoder and a carefully staged fine-tuning schedule, yields state-of-the-art character error rates on multiple Chinese dialect and accent test sets, including Kespeech at 6.48% versus 7.52% for the prior best system. It also reports three supporting findings: fully connected projectors beat convolutional, transformer, and Q-Former projectors in the early training stage; CTC fine-tuning of the encoder before LLM alignment hurts rather than helps; and adding a fourth fine-tuning stage, in which the entire model is unfrozen, improves over the conventional three-stage approach.

Load-bearing premise

The internal dialect test sets are accurate and representative, and the Kespeech evaluation uses the same test split as the prior best system; every headline CER depends on those comparisons being fair.

Editorial extensions

If this is right

  • The 0.5B-parameter LLM already beats the prior best Kespeech result, so the recipe does not require a large decoder.
  • Larger LLMs within the same recipe continue to lower error rates, so scaling the decoder is a clear further lever.
  • The 6.25 Hz operating frame rate makes the approach practical for deployment compared with higher-frame-rate speech LLMs.
  • Full fine-tuning of the LLM gives a significant gain over LoRA for the smallest model, indicating that LoRA capacity can be a bottleneck.
  • The four-stage schedule yields consistent improvements over three-stage training across dialect test sets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The finding that CTC fine-tuning hurts may generalize to other self-supervised encoders paired with LLM decoders, suggesting that task-specific supervised adaptation can conflict with semantic alignment learned during LLM fine-tuning.
  • Because projector differences shrink once the encoder and LLM are unfrozen, projector architecture may be a minor design choice in the long run, and its main role is managing early-stage convergence and frame rate.
  • If the internal dialect test sets are representative, the same recipe could transfer to other under-resourced languages, but this depends on public validation against unbiased test splits.
  • The Kespeech comparison would be more decisive if the exact test split and decoding settings were released; a reproducibility check on that split is the key next step.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. This paper proposes an ASR system for Chinese dialects that couples a Data2Vec2 speech encoder pre-trained on about 300,000 hours of unlabeled speech, one of four projection layers, and a Qwen2/Qwen2.5 LLM decoder (0.5B-7B) trained on 40,000 hours of supervised multi-dialect data in a four-stage schedule. It reports ablations of projector architecture, frame rate, encoder fine-tuning, and LLM size, and claims state-of-the-art results on the public Kespeech benchmark (6.48% CER vs. 7.52% for MMGER) as well as strong performance on four internal dialect test sets. The authors state that training, inference, evaluation scripts, and pre-trained models will be released.

Significance. If the main result is correct, the paper is a useful empirical data point: a relatively small LLM decoder on a 6.25 Hz stream from a self-supervised encoder can beat a cascade error-correction method on accented Mandarin, and the four-stage recipe plus the projector comparison would be of practical value to the community. The strengths are the unusually large (300k-hour) self-supervised pretraining, the 40k-hour supervised alignment data, the systematic variation of projectors and LLM sizes, and the inclusion of public baselines. The weakness is that the headline comparisons rest on measurements whose test conditions are not disclosed: four internal test sets and an unspecified Kespeech split. Because the central SOTA claim is not currently checkable, and because at least one ablation conclusion is contradicted by the paper's own Table 1, the paper needs major revisions before the results can be assessed.

major comments (5)
  1. [Section 3.1 and Table 3] The Kespeech comparison that supports the SOTA claim is not reproducible. The paper never states which Kespeech split (official test set, a custom split, or a filtered subset) is used, and the authors themselves note that the accent labels in Kespeech are noisy. Since MMGER's reported 7.52% was obtained under conditions that may differ (e.g., a different split or scoring tool), the 1.04-percentage-point gap cannot be attributed to the proposed method. Please report the exact split, subdialect composition, filtering rules, and scoring configuration, and make the evaluation scripts available.
  2. [Section 3.1 and Table 3] The four internal dialect test sets are described only as 'approximately 2,000 audio samples' that are 'internally annotated,' with no details on speakers, recording conditions, transcriptions, overlap with the training data, or a release plan. All the non-Kespeech CER numbers in Table 3 are hence unverifiable and cannot be compared with future work. A general promise to open-source models and scripts is insufficient if the test sets themselves remain private; please release the test sets or state clearly that the numbers are not independently reproducible.
  3. [Section 4.3 and Table 1] The claim that CTC-finetuned encoders 'perform worse across all four projection layer types' is contradicted by Table 1. The finetuned encoder improves on Test-net for the Linear projector (14.09 vs. 16.47) and improves on both Test-net and Test-meeting for the Transformer projector (14.34 vs. 17.32 and 11.32 vs. 12.37). This means either the conclusion or the table is wrong; please correct the inconsistency and re-state the resulting conclusion about CTC fine-tuning.
  4. [Section 4.4 and Table 4] The multi-stage analysis is not auditable as written. Table 4 does not state which LLM and fine-tuning method are used, and the Stage-4 rows do not match any row in Table 3 (e.g., Stage-4 Conv1d reports 25.98/20.19/18.56 for Henan/Shanghai/Hunan, while the Qwen2 0.5B Conv1D Full row in Table 3 reports 19.97/14.58/11.73). Moreover, the text quotes 31.46/27.87 as the worst first-stage configuration, but that number is the CTC-finetuned Conv1d row of Table 1, not the worst pretrained configuration (18.19/16.64). Please reconcile the tables and specify the exact configurations and the reported quantity for every row.
  5. [Section 4.5 and Table 3] There is no matched supervised ASR baseline trained on the same 40,000-hour set, so the contribution of the LLM decoder is not isolated. The gains over Whisper-Large-V3 and Qwen2-Audio may reflect the additional supervised data, the 300k-hour encoder, or the four-stage schedule rather than the LLM-decoder paradigm. Please add a strong non-LLM ASR baseline (e.g., CTC or AED model with the same encoder and training data) to Table 3.
minor comments (6)
  1. [Section 3.1] The word 'Chongqing' appears twice in the list of training dialects; delete the duplicate.
  2. [Tables 1 and 2] Tables 1 and 2 use inconsistent column orders (Test-net/Test-meeting vs. Test-meeting/Test-net); make the order uniform across all tables.
  3. [Section 4.5] The sentence 'even the smallest model, the 0.5B LLM, achieved competitive results, surpassing ... MMGER' is only supported by the full-finetuned 0.5B row (7.20 vs. 7.52); the LoRA rows at 6.25/12.5 Hz report 7.85 and 7.64, which do not surpass MMGER. Please rephrase to match Table 3.
  4. [Sections 4.1 and 4.3] The stated 1,000,000-step first-stage runs are difficult to reconcile with an effective batch size of 100 and a 40,000-hour supervised set; please provide the actual number of update steps or correct the typo.
  5. [Section 2, Eq. (5)] Equation (5) uses the function name 'Regulation' without a definition; this is presumably concatenation or interleaving and should be stated.
  6. [Table 4] The caption says 'improvement in CER' but the entries appear to be absolute CER values, with one value above 100 (169.03 for Conv1d/Shanghai in Stage 1); define the quantity, the baseline, and the model configuration for every row.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the SOTA and ablation claims are benchmarked against external baselines, and the only self-citation (TeleSpeechPT encoder) is non-load-bearing and externally published.

full rationale

The paper's derivation is empirical rather than definitional. Section 3.3 defines a four-stage fine-tuning recipe; Section 4 reports CER on external/public test sets (Kespeech, Test-net, Test-meeting) and internal dialect test sets. The headline Kespeech claim (Section 4.5: 'surpassing the current SOTA MMGER [18] in Kespeech', 6.48 vs 7.52) is an external comparison, not a fitted parameter renamed as a prediction; no equation in Sections 2-4 defines any reported CER in terms of an input it is supposed to predict. The ablations over projectors, sampling rates, encoder finetuning, and LLM sizes (Tables 1-4) are self-contained experiments whose conclusions follow from the measured numbers, with the exception of an internal inconsistency in Section 4.3's verbal claim ('ASR-finetuned encoders perform worse across all four projection layer types') versus Table 1, where finetuned Linear, Transformer, and Q-Former improve Test-net; this is a correctness/consistency issue, not circularity. The only self-citation is [29] TeleSpeechPT in Section 3.2.2 ('For the current study, we use the TeleSpeechPT variant of Data2vec2'), whose authors overlap with this paper. The citation motivates the encoder choice, but the SOTA result is benchmarked externally and would stand or fall on the measured CERs, so the citation is not load-bearing in a derivation sense and is externally published (NCMMSC 2024). Reproducibility/validity limitations are flagged as non-circularity concerns: Kespeech split is not specified in Section 3.1, internal dialect test sets ('approximately 2,000 audio samples', 'internally annotated') are not released and come from the same dialect families as training data, and no data-contamination analysis is provided. These concerns affect falsifiability, not circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper's results depend on the quality and composition of the pre-trained encoder, the supervised dataset, and the test sets. No closed-form derivation is involved; the main analytical claims are empirical comparisons. The ledger therefore captures the hand-chosen hyperparameters and unstated assumptions about data integrity rather than fitted constants in a mathematical derivation.

free parameters (4)
  • projector downsampling rate = 4 (6.25 Hz at LLM input)
    Chosen post hoc as the best balance between accuracy and compute (Section 4.2); the paper frames it as a trade-off rather than a fitted constant.
  • number of training stages = 4
    The additive fourth stage is a design choice, not fit to a closed-form target; the authors claim it improves CER across datasets.
  • LoRA rank and alpha = rank=12, alpha=32
    Fixed hyperparameters for LLM fine-tuning (Section 3.4), hand-set without sensitivity analysis.
  • Q-Former query count = 64
    Follows SLAM; used in the Q-Former projector comparison, not tuned here.
assumptions (4)
  • domain assumption TeleSpeechPT Data2Vec2 pre-trained on 300,000 hours of unlabeled Chinese dialect and accented speech improves dialect representations.
    Section 3.2.2 relies on prior work [29] with no in-paper ablation against a smaller or differently pre-trained encoder.
  • domain assumption Character error rate (CER) is computed consistently across all compared systems, including the authors' models and the baselines.
    Section 4 reports CER for all models with no description of normalization, punctuation stripping, or Chinese character segmentation; a mismatch here would invalidate the comparisons.
  • domain assumption The supervised 40,000-hour dataset's composition (dialects, domains, label quality) matches the test dialects.
    Section 3.1 lists dialect regions but gives no per-dialect hours, no label verification, and no handling of label noise.
  • domain assumption The LLM tokenizer and text prompt do not bias transcription output.
    Section 2 defines text prompts and tokenization but provides no analysis of prompt sensitivity or tokenization effects on dialectal characters.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Leveraging LLM and Self-Supervised Training Models for Speech Recognition in Chinese Dialects: A Comparative Analysis." pith.science (2026). https://pith.science/paper/QFZJ57A4

@misc{pith2026250521138,
  author       = {Pith},
  title        = {Pith review of: Leveraging LLM and Self-Supervised Training Models for Speech Recognition in Chinese Dialects: A Comparative Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QFZJ57A4}},
  note         = {Machine review of arXiv:2505.21138}
}
read the original abstract

Large-scale training corpora have significantly improved the performance of ASR models. Unfortunately, due to the relative scarcity of data, Chinese accents and dialects remain a challenge for most ASR models. Recent advancements in self-supervised learning have shown that self-supervised pre-training, combined with large language models (LLM), can effectively enhance ASR performance in low-resource scenarios. We aim to investigate the effectiveness of this paradigm for Chinese dialects. Specifically, we pre-train a Data2vec2 model on 300,000 hours of unlabeled dialect and accented speech data and do alignment training on a supervised dataset of 40,000 hours. Then, we systematically examine the impact of various projectors and LLMs on Mandarin, dialect, and accented speech recognition performance under this paradigm. Our method achieved SOTA results on multiple dialect datasets, including Kespeech. We will open-source our work to promote reproducible research

Figures

Figures reproduced from arXiv: 2505.21138 by the authors.

Figure 1
Figure 1. Overall model structure. Data2Vec2 as the speech encoder and incorporates convolu￾tional layers as the projector. Combined with a four-stage train￾ing strategy, the proposed method achieves SOTA performance on multiple Chinese dialect and accent test sets, including Ke￾speech (with CER of 6.48% vs. 7.52%), even with a small LLM (0.5B) and a low frame rate (6.25Hz). To promote reproducibil￾ity, we will release all as… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 26 canonical work pages

  1. [29]

    Qwen-Audio: Advancing Universal Audio Understand- ing via Unified Large-Scale Audio-Language Models,

    Y . Chu, J. Xu, X. Zhou, Q. Yang, S. Zhang, Z. Yan, C. Zhou, and J. Zhou, “Qwen-Audio: Advancing Universal Audio Understand- ing via Unified Large-Scale Audio-Language Models,”CoRR, 2023

    work page 2023

  2. [1]

    Leveraging LLM and Self-Supervised Training Models for Speech Recognition in Chinese Dialects: A Comparative Analysis

    Introduction Large-scale training corpora have significantly improved the performance of automatic speech recognition (ASR) mod- els [1, 2, 3, 4]. However, labeled dialect data remains pro- hibitively expensive, making it challenging to obtain in large quantities. As a result, most pre-trained models in Chinese speech processing communities, such as WeNet...

    work page Pith review arXiv 2025

  3. [2]

    Method As illustrated in Figure 1, the model architecture consists of an audio encoder and a LLM. For each sample during training, we define the text prompt (e.g., ”Transcribe the following speech”), the speech utterance, and the corresponding transcription asP, SandT, respectively. The text prompt and transcription are tokenized using the LLM’s tokenizer...

    work page

  4. [3]

    Experimental Setup 3.1. Datasets Training Set:The training data comprises a diverse set of inter- nal dialect and accent datasets, including samples from Anhui, Gansu, Hebei, Shandong, Shanxi, Tianjin, Cantonese, Henan, Sichuan, Chongqing, Northeast China, Shaanxi, Hubei, Fujian, Guizhou, Hangzhou, Hunan, Jiangxi, Shanghai, Suzhou, Yun- nan, Chongqing and...

    work page

  5. [4]

    Comparison of Projector Architectures We evaluate the effectiveness of different projection layers through small-scale experiments

    Experimental Results 4.1. Comparison of Projector Architectures We evaluate the effectiveness of different projection layers through small-scale experiments. Specifically, four types of projection layers—Linear, Conv1d, Transformer, and Q- Former are compared using the Data2Vec2 model pre-trained on 300,000 hours of data and tested on the Test-meeting and...

    work page arXiv

  6. [5]

    For the encoder, we employ Data2Vec2, pre-trained on 300,000 hours of unlabeled dialect and accented speech data

    Conclusion In this study, we investigate the performance of various struc- tural configurations within the paradigm of self-supervised training encoders paired with LLM decoders, leveraging a large-scale dataset comprising 40,000 hours of Chinese dialect and accented speech. For the encoder, we employ Data2Vec2, pre-trained on 300,000 hours of unlabeled d...

    work page

  7. [6]

    Zipformer: A faster and better encoder for automatic speech recognition,

    Z. Yao, L. Guo, X. Yang, W. Kang, F. Kuang, Y . Yang, Z. Jin, L. Lin, and D. Povey, “Zipformer: A faster and better encoder for automatic speech recognition,” inThe Twelfth International Conference on Learning Representations, 2023

    2023

  8. [7]

    Funasr: A fundamental end-to-end speech recognition toolkit,

    Z. Gao, Z. Li, J. Wang, H. Luo, X. Shi, M. Chen, Y . Li, L. Zuo, Z. Du, and S. Zhang, “Funasr: A fundamental end-to-end speech recognition toolkit,” inInterspeech 2023, 2023, pp. 1593–1597

    work page 2023

Show all 42 references
  1. [8]

    Robust Speech Recognition via Large-Scale Weak Supervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust Speech Recognition via Large-Scale Weak Supervision,” inICML, 2023

    2023

  2. [9]

    Google usm: Scaling auto- matic speech recognition beyond 100 languages,

    Y . Zhang, W. Han, J. Qin, Y . Wang, A. Bapna, Z. Chen, N. Chen, B. Li, V . Axelrod, G. Wanget al., “Google usm: Scaling auto- matic speech recognition beyond 100 languages,”arXiv preprint arXiv:2303.01037, 2023

    2023 arXiv

  3. [10]

    Scaling speech technology to 1,000+ languages,

    V . Pratap, A. Tjandra, B. Shi, P. Tomasello, A. Babu, S. Kundu, A. Elkahky, Z. Ni, A. Vyas, M. Fazel-Zarandiet al., “Scaling speech technology to 1,000+ languages,”Journal of Machine Learning Research, vol. 25, no. 97, pp. 1–52, 2024

    2024

  4. [11]

    Wavlm: Large-scale self- supervised pre-training for full stack speech processing,

    S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiaoet al., “Wavlm: Large-scale self- supervised pre-training for full stack speech processing,”IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022

    2022

  5. [12]

    WeNet 2.0: More Productive End- to-End Speech Recognition Toolkit,

    B. Zhang, D. Wu, Z. Peng, X. Song, Z. Yao, H. Lv, L. Xie, C. Yang, F. Pan, and J. Niu, “WeNet 2.0: More Productive End- to-End Speech Recognition Toolkit,” inInterspeech, 2022

    2022

  6. [13]

    HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units,

    W. Hsu, B. Bolte, Y . H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units,”ACM, 2021

    2021

  7. [14]

    Data2vec: A general framework for self-supervised learning in speech, vision and language,

    A. Baevski, W.-N. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli, “Data2vec: A general framework for self-supervised learning in speech, vision and language,” inInternational Conference on Ma- chine Learning. PMLR, 2022, pp. 1298–1312

    2022

  8. [15]

    Gpt-3: Its nature, scope, limits, and consequences,

    L. Floridi and M. Chiriatti, “Gpt-3: Its nature, scope, limits, and consequences,”Minds and Machines, vol. 30, pp. 681–694, 2020

    2020

  9. [16]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azharet al., “Llama: Open and efficient foundation language models,”arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  10. [17]

    GPT-4 Technical Report,

    OpenAI, “GPT-4 Technical Report,”CoRR, 2023

    2023

  11. [18]

    Gemini 1.5: Un- locking multimodal understanding across millions of tokens of context,

    G. Team, P. Georgiev, V . I. Lei, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan, S. Wanget al., “Gemini 1.5: Un- locking multimodal understanding across millions of tokens of context,”arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  12. [19]

    wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations,

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations,”Advances in neural information processing systems, vol. 33, pp. 12 449–12 460, 2020

    2020

  13. [20]

    SALMONN: Towards Generic Hearing Abilities for Large Language Models,

    C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. Ma, and C. Zhang, “SALMONN: Towards Generic Hearing Abilities for Large Language Models,”CoRR, 2023

    2023

  14. [21]

    An Embarrassingly Simple Approach for LLM with Strong ASR Capacity,

    Z. Ma, G. Yang, Y . Yang, Z. Gao, J. Wang, Z. Du, F. Yu, Q. Chen, S. Zheng, S. Zhang, and X. Chen, “An Embarrassingly Simple Approach for LLM with Strong ASR Capacity,”CoRR, 2024

    2024

  15. [22]

    Audiogpt: Understanding and generating speech, music, sound, and talking head,

    R. Huang, M. Li, D. Yang, J. Shi, X. Chang, Z. Ye, Y . Wu, Z. Hong, J. Huang, J. Liuet al., “Audiogpt: Understanding and generating speech, music, sound, and talking head,” inProceed- ings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 21, 2024, pp. 23 802–23 804

    2024

  16. [23]

    Leveraging large language models for exploiting asr uncer- tainty,

    P. Dighe, Y . Su, S. Zheng, Y . Liu, V . Garg, X. Niu, and A. Tew- fik, “Leveraging large language models for exploiting asr uncer- tainty,” inICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 12 231–12 235

    2024

  17. [24]

    Can gener- ative large language models perform asr error correction?

    R. Ma, M. Qian, P. Manakul, M. Gales, and K. Knill, “Can gener- ative large language models perform asr error correction?”arXiv preprint arXiv:2307.04172, 2023

    2023 arXiv

  18. [25]

    Mmger: Multi-modal and multi-granularity generative error correction with llm for joint accent and speech recognition,

    B. Mu, Y . Li, Q. Shao, K. Wei, X. Wan, N. Zheng, H. Zhou, and L. Xie, “Mmger: Multi-modal and multi-granularity generative error correction with llm for joint accent and speech recognition,” arXiv preprint arXiv:2405.03152, 2024

    2024 arXiv

  19. [26]

    Kespeech: An open source speech dataset of mandarin and its eight subdialects,

    Z. Tang, D. Wang, Y . Xu, J. Sun, X. Lei, S. Zhao, C. Wen, X. Tan, C. Xie, S. Zhouet al., “Kespeech: An open source speech dataset of mandarin and its eight subdialects,” inThirty-fifth Conference on Neural Information Processing Systems Datasets and Bench- marks Track (Round 2), 2021

    2021

  20. [27]

    AISHELL-1: An open-source Mandarin speech corpus and a speech recognition baseline,

    H. Bu, J. Du, X. Na, B. Wu, and H. Zheng, “AISHELL-1: An open-source Mandarin speech corpus and a speech recognition baseline,” inO-COCOSDA, 2017

    2017

  21. [28]

    Qwen2. 5 technical report,

    A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Weiet al., “Qwen2. 5 technical report,”arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  22. [30]

    BEATs: Audio pre-training with acoustic tokenizers,

    S. Chen, Y . Wu, C. Wang, S. Liu, D. Tompkins, Z. Chen, W. Che, X. Yu, and F. Wei, “BEATs: Audio pre-training with acoustic tokenizers,” inProceedings of the 40th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, A. Krause, E. Brunski...

    2023

  23. [31]

    Qwen2-audio technical report,

    Y . Chu, J. Xu, Q. Yang, H. Wei, X. Wei, Z. Guo, Y . Leng, Y . Lv, J. He, J. Linet al., “Qwen2-audio technical report,”arXiv preprint arXiv:2407.10759, 2024

    2024 arXiv

  24. [32]

    Unveiling the potential of llm-based asr on chinese open-source datasets,

    X. Geng, T. Xu, K. Wei, B. Mu, H. Xue, H. Wang, Y . Li, P. Guo, Y . Dai, L. Liet al., “Unveiling the potential of llm-based asr on chinese open-source datasets,” in2024 IEEE 14th International Symposium on Chinese Spoken Language Processing (ISCSLP). IEEE, 2024, pp. 26–30

    2024

  25. [33]

    WENETSPEECH: A 10000+ Hours Multi-Domain Mandarin Corpus for Speech Recognition,

    B. Zhang, H. Lv, P. Guo, Q. Shao, C. Yang, L. Xie, X. Xu, H. Bu, X. Chen, C. Zeng, D. Wu, and Z. Peng, “WENETSPEECH: A 10000+ Hours Multi-Domain Mandarin Corpus for Speech Recognition,” inICASSP, 2022

    2022

  26. [34]

    Why Gradient Clip- ping Accelerates Training: A Theoretical Justification for Adap- tivity,

    J. Zhang, T. He, S. Sra, and A. Jadbabaie, “Why Gradient Clip- ping Accelerates Training: A Theoretical Justification for Adap- tivity,” inICLR, 2020

    2020

  27. [35]

    LoRA: Low-Rank Adaptation of Large Language Models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-Rank Adaptation of Large Language Models,” inICLR, 2022

    2022

  28. [36]

    Tele- speechpt: Large-scale chinese multi-dialect and multi-accent speech pre-training,

    H. Chen, Z. Li, G. Xia, B. Liu, Y . Yang, J. Kang, and J. Li, “Tele- speechpt: Large-scale chinese multi-dialect and multi-accent speech pre-training,” inNational Conference on Man-Machine Speech Communication. Springer, 2024, pp. 183–190

    2024

  29. [37]

    Self-supervised learning with random-projection quantizer for speech recogni- tion,

    C.-C. Chiu, J. Qin, Y . Zhang, J. Yu, and Y . Wu, “Self-supervised learning with random-projection quantizer for speech recogni- tion,” inInternational Conference on Machine Learning. PMLR, 2022, pp. 3915–3924

    2022

  30. [38]

    Di- nosr: Self-distillation and online clustering for self-supervised speech representation learning,

    A. H. Liu, H.-J. Chang, M. Auli, W.-N. Hsu, and J. Glass, “Di- nosr: Self-distillation and online clustering for self-supervised speech representation learning,”Advances in Neural Information Processing Systems, vol. 36, pp. 58 346–58 362, 2023

    2023

  31. [39]

    Connecting speech encoder and large language model for asr,

    W. Yu, C. Tang, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. Ma, and C. Zhang, “Connecting speech encoder and large language model for asr,” inICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 12 637–12 641

    2024

  32. [40]

    Decoupled Weight Decay Regular- ization,

    I. Loshchilov and F. Hutter, “Decoupled Weight Decay Regular- ization,” inICLR, 2019

    2019

  33. [64]

    However, lower down-sampling rates also increase the computational load on the LLM, requiring more resources during both training and in- ference

    The results reveal that lower down-sampling rates preserve more information, resulting in lower CER. However, lower down-sampling rates also increase the computational load on the LLM, requiring more resources during both training and in- ference. Subsequent experiments use a ...

  34. [100]

    We configure LoRA with alpha = 32, rank = 12

    Regarding the training approach, when training the LLM, we freeze the LLM body and only update the LLM using LoRA fine-tuning [35]. We configure LoRA with alpha = 32, rank = 12. The alpha parameter controls the weight of the LoRA matrix, and the rank parameter determines the d...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.