Pith. sign in

REVIEW 4 major objections 5 minor 37 references

GigaAM: Efficient Self-Supervised Learner for Speech Recognition

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Supervised cluster targets from a CTC-fine-tuned ASR teacher give a speech encoder that cuts Russian word error rates by roughly half versus Whisper-large-v3.

desk verdict Worth a serious referee: the HuBERT-CTC recipe is a real contribution, but the headline WERs depend on an undescribed pretraining corpus that may overlap the test sets. read the letter →

arxiv 2506.01192 v1 pith:A4HRHOST submitted 2025-06-01 eess.AS cs.SD

Aleksandr Kutsakov , Alexandr Maximenko , Georgii Gospodinov , Pavel Bogomolov , Fyodor Minkin This is my paper · ORCID
classification eess.AScs.SD
keywords self-supervisedlearningspeechrecognitionHuBERT-CTCmaskedlanguagemodelinglow-resourceASRRussianstreamingConformer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces HuBERT-CTC, a self-supervised pretraining recipe for speech recognition in which masked audio is predicted not from low-level acoustic units but from discrete cluster IDs produced by K-means on the top layer of a CTC-fine-tuned ASR model. These targets carry ASR-specific semantic information, so the student's representations improve monotonically to the last layer instead of peaking mid-network as they do with HuBERT and BEST-RQ. The authors train the GigaAM family on 100k hours of Russian audio and 2k hours of labeled data, reporting 3.35 WER with CTC decoding and 3.9 with RNNT on Golos Farfield, 3.1/2.7 on Common Voice Russian, and 5.5 on Russian LibriSpeech, with the paper's headline claim being a nearly 50% improvement over Whisper-large-v3 on their benchmarks. They also show that dynamic chunk-size sampling during pretraining lets one model serve both full-context and streaming fine-tuning. The claim matters because, if correct, supervision-derived cluster targets are a stronger and more sample-efficient pretraining signal than purely unsupervised targets, especially for low-resource languages.

What carries the argument

The load-bearing mechanism is HuBERT-CTC pretraining: masked language modeling whose targets at masked positions are K-means cluster IDs computed on the final encoder layer of a CTC-fine-tuned ASR teacher, rather than MFCC clusters or intermediate-layer clusters. This target choice is what shifts the learned representations toward ASR-relevant semantics and produces the monotonic improvement to the last layer. A second mechanism is chunkwise attention with dynamic chunk-size sampling during pretraining, which partitions audio into segments and samples chunk sizes (1s, 2s, 4s, 8s) so the same pretrained Conformer encoder can later be fine-tuned for full-context, long-form, and streaming inference.

What would settle it

Transcribe a set of newly recorded Russian sentences that are verified to be absent from both the pretraining and fine-tuning corpora, then compare HuBERT-CTC's word error rate with Whisper-large-v3's on that set; if the relative gain falls far below the reported roughly 50%, the claim that supervision-derived targets drive the improvement would be contradicted.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that defining the masked-prediction targets with a supervised ASR model makes self-supervised speech representations more useful for downstream recognition. Concretely, HuBERT-CTC applies K-means clustering to the last hidden layer of a 240M wav2vec2-style encoder that has been fine-tuned with CTC on 2k hours of transcribed Russian, and the resulting cluster assignments become the prediction targets at masked positions during pretraining. Unlike HuBERT, which clusters MFCCs in stage 1 and intermediate-layer features in stage 2, and unlike BEST-RQ, which uses a frozen random-projection quantizer, the HuBERT-CTC encoder keeps improving in ASR quality all the way to its final layer. The paper reports WERs of 3.35 (CTC) and 3.9 (RNNT) on Golos Farfield, 3.1 and 2.7 on Common Voice Russian, 5.5 on Russian LibriSpeech, and 7.41 WER at 0.1% of the supervised data compared with 16.80 for BEST-RQ and 16.70 for HuBERT. The scaling study also finds that a 100M-parameter student can outperform the 240M teacher, and that WER plateaus once pretraining data exceeds roughly 6k hours.

Load-bearing premise

The load-bearing premise is that the 100,000 hours of unlabeled Russian audio used for pretraining does not overlap with the test recordings of Golos, Common Voice, and Russian LibriSpeech; the paper never describes this corpus or reports that duplicates were removed, and the same 2,000 hours of labeled audio are used to fine-tune both the teacher and the student.

Editorial extensions

If this is right

  • At 0.1% labeled data, HuBERT-CTC's 7.41 WER is less than half of BEST-RQ's 16.80 and HuBERT's 16.70, implying the advantage of supervision-derived targets grows as supervision shrinks.
  • Because WER stabilizes above roughly 6k hours of pretraining audio, distilling a large multilingual ASR model into a smaller domain-specific one may be feasible with a modest amount of unlabeled in-domain data.
  • A 100M-parameter student beats the 240M teacher, indicating the student can exceed its teacher's downstream performance.
  • Dynamic chunk-size pretraining supports full-context, long-form, and streaming inference from a single checkpoint, with chunk sizes as small as 1s and only 200ms excluded as unstable during pretraining.
  • Consistency regularization (CR-CTC) helps small models trained from scratch but gives little or negative gain after full HuBERT-CTC pretraining, suggesting pretraining itself provides similar regularization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the 100k-hour pretraining corpus overlaps with the Golos, Common Voice, or Russian LibriSpeech test recordings, the reported WER gaps, including the nearly 50% improvement over Whisper, would be partly an artifact of data leakage; the paper does not demonstrate disjointness.
  • The same target-generation scheme should transfer to other languages with small labeled sets, since nothing in the method is Russian-specific and the plateau above 6k hours suggests a small transcribed set may suffice to build a useful target generator.
  • The streaming results at 200ms chunk size degrade to roughly 10.4 WER, so extending this approach to low-latency streaming may require architectural changes rather than more pretraining data.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes HuBERT-CTC, a speech pretraining method that applies K-means clustering to the last-layer hidden states of a CTC-fine-tuned ASR teacher and uses the resulting cluster IDs as masked-prediction targets, building on the HuBERT framework. The authors train the GigaAM family of Conformer models on up to 100k hours of Russian audio and report state-of-the-art WER results on Russian benchmarks, including Golos Farfield, Common Voice (ru), and Russian LibriSpeech, with a claimed improvement of nearly 50% relative to Whisper-large-v3. The paper also introduces dynamic chunk-size sampling during pretraining to support both full-context and streaming fine-tuning, and it includes scaling experiments over pretraining data size, fine-tuning data size, model size, and pretraining steps. The models and inference code are released under the MIT license.

Significance. If the reported results are valid, the paper would make a practically valuable contribution: it presents a recipe for using supervision-derived cluster targets to obtain strong Russian ASR performance with relatively small models, and it releases usable open-source models and code. The scaling analysis and the dynamic-chunking study are useful empirical additions, and the explicit comparison across several pretraining objectives at multiple fine-tuning data sizes is informative. However, the central comparison is currently not verifiable because the pretraining corpus is not described and no deduplication against evaluation sets is reported; moreover, the self-supervised framing is weakened by the fact that the teacher is fine-tuned on the same labeled data used for downstream fine-tuning. These issues are external-validity concerns rather than internal contradictions, and they can in principle be addressed with additional evidence and clarification.

major comments (4)
  1. [Section 4] The pretraining corpus is described only as '100k hours of Russian audio'; no composition, source, or filtering beyond VAD is provided, and no deduplication against the Golos, Common Voice (ru), or Russian LibriSpeech test sets is reported. Because the headline WER gains in Tables 1 and 2 depend on this corpus, the paper must report its provenance and present a deduplication analysis (e.g., exact or near-duplicate audio between pretraining and test sets). Without this, the comparison with baselines that were not pretrained on the same audio could be substantially inflated.
  2. [Section 3.1 and Section 4] The method is presented as self-supervised, but the cluster targets are generated by a wav2vec2 teacher that was 'fine-tuned ... with the same supervised data as the other models in the paper.' This means the pretraining signal originates from labeled data, making the approach closer to knowledge distillation than to pure SSL. It also creates a potential leakage path: if any pretraining utterance overlaps a test set, the teacher has already seen the corresponding transcription, and the student can inherit that information through the cluster targets. Please clarify the exact data flow and provide an ablation where the teacher is not fine-tuned on the evaluation-related supervised data, or otherwise quantify how much of the observed gain comes from the supervised teacher.
  3. [Section 4.1, Table 2] The controlled comparison in Table 2 does not state whether all pretraining methods (BEST-RQ, wav2vec2, HuBERT, and HuBERT-CTC) used the identical pretraining corpus, VAD filtering, number of steps, model size, and fine-tuning recipe. Without this information, the large low-data gap (7.41 vs 16.80 and 16.70 at 0.1% labeled data) could be attributable to differences in pretraining data or protocol rather than to the target-generation method. Please specify the exact protocol for every baseline, including the source and amount of unlabeled audio and the fine-tuning details.
  4. [Section 4, Tables 1 and 2] No error bars, confidence intervals, or repeated-seed results are reported for any of the headline WER numbers; averaging over the last ten epochs is not a measure of run-to-run variability. Given that the reported gains are large, the paper should report at least two or three independent fine-tuning runs for the central comparisons, or explicitly present checkpoint-averaging statistics and distinguish them from seed variance.
minor comments (5)
  1. [Section 4.3, Table 4] The 'pretraining' column in Table 4 is ambiguous, especially rows such as 'Full-context 8s' and 'Full-context' used as a pretraining method; please clarify the table structure and add a legend explaining the relationship between pretraining chunking, fine-tuning chunking, and the 'Long-form'/'Streaming' columns.
  2. [Section 3.2.1] The VAD filtering procedure is described without specifying the actual silence threshold or the VAD model used; these are free parameters of the pipeline and should be reported for reproducibility.
  3. [Section 4.2.3, Table 3] In Table 3, the column labeled '–' reports single numbers without standard deviations, while the other columns report means over pretraining data quantities; please state how many runs produced the '–' numbers and why no variability is reported for them.
  4. [References] Reference [25] includes the word 'rejected' in the bibliographic entry; this is non-standard and should be removed or replaced with the appropriate publication status.
  5. [Throughout] There are several typographical issues, including 'use100k hours' in Section 4, 'V oice Activity Detection' in Section 3.2.1, and inconsistent spacing around dashes and percentages; these should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation found: HuBERT-CTC's supervised-teacher targets are a disclosed design choice, not an instance of the student's own output, and the headline WER claims rest on public benchmarks rather than on the paper's own definitions.

full rationale

I find no circular step under the strict reduction standard. The pretraining targets are cluster IDs of a CTC teacher's last-layer features; the student predicts these targets on masked unlabeled audio. This is a supervised-distillation-style pretraining objective, not a quantity that is defined in terms of the student's own downstream WER. The downstream WERs are computed on public test sets (Golos, Common Voice ru, Russian LibriSpeech) after separate fine-tuning; they are not fit parameters of the pretraining objective. The only load-bearing self-link is the Golos benchmark [11], whose author team includes F. Minkin, but Golos is an externally available dataset and the evaluation is against public test splits; this does not make the derivation circular. The absence of deduplication evidence between the 100k-hour pretraining corpus and evaluation sets (Section 4) is a legitimate data-integrity and external-validity concern, and could inflate WER gains if overlap exists, but it is not a case where the paper's equations reduce to their inputs. The label-informed teacher is disclosed ('employ an encoder fine-tuned on ASR tasks'; 'fine-tuned ... with the same supervised data'), and the method therefore may be better described as distillation than as pure SSL, but that is a naming/attribution issue, not a circular derivation. No self-citation chain or imported uniqueness theorem forces the central claim.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims rest on empirical assumptions about target quality, data hygiene, and chunking transfer, none of which are formally proven. The method also depends on several hand-chosen hyperparameters (VAD threshold, cluster count, chunk set, kernel size). No new physical entities are introduced.

free parameters (4)
  • VAD silence threshold = 60%
    Segments with more than 60% silence were discarded; this threshold is hand-chosen and affects the pretraining data composition (Section 3.2.1).
  • K-means cluster count
    The number of clusters for generating teacher targets is not reported in the paper; it is a free parameter of the pretraining method (Section 3.1).
  • Dynamic chunk size set = 1s, 2s, 4s, 8s
    Chunk sizes were uniformly sampled from this set during pretraining; 200ms was excluded because training diverged, making the choice partly post-hoc (Section 4.3).
  • Conformer kernel size = 5
    A small kernel size of 5 (200ms) was chosen after an ablation showing kernel 31 gave slightly worse WER (3.35 to 3.49); this constrains the receptive field (Section 4.3.1).
assumptions (4)
  • domain assumption K-means on the last layer of a CTC-fine-tuned teacher produces semantically informative targets for masked prediction.
    Section 3.1 and Figure 1 motivate this empirically, but there is no theoretical or formal justification that last-layer CTC hidden states cluster into useful phonetic/lexical units.
  • domain assumption The 100k-hour pretraining corpus is disjoint from the test sets used for evaluation.
    Section 4 describes the pretraining data only as 100k hours of Russian audio; no composition, source list, or deduplication against Golos, Common Voice, or LibriSpeech is provided, so test-set contamination cannot be ruled out.
  • domain assumption Dynamic chunk-size sampling during pretraining transfers to fixed-chunk fine-tuning for both full-context and streaming inference.
    Section 4.3 and Table 4 support this empirically for the tested chunk sizes, but the mechanism is not analyzed and the 200ms exclusion shows a boundary to the assumption.
  • domain assumption VAD filtering that discards high-silence segments improves pretraining without biasing the evaluation domain.
    Section 3.2.1 reports a 5-10% average improvement from VAD filtering, but the interaction with test-domain silence characteristics is not discussed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GigaAM: Efficient Self-Supervised Learner for Speech Recognition." pith.science (2026). https://pith.science/paper/A4HRHOST

@misc{pith2026250601192,
  author       = {Pith},
  title        = {Pith review of: GigaAM: Efficient Self-Supervised Learner for Speech Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/A4HRHOST}},
  note         = {Machine review of arXiv:2506.01192}
}
read the original abstract

Self-Supervised Learning (SSL) has demonstrated strong performance in speech processing, particularly in automatic speech recognition. In this paper, we explore an SSL pretraining framework that leverages masked language modeling with targets derived from a speech recognition model. We also present chunkwise attention with dynamic chunk size sampling during pretraining to enable both full-context and streaming fine-tuning. Our experiments examine scaling with respect to model size and the amount of data. Using our method, we train the GigaAM family of models, including a state-of-the-art model for Russian speech recognition that outperforms Whisper-large-v3 by 50%. We have released our foundation and ASR models, along with the inference code, under the MIT license as open-source resources to the research community. Available at https://github.com/salute-developers/gigaam.

Figures

Figures reproduced from arXiv: 2506.01192 by the authors.

Figure 1
Figure 1. Intermediate layers probing of pretrained encoders with different pretraining approaches the performance of our SSL model on downstream task. 3. Our methodology 3.1. Pretraining method Our pretraining method HuBERT-CTC builds on HuBERT, where KMeans clustering is applied to the hidden states of the encoder. However, unlike the standard HuBERT approach, which uses a pretrained encoder, we employ an encoder fine￾tuned… view at source ↗
Figure 2
Figure 2. The dependency of WER on the pretraining dataset size with fine-tuning set scaling • Fine-tuning dataset size: 2000, 200, 20, 2 hours. • Model size: 30M, 60M, 100M, 240M, 500M. • Number of pretraining steps: 25k, 100k, 400k steps. 4.2.1. Pretraining data The reduction of supervised data quantity significantly affects speech recognition quality, but the dependence on the amount of unsupervised data is less frequently… view at source ↗
Figure 3
Figure 3. Quality dependency on model size with fine-tuning set scaling. For each configuration, we average the metrics across pre-training data quantities ranging from 6k to 100k hours and include the standard deviation on the graph training durations. Notably, the application of CR-CTC does not benefit larger models, while for smaller models, pretraining effectively replaces the need for regularization, resulting in only ma… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 31 canonical work pages

  1. [1]

    Numerous methods have emerged [1, 2, 3, 4, 5, 6, 7], but some of them suffer from training instability due to representation collapse [1, 5, 7]

    Introduction Self-supervised learning (SSL) has transformed ASR by lever- aging vast amounts of unlabeled data, reducing the need for manual annotation. Numerous methods have emerged [1, 2, 3, 4, 5, 6, 7], but some of them suffer from training instability due to representation collapse [1, 5, 7]. On the other hand, BEST- RQ [3] has proven effective due to...

    work page

  2. [2]

    Related Work The field of self-supervised learning for speech encompasses a wide range of methods, from early approaches such as wav2vec

    work page

  3. [3]

    Pretraining method Our pretraining method HuBERT-CTC builds on HuBERT, where KMeans clustering is applied to the hidden states of the encoder

    Our methodology 3.1. Pretraining method Our pretraining method HuBERT-CTC builds on HuBERT, where KMeans clustering is applied to the hidden states of the encoder. However, unlike the standard HuBERT approach, which uses a pretrained encoder, we employ an encoder fine- tuned on ASR tasks. We compare the properties of these ap- proaches by probing the inte...

    work page

  4. [4]

    pretraining

    Experiments Our experimental setup is as follows: for self-supervised pre- training with HuBERT-CTC, we use100k hours of Russian au- dio and train for 400k steps with a virtual batch size of 9 hours. For supervised fine-tuning, we utilize datasets including Golos [11], Mozilla Common V oice (ru) [12], Russian LibriSpeech [13], and SOV A [28], totaling 2k ...

    work page

  5. [5]

    By integrating dynamic chunk size training, our method supports both full-context and streaming fine-tuning ability within a single pretraining run

    Conclusion We presented an SSL pretraining framework for ASR which leverages semantically enriched targets from a CTC-based ASR model. By integrating dynamic chunk size training, our method supports both full-context and streaming fine-tuning ability within a single pretraining run. Experiments on Russian speech recognition demonstrate that our HuBERT-CTC...

    work page

  6. [6]

    GigaAM: Efficient Self-Supervised Learner for Speech Recognition

    to more recent advances. Although many excellent tech- niques have been proposed, in this work we focus on two rep- resentative frameworks – HuBERT [1] and BEST-RQ – due to their complementary strengths. HuBERT, which forms the ba- sis of our approach, employs a multi-stage masked language modeling pipeline to learn robust representations. At the same tim...

    work page Pith review arXiv 2025

  7. [7]

    Hubert: Self-supervised speech rep- resentation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhut- dinov, and A. Mohamed, “Hubert: Self-supervised speech rep- resentation learning by masked prediction of hidden units,” IEEE/ACM Transactions on Audio, Speech, and Language Pro- cessing, vol. 29, pp. 3451–3460, 2021

    2021

  8. [8]

    Wavlm: Large-scale self- supervised pre-training for full stack speech processing,

    S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y . Qian, Y . Qian, M. Zeng, and F. Wei, “Wavlm: Large-scale self- supervised pre-training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing , vol. 16, pp. 1505–1518, 2021

    work page 2021

Show all 37 references
  1. [9]

    Self-supervised learning with random-projection quantizer for speech recogni- tion,

    C. Chiu, J. Qin, Y . Zhang, J. Yu, and Y . Wu, “Self-supervised learning with random-projection quantizer for speech recogni- tion,” in International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , ser. Proceed- ings of Machine Learning R...

    2022

  2. [10]

    Supervision-guided codebooks for masked prediction in speech pre-training,

    C. Wang, Y . Wang, Y . Wu, S. Chen, J. Li, S. Liu, and F. Wei, “Supervision-guided codebooks for masked prediction in speech pre-training,” in Interspeech 2022, 2022, pp. 2643–2647

    2022

  3. [11]

    wav2vec 2.0: A framework for self-supervised learning of speech representa- tions,

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representa- tions,” in Advances in Neural Information Processing Systems , vol. 33. Curran Associates, Inc., 2020, pp. 12 449–12 460

    2020

  4. [12]

    wav2vec: Unsupervised pre-training for speech recognition,

    S. Schneider, A. Baevski, R. Collobert, and M. Auli, “wav2vec: Unsupervised pre-training for speech recognition,” in Proc. of In- terspeech, 2019

    2019

  5. [13]

    data2vec: A general framework for self-supervised learning in speech, vision and language,

    A. Baevski, W.-N. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli, “data2vec: A general framework for self-supervised learning in speech, vision and language,” in Proceedings of the 39th Inter- national Conference on Machine Learning , ser. Proceedings of Machine Learning Research, vo...

    2022

  6. [14]

    Layer-wise analysis of a self-supervised speech representation model,

    A. Pasad, J.-C. Chou, and K. Livescu, “Layer-wise analysis of a self-supervised speech representation model,” in2021 IEEE Auto- matic Speech Recognition and Understanding Workshop (ASRU), 2021, pp. 914–921

    2021

  7. [15]

    Resource-efficient transfer learning from speech foundation model using hierarchical feature fusion,

    Z. Huo, K. C. Sim, B. Li, D. Hwang, T. N. Sainath, and T. Strohman, “Resource-efficient transfer learning from speech foundation model using hierarchical feature fusion,” ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. ...

    2023

  8. [16]

    Connectionist temporal classification: labelling unsegmented se- quence data with recurrent neural networks,

    A. Graves, S. Fern ´andez, F. J. Gomez, and J. Schmidhuber, “Connectionist temporal classification: labelling unsegmented se- quence data with recurrent neural networks,” inProceedings of the 23rd International Conference on Machine Learning , ser. ACM International Conference...

    2006

  9. [17]

    Golos: Russian dataset for speech research,

    N. Karpov, A. Denisenko, and F. Minkin, “Golos: Russian dataset for speech research,” in Interspeech 2021, 2021, pp. 1419–1423

    2021

  10. [18]

    Com- mon voice: A massively-multilingual speech corpus,

    R. Ardila, M. Branson, K. Davis, M. Kohler, J. Meyer, M. Hen- retty, R. Morais, L. Saunders, F. Tyers, and G. Weber, “Com- mon voice: A massively-multilingual speech corpus,” inProceed- ings of The 12th Language Resources and Evaluation Conference. Marseille, France: European ...

    2020

  11. [19]

    Mls: A large-scale multilingual dataset for speech research,

    V . Pratap, Q. Xu, A. Sriram, G. Synnaeve, and R. Collobert, “Mls: A large-scale multilingual dataset for speech research,” in Inter- speech 2020, 2020, pp. 2757–2761

    2020

  12. [20]

    Xls-r: Self-supervised cross-lingual speech representation learning at scale,

    A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. von Platen, Y . Saraf, J. Pino, A. Baevski, A. Con- neau, and M. Auli, “Xls-r: Self-supervised cross-lingual speech representation learning at scale,” in Interspeech 2022, 2022, pp. 2278–2282

    2022

  13. [21]

    Google usm: Scaling automatic speech recognition beyond 100 languages,

    Y . Zhang, W. Han, J. Qin, Y . Wang, A. Bapna, Z. Chen, N. Chen, B. Li, V . Axelrod, G. Wang, Z. Meng, K. Hu, A. Rosenberg, R. Prabhavalkar, D. S. Park, P. Haghani, J. Riesa, G. Perng, H. Soltau, T. Strohman, B. Ramabhadran, T. Sainath, P. Moreno, C.-C. Chiu, J. Schalkwyk, F. ...

  14. [22]

    Scaling asr improves zero and few shot learning,

    W. Zheng, A. Xiao, G. Keren, D. Le, F. Zhang, C. Fuegen, O. Kalinli, Y . Saraf, and A. Mohamed, “Scaling asr improves zero and few shot learning,” inInterspeech 2022, 2022, pp. 5135–5139

    2022

  15. [23]

    w2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training,

    Y .-A. Chung, Y . Zhang, W. Han, C.-C. Chiu, J. Qin, R. Pang, and Y . Wu, “w2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training,” in 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2021, pp. 244–250

    2021

  16. [24]

    Unsupervised data selection via discrete speech representation for asr,

    Z. Lu, Y . Wang, Y . Zhang, W. Han, Z. Chen, and P. Haghani, “Unsupervised data selection via discrete speech representation for asr,” in Interspeech 2022, 2022, pp. 3393–3397

    2022

  17. [25]

    Scaling Laws for Neural Language Models,

    J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling Laws for Neural Language Models,” 1 2020

    2020

  18. [26]

    Training compute-optimal large language models,

    J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driess- che, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, O. Vinyals, J. W. Rae, and L. Si...

    2022

  19. [27]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” in Interna- tional Conference on Learni...

    2021

  20. [28]

    Scaling end-to-end models for large-scale multilingual asr,

    B. Li, R. Pang, T. N. Sainath, A. Gulati, Y . Zhang, J. Qin, P. Haghani, W. R. Huang, M. Ma, and J. Bai, “Scaling end-to-end models for large-scale multilingual asr,” 2021

    2021

  21. [29]

    Conformer: Convolution-augmented transformer for speech recognition,

    A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y . Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y . Wu, and R. Pang, “Conformer: Convolution-augmented transformer for speech recognition,” in Interspeech 2020, 2020, pp. 5036–5040

    2020

  22. [30]

    Robust speech recognition via large-scale weak su- pervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. Mcleavey, and I. Sutskever, “Robust speech recognition via large-scale weak su- pervision,” in Proceedings of the 40th International Conference on Machine Learning, ser. Proceedings of Machine Learning Re- search, vol. 202. PMLR, 2...

    2023

  23. [31]

    Conformer-1: Robust asr via large-scale semisupervised bootstrapping,

    K. Zhang, L. Chkhetiani, F. M. Ramirez, Y . Khare, A. Vanzo, M. Liang, S. R. Martin, G. Oexle, R. Bousbib, T. Peyash, M. Nguyen, D. Pulliam, and D. Donato, “Conformer-1: Robust asr via large-scale semisupervised bootstrapping,” 2024

    2024

  24. [32]

    Semdedup: Data-efficient learning at web-scale through seman- tic deduplication,

    R. Ardila, M. Branson, K. Davis, M. Henretty, and M. K. et al, “Semdedup: Data-efficient learning at web-scale through seman- tic deduplication,” Submitted to Transactions on Machine Learn- ing Research, 2023, rejected

    2023

  25. [33]

    Beyond neural scaling laws: beating power law scaling via data pruning,

    B. Sorscher, R. Geirhos, S. Shekhar, S. Ganguli, and A. Morcos, “Beyond neural scaling laws: beating power law scaling via data pruning,” in Advances in Neural Information Processing Systems, vol. 35. Curran Associates, Inc., 2022, pp. 19 523–19 536

    2022

  26. [34]

    Dy- namic chunk convolution for unified streaming and non-streaming conformer asr,

    X. Li, G. Huybrechts, S. Ronanki, J. Farris, and S. Bodapati, “Dy- namic chunk convolution for unified streaming and non-streaming conformer asr,” in ICASSP 2023 - 2023 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP) , 2023, pp. 1–5

    2023

  27. [35]

    Sova dataset,

    E. Zubarev and T. Moskalets, “Sova dataset,” https://github.com/ sovaai/sova-dataset, 2022

    2022

  28. [37]

    Cr-ctc: Consistency regularization on ctc for improved speech recognition,

    Z. Yao, W. Kang, X. Yang, F. Kuang, L. Guo, H. Zhu, Z. Jin, Z. Li, L. Lin, and D. Povey, “Cr-ctc: Consistency regularization on ctc for improved speech recognition,” 2024

    2024

  29. [2023]

    Available: https://arxiv.org/abs/2303.01037

    [Online]. Available: https://arxiv.org/abs/2303.01037

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.