Pith. sign in

REVIEW 2 major objections 2 minor 29 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

Reverse multi-stage fine-tuning lets a 244M Whisper model match or exceed 769M counterparts on Indic spontaneous speech.

2026-06-30 21:51 UTC pith:3OXKAQ6V

load-bearing objection Vividh-ASR gives a useful tiered benchmark for Indic ASR and R-MFT shows a workable path for smaller models on spontaneous speech, but the gains look tied to the tested Whisper setups. the 2 major comments →

arxiv 2605.13087 v2 pith:3OXKAQ6V submitted 2026-05-13 cs.CL cs.AI

Vividh-ASR: A Complexity-Tiered Benchmark and Optimization Dynamics for Robust Indic Speech Recognition

Kush Juvekar , Kavya Manohar , Aditya Srinivas Menon , Arghya Bhattacharya , Kumarmanas Nethil This is my paper
classification cs.CL cs.AI
keywords Vividh-ASRIndic ASRWhisper fine-tuningspontaneous speechreverse multi-stage fine-tuningcomplexity-tiered benchmarkHindi Malayalam speech recognitionparameter-efficient ASR
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Standard fine-tuning of models like Whisper improves read speech in low-resource languages but degrades spontaneous audio performance. The paper introduces the Vividh-ASR benchmark stratified into studio, broadcast, spontaneous, and synthetic noise tiers for Hindi and Malayalam to diagnose the mismatch. Controlled experiments on learning-rate timing and curriculum ordering show that early large parameter updates cut global word error rate by 12 points and a hard-to-easy curriculum boosts spontaneous results. These observations motivate reverse multi-stage fine-tuning (R-MFT), which concentrates adaptation in the decoder and lets the smaller model compete with or surpass larger ones while preserving the encoder's acoustic geometry.

Core claim

The authors establish that reverse multi-stage fine-tuning (R-MFT), built from early large updates followed by a hard-to-easy curriculum, enables a parameter-efficient 244M Whisper model to match or exceed conventionally fine-tuned 769M counterparts across the four complexity tiers of the Vividh-ASR benchmark. Representational analysis via CKA and SVD shows that successful schedules concentrate adaptation in the decoder while leaving the pre-trained encoder's acoustic geometry intact.

What carries the argument

Reverse multi-stage fine-tuning (R-MFT), a training recipe of learning-rate timing and curriculum ordering that reverses typical progression to prioritize early large parameter updates.

Load-bearing premise

The controlled study of learning-rate timing and curriculum ordering identifies generalizable factors for spontaneous speech gains rather than effects limited to the four benchmark tiers or the tested Whisper variants.

What would settle it

An experiment in which R-MFT applied to a different multilingual ASR model or an additional low-resource language pair produces no spontaneous-speech gains relative to standard fine-tuning would falsify the central claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Early large parameter updates improve global WER by 12 absolute points on the benchmark.
  • A hard-to-easy curriculum supplies additional gains for spontaneous speech.
  • Effective schedules concentrate adaptation in the decoder while preserving the pre-trained encoder.
  • The 244M model reaches or surpasses 769M model performance without extra parameters.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • R-MFT may reduce reliance on model scale when adapting multilingual ASR systems to spontaneous speech in other Indic or non-Indic languages.
  • The four-tier benchmark structure could serve as a template for diagnosing similar read-versus-spontaneous mismatches in other sequence-to-sequence tasks.
  • Future ablation studies might isolate whether the decoder-focused adaptation pattern holds when R-MFT is applied to non-Whisper encoder-decoder architectures.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper introduces Vividh-ASR, a complexity-stratified benchmark for Hindi and Malayalam ASR spanning studio, broadcast, spontaneous, and synthetic tiers. Through controlled experiments on learning-rate timing and curriculum ordering during Whisper fine-tuning, it identifies that early large updates yield 12-point absolute WER gains and hard-to-easy curricula benefit spontaneous speech. These motivate reverse multi-stage fine-tuning (R-MFT), claimed to let a 244M-parameter Whisper model match or exceed a conventionally fine-tuned 769M model. Representational analysis via CKA and SVD indicates decoder-focused adaptation that preserves encoder geometry. The benchmark and models are released.

Significance. If the R-MFT recipe and its decoder-centric adaptation mechanism prove robust, the work would offer a practical, parameter-efficient path for improving spontaneous-speech performance in low-resource Indic ASR without scaling model size, supported by the release of the tiered benchmark and trained models as reusable artifacts.

major comments (2)
  1. [Abstract / Experiments] Abstract and experimental results: the central claim that R-MFT enables the 244M model to match/exceed the 769M model rests on a controlled study of learning-rate timing and curriculum ordering, yet the manuscript provides no cross-model (beyond the specific Whisper small/medium pair) or cross-language ablations to show these factors transfer beyond the four Vividh-ASR tiers; without such tests the parameter-efficiency conclusion does not follow from the reported 12-point WER shift.
  2. [Methods / Results] Methods and results sections: the reported 12 absolute WER improvement from early large updates and the spontaneous-speech gains from hard-to-easy curriculum lack accompanying dataset statistics, error bars, full ablation tables, or statistical significance tests, making it impossible to verify that the gains are load-bearing for the R-MFT recipe rather than specific to the tested conditions.
minor comments (2)
  1. [Benchmark description] The four-tier naming (studio/broadcast/spontaneous/synthetic) is introduced without an explicit table listing per-tier utterance counts, durations, or speaker demographics, which would aid reproducibility.
  2. [Model descriptions] Notation for model sizes (244M vs 769M) should be cross-referenced to the exact Whisper variants (small/medium) in a dedicated table for clarity.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their insightful comments on the generalizability of our R-MFT recipe and the statistical presentation of results. We address each major comment in detail below.

read point-by-point responses

  1. Referee: [Abstract / Experiments] Abstract and experimental results: the central claim that R-MFT enables the 244M model to match/exceed the 769M model rests on a controlled study of learning-rate timing and curriculum ordering, yet the manuscript provides no cross-model (beyond the specific Whisper small/medium pair) or cross-language ablations to show these factors transfer beyond the four Vividh-ASR tiers; without such tests the parameter-efficiency conclusion does not follow from the reported 12-point WER shift.

    Authors: The experiments focus on demonstrating the effectiveness of R-MFT within the Whisper family for the two languages and four tiers in Vividh-ASR. The 12-point WER gain and the matching performance are shown specifically for these models. We agree that the absence of broader ablations limits the strength of the general parameter-efficiency claim. In the revision, we will clarify the scope of the claims to the tested conditions and add a limitations paragraph discussing the need for future cross-model and cross-lingual validation. revision: partial

  2. Referee: [Methods / Results] Methods and results sections: the reported 12 absolute WER improvement from early large updates and the spontaneous-speech gains from hard-to-easy curriculum lack accompanying dataset statistics, error bars, full ablation tables, or statistical significance tests, making it impossible to verify that the gains are load-bearing for the R-MFT recipe rather than specific to the tested conditions.

    Authors: We acknowledge this limitation in the current manuscript. We will revise the methods and results sections to include relevant dataset statistics (e.g., hours per tier), error bars from repeated runs with different seeds, expanded ablation tables, and statistical significance tests (e.g., paired t-tests) for the key improvements reported. revision: yes

Circularity Check

0 steps flagged

No circularity: claims derive from empirical experiments on released benchmark and models.

full rationale

The paper's central claims rest on controlled experiments measuring WER improvements from learning-rate timing and curriculum ordering across the four Vividh-ASR tiers, followed by post-hoc CKA/SVD analysis of the resulting models. These are direct observational results from training runs, not reductions via equations, fitted parameters renamed as predictions, or self-citation chains. R-MFT is presented as a recipe motivated by those measurements rather than derived by construction from any input definition. The work is self-contained against external benchmarks and artifacts, with no load-bearing step that collapses to its own inputs.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No free parameters, axioms, or invented entities are described in the abstract; the work consists of empirical benchmark creation and training experiments.

pith-pipeline@v0.9.1-grok · 5695 in / 1137 out tokens · 30642 ms · 2026-06-30T21:51:34.783725+00:00 · methodology

0 comments
read the original abstract

Fine-tuning multilingual ASR models like Whisper for low-resource languages often improves read speech but degrades spontaneous audio performance. To diagnose this mismatch, we introduce Vividh-ASR, a complexity-stratified benchmark for Hindi and Malayalam across four tiers: studio, broadcast, spontaneous, and synthetic noise. Through a controlled study of learning-rate timing and curriculum ordering, we find that early large parameter updates improve global WER by 12 absolute points, while a hard-to-easy curriculum adds gains for spontaneous speech. These findings motivate reverse multi-stage fine-tuning (R-MFT), a training recipe that enables a parameter-efficient 244M Whisper model to match or exceed conventionally fine-tuned 769M counterparts. Representational analysis via CKA and SVD reveals effective schedules concentrate adaptation in the decoder, preserving the pre-trained encoder's acoustic geometry. We release the benchmark and models.

Figures

Figures reproduced from arXiv: 2605.13087 by Aditya Srinivas Menon, Arghya Bhattacharya, Kavya Manohar, Kumarmanas Nethil, Kush Juvekar.

Figure 1
Figure 1. Figure 1: Comparison of training curricula. Both use identical decreasing LR schedules. R-MFT (right) places spontaneous data in the high-LR phase. data, model architecture, and optimizer configuration constant. Our findings challenge the status quo: we demonstrate that re￾versing standard heuristics—applying high-magnitude updates initially on the hardest data—yields drastic WER improvements on identical data distr… view at source ↗
Figure 2
Figure 2. Figure 2: shows training loss for the Malayalam Whisper￾medium model (representative; Hindi and Whisper-small ex￾hibit identical trends). The conservative LR (1e−5) plateaus within the first 7K steps at a loss an order of magnitude higher than the 2e−4 schedule, consistent with the hypothesis that the pre-trained prior creates a deep, narrow basin from which small gradients cannot escape [PITH_FULL_IMAGE:figures/fu… view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

29 extracted references · 29 canonical work pages · 1 internal anchor

  1. [1]

    However, zero-shot word er- ror rates (WER) for many Indic languages often exceed 100%

    Introduction Large-scale weakly supervised pre-training has brought auto- matic speech recognition (ASR) close to human performance in high-resource languages [1, 2]. However, zero-shot word er- ror rates (WER) for many Indic languages often exceed 100%. Fine-tuned models such as IndicWhisper [3] reduce this gap, but are trained primarily on studio-record...

    work page

  2. [2]

    Related Work Indic ASR corpora and benchmarks.Open-source datasets for Indian languages have expanded rapidly. Kathbath [8] pro- arXiv:2605.13087v1 [cs.CL] 13 May 2026 vides large-scale read speech, Shrutilipi [9] broadcast news transcriptions, and Indic V oices [10] crowdsourced sponta- neous speech. Benchmarks such as Vistaar [3] evaluate models across ...

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  3. [3]

    The Vividh-ASR Benchmark Vividh-ASR is a diagnostic benchmark organized byacous- tic and prosodic complexityrather than by domain. It targets Hindi and Malayalam, representing the Indo-Aryan and Dra- vidian language families respectively, and aggregates data from Kathbath [8], Shrutilipi [9], Indic V oices [10], FLEURS [17], and additional publicly availa...

    work page

  4. [4]

    However, when adapting to low-resource languages with complex phonotactics, the model Table 1:Data distribution in hours

    Methodology Standard Whisper fine-tuning relies on conservative learning rates (1e−5) under the assumption that large updates will de- stroy the pre-trained priors [19]. However, when adapting to low-resource languages with complex phonotactics, the model Table 1:Data distribution in hours. The corpus is intention- ally weighted toward spontaneous speech ...

    work page

  5. [5]

    Learning Rate Effect Figure 2 shows training loss for the Malayalam Whisper- medium model (representative; Hindi and Whisper-small ex- hibit identical trends)

    Results 5.1. Learning Rate Effect Figure 2 shows training loss for the Malayalam Whisper- medium model (representative; Hindi and Whisper-small ex- hibit identical trends). The conservative LR (1e−5) plateaus within the first 7K steps at a loss an order of magnitude higher than the2e−4schedule, consistent with the hypothesis that the pre-trained prior cre...

    work page

  6. [6]

    over-parameterization

    Analysis We hypothesize that successful adaptation to low-resource In- dic phonotactics requires a structural asymmetry: learning new linguistic priors in the decoder while preserving the pre-trained Table 4:WER (%) across Vividh-ASR tiers for Malayalam (Mal) and Hindi (Hi). Best per-column inbold.†: uses less Tier C training data than our models (see tex...

    work page

  7. [7]

    Conclusion We introduced Vividh-ASR, a complexity-tiered benchmark designed to diagnosestudio-biasin Indic speech recognition. Through a2×2factorial study on Hindi and Malayalam, we demonstrated that optimization plasticity dominates curriculum ordering: early large parameter updates yield∼12 absolute WER points of improvement, while a hard-to-easy curric...

    work page

  8. [8]

    All final content was re- viewed, verified, and approved by the authors, who take full responsibility for the integrity of the research and its presenta- tion

    Generative AI Use Disclosure The authors utilized large language model (LLM) tools, specif- ically Gemini 2.5 Pro, to assist in the linguistic refinement and technical polishing of the manuscript. All final content was re- viewed, verified, and approved by the authors, who take full responsibility for the integrity of the research and its presenta- tion

    work page

  9. [9]

    Robust speech recognition via large-scale weak supervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” inInternational conference on machine learning. PMLR, 2023, pp. 28 492–28 518

    work page 2023

  10. [10]

    Unsupervised cross-lingual representation learning for speech recognition,

    A. Conneau, A. Baevski, R. Collobert, A. Mohamed, and M. Auli, “Unsupervised cross-lingual representation learning for speech recognition,” inInterspeech, 2021

    work page 2021

  11. [11]

    Vistaar: Diverse Benchmarks and Training Sets for Indian Language ASR,

    K. Bhogale, S. Sundaresan, A. Raman, T. Javed, M. M. Khapra, and P. Kumar, “Vistaar: Diverse Benchmarks and Training Sets for Indian Language ASR,” inInterspeech 2023, 2023, pp. 4384– 4388

    work page 2023

  12. [12]

    whisper-finetune: Hyperparameter tun- ing,

    V . Lodagala, “whisper-finetune: Hyperparameter tun- ing,” https://github.com/vasistalodagala/whisper-finetune# hyperparameter-tuning, 2024, gitHub repository, accessed 2026-03-05

    work page 2024

  13. [13]

    Overcoming catastrophic forgetting in neural networks,

    J. e. a. Kirkpatrick, “Overcoming catastrophic forgetting in neural networks,” inPNAS, 2017

    work page 2017

  14. [14]

    Curriculum learning,

    Y . Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learning,” inICML, 2009

    work page 2009

  15. [15]

    Similarity of neural network representations revisited,

    S. Kornblith, M. Norouzi, H. Lee, and G. Hinton, “Similarity of neural network representations revisited,” inInternational Con- ference on Machine Learning (ICML). PMLR, 2019, pp. 3519– 3529

    work page 2019

  16. [16]

    Indicsuperb: A speech processing universal performance benchmark for indian languages,

    T. Javed, K. S. Bhogale, A. Raman, A. Kunchukuttan, P. Kumar, and M. M. Khapra, “Indicsuperb: A speech processing universal performance benchmark for indian languages,” 2022. [Online]. Available: https://arxiv.org/abs/2208.11761

    work page arXiv 2022

  17. [17]

    Effectiveness of mining au- dio and text pairs from public data for improving asr systems for low-resource languages,

    K. Bhogale, A. Raman, T. Javed, S. Doddapaneni, A. Kunchukut- tan, P. Kumar, and M. M. Khapra, “Effectiveness of mining au- dio and text pairs from public data for improving asr systems for low-resource languages,” inIcassp 2023-2023 ieee international conference on acoustics, speech and signal processing (icassp). IEEE, 2023, pp. 1–5

    work page 2023

  18. [18]

    Indicvoices: Towards building an inclusive multilingual speech dataset for indian languages,

    T. Javed, J. Nawale, E. George, S. Joshi, K. Bhogale, D. Mehen- dale, I. Sethi, A. Ananthanarayanan, H. Faquih, P. Palitet al., “Indicvoices: Towards building an inclusive multilingual speech dataset for indian languages,” inFindings of the Association for Computational Linguistics: ACL 2024, 2024, pp. 10 740–10 782

    work page 2024

  19. [19]

    Esb: A benchmark for multi-domain end-to-end speech recognition,

    S. Gandhi, P. V on Platen, and A. M. Rush, “Esb: A benchmark for multi-domain end-to-end speech recognition,”arXiv preprint arXiv:2210.13352, 2022

    work page arXiv 2022

  20. [20]

    Cba-whisper: Curriculum learning-based adalora fine-tuning on whisper for low-resource dysarthric speech recognition,

    T. Tan, X. Chen, X. Le, W. Fan, X. Xia, C. Huang, and J. Lu, “Cba-whisper: Curriculum learning-based adalora fine-tuning on whisper for low-resource dysarthric speech recognition,” inInter- speech 2025, 2025, pp. 3309–3313

    work page 2025

  21. [21]

    Task-informed anti- curriculum by masking improves downstream performance on text,

    A. Jarca, F. A. Croitoru, and R. T. Ionescu, “Task-informed anti- curriculum by masking improves downstream performance on text,”arXiv preprint arXiv:2502.12953, 2025, accepted at ACL 2025

    work page arXiv 2025

  22. [22]

    Layer-wise analysis of a self-supervised speech representation model,

    A. Pasad, J.-C. Chou, and K. Livescu, “Layer-wise analysis of a self-supervised speech representation model,” inIEEE Auto- matic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2021, pp. 914–921

    work page 2021

  23. [23]

    Comparative layer-wise analy- sis of self-supervised speech models,

    A. Pasad, B. Shi, and K. Livescu, “Comparative layer-wise analy- sis of self-supervised speech models,” inIEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5

    work page 2023

  24. [24]

    What happens to BERT embeddings during fine-tuning?

    A. Merchant, E. Rahimtoroghi, E. Pavlick, and I. Tenney, “What happens to BERT embeddings during fine-tuning?” inProceed- ings of the Third BlackboxNLP Workshop on Analyzing and Inter- preting Neural Networks for NLP, 2020, pp. 33–44

    work page 2020

  25. [25]

    Fleurs: Few-shot learning evaluation of universal representations of speech,

    A. Conneau, M. Ma, S. Khanuja, Y . Zhang, V . Axelrod, S. Dalmia, J. Riesa, C. Rivera, and A. Bapna, “Fleurs: Few-shot learning evaluation of universal representations of speech,” arXiv preprint arXiv:2205.12446, 2022. [Online]. Available: https://arxiv.org/abs/2205.12446

    work page arXiv 2022

  26. [26]

    Re- sources for Indian languages,

    A. Baby, A. L. Thomas, N. Nishanthi, T. Consortiumet al., “Re- sources for Indian languages,” inProceedings of Text, Speech and Dialogue. CBBLR Workshop, 2016

    work page 2016

  27. [27]

    Enhancing whisper’s accu- racy and speed for indian languages through prompt-tuning and tokenization,

    K. Tripathi, R. Gothi, and P. Wasnik, “Enhancing whisper’s accu- racy and speed for indian languages through prompt-tuning and tokenization,” inICASSP 2025-2025 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5

    work page 2025

  28. [28]

    A metric for distributions with applications to image databases,

    Y . Rubner, C. Tomasi, and L. Guibas, “A metric for distributions with applications to image databases,” inSixth International Con- ference on Computer Vision (IEEE Cat. No.98CH36271), 1998, pp. 59–66

    work page 1998

  29. [29]

    Conformer: Convolution-augmented transformer for speech recognition,

    A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y . Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y . Wu, and R. Pang, “Conformer: Convolution-augmented transformer for speech recognition,” in Interspeech, 2020, pp. 5036–5040

    work page 2020