Vedavani: A Benchmark Corpus for ASR on Vedic Sanskrit Poetry

Abhinay Beerukuri; Manoj Balaji Jagadeeshan; Pawan Goyal; Pretam Ray; Shrey Kamoji; Sujeet Kumar

REVIEW 3 major objections 7 minor 13 references

Vedavani: A Benchmark Corpus for ASR on Vedic Sanskrit Poetry

T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Vedavani, a 54-hour corpus of Vedic Sanskrit poetry, cuts ASR word error from roughly 100 percent to about 23 percent.

desk verdict Vedavani fills a real gap with a public 54-hour Vedic poetry ASR dataset, but the unvalidated manual alignment in §2.3 makes the benchmark numbers provisional until checked. read the letter →

arxiv 2506.00145 v1 pith:XPL4I4SI submitted 2025-05-30 cs.CL cs.SDeess.AS

Sujeet Kumar , Pretam Ray , Abhinay Beerukuri , Shrey Kamoji , Manoj Balaji Jagadeeshan , Pawan Goyal This is my paper · ORCID

classification cs.CLcs.SDeess.AS

keywords VedicSanskritautomaticspeechrecognitioncorpuspoetryRigVedaAtharvaIndicWhisperlow-resourceASR

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Vedavani, a 54-hour benchmark corpus of 30,779 labelled audio samples drawn from Rig Veda and Atharva Veda recitations, and argues it is the first ASR dataset aimed specifically at Sanskrit Vedic poetry. Existing Sanskrit ASR work has focused on prose, and off-the-shelf multilingual models fail on this poetic register, with zero-shot WER around 99 to 110. After fine-tuning on Vedavani, the best model, IndicWhisper, reaches a word error rate of 23.14 on IAST-script test data and 21.76 on Devanagari. The paper's central claim is that the corpus, with its manually aligned verse-level audio–text pairs, makes Vedic poetry a tractable ASR target and gives future systems a standard benchmark.

What carries the argument

The load-bearing object is Vedavani itself: 30,779 verse-level audio–text pairs totalling 54 hours, drawn from Internet Archive recitations of the Rig Veda (20,782 verses) and Atharva Veda (9,997 verses), with text from Wikisource, split into 80% training, 10% validation, and 10% test. Its distinctive feature is manual alignment: each audio chunk was listened to and matched to its transcript after automated silence-based segmentation failed, with segments under 0.25 seconds removed and 500 ms of silence padded. Around this corpus, the benchmark machinery consists of fine-tuning encoder-only models (XLSR, MMS, HuBERT, IndicWav2Vec, and the Sanskrit-tuned SPRING-INX variants, optionally with KenLM n-gram language models) and encoder-decoder Whisper variants (small, medium, large, Distil-Whisper, and IndicWhisper), evaluated in SLP1 transliteration with word error rate and character error rate. The manual alignment is what lets the corpus function as both training data and a trustworthy test set.

What would settle it

Take a random sample of, say, 200 Vedavani test clips, have two independent Sanskrit-literate annotators re-align each clip to the source text from the audio alone, and measure agreement with the released alignment; if agreement is low, or if the recitation is found to be a different rescension than the Wikisource text, the reported WERs and CERs are not trustworthy as stated.

Watch

Extended reading notes

Core claim

On its own terms, this paper establishes that Vedic poetry is a distinct ASR register that standard models cannot touch zero-shot: the four existing Sanskrit and multilingual systems tested before adaptation give word error rates of 99.84 to 109.80 and character error rates of 39.38 to 46.05. Fine-tuning on Vedavani pulls the best results down to word error rates of 20.71 to 23.14 for the Whisper family, with IndicWhisper the best overall on the paper's headline comparison while Whisper large is marginally ahead on Devanagari-script WER. Encoder-only models improve further when an n-gram language model is added, by an average of 8.87 WER points. The remaining errors cluster into phonetic confusions such as sibilant and short-long vowel substitutions, structural errors such as sandhi splitting and merging, and intonational discrepancies, which the authors identify as the next target.

Load-bearing premise

The whole benchmark's numbers stand on the accuracy of the manual text–audio alignment: if the Internet Archive recitations do not match the Wikisource texts, or if hand alignment is systematically off, every training and test transcript is wrong and every reported word error and character error changes.

Editorial extensions

If this is right

Existing multilingual and Sanskrit ASR systems cannot handle Vedic poetry zero-shot, so any practical Vedic ASR must be fine-tuned on in-register data.
A 54-hour, manually aligned verse-level corpus is enough to reduce word error on this register from roughly 100 percent to the 20–30 percent range, making data quantity secondary to alignment quality and transcript fidelity.
Devanagari-script fine-tuning is consistently better than IAST for Whisper-family models by an average of 4.03 WER points, so script choice is a practical lever for Sanskrit ASR.
Adding an n-gram language model improves encoder-only models by an average of 8.87 WER and 1.19 CER, showing that acoustic models alone leave systematic lexical error on the table.
Vedavani gives the field a common benchmark on Vedic poetry, making future model comparisons meaningful and opening the way to intonation-aware ASR.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the dataset was built from a single recitation source, its numbers should be read as performance on that chanting style; recording additional reciters or regional styles would test whether fine-tuned models generalise across Vedic prosody rather than memorise one voice.
The paper's error taxonomy isolates sandhi splitting and merging as a structured failure mode; tagging the reference transcripts with sandhi boundaries would let future work decompose word error into acoustic error versus morphophonemic segmentation error.
The planned intonation markers could turn Vedavani into a prosody benchmark as well: one could test whether Whisper-family models can be adapted to recognise the three Vedic accents, a task that word error and character error metrics do not capture.
Since alignment was done manually with no reported inter-annotator agreement, a natural next step is a small re-alignment study in which two or three annotators independently align a random sample of clips, producing a confidence interval around every reported WER.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First thing to know: Vedavani is the first poetry-specific Sanskrit ASR benchmark, 54 hours, ~31k labelled verses, with code and data public. That alone makes it a useful resource for the Sanskrit speech community. The paper also does something mildly unusual: it benchmarks a wide set of encoder-only and encoder-decoder models, with and without language models, and reports WER/CER on both Devanagari and IAST scripts. The public release is a real contribution; the data collection pipeline, while standard, is documented.

What worried me is the same thing the stress-test flags: §2.3. The authors manually aligned each audio chunk to a transcript scraped from Wikisource, but they report no inter-annotator agreement, no second-pass verification, and no check that the Internet Archive recitation follows the same recension as the printed text. Vedic recitation varies across shakhas, so this is not academic hand-wringing. If the alignment is systematically wrong, every WER and CER in Tables 3 and 4 changes, and the 'IndicWhisper best' headline may not survive. That said, nothing in the paper suggests the alignment is wrong; it is just unvalidated. This is a fixable flaw, not a dead end.

The smaller soft spots are real but minor. There are numeric inconsistencies: Table 4 lists Whisper-S Devanagari WER as 28.77, the text says 28.88; and the claimed '0.05 difference' between IndicWhisper and Whisper-L on Devanagari WER is actually 1.05 (21.76 vs 20.71). The paper reports no error bars or significance tests, so differences of a few WER points should not be over-read. The abstract's claim about 'precise prosodic and rhythmic features' is also not demonstrated; the dataset is just audio with text, no prosody annotations.

Overall, this is a solid resource paper with a clear gap to fill. The central claim—new poetry-specific benchmark—holds up. The numbers should be treated as provisional until alignment is independently checked. I'd send it to peer review, with the requirement that the authors either validate the alignment with a second annotator or report inter-annotator agreement on a subset, and fix the numeric inconsistencies. If they do that, it becomes a citable benchmark. Quick read, worth the hour.

Referee Report

3 major / 7 minor

Summary. The paper introduces Vedavani, a 54-hour ASR corpus of 30,779 labeled audio samples from Vedic Sanskrit poetry, drawn from the Rig Veda and Atharva Veda. Audio was obtained from Internet Archive recitations and text from Sanskrit Wikisource; the authors manually aligned each audio chunk to its transcript. The paper reports corpus statistics and splits, fine-tunes a range of encoder-only models (XLSR, MMS, HuBERT, IndicWav2Vec, and SPRING-INX variants) with KenLM rescoring, and fine-tunes Whisper variants in IAST and Devanagari scripts. The headline result is that IndicWhisper performs best among the benchmarked SOTA models, mainly on IAST; on Devanagari the comparison is closer and Whisper-Large achieves a lower WER. The paper closes with a WER/CER distribution analysis and a qualitative error taxonomy.

Significance. If the manual alignments are accurate, Vedavani is a genuinely useful resource: it is, to my knowledge, the first ASR benchmark aimed specifically at Vedic poetry, it is publicly released, and it provides a reproducible standard evaluation over existing multilingual speech models. The paper makes no parametric claims and fits no free parameters, so circularity is not a concern. The empirical conclusions are, however, only as trustworthy as the alignment described in Section 2.3, and the reported numbers contain internal inconsistencies. The dataset and baseline results are potentially valuable, but the alignment-validation gap and the numeric errors must be addressed before the benchmark claims can be relied upon.

major comments (3)

[§2.3] The load-bearing premise of every WER/CER in Tables 1, 3, and 4 is that the manually aligned text-audio pairs are correct. The paper reports no inter-annotator agreement, no second-pass verification, and no check that the Internet Archive recordings follow the same recension or edition as the Wikisource text. Since Vedic recitations differ across shakhas in wording, sandhi, and verse ordering, a systematic alignment mismatch would change all reported metrics and the 'IndicWhisper performed best' claim. Please add quantitative validation, such as independent re-annotation of a random sample with agreement statistics, and a recension-level consistency check between the audio source and the text source.
[§3.1 / Table 4] The results text is inconsistent with Table 4: the text states that Whisper-Small on Devanagari achieves a WER of 28.88, while the table lists 28.77; the text states that Whisper-Large outperforms IndicWhisper on Devanagari WER by 'just a 0.05 difference', but the table values are 20.71 and 21.76, a 1.05-point difference. These numbers must be corrected, and the abstract's blanket claim that 'IndicWhisper performed the best' should be qualified, because on the Devanagari script Whisper-Large has the lower WER.
[§3 / Tables 2–4] The preprocessing description changes the dataset from the 24,623/3,078/3,078 split in Table 2 to 24,590/3,073/3,075 after removing audio files over 30 seconds, but the paper does not state whether the zero-shot results in Table 1 and the encoder-only results in Table 3 were evaluated on the same filtered test set as the encoder-decoder results in Table 4. Please report the exact test set used for each table. In addition, several headline differences are small (e.g., SPG-INXS-MMS vs SPG-INXS-W2V with LM, and Whisper-Large vs IndicWhisper on Devanagari) and no confidence intervals, significance tests, or repeated-run variation are reported; the benchmarking conclusions would be more robust with this information.

minor comments (7)

[§1, §3, Table 3] The manuscript uses 'SPG-INXS' and 'SPRING-INX' interchangeably for the same models; please unify the naming.
[§2.1] The text says the data were obtained from 'Wikipedia', but the URLs point to Sanskrit Wikisource; please cite the actual source consistently.
[Throughout] The spelling 'Devanagri' should be 'Devanagari' (e.g., in Table 4 and Section 3).
[§4 / Table 6] Several Sanskrit examples in Tables 6 and 7 appear as garbled glyphs in the PDF; please ensure proper Unicode rendering so the error examples are legible.
[§2.3] The statement that 500 ms of silence was added 'for aesthetic purposes' should be rephrased as an audio-processing decision, and the paper should state whether that padding is included in the reported segment durations.
[§1] The 'first comprehensive ASR study focused on Sanskrit Vedic poetry' claim should be supported by a brief comparison of the genre composition of the existing Sanskrit corpora cited in Section 1 (e.g., Vakysancayah, Shrutilipi, Kathbath), rather than asserted without contrast.
[§2.3 / §3] There are a few grammatical and typographical errors, such as 'exhibite' in Section 1 and 'The dataset have' in the statistics paragraph; these should be corrected in revision.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Vedavani is an empirical benchmark corpus with no derivation chain and no fitted parameter masquerading as a prediction.

full rationale

The paper's central contribution is a manually constructed dataset and an empirical comparison of pretrained ASR models. There is no mathematical derivation, no fitted parameter that is later renamed as a prediction, and no uniqueness theorem imported from the authors' prior work. The load-bearing premise, the §2.3 manual text-audio alignment, is a data-construction step, not a circular argument: label accuracy affects all WER/CER numbers, but that is a validity or reliability limitation (unvalidated alignment, no inter-annotator agreement), not a circularity. The few co-authored citations (Krishna et al. 2018, 2021; Adiga et al. 2021) are background context for word-order and prior Sanskrit ASR work; the reported benchmark results are computed against external model checkpoints (SPRING-INX, Whisper, IndicWhisper, etc.) and are not forced by any self-citation or by construction. Concerns about recension mismatch between the Internet Archive audio and Wikisource text are correctness risks and should be scored under data validation, not circularity. Therefore the appropriate finding is no significant circularity (score 0).

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central contribution is a dataset, not a mathematical derivation, so there are no fitted constants. The load-bearing assumptions are about data provenance and label quality, not about equations.

assumptions (3)

domain assumption The Internet Archive recitations by Veda Prasara Samithi are faithful, complete renderings of the Rig Veda and Atharva Veda texts scraped from Wikipedia.
Section 2.2 and 2.3; mismatches between audio and text would corrupt labels, and no verification is reported.
domain assumption Manual text-audio alignment is correct for all 30,779 samples.
Section 2.3; the data quality rests entirely on this manual process, without inter-annotator checks.
domain assumption Computing WER and CER on SLP1 transliteration captures the intended evaluation.
Section 3; transliteration to SLP1 is a modeling choice and could introduce errors, though it is a reasonable standard.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Vedavani: A Benchmark Corpus for ASR on Vedic Sanskrit Poetry." pith.science (2026). https://pith.science/paper/XPL4I4SI

@misc{pith2026250600145,
  author       = {Pith},
  title        = {Pith review of: Vedavani: A Benchmark Corpus for ASR on Vedic Sanskrit Poetry},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XPL4I4SI}},
  note         = {Machine review of arXiv:2506.00145}
}

abstract

Sanskrit, an ancient language with a rich linguistic heritage, presents unique challenges for automatic speech recognition (ASR) due to its phonemic complexity and the phonetic transformations that occur at word junctures, similar to the connected speech found in natural conversations. Due to these complexities, there has been limited exploration of ASR in Sanskrit, particularly in the context of its poetic verses, which are characterized by intricate prosodic and rhythmic patterns. This gap in research raises the question: How can we develop an effective ASR system for Sanskrit, particularly one that captures the nuanced features of its poetic form? In this study, we introduce Vedavani, the first comprehensive ASR study focused on Sanskrit Vedic poetry. We present a 54-hour Sanskrit ASR dataset, consisting of 30,779 labelled audio samples from the Rig Veda and Atharva Veda. This dataset captures the precise prosodic and rhythmic features that define the language. We also benchmark the dataset on various state-of-the-art multilingual speech models.$^{1}$ Experimentation revealed that IndicWhisper performed the best among the SOTA models.

Figures

Figures reproduced from arXiv: 2506.00145 by the authors.

**Figure 1.** Statistical analysis of Sanskrit audio and text data, including the distribution of audio duration, and number of characters per sentence. All frequencies are presented on a logarithmic scale. characters, with variations observed across the dataset. The dataset have a rich vocabulary of 64,082 unique words, ranging from single-character words to those with 59 characters. These metrics provide insight into the datase… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 10 canonical work pages

[1]

[Adiga et al.2021] Devaraja Adiga, Rishabh Kumar, Amrith Krishna, Preethi Jyothi, Ganesh Ramakrish- nan, and Pawan Goyal

work page 2021
[8]

In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 12942–12950

Indicsuperb: A speech processing universal performance benchmark for indian languages. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 12942–12950. [Krishna et al.2018] Amrith Krishna, Bishal Santra, Sasi Prasanth Bandaru, Gaurav Sahu, Vishnu Dutt Sharma, Pavankumar Satuluri, and Pawan Goyal

work page 2018
[11]

Journal of Machine Learning Research , 25(97):1–52

Scaling speech technology to 1,000+ languages. Journal of Machine Learning Research , 25(97):1–52. [Radford et al.2023] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever

work page 2023
[13]

Ctc-based end-to-end speech recog- nition for low resource language sanskrit. In 2023 26th Conference of the Oriental COCOSDA Inter- national Committee for the Co-ordination and Standardisation of Speech Databases and Assessment Techniques (O-COCOSDA), pages 1–5. [Wright1968] J. C. Wright

work page 2023
[20]

[Suhani et al.2023] Suhani, Amita Dev, and Poonam Bansal

work page 2023
[2011]

In Proceed- ings of the Sixth Workshop on Statistical Machine Translation , pages 187–197, Edinburgh, Scotland, July

KenLM: Faster and smaller language model queries. In Proceed- ings of the Sixth Workshop on Statistical Machine Translation , pages 187–197, Edinburgh, Scotland, July. Association for Computational Linguistics. [Hsu et al.2021] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed

work page 2021
[2018]

[Krishna et al.2021] Amrith Krishna, Bishal Santra, Ashim Gupta, Pavankumar Satuluri, and Pawan Goyal

Free as in free word order: An energy based model for word segmentation and morphological tagging in sanskrit. [Krishna et al.2021] Amrith Krishna, Bishal Santra, Ashim Gupta, Pavankumar Satuluri, and Pawan Goyal

work page 2021
[2019]

In 2019 2nd International Conference on Intelligent Computing, Instrumentation and Control Technologies (ICICICT), volume 1, pages 1146–1151

Automatic speech recognition for sanskrit. In 2019 2nd International Conference on Intelligent Computing, Instrumentation and Control Technologies (ICICICT), volume 1, pages 1146–1151. IEEE. [Bhogale et al.2023a] Kaushal Bhogale, Abhigyan Raman, Tahir Javed, Sumanth Doddapaneni, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh M Khapra. 2023a. Effectiveness...

arXiv 2019

Show all 13 references

[2020]

arXiv preprint arXiv:2006.13979

Unsupervised cross-lingual representation learning for speech recognition. arXiv preprint arXiv:2006.13979 . [Gandhi et al.2023] Sanchit Gandhi, Patrick von Platen, and Alexander M Rush

2006 arXiv
[2021]

IEEE/ACM transactions on audio, speech, and lan- guage processing, 29:3451–3460

Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and lan- guage processing, 29:3451–3460. [Javed et al.2022] Tahir Javed, Sumanth Doddapaneni, Abhigyan Raman, Kaushal Santosh Bhogale, Gowtham R...

2022
[2022]

In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 10813–10821

Towards building asr systems for the next billion users. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 10813–10821. [Javed et al.2023] Tahir Javed, Kaushal Bhogale, Abhigyan Raman, Pratyush Kumar, Anoop Kunchukut- tan, and Mitesh M Khapra

2023
[2023]

arXiv preprint arXiv:2311.00430

Distil-whisper: Robust knowledge distillation via large-scale pseudo labelling. arXiv preprint arXiv:2311.00430 . [Heafield2011] Kenneth Heafield

arXiv
[2024]

In 2024 IEEE Spoken Language Technology Workshop (SLT), pages 1217–1223

Inx-speakerhub: A 2000-hour indian multiligual speaker verification corpus. In 2024 IEEE Spoken Language Technology Workshop (SLT), pages 1217–1223. IEEE. [Pratap et al.2024] Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoh...

2000

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

[Adiga et al.2021] Devaraja Adiga, Rishabh Kumar, Amrith Krishna, Preethi Jyothi, Ganesh Ramakrish- nan, and Pawan Goyal

work page 2021

[2] [8]

In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 12942–12950

Indicsuperb: A speech processing universal performance benchmark for indian languages. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 12942–12950. [Krishna et al.2018] Amrith Krishna, Bishal Santra, Sasi Prasanth Bandaru, Gaurav Sahu, Vishnu Dutt Sharma, Pavankumar Satuluri, and Pawan Goyal

work page 2018

[3] [11]

Journal of Machine Learning Research , 25(97):1–52

Scaling speech technology to 1,000+ languages. Journal of Machine Learning Research , 25(97):1–52. [Radford et al.2023] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever

work page 2023

[4] [13]

Ctc-based end-to-end speech recog- nition for low resource language sanskrit. In 2023 26th Conference of the Oriental COCOSDA Inter- national Committee for the Co-ordination and Standardisation of Speech Databases and Assessment Techniques (O-COCOSDA), pages 1–5. [Wright1968] J. C. Wright

work page 2023

[5] [20]

[Suhani et al.2023] Suhani, Amita Dev, and Poonam Bansal

work page 2023

[6] [2011]

In Proceed- ings of the Sixth Workshop on Statistical Machine Translation , pages 187–197, Edinburgh, Scotland, July

KenLM: Faster and smaller language model queries. In Proceed- ings of the Sixth Workshop on Statistical Machine Translation , pages 187–197, Edinburgh, Scotland, July. Association for Computational Linguistics. [Hsu et al.2021] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed

work page 2021

[7] [2018]

[Krishna et al.2021] Amrith Krishna, Bishal Santra, Ashim Gupta, Pavankumar Satuluri, and Pawan Goyal

Free as in free word order: An energy based model for word segmentation and morphological tagging in sanskrit. [Krishna et al.2021] Amrith Krishna, Bishal Santra, Ashim Gupta, Pavankumar Satuluri, and Pawan Goyal

work page 2021

[8] [2019]

In 2019 2nd International Conference on Intelligent Computing, Instrumentation and Control Technologies (ICICICT), volume 1, pages 1146–1151

Automatic speech recognition for sanskrit. In 2019 2nd International Conference on Intelligent Computing, Instrumentation and Control Technologies (ICICICT), volume 1, pages 1146–1151. IEEE. [Bhogale et al.2023a] Kaushal Bhogale, Abhigyan Raman, Tahir Javed, Sumanth Doddapaneni, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh M Khapra. 2023a. Effectiveness...

arXiv 2019