Seeds Before Objectives: Rethinking Evaluation for Low-Resource Garhwali ASR

Ashima Sood; Jasmeet Singh; Karamvir Singh Batra; Prathamjyot Singh; Sahil Sharma

REVIEW 1 major objections 4 minor 14 references

Seeds Before Objectives: Rethinking Evaluation for Low-Resource Garhwali ASR

T0 review · 1 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that single-run comparisons are unreliable in low-resource dialectal ASR and proves it for Garhwali: under a five-seed protocol, Focal CTC, a matra-weighted objective, and Hindi-to-Garhwali transfer all fail to beat…

desk verdict Solid multi-seed Garhwali benchmark with honest null results for Focal CTC, matra weighting, and Hindi transfer; the 'pretraining design, not parameter count' claim is overreach from a confounded four-model comparison. read the letter →

arxiv 2608.10670 v1 pith:UZEXC72J submitted 2026-08-11 cs.CL

Karamvir Singh Batra , Prathamjyot Singh , Ashima Sood , Jasmeet Singh , Sahil Sharma This is my paper · ORCID

classification cs.CL

keywords low-resourceASRdialectalspeechrecognitionGarhwalimulti-seedevaluationCTCfocallosscross-lingualtransferreproducibility

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

At the small corpus sizes typical of low-resource dialects, a single training run cannot tell whether an ASR change helps. The paper shows this for Garhwali, building the first reproducible multi-seed benchmark on the official released splits and testing three plausible interventions: Focal CTC (a loss that down-weights easy examples), a matra-weighted objective that up-weights vowel-diacritic errors, and Hindi-to-Garhwali transfer. All three dissolve into seed noise: once results are averaged over five seeds with paired significance testing, none beats standard CTC, and the matra-weighted loss does not even reduce the vowel-diacritic errors it was built to attack. What survives is ordinary: a 580M-parameter w2v-BERT 2.0 encoder reaches 47.0% word error rate while beating larger multilingual models, and speed augmentation gives a small, largely consistent gain. The broader point a reader can take away is that multi-seed reporting with per-seed outputs and significance tests is the minimum needed to trust claimed gains in low-resource dialectal ASR.

What carries the argument

The central mechanism is the multi-seed paired evaluation protocol: every primary system is fine-tuned over five fixed seeds (42, 123, 777, 2025, 1234) on the same official train/validation/test splits, with per-seed WER and CER reported, paired Wilcoxon signed-rank tests with Holm-Bonferroni correction, and a post-hoc power analysis that states how many seeds a real effect would require. This is what carries the argument, because the within-system seed spread (1.6 WER points for standard CTC) is comparable to or larger than the gaps between objectives (0.4 to 0.8 points). The design makes visible what a single lucky seed can hide: the best Focal seed and the best standard seed are within noise of one another, while the multi-seed means are not.

What would settle it

Run the three interventions and standard CTC under the same official splits but with 20 seeds; if any intervention's Holm-corrected paired test gives a significantly lower mean WER than standard CTC, the paper's central claim that these objectives do not reliably help is falsified.

Watch

Extended reading notes

Core claim

Stated on the paper's own terms: on the official splits, with five seeds per system, standard CTC is the floor that the trained objectives cannot beat. The five-seed mean is 47.0% WER for standard CTC, 47.83% for Focal CTC, 47.42% for the matra-weighted objective, and 47.22% for Hindi-to-Garhwali transfer; the matra objective's targeted error rate is essentially unchanged (22.26% versus 22.31%), and the only seed-consistent gap, standard versus focal, favors the baseline. Across encoders, w2v-BERT 2.0 (580M) beats MMS-1B (49.0%), XLS-R (50.2%), and HuBERT (60.9%), which the paper reads as evidence that pretraining design, not parameter count, drives performance. Speed augmentation lowers WER by 1.1 to 1.5 points under every objective, making it the only dependable intervention tested. The paper concludes that at this corpus size the residual bottleneck is representational, involving fine phonological contrasts such as vowel length and conjunct marking, rather than procedural, and that single-seed numbers in this regime should not be trusted as evidence of an effect.

Load-bearing premise

The load-bearing premise is that five seeds on one 8.8-hour corpus capture enough run-to-run noise to judge what is real; the paper's own power analysis shows the seed-level tests cannot reach $p < .05$ at that budget, so the negative results are statements about reliability rather than proof of absence.

Editorial extensions

If this is right

Future low-resource dialectal ASR results should report multi-seed means, per-seed values, and significance tests before claiming an objective or transfer benefit; single-run gains should be treated as hypotheses rather than results.
At Garhwali-scale data, objective engineering such as focal or class-weighted losses is unlikely to yield reliable gains, so effort should move to pretrained encoder choice, augmentation, and richer acoustic or data evidence.
Model comparisons from single runs should not be trusted: the same encoders swapped relative order between the two studies, so rankings need variance reporting to replicate.
Speed perturbation is a cheap, consistent improvement of roughly one to one and a half WER points across all objectives and should be part of the default training recipe.
Hindi-to-Garhwali transfer is not a dependable lever; direct fine-tuning is as good or better, so transfer studies need the same multi-seed scrutiny before claiming gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: if the representational-bottleneck reading is right, the next gains for Garhwali should come from better acoustic representations or additional unlabeled audio, not from further loss engineering; the paper leaves this as future work, but its own evidence points that way.
Beyond the paper: the same multi-seed protocol could be applied to other low-resource dialect benchmarks to check whether their reported single-run gains replicate; the paper argues the protocol transfers, but the specific null results are Garhwali-only.
Beyond the paper: the power analysis implies that a real effect smaller than about one WER point would need roughly 11 to 20 seeds to detect, so any future study claiming a small gain in this regime should budget its seed count accordingly.
Beyond the paper: the silence hallucination finding, where all eight true-silence segments receive spurious text across all five seeds, suggests a cheap and testable mitigation: voice-activity gating before decoding.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Read this paper if you care about low-resource dialectal ASR evaluation. It does what it advocates: all primary systems run over five seeds on the official VAANI splits, per-seed WER/CER values are in the appendix, paired Wilcoxon with Holm correction is applied, a bootstrap cross-check is included with its limitations spelled out, and the post-hoc power analysis is reported with the five-seed p-value floor disclosed. The null results for Focal CTC, matra-weighted CTC, and Hindi-to-Garhwali transfer are well supported by the per-seed tables; the authors even note that five-seed Wilcoxon cannot reach p<0.05 and rest their conclusions on paired differences, effect sizes, and the t-based power analysis. That is honest and reproducible. The divergence from the prior single-run Garhwali study, where encoder rankings swap between the two papers, makes the methodological point effectively. Speed augmentation is direction-consistent (13 of 15 seeds) and they correctly describe it as small and not statistically certified. The benchmark itself is genuinely useful; I would cite it. The main soft spot is contribution 3 and the abstract sentence 'pretraining design, not parameter count, drives performance.' The evidence is four checkpoints - w2v-BERT 2.0 (580M), MMS-1B, XLS-R 300M, and HuBERT Large - that differ simultaneously in architecture, pretraining corpus and coverage, tokenizer or quantization, and scale. No axis is varied alone. The observed ordering could be driven by pretraining data, architecture, or fine-tuning recipe just as easily as by 'design' versus 'parameter count'; the attribution is not identified. This overclaim does not damage the core negative results about objectives and transfer, which stand independently, but the abstract should be reworded to say that w2v-BERT 2.0 was the best encoder among those tested and that larger scale did not help across these checkpoints. A controlled scaling study within one model family would be needed to support the causal sentence. Other soft spots are minor and mostly self-flagged: the layer-wise probing is single-seed and exploratory; the Whisper reference is unstable with two failed seeds; the LLM-based error categorization is single-seed and only directional. They are labeled as such. The focal and matra hyperparameters were fixed before the multi-seed comparison, so there is no fitting-to-significance concern. Final take: this is a careful, well-scoped empirical paper with a real contribution as a reproducible benchmark and a useful caution about single-run reporting in this regime. The central null results hold up. The pretraining-design claim needs softening. I would send it to serious peer review with the expectation of minor-to-moderate revision; my main request as referee would be to fix the abstract and contribution 3, and to acknowledge the confound or add one controlled comparison.

Referee Report

1 major / 4 minor

Summary. This paper argues that at low-resource dialect corpus sizes, single-run ASR comparisons are unreliable, and it supports this by building a five-seed Garhwali benchmark on the official VAANI splits. Under this protocol, the paper reports that Focal CTC, a matra-weighted CTC objective, and Hindi-to-Garhwali transfer do not reliably improve over standard CTC, that speed augmentation gives a small and largely consistent mean gain, and that w2v-BERT 2.0 (580M) reaches the lowest WER among the compared encoders (47.0% over five seeds). The paper also provides per-seed outputs, Holm-corrected paired Wilcoxon tests, a bootstrap cross-check, a post-hoc power analysis, and a per-category error analysis indicating that matra and conjunct errors dominate.

Significance. If accepted with the proposed correction, this is a valuable methodological contribution to low-resource ASR evaluation. The study is exemplary in transparency: per-seed results are reported in the appendix, the five-seed Wilcoxon floor is disclosed in Section 3.2, the power analysis in Section 4.5 is honest about what the seed budget can and cannot establish, and code plus per-seed outputs are released. The central null results for the three objective/transfer interventions are empirically grounded and are worded as statistical rather than absolute. The main weakness is the encoder-comparison claim, which is not identified by the evidence as currently presented.

major comments (1)

[Abstract; §5.2, Table 7, Fig. 2] The claim that "pretraining design, not parameter count, drives performance" is not identified by the evidence offered. The four models in Table 7 and Figure 2 — w2v-BERT 2.0 (580M), MMS-1B (1B), XLS-R 300M, and HuBERT Large — differ simultaneously in architecture (w2v-BERT versus wav2vec2-style versus HuBERT), pretraining corpus and language coverage, tokenizer or quantization details, fine-tuning recipe, and parameter count. No axis is varied alone: there is no same-family checkpoint at a different scale, and no two checkpoints are matched on everything except pretraining design. The observed ordering could plausibly be driven by pretraining data, architecture, or scale rather than by pretraining design per se. A concrete test would be to run a same-family scaling comparison (e.g., XLS-R 300M versus XLS-R 2B under the same protocol, or a 1B w2v-BERT-style checkpoint) or to match parameter counts across architectures. Until such a comparison is added, the claim in the abstract and in Section 5.3 should be softened to, for example, "the best-performing checkpoint in our set is not the largest" or "pretraining design is a plausible factor, but scale and design are confounded in our comparison." This overclaim does not affect the multi-seed objective and transfer null results, which rest on Tables 2 and 3 and are supported by the per-seed data.

minor comments (4)

[Appendix E] The citation "(K et al., 2025)" should be expanded to the full author name used in the reference list ("Thennal D K et al., 2025") so the reference is unambiguous.
[Throughout] The text repeatedly uses the nonstandard ligature "oﬀicial"; please replace it with "official" for consistency with standard notation.
[Fig. 2] The broken y-axis is not explicitly labeled as broken in the figure; adding a clear break marker or a caption note would prevent misreading the vertical distances near WER 46–48.
[Table 10] The caption says percentages are "of error chunks," but the table lists both chunk counts and percentages; clarifying the denominator in the caption or a footnote would help readers avoid interpreting the percentages as row-level rates.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the multi-seed benchmark, objective comparisons, transfer test, and augmentation comparison are empirical evaluations against a held-out official test set, with no parameter fit renamed as a prediction and no load-bearing self-citation.

full rationale

The paper's central claims—that single-seed comparisons are unreliable at this corpus size, that neither Focal CTC nor the matra-weighted objective beats standard CTC under seed-level testing, that Hindi-to-Garhwali transfer gives no reliable gain, and that speed augmentation is the most consistent lever—are supported by held-out test-set evaluations on the official VAANI splits over five seeds. The objective hyperparameters (α=1.0, γ=0.5, τ=10, λ=0.3, class weights) are explicitly stated to have been fixed before the multi-seed comparison, so no result is obtained by fitting a quantity defined in terms of the claim. The matra-weighted objective is tested on the category it targets, but the matra error rate is an independent held-out measurement rather than the training loss or the class-weight input, so the test is not self-definitional. The comparison against Dhasmana et al. (2026) is external prior work by different authors, used as a baseline and as independent corroboration of the error profile; it does not carry the derivation. The 'pretraining design, not parameter count' conclusion is a confounded inference because the four compared models differ simultaneously in architecture, pretraining data, tokenizer, and scale, but this is a correctness risk, not a circular reduction: no equation equates an output to an input, and no load-bearing premise is justified solely by a self-citation. The post-hoc power analysis is explicitly descriptive and the paper explicitly notes the five-seed Wilcoxon floor and states its negative results are statistical rather than absolute. No specific circular step can be quoted, so the circularity score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The main hand-set inputs are the focal, matra, and speed-augmentation hyperparameters. The central empirical claims are benchmarked against an external held-out test set, but the pretraining-design-not-scale claim relies on an uncontrolled comparison across four heterogeneous models.

free parameters (5)

Focal CTC modulation strength gamma = 0.5
Chosen by hand before the multi-seed comparison; the null result for Focal CTC is conditional on this value and the temperature.
Focal CTC temperature tau = 10
Chosen to avoid underflow of exp(-ell/L); it defines the confidence p and therefore the focal weighting.
Matra auxiliary loss weight lambda = 0.3
Hand-set balance between the focal CTC base and the auxiliary class-weighted NLL; the matra objective null depends on it.
Matra/aspirated/retroflex class weights = 3.0/2.0/2.5
Hand-set weights in the auxiliary loss that emphasize phonologically salient tokens; different weights could change targeted error rates.
Speed augmentation rates = 0.9x, 1.0x, 1.1x
Chosen augmentation schedule tripling the training set; the positive result for speed augmentation is conditional on these rates and the 3x replication.

assumptions (4)

domain assumption The official VAANI train/validation/test split (4,778/666/450) is a valid, leakage-free benchmark for Garhwali ASR.
All comparisons use these splits; if the split or the normalization pipeline introduces artifacts, reported rankings and null results could change. See Section 3.1.
standard math Paired Wilcoxon signed-rank tests on five shared seeds are usable as a conservative descriptive summary despite the p-value floor.
The paper explicitly notes that p cannot reach 0.05 after Holm correction at n=5, so it rests conclusions on paired differences and power analysis rather than on the test alone. Section 3.2.
ad hoc to paper The four compared encoders are assumed to differ in ways attributable to pretraining design rather than to confounded architecture, data, and size.
The causal claim that pretraining design, not parameter count, drives performance requires this assumption, which is not controlled in the experiment. Section 5.2 and Figure 2.
domain assumption The three tested interventions are representative of objective engineering and cross-lingual transfer for low-resource dialectal ASR.
The paper generalizes to a practical message that objective engineering and transfer are not dependable levers; this depends on the chosen objective variants and the single transfer source (Hindi). Section 3.4 and Limitations.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Seeds Before Objectives: Rethinking Evaluation for Low-Resource Garhwali ASR." pith.science (2026). https://pith.science/paper/UZEXC72J

@misc{pith2026260810670,
  author       = {Pith},
  title        = {Pith review of: Seeds Before Objectives: Rethinking Evaluation for Low-Resource Garhwali ASR},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UZEXC72J}},
  note         = {Machine review of arXiv:2608.10670}
}

read the original abstract

At corpus sizes typical of low-resource dialects, single-run comparisons can yield gains that do not replicate. We show this for Garhwali, an under-resourced Indo-Aryan language of the central Himalaya, building the first reproducible multi-seed ASR benchmark on the official VAANI splits, with per-seed outputs and significance testing. Re-examining plausible gains, we find them fragile: neither Focal CTC nor a matra-weighted objective beats standard CTC under seed-level testing, the matra objective fails to cut even its targeted errors, and Hindi-to-Garhwali transfer gives no gain over direct fine-tuning. What holds up is mundane: w2v-BERT 2.0 with standard CTC reaches 47.0% WER over five seeds, beating the larger MMS-1B and comparable models; pretraining design, not parameter count, drives performance, and speed augmentation gives a small, largely consistent gain. Multi-seed evaluation on official splits separates real gains from seed noise.

Figures

Figures reproduced from arXiv: 2608.10670 by the authors.

**Figure 2.** Encoder WER vs. parameter count on the official Garhwali test set (five-seed means, [PITH_FULL_IMAGE:figures/full_fig_p015_2.png] view at source ↗

**Figure 3.** Layer-wise linear probing of w2v-BERT 2.0 (67,650-parameter probe; seed 42, standard CTC). [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 10 canonical work pages

[4]

A methodological difference from Dhasmana et al

Early-stopping metric. A methodological difference from Dhasmana et al. (2026) is the early-stopping criterion: we stop on validation WER, they on validation CER, a metric in- creasingly argued to be more appropriate for multilingual and Indic ASR ( K et al. , 2025). To confirm this does not drive our numbers, we re-run standard CTC under CER-based stoppi...

work page 2026
[5]

Primary test: per-seed paired Wilcoxon signed-rank on corpus WER, Holm–Bonferroni corrected

on val WER Decoding per-utterance greedy Seeds 42, 123, 777, 2025, 1234 Hardware 1× NVIDIA A100-SXM4 (80GB), 4 CPU, 48 GB RAM Table 12: Pairwise significance for the augmented objective comparison. Primary test: per-seed paired Wilcoxon signed-rank on corpus WER, Holm–Bonferroni corrected. No pair is significant at α = 0.05. Pair ∆WER p (Holm) Standard vs...

work page 2025
[6]

arXiv preprint arXiv:2512.10967

Asr under the stethoscope: Evaluating biases in clinical speech recognition across indian lan- guages. arXiv preprint arXiv:2512.10967 . Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaim- ing He, and Piotr Dollar

work page arXiv
[7]

manual_seed (3407) is all you need: On the influence of random seeds in deep learning architectures for computer vi- sion

Torch. manual_seed (3407) is all you need: On the influence of random seeds in deep learning architectures for computer vi- sion. arXiv preprint arXiv:2109.08203 . Vineel Pratap, Anuroop Sriram, Paden Tomasello, Awni Hannun, Vitaliy Liptchinsky, Gabriel Syn- naeve, and Ronan Collobert

arXiv
[8]

Seeds: 25 = 2025, 34 =

work page 2025
[10]

arXiv preprint arXiv:2510.06961

Open asr leaderboard: To- wards reproducible and transparent multilingual speech recognition evaluation. arXiv preprint arXiv:2510.06961. Appendix This appendix provides the complete evidence behind the aggregate numbers in the main pa- per. Because our central claim is methodolog- ical that single-seed comparisons are unreli- able at this corpus size, we...

arXiv
[16]

Categorized

to the top of the fine- tuned model (layer 24, near full-model perfor- mance), while lower layers stay close to their pretrained state. We treat this as exploratory and draw no quantitative conclusions from it. B.2 LLM-Assisted Cross-Model Error Categorisation To obtain a functional view of the residual er- rors, grouping each reference–hypothesis mis- ma...

work page 2025
[2019]

arXiv preprint arXiv:1910.05453

vq-wav2vec: Self-supervised learn- ing of discrete speech representations. arXiv preprint arXiv:1910.05453 . Alexei Baevski, Yuhao Zhou, Abdelrahman Mo- hamed, and Michael Auli

arXiv 1910

Show all 14 references

[2020]

In Interspeech 2020, pages 4751–4755

Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameters . In Interspeech 2020, pages 4751–4755. Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, Alexei Baevski, Yoss...

2020
[2021]

In 2021 IEEE Automatic Speech Recognition and Understand- ing Workshop (ASRU) , pages 244–250

w2v-bert: Combining contrastive learning and masked language modeling for self- supervised speech pre-training . In 2021 IEEE Automatic Speech Recognition and Understand- ing Workshop (ASRU) , pages 244–250. Akriti Dhasmana, Aarohi Srivastava, and David Chiang

2021
[2022]

In Interspeech 2022 , pages 2278–2282

XLS-R: Self- supervised Cross-lingual Speech Representation Learning at Scale . In Interspeech 2022 , pages 2278–2282. Alexei Baevski, Steffen Schneider, and Michael Auli

2022
[2024]

In Findings of the Association for Computa- tional Linguistics: ACL 2024 , pages 10740– 10782, Bangkok, Thailand

In- dicVoices: Towards building an inclusive mul- tilingual speech dataset for Indian languages . In Findings of the Association for Computa- tional Linguistics: ACL 2024 , pages 10740– 10782, Bangkok, Thailand. Association for Com- putational Linguistics. Thennal D K, Jesin J...

2024
[2025]

In Findings of the Association for Computational Linguistics: NAACL 2025 , pages 4941–4950, Albuquerque, New Mexico

Ad- vocating character error rate for multilingual ASR evaluation. In Findings of the Association for Computational Linguistics: NAACL 2025 , pages 4941–4950, Albuquerque, New Mexico. As- sociation for Computational Linguistics. Tom Ko, Vijayaditya Peddinti, Daniel Povey, Mich...

2025
[2026]

Preprint, arXiv:2603.28714

Vaani: Captur- ing the language landscape for an inclusive digi- tal india . Preprint, arXiv:2603.28714. Vaibhav Srivastav, Steven Zheng, Eric Bezzam, Eustache Le Bihan, Adel Moumen, and San- chit Gandhi

arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [4]

A methodological difference from Dhasmana et al

Early-stopping metric. A methodological difference from Dhasmana et al. (2026) is the early-stopping criterion: we stop on validation WER, they on validation CER, a metric in- creasingly argued to be more appropriate for multilingual and Indic ASR ( K et al. , 2025). To confirm this does not drive our numbers, we re-run standard CTC under CER-based stoppi...

work page 2026

[2] [5]

Primary test: per-seed paired Wilcoxon signed-rank on corpus WER, Holm–Bonferroni corrected

on val WER Decoding per-utterance greedy Seeds 42, 123, 777, 2025, 1234 Hardware 1× NVIDIA A100-SXM4 (80GB), 4 CPU, 48 GB RAM Table 12: Pairwise significance for the augmented objective comparison. Primary test: per-seed paired Wilcoxon signed-rank on corpus WER, Holm–Bonferroni corrected. No pair is significant at α = 0.05. Pair ∆WER p (Holm) Standard vs...

work page 2025

[3] [6]

arXiv preprint arXiv:2512.10967

Asr under the stethoscope: Evaluating biases in clinical speech recognition across indian lan- guages. arXiv preprint arXiv:2512.10967 . Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaim- ing He, and Piotr Dollar

work page arXiv

[4] [7]

manual_seed (3407) is all you need: On the influence of random seeds in deep learning architectures for computer vi- sion

Torch. manual_seed (3407) is all you need: On the influence of random seeds in deep learning architectures for computer vi- sion. arXiv preprint arXiv:2109.08203 . Vineel Pratap, Anuroop Sriram, Paden Tomasello, Awni Hannun, Vitaliy Liptchinsky, Gabriel Syn- naeve, and Ronan Collobert

arXiv

[5] [8]

Seeds: 25 = 2025, 34 =

work page 2025

[6] [10]

arXiv preprint arXiv:2510.06961

Open asr leaderboard: To- wards reproducible and transparent multilingual speech recognition evaluation. arXiv preprint arXiv:2510.06961. Appendix This appendix provides the complete evidence behind the aggregate numbers in the main pa- per. Because our central claim is methodolog- ical that single-seed comparisons are unreli- able at this corpus size, we...

arXiv

[7] [16]

Categorized

to the top of the fine- tuned model (layer 24, near full-model perfor- mance), while lower layers stay close to their pretrained state. We treat this as exploratory and draw no quantitative conclusions from it. B.2 LLM-Assisted Cross-Model Error Categorisation To obtain a functional view of the residual er- rors, grouping each reference–hypothesis mis- ma...

work page 2025

[8] [2019]

arXiv preprint arXiv:1910.05453

vq-wav2vec: Self-supervised learn- ing of discrete speech representations. arXiv preprint arXiv:1910.05453 . Alexei Baevski, Yuhao Zhou, Abdelrahman Mo- hamed, and Michael Auli

arXiv 1910