Pith. sign in

REVIEW 2 major objections 5 minor 63 references

A 10KB text memory teaches a frozen ASR corrector when to hold still, cutting word error without touching weights.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 16:32 UTC pith:DJKLGBTK

load-bearing objection A genuinely useful inference-only ASR correction method with a nice restraint story, but the headline appendix table includes an undefined 'oracle gating' that must be pinned down before the central claim is clean. the 2 major comments →

arxiv 2607.26410 v1 pith:DJKLGBTK submitted 2026-07-29 cs.CL cs.AIcs.SDeess.AS

Voice Memory for Agentic Speech Recognition

Chao-Han Huck Yang , Zih-Ching Chen , Piotr Zelasko , Zhehuai Chen , Jagadeesh Balam , Boris Ginsburg This is my paper
classification cs.CL cs.AIcs.SDeess.AS
keywords voice agentsmemory modelingon-device adaptationlanguage modelstest-time adaptationgenerative error correctionlistener-thinker architecturerestraint policy
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Voice Memory proposes that the main obstacle to fixing modern ASR errors is over-correction: LLM correctors rewrite tokens that were already right. The paper's solution is to keep the corrector frozen and store the correction skill in a small per-domain Markdown file that the model reads at inference. An offline optimizer revises that file with bounded edits, accepting only changes that strictly improve a held-out score. The loop discovers restraint — mostly negative rules like 'keep the verbatim ASR token' — and across ten speech domains it lowers weighted WER from 8.36% to 7.52% with zero added parameters, dropping harmful edit rate from 64% to 35% on financial news. The contribution is framing correction as a per-utterance act/abstain decision, with the learned skill living in an auditable, portable artifact rather than in weights.

Core claim

On the paper's own account, the task of modern ASR error correction has inverted: at single-digit word-error rates the decision is not what to fix but whether to touch the hypothesis at all. The authors formalize an agentic listener-thinker split in which a frozen corrector reads a persistent text memory and chooses, per utterance, to act on its hypothesis or abstain and keep the 1-best. The memory is learned asynchronously by a score-gated optimizer that makes bounded add/delete/replace edits and accepts an edit only when it strictly improves a held-out score. The learned policy is mostly restraint: suppressive rules such as 'do not normalize' stop the corrector from breaking correct tokens

What carries the argument

The load-bearing object is the per-domain text memory (memory.md) — a short, human-readable Markdown rule set read by the frozen corrector before each utterance — together with the score-gated optimization loop that revises it. The optimizer proposes bounded edits; the gate accepts an edit only when a held-out score (WER or semantic cosine) strictly improves; accepted rules accumulate, mostly as suppressive/negative instructions. The act/abstain decision over corrected hypothesis versus 1-best turns correction into a policy, and the Recoverable Information Ratio and Harmful Edit Rate metrics measure whether the loop closes recoverable headroom or breaks correct tokens.

Load-bearing premise

The 8.36→7.52 result relies on an undocumented 'oracle gating' component that must not consult test references when deciding whether to correct, and on an 80-utterance held-out split being representative of full-test scores.

What would settle it

Remove or audit the 'oracle gating' step in the combined configuration and rerun the ten-domain evaluation: if weighted WER returns near the negative-rules-only level (about 7.66) without access to references, the headline gain comes from oracle access, not the learned memory. A simpler check: verify that act/abstain decisions for test utterances are made before any reference transcript is read.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim holds, ASR-LLM cascades can be adapted to new domains with a few hundred utterances, forward passes only, and a sub-10KB auditable artifact — no training infrastructure needed.
  • The learned memory is a portable asset: a memory written by one corrector family improves a different frozen corrector at inference, and a stronger reader can re-form a better memory with the same loop.
  • Because the optimal policy is mostly restraint, gains concentrate where recoverable headroom is large (air-travel commands from 8.40% to 3.40%; noisy far-field speech from 12.69% to 10.46%) and vanish at clean-speech floors, guiding where correction is worth deploying.
  • Gating the optimizer on semantic similarity rather than WER gives slightly better and more seed-stable WER, suggesting that optimizing the reported metric directly can be worse than optimizing the underlying meaning.
  • The same loop transfers to translation n-best correction, recovering 13–32% of oracle BLEU headroom in a small study, implying a general recipe for n-best language correction.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Not stated by the paper: the act/abstain policy could generalize as a test-time mechanism for any LLM agent with a verifiable score, tuning retrieval or tool-use policies by editing an external text memory with no gradient updates.
  • A testable extension follows from the finding that negative rules carry most of the gain: initialize a memory with generic restraint rules (e.g., 'do not normalize spelling') and measure how much benefit remains, separating universal over-correction from domain-specific knowledge.
  • If the oracle-gating ambiguity is resolved in the authors' favor, a practical calibration recipe emerges: run the loop on a few dozen held-out utterances of a new domain, inspect the generated memory file before deployment, and edit it since provenance quality beats raw coverage.
  • The semantic-gate result points to a design principle beyond ASR: when a metric (WER) poorly tracks the true objective (meaning), optimizing the metric can be counterproductive; this may apply to any task with meaning-preserving rewrite ambiguity.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper presents Voice Memory, a method for adapting an ASR corrector at inference time by learning a per-domain text memory via a score-gated optimization loop. A frozen LLM reads the memory and decides per utterance whether to correct the 1-best hypothesis or abstain. The method is evaluated on HyPoradise and Robust HyPoradise benchmarks, with reported gains across domains, portability across correctors, and transfer to translation. The main result is a weighted WER reduction from 8.36% to 7.52% (7.47% with examples) without regression below the 1-best baseline, using an open Qwen3-30B-A3B corrector (Table 9).

Significance. Assuming the results hold, the paper introduces a promising new paradigm: external, auditable, inference-only memory for ASR correction. Strengths include the disjoint train/selection/test splits, the strict held-out acceptance gate, the open-source code release, and the introduction of diagnostic metrics (ρ, HER). The claimed WER reduction and no-regression property are notable and would be of broad interest. However, the undefined 'oracle gating' in Table 9 is a serious omission that directly affects the headline result; it must be resolved before the central claim is credible.

major comments (2)
  1. [Table 9, §3.2/Algorithm 1] The VM (All Combined) configuration in Table 9 is described as applying 'negative rules, filtered memory, domain prompts, and oracle gating.' Neither §3.2 nor Algorithm 1 defines 'oracle gating.' If this refers to a per-utterance decision that consults the test reference to choose act/abstain, the 'no regression below baseline' property holds by construction and the 8.36→7.52 result is not an inference-only evaluation; the learned-restraint conclusion would be invalid. Please define the term, state whether test references are used, and report the result without this component.
  2. [§4 / Appendix B] The paper claims 'Negative rules are the single load-bearing component: alone, they drive weighted WER from 7.66 to 7.52' but provides no table or numbers for this ablation. Since the central narrative is that restraint (negative rules) is the discovered policy, the absence of an ablation table prevents verification of the claim. Please include the full component ablation with WER for each individual component and their combinations.
minor comments (5)
  1. [§4 / Figure 2] The text states 'up to 0.64 on wsj (Figure 2)', but Figure 2 shows static GER HER=0.74 on ls_clean, which is higher. Please correct the maximum or adjust the wording.
  2. [§3.3 / Table 3] §3.3 lists SWBD among the ten HyPoradise domains, but Table 3 reports only nine domains (missing SWBD). Please clarify whether SWBD was excluded and why, or add it to Table 3.
  3. [§6 / Table 4] VoiceBank-DEMAND and LS-FreeSound results are discussed in text (party ρ=0.03) but not tabulated; the comparison to RobustGER is indirect. Please add a complete table or explicit numbers.
  4. [Abstract / Appendix C] The transfer study uses only 50-100 utterances and is labeled 'illustrative' in Appendix C, yet the abstract states 'The memory transfers across corrector families' without this caveat. Consider hedging the claim in the abstract.
  5. [Abstract / §1] The phrase 'a inference-only' appears in the abstract and introduction; it should be 'an inference-only.'

Circularity Check

2 steps flagged

Table 9's undefined 'oracle gating' makes the headline no-regression claim true by construction if it uses test references; the 'restraint' conclusion is also partly the optimizer's acceptance rule, not a discovered behavior.

specific steps
  1. fitted input called prediction [Appendix B, Table 9 and surrounding text; compare §3.2 and Algorithm 1]
    "Raw ASR is the Whisper-v2-Large 1-best; Voice Memory (All Combined) applies negative rules, filtered memory, domain prompts, and oracle gating; Few-shot adds three in-context correction examples per dataset. ... Table 9 confirms the depth-study pattern at breadth: Voice Memory cuts weighted WER from 8.36 to 7.47, and no dataset regresses below its raw baseline."

    §3.2 and Algorithm 1 define only a held-out score gate ('accepting an edit only when it strictly improves a held-out score'; 'if v′ > v⋆ then ... accept'). 'Oracle gating' is never defined in §3.2 or Algorithm 1. If it means a per-utterance act/abstain decision that consults the test reference, then the 'no dataset regresses below its raw baseline' property is true by construction, and the reported 8.36→7.52 reduction is not an inference-only result attributable to the memory. The abstract and conclusion rest on this table, so the omission is load-bearing.

  2. self definitional [§3.2 ('accepting an edit only when it strictly improves a held-out score'; 'The rules the loop accepts are mostly suppressive') and §4 ('the learned policy is restraint')]
    "accepting an edit only when it strictly improves a held-out score ... The rules the loop accepts are mostly suppressive; §4 shows the learned policy is restraint."

    The optimizer is defined to reject every candidate edit that does not strictly improve the held-out score (Algorithm 1: 'if v′ > v⋆ then ... accept'). Therefore the final memory is, by construction, one that avoids score-decreasing edits on the selection split. The observed 'restraint' — low harmful edit rate and suppressive rules — is the acceptance rule's direct consequence, not an independently discovered behavioral property of the model. The test-set WER gains are not forced by the gate, but the paper's interpretive claim that 'restraint turns out to be the operative skill this loop discovers' reduces to the loop's definition.

full rationale

The main empirical protocol is largely sound: train/selection/test splits are disjoint, the selection gate uses held-out scores, and the open-corrector breadth study is benchmarked on HyPoradise, so the core 8.36→7.52 weighted-WER reduction is not simply a fit to the test set. However, two load-bearing steps are circular or potentially so. First, Table 9 — the source of the abstract's headline 'without regressing any dataset below its 1-best baseline' — lists 'oracle gating' as part of the VM (All Combined) configuration, but §3.2 and Algorithm 1 never define it. If that gating uses test references to choose act/abstain, the no-regression result is true by construction and the inference-only claim collapses. Second, the paper repeatedly concludes that 'restraint is the operative skill this loop discovers,' but the optimizer's acceptance rule already forbids any edit that does not strictly improve the held-out score; the final policy is therefore restrained by definition. The quantitative WER improvements are not fully forced because the selection split differs from the test split, so the score is 6 (partial circularity) rather than higher. No self-citation chain or imported uniqueness theorem is load-bearing here.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The method's real free object is the per-domain text memory plus optimizer hyperparameters. The main assumptions are n-best sufficiency, LLM instruction-following, representativeness of the small selection split, and semantic-encoder validity. No new physical or conceptual entities are postulated beyond the memory file itself.

free parameters (4)
  • Per-domain memory.md rules = WSJ 776 B; Claude memory 5.4 KB; auto/expert variants up to 9.0 KB
    The central learned artifact is a set of text rules selected by the validation-gated optimizer; it is the object fitted on each domain's train/selection splits.
  • Optimization hyperparameters: epochs=4, rollout_batch=24, selection_size=80, text_lr=4 with cosine decay floor 2 = as stated in Appendix A
    Hand-chosen defaults; no sweeps are reported, and they control which memory is accepted and hence the headline WER.
  • Edit budget (ell_e) = not specified
    Algorithm 1 takes a bounded edit budget as input but its value is not reported.
  • Benign-mass semantic threshold = 0.15
    Used in §7.2 to label residual error mass as meaning-preserving; the threshold is arbitrary and not justified.
axioms (5)
  • domain assumption The n-best hypotheses H from the frozen decoder are the only evidence; no audio features are used at inference.
    Definition 1/§3.1: the corrector sees H and memory s only; if recoverable acoustic information is absent from H, no text memory can recover it.
  • domain assumption A frozen LLM (MiniMax-M3 or Qwen3-30B-A3B) reliably follows memory.md instructions, so the rules modulate act/abstain behavior.
    §3.2/Algorithm 1: the method assumes placing s in context changes M's outputs in the intended direction.
  • domain assumption The held-out selection split (80 utterances) is representative of the full test split, and the 'strictly greater' gate on it selects memories that generalize.
    §3.2/Appendix A: all memories are chosen by this gate; if the selection split is unrepresentative, the reported test gains are overfits.
  • domain assumption Semantic cosine similarity from a 4-bit all-MiniLM encoder is a valid proxy for meaning; it is used both as a gate and as the benign-mass measure.
    §7: the semantic gate is compared to a WER gate on two domains and two seeds only; the benign-mass measure depends on the encoder.
  • domain assumption The reference transcripts and n-best alignments in HyPoradise/Robust HyPoradise are ground truth, and annotation artifacts should not be corrected.
    §7.3 and Appendix B: residual WER is split into acoustic errors and annotation artifacts, which requires that artifact labels are known.

pith-pipeline@v1.3.0-daily-deepseek · 16861 in / 19463 out tokens · 159762 ms · 2026-08-01T16:32:55.601170+00:00 · methodology

0 comments
read the original abstract

We present Voice Memory, a inference-only scheme for agentic speech recognition: at stream time, a frozen corrector reads a single per-domain memory.md and decides per utterance whether to act on the hypothesis or abstain and keep the 1-best. Asynchronously, a score-gated optimizer revises that file through bounded edits, accepting an edit only when it strictly improves a held-out score. Extended from classical ASR-LM framework, we refer this split the listener-thinker architecture; the two roles are coupled only through the memory, so no weights change and the learned skill stays auditable and portable. Restraint turns out to be the operative skill this loop discovers: unconstrained generative error correction (GER) over-corrects, breaking correct tokens on up to 64% of its edits on financial news, and Voice Memory, reduces this rate to 35%. Across ten HyPoradise domains with an open corrector, Voice Memory, lowers weighted word error rate from 8.36% to 7.52% (7.47% with three added in-context examples) without regressing any dataset below its 1-best baseline; gains concentrate where recoverable headroom is largest, including air-travel commands (8.40% to 3.40%) and noisy far-field speech (CHiME-4, 12.69% to 10.46%). The memory transfers across corrector families and adds zero parameters to the inference path. A demo and example code are provided for future studies.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

63 extracted references · 1 canonical work pages

  1. [1]

    Gepa: Reflective prompt evolution can outperform reinforcement learning.arXiv preprint arXiv:2507.19457, 2025

    Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, et al. Gepa: Reflective prompt evolution can outperform reinforcement learning.arXiv preprint arXiv:2507.19457, 2025

    Pith/arXiv arXiv 2025

  2. [2]

    Bringing contextual information to google speech recognition

    Petar Aleksic, Mohammadreza Ghodsi, Assaf Michaely, Cyril Allauzen, Keith Hall, Brian Roark, David Rybach, and Pedro Moreno. Bringing contextual information to google speech recognition. InProc. Interspeech, pp. 468–472, 2015

    2015

  3. [3]

    wav2vec 2.0: A framework for self-supervised learning of speech representations

    Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. InAdvances in Neural Information Processing Systems (NeurIPS), 2020

    2020

  4. [4]

    Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, et al

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, et al. Language models are few-shot learners. InAdvances in Neural Information Processing Systems (NeurIPS), 2020

    2020

  5. [5]

    Text-only adaptation in llm-based asr through text denoising.arXiv preprint arXiv:2601.20900, 2026

    Andrés Carofilis, Sergio Burdisso, Esaú Villatoro-Tello, Shashi Kumar, Kadri Hacioglu, Srikanth Madikeri, Pradeep Rangappa, Petr Motlicek, Shankar Venkatesan, Andreas Stolcke, et al. Text-only adaptation in llm-based asr through text denoising.arXiv preprint arXiv:2601.20900, 2026

    arXiv 2026

  6. [6]

    HyPoradise: An open baseline for generative speech recognition with large language models

    Chen Chen, Yuchen Hu, Chao-Han Huck Yang, Sabato Marco Siniscalchi, Pin-Yu Chen, and Eng Siong Chng. HyPoradise: An open baseline for generative speech recognition with large language models. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2023

    2023

  7. [7]

    Moshi: a speech-text foundation model for real-time dialogue.arXiv preprint arXiv:2410.00037, 2024

    Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation model for real-time dialogue.arXiv preprint arXiv:2410.00037, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Lipger: Visually-conditioned generative error correction for robust automatic speech recognition

    Sreyan Ghosh, Sonal Kumar, Ashish Seth, Purva Chiniya, Utkarsh Tyagi, Ramani Duraiswami, and Dinesh Manocha. Lipger: Visually-conditioned generative error correction for robust automatic speech recognition. InProc. Interspeech 2024, pp. 1920–1924, 2024

    2024

  9. [9]

    Words and voices: episodic traces in spoken word identification and recognition memory.Journal of experimental psychology: Learning, memory, and cognition, 22(5):1166, 1996

    Stephen D Goldinger. Words and voices: episodic traces in spoken word identification and recognition memory.Journal of experimental psychology: Learning, memory, and cognition, 22(5):1166, 1996

    1996

  10. [10]

    Conformer: Convolution-augmented transformer for speech recognition

    Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. Conformer: Convolution-augmented transformer for speech recognition. InProc. Interspeech, pp. 5036–5040, 2020

    2020

  11. [11]

    The sapir-whorf hypothesis.Language in culture, 4:92–105, 1954

    Harry Hoijer. The sapir-whorf hypothesis.Language in culture, 4:92–105, 1954

    1954

  12. [12]

    Let’s fuse step by step: A generative fusion decoding algorithm with llms for robust and instruction-aware asr and ocr

    Chan-Jan Hsu, Yi-Chang Chen, Feng-Ting Liao, Pei-Chen Ho, Yu-Hsiang Wang, Po-Chun Hsu, and Da-shan Shiu. Let’s fuse step by step: A generative fusion decoding algorithm with llms for robust and instruction-aware asr and ocr. InFindings of the Association for Computational Linguistics: ACL 2025, pp. 24959–24973, 2025

    2025

  13. [13]

    HuBERT: Self-supervised speech representation learning by masked prediction of hidden units.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451–3460, 2021

    Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. HuBERT: Self-supervised speech representation learning by masked prediction of hidden units.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451–3460, 2021

    2021

  14. [14]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. InInternational Conference on Learning Representations (ICLR), 2022

    2022

  15. [15]

    Scaling up deliberation for multilingual asr

    Ke Hu, Bo Li, and Tara N Sainath. Scaling up deliberation for multilingual asr. In2022 IEEE Spoken Language Technology Workshop (SLT), pp. 771–776. IEEE, 2023. 11 Voice Memory for Agentic Speech Recognition

    2023

  16. [16]

    Chain-of-thought prompting for speech translation

    KeHu, ZhehuaiChen, Chao-HanHuckYang, PiotrŻelasko, OleksiiHrinchuk, VitalyLavrukhin, Jagadeesh Balam, and Boris Ginsburg. Chain-of-thought prompting for speech translation. InICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5. IEEE, 2025

    2025

  17. [17]

    Subjective comparison and evaluation of speech enhancement algorithms

    Yi Hu and Philipos C Loizou. Subjective comparison and evaluation of speech enhancement algorithms. Speech communication, 49(7-8):588–601, 2007

    2007

  18. [18]

    Large language models are efficient learners of noise-robust speech recognition

    Yuchen Hu, Chen Chen, Chao-Han Huck Yang, Ruizhe Li, Chao Zhang, Pin-Yu Chen, and Eng Siong Chng. Large language models are efficient learners of noise-robust speech recognition. InInternational Conference on Learning Representations (ICLR), 2024

    2024

  19. [19]

    GenTranslate: Large language models are generative multilingual speech and machine translators

    Yuchen Hu, Chen Chen, Chao-Han Huck Yang, Ruizhe Li, Dong Zhang, Zhehuai Chen, and Eng Siong Chng. GenTranslate: Large language models are generative multilingual speech and machine translators. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024

    2024

  20. [20]

    Huang et al

    M. Huang et al. DuplexOmni: Real-time listening, seeing, thinking, and speaking for full-duplex interaction.arXiv preprint arXiv:2606.09186, 2026

    Pith/arXiv arXiv 2026

  21. [21]

    Evaluating open-source asr systems: Performance across diverse audio conditions and error correction methods

    Saki Imai, Tahiya Chowdhury, and Amanda Stent. Evaluating open-source asr systems: Performance across diverse audio conditions and error correction methods. InProceedings of the 31st International Conference on Computational Linguistics, pp. 5027–5039, 2025

    2025

  22. [22]

    Contextual RNN-T for open domain ASR

    Mahaveer Jain, Gil Keren, Jay Mahadeokar, Geoffrey Zweig, Florian Metze, and Yatharth Saraf. Contextual RNN-T for open domain ASR. InProc. Interspeech, 2020

    2020

  23. [23]

    Language, Speech and Communication

    Frederick Jelinek.Statistical Methods for Speech Recognition. Language, Speech and Communication. MIT Press, Cambridge, MA, 1997

    1997

  24. [24]

    Two heads are better than one: Audio-visual speech error correction with dual hypotheses

    Sungnyun Kim, Kangwook Jang, Sungwoo Cho, Joon Son Chung, Hoi-Rin Kim, and Se-Young Yun. Two heads are better than one: Audio-visual speech error correction with dual hypotheses. InFindings of the Association for Computational Linguistics: ACL 2026, pp. 544–564, 2026

    2026

  25. [25]

    Semantic distance: A new metric for ASR performance analysis towards spoken language understanding

    Suyoun Kim, Abhinav Arora, Duc Le, Ching-Feng Yeh, Christian Fuegen, Ozlem Kalinli, and Michael L Seltzer. Semantic distance: A new metric for ASR performance analysis towards spoken language understanding. InInterspeech, 2021

    2021

  26. [26]

    Preference-asr: A preference-aware test set for benchmarking asr in the era of speech llms.arXiv preprint arXiv:2606.29534, 2026

    Nithin Rao Koluguri, Sasha Meister, Nikolay Karpov, Piotr Zelasko, Desh Raj, Jagadeesh Balam, and Boris Ginsburg. Preference-asr: A preference-aware test set for benchmarking asr in the era of speech llms.arXiv preprint arXiv:2606.29534, 2026

    Pith/arXiv arXiv 2026

  27. [27]

    Interaction models: A scalable approach to human-ai collabora- tion.Thinking Machines Lab: Connectionism, May 2026

    Thinking Machines Lab. Interaction models: A scalable approach to human-ai collabora- tion.Thinking Machines Lab: Connectionism, May 2026. doi: 10.64434/tml.20260511. https://thinkingmachines.ai/blog/interaction-models/

    doi:10.64434/tml.20260511 2026

  28. [28]

    MiniMax sparse attention.arXiv preprint arXiv:2606.13392, 2026

    Xunhao Lai, Weiqi Xu, Yufeng Yang, Qiaorui Chen, Yang Xu, Lunbin Zeng, Xiaolong Li, Haohai Sun, Haichao Zhu, Vito Zhang, and Pengyu Zhao. MiniMax sparse attention.arXiv preprint arXiv:2606.13392, 2026

    Pith/arXiv arXiv 2026

  29. [29]

    Duc Le, Mahaveer Jain, Gil Keren, Suyoun Kim, Yangyang Shi, Jay Mahadeokar, Julian Chan, Yuan Shangguan, Christian Fuegen, Ozlem Kalinli, Yatharth Saraf, and Michael L. Seltzer. Contextualized streaming end-to-end speech recognition with trie-based deep biasing and shallow fusion. InProc. Interspeech, pp. 1772–1776, 2021

    2021

  30. [30]

    FastCorrect: Fast error correction with edit alignment for automatic speech recognition

    Yichong Leng, Xu Tan, Linchen Zhu, Jin Xu, Renqian Luo, Linquan Liu, Tao Qin, Xiang-Yang Li, Ed Lin, and Tie-Yan Liu. FastCorrect: Fast error correction with edit alignment for automatic speech recognition. InAdvances in Neural Information Processing Systems (NeurIPS), volume 34, pp. 21708–21719, 2021

    2021

  31. [31]

    The power of scale for parameter-efficient prompt tuning

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. InProceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021. 12 Voice Memory for Agentic Speech Recognition

    2021

  32. [32]

    Investigating asr error correction with large language model and multilingual 1-best hypotheses

    Sheng Li, Chen Chen, Chin Yuen Kwok, Chenhui Chu, Eng Siong Chng, and Hisashi Kawai. Investigating asr error correction with large language model and multilingual 1-best hypotheses. InProc. Interspeech 2024, pp. 1315–1319, 2024

    2024

  33. [33]

    Rethinking evaluation in ASR: Are our models robust enough? InProc

    Tatiana Likhomanenko, Qiantong Xu, Vineel Pratap, Paden Tomasello, Jacob Kahn, Gilad Avidov, Ronan Collobert, and Gabriel Synnaeve. Rethinking evaluation in ASR: Are our models robust enough? InProc. Interspeech, pp. 311–315, 2021. doi: 10.21437/Interspeech.2021-1758

    work page doi:10.21437/interspeech.2021-1758 2021

  34. [34]

    Neko: Cross-modality post-recognition error correction with tasks-guided mixture-of-experts language model

    Yen-Ting Lin, Zhehuai Chen, Piotr Żelasko, Zhen Wan, Xuesong Yang, Zih-Ching Chen, Krishna C Puvvada, Ke Hu, Szu-Wei Fu, Jun Wei Chiu, et al. Neko: Cross-modality post-recognition error correction with tasks-guided mixture-of-experts language model. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Indust...

    2025

  35. [35]

    N-best t5: Robust asr error correction using multiple input hypotheses and constrained decoding space.arXiv preprint arXiv:2303.00456, 2023

    Rao Ma, Mark JF Gales, Kate M Knill, and Mengjie Qian. N-best t5: Robust asr error correction using multiple input hypotheses and constrained decoding space.arXiv preprint arXiv:2303.00456, 2023

    Pith/arXiv arXiv 2023

  36. [36]

    Typicality effects on memory for voice: Implications for earwitness testimony.Applied Cognitive Psychology, 25(1):29–34, 2011

    JOHN W Mullennix, A Ross, C Smith, K Kuykendall, J Conard, and S Barb. Typicality effects on memory for voice: Implications for earwitness testimony.Applied Cognitive Psychology, 25(1):29–34, 2011

    2011

  37. [37]

    Trace2skill: Distill trajectory-local lessons into transferable agent skills.arXiv preprint arXiv:2603.25158, 2026

    Jingwei Ni, Yihao Liu, Xinpeng Liu, Yutao Sun, Mengyu Zhou, Pengyu Cheng, Dexin Wang, Erchao Zhao, Xiaoxi Jiang, and Guanjun Jiang. Trace2skill: Distill trajectory-local lessons into transferable agent skills.arXiv preprint arXiv:2603.25158, 2026

    Pith/arXiv arXiv 2026

  38. [38]

    Long-term memory for unfamiliar voices.The Journal of the Acoustical Society of America, 85(2):913–925, 1989

    George Papcun, Jody Kreiman, and Anthony Davis. Long-term memory for unfamiliar voices.The Journal of the Acoustical Society of America, 85(2):913–925, 1989

    1989

  39. [39]

    Sainath, Rohit Prabhavalkar, Anjuli Kannan, and Ding Zhao

    Golan Pundak, Tara N. Sainath, Rohit Prabhavalkar, Anjuli Kannan, and Ding Zhao. Deep context: End-to-end contextual speech recognition. InIEEE Spoken Language Technology Workshop (SLT), pp. 418–425, 2018

    2018

  40. [40]

    Less is more: Accurate speech recognition & translation without web-scale data

    Krishna C Puvvada, Piotr Żelasko, He Huang, Oleksii Hrinchuk, Nithin Rao Koluguri, et al. Less is more: Accurate speech recognition & translation without web-scale data. InInterspeech, 2024

    2024

  41. [41]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    Qwen Team. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  42. [42]

    Lawrence R. Rabiner. A tutorial on hidden markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(2):257–286, 1989

    1989

  43. [43]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. InProceedings of the 40th International Conference on Machine Learning (ICML 2023), 2023

    2023

  44. [44]

    Whispering LLaMA: A cross-modal generative error correction framework for speech recognition

    Srijith Radhakrishnan, Chao-Han Huck Yang, Sumeer Ahmad Khan, Rohit Kumar, Narsis A Kiani, David Gomez-Cabrero, and Jesper Tegner. Whispering LLaMA: A cross-modal generative error correction framework for speech recognition. InProceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023

    2023

  45. [45]

    Sentence-bert: Sentence embeddings using siamese bert-networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. InProceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pp. 3982–3992, 2019

    2019

  46. [46]

    Fast conformer with linearly scalable attention for efficient speech recognition

    Dima Rekesh, Nithin Rao Koluguri, Samuel Kriman, Somshubra Majumdar, Vahid Noroozi, He Huang, Oleksii Hrinchuk, Krishna Puvvada, Ankur Kumar, Jagadeesh Balam, et al. Fast conformer with linearly scalable attention for efficient speech recognition. In2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 1–8. IEEE, 2023

    2023

  47. [47]

    Nguyen, and Katrin Kirchhoff

    Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. Masked language model scoring. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 2699–2712, 2020. 13 Voice Memory for Agentic Speech Recognition

    2020

  48. [48]

    The effects of voice and visible speaker change on memory for spoken words.Journal of Memory and Language, 34(5):665–685, 1995

    Sonya M Sheffert and Carol A Fowler. The effects of voice and visible speaker change on memory for spoken words.Journal of Memory and Language, 34(5):665–685, 1995

    1995

  49. [49]

    Dialog act modeling for automatic tagging and recognition of conversational speech.Computational Linguisitics, 26(3):339, 2000

    Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. Dialog act modeling for automatic tagging and recognition of conversational speech.Computational Linguisitics, 26(3):339, 2000

    2000

  50. [50]

    WER we are and WER we think we are

    Piotr Szymański, Piotr Żelasko, Mikolaj Morzy, Adrian Szymczak, Marzena Zyła-Hoppe, et al. WER we are and WER we think we are. InFindings of the Association for Computational Linguistics: EMNLP, 2020

    2020

  51. [51]

    Salmonn: Towards generic hearing abilities for large language models

    Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Salmonn: Towards generic hearing abilities for large language models. InInternational Conference on Learning Representations, volume 2024, pp. 16607–16629, 2024

    2024

  52. [52]

    Interaction models: A scalable approach to human-AI collaboration, May 2026

    Thinking Machines Lab. Interaction models: A scalable approach to human-AI collaboration, May 2026. Technical report

    2026

  53. [53]

    Leveraging llms for post-ocr correction of historical newspapers

    Alan Thomas, Robert Gaizauskas, and Haiping Lu. Leveraging llms for post-ocr correction of historical newspapers. InProceedings of the third workshop on language Technologies for Historical and Ancient Languages (LT4HALA)@ LREC-COLING-2024, pp. 116–121, 2024

    2024

  54. [54]

    Speech-hands: A self-reflection voice agentic approach to speech recognition and audio reasoning with omni perception

    Zhen Wan, Chao-Han Huck Yang, Jinchuan Tian, Hanrong Ye, Ankita Pasad, Szu-wei Fu, Arushi Goel, Shizhe Diao, Chenhui Chu, Shinji Watanabe, Boris Ginsburg, and Yu-Chiang Frank Wang. Speech-hands: A self-reflection voice agentic approach to speech recognition and audio reasoning with omni perception. arXiv preprint arXiv:2601.09413, 2026

    Pith/arXiv arXiv 2026

  55. [55]

    Audio-mind: An auditable agentic framework for audio understanding.arXiv preprint arXiv:2605.28480, 2026

    Yucheng Wang, Jing Peng, Hanqi Li, Chenghao Wang, Wenming Tu, Yu Xi, Zhaokai Sun, Kai Yu, and Shuai Wang. Audio-mind: An auditable agentic framework for audio understanding.arXiv preprint arXiv:2605.28480, 2026

    Pith/arXiv arXiv 2026

  56. [56]

    Speecheq: Benchmarking emotional intelligence quotient in socially aware voice conversational models.arXiv preprint arXiv:2606.25990, 2026

    Liang-Yuan Wu, Zih-Ching Chen, Tongshuang Wu, Chao-Han Huck Yang, and Hua Shen. Speecheq: Benchmarking emotional intelligence quotient in socially aware voice conversational models.arXiv preprint arXiv:2606.25990, 2026

    Pith/arXiv arXiv 2026

  57. [57]

    Generative speech recognition error correction with large language models and task-activating prompting

    Chao-Han Huck Yang, Yile Gu, Yi-Chieh Liu, Shalini Ghosh, Ivan Bulyko, and Andreas Stolcke. Generative speech recognition error correction with large language models and task-activating prompting. InIEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2023

    2023

  58. [58]

    Large language models as optimizers

    Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. InInternational Conference on Learning Representations, volume 2024, pp. 12028–12068, 2024

    2024

  59. [59]

    SkillOpt: Executive strategy for self-evolving agent skills.arXiv preprint arXiv:2605.23904, 2026

    Yifan Yang et al. SkillOpt: Executive strategy for self-evolving agent skills.arXiv preprint arXiv:2605.23904, 2026

    Pith/arXiv arXiv 2026

  60. [60]

    Covoger: A multilingual multitask benchmark for speech-to-text generative error correction with large language models

    Zhengdong Yang, Zhen Wan, Sheng Li, Chao-Han Huck Yang, and Chenhui Chu. Covoger: A multilingual multitask benchmark for speech-to-text generative error correction with large language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 6313–6325, 2025

    2025

  61. [61]

    Mms-llama: Efficient llm-based audio-visual speech recognition with minimal multimodal speech tokens

    Jeong Hun Yeo, Hyeongseop Rha, Se Jin Park, and Yong Man Ro. Mms-llama: Efficient llm-based audio-visual speech recognition with minimal multimodal speech tokens. InFindings of the Association for Computational Linguistics: ACL 2025, pp. 20724–20735, 2025

    2025

  62. [62]

    differentiation

    Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. TextGrad: Automatic “differentiation” via text.arXiv preprint arXiv:2406.07496, 2024

    Pith/arXiv arXiv 2024

  63. [63]

    strictly greater

    Ding Zhao, Tara N. Sainath, David Rybach, Pat Rondon, Deepti Bhatia, Bo Li, and Ruoming Pang. Shallow-fusion end-to-end contextual biasing. InProc. Interspeech, pp. 1418–1422, 2019. 14 Voice Memory for Agentic Speech Recognition A. Reproducibility and Hyperparameters We default to 4 epochs, rollout batch 24, selection split 80, and textual learning rate 4...

    2019