When Helpful Context Leaks: Privacy Risks in Domain-Adapted ASR

Jan Niehues; Maike Z\"ufle

arxiv: 2605.28211 · v1 · pith:4BDU5JLI · submitted 2026-05-27 · cs.CL

When Helpful Context Leaks: Privacy Risks in Domain-Adapted ASR

Maike Z\"ufle , Jan Niehues This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-06-29 12:32 UTCgrok-4.3pith:4BDU5JLI record.json open to challenge →

classification cs.CL

keywords privacy risksdomain adaptationautomatic speech recognitionSpeechLLMspromptingfine-tuningdata leakagephonetic similarity

0 comments

The pith

Domain-adapted speech models leak private information by transcribing phonetically similar words from context or training data instead of spoken terms.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows that customising speech recognition models with domain-specific context or fine-tuning data creates a privacy leak. Adapted models can output private terms that sound like the actual spoken words, even when those private terms were never uttered. This happens through both prompting and fine-tuning, and the effects add up when the two are used together. The authors test a mitigation approach at the prompt level and find that fine-tuning without context prompts keeps accuracy high while limiting the leakage most effectively. The risk arises because domain customisation is routine in professional settings that handle sensitive recordings and terminology.

Core claim

A model adapted to recognise domain-specific terminology can be nudged into transcribing a phonetically similar word from its context or training data, even when a different word is spoken, thereby leaking private information. The authors build a controlled dataset to measure this leakage under prompting and fine-tuning, observe measurable rates for each method, and find that the rates rise when both are applied together. They also evaluate a prompt-level mitigation and conclude that fine-tuning without context prompts yields the strongest accuracy-leakage trade-off.

What carries the argument

Phonetic leakage mechanism in domain-adapted SpeechLLMs, where supplied context or fine-tuning data supplies substitute terms that match the acoustics of spoken input.

If this is right

Prompting with sensitive context produces measurable leakage of private terms.
Fine-tuning on proprietary recordings also produces measurable leakage.
Using both prompting and fine-tuning together raises leakage rates further.
Fine-tuning without context prompts reduces leakage while preserving higher transcription accuracy than other combinations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Deployments handling confidential terminology should default to fine-tuning alone and avoid embedding private details in prompts.
An adversary could potentially craft spoken inputs to trigger extraction of specific private terms stored in the model context.
The same substitution risk may appear in other customised sequence models that rely on phonetic or embedding similarity.
Mitigations beyond prompt editing, such as filtering training data for phonetic near-matches, remain untested in this setup.

Load-bearing premise

The controlled dataset and leakage measurement protocol accurately reflect real-world leakage rates and attacker capabilities when models are deployed with domain customisation.

What would settle it

A production deployment of domain-adapted models in which no private terms from context or training data appear in transcripts when phonetically similar words are spoken.

Figures

Figures reproduced from arXiv: 2605.28211 by Jan Niehues, Maike Z\"ufle.

**Figure 2.** Figure 2: Acoustic word accuracy scores confirm the [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Figure 3: Leakage rate under leakage conditions. (Con [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

**Figure 4.** Figure 4: Leakage rate stratified by lexical similarity [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗

**Figure 5.** Figure 5: Leakage vs. accuracy for different models. [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗

**Figure 6.** Figure 6: Prompts used for Gemma-3-12B context sen [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗

**Figure 7.** Figure 7: Inference prompt templates for both models. [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗

**Figure 8.** Figure 8: Baseline results: acoustic accuracy (top) and background WER (bottom). [PITH_FULL_IMAGE:figures/full_fig_p010_8.png] view at source ↗

**Figure 9.** Figure 9: Leakage rate under leakage conditions (context word injected in prompt and/or finetuning on the context). [PITH_FULL_IMAGE:figures/full_fig_p011_9.png] view at source ↗

**Figure 10.** Figure 10: WER for the leakage conditions (acoustic word as context). [PITH_FULL_IMAGE:figures/full_fig_p011_10.png] view at source ↗

**Figure 11.** Figure 11: Leakage rate broken down by context sentence similarity (top) and phoneme edit distance (bottom), [PITH_FULL_IMAGE:figures/full_fig_p012_11.png] view at source ↗

**Figure 12.** Figure 12: Acoustic Accuracy for models with and without mitigation strategy. [PITH_FULL_IMAGE:figures/full_fig_p013_12.png] view at source ↗

**Figure 13.** Figure 13: Leakage/Accuracy trade-off for Qwen and Phi. [PITH_FULL_IMAGE:figures/full_fig_p013_13.png] view at source ↗

read the original abstract

SpeechLLMs are increasingly deployed in professional settings where domain customisation is standard practice: users supply context in prompts with sensitive information, fine-tune on proprietary recordings, or both. We identify and systematically investigate an overlooked privacy risk of such customisation: a model adapted to recognise domain-specific terminology can be nudged into transcribing a phonetically similar word from its context or training data, even when a different word is spoken, thereby leaking private information. To evaluate this risk, we construct a controlled dataset and measure leakage rates across two customisation mechanisms, prompting and fine-tuning. Both mechanisms cause measurable leakage, compounding when combined. We evaluate a prompt-level mitigation strategy and analyse the accuracy-leakage trade-off across customisation approaches, finding that fine-tuning without context prompts offers the best balance. We release our code and dataset publicly.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper flags a phonetic leakage path in domain-adapted ASR that prior work missed, but the controlled measurements leave the practical scale unclear.

read the letter

The core observation is that prompting or fine-tuning a speech model on domain terms can make it output a word from that material when the spoken audio is only phonetically close. Both adaptation methods produce measurable leakage, and they add up when used together. The authors test a prompt-level fix and report that fine-tuning alone gives the better accuracy versus leakage trade-off. They also release the dataset and code.

This is a clean empirical measurement study on a new leakage vector. The controlled construction lets them isolate the phonetic substitution effect without confounding factors, and checking both prompting and fine-tuning is the right scope. Releasing the artifacts is the right move for follow-up work.

The soft spot is the gap between the lab protocol and deployed conditions. The dataset is built for the experiment, so the leakage rates depend on how phonetic similarity was scored, how the spoken items were chosen, and whether an attacker would actually notice or exploit the substitutions. Without those details the numbers are hard to translate to real privacy impact. The abstract gives no quantitative results or error bars, which makes it difficult to judge effect size.

The work is aimed at people who deploy or audit customized speech systems in sensitive domains. It is worth sending to referees because the risk is specific and testable, even if the current evidence needs tighter grounding in realistic attacker models and larger-scale checks.

Referee Report

2 major / 0 minor

Summary. The paper claims that domain customization of SpeechLLMs via context prompts or fine-tuning on proprietary data creates a privacy risk: the adapted model can transcribe a phonetically similar word drawn from the supplied context or training data even when a different word is spoken. The authors construct a controlled dataset, measure leakage rates under prompting, fine-tuning, and their combination, report that both mechanisms produce measurable leakage that compounds when used together, evaluate a prompt-level mitigation, and conclude that fine-tuning without context prompts yields the best accuracy-leakage trade-off. Code and dataset are released publicly.

Significance. If the measured leakage rates prove robust, the work identifies a concrete and practically relevant privacy vulnerability in the standard practice of domain-adapting ASR systems for professional use. The empirical focus on two common customization mechanisms, the comparison of their leakage profiles, and the public release of code and dataset constitute clear strengths that support reproducibility and follow-on work.

major comments (2)

[Abstract / Evaluation] Abstract and Evaluation section: the central claim rests on leakage rates measured on a constructed dataset, yet the manuscript provides no quantitative results, error bars, or explicit description of how phonetic similarity is operationalized, how leakage is scored (exact match versus semantic), or how baseline accuracy is established; without these details the validity of the reported rates cannot be assessed.
[Evaluation] Evaluation protocol: the claim that the observed transcription constitutes a practical privacy leak depends on the dataset and measurement protocol accurately reflecting real-world attacker capabilities (e.g., whether the attacker is assumed to know the spoken audio or only observes the output transcription). The manuscript does not specify these assumptions, which are load-bearing for the privacy-risk conclusion.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments identifying areas where our evaluation details require greater clarity. We address each point below and will revise the manuscript accordingly to strengthen the presentation of our results and threat model.

read point-by-point responses

Referee: [Abstract / Evaluation] Abstract and Evaluation section: the central claim rests on leakage rates measured on a constructed dataset, yet the manuscript provides no quantitative results, error bars, or explicit description of how phonetic similarity is operationalized, how leakage is scored (exact match versus semantic), or how baseline accuracy is established; without these details the validity of the reported rates cannot be assessed.

Authors: We agree that the manuscript would benefit from more explicit descriptions of these elements to allow readers to fully assess the reported rates. The Evaluation section contains the quantitative leakage rates (including comparisons across prompting, fine-tuning, and combined settings) along with baseline accuracy measurements, but we will expand the text to detail: (i) phonetic similarity via phoneme-level Levenshtein distance with a threshold of 2, (ii) leakage scoring as exact token match to the context word, (iii) baseline accuracy computed on the same audio without domain adaptation, and (iv) error bars from 5 random seeds. We will also insert a concise summary of the key numerical findings into the abstract. revision: yes
Referee: [Evaluation] Evaluation protocol: the claim that the observed transcription constitutes a practical privacy leak depends on the dataset and measurement protocol accurately reflecting real-world attacker capabilities (e.g., whether the attacker is assumed to know the spoken audio or only observes the output transcription). The manuscript does not specify these assumptions, which are load-bearing for the privacy-risk conclusion.

Authors: We will add an explicit 'Threat Model' subsection at the start of the Evaluation section. Our model assumes a passive attacker who (a) knows the domain-adaptation context or fine-tuning data, (b) can submit arbitrary audio queries to the adapted model, and (c) observes only the resulting transcription. The constructed dataset operationalizes this by supplying phonetically similar distractors exclusively in the context while the spoken audio contains a different word; leakage is measured when the model outputs the context word. This matches common real-world scenarios where an adversary has access to the customized model but not the raw audio. We will also discuss the limitations of this model relative to stronger attackers who might have partial audio knowledge. revision: yes

Circularity Check

0 steps flagged

Empirical measurement study with no derivation chain or fitted predictions

full rationale

The paper is an empirical measurement study that constructs a controlled dataset and directly measures leakage rates under prompting and fine-tuning. No equations, first-principles derivations, parameter fitting presented as predictions, or self-citation load-bearing steps are described. Central quantities (leakage rates) are obtained by direct observation on the new dataset rather than by reduction to prior inputs or self-referential definitions, so the work is self-contained.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review; no explicit free parameters, axioms, or invented entities are stated in the provided text.

pith-pipeline@v0.9.1-grok · 5667 in / 999 out tokens · 20358 ms · 2026-06-29T12:32:30.656162+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

3 extracted references · 2 canonical work pages · 1 internal anchor

[1]

Fleurs: Few-shot learning evaluation of universal representations of speech,

Quantifying memorization across neural lan- guage models. InThe Eleventh International Confer- ence on Learning Representations. Nicholas Carlini and David Wagner. 2018. Audio ad- versarial examples: Targeted attacks on speech-to- text. In2018 IEEE Security and Privacy Workshops (SPW), pages 1–7. Feng-Ju Chang, Jing Liu, Martin Radfar, Athanasios Mouchtar...

work page arXiv 2018
[2]

LoRA: Low-Rank Adaptation of Large Language Models

Contextual Biasing Speech Recognition in Speech-enhanced Large Language Model. InInter- speech 2024, pages 257–261. Xun Gong, Anqi Lv, Wangyou Zhang, Zhiming Wang, Huijia Zhu, and Yanmin Qian. 2025. BR-ASR: effi- cient and scalable bias retrieval framework for con- textual biasing ASR in speech LLM. In26th Annual Conference of the International Speech Com...

work page internal anchor Pith review Pith/arXiv arXiv 2024
[3]

{transcript}

Llamafactory: Unified efficient fine-tuning of 100+ language models. InProceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (V olume 3: System Demonstra- tions), Bangkok, Thailand. Association for Computa- tional Linguistics. Zoom Video Communications. 2025. Zoom launches AI companion 3.0 with agentic workflows, transf...

2025

[1] [1]

Fleurs: Few-shot learning evaluation of universal representations of speech,

Quantifying memorization across neural lan- guage models. InThe Eleventh International Confer- ence on Learning Representations. Nicholas Carlini and David Wagner. 2018. Audio ad- versarial examples: Targeted attacks on speech-to- text. In2018 IEEE Security and Privacy Workshops (SPW), pages 1–7. Feng-Ju Chang, Jing Liu, Martin Radfar, Athanasios Mouchtar...

work page arXiv 2018

[2] [2]

LoRA: Low-Rank Adaptation of Large Language Models

Contextual Biasing Speech Recognition in Speech-enhanced Large Language Model. InInter- speech 2024, pages 257–261. Xun Gong, Anqi Lv, Wangyou Zhang, Zhiming Wang, Huijia Zhu, and Yanmin Qian. 2025. BR-ASR: effi- cient and scalable bias retrieval framework for con- textual biasing ASR in speech LLM. In26th Annual Conference of the International Speech Com...

work page internal anchor Pith review Pith/arXiv arXiv 2024

[3] [3]

{transcript}

Llamafactory: Unified efficient fine-tuning of 100+ language models. InProceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (V olume 3: System Demonstra- tions), Bangkok, Thailand. Association for Computa- tional Linguistics. Zoom Video Communications. 2025. Zoom launches AI companion 3.0 with agentic workflows, transf...

2025