{"id":"0c926151-880d-4866-bf87-d1ac8a820c37","arxiv_id":"2607.16870","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Three seconds of speech tokens from Moshi, Higgs3, Kimi-Audio, or Qwen3-Omni allow a trained inversion model to recover speaker embeddings with cosine similarity above 0.70 against a pretrained speaker encoder.","lead":"This paper shows that speech tokens emitted by the frontends of four speech AI models carry enough speaker information to reconstruct a usable voiceprint embedding. A trained attacker model needs only three seconds of intercepted tokens to recover a speaker embedding that closely matches the output of a speaker-verification encoder.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Speaker-verification metrics may use same-utterance positive trials; if so, recovered embeddings are not yet shown to act as cross-utterance voiceprints.","rationale":"The reader's weakest assumption was the adversary's query access to the exact frozen frontend. That is a valid deployment limitation, and the paper states it clearly in the threat model. However, a more load-bearing concern for the paper's central 'voiceprint' claim is the evaluation protocol for speaker verification: if positive trials are same-utterance, then the high CosSim and low EER may reflect utterance-specific information rather than speaker identity. This is not an internal contradiction; it is an unspecified and testable evaluation detail. The paper does provide independent support through the speaker-disjoint leakage diagnostic (Table I), which suggests tokens do retain identity. But Table I does not validate the recovered embeddings in the attacker-specified encoder space. A cross-utterance trial protocol would settle whether the SpInv embeddings truly behave as voiceprints. The verdict remains conditional: the central claim is plausible, but the verification-sense metrics need clarification before acceptance.","tokens_in":12310,"tokens_out":9966,"duration_ms":120904,"concrete_test":"Re-run the Tables II/III/IV evaluation with positive trials formed from two distinct utterances of the same speaker: extract the target embedding z from utterance A's waveform, recover \\hat{z} from a 3-second token slice of utterance B (a different recording of the same speaker), and compute EER/Acc. If EER remains near 0.017-0.03, the voiceprint claim is confirmed; if EER degrades materially (e.g., above 0.08) relative to the same-utterance setup, the current metrics overstate identity-level leakage.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central practical claim is that SpInv recovers a usable voiceprint from a 3-second token slice. In Section IV.C, the evaluation compares the embedding recovered from a local view of utterance i with the target embedding extracted by the frozen speaker encoder from the teacher view of the same utterance. The paper never states that positive verification trials are formed from two different utterances of the same speaker. If they are same-utterance, the reported CosSim, Acc, and EER partly reflect acoustic, channel, and content cues of that one recording, not speaker identity across sessions. A voiceprint-leakage claim requires cross-utterance matching: enrollment from one recording and probe from another. Table I's speaker-disjoint classification gives independent evidence that tokens carry identity, but it does not show that the recovered embeddings in the attacker-specified ECAPA space generalize across utterances. The current reporting therefore leaves the main 'speaker-verification sense' claim under-specified and potentially inflated.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper asks whether speech tokens exposed by end-to-end speech language models leak voiceprints. It proposes Audio BERT (AuB), a trainable model that maps discrete codebook IDs and/or continuous frontend features into token embeddings and aggregates them with a BERT backbone, and SpInv, a two-stage training procedure (DINO/VICReg-style distillation pretraining plus ArcMargin discriminative fine-tuning) to recover a speaker embedding in an attacker-specified speaker-encoder space from a short intercepted token slice. The method is evaluated on Moshi, Higgs3, Kimi-Audio, and Qwen3-Omni frontends under speaker-disjoint VoxCeleb protocols, reporting cosine similarity above 0.70 on the three token-based frontends, Acc above 0.93, and EER as low as about 0.017 with 3-second input. Ablations vary input duration and target speaker encoder.","tokens_in":12516,"tokens_out":5982,"duration_ms":63340,"significance":"If the reported results hold, the paper demonstrates a timely and practically relevant privacy channel: discrete speech tokens exposed at the frontend/downstream boundary of speech language models preserve enough speaker information for an adversary to recover useful embeddings in a chosen speaker space. The work is stronger than much prior audio-privacy work in evaluating on speaker-disjoint VoxCeleb splits, covering four different frontend interface types, and testing multiple target speaker encoders. The threat model is explicit, and the experiments use public datasets and pretrained models. The main weakness is that the verification-style metrics appear to compare embeddings extracted from the same utterance, so the central claim that the recovered embeddings act as cross-utterance voiceprints is not yet established. If this is corrected with cross-utterance trial evaluation, the contribution would be significant for the speech-privacy community.","major_comments":[{"comment":"The verification protocol is underspecified and appears to use same-utterance enrollment and probe. In Section III.D the student model receives local views derived from utterance i and the frozen teacher receives a teacher view of the same utterance i; the alignment loss and the reported CosSim compare the recovered embedding \\hat z_i with z_i = F(x_i^t) for the same utterance. The paper never states that positive verification trials use different utterances of the same speaker. If Acc/EER are computed from same-utterance pairs, the numbers can reflect acoustic, channel, and content cues of one recording rather than speaker identity across sessions. A voiceprint-leakage claim requires cross-utterance matching: enrollment from one recording, probe from another. Please state the trial construction explicitly and report cross-utterance CosSim, Acc, EER, and score distributions, with at leas","section":""},{"comment":"All headline numbers come from a single training run without error bars or seed variance. Given the random initialization, DINO prototype/EMA schedules, and multi-loss training, it is not possible to tell whether differences such as Moshi EER 0.0305 vs Higgs3 EER 0.0174 are meaningful or within run-to-run noise. Reporting at least three seeds with mean and std (or a similar stability measure) for the main tables would materially strengthen the empirical claim.","section":""}],"minor_comments":[{"comment":"For Acc, the threshold is said to be the optimal cosine-score decision boundary, which is optimistic if chosen on the test set. Please clarify whether EER/minDCF thresholds are selected on held-out development data or on the test set, and report calibration details.","section":""},{"comment":"For purely continuous frontends such as Qwen3-Omni, there are no codebooks; the text should state explicitly that \\mathbf{e}_n is omitted (or set to zero) and clarify how AuB handles a stream with only the MLP-projected continuous feature.","section":""},{"comment":"The minDCF values at 1 s and 2 s (0.6670, 0.4050) are much larger than the 3 s value and look qualitatively inconsistent with the reported Acc values. Please check the definition/normalization and provide an explanation, since this may be a typo or an artifact of the evaluation threshold.","section":""},{"comment":"Please state the number of evaluation utterances N in Tables II--III, how many random crops are used per utterance at evaluation time, and whether the target frontend outputs are used deterministically. This is needed for reproducibility and for interpreting CosSim as an average over utterances or crops.","section":""},{"comment":"The paper does not mention release of code or trained checkpoints. Given the multi-stage training and many hyperparameters, a public release or a detailed reproducibility appendix would be valuable.","section":""}],"recommendation":"major_revision","confidential_remarks":""},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Chao, here's my read on the speech-token inversion paper.\n\nThe central claim—three seconds of frontend output from Moshi, Higgs3, Kimi-Audio, or Qwen3-Omni can be inverted into an attacker-specified speaker embedding with CosSim > 0.7—is plausible and worth taking seriously. The paper does a genuinely useful thing: it's the first systematic study I know of that treats speech tokens as a privacy surface, and it evaluates four different frontend types under a speaker-disjoint protocol on VoxCeleb. The SpInv/AuB architecture is not a conceptual breakthrough—codebook embedding lookup plus BERT aggregation plus a two-stage loss is sensible engineering—but the evaluation is honest, the ablations on duration and target encoder space are useful, and the authors avoid overclaiming waveform reconstruction. The limitation and ethics statements are appropriately scoped.\n\nThe soft spots are real, though. The biggest one is the verification protocol. Section IV.C reports cosine similarity between the embedding recovered from a local view of utterance i and the target embedding from the teacher view of the same utterance i. But the paper never says whether the Acc/EER/minDCF numbers come from cross-utterance trials—matching a probe from one recording to an enrollment from another—or from same-utterance pairings. If they're same-utterance, the high accuracy partly reflects acoustic and channel match of one recording, not a voiceprint that generalizes across sessions. The practical 'voiceprint leakage' claim needs cross-utterance evidence. Table I's speaker-disjoint classification gives independent evidence that tokens carry identity, so I don't think the whole paper collapses, but the verification metrics as reported are ambiguous. This is the first thing I'd push a referee to ask about.\n\nSecond, there are no error bars or variance across training runs. With that many loss weights and schedule parameters, single numbers without seed variance make it hard to judge stability. Third, there's no baseline—e.g., averaging token embeddings or a shallow classifier on token statistics—so the increment attributable to SpInv is unquantified. Fourth, the threat model gives the attacker query access to the exact target frontend and knowledge of codebook sizes; that's plausible for open models but not for encrypted or rate-limited interfaces, and the paper could state this boundary more explicitly.\n\nMy overall take: this deserves a serious referee. The core result is likely real, but the current write-up doesn't nail down the most important metric. I'd send it to review with a request for clarification on the trial construction, plus added variance and a baseline. Good reading-group material.","headline":"Plausible and important claim about speech-token voiceprint leakage, but the verification protocol is under-specified—same-utterance trials could inflate the headline numbers.","tokens_in":13081,"tokens_out":4367,"would_cite":true,"duration_ms":43686,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Speech tokens exposed by end-to-end speech models leak voiceprints: three seconds of frontend output lets SpInv recover a speaker embedding with cosine similarity above 0.70.","keywords":["speech tokens","speaker inversion attack","voiceprint leakage","speech language models","Audio BERT","speaker embedding recovery","speech privacy","discrete speech frontends"],"falsifier":"Take one of the four frontends and re-run the same three-second protocol while encrypting or one-time-padding the token IDs between frontend and backbone (or after a single codebook-vocabulary update), then measure whether cosine similarity stays above 0.70; if it drops, the leakage is conditional on the plaintext stable-interface assumption rather than inherent to the acoustic content.","tokens_in":12149,"feed_emoji":"🎙️","tokens_out":6345,"duration_ms":60292,"temperature":0.7,"pith_summary":"The paper asks whether the discrete speech tokens that end-to-end speech language models use to carry audio still harbor enough speaker identity to be inverted back into a voiceprint. It claims yes: an attacker who observes three seconds of token output can train a recovery model, SpInv, to produce an embedding in the attacker's chosen speaker-recognition space that matches the victim's true embedding at cosine similarity above 0.70 for Moshi, Higgs3, Kimi-Audio, and Qwen3-Omni. This matters because such tokens are often the natural hand-off point in split inference, where user speech is tokenized locally and sent to a cloud model; if the claim holds, that hand-off is a practical voiceprint-leakage channel. The paper further argues that the leakage appears across discrete codebook, hybrid discrete-continuous, and fully continuous frontend interfaces, and that training with query access to the frozen frontend on public data is sufficient to build the attack.","feed_headline":"Three seconds of speech tokens recover a voiceprint","feed_subtitle":"Intercepted token streams from four speech LMs yield speaker embeddings with cosine similarity above 0.70.","key_machinery":"AuB (Audio BERT) is the trainable component that carries the inversion. For each frame it maps every discrete codebook ID to a learned embedding, sums the embeddings across codebooks, fuses in any time-aligned continuous features through an MLP, and feeds the sequence through a BERT backbone; the CLS output is projected into the attacker-selected speaker-embedding space. SpInv trains AuB in two stages: a distillation stage aligns recovered embeddings to a frozen teacher speaker encoder while EMA self-distillation (DINO-style) and variance/covariance regularizers prevent collapse, followed by a discriminative fine-tuning stage with an ArcMargin speaker classifier. Only AuB and the projection","core_discovery":"SpInv shows that the high-level speech representations emitted by end-to-end speech language models preserve speaker-discriminative information that can be inverted without the waveform. With only a three-second slice of exposed frontend output, the trained AuB model plus projection head recovers an embedding in an attacker-specified speaker-encoder space: cosine similarity exceeds 0.70 on unseen speakers for the three token-based frontends, speaker-verification accuracy reaches 0.97-0.98 on VoxCeleb2-test, and equal error rates drop as low as 0.017. A dedicated leakage diagnostic on the VoxCeleb1 verification split confirms that all four interfaces, including the purely continuous Qwen3-Omn","pith_inferences":["A natural next step the paper leaves implicit: if recovered embeddings transfer even partially to enrollment in commercial voice-biometric systems, intercepted token bursts could support unauthorized speaker matching without any waveform reconstruction.","The dependency on plain, stable token interfaces suggests a cheap defensive test: encrypt, one-time-pad, or periodically remap token IDs between frontend and backbone, then see whether recovery collapses; if it does, the leakage is largely an interface-design problem.","Because any legitimate downstream model that consumes speech tokens internally could run the same inversion, privacy protection likely has to live inside the tokenizer itself, for instance through explicit speaker-suppression objectives rather than by hiding the token stream.","The method's compatibility with continuous frontend features implies the attack surface may extend beyond discrete speech tokens to any learned audio representation passed between modules, which is worth probing on other model families."],"forward_implications":["If the claim holds, split-inference deployments that transmit frontend token streams to a downstream model expose speaker identity to any party able to observe that stream.","Three seconds of exposed tokens is enough for near-practical speaker verification: recovered embeddings reach EER as low as 0.017 and accuracy above 0.96 on held-out speakers.","The leakage is not tied to one codec design; residual-codebook, eight-codebook, hybrid semantic-plus-continuous, and fully continuous frontends all show it.","Longer observations strengthen recovery up to saturation around eight to ten seconds, so even short intercepted utterances are a meaningful risk.","Changing the attacker's target speaker encoder does not neutralize the attack; SpInv adapts to ECAPA-TDNN, ERes2Net, and CAM++ spaces with similar recovery quality."],"fun_headline_variants":["3 seconds of speech tokens can recover a voiceprint","Speaker inversion attack exposes voiceprints from tokens","SpInv: turn 3s of speech tokens into a speaker embedding","Token streams leak voiceprints—SpInv hits 0.70 cosine","Speech LMs leak speaker identity via invertible tokens"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The attack's success depends on the adversary being able to query the victim's exact frozen frontend on public audio during attack training, and on the deployed interface continuing to expose plain, unchanged token IDs with known codebook counts and vocabulary sizes when the attack runs.","fun_headline_variants_meta":{"raw":{"variants":["3 seconds of speech tokens can recover a voiceprint","Speaker inversion attack exposes voiceprints from tokens","SpInv: turn 3s of speech tokens into a speaker embedding","Token streams leak voiceprints—SpInv hits 0.70 cosine","Speech LMs leak speaker identity via invertible tokens"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000312,"raw_usage":{"total_tokens":1591,"prompt_tokens":701,"completion_tokens":890,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":445,"completion_tokens_details":{"reasoning_tokens":808}},"tokens_in":445,"tokens_out":890,"duration_ms":7635,"temperature":1.0,"reasoning_tokens":808,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T19:41:15.349591+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take one of the four frontends and re-run the same three-second protocol while encrypting or one-time-padding the token IDs between frontend and backbone (or after a single codebook-vocabulary update), then measure whether cosine similarity stays above 0.70; if it drops, the leakage is conditional on the plaintext stable-interface assumption rather than inherent to the acoustic content.","supporting_citations":[],"review_version":1}