{"id":"86dba744-933d-4892-8319-b49110912d5d","arxiv_id":"2504.16609","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"GEIA embedding-inversion results are reproducible across five sentence embedders, and a log-likelihood comparison of masked versus alternative sentences indicates sentence embeddings can expose masked named entities, though attribution to model training data is not fully established.","lead":"This study repeats the GEIA attack that reconstructs sentences from their embedding vectors and finds its results mostly match the original paper across five sentence embedding models. It then asks whether the same attack can recover private details that were masked out of the input but are known from the embedding model's training data, with suggestive but incomplete evidence.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The training-data leakage claim is not yet supported: AltLex membership in victim pretraining is asserted (Section 4.1) without audit, and the Section 5.2 experiment cannot rule out that the masked embedding encodes world-knowledge inference rather than specific memorized training examples.","rationale":"The reader's assessment identifies the same load-bearing weakness: the AltLex membership assumption is unverified, and the experiment lacks a control that separates memorized training data from general world knowledge. I agree with the CONDITIONAL verdict because the reproduction portion of the paper is valuable and largely well-executed: the GEIA metrics closely match the original paper, the authors transparently report baseline discrepancies, and the leakage experiment includes a useful with/without-embedding control. However, the paper's new contribution -- training-data leakage -- is precisely the part that needs re-framing and additional controls. The concern I highlight is not that the experiment is wrong, but that the interpretation as 'exact sensitive information that the embedding model saw during training' overreaches: without membership verification and a synthetic-entity control, the observed likelihood gap could equally reflect the embedding's encoding of generic entity knowledge from Wikipedia-like text. The proposed concrete tests directly target this distinction: a fictitious-entity control would show whether the effect is training-data-specific, and a canary fine-tuning test would establish a causal link to training set membership. Until such evidence is provided, the paper should be considered a solid reproduction with a suggestive but unproven leakage extension, which matches the reader's CONDITIONAL stance.","tokens_in":15227,"tokens_out":3662,"duration_ms":35676,"concrete_test":"Construct a matched control set of sentences with fictitious entities (e.g., synthetic names, dates, and locations that cannot appear in any victim's pretraining data), mask them with the same LLM reasoner, and run the Section 3.2 pipeline. If original-vs-similar masked-token likelihood differences remain as large (≥20%) when f(x) is present, the effect is not training-data-specific. Conversely, run a canary test: fine-tune one victim (e.g., SimCSE-BERT) on a small set of synthetic AltLex-style sentences, then test whether the attack succeeds only on fine-tuned (seen) sentences and not on held-out synthetic sentences; success only on seen sentences would validate the training-leakage interpretation. Additionally, report exact n-gram overlap between AltLex and each victim's known pretraining corpus to substantiate the Section 4.1 membership assertion.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's new central claim (Section 5.2: 'masked sentence embeddings do carry significant sensitive information regarding the train set') requires that (a) AltLex sentences were actually seen by SRoBERTa, SimCSE, ST5, and MPNet during pretraining, and (b) the higher conditional likelihood of original vs similar masked tokens is caused by recollection of those training examples, not by the victim model's general linguistic or world knowledge. Neither condition is established. Section 4.1 states 'Altlex also formed part of the training set for the sentence embedding models used' with no citation, dataset audit, or overlap analysis, despite the victims having heterogeneous pretraining corpora (RoBERTa/SimCSE: Wikipedia+BookCorpus; ST5: C4; MPNet: BookCorpus/Wikipedia). More importantly, the experiment compares the attacker's likelihood for the true original entity versus a reasoner-generated plausible alternative, conditioned on the masked embedding. Since the masked sentence retains rich context (e.g., 'was a commander of the <ORGANIZATION> during the <LOCATION> campaign... seeing much of <PERSON>'), the victim embedding may simply encode strong contextual and entity priors learned from general text, which is not 'exact sensitive information that the embedding model saw during training.' The control without f(x) shows the attacker's own language-model prior is neutral, but it does not control for knowledge already encoded in the victim's parameters; the observed +25-33% masked-token likelihood gap is consistent with the embedding functioning as a knowledge-retrieval cue rather than as a leaked training example. Thus the headline training-data-leakage claim is over-strong; the evidence supports only embedding-conditioned entity inference.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper is a reproducibility study of the GEIA (Generative Embedding Inversion Attack) framework for sentence embeddings, combined with a new research question: whether sentence embeddings leak information from the embedding model's training data. The authors replicate GEIA's original experiments on PersonaChat and QNLI with four victim sentence-embedding models (SRoBERTa, SimCSE-BERT, SimCSE-RoBERTa, ST5, MPNet) and confirm the original paper's four claims, with small discrepancies attributed to stochasticity and implementation details. For the new analysis, they sample sentences from AltLex, use an LLM reasoner (GLM-4 or Llama-3.1) to create masked versions (sensitive entities replaced with placeholders) and alternative versions (entities replaced with plausible substitutes), embed the masked sentences with the victim model, and feed the embedding concatenated with either the original or alternative text to the GEIA attacker. They compare the attacker's log-likelihoods for the original versus alternative tokens, both with and without access to the sentence embedding.","tokens_in":15486,"tokens_out":4401,"duration_ms":42078,"significance":"If the new claim were established, this paper would extend embedding inversion attacks from input reconstruction to training-data extraction, which would be a meaningful privacy contribution. The reproduction component is useful and appears mostly faithful to the original GEIA results, with transparent reporting of discrepancies. However, the central novel claim is not yet supported: the assertion that AltLex belongs to the pre-training corpora of all victim models is unverified, and the experimental design does not rule out that the observed likelihood differences arise from contextual inference or world knowledge encoded in the victim's parameters rather than from memorization of specific training examples. The paper is therefore a solid reproducibility study with an interesting but currently unproven extension.","major_comments":[{"comment":"The statement that 'Altlex also formed part of the training set for the sentence embedding models used' is asserted without citation, dataset audit, or overlap analysis. The victim models were pre-trained on heterogeneous corpora (e.g., RoBERTa/SimCSE on Wikipedia and BookCorpus, ST5 on C4, MPNet on BookCorpus/Wikipedia), and no evidence shows that the specific AltLex sentences sampled for the experiments actually appear in those corpora. This assumption is load-bearing for the Section 5.2 conclusion that the attacker recovers 'exact sensitive information that the embedding model saw during training.' Without verification, the observed likelihood differences could reflect general knowledge (e.g., that Rommel was a German commander in WWII) rather than training-data memorization. The authors should provide an overlap analysis (e.g., exact or near-duplicate sentence matching against the pre-training data) or include a control set of sentences known to be absent from the training corpora.","section":"Section 4.1"},{"comment":"The without-f(x) control is insufficient to support the training-data-leakage claim. This control shows that the attacker's parametric memory alone does not produce the difference, but it does not rule out information already stored in the victim model's parameters. The masked sentences retain rich context (e.g., 'Rommel was a commander of the <ORGANIZATION> during the <LOCATION> campaign... seeing much of <PERSON>'), so the victim embedding may encode strong contextual priors over entities that are plausible in that context, which is not equivalent to memorizing specific training examples. To isolate memorization, the authors should include a condition where the masked entity is unlikely to appear in the victim's pre-training data, or compare against a set of alternatives matched for contextual plausibility.","section":"Section 5.2, Table 6"},{"comment":"The analysis appears to select the masked-token subset in a post-hoc manner: the whole-sentence aggregation is reported as not indicating leakage-specific differences (the difference persists without f(x)), and only then the masked-token subset is examined, where the pattern is observed. No correction for multiple comparisons is applied, and only p-values are reported. The paper should provide the number of samples, effect sizes, and confidence intervals for each cell in Table 6, and clarify whether the masked-token analysis was planned a priori or is exploratory.","section":"Section 5.2"},{"comment":"The operational definition of training-data leakage is circular in practice: a higher likelihood for the original masked tokens is taken as evidence of training-data leakage, but this pattern could also arise from contextual inference or from the victim embedding encoding common-sense factual knowledge that is not tied to any specific training example. The paper should formulate a sharper null hypothesis (e.g., that the embedding provides no more information about the masked entity than a well-calibrated language model with access to the same context) and test it with appropriate controls, such as comparing embeddings of sentences that are paraphrases but not verbatim training examples.","section":"Section 3.2"}],"minor_comments":[{"comment":"Table 2 reports that 10% of AltLex is used (9,851 sentences) but does not describe the sampling procedure; please specify how this subsample was drawn and whether it is representative of the full dataset.","section":"Section 4.1"},{"comment":"The phrase 'make its an excellent candidate' contains a typo; it should be 'make it an excellent candidate.'","section":"Section 4.3"},{"comment":"The statement 'the projection was not parametrized' is ambiguous; it likely means that the original authors did not provide a parameterized projection module, but the sentence is unclear. Please clarify the intended meaning.","section":"Section 6.2"},{"comment":"The caption does not explain the sign convention of the 'Distr. Comparison Original vs Similar' column. It should state explicitly that positive values indicate higher likelihood for the original tokens, and specify the units (percentage points).","section":"Table 6"},{"comment":"The procedure for 'pre-appending' the sentence embedding to the original and alternative sentences is not described precisely in the text. Figure 2 is too coarse; a precise description of the concatenation mechanism (e.g., how the embedding is projected and inserted into the decoder's token sequence) is needed for reproducibility.","section":"Section 3.2 and Figure 2"},{"comment":"The abstract and the conclusion state that the method reveals 'data that sentence embedding models have been pre-trained on,' but the evidence presented does not yet establish a connection to pre-training data specifically. This phrasing overstates the current results and should be tempered until the training-data overlap is verified.","section":"Abstract and Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"The reproduction of GEIA is a useful contribution, and the authors are transparent about the discrepancies they encountered. The novel leakage claim, however, is the paper's main selling point and it is not yet convincing. The core problem is the unsupported assertion that AltLex is in the victim models' pre-training data; without that, the experiments could be explained by contextual inference. I would encourage the editor to ask the authors to either provide a data audit with concrete overlap evidence or add a control dataset of sentences not in the pre-training corpora. If they can do that, the paper could be a strong contribution; as it stands, the central claim overreaches."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Good reproduction of GEIA, and a useful new probe for embedding leakage — but the headline training-data claim is ahead of the evidence. The four GEIA reproduction claims mostly hold up: reported metrics are close to the original, code is released, and the authors are transparent about where their baselines diverge. That alone makes this a worthwhile paper for the reproducibility track.\n\nThe new experiment is a clever extension: mask sensitive entities, embed the masked sentence, then ask the trained GEIA attacker to score the original tokens against a plausible alternative, with and without the embedding. The without-embedding control is the right instinct, and the t-test results indicate the embedding does bias generation toward the original masked tokens.\n\nThe problem is what that bias means. Section 4.1 asserts AltLex 'formed part of the training set' for SRoBERTa, SimCSE, ST5, and MPNet, with no citation, overlap analysis, or audit. Those models were trained on different corpora (Wikipedia+BookCorpus, C4, etc.), and AltLex is a processed Wikipedia-derived resource, not a guaranteed pretraining ingredient. Without verified membership, the +25–33% masked-token likelihood gap can be explained just as well by the victim model's contextual inference: the unmasked context ('was a commander of the <ORGANIZATION> during the <LOCATION> campaign') is enough to make Rommel likely. That is knowledge stored in the parameters, not evidence of direct exposure to that specific AltLex sentence. The paper's own framing — 'exact sensitive information that the embedding model saw during training' — is stronger than the experiment supports.\n\nThe fix is not hard: re-frame the contribution as embedding-conditioned entity inference, or add controls that separate memorization from general knowledge — for example, use masked entities that the context does not disambiguate, or compare against a verified membership set with known duplicate/near-duplicate sentences. The reproduction half of the paper stands regardless.\n\nThis paper deserves a serious referee. The reproducibility results are solid, and the leakage question is important even if this first pass over-claims. Send it to review, but the authors should be pushed to temper the training-data claim or provide the missing audit.","headline":"Solid GEIA reproduction plus a clever leakage probe, but the training-data-leak claim is not yet supported; worth refereeing with a demand to temper it.","tokens_in":16100,"tokens_out":2842,"would_cite":false,"duration_ms":28625,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that masked sentence embeddings leak sensitive information from the embedding model's training data, and that the leakage is tied to the embedding itself.","keywords":["sentence embeddings","embedding inversion attack","training data leakage","privacy","large language models","adversarial retrieval","reproducibility","named entity recovery"],"falsifier":"Run the same masked-token log-likelihood comparison on a corpus verifiably absent from the victim models' training data, or audit AltLex's overlap with each victim's pre-training corpus. If the original-over-similar gap persists for absent sentences, the effect is contextual inference rather than training-data leakage; if AltLex was never in the training data, the paper's central interpretation collapses.","tokens_in":15020,"feed_emoji":"🔓","tokens_out":6107,"duration_ms":56481,"temperature":0.7,"pith_summary":"This paper is a reproducibility study plus an extension of the Generative Embedding Inversion Attack (GEIA), which trains a GPT-2 decoder to reconstruct sentences from frozen sentence embeddings. The authors reproduce GEIA across five victim models and largely confirm its original claims, with GEIA's own metrics landing within 1–2% of the reported values. Their new contribution is a training-data leakage test: using LLM reasoners to mask named entities in sentences from AltLex, they show that a masked sentence's embedding makes the attacker assign higher log-likelihood to the original masked entity than to a plausible alternative, by 25–34% when averaging over masked tokens. When the embedding is withheld, the gap disappears, which they take as evidence that the leakage comes from the victim embedding, not the attacker's memory. If correct, this means sentence embeddings can reveal facts the embedding model memorized during pre-training, even when the input sentence no longer contains those facts.","feed_headline":"Embeddings leak masked names from training data","feed_subtitle":"Recovered masked entities are 25–34% more likely than plausible alternatives, but only when the embedding is present.","key_machinery":"The central object is the GEIA attacker: a randomly initialized GPT-2 medium decoder, trained from scratch on PersonaChat, with a projection module that aligns the victim sentence embedding to the decoder's embedding space. The extension's mechanism is a likelihood-difference test: for each AltLex sentence, two LLM reasoners (GLM-4 and Llama-3.1) produce a masked version and a similar alternative; the masked sentence embedding f(x) is concatenated to each candidate, and the attacker's log-likelihood for the masked tokens is compared. A control condition withholds f(x), isolating leakage from the parametric knowledge of the attacker itself.","core_discovery":"On the paper's own terms, the central finding is that masked sentence embeddings do carry significant sensitive information regarding the training set, posing a considerable threat for embedding models. Concretely, for sentences sampled from AltLex, an LLM masks named entities; the victim model embeds the masked sentence; and the GEIA attacker computes generation log-likelihoods for the original sentence versus a similar alternative. Averaging only over the masked tokens, the original tokens are 25–34% more likely than the alternative when the embedding is provided, whereas without the embedding the difference is near zero or slightly negative. This pattern holds for SRoBERTa and SimCSE-BERT victims with both GLM-4 and Llama-3.1 reasoners, and the paper reports t-tests with p-values close to zero. The authors conclude that attackers are able to identify exact sensitive information that the embedding model saw during training.","pith_inferences":["One extension the authors do not pursue is a top-1 recovery rate: enumerate candidate entity fillings for a masked slot, rank them by the attacker's log-likelihood, and measure how often the true entity is the most likely; that would turn the likelihood gap into a concrete extraction rate.","A natural control the paper does not run is the same comparison on sentences from a corpus verifiably absent from pre-training data; that control would separate memorization from the contextual reasoning power of the attacker.","Because the gap appears only for masked tokens and only when f(x) is supplied, the method could be developed into an auditing tool that tests which entities an embedding model has memorized, by masking candidates and observing which fillings receive high likelihood."],"forward_implications":["An adversary with access to sentence embeddings can recover named entities that are absent from the input text but present in the victim model's training data.","Masking sensitive entities at inference time does not by itself protect against embedding inversion, because the embedding still encodes the masked entity.","The leakage signal is attributable to the embedding itself: without f(x), original and alternative tokens are equally likely, so the attacker's memory is not the source.","The effect replicates across two victim model families and two LLM reasoners, suggesting a general vulnerability rather than a quirk of one model."],"supporting_citations":[{"why":"Introduces GEIA, the generative inversion attack whose claims this paper reproduces and whose trained attacker is reused for the leakage test.","marker":"[21]"},{"why":"Supplies AltLex, the Wikipedia-derived dataset of sentences claimed to be part of the victim models' pre-training data.","marker":"[14]"},{"why":"Defines the MLC and MSP baseline attacks that GEIA is compared against in the reproducibility study.","marker":"[30]"},{"why":"Provides the Sentence-RoBERTa victim model used to produce embeddings.","marker":"[27]"},{"why":"Provides the Sentence-T5 victim model used to produce embeddings.","marker":"[23]"},{"why":"Provides the MPNet victim model used to produce embeddings.","marker":"[31]"},{"why":"Provides the SimCSE victim models used to produce embeddings.","marker":"[9]"},{"why":"Provides PersonaChat, the auxiliary dataset on which the attacker is trained from scratch.","marker":"[15]"},{"why":"Provides the GLM-4 few-shot reasoner that creates masked and alternative variants of AltLex sentences.","marker":"[10]"},{"why":"Provides the Llama-3.1 reasoner used as a second masking and alternative-generation model.","marker":"[8]"}],"fun_headline_variants":["Masked names leak from sentence embeddings","Embeddings expose masked training data","Attack recovers masked entities from embeddings","Embedding inversion reveals masked names","Masked tokens leak via embedding inversion"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion that the recovered entities are training-data leakage assumes AltLex actually appears in the pre-training corpus of every victim model; the paper asserts this without a membership audit, and if that membership is wrong the same likelihood gap could be contextual inference from the unmasked parts of the sentence.","fun_headline_variants_meta":{"raw":{"variants":["Masked names leak from sentence embeddings","Embeddings expose masked training data","Attack recovers masked entities from embeddings","Embedding inversion reveals masked names","Masked tokens leak via embedding inversion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000226,"raw_usage":{"total_tokens":1456,"prompt_tokens":918,"completion_tokens":538,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":534,"completion_tokens_details":{"reasoning_tokens":479}},"tokens_in":534,"tokens_out":538,"duration_ms":5394,"temperature":1.0,"reasoning_tokens":479,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:59:33.741835+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same masked-token log-likelihood comparison on a corpus verifiably absent from the victim models' training data, or audit AltLex's overlap with each victim's pre-training corpus. If the original-over-similar gap persists for absent sentences, the effect is contextual inference rather than training-data leakage; if AltLex was never in the training data, the paper's central interpretation collapses.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies AltLex, the Wikipedia-derived dataset of sentences claimed to be part of the victim models' pre-training data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the MLC and MSP baseline attacks that GEIA is compared against in the reproducibility study."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides PersonaChat, the auxiliary dataset on which the attacker is trained from scratch."}],"review_version":1}