{"id":"d7006c8d-f25e-4128-bc13-bb9e7830a02a","arxiv_id":"2505.13819","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Fine-tuned LLMs leak private fragment-level information to adversaries holding only a few unordered public fragments, as shown by two probe attacks (LR-Attack and PRISM) on medical and legal summarization tasks.","lead":"An attacker who knows only a few public facts from someone's record, such as 'hypertension' from a medical note, can query a fine-tuned language model to infer additional private facts, such as 'osteoporosis'. This paper introduces the 'partial-information fragment inference' (PIFI) threat model and shows that Llama, Qwen, and Mistral models leak such fragment-level information at rates well above random.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Negative examples in §5.1 confound fragment co-occurrence with fragment-level membership; noise ablation (T5) shows LR-Attack is insensitive to S, so the 'fragment-specific' central claim is not established.","rationale":"The paper's central claim, as stated in the abstract, is that fine-tuned LLMs are susceptible to 'fragment-specific extraction attacks' under the PIFI threat model, where the adversary possesses only partial, unordered fragments S of an individual's record. For this claim to hold, the attack statistic must be driven by the joint occurrence of S and y* in the specific training record s, not merely by the marginal memorization of y* elsewhere in D. The evaluation in §5.1 does not enforce this: negative examples are sampled from notes outside the fine-tuning set, so negative y* are mostly absent from D. This creates a shortcut: LR-Attack can achieve high TPR by detecting whether y* was fine-tuned on at all. The paper's own noise ablation (T5, §8.5) shows that LR-Attack's AUC is unchanged when 75% of S is replaced with wrong fragments, which strongly suggests S is not being used. Table 3 reinforces this: LR-Attack's success is concentrated in rare fragments—precisely those whose presence in D is easiest to detect—while its TPR for common fragments is near chance. The reader's concern about PRISM's ad hoc proportionality assumptions is legitimate but affects only one of the three methods and does not bear on the more fundamental confound that undermines both data-blind attacks and the interpretation of the Classifier baseline. The paper has real strengths: the threat model is clearly formalized, experiments span multiple model families and fine-tuning schemes, code is released, and the synthetic experiment in Appendix B validates PRISM's closed form under known distributions. However, these do not compensate for the missing matched-negative control. A single re-run with y* drawn from other training records (and the noise ablation repeated on that set) would determine whether the reported vulnerability is genuinely fragment-specific or an artifact of membership detection. Until then, the central claim should be regarded as unverified rather than conditionally accepted on the PRISM assumptions.","tokens_in":25287,"tokens_out":11676,"duration_ms":101706,"concrete_test":"Re-run the medical summarization evaluation with a matched negative control: for each training record s used as a positive, sample candidate y* from fragments of other training records (ensuring y* ∈ D but y* ∉ s), while keeping S = Frag(s). Recompute LR-Attack and PRISM TPR@2%FPR, TPR@5%FPR, and ROC-AUC on this controlled test set. In addition, repeat the 75%-noise ablation of §A.2 on this controlled set. If LR-Attack's TPR@2%FPR drops to near the FPR (≤2-3%) or the noise ablation now changes its AUC, the reported attacks exploit marginal fragment membership rather than the co-occurrence of S and y*; the central 'fragment-specific' claim would then be unsupported. If performance is unchanged and LR-Attack remains insensitive to S, the paper still lacks evidence that S is used, and the conclusion should be revised accordingly.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"Section 5.1 constructs negative examples as 'a second disjoint set of unseen fragments, i.e., fragments from notes that did not appear in the fine-tuning subset and which are not part of the samples that we test as true positives.' This means negative y* are primarily out-of-distribution relative to D; for the rare fragments on which LR-Attack performs best (Table 3, 17.5% TPR@2%FPR), they are almost certainly absent from D. The attack can then succeed by thresholding the marginal memorization of y* in fθ,D vs fθ,D', with no need to use the conditioning set S. Section 8.5/T5 provides direct evidence: replacing 75% of S with inaccurate fragments leaves LR-Attack AUC unchanged (0.54 for the 1-epoch Llama-3-8B model; 0.64→0.66 for the convergence model), so the content of S is irrelevant for LR-Attack. This is incompatible with the paper's account that the attacker uses the known fragments S to infer co-occurring y* in the same record; instead, the statistic appears to detect whether y* was seen during fine-tuning. The proper control, required by the threat model in Algorithm 1 (which conditions on s ∈ D), is to draw negative y* from other training records so that y* ∈ D but y* ∉ s. Without this control, Tables 1-5 do not demonstrate fragment-specific inference. The PRISM derivation concerns raised by the reader are secondary; a correct PRISM score would still not rescue an evaluation whose negatives are confounded with membership.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PIFI (Partial-Information Fragment Inference), a threat model in which an attacker who knows a small unordered set S of fragments from an individual's training record queries a fine-tuned LLM's output probabilities to infer whether a candidate private fragment y* also occurs in that record. Two data-blind scoring functions are proposed: LR-Attack, which thresholds the ratio p_D/p_D' against a shadow model, and PRISM, which incorporates a world-model prior through a Bayesian update. The authors evaluate on medical and legal summarization tasks using Llama-3.1-8B, Qwen-2-7B, Mistral-7B, and Llama-3.2-3B, with full and LoRA fine-tuning, and also report a DP-SGD defense experiment. The central empirical claim is that fine-tuned LLMs leak under PIFI, with TPRs around 5-17% at 2-5% FPR across model families.","tokens_in":25609,"tokens_out":5337,"duration_ms":48494,"significance":"The PIFI threat model is a timely and meaningful relaxation of prior memorization and membership-inference assumptions: it asks whether an adversary with only a few unordered fragments can infer additional co-occurring fragments. The empirical effort is substantial, covering three model families, multiple fine-tuning regimes, LoRA, DP fine-tuning, world-model ablations, and a public code release. If the fragment-specific claim were supported by the evaluation, the results would constitute an important contribution to LLM privacy. However, as detailed in the major comments, the current evaluation does not establish the fragment-specific mechanism, and the PRISM derivation rests on unreported and unjustified choices. The work is potentially salvageable but requires a re-designed evaluation before the central claim can be accepted.","major_comments":[{"comment":"The negative examples used to compute FPR are fragments drawn from notes that 'did not appear in the fine-tuning subset,' so all negative y* are out-of-distribution with respect to D while all positive y* are in D. This allows LR-Attack to separate positives from negatives by detecting whether y* was memorized at all, without using the conditioning set S, which contradicts Algorithm 1's requirement to score y*∈s given s∈D. The rare-fragment stratum in Table 3, where LR-Attack reaches 17.5% TPR@2%FPR, is exactly the regime where this confound is strongest because rare y* are almost certainly absent from the shadow model's training data, making p_D/p_D' spike from membership alone. The proper control, required by the threat model, is to draw negative y* from other training records so that y*∈D but y*∉s.","section":"§5.1, Tables 1 and 3"},{"comment":"The noise ablation provides direct evidence that the content of S is not the signal driving the attacks: replacing 75% of the fragments in S with inaccurate fragments leaves LR-Attack AUC essentially unchanged (0.54 for the 1-epoch Llama-3-8B model and 0.64→0.66 for the convergence model), while PRISM changes only mildly (0.68→0.65 for 1 epoch, 0.67→0.66 for convergence). This is incompatible with the paper's account that the attacker uses the known fragments S to infer co-occurring y* in the same record; instead, the statistic appears to detect whether y* was seen during fine-tuning. This ablation should be reported prominently, and the central claim should be revisited in light of it.","section":"§8.5 and Appendix A.2"},{"comment":"The PRISM score is derived from two unjustified proportionality assumptions, Pr(ℓ̂|s∈D)∝ℓ̂ and Pr(ℓ̂|s∉D)∝1/ℓ̂, and the prior β=Pr(s∈D) is never reported for the LLM experiments. The only validation of this closed form is on a synthetic trigram model in Appendix B, which is a count-based model rather than an LLM and does not establish that the functional form is appropriate for the reported TPR/AUC values. The paper should report β, provide a sensitivity analysis over β and the assumed proportionalities, or explicitly reframe PRISM as a heuristic rather than a posterior estimate.","section":"§4.3 (PRISM)"}],"minor_comments":[{"comment":"All results are reported as point estimates without error bars, confidence intervals, or a statement about the number of fine-tuning runs and shadow-model draws; given the stochasticity of fine-tuning, the absence of any variance estimate weakens the quantitative comparisons.","section":"Tables 1, 3, 4, 5"},{"comment":"The notation Pr(fθ,D*(y*|S)|s∈D*) is not a well-defined probability as written; it appears to denote the distribution of the model-assigned probability under random draws of D*, and this should be clarified.","section":"§4.3"},{"comment":"The memorization results reference 'Table??' instead of a numbered table; the cross-reference should be fixed.","section":"Appendix D"},{"comment":"The entity category examples contain apparent NER labeling errors: 'Abnormal Heart Valve' is listed under B-Lab_value and 'Darvocet' under B-Biological_structure; these should be verified because the fragment categories are used in the analysis.","section":"Appendix C, Table 8"},{"comment":"The citation to Jiang et al. [2024] for the claim that LoRA mitigates memorization appears to point to a paper on high-rank updating for parameter-efficient fine-tuning; please verify that this citation supports the stated claim.","section":"§8.3"}],"recommendation":"major_revision","confidential_remarks":"The evaluation confound in §5.1 is the central issue: until the authors run the proper control where negative y* are drawn from other training records, the headline 'fragment-specific extraction' claim is not supported, and the noise ablation in §8.5 suggests the result may not survive that control. I would ask for that experiment before further consideration. The PRISM derivation issue is secondary but should be fixed by reporting β or demoting PRISM to a heuristic."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nRead the Rosenblatt et al. paper. The PIFI threat model is a genuinely useful reformulation—relaxing the full-sample and ordered-prefix assumptions of membership inference and memorization to unordered public fragment sets is a real gap in the literature, and the paper works through it carefully. The experimental footprint is substantial too: three model families, full fine-tuning and LoRA, multiple epochs, a DP ablation, and a second legal domain. If the empirical claims held up, this would be a solid contribution.\n\nBut the central empirical claim doesn't hold up as stated, and the problem is load-bearing rather than cosmetic. The negatives in §5.1 are fragments from notes that never appeared in the fine-tuning set—so the attack is distinguishing fragments that are in D from fragments that are entirely out-of-distribution. That's membership inference on the fragment y*, not inference about co-occurrence with S within s. The paper's own noise ablation (§8.5) is the smoking gun: replacing 75% of S with wrong fragments leaves LR-Attack AUC essentially unchanged (0.54 in the 1-epoch setting; 0.64→0.66 in the convergence setting). If S mattered, corrupting it should degrade performance. The paper calls this \"robustness,\" but the more parsimonious reading is that LR-Attack is ignoring S entirely and just measuring whether y* was memorized during fine-tuning. The correct control—negative y* drawn from other training records so that y* ∈ D but y* ∉ s—is absent from every table in the paper.\n\nThe PRISM concerns are real but secondary. The derivation in §4.3 is openly heuristic: \"we pragmatically assume\" proportionality, the prior β is never reported for the LLM experiments, and the only validation is a synthetic trigram model. That's a weakness, not a fatal flaw, but it makes the PRISM numbers hard to interpret. Missing error bars and the data-aware Classifier outperforming the data-blind methods in most settings are also worth flagging, though the \"competitive\" claim is roughly accurate.\n\nWhere does that leave us? The threat model is worth taking seriously, and the paper deserves referee time—the flaw is addressable and the authors seem capable of fixing it. But as written, the abstract's claim about \"fragment-specific extraction attacks\" is not supported by the evidence. I'd send this to peer review with a strong request to redo the negative construction and re-report the noise ablation as a control rather than a robustness result.","headline":"A genuinely new threat model, but the evaluation confounds fragment co-occurrence with fragment membership—LR-Attack appears to ignore the conditioning set entirely.","tokens_in":26179,"tokens_out":5071,"would_cite":false,"duration_ms":45064,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuned LLMs leak private fragments to attackers who know only a few facts","keywords":["partial-information fragment inference","PIFI","LLM privacy","membership inference","memorization","likelihood ratio attack","PRISM","fine-tuning leakage"],"falsifier":"On a fine-tuned LLM with known training data, compute the empirical conditional distributions of $\\hat{\\ell} = p_D/p_{D'}$ for fragments that are in the target individual's record versus fragments that are not; if the ratio of these densities is not approximately $\\hat{\\ell}$ versus $1/\\hat{\\ell}$, or if PRISM's AUC collapses when $\\beta$ is swept across plausible values, then the closed-form posterior is not a valid estimate of $\\Pr(s \\in D \\mid \\hat{\\ell})$.","tokens_in":25018,"feed_emoji":"🩺","tokens_out":5015,"duration_ms":41318,"temperature":0.7,"pith_summary":"This paper tries to show that a weak adversary — one who knows only a handful of unordered text fragments about a person, such as a few medical conditions — can query a fine-tuned LLM and infer additional private fragments from that same record. It introduces the partial-information fragment inference (PIFI) threat model and two data-blind attacks: LR-Attack, a likelihood ratio between target and shadow model probabilities, and PRISM, which adds a world-model prior to curb false positives. On medical summarization models, the attacks reach true-positive rates around 5–17% at 2–5% false positives, approaching a baseline classifier that assumes labeled in-distribution data. The paper argues that these rates justify treating fragment-level leakage as a distinct privacy risk in deployed fine-tuned models, especially after repeated epochs of fine-tuning.","feed_headline":"Attackers infer hidden health facts from fragmentary LLM prompts","feed_subtitle":"Two probability-ratio attacks on fine-tuned medical models reach up to 17.5% true-positive rate at 2% false positives.","key_machinery":"The statistical engine is the likelihood ratio $\\hat{\\ell} = p_D / p_{D'}$, where $p_D = f_{\\theta,D}(y^* \\mid \\mathrm{Prompt}(S))$ is the target model's probability for the candidate fragment given the known fragments, and $p_{D'}$ is the same probability under a shadow model trained without the individual's record. PRISM regularizes this ratio with a world-model probability $p_{\\mathrm{world}}$, computed as the mean probability over models not fine-tuned on the data, and applies a closed-form Bayesian update under the pragmatic assumptions $\\Pr(\\hat{\\ell} \\mid s \\in D) \\propto \\hat{\\ell}$ and $\\Pr(\\hat{\\ell} \\mid s \\notin D) \\propto 1/\\hat{\\ell}$, with a prior $\\beta = \\Pr(s \\in D)$. The Classifier baseline, a LightGBM trained on the same three probabilities with labeled data, serves as the data-aware upper reference.","core_discovery":"The central claim is that fine-tuned language models leak sensitive fragments under PIFI: given a public fragment set S from an individual's record, an attacker who can query the model's output probabilities can decide whether a candidate private fragment y* also appears in the record, conditioned on the record being in the training set. The evidence is empirical: across Llama-3.1-8B, Qwen-2-7B, and Mistral-7B fine-tuned on medical notes, LR-Attack and PRISM exceed random guessing by roughly two to four times at low FPR, with LR-Attack reaching 17.5% TPR at 2% FPR for fragments that appear once in the training data. The authors also show that repeated fine-tuning epochs increase leakage, LoRA reduces but does not eliminate it, larger models leak more, and PRISM outperforms LR-Attack on common fragments while LR-Attack excels on rare ones.","pith_inferences":["The reported per-fragment TPRs, applied across many victims, imply that an attacker querying thousands of records could recover a substantial number of sensitive facts; multiplying the low-FPR TPR by population size gives concrete expected leak counts.","Because the PIFI attack needs only output probabilities, defenses that hide logits (e.g., API restrictions) would blunt LR-Attack and PRISM, a testable prediction not explored in the paper.","The same fragment-inference machinery could be turned toward constructive ends, such as auditing what a fine-tuned model has absorbed about a cohort, by treating the attack score as a per-fragment memorization diagnostic.","A direct calibration check — computing $\\Pr(\\hat{\\ell} \\mid s \\in D)$ and $\\Pr(\\hat{\\ell} \\mid s \\notin D)$ on an open model with known training data — would tell whether PRISM's closed-form posterior is reliable; the paper only validates it on a synthetic trigram model."],"forward_implications":["More fine-tuning epochs raise fragment-leakage rates, so deployment of repeatedly fine-tuned models carries higher PIFI risk than single-epoch models.","LR-Attack offers a practical scare for rare fragments: a single-occurrence term yields 17.5% TPR at 2% FPR, meaning roughly one in six rare facts is recoverable at a tolerable false-positive budget.","PRISM's world-model prior suppresses false positives on common fragments, so the choice of attack should depend on fragment frequency.","Differential privacy with $\\epsilon=3$ suppresses LR-Attack to 0.9% TPR at 2% FPR but leaves PRISM and Classifier at roughly 4% TPR, so DP alone does not close the gap.","LoRA fine-tuning reduces but does not eliminate vulnerability, so parameter-efficient deployment should not be assumed private."],"supporting_citations":[{"why":"Supplies the likelihood-ratio membership inference principle and shadow-model methodology that LR-Attack adapts to fragment-level inference.","marker":"[Carlini et al., 2022a]"},{"why":"Provides the Neyman-Pearson lemma justifying thresholding likelihood ratios for optimal hypothesis testing at a fixed false-positive rate.","marker":"[Neyman and Pearson, 1933]"},{"why":"Establishes k-eidetic memorization in LLMs and the repeated-exposure effect that motivates the vulnerability and the more-epochs finding.","marker":"[Carlini et al., 2021]"},{"why":"Supplies the MTS-Dialog medical dataset used for the main experiments and the clinical summarization task.","marker":"[Abacha et al., 2023]"},{"why":"Introduces LoRA, the parameter-efficient fine-tuning method the paper tests as a potential mitigation.","marker":"[Hu et al., 2022]"},{"why":"Provides the differential-privacy fine-tuning approach and PII-leakage results that the authors compare against in their defense experiments.","marker":"[Lukas et al., 2023]"},{"why":"LightGBM is the model behind the data-aware Classifier baseline that the attacks are compared with.","marker":"[Ke et al., 2017]"},{"why":"Defines extractable memorization with ordered prefixes, the threat model that PIFI generalizes by relaxing to partial unordered fragments.","marker":"[Carlini et al., 2022b]"}],"fun_headline_variants":["Partial prompts leak hidden patient facts from fine-tuned LLMs","Sparse data snippets enable LLM leaks of sensitive medical info","Even partial patient info lets attackers pry facts from LLMs","Fine-tuned models leak secrets from just a few known fragments","Fragment clues let attackers infer private health details from LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"PRISM's score is a meaningful posterior only if the ad hoc assumptions $\\Pr(\\hat{\\ell} \\mid s \\in D) \\propto \\hat{\\ell}$ and $\\Pr(\\hat{\\ell} \\mid s \\notin D) \\propto 1/\\hat{\\ell}$ hold and the prior $\\beta$ is set sensibly; these are validated only on a synthetic trigram model, not on any LLM.","fun_headline_variants_meta":{"raw":{"variants":["Partial prompts leak hidden patient facts from fine-tuned LLMs","Sparse data snippets enable LLM leaks of sensitive medical info","Even partial patient info lets attackers pry facts from LLMs","Fine-tuned models leak secrets from just a few known fragments","Fragment clues let attackers infer private health details from LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000173,"raw_usage":{"total_tokens":1275,"prompt_tokens":937,"completion_tokens":338,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":553,"completion_tokens_details":{"reasoning_tokens":254}},"tokens_in":553,"tokens_out":338,"duration_ms":3495,"temperature":1.0,"reasoning_tokens":254,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:09:57.971406+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a fine-tuned LLM with known training data, compute the empirical conditional distributions of $\\hat{\\ell} = p_D/p_{D'}$ for fragments that are in the target individual's record versus fragments that are not; if the ratio of these densities is not approximately $\\hat{\\ell}$ versus $1/\\hat{\\ell}$, or if PRISM's AUC collapses when $\\beta$ is swept across plausible values, then the closed-form posterior is not a valid estimate of $\\Pr(s \\in D \\mid \\hat{\\ell})$.","supporting_citations":[],"review_version":1}