Pith. sign in

REVIEW 5 major objections 3 minor 37 references

Automatic Calibration for Membership Inference Attack on Large Language Models

T0 review · 5 major / 3 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A tunable temperature on a model's own output probabilities can reveal which texts were in its training data, with no reference model.

desk verdict Useful temperature calibration for MIA, but DerivAC has a sign contradiction that makes its reported AUROC unreproducible; fix that and it earns a serious review. read the letter →

arxiv 2505.03392 v1 pith:VYWJ7TFN submitted 2025-05-06 cs.LG cs.AI

Saleh Zare Zade , Yao Qiang , Xiangyu Zhou , Hui Zhu , Mohammad Amin Roshani , Prashant Khanduri , Dongxiao Zhu This is my paper · ORCID
classification cs.LGcs.AI
keywords membershipinferenceattackslargelanguagemodelstemperaturescalingpretrainingdatadetectionprivacyauditingcalibrationMIMIRbenchmarkWikiMIA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that membership in a large language model's (LLM's) pretraining data can be detected by rescaling the model's own token probabilities with a tunable temperature, rather than by comparing against an external reference model. It presents three scoring variants — AC, DerivAC, and NormAC — that either compare temperature-scaled log probabilities against the raw ones, measure how the scaled probability changes as temperature moves, or normalize each token's scaled score by its mean and variance. On the WikiMIA, MIMIR, and PatentMIA benchmarks, across several open-weight models, the paper reports that all three variants outperform existing score-based, calibration-based, and reference-model baselines. If the claim is right, privacy auditors and contamination checkers gain a tool that needs only the target model's outputs.

What carries the argument

The load-bearing object is the temperature-scaled token probability $\mathrm{TSP}(z|x_{<t},\tau) = \frac{\exp(\log p(z|x_{<t})/\tau)}{\sum_{i=1}^{K} \exp(\log p(V_i|x_{<t})/\tau)}$, which reshapes the model's categorical distribution without retraining. ACMIA's three scorers use this object in different ways: AC takes a sign-weighted average of the difference between the temperature-adjusted log probability and the original log probability; DerivAC measures the finite-difference derivative of the scaled log probability with respect to temperature; NormAC z-normalizes each token's scaled log probability under the scaled distribution itself, using the mean and variance defined in equations (8) and (9). Tuning $\tau$ changes the balance between first-order and second-order terms in the maximum-likelihood score of equation (3), which the paper argues amplifies the probability gap between training and non-training texts even when both come from the same distribution.

What would settle it

Run ACMIA on a fresh model with the temperature fixed by a label-free rule, or with one temperature held constant across all datasets; if the AUROC drops to the level of raw loss or random guessing, then the reported gain is an artifact of tuning on labeled members and non-members rather than a property of the calibration mechanism itself.

Watch

Extended reading notes

Core claim

The paper's central claim is that a single tunable temperature applied to the target LLM's own output probabilities is enough to convert raw likelihood scores into calibrated membership signals: reshaping the token distribution with $\tau$ widens the separation between member and non-member texts, and the three ACMIA variants that exploit this reshaping consistently rank above seven baselines, as measured by area under the ROC curve, on WikiMIA, MIMIR, and PatentMIA across Baichuan, Qwen1.5, OPT, Pythia, and GPT-NeoX. The authors derive the temperature adjustment from a maximum-likelihood and implicit-score-matching view of pretraining, in which training samples sit near local maxima of the likelihood landscape, and they show that ACMIA also reduces false positives at strict operating points, including the true-positive rate at a fixed 5% false-positive rate and the false-positive rate at a fixed 95% true-positive rate. They further state that AC and DerivAC need only the sample loss, so they remain applicable when only API-level access is available.

Load-bearing premise

The reported scores take the temperature as tuned on a labeled set of known members and non-members for each dataset and model, so the attack's performance without any labeled examples is not established.

Editorial extensions

If this is right

  • If ACMIA works as claimed, pre-training-data auditing no longer requires a held-out reference model or a reference corpus, since the target model's own temperature-adjusted outputs serve as the reference.
  • Because AC and DerivAC can be computed from sample-level loss alone, the attack carries over to API-only access where token log-probabilities are hidden.
  • The reported consistency across original and paraphrased WikiMIA texts implies the calibrated scores track memorization structure rather than surface lexical overlap.
  • The reported correlation between model size and detection accuracy means larger models are more exposed to this style of attack, which the paper frames as an urgent privacy concern.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Since the optimal temperature values are reported to be stable across models trained on similar data, one could test whether a temperature tuned on one model transfers to another model trained on the same corpus; the paper does not run this transfer experiment.
  • A defense direction the paper leaves implicit: if temperature scaling is what exposes membership, then smoothing logits at inference time, for instance by raising the model's own temperature, might shrink the gap the attack exploits.
  • The label-free variant is only sketched, so an obvious extension is to replace the labeled tuning set with a heuristic such as perplexity-based or frequency-based temperature selection and measure how much detection accuracy is lost.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 3 minor

Summary. The paper introduces ACMIA, a family of membership-inference attacks against LLMs that applies temperature scaling to token log-probabilities and then calibrates the resulting scores to separate pretraining members from non-members. Three variants are proposed: AC (difference between temperature-scaled and original log probabilities with a sign depending on whether τ>1), DerivAC (a finite-difference in temperature), and NormAC (a z-score over the temperature-scaled distribution). The method is evaluated on WikiMIA, MIMIR, and PatentMIA with several open-source LLMs, reporting AUROC, TPR@5%FPR, and FPR@95%TPR, and the authors claim consistent gains over score-based, calibration-based, and reference-model baselines.

Significance. If the results hold, ACMIA would be a useful practical MIA: it avoids external reference models, has variants for different levels of API access, and is evaluated on three benchmarks with a large model sweep. The paper ships code and provides detailed per-subset tables in the appendix; the temperature robustness ablation (Figure 3) is a useful check. However, the current version has internal inconsistencies in the definition of DerivAC, a theoretical motivation that does not support the stated claims, and headline results that depend on labeled temperature tuning; these issues must be resolved before the empirical claims can be relied upon.

major comments (5)
  1. [Section 4.1, Eq. (2) and Appendix B.1] The statement that 'the maximum likelihood estimation used in the LLM pre-training process can be formulated as' Eq. (2) is not correct. Eq. (2) is the implicit score matching objective of Hyvärinen and Dayan, which is a surrogate for estimating unnormalized continuous densities, not an equivalent form of MLE for autoregressive language models. Moreover, the input x is a discrete token sequence, so derivatives of log p(x) with respect to x are not defined in the usual sense. In addition, under the paper's own definition ψ′(x)=τ·ψ(x,τ), the explicit temperature coefficients in Eq. (3) cancel: the first term becomes (1/2)||ψ(x,τ)||² and the second becomes ∑∂ψ_i(x,τ)/∂x_i. The claimed role of τ as a balancing coefficient between first- and second-order terms is therefore not supported. The theoretical motivation should be corrected or explicitly reframed as a heuristic.
  2. [Section 4.2 Eq. (6), Appendix B.2, Appendix C] The definition of DerivAC is internally contradictory. Eq. (6) defines DerivAC as the average of log TSP(τ+δ) − log TSP(τ) over first-occurrence tokens, and the text states that a higher derivative indicates membership. Appendix B.2 derives ∂ log p(x,τ)/∂τ = (µ_z − z)/τ², where µ_z is the softmax-weighted mean logit. For a token with above-average logit (z > µ_z), which is exactly a token the model considers likely, this derivative is negative, so the Eq. (6) score ranks likely tokens lower and surprising tokens higher, the opposite of the intended ranking. The sample-level version in Appendix C instead defines DerivAC(x,τ;M)=L(x;M_{τ+δ})−L(x;M_τ), i.e., the negative of the log-probability difference, which has the opposite sign. These two definitions cannot both describe the evaluated attack, and the reported AUROC for DerivAC (e.g., 81.4 on PatentMIA/Qwen1.5-32B in Table 1) is not reproducible from Eq. (6) without an additional sign flip.
  3. [Section 5.5, Table 4] The headline results are obtained with the optimal temperature selected on a set of labeled member and non-member samples for each benchmark and model. This makes the 'automatic' and practical claims conditional on access to membership labels that an attacker would not normally have. The paper should report AUROC with a fixed default temperature or with a temperature selected without membership labels, and should state clearly in the main text that Tables 1–3 use labeled tuning. Figure 3 shows robustness on WikiMIA and PatentMIA for two models, but it does not cover the full benchmark/model grid; without unlabeled-tuning results, the practical relevance of the reported numbers is not established.
  4. [Section 6.1 vs. Appendix E] The claim that the ACMIA variants 'consistently outperform baselines ... across all benchmarks and models' is too strong. In the MIMIR appendix tables, several cells show a baseline equal to or better than ACMIA. For example, in Table 5 (≤20% 7-gram overlap), GitHub 12B has DC-PDD 92.5 equal to AC and DerivAC, and Wikipedia 160M has DC-PDD 63.3 vs. AC 62.9; in Table 7 (≤80% 13-gram overlap), Wikipedia 160M has Ref 50.8 vs. AC 49.9 and DerivAC 50.2. The authors should soften the claim and discuss the settings in which calibrated baselines remain competitive.
  5. [Section 4.2 Eq. (4), Appendix C] The claim that AC and DerivAC 'do not require access to the full log-likelihood distribution' is inconsistent with Eq. (4), whose normalization constant requires the full vocabulary distribution over all V_i. The loss-only versions in Appendix C remove the FOS restriction and are not identical to Eqs. (5)–(6). The paper should clarify exactly which variant is evaluated in each table and specify the precise access requirements, including whether the model API must return full log-probabilities or only losses under modified temperatures.
minor comments (3)
  1. [Tables 1–10] AUROC values are reported to one decimal place without confidence intervals or significance tests; many ACMIA-vs-baseline differences are 0.2–0.5 points, so the claimed systematic gains should be supported with repeated runs or variance estimates.
  2. [Table 4] The optimal-temperature table covers only PatentMIA; to support the claim that optimal τ is determined by the dataset rather than the model, the same analysis should be shown for WikiMIA and MIMIR.
  3. [Figure 2 and Eq. (13)] The normalization used in Figure 2 is not described, and Appendix A's Min-K%++ score in Eq. (13) does not define how the 'min-k%' subset is selected; these details should be added for reproducibility.

Circularity Check

2 steps flagged · score 3.0 of 10

No load-bearing circularity; one self-definitional reparameterization in the theory and an internal sign contradiction in DerivAC, with disclosed label-tuned temperature, but the central AUROC comparison is not forced.

  1. self definitional [Section 4.2, Eq. (3) and definition of ψ′(x)=τ·∂logp(x,τ)/∂x; Appendix B.1]
    "Our method dynamically adjusts both first-order and second-order derivatives through temperature scaling, refining the maximum likelihood estimation as: [1/N Σ_x (1/(2τ²)||ψ′(x)||² + (1/τ)Σ_i ∂ψ′_i(x)/∂x_i)], where ψ′(x) = τ·∂ logp(x,τ)/∂x denotes the scaled gradient ... Here, the temperature τ plays a crucial role in balancing the influence of first-order and second-order derivatives with respect to the inputs."

    Substituting ψ′=τψ into Eq. (3) gives (1/(2τ²))||τψ||² + (1/τ)Σ∂(τψ_i)/∂x_i = (1/2)||ψ||² + Σ∂ψ_i/∂x_i, which is exactly Eq. (2). The τ coefficients cancel, so the claimed 'balancing' of first- and second-order terms is an algebraic identity. As written, the 'refined' MLE objective is the same functional as the original under the stated definition; no temperature-dependent effect follows from Eq. (3) itself.

  2. other [Section 4.2 Eq. (6); Appendix B.2; Appendix C]
    "Specifically, a higher derivative indicates that the sample is more likely to belong to the pre-training data. ... DerivAC(x,τ) = 1/|FOS(x)| Σ_{xt∈FOS(x)} (log TSP(xt|x<t,τ+δ) − log TSP(xt|x<t,τ)) ... DerivAC(x,τ;M) = L(x;M_{τ+δ}) − L(x;M_τ)."

    Appendix B.2 derives ∂logp(x,τ)/∂τ = (µz−z)/τ². For a high-probability token z>µz this derivative is negative, so Eq. (6) assigns lower scores to exactly the tokens that are most typical of members, contradicting the claim that a higher derivative indicates training data. Appendix C defines DerivAC as the loss difference L(M_{τ+δ})−L(M_τ), which is the negative of the Eq. (6) log-probability difference. The reported high AUROC for DerivAC is therefore not generated by Eq. (6) unless an unstated sign flip is applied; the evaluated score is, by construction, the opposite of the formula in Eq. (6). This breaks the derivation chain for a central variant, although it is an internal inconsistency rather than a reduction of the result to its fitted inputs.

full rationale

The core ACMIA scoring functions are concrete transformations of the target model's token probabilities (temperature-scaled softmax, log-ratio, finite difference, or mean/variance normalization), and the headline AUROC numbers are measured against external benchmarks and standard baselines. The membership labels enter only through the disclosed per-dataset/model selection of the temperature in Section 5.5 ('we find the optimal temperature using a set of labeled member and non-member samples'); this limits the 'automatic' and 'practical' claims and should have been reported as a labeled development-set assumption, but it is not a circular reduction because the score ranking itself is not fitted to the labels. The two flagged steps are not load-bearing circularity: Eq. (3) is a tautological reparameterization of Eq. (2) used only as motivation, and the DerivAC sign inconsistency in Eq. (6), Appendix B.2, and Appendix C is a reproducibility/correctness defect in a central variant, not an equivalence of the result to its inputs. On balance, the paper's central empirical claim has independent content, so the circularity score is 3 rather than 0-2.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central empirical results rest on one tuned hyperparameter (temperature) and several domain assumptions about LLM training dynamics and benchmark validity. The theoretical motivation in Section 4.1 is not a derivation of the attack; it is a heuristic analogy to score matching.

free parameters (1)
  • temperature tau (log2 tau) = roughly 1.1 to 1.4 on PatentMIA across models (log2-tau), see Table 4
    Chosen per dataset/model by maximizing AUROC on labeled member/non-member validation samples (Section 5.5); reported results are conditional on this tuning.
assumptions (4)
  • ad hoc to paper LLM maximum likelihood pre-training is approximated by the implicit score matching objective in Eq (2), including smoothness of log p(x) around training samples.
    Section 4.1 asserts this to motivate temperature scaling; standard autoregressive LLMs use next-token cross-entropy, and the derivative with respect to the discrete text x is not defined in the paper.
  • domain assumption Temperature scaling with tau > 1 simulates underfitting and tau < 1 simulates overfitting, so the gap between the target and temperature-adjusted scores indicates membership.
    Section 4.2 heuristic; no proof that the simulated model family spans the relevant training dynamics.
  • domain assumption Only first-occurrence tokens should be scored because later repetitions are easier to predict.
    Section 4.2, Eq (5); borrowed from DC-PDD [37], with no evidence given for this specific benchmark set.
  • domain assumption The non-member texts in WikiMIA, MIMIR, and PatentMIA are genuinely absent from pre-training.
    The benchmarks assume temporal or structural separation; if leakage exists, AUROC values are inflated. This is a standard assumption for these datasets.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Automatic Calibration for Membership Inference Attack on Large Language Models." pith.science (2026). https://pith.science/paper/VYWJ7TFN

@misc{pith2026250503392,
  author       = {Pith},
  title        = {Pith review of: Automatic Calibration for Membership Inference Attack on Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VYWJ7TFN}},
  note         = {Machine review of arXiv:2505.03392}
}
read the original abstract

Membership Inference Attacks (MIAs) have recently been employed to determine whether a specific text was part of the pre-training data of Large Language Models (LLMs). However, existing methods often misinfer non-members as members, leading to a high false positive rate, or depend on additional reference models for probability calibration, which limits their practicality. To overcome these challenges, we introduce a novel framework called Automatic Calibration Membership Inference Attack (ACMIA), which utilizes a tunable temperature to calibrate output probabilities effectively. This approach is inspired by our theoretical insights into maximum likelihood estimation during the pre-training of LLMs. We introduce ACMIA in three configurations designed to accommodate different levels of model access and increase the probability gap between members and non-members, improving the reliability and robustness of membership inference. Extensive experiments on various open-source LLMs demonstrate that our proposed attack is highly effective, robust, and generalizable, surpassing state-of-the-art baselines across three widely used benchmarks. Our code is available at: \href{https://github.com/Salehzz/ACMIA}{\textcolor{blue}{Github}}.

Figures

Figures reproduced from arXiv: 2505.03392 by the authors.

Figure 1
Figure 1. Conversely, when non-members are far from a local maxi [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 12 canonical work pages

  1. [1]

    Achiam, S

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 tech- nical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Biderman, U

    S. Biderman, U. Prashanth, L. Sutawika, H. Schoelkopf, Q. Anthony, S. Purohit, and E. Raff. Emergent and predictable memorization in large language models. Advances in Neural Information Processing Systems, 36, 2024

    work page 2024

  3. [3]

    Black, S

    S. Black, S. Biderman, E. Hallahan, Q. Anthony, L. Gao, L. Gold- ing, H. He, C. Leahy, K. McDonell, J. Phang, et al. Gpt-neox- 20b: An open-source autoregressive language model. arXiv preprint arXiv:2204.06745, 2022

    arXiv 2022

  4. [4]

    Brown, B

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language mod- els are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  5. [5]

    Carlini, F

    N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-V oss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633–2650, 2021

    work page 2021

  6. [6]

    Carlini, S

    N. Carlini, S. Chien, M. Nasr, S. Song, A. Terzis, and F. Tramer. Mem- bership inference attacks from first principles. In2022 IEEE Symposium on Security and Privacy (SP) , pages 1897–1914. IEEE, 2022

    work page 1914

  7. [7]

    Carlini, D

    N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramer, and C. Zhang. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646, 2022

    arXiv 2022

  8. [8]

    M. Duan, A. Suri, N. Mireshghallah, S. Min, W. Shi, L. Zettlemoyer, Y . Tsvetkov, Y . Choi, D. Evans, and H. Hajishirzi. Do membership inference attacks work on large language models? arXiv preprint arXiv:2402.07841, 2024

    arXiv 2024

Show all 37 references
  1. [9]

    A. V . Duarte, X. Zhao, A. L. Oliveira, and L. Li. De-cop: Detecting copyrighted content in language models training data. arXiv preprint arXiv:2402.09910, 2024

    2024 arXiv

  2. [10]

    W. Fu, H. Wang, C. Gao, G. Liu, Y . Li, and T. Jiang. Practical mem- bership inference attacks against fine-tuned large language models via self-prompt calibration. arXiv preprint arXiv:2311.06062, 2023

    2023 arXiv

  3. [11]

    L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020

    2020 arXiv

  4. [12]

    M. M. Grynbaum and R. Mac. The times sues openai and microsoft over ai use of copyrighted work. The New York Times, 27, 2023

    2023

  5. [13]

    Hyvärinen and P

    A. Hyvärinen and P. Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research , 6 (4), 2005

    2005

  6. [14]

    S. Liu, Y . Yao, J. Jia, S. Casper, N. Baracaldo, P. Hase, X. Xu, Y . Yao, H. Li, K. R. Varshney, et al. Rethinking machine unlearning for large language models. arXiv preprint arXiv:2402.08787, 2024

    2024 arXiv

  7. [15]

    Z. Liu, T. Zhu, C. Tan, H. Lu, B. Liu, and W. Chen. Prob- ing language models for pre-training data detection. arXiv preprint arXiv:2406.01333, 2024

    2024 arXiv

  8. [16]

    Mattern, F

    J. Mattern, F. Mireshghallah, Z. Jin, B. Schölkopf, M. Sachan, and T. Berg-Kirkpatrick. Membership inference attacks against language models via neighbourhood comparison. arXiv preprint arXiv:2305.18462, 2023

    2023 arXiv

  9. [17]

    Meeus, S

    M. Meeus, S. Jain, M. Rei, and Y .-A. de Montjoye. Did the neurons read your book? document-level membership inference for large language models. arXiv preprint arXiv:2310.15007, 2023

    2023 arXiv

  10. [18]

    Mireshghallah, K

    F. Mireshghallah, K. Goyal, A. Uniyal, T. Berg-Kirkpatrick, and R. Shokri. Quantifying privacy risks of masked language models us- ing membership inference attacks. arXiv preprint arXiv:2203.03929 , 2022

    2022 arXiv

  11. [19]

    M. Nasr, N. Carlini, J. Hayase, M. Jagielski, A. F. Cooper, D. Ippolito, C. A. Choquette-Choo, E. Wallace, F. Tramèr, and K. Lee. Scalable extraction of training data from (production) language models. arXiv preprint arXiv:2311.17035, 2023

    2023 arXiv

  12. [20]

    Y . Oren, N. Meister, N. Chatterji, F. Ladhak, and T. B. Hashimoto. Prov- ing test set contamination in black box language models.arXiv preprint arXiv:2310.17623, 2023

    2023 arXiv

  13. [21]

    Qiang, X

    Y . Qiang, X. Zhou, and D. Zhu. Hijacking large language models via adversarial in-context learning. arXiv preprint arXiv:2311.09948, 2023

    2023 arXiv

  14. [22]

    Qiang, X

    Y . Qiang, X. Zhou, S. Z. Zade, M. A. Roshani, P. Khanduri, D. Zytko, and D. Zhu. Learning to poison large language models during instruc- tion tuning. arXiv preprint arXiv:2402.13459, 2024

    2024 arXiv

  15. [23]

    M. A. Roshani, X. Zhou, Y . Qiang, S. Suresh, S. Hicks, U. Sethuraman, and D. Zhu. Generative large language model—powered conversational ai app for personalized risk assessment: Case study in covid-19. JMIR AI, 4(1):e67363, 2025

    2025

  16. [24]

    W. Shi, A. Ajith, M. Xia, Y . Huang, D. Liu, T. Blevins, D. Chen, and L. Zettlemoyer. Detecting pretraining data from large language models. arXiv preprint arXiv:2310.16789, 2023

    2023 arXiv

  17. [25]

    Shokri, M

    R. Shokri, M. Stronati, C. Song, and V . Shmatikov. Membership infer- ence attacks against machine learning models. In2017 IEEE symposium on security and privacy (SP) , pages 3–18. IEEE, 2017

    2017

  18. [26]

    Steinke, M

    T. Steinke, M. Nasr, and M. Jagielski. Privacy auditing with one (1) training run. Advances in Neural Information Processing Systems , 36, 2024

    2024

  19. [27]

    Q. Team. Introducing qwen1.5, February 2024. URL https://qwenlm. github.io/blog/qwen1.5/

    2024

  20. [28]

    Touvron, L

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  21. [29]

    T. Wang, A. Roberts, D. Hesslow, T. Le Scao, H. W. Chung, I. Beltagy, J. Launay, and C. Raffel. What language model architecture and pre- training objective works best for zero-shot generalization? In Interna- tional Conference on Machine Learning , pages 22964–22984. PMLR, 2022

    2022

  22. [30]

    Watson, C

    L. Watson, C. Guo, G. Cormode, and A. Sablayrolles. On the impor- tance of difficulty calibration in membership inference attacks. arXiv preprint arXiv:2111.08440, 2021

    2021 arXiv

  23. [31]

    X. Wu, R. Duan, and J. Ni. Unveiling security, privacy, and ethical concerns of chatgpt. Journal of Information and Intelligence, 2(2):102– 115, 2024

    2024

  24. [32]

    A. Yang, B. Xiao, B. Wang, B. Zhang, C. Bian, C. Yin, C. Lv, D. Pan, D. Wang, D. Yan, et al. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305, 2023

    2023 arXiv

  25. [33]

    J. Ye, A. Maddi, S. K. Murakonda, V . Bindschaedler, and R. Shokri. En- hanced membership inference attacks against machine learning models. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, pages 3093–3106, 2022

    2022

  26. [34]

    S. Yeom, I. Giacomelli, M. Fredrikson, and S. Jha. Privacy risk in ma- chine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF) , pages 268–282. IEEE, 2018

    2018

  27. [35]

    Zhang, J

    J. Zhang, J. Sun, E. Yeats, Y . Ouyang, M. Kuo, J. Zhang, H. Yang, and H. Li. Min-k%++: Improved baseline for detecting pre-training data from large language models. arXiv preprint arXiv:2404.02936, 2024

    2024 arXiv

  28. [36]

    Zhang, S

    S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V . Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022

    2022 arXiv

  29. [37]

    Zhang, R

    W. Zhang, R. Zhang, J. Guo, M. de Rijke, Y . Fan, and X. Cheng. Pre- training data detection for large language models: A divergence-based calibration method. arXiv preprint arXiv:2409.14781, 2024. A Baselines Details We consider seven MIAs from the three categories as the bas...

    2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.