Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

How Contaminated Is Your Benchmark? Quantifying Dataset Leakage in Large Language Models with Kernel Divergence

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that dataset contamination in LLMs can be scored by the divergence between kernel similarity matrices of sample embeddings before and after fine-tuning, and that the score tracks the true contamination fraction almost…

desk verdict A genuinely new, parameter-free contamination score with strong controlled results, but the one 'clean' IID validation simply assumes Pile train is seen by Mistral, leaving the central mechanism less verified than the paper claims. read the letter →

arxiv 2502.00678 v2 pith:UEPS22F7 submitted 2025-02-02 cs.LG cs.AIcs.CL

Hyeong Kyu Choi , Maxim Khanov , Hongxin Wei , Yixuan Li This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords datasetcontaminationbenchmarkleakagekerneldivergencescoremembershipinferencefine-tuningembeddingsimilarityLLMevaluationmonotonicity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models are often evaluated on benchmarks that may already appear in pre-training data, inflating reported scores. This paper proposes the Kernel Divergence Score (KDS), a dataset-level measure that quantifies how contaminated a benchmark is with respect to a specific model. KDS compares the kernel similarity matrix of sample embeddings before and after fine-tuning on the benchmark, on the premise that fine-tuning reshapes relationships among unseen samples more than among seen ones. The paper claims that KDS satisfies monotonicity and consistency, correlating almost perfectly (average Spearman 0.990) with the true contamination fraction across three benchmarks and several model families. If correct, KDS gives researchers a practical way to rank benchmarks by leakage and choose less contaminated ones for evaluation.

What carries the argument

The load-bearing object is the kernel similarity matrix Phi(Z) with entries exp(-gamma||Zi - Zj||^2), computed from normalized embeddings Z of the dataset under the pre-trained model and again after fine-tuning (Z'). The Kernel Divergence Score is S(D,M) = -(1/E) sum_{i,j} |Phi(Z)_{i,j} log(Phi(Z)_{i,j}/Phi(Z')_{i,j})|, where E is a square-root normalizer. Equivalently, when gamma=1 the score weights the absolute change in squared pairwise distance by a soft gate exp(-||Zi - Zj||^2), so pairs that were originally close contribute more. The mechanism that separates seen from unseen samples is the differential effect of fine-tuning on pairwise embedding distances; the score's job is to aggregate that differential effect into one number.

What would settle it

Fine-tune a model on one half of an i.i.d. train/val split (for example a Pile subset) and compute KDS on mixtures with known contamination fractions from 0 to 1; if the score does not increase monotonically with the fraction, or if the histogram of pairwise embedding-distance changes shows seen-unseen and unseen-unseen pairs shifting by similar amounts, the central claim is falsified.

Watch

Extended reading notes

Core claim

The central claim is that dataset contamination can be scored without knowing the pre-training corpus, by measuring how much fine-tuning on the benchmark rearranges the embedding geometry of the samples. The paper defines the score as the negative of a normalized kernel divergence between the RBF kernel matrix computed from a pre-trained model's embeddings and the kernel matrix computed after LoRA fine-tuning on the dataset. A larger fraction of previously seen samples (higher lambda) yields a smaller kernel divergence, hence a larger KDS. In controlled mixtures with lambda from 0 to 1, KDS tracks the contamination fraction with near-perfect Spearman correlation on WikiMIA, BookMIA, and ArxivTection, and remains stable across random subsets, while several likelihood-based baselines show inconsistent or even reversed correlations.

Load-bearing premise

The load-bearing premise is that fine-tuning alters embedding relationships among samples the model has not seen more strongly than relationships among samples it has already seen; if that differential effect is missing or reversed, KDS stops measuring contamination.

Editorial extensions

If this is right

  • Benchmark providers can report a KDS value for each model, letting users see which evaluation results are trustworthy before running expensive tests.
  • Model developers can screen candidate evaluation suites and drop datasets whose scores indicate heavy overlap with pre-training data.
  • KDS can serve as a validation tool for synthetic contamination studies, because it recovers the known contamination fraction almost monotonically.
  • Because performance is stable across kernel choices, bandwidths, and embedding layers, KDS can be applied to new models without per-model tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same fine-tuning-differential signal could be adapted to per-sample contamination estimates, not just dataset-level scores, by examining how individual embeddings move rather than pairwise structure.
  • KDS may work outside language: any model family with an embedding space and a fine-tuning procedure could use the same score to audit benchmark overlap.
  • Because the score depends on the fine-tuning recipe (optimizer, epochs, LoRA rank), KDS values are probably comparable within a fixed recipe but not directly across different fine-tuning setups.
  • The monotonicity evidence comes from benchmarks whose seen/unseen labels may carry temporal cues; an i.i.d. split with no temporal signals is the sharper test of the method's premise.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Kernel Divergence Score (KDS), a dataset-level contamination score S(D, M) defined by Eq. (5) as the negative of a kernel-weighted divergence between RBF similarity matrices of sample embeddings before and after LoRA fine-tuning on the benchmark. The central claim is that KDS satisfies two requirements: monotonicity (positive correlation with the contamination fraction λ) and consistency (stability across random subsets). Controlled experiments on WikiMIA, BookMIA, and ArxivTection report near-perfect Spearman correlations (average 0.990), and an additional evaluation on six Pile subsets is presented as a temporal-shift-free IID test. Ablation studies analyze the roles of the soft gate, fine-tuning, kernel choice, bandwidth, embedding layer, and training configuration.

Significance. If the central claim holds, KDS would be a practically useful tool for ranking benchmarks by their leakage with respect to a given model, requiring only model embeddings and fine-tuning rather than access to pretraining data. The paper's strengths include well-controlled contamination-rate sweeps, direct comparison against several baselines, a thorough ablation of design choices, and release of code. The reported average Spearman of 0.990 on the three MIA-style benchmarks is strong. However, the significance is tempered by the fact that the cleanest IID validation (Pile subsets) rests on an unverified membership assumption, and the only direct evidence for the underlying differential-effect mechanism uses the very benchmarks that the paper acknowledges have temporal-shift problems.

major comments (4)
  1. [Section 6, Tables 5 and 6] The IID validation on Pile subsets assumes that the Pile train split is 'seen' data for Mistral-7B-Instruct-v0.2, but Mistral's pretraining corpus is not public and no membership verification is provided. If Mistral was not actually trained on these Pile documents (or trained on only a subset), then both 'train' and 'val' samples are unseen, and the high monotonic correlations in Table 6 cannot be attributed to contamination. This is a load-bearing issue because this experiment is the only attempt to rule out temporal-shift confounds. The paper should either use a model with publicly known training data (e.g., Pythia, OLMo), or verify membership through a reliable detection method, or explicitly acknowledge that the Pile experiment does not establish contamination-driven monotonicity for Mistral.
  2. [Section 3, Figure 7, Appendix E] The claimed mechanism that 'fine-tuning affects embedding relationships involving unseen samples more significantly than those involving seen samples' is supported only by the histogram in Figure 7, which uses seen/unseen labels from WikiMIA/BookMIA. Section 6 itself states that these benchmarks are susceptible to temporal shift issues, so the differential effect in Figure 7 may be driven by temporal cues rather than by pretraining membership. The paper should quantify the separation between pair-type distributions and demonstrate that the effect persists under a temporal-shift-free setting (e.g., the Pile setup with verified membership). Without this, the causal interpretation of KDS is not established.
  3. [Tables 1, 5, 6, and Figure 3] The tables report correlations averaged over five random subsets but do not provide standard deviations or confidence intervals, and Figure 3 shows variability only for WikiMIA. Consequently, the reader cannot assess the statistical significance of the reported averages, particularly for the 500-sample Pile runs in Table 6. Additionally, the text states that Pile correlations are 'near 1.0, averaging at 0.944 and 0.948,' but an average of 0.944 is not 'near 1.0' in the same sense as the WikiMIA value of 0.999. The claims should be softened or error bars should be reported.
  4. [Appendix D] The normalization factor E uses a square-root to compensate for the linear scaling of the kernel-entry sum, but the linearity is demonstrated only on WikiMIA (Figure 6, R^2=0.9766). If KDS is intended to compare benchmarks of different sizes, the paper should validate size-invariance on additional datasets or show that the normalization is theoretically justified. Currently, the consistency requirement is only tested on fixed-size subsets, so the behavior of KDS across varying dataset sizes is not established.
minor comments (5)
  1. [Table 5] The header 'HackeerNews' is a typo for 'HackerNews'.
  2. [Appendix E] The phrase 'highlighting and confirmaing our intuition' contains a typo: 'confirmaing' should be 'confirming'.
  3. [Appendix F] The word 'appraoches' is a typo for 'approaches'.
  4. [Eq. (6), Section 5] The text says 'the effect of γ is limited to a constant multiplicative factor,' but Eq. (6) shows γ also appears inside the exponential, changing the sharpness of the soft gate. The empirical invariance is convincing, but the explanation is imprecise.
  5. [Eq. (3)] The displayed equation does not clearly show the absolute-value bars that appear in Eq. (4); please ensure the notation is consistent.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: KDS is defined from embedding kernel matrices independently of contamination labels, and its monotonicity/consistency results are measured empirical properties rather than fitted or definitional identities.

full rationale

The derivation chain is self-contained at the construction level. Section 2.1 models a benchmark as a mixture with contamination fraction λ, and Section 3 defines S(D, M) in Eq. (5) directly from the RBF kernel matrices before and after LoRA fine-tuning. No parameter of S is fitted to λ or to any seen/unseen label; the near-perfect Spearman correlations in Tables 1 and 6 are observed relationships, not regression fits renamed as predictions. The motivating statement that fine-tuning moves unseen embeddings more than seen embeddings is a hypothesis supported post hoc by Figure 7, and the Section 6 Pile validation relies on the proxy that the Pile train split is seen data for Mistral ('the train set is regarded as seen data'), which is a ground-truth-validity limitation rather than a circular construction: even if that proxy is wrong, the score itself is not defined in terms of the validation labels. The self-citations present (Li 2023 perplexity baseline; Zhang et al. 2025 fine-tuned score deviation) appear as comparison methods and related work, not as justification of Eq. (5), so they are not load-bearing. No equation reduces to its own input, and no claimed prediction is forced by construction.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method has no invented entities and no parameters fit to the contamination labels. It rests on the empirical claim of differential fine-tuning effects and on the correctness of benchmark membership labels, both domain assumptions. The remaining choices (bandwidth, LoRA settings) are fixed a priori and shown to be broadly robust, though not universally.

free parameters (2)
  • RBF kernel bandwidth gamma = inverse of median pairwise distance (median heuristic)
    Set by a standard heuristic, not fit to contamination labels. Robust across a wide range (Table 3); only gamma=10 degrades performance due to numerical issues.
  • LoRA fine-tuning hyperparameters (rank, alpha, dropout, learning rate, batch size, epochs) = rank=8, alpha=32, dropout=0.1, LR=1e-4, batch size=4, 1 epoch
    Fixed a priori for all experiments, not tuned on the target. Performance depends on these choices: batch GD and 4 epochs reduce correlation (Table 4).
assumptions (4)
  • domain assumption Fine-tuning shifts embedding relationships of unseen samples more than seen samples
    Section 3 states this as the motivation. It is the load-bearing premise; if false, KDS cannot separate seen from unseen. Evidence is indirect (correlations) plus one histogram (Figure 7).
  • domain assumption The benchmark's seen/unseen labels correctly reflect pre-training exposure, so D = D_seen union D_unseen with fraction lambda as in Eq. (1)
    Section 4.1 uses WikiMIA, BookMIA, ArxivTection labels as ground truth. The paper acknowledges these benchmarks have temporal-shift issues (Section 6), so label correctness is not guaranteed.
  • standard math Embeddings are normalized and the kernel matrix is computed with the RBF kernel as defined
    Section 3 fixes this preprocessing; normalization is a standard choice and the kernel is positive semidefinite.
  • standard math The algebraic expansion in Eq. (4) is valid for gamma=1
    Direct log-ratio expansion of RBF kernels; verified by inspection.

how reviews work

0 comments
Cite this review

Pith. "Pith review of How Contaminated Is Your Benchmark? Quantifying Dataset Leakage in Large Language Models with Kernel Divergence." pith.science (2026). https://pith.science/paper/UEPS22F7

@misc{pith2026250200678,
  author       = {Pith},
  title        = {Pith review of: How Contaminated Is Your Benchmark? Quantifying Dataset Leakage in Large Language Models with Kernel Divergence},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UEPS22F7}},
  note         = {Machine review of arXiv:2502.00678}
}
read the original abstract

Dataset contamination, where evaluation datasets overlap with pre-training corpora, inflates performance metrics and undermines the reliability of model evaluations. Measuring dataset contamination thus becomes essential to ensure that performance evaluations genuinely reflect a model's ability to generalize to unseen data, rather than relying on memorized examples. To address this problem, we propose Kernel Divergence Score (KDS), a novel method that evaluates dataset contamination by computing the divergence between the kernel similarity matrix of sample embeddings, before and after fine-tuning on the benchmark dataset. Leveraging the insight that fine-tuning affects unseen samples more significantly than seen ones, KDS provides a reliable measure of contamination. Through extensive experiments on controlled contamination scenarios, KDS demonstrates a near-perfect correlation with contamination levels and outperforms existing baselines. Additionally, we perform comprehensive ablation studies to analyze the impact of key design choices, providing deeper insights into the components and effectiveness of KDS. These ablations highlight the importance of leveraging fine-grained kernel-based information and confirm the reliability of the proposed framework across diverse datasets and settings. Code is released in https://github.com/deeplearning-wisc/kernel-divergence-score.

Figures

Figures reproduced from arXiv: 2502.00678 by the authors.

Figure 1
Figure 1. Overview of the proposed Kernel Divergence Score (KDS) framework for measuring dataset contamination in large language models. The process involves extracting sample embeddings from the model before and after fine-tuning on the benchmark dataset D, computing the kernel similarity matrix for each stage, and measuring the divergence between the two matrices Φ(Z) and Φ(Z ′ ). By capturing the changes of embeddings indu… view at source ↗
Figure 2
Figure 2. Decomposition of the Kernel Divergence Score. Each component of the Kernel Divergence Score function is shown. Φ(·) denotes the kernel similarity matrix, Z and Z ′ represent normalized sample embeddings before and after fine-tuning, and ⊙ is the Hadamard product. Score and embeddings are based on Llama-3.1-8B-Instruct (Dubey et al., 2024). (Left) shows that the original kernel similarity matrix before fine-tuning. N… view at source ↗
Figure 3
Figure 3. Trend of Kernel Divergence Scores on WikiMIA. The score shows monotonic increase with respect to contamination rate, and the standard deviation over 5 runs is low. Kernel divergence score satisfies the consistency require￾ment. To verify the Consistency requirement, we test whether our score remains stable across independently and identically distributed datasets sampled from the same dis￾tribution and with the same… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Scoring performance across embedding location. Cor￾relation coefficients from different layers are retrieved using Mistral-7B-Instruct-v0.2 on WikiMIA. How does the kernel bandwidth γ impact the perfor￾mance? In an RBF kernel, the bandwidth parameter γ controls the sha…
Figure 5
Figure 5. Figure 5: Decomposition of the Kernel Divergence Score - full list. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Sum of kernel entries by data subset size. The data subsets are randomly sampled from the WikiMIA dataset. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Histogram on the values of embedding distance change. F. Extended Literature Review Here, we provide a more descriptive review of previous works on membership inference attack (MIA) (Shokri et al., 2017; Truex et al., 2019), as it is related to the objective of our wor…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Economics of AI Training Data: A Research Agenda

    cs.CY 2025-10 unverdicted novelty 5.0 of 10

    The paper synthesizes fragmented research to frame data economics around data's nonrivalry and context dependence, catalogs 2020-2025 AI training data deals, and proposes a hierarchy of data units while listing four f...

  2. Maintaining MTEB: Towards Long Term Usability and Reproducibility of Embedding Benchmarks

    cs.CL 2025-06 conditional novelty 4.0 of 10

    The MTEB maintainers document their infrastructure for versioning and validating benchmark components, plus a zero-shot score that flags models trained on benchmark tasks.

Reference graph

Works this paper leans on

26 extracted references · 13 canonical work pages · cited by 2 Pith papers

  1. [1]

    A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl, H., et al

    Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan, A. A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl, H., et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219,

    arXiv

  2. [4]

    (Zhang et al., 2024b) Following the general guideline from Shi et al

    (Min-K%++ Score) is the negated mean normalized probability from bottom-k% tokens averaged across samples: − 1 n · |Ki| nX i=1 X xj ∈Ki log Pθ(xj|x<j) − µx<j σx<j , (10) where Ki is the set of bottom-k% tokens from sample i, µx<j = Ez∼p(·|x<j )[log p(z|x<j)] is the expected log probability over the vocabulary of the model, and σx<j = q Ez∼p(·|x<j )[(log p...

    work page 2023

  3. [5]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  4. [7]

    Histogram on the values of embedding distance change. F. Extended Literature Review Here, we provide a more descriptive review of previous works on membership inference attack (MIA) (Shokri et al., 2017; Truex et al., 2019), as it is related to the objective of our work in quantifying leakage (i.e., contamination) in datasets (Magar & Schwartz, 2022; Xu e...

    work page 2019

  5. [8]

    Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825,

    arXiv

  6. [10]

    Llm dataset inference: Did you train on my dataset? arXiv preprint arXiv:2406.06443,

    Maini, P., Jia, H., Papernot, N., and Dziedzic, A. Llm dataset inference: Did you train on my dataset? arXiv preprint arXiv:2406.06443,

    arXiv

  7. [11]

    Membership inference attacks against language models via neighbourhood comparison

    Mattern, J., Mireshghallah, F., Jin, Z., Schoelkopf, B., Sachan, M., and Berg-Kirkpatrick, T. Membership inference attacks against language models via neighbourhood comparison. In Findings of the Association for Computational Linguistics: ACL 2023 , pp. 11330–11343,

    work page 2023

  8. [12]

    Sok: Membership inference attacks on llms are rushing nowhere (and how to fix it)

    Meeus, M., Shilov, I., Jain, S., Faysse, M., Rei, M., and de Mon- tjoye, Y .-A. Sok: Membership inference attacks on llms are rushing nowhere (and how to fix it). arXiv preprint arXiv:2406.17975,

    arXiv

Show all 26 references
  1. [13]

    Ml-leaks: Model and data independent membership inference attacks and defenses on machine learning models

    Salem, A., Zhang, Y ., Humbert, M., Berrang, P., Fritz, M., and Backes, M. Ml-leaks: Model and data independent membership inference attacks and defenses on machine learning models. In Proceedings 2019 Network and Distributed System Security Symposium. Internet Society,

    2019

  2. [14]

    Membership inference attacks against machine learning models

    10 Kernel Divergence Score Shokri, R., Stronati, M., Song, C., and Shmatikov, V . Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pp. 3–18. IEEE,

    2017

  3. [17]

    Benchmark data contamination of large language models: A survey

    Xu, C., Guan, S., Greene, D., Kechadi, M., et al. Benchmark data contamination of large language models: A survey. arXiv preprint arXiv:2406.04244,

    arXiv

  4. [18]

    E., and Stoica, I

    Yang, S., Chiang, W.-L., Zheng, L., Gonzalez, J. E., and Stoica, I. Rethinking benchmark and contamination for language mod- els with rephrased samples. arXiv preprint arXiv:2311.04850,

    arXiv

  5. [19]

    Data contamination calibration for black-box llms

    Ye, W., Hu, J., Li, L., Wang, H., Chen, G., and Zhao, J. Data contamination calibration for black-box llms. arXiv preprint arXiv:2405.11930,

    arXiv

  6. [20]

    Privacy risk in machine learning: Analyzing the connection to overfitting

    Yeom, S., Giacomelli, I., Fredrikson, M., and Jha, S. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF), pp. 268–282. IEEE,

    2018

  7. [23]

    12 A.2 Baseline Definitions

    11 Kernel Divergence Score Appendix Contents A Further Experimental Details 12 A.1 Implementation Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 A.2 Baseline Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . ....

    2021

  8. [25]

    The average Mean Absolute Percentage Error (MAPE) over 5 independent runs is calculated

    Consistency Requirement in terms of average MAPE. The average Mean Absolute Percentage Error (MAPE) over 5 independent runs is calculated. Among non-FSD baselines, our Kernel Divergence Score achieves the lowest average MAPE. Methods WikiMIA BookMIA ArxivTection Average Non-FS...

    1974

  9. [2008]

    The pile: An 800gb dataset of diverse text for language modeling

    Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027,

    arXiv

  10. [2017]

    E., Yu, L., and Wei, W

    Truex, S., Liu, L., Gursoy, M. E., Yu, L., and Wei, W. Demystifying membership inference attacks in machine learning as a service. IEEE transactions on services computing , 14(6):2073–2089,

  11. [2018]

    Pacost: Paired confidence signifi- cance testing for benchmark contamination detection in large language models

    Zhang, H., Lin, Y ., and Wan, X. Pacost: Paired confidence signifi- cance testing for benchmark contamination detection in large language models. In Findings of the Association for Computa- tional Linguistics: EMNLP 2024, pp. 1794–1809, 2024a. Zhang, H., Zhang, S., Jing, B., a...

    2024

  12. [2019]

    Recall: Membership inference via relative conditional log-likelihoods

    Xie, R., Wang, J., Huang, R., Zhang, M., Ge, R., Pei, J., Gong, N., and Dhingra, B. Recall: Membership inference via relative conditional log-likelihoods. In Proceedings of the 2024 Confer- ence on Empirical Methods in Natural Language Processing, pp. 8671–8689,

    2024

  13. [2020]

    Large sample anal- ysis of the median heuristic

    Garreau, D., Jitkrittum, W., and Kanagawa, M. Large sample anal- ysis of the median heuristic. arXiv preprint arXiv:1707.07269,

    arXiv

  14. [2021]

    Membership inference attacks from first principles

    Carlini, N., Chien, S., Nasr, M., Song, S., Terzis, A., and Tramer, F. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 1897–1914. IEEE,

    2022

  15. [2022]

    Blind baselines beat member- ship inference attacks for foundation models

    Das, D., Zhang, J., and Tram`er, F. Blind baselines beat member- ship inference attacks for foundation models. arXiv preprint arXiv:2406.16201,

    arXiv

  16. [2023]

    Estimating contamination via perplexity: Quantifying memorisation in language model evaluation

    Li, Y . Estimating contamination via perplexity: Quantifying memorisation in language model evaluation. arXiv preprint arXiv:2309.10677,

    arXiv

  17. [2024]

    Do membership inference attacks work on large language models? arXiv preprint arXiv:2402.07841,

    Duan, M., Suri, A., Mireshghallah, N., Min, S., Shi, W., Zettle- moyer, L., Tsvetkov, Y ., Choi, Y ., Evans, D., and Hajishirzi, H. Do membership inference attacks work on large language models? arXiv preprint arXiv:2402.07841,

    arXiv

  18. [2025]

    F., and Li, H

    Zhang, J., Sun, J., Yeats, E., Ouyang, Y ., Kuo, M., Zhang, J., Yang, H. F., and Li, H. Min-k%++: Improved baseline for detecting pre-training data from large language models. arXiv preprint arXiv:2404.02936, 2024b. Zhou, K., Zhu, Y ., Chen, Z., Chen, W., Zhao, W. X., Chen, X....

    arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.