Pith. sign in

REVIEW 2 major objections 6 minor 18 references

The Poisoned Chalice of LLM Evaluation Report

T0 review · 2 major / 6 minor · reviewed 2026-07-09 · glm-5.2

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Structure beats statistics for detecting LLM memorization

desk verdict Competition report showing structure-aware MIA beats likelihood-based methods on code, but a cross-dataset confound undermines the central claim. read the letter →

arxiv 2607.07481 v1 pith:ESOMXAYL submitted 2026-07-08 cs.SE

Jonathan Katzy , Ali Al-Kaswan , Razvan Mihai Popescu , Zhou Yang This is my paper · ORCID
classification cs.SE
keywords membershipinferencedatacontaminationcodelanguagemodelssoftwareengineeringevaluationwhite-boxattacksSERSEMTheHeapStarCoder2
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports the results of a competition designed to test whether one can reliably detect if a specific code file was part of a large language model's training data, without having access to the training corpus itself. The competition frames this as a white-box membership inference task: participants are given open-weight code models (StarCoder2-3B and 7B) and curated sets of files known to be either in or out of the training data, and must build a method that distinguishes them. The central finding is that simple likelihood-based methods, which check whether a model finds certain tokens easy or hard to predict, perform barely above random chance (AUC 0.56-0.61) on carefully curated code data. In contrast, a method called SERSEM, which combines entropy-weighted token scoring with probing of the model's internal hidden-layer representations and focuses on human-authored signals like comments and identifiers, achieves substantially better performance (AUC 0.773 on StarCoder2-3B, 0.753 on a held-out model). The paper establishes that detecting code contamination requires structural and representational analysis rather than surface-level probability signals, and that this approach generalizes across different models and programming languages.

What carries the argument

SERSEM (Selective Entropy-Weighted Scoring for Membership Inference), a method that down-weights predictable syntactic boilerplate, emphasizes human-authored signals such as comments and developer markers, and combines weighted token-level scoring from output logits with probing of intermediate transformer activations. The competition evaluation framework itself is also a central object: curated member/non-member datasets built from The Stack Edu (seen) and The Heap (unseen, deduplicated against The Stack v2), with a bag-of-words classifier used to filter out files distinguishable by keyword distribution shifts, plus a held-out model and dataset for final evaluation.

What would settle it

If a subsequent study found that The Heap contains files that are near-duplicates of training data (e.g., through contamination pathways not covered by the locality-sensitive hashing deduplication), the non-member labels would be incorrect, and the AUC scores for all methods, including SERSEM, would be unreliable. Alternatively, if SERSEM's performance dropped to the baseline range (AUC ~0.56-0.61) when tested on additional held-out models beyond StarCoder2 and Mellum, the generalization claim would not hold.

Watch

Extended reading notes

Core claim

The core discovery is that membership inference on source code separates into two performance regimes: simple probability-based signals (loss, MinK%Prob, calibrated probabilities, perturbation-based calibration) cluster near random guessing with AUC scores between 0.56 and 0.61 on curated data, while SERSEM's structure-aware approach combining entropy-weighted token scoring with hidden-layer probing reaches AUC 0.773 on StarCoder2-3B and 0.753 on the held-out Mellum model. This gap demonstrates that memorization in code LLMs is more detectable through internal model representations and human-centric code artifacts (comments, long identifiers, string literals) than through aggregate token-lik

Load-bearing premise

The ground truth labels for non-membership depend entirely on The Heap being truly free of contamination from the training corpora of all target models. The Heap was deduplicated against The Stack v2 using locality-sensitive hashing, but if that deduplication missed near-duplicates or if indirect contamination pathways exist through other training data sources, the member/non-member labels would be noisy and all AUC scores would be affected.

Editorial extensions

If this is right

  • If SERSEM's approach generalizes as reported, benchmark maintainers could run it on their evaluation sets against candidate models to estimate contamination risk without needing access to training corpora, which is especially relevant for closed or partially closed models.
  • The finding that hidden-layer representations carry stronger memorization signals than output probabilities suggests that future code LLMs designed to resist membership inference may need to address internal representation leakage, not just output-level calibration.
  • The precision-recall trade-off noted in the conclusion implies that practical contamination detection may be best used as a triage tool (identifying suspicious files for manual review) rather than a definitive classifier, since high-confidence detection requires accepting low recall.
  • The competition's dataset curation methodology (using a bag-of-words classifier to remove distribution-shift artifacts) could become a standard preprocessing step for future membership inference benchmarks across modalities beyond code.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the gap between likelihood-based and structure-aware methods holds across more models and training scales, it would suggest that memorization in code models is fundamentally a representational phenomenon that surface-level outputs only weakly reflect, which could redirect the entire membership inference research agenda toward internal probing.
  • The dependency on The Heap's deduplication quality means that if near-duplicate contamination pathways exist (e.g., files appearing in training data through sources not covered by the deduplication), all reported AUC scores could be systematically biased. A sensitivity analysis on label noise would clarify whether the SERSEM-baseline gap is robust to this risk.
  • SERSEM's reliance on human-centric signals (comments, identifiers, TODO markers) implies that contamination detection may be easier in code with high developer-authored content and harder in boilerplate-heavy or auto-generated code, which could bias contamination assessments toward certain project types.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. This paper reports the setup and results of the 'Poisoned Chalice of LLM Evaluation Competition,' which frames code contamination detection as a white-box membership inference task. The competition provides curated member/non-member datasets (derived from The Stack Edu and The Heap), target models (StarCoder2-3B/7B), baselines, and a held-out evaluation on Mellum-4B. Two submitted methods are described: SERSEM, which combines entropy-weighted token scoring with hidden-layer probing, and CalibratedProbs, which adjusts log-probability by token diversity. SERSEM achieves AUC-ROC of 0.773 on StarCoder2-3B and 0.753 on held-out Mellum, substantially outperforming probability-based baselines that cluster near random guessing (AUC 0.56–0.61). The paper is concise and the competition design is well-motivated.

Significance. The paper makes a useful community contribution by organizing a competition with a held-out model and dataset, uniform execution environment, and curated data designed to remove keyword artifacts. The finding that simple likelihood-based MIA signals are insufficient for code contamination detection (AUC near 0.56–0.61) while structure-aware probing achieves substantially higher AUC is a meaningful empirical result. The reproducible evaluation setup and public dataset release are strengths. However, the central claim that SERSEM detects genuine membership rather than dataset-origin artifacts is not fully established by the current experimental design.

major comments (2)
  1. §2.2: The member set is drawn from The Stack Edu and the non-member set from The Heap. These are fundamentally different data sources with potentially different code distributions (comment density, identifier length, formatting conventions). The BoW curation step (§2.2) removes files distinguishable by keyword presence but does not address deeper distributional shifts. SERSEM specifically targets 'comments, long identifiers, string literals, formatting anomalies, and developer markers like TODO' (§3.1)—precisely the features most likely to differ systematically between educational/curated code (The Stack Edu) and general deduplicated code (The Heap). A method that detects 'is this file from The Stack Edu?' would generalize across both StarCoder2 and Mellum (both trained on The Stack v2) and achieve high AUC without performing genuine membership inference. The held-out Mellum evaluation (
  2. Footnote 3: PAC was evaluated on only 1,000 samples while all other methods used 5,000. This 5× difference in evaluation set size creates non-comparable AUC estimates with potentially higher variance for PAC. While PAC is a baseline rather than the central submission, the paper presents all five methods side-by-side in Figure 1 and §4 without flagging this caveat in the discussion. A note in the results section (not just the footnote) and confidence intervals or standard errors would allow readers to assess whether the AUC differences between PAC (0.563–0.568) and the other baselines (0.591–0.613) are meaningful.
minor comments (6)
  1. §5: The conclusion states that 'limited generalizability and sensitivity to keyword drift identified in prior work become less problematic when such confounds are explicitly addressed during dataset construction.' Given that the dataset-origin confound (Major Comment 1) is not addressed, this claim is overstated. Consider softening.
  2. §3.1 and §3.2: The AUC values reported in the method summaries (0.79 for SERSEM on both StarCoder2-3B and 7B) differ from the final evaluation values in §4 (0.773 on StarCoder2-3B). Clarify whether the §3 values are from participant-side experiments on the development set, while §4 values are from the held-out evaluation set.
  3. §2.2: The paper states that The Heap was 'deduplicated against The Stack v2' using locality-sensitive hashes. No false negative rate or sensitivity analysis for the deduplication is reported. While this may be outside the scope of the competition report, a brief acknowledgment of this assumption's importance would strengthen the paper.
  4. Figure 1: The ROC curves would benefit from confidence intervals or shaded regions, particularly given the different sample sizes across methods.
  5. §2.4: The baselines are described briefly but methodological details are deferred to external references. A sentence or two on the perturbation strategy for PAC would improve readability.
  6. The paper does not report per-language breakdowns of AUC, despite selecting five languages (Go, Java, Python, Ruby, Rust) 'to provide a broader overview of the generalizability of the submitted approaches across languages' (§2.2). A table or figure showing per-language performance should be included.

Simulated Author's Rebuttal

2 responses · 1 unresolved

We thank the referee for the careful review and agree that the dataset-origin confound is a serious concern warranting explicit discussion in the revision. We accept the PAC sample-size caveat and will surface it more prominently. On the core confound issue, we agree the current design cannot fully rule it out and will revise the manuscript's claims accordingly; a complete fix requires additional experiments that are beyond what we can conduct within the revision window.

read point-by-point responses

  1. Referee: §2.2: The member set is drawn from The Stack Edu and the non-member set from The Heap. These are fundamentally different data sources with potentially different code distributions. The BoW curation step removes keyword-distinguishable files but does not address deeper distributional shifts. SERSEM targets features (comments, long identifiers, formatting anomalies, TODO markers) that may differ systematically between educational/curated code and general deduplicated code. A method detecting 'is this from The Stack Edu?' would generalize across StarCoder2 and Mellum and achieve high AUC without performing genuine membership inference. The held-out Mellum evaluation does not rule this out.

    Authors: The referee raises a valid and important concern. We acknowledge that the current experimental design cannot definitively distinguish genuine membership inference from detection of dataset-origin artifacts. The referee is correct that The Stack Edu (educational/curated code) and The Heap (general deduplicated code) may differ in distributional properties—comment density, identifier length, formatting conventions—that align precisely with the features SERSEM targets. The BoW classifier we employed removes files distinguishable by keyword presence, but it does not address deeper stylistic or structural distributional shifts. Furthermore, the referee's observation that both StarCoder2 and Mellum were trained on The Stack v2 (which contains The Stack Edu) means that a method detecting 'is this file from The Stack Edu?' would indeed generalize across both models and achieve high AUC without performing genuine membership inference. The held-out Mellum evaluation, which we presented as evidence of generalization, does not rule out this alternative explanation. We will revise the manuscript to explicitly acknowledge this limitation. Specifically: (1) We will add a dedicated discussion in §4 or §5 noting that the dataset construction creates a potential confound between membership status and dataset origin, and that SERSEM's feature design (targeting comments, long identifiers, developer markers) is precisely the type of approach that could exploit this confound. (2) We will soften the claim that SERSEM 'detects genuine membership' and instead frame the result as: SERSEM achieves high AUC, but the current design cannot determine whether this reflects genuine memorization signals or dataset-origin artifacts. (3) We will note that a definitive test would require member and non成员文件 revision: partial

  2. Referee: Footnote 3: PAC was evaluated on only 1,000 samples while all other methods used 5,000. This 5× difference creates non-comparable AUC estimates with potentially higher variance for PAC. The paper presents all five methods side-by-side in Figure 1 and §4 without flagging this caveat in the discussion. A note in the results section (not just the footnote) and confidence intervals or standard errors would allow readers to assess whether the AUC differences between PAC (0.563–0.568) and the other baselines (0.591–0.613) are meaningful.

    Authors: The referee is correct. The 5× difference in evaluation set size between PAC and all other methods is a methodological inconsistency that should be surfaced in the results discussion, not buried in a footnote. We will make two changes: (1) We will add an explicit caveat in §4 (the results discussion) noting that PAC was evaluated on 1,000 samples due to runtime constraints, while all other methods used 5,000 samples, and that this makes PAC's AUC estimates not directly comparable. (2) We will add confidence intervals (bootstrap or DeLong) for all AUC estimates in Figure 1 and the results table, so readers can assess whether the differences between PAC (0.563–0.568) and the other baselines (0.591–0.613) are within noise. We note that even with wider confidence intervals, PAC's AUC remains near random guessing and the qualitative conclusion—that simple probability-based and perturbation-based baselines are insufficient—does not change. But the referee is right that readers should be able to make this assessment themselves. revision: yes

standing simulated objections not resolved
  • The dataset-origin confound (Major Comment 1) cannot be fully resolved within this revision. A definitive test would require member and non-member files drawn from the same source distribution (e.g., both from The Stack Edu, with non-members being files confirmed excluded from training), or a control experiment where SERSEM is evaluated on Stack Edu vs. Heap classification without model involvement. These experiments require new dataset construction and re-evaluation that are beyond what we can complete in the revision window. We will honestly acknowledge this limitation in the revised manuscript rather than claim it is resolved.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity found; the paper is an empirical competition report with self-cited dataset infrastructure that is independently verifiable.

full rationale

This paper reports the setup and results of a competition. Its central claims (SERSEM achieves AUC 0.773 on StarCoder2-3B, 0.753 on held-out Mellum; probability-based baselines cluster near random) are measured empirical results on held-out evaluation data, not derivations from first principles. The self-citation to The Heap [11] is load-bearing for non-member labels, but The Heap is a dataset construction whose methodology (LSH deduplication against The Stack v2) is published and independently verifiable — the paper does not claim to 'predict' any property of The Heap from its own prior work. The BoW curation step cites external work [13] (Meeus et al.). SERSEM itself was submitted by external participants (Dikici et al. [8]), not the paper's authors. The member labels rest on the factual composition of The Stack v2 (The Stack Edu is a known subset), not on a derived result. No step in the paper reduces a 'prediction' to its inputs by construction. The reader's concerns about deduplication quality and the skeptic's concerns about distributional confounds between The Stack Edu and The Heap are validity/correctness risks, not circularity — they question whether the labels are correct, not whether the paper's claims are tautologically forced by self-citation. The one point is for the self-citation to The Heap being load-bearing for all non-member labels without sensitivity analysis on label noise, but this is a minor concern that does not make the results circular.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No new entities are invented. The competition uses existing models, datasets, and evaluation metrics.

free parameters (2)
  • Bag-of-words classifier threshold for curation = not specified
    The paper states files were retained if misclassified by a bag-of-words classifier, but the classifier's decision threshold and training details are not specified. This affects which files enter the evaluation set.
  • Language subset selection = Go, Java, Python, Ruby, Rust
    Five languages were selected from a larger pool. The selection rationale ('broader overview') is not parameter-free; it introduces a design choice that affects generalizability claims.
assumptions (4)
  • domain assumption The Heap is contamination-free relative to The Stack v2 and all target model training corpora.
    Stated in §2.2: 'We selected The Heap as the source of unseen data because it has been deduplicated against The Stack v2.' This is the load-bearing assumption for all non-member labels.
  • domain assumption Locality-sensitive hashing deduplication removes all near-duplicate files that could create label noise.
    §2.2: 'we select files based on the near duplicates column, which uses locality sensitive hashes to also remove files that had only minor changes made to them.' No false-negative rate is reported.
  • domain assumption AUC-ROC is the appropriate metric for this task.
    §2.3 justifies AUC-ROC over TPR@xFPR because 'the community does not yet agree on a single acceptable false positive rate.' This is reasonable but means threshold-dependent practical performance is not assessed.
  • domain assumption Bag-of-words misclassification removes all superficial distribution shifts between member and non-member sets.
    §2.2: 'This resulted in member and non-member sets that could not be distinguished simply by the presence of specific keywords.' The claim that this removes all distribution shift is stronger than what a bag-of-words classifier can guarantee.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Poisoned Chalice of LLM Evaluation Report." pith.science (2026). https://pith.science/paper/ESOMXAYL

@misc{pith2026260707481,
  author       = {Pith},
  title        = {Pith review of: The Poisoned Chalice of LLM Evaluation Report},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ESOMXAYL}},
  note         = {Machine review of arXiv:2607.07481}
}
read the original abstract

Large language models are increasingly used to evaluate and support software engineering tasks, yet the validity of these evaluations is often undermined by uncertainty about whether benchmark instances were seen during pretraining. This can lead to data contamination, which may inflate performance and result in misleading conclusions about model capability. Despite this, the training corpora of many modern models are only partially disclosed, making direct decontamination infeasible. This creates a need for practical methods that can detect a large language models' prior exposure to training data without access to the full training corpus. To address this challenge, we organize the first Poisoned Chalice of LLM Evaluation Competition, co-located with the FSE-AIWare 2026 Competition Track. The competition frames contamination detection as a white-box membership inference task on source code and provides participants with curated datasets, target models, baseline attacks, and a final evaluation on a held-out model and dataset. This design encourages methods that generalize beyond superficial dataset artifacts and beyond a single training setting. This paper reports the setup and results of the competition. More broadly, the competition aims to catalyze the community around trustworthy LLM evaluation for software engineering.

Figures

Figures reproduced from arXiv: 2607.07481 by the authors.

Figure 1
Figure 1. ROC curves for all evaluated membership inference methods on the final test sets for [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 18 canonical work pages

  1. [11]

    Jonathan Katzy, Razvan Popescu, Arie Van Deursen, and Maliheh Izadi. 2024. An Exploratory Investigation into Code License Infringements in Large Language Model Training Datasets. InProceedings of the 2024 IEEE/ACM First International Conference on AI Foundation Models and Software Engineering(Lisbon, Portugal) (FORGE ’24). Association for Computing Machin...

    work page doi:10.1145/3650105.3652298 2024

  2. [1]

    Ali Al-Kaswan and Maliheh Izadi. 2023. The (ab) use of open source code to train large language models. In2023 IEEE/ACM 2nd International Workshop on Natural Language-Based Software Engineering (NLBSE). IEEE, 9–10

    work page 2023

  3. [2]

    Ali Al-Kaswan, Maliheh Izadi, and Arie Van Deursen. 2023. Targeted attack on gpt-neo for the satml language model data extraction challenge.arXiv preprint arXiv:2302.07735(2023)

    work page Pith review arXiv 2023

  4. [3]

    Ali Al-Kaswan, Maliheh Izadi, and Arie Van Deursen. 2024. Traces of memori- sation in large language models for code. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering. 1–12

    work page 2024

  5. [4]

    Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guil- herme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíček, Agustín Pi- queres Lajarín, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathieu Morlon, Colin Raffel, Leandro von ...

    work page Pith review arXiv 2025

  6. [5]

    Alexander Berndt, Vekil Bekmyradov, Thomas Bach, and Sebastian Baltes. [n. d.]. Evaluating Signals for Membership Inference Attacks on Source Code

    work page

  7. [6]

    Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. 2022. Membership inference attacks from first principles. In2022 IEEE symposium on security and privacy (SP). IEEE, 1897–1914

    work page 2022

  8. [7]

    Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. Extracting training data from large language models. In30th USENIX security symposium (USENIX Security 21). 2633–2650

    work page 2021

Show all 18 references
  1. [8]

    Kıvanç Kuzey Dikici, Serdar Kara, Semih Çağlar, Eray Tüzün, and Sinem Sav

  2. [9]

    InCompanion Proceedings of the 34th ACM Symposium on the Foundations of Software Engineering (FSE ’26)

    SERSEM: Selective Entropy-Weighted Scoring for Membership Inference in Code Language Models. InCompanion Proceedings of the 34th ACM Symposium on the Foundations of Software Engineering (FSE ’26). ACM

  3. [10]

    Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large language models for software engineering: A systematic literature review.ACM Transactions on Software Engineering and Methodology33, 8 (2024), 1–79

    2024

  4. [12]

    Jonathan Katzy, Razvan Mihai Popescu, Arie van Deursen, and Maliheh Izadi

  5. [13]

    In2025 IEEE/ACM Second International Conference on AI Foundation Models and Software Engineering (Forge)

    The heap: A contamination-free multilingual code dataset for evaluating large language models. In2025 IEEE/ACM Second International Conference on AI Foundation Models and Software Engineering (Forge). IEEE, 151–155

  6. [14]

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy- Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zi- jian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul...

    2024 arXiv

  7. [15]

    Matthieu Meeus, Igor Shilov, Shubham Jain, Manuel Faysse, Marek Rei, and Yves-Alexandre de Montjoye. 2025. Sok: Membership inference attacks on llms are rushing nowhere (and how to fix it). In2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML). IEEE, 385–401

    2025

  8. [16]

    Nikita Pavlichenko, Iurii Nazarov, Ivan Dolgov, Ekaterina Garanina, Dmitry Ustalov, Ivan Bondyrev, Kseniia Lysaniuk, Evgeniia Vu, Kirill Chekmenev, Joseph Shtok, et al. 2025. Mellum: Production-Grade in-IDE Contextual Code Completion with Multi-File Project Understanding.arXiv...

    2025

  9. [17]

    Zhou Yang, Jieke Shi, Premkumar Devanbu, and David Lo. 2025. Ecosystem of large language models for code.ACM Transactions on Software Engineering and Methodology35, 1 (2025), 1–30

    2025

  10. [18]

    Zhou Yang, Zhipeng Zhao, Chenyu Wang, Jieke Shi, Dongsun Kim, Donggyun Han, and David Lo. 2024. Unveiling memorization in code models. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering. 1–13

    2024

Pith tools

Reviewed July 9, 2026 · model on record in the stance chip above.