Pith. sign in

REVIEW 4 major objections 6 minor 23 references

Addressing Data Leakage in HumanEval Using Combinatorial Test Design

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that all four tested LLMs perform consistently worse on reworded HumanEval tasks, indicating likely data leakage in the original benchmark.

desk verdict Sensible template-based benchmark idea, but the data-leakage claim is not supported because the HumanEval_T variants differ from HumanEval in prompt format and task equivalence is never validated. read the letter →

arxiv 2412.01526 v1 pith:KDYZWPWY submitted 2024-12-02 cs.SE cs.AI

Jeremy S. Bradbury , Riddhi More This is my paper · ORCID
classification cs.SEcs.AI
keywords dataleakagecontaminationHumanEvalbenchmarkconstructioncombinatorialtestdesigntemplatetasksLLMevaluationcodegeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a way to build LLM benchmarks that are harder to game by memorization: turn each task into a template, use combinatorial test design to generate several concrete wordings, and assemble evaluation variants from those wordings. The authors build HumanEval_T from 10 HumanEval problems and test four LLMs on five variants. Every model scores lower on the variants than on the original tasks, with drops from 4.75 to 13.75 percentage points. The paper reads the consistency of the drop as likely evidence that HumanEval has leaked into training data, and argues that template-variant benchmarks can separate real capability from memorized performance.

What carries the argument

The central mechanism is the template task: a HumanEval problem generalized by replacing specific words, such as input type, threshold descriptor, and value descriptor, with variables. Combinatorial test design then systematically instantiates those variables into multiple concrete tasks, and benchmark variant assembly randomly selects one concrete task per template into each evaluation variant. This produces textually distinct but purportedly equivalent tasks, so any performance gap between original and variant benchmarks can be attributed to memorization of the original wording. That difference is what the comparison measures.

What would settle it

Run the same models on template variants of tasks that were released after the models' training cutoffs and thus could not have leaked; if those variants also score lower, the gap is about wording or difficulty, while if they score the same, the original gap is consistent with memorization.

Watch

Extended reading notes

Core claim

The paper's central claim is stated in Section V.B.1: the consistent performance drop across all models indicates likely data leakage in the original HumanEval benchmark. Since the concrete variants were designed to preserve semantic behaviour and difficulty, the authors attribute the lower pass rates to memorized exposure to the original tasks rather than to the variants being harder. They also claim that the five concrete versions of each template task are interchangeable enough for evaluation, based on consistency analysis across variants, while acknowledging that the results are preliminary and not conclusive.

Load-bearing premise

The load-bearing premise is that the reworded concrete tasks are semantically and difficulty-equivalent to the original HumanEval tasks, so a lower score can only be explained by memorization of the original wording.

Editorial extensions

If this is right

  • If HumanEval scores are inflated by leakage, published comparisons of code-generation LLMs on HumanEval need to be re-read as partly measuring memorization rather than capability.
  • A template-variant benchmark can be refreshed over time by generating new concrete tasks, allowing longitudinal model comparisons without freezing one static test.
  • The method gives a practical contamination check: a large gap between original and reworded variants flags possible leakage even when training data is black-box.
  • If all variants are truly interchangeable, smaller benchmark subsets can be reused across evaluation rounds, since new variants prevent direct memorization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not run is to apply the same template process to tasks that postdate the models' training cutoffs; if the original-versus-variant gap disappears there, the leakage explanation is confirmed rather than inferred.
  • A control using non-leaked tasks with similar wording difficulty would distinguish leakage from general sensitivity to rewording, since the current design has no such control.
  • The same construction method could transfer to other static benchmarks in program repair or clone detection, where contamination is equally hard to detect.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a benchmark-construction method in which template tasks are instantiated into multiple concrete tasks using combinatorial test design, and instantiates this method on a 10-problem subset of HumanEval to create HumanEval_T with five variants. The authors compare four LLMs on the original HumanEval subset and on the HumanEval_T variants, report consistent performance drops on the variants, and interpret these drops as evidence of likely data leakage in HumanEval. They also examine cross-variant consistency and conclude that the concrete tasks are likely interchangeable. The paper explicitly states that the results are not conclusive and identifies future work.

Significance. If the equivalence of the HumanEval_T variants with the original tasks were validated, the proposed method would be a practical way to build evolving benchmark variants and to probe for data contamination. The manuscript is honest in reporting preliminary results and the data are presented in a readable form. However, the central inference from performance drops to data leakage rests on an unvalidated equivalence assumption and on a confounded comparison, so the empirical contribution, as it stands, does not support the headline claim.

major comments (4)
  1. [Section IV.B and Section V.B.2] The evidence for data leakage is confounded with a prompt-format mismatch. The original HumanEval prompts include a function signature and example input/output pairs, while the HumanEval_T variants, per the footnote in Section IV.A, exclude the example pairs. A performance drop under this comparison can be caused by the absence of examples or by reworded descriptions that LLMs handle less well, not by memorization of the original tasks. The paper needs a control condition, such as reworded versions of the original tasks that keep the signature and examples, original-format prompts on held-out tasks, or example-free versions of the original HumanEval prompts. Without such a control, the phrase 'likely data leakage' in Section V.B.1 is not warranted.
  2. [Section V.B.1 and Table I] The load-bearing premise that HumanEval_T variants are semantically equivalent and of comparable complexity to the original tasks is asserted but not validated. Section IV.A states the goal of maintaining semantic behavioural equivalence and Section IV.B claims 'semantic equivalence and comparable complexity,' but there is no calibration, human study, difficulty metric, or control condition. If the reworded tasks are slightly harder or phrased in a way that LLMs process less effectively, the observed drops are fully explained without invoking data leakage. The authors need to demonstrate equivalence, for example through human ratings, difficulty measurements on uncontaminated models, or per-template pass-rate comparisons on a larger task set.
  3. [Section V.B.2] The reported numbers are internally inconsistent and lack uncertainty quantification. The GPT-3.5 row lists AVG=76.7 and HE=80.0, but the text reports a 4.75-point drop from 80.0 to 75.25; neither the printed AVG nor the row average of V1–V5 (75.92) equals 75.25. The paper says all results are averages of five runs, but it does not report per-run or per-task data, making it impossible to verify the averages, the 'consistent drop' claim, or the outlier assertion. With n=10 tasks, paired significance tests (e.g., Wilcoxon signed-rank) and confidence intervals are needed for each model before a leakage conclusion can be drawn.
  4. [Section V.B.2] The consistency analysis in RQ2 does not test interchangeability. Standard deviations of whole-benchmark averages (sigma from 1.8% to 8.9%) describe variation across five 10-task variants, not whether concrete tasks generated from the same template are interchangeable. For example, a model could pass exactly the same 8 of 10 tasks in every variant and still show a small standard deviation, which would not demonstrate that the hard tasks are equivalent across variants. The authors should report per-template results across variants and use a formal equivalence criterion, such as paired comparisons per task or an item-difficulty model.
minor comments (6)
  1. [Section IV.A] Please report which 10 HumanEval tasks were randomly sampled and provide the sampling procedure or seed, since the random subset strongly influences all reported gaps.
  2. [Section IV.A] The footnote stating that example input/output pairs were excluded from the analysis is easy to miss; this design decision should be moved to the experimental setup and discussed as a limitation, because it is central to interpreting Table I.
  3. [Table I] The 'AVG' column should be defined explicitly as the mean of V1–V5, and the GPT-3.5 row should be corrected to match the stated drop of 4.75 points.
  4. [Section IV.B] The combinatorial test design is described only by example; please specify the parameters, value sets, and coverage criterion used for the 10 tasks, or provide them in an appendix for reproducibility.
  5. [Figure 3] The boxplots would be more informative if per-task or per-variant points were overlaid, and the caption should clarify whether the boxes are computed across problems or across variants.
  6. [Section V.B.2] The text uses 'GPT-4' in the RQ2 discussion while Table I and Section V.A use 'GPT-4o'; please standardize the model name.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the observed performance gaps are measured outcomes, not fitted inputs, and no load-bearing claim reduces to its own premises by construction.

full rationale

The paper's central empirical result is a measured pass@1 gap between HumanEval and the HumanEval_T variants; no model parameter is fitted to produce that gap, and no quantity is predicted from the same data used to define it. The inference that the consistent drop 'indicates likely data leakage' rests on the unvalidated premise that the variants are semantically and difficulty-equivalent to the original tasks, which is a validity threat (format mismatch, missing example I/O, no control condition) rather than a circular reduction in the sense of an equation-level or definition-level identity. RQ2's consistency analysis reports measured variation across variants and is likewise empirical. The only self-citation, [19], describes prior use of combinatorial benchmark construction and is not load-bearing for the present results. No fitted input is renamed as a prediction, no uniqueness theorem is imported from the authors, and no ansatz is smuggled in via citation. The paper therefore exhibits no significant circularity under the review's criteria.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claims rest on the unvalidated equivalence of template-generated variants and original tasks, and on the representativeness of a 10-task sample. No free parameters are fitted and no new entities are introduced.

assumptions (3)
  • domain assumption Concrete task variants generated from a template are semantically and difficulty-equivalent to the original HumanEval task and to each other.
    Section IV.A states template variables are chosen so they do not impact difficulty or purpose, but no calibration or human study supports this.
  • domain assumption Any systematic performance gap between HumanEval and HumanEval_T is attributable to data leakage rather than to prompt wording, tokenization, or task framing.
    This is the inference made in Section V.B.1 and V.C; it depends on the first axiom and on the absence of other confounds.
  • domain assumption A 10-problem random sample from the 164 HumanEval tasks is representative enough to support cross-model conclusions.
    Section V.A selects a random subset of 10; with high per-task variance, n=10 and n=5 variants limit generality.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Addressing Data Leakage in HumanEval Using Combinatorial Test Design." pith.science (2026). https://pith.science/paper/KDYZWPWY

@misc{pith2026241201526,
  author       = {Pith},
  title        = {Pith review of: Addressing Data Leakage in HumanEval Using Combinatorial Test Design},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KDYZWPWY}},
  note         = {Machine review of arXiv:2412.01526}
}
read the original abstract

The use of large language models (LLMs) is widespread across many domains, including Software Engineering, where they have been used to automate tasks such as program generation and test classification. As LLM-based methods continue to evolve, it is important that we define clear and robust methods that fairly evaluate performance. Benchmarks are a common approach to assess LLMs with respect to their ability to solve problem-specific tasks as well as assess different versions of an LLM to solve tasks over time. For example, the HumanEval benchmark is composed of 164 hand-crafted tasks and has become an important tool in assessing LLM-based program generation. However, a major barrier to a fair evaluation of LLMs using benchmarks like HumanEval is data contamination resulting from data leakage of benchmark tasks and solutions into the training data set. This barrier is compounded by the black-box nature of LLM training data which makes it difficult to even know if data leakage has occurred. To address the data leakage problem, we propose a new benchmark construction method where a benchmark is composed of template tasks that can be instantiated into new concrete tasks using combinatorial test design. Concrete tasks for the same template task must be different enough that data leakage has minimal impact and similar enough that the tasks are interchangeable with respect to performance evaluation. To assess our benchmark construction method, we propose HumanEval_T, an alternative benchmark to HumanEval that was constructed using template tasks and combinatorial test design.

Figures

Figures reproduced from arXiv: 2412.01526 by the authors.

Figure 1
Figure 1. Benchmark Construction Approach Using Combinatorial Test Design [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Performance Distribution Across HumanEval [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Performance Comparison of Models with HumanEval scores [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 12 canonical work pages

  1. [1]

    Large language models for software engineering: Sur- vey and open problems,

    A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, and J. M. Zhang, “Large language models for software engineering: Sur- vey and open problems,” in 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE), 2023, pp. 31–53

    work page 2023

  2. [2]

    Software testing with large language models: Survey, landscape, and vision,

    J. Wang, Y . Huang, C. Chen, Z. Liu, S. Wang, and Q. Wang, “Software testing with large language models: Survey, landscape, and vision,”IEEE Transactions on Software Engineering, vol. 50, no. 4, pp. 911–936, 2024

    2024

  3. [3]

    Large language models for software engineering: A systematic literature review,

    X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engineering: A systematic literature review,” ACM Trans. Softw. Eng. Methodol. , Sep. 2024, just Accepted. [Online]. Available: https://doi-org.uproxy.library.dc-uoit.ca/10.1145/3695988

    doi:10.1145/3695988 2024

  4. [4]

    A Survey on Evaluation of Large Language Models,

    Y . Chang, X. Wang, J. Wang, Y . Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y . Wang, W. Ye, Y . Zhang, Y . Chang, P. S. Yu, Q. Yang, and X. Xie, “A Survey on Evaluation of Large Language Models,” ACM Trans. Intell. Syst. Technol. , vol. 15, no. 3, pp. 39:1–39:45, Mar. 2024. [Online]. Available: https://dl.acm.org/doi/10.1145/3641289

    doi:10.1145/3641289 2024

  5. [5]

    A Software Engineering Perspective on Testing Large Language Models: Research, Practice, Tools and Benchmarks,

    S. Hudson, S. Jit, B. C. Hu, and M. Chechik, “A Software Engineering Perspective on Testing Large Language Models: Research, Practice, Tools and Benchmarks,” Jun. 2024. [Online]. Available: https://arxiv.org/abs/2406.08216v1

    arXiv 2024

  6. [6]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Her...

    arXiv 2021

  7. [7]

    Codereval: A benchmark of pragmatic code generation with generative pre-trained models,

    H. Yu, B. Shen, D. Ran, J. Zhang, Q. Zhang, Y . Ma, G. Liang, Y . Li, Q. Wang, and T. Xie, “Codereval: A benchmark of pragmatic code generation with generative pre-trained models,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ser. ICSE ’24. New York, NY , USA: Association for Computing Machinery,

  8. [8]

    Towards standarized benchmarks of LLMs in software modeling tasks: a conceptual framework,

    J. C ˜A¡mara, L. Burgue ˜A±o, and J. Troya, “Towards standarized benchmarks of LLMs in software modeling tasks: a conceptual framework,” Software and Systems Modeling , Sep. 2024. [Online]. Available: https://doi.org/10.1007/s10270-024-01206-9

    doi:10.1007/s10270-024-01206-9 2024

Show all 23 references
  1. [9]

    Using benchmarking infrastructure to evaluate llm performance on cs concept inventories: Challenges, opportunities, and critiques,

    M. Ali, P. Rao, Y . Mai, and B. Xie, “Using benchmarking infrastructure to evaluate llm performance on cs concept inventories: Challenges, opportunities, and critiques,” in Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1 , ser. I...

    2024

  2. [10]

    Condefects: A complementary dataset to address the data leakage concern for llm-based fault localization and program repair,

    Y . Wu, Z. Li, J. M. Zhang, and Y . Liu, “Condefects: A complementary dataset to address the data leakage concern for llm-based fault localization and program repair,” in Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering,...

    2024

  3. [11]

    BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation,

    J. Dhamala, T. Sun, V . Kumar, S. Krishna, Y . Pruksachatkun, K.-W. Chang, and R. Gupta, “BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation,” in Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, ser. FAccT ’21....

    2021

  4. [12]

    Holistic Evaluation of Language Models,

    P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y . Zhang, D. Narayanan, Y . Wu, A. Kumar, B. Newman, B. Yuan, B. Yan, C. Zhang, C. Cosgrove, C. D. Manning, C. R ˜A©, D. Acosta- Navas, D. A. Hudson, E. Zelikman, E. Durmus, F. Ladhak, F. Rong, H. Ren, H. Yao,...

    2022 arXiv

  5. [13]

    Don’t Make Your LLM an Evaluation Benchmark Cheater,

    K. Zhou, Y . Zhu, Z. Chen, W. Chen, W. X. Zhao, X. Chen, Y . Lin, J.-R. Wen, and J. Han, “Don’t Make Your LLM an Evaluation Benchmark Cheater,” Nov. 2023. [Online]. Available: https://arxiv.org/abs/2311.01964v1

    2023 arXiv

  6. [14]

    NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark,

    O. Sainz, J. Campos, I. Garc ´ıa-Ferrero, J. Etxaniz, O. L. de Lacalle, and E. Agirre, “NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark,” in Findings of the Association for Computational Linguistics: EMNLP 2023 , H. Bouamor, J. Pino,...

    2023

  7. [15]

    Task Contamination: Language Models May Not Be Few-Shot Anymore,

    C. Li and J. Flanigan, “Task Contamination: Language Models May Not Be Few-Shot Anymore,” Proceedings of the AAAI Conference on Artificial Intelligence , vol. 38, no. 16, pp. 18 471–18 480, Mar. 2024, number: 16. [Online]. Available: https://ojs.aaai.org/index.php/AAAI/ articl...

    2024

  8. [16]

    A test generation strategy for pairwise testing,

    K.-C. Tai and Y . Lei, “A test generation strategy for pairwise testing,” IEEE Transactions on Software Engineering, vol. 28, no. 1, pp. 109–111, 2002

    2002

  9. [17]

    Combinatorial test design in practice,

    M. B. Cohen and S. Ur, “Combinatorial test design in practice,” in 2010 ACM/IEEE 32nd International Conference on Software Engineering , vol. 2, 2010, pp. 495–496

    2010

  10. [18]

    Using benchmarking to advance research: a challenge to software engineering,

    S. Sim, S. Easterbrook, and R. Holt, “Using benchmarking to advance research: a challenge to software engineering,” in 25th International Conference on Software Engineering, 2003. Proceedings. , May 2003, pp. 74–83, iSSN: 0270-5257. [Online]. Available: https://ieeexplore.ieee...

    2003

  11. [19]

    Using combinatorial benchmark construction to improve the assessment of concurrency bug detection tools,

    J. S. Bradbury, I. Segall, E. Farchi, K. Jalbert, and D. Kelk, “Using combinatorial benchmark construction to improve the assessment of concurrency bug detection tools,” in Proceedings of the 2012 Workshop on Parallel and Distributed Systems: Testing, Analysis, and Debugging, ...

    2012

  12. [20]

    Applying pairwise combinatorial testing to large language model testing,

    B. Garn, L. Kampel, M. Leithner, B. Celic, C. C ¸ ulha, I. Hiess, K. Kieseberg, M. Koelbing, D.-P. Schreiber, M. Wagner, C. Wech, J. Zivanovic, and D. E. Simos, “Applying pairwise combinatorial testing to large language model testing,” in Testing Software and Systems , S. Bonf...

    2023

  13. [21]

    Benchmark Self- Evolving: A Multi-Agent Framework for Dynamic LLM Evaluation,

    S. Wang, Z. Long, Z. Fan, Z. Wei, and X. Huang, “Benchmark Self- Evolving: A Multi-Agent Framework for Dynamic LLM Evaluation,” Feb. 2024, arXiv:2402.11443. [Online]. Available: http://arxiv.org/abs/ 2402.11443

    2024 arXiv

  14. [22]

    Dynamic Evaluation of Large Language Models by Meta Probing Agents,

    K. Zhu, J. Wang, Q. Zhao, R. Xu, and X. Xie, “Dynamic Evaluation of Large Language Models by Meta Probing Agents,” Jun. 2024, arXiv:2402.14865. [Online]. Available: http://arxiv.org/abs/2402.14865

    2024 arXiv

  15. [2024]

    Available: https://doi.org/10.1145/3597503.3623316

    [Online]. Available: https://doi.org/10.1145/3597503.3623316

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.