Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact on Machine Translation

Colin Cherry; Daniel Deutsch; Eleftheria Briakou; Jiaming Luo; Markus Freitag; Muhammed Yusuf Kocyigit

REVIEW 4 major objections 6 minor 23 references

Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact on Machine Translation

T0 review · 4 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Test-set leakage inflates translation scores by up to 30 BLEU.

desk verdict First controlled large-scale contamination study for MT at 1B/8B scale; core inflation findings are credible, though the magnitudes need variance estimates and the abstract overstates timing-independence. read the letter →

arxiv 2501.18771 v1 pith:V6G5IIYU submitted 2025-01-30 cs.CL cs.AI

Muhammed Yusuf Kocyigit , Eleftheria Briakou , Daniel Deutsch , Jiaming Luo , Colin Cherry , Markus Freitag This is my paper · ORCID

classification cs.CLcs.AI

keywords datacontaminationmachinetranslationevaluationBLEUpre-trainingLLMmodelscalemultilingualbenchmarkleakage

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether evaluation benchmarks for machine translation overstate real model quality when test sentences accidentally appear in pre-training data. Using a controlled setup that starts from a decontaminated train/test split, the authors inject WMT'23 test examples back into the pre-training mixture under many conditions and measure how BLEU changes. They find that contaminating both source and target sides of test pairs inflates BLEU by up to 30 points for 8B-parameter models, with average inflation roughly 2.5 times larger than for 1B models. Source-only or target-only contamination produces smaller, less consistent overestimates. These results matter because they quantify how much benchmark scores can lie when test data leaks into training corpora.

What carries the argument

The argument is carried by a controlled contamination-injection pipeline built on checkpoint branching. From a single baseline model trained on a decontaminated 325B-token multilingual mixture, the authors continue training from checkpoints taken at 30%, 60%, and 90% of training on data mixtures where test examples are inserted in one of several formats: full prompted source-target pairs, source-only, target-only, or source and target as separate unpaired examples in the same or different batches. An 8-gram subword search with a 70% overlap threshold strips pre-existing test overlap from the evaluation sets before the baseline is trained. By fixing initialization and data order and comparing contaminated checkpoints against the baseline on both contaminated (WMT'23) and non-contaminated (WMT'24) test sets, they isolate the effect of contamination and attribute observed BLEU differences to it.

What would settle it

Train an 8B model from scratch on a mixture containing Full contamination from the first step and compare its WMT'23 BLEU to the branched checkpoint model's final BLEU under the same contamination format; if the from-scratch inflation differs by more than the paper's reported variance, the temporal conclusions and the 30-point ceiling would need revision.

Watch

Extended reading notes

Core claim

The central claim is that data contamination during pre-training inflates machine-translation evaluation scores, and the inflation scales with model size and contamination format. When both source and target of a WMT'23 test example are inserted as a prompted parallel pair, the 8B model's BLEU on that test set rises by as much as 30 points relative to an uncontaminated baseline, while the 1B model's largest rise is about 9 points; averaged across settings, the 8B inflation is 2.5 times the 1B inflation. Contaminating only one side yields small and inconsistent gains, and on non-contaminated WMT'24 sets the improvements largely disappear, confirming the gains are benchmark inflation rather than genuine translation ability. Contamination introduced uniformly across training leaves the largest final footprint, while concentrated early exposure produces sharp temporary spikes that decay; and languages with no representation in pre-training show no measurable contamination effect. The paper argues these patterns hold under both BLEU and the learned metric MetricX.

Load-bearing premise

The checkpoint-branching method assumes that contamination introduced from 30% of training onward produces the same effect as contamination present from the start of pre-training, whereas real web-scale leakage begins at step zero.

Editorial extensions

If this is right

Leaderboard comparisons on MT benchmarks can be distorted by accidental test-set leakage, so published BLEU improvements should be read as upper bounds when contamination status is unknown.
Contamination risk grows with model scale, meaning larger LLMs need stricter data hygiene before evaluation.
Uniformly distributed leakage is the most persistent inflation source, so random shuffling of web-scale data does not neutralize contamination.
Low-resource languages are a safer evaluation ground in the sense that contamination has little effect until the language is represented in pre-training data.
Both string-based and learned metrics show the same inflation pattern, so the effect is not an artifact of BLEU.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the 2.5x scale ratio continues beyond 8B parameters, frontier models could show even larger benchmark inflation, making private hold-out test sets increasingly necessary.
The same controlled-injection methodology could estimate contamination inflation for summarization or code benchmarks, where paired input-output test examples play a similar role.
Because contamination only matters once a language is represented, contamination-detection methods that compare per-language behavior might identify leaked test sets across languages.
The branching design may understate contamination that is present from the very first training step, so the reported magnitudes may be conservative relative to real web-scale leakage.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The short version: this is the first controlled, large-scale pre-training contamination study for machine translation, and the core findings are credible. Full source-target contamination inflates BLEU substantially, more for 8B than 1B models, while source-only or target-only contamination does not, and contamination only bites when the language already has representation in the training data. The zero-resource negative result is clean, and the MetricX appendix corroborates the BLEU numbers.

The design is the real contribution. Decontaminating the test sets first (they found roughly 10% of WMT'23 was already contaminated and dropped those examples), branching from a shared checkpoint to afford 40-plus conditions, and using WMT'24 as a non-contaminated control to separate genuine gains from inflation, that is careful work, and the checkpoint-branching trick is worth borrowing.

Soft spots, in proportion. First, no uncertainty quantification: single seed, single data ordering, no error bars or significance tests in the main text. The authors admit this in Limitations, and the qualitative trends are consistent enough that conclusions are unlikely to flip, but "up to 30 BLEU" and "2.5x" carry more precision than the design supports. Second, checkpoint-branching means contamination always enters after 30% of training; no condition tests from-scratch contamination. That is a real limitation. But the stress-test companion's stronger version does not land: Section 5.2 reports the Uniform condition, which the paper rightly calls the realistic scenario, produces the largest end-of-training inflation, not Late. So the headline magnitude is not an artifact of the unrepresentative late-insertion setting, though the abstract would benefit from one sentence making the timing-dependence explicit. Third, housekeeping: the intro says 42 contamination conditions, the conclusion says 46, and the WMT'24 appendix tables claim confidence intervals that are not actually shown. No code or data are released, which limits independent replication, though that is standard for Google papers. The citation pattern is fine; the n-gram search builds on published work, including Singh et al., legitimately.

This is for anyone working on benchmark validity, contamination detection, or MT evaluation. It deserves serious refereeing. I would send it to review, with the ask that a revision adds variance information and tightens the abstract's conditional language. The core claims should survive.

Referee Report

4 major / 6 minor

Summary. The paper reports a controlled large-scale study of data contamination in pre-training for machine translation, using 1B and 8B decoder-only models trained on a 325B-token multilingual mixture. After decontaminating WMT'23 and WMT'24 test sets, the authors branch from base-model checkpoints and insert test examples under varied formats (full source-target prompts, source-only, target-only), temporal placements (Early/Middle/Late/Uniform), and frequencies (1/10/100 copies). They measure BLEU inflation on contaminated WMT'23 sets and generalize to non-contaminated WMT'24 sets, with MetricX as a robustness check. The main claims are that full source-target contamination substantially inflates BLEU, that this inflation is about 2.5x larger for 8B than for 1B models (up to 30 BLEU points), that partial contamination has smaller and less consistent effects, and that temporal placement and language-resource availability modulate the effect.

Significance. If the claims hold, this is one of the largest controlled studies of pre-training contamination to date, and it goes beyond prior work in several ways: it decontaminates the training/test splits, includes both a contaminated and a non-contaminated test set to separate inflation from genuine gains, spans 13 language pairs and two model scales, and supplements BLEU with MetricX. The checkpoint-branching design is a pragmatic and clearly explained way to make 42-46 contamination conditions feasible at 325B tokens, while reducing variance between contaminated and baseline runs. The study also provides useful evidence on when contamination matters: late-inserted contamination has a larger end-of-training footprint, uniform contamination has the most persistent effect, and near-zero-resource languages show no measurable inflation. The main weaknesses are interpretive: the headline quantitative claims aggregate over temporal windows that the paper itself shows are not interchangeable, and the model-scale comparison rests on absolute BLEU differences that are not scale-invariant.

major comments (4)

[Section 3.4, Section 5.2, Abstract] The headline magnitudes—'up to 30 BLEU points' and '2.5× larger' for 8B versus 1B—are aggregated over the Early/Middle/Late/Uniform contamination windows defined in Table 2. Because all conditions branch from a partially trained checkpoint (never before 30% of training), and because Section 5.2 itself reports that later contamination leaves a larger end-of-training footprint and that extended pre-training can exaggerate contamination's impact, the abstract's numbers may be driven by the least realistic Late condition. Please report the headline statistics separately for each temporal window (at minimum for Uniform, which the paper describes as more realistic), and state explicitly in the abstract and Section 5 that the measurement protocol is checkpoint-branching from 30-90% of training rather than contamination present from step zero. Without this qualification, the quantitative central claim overstates what is established about from-scratch pre-training contamination.
[Section 5, Figure 2] The '2.5×' inflation ratio is presented as absolute BLEU-point differences between 1B and 8B models, but Section 6 correctly notes that BLEU differences are not comparable across models operating on different BLEU scales. Since the 8B baselines are substantially higher (e.g., Table 15, EN-DE baseline 30.95 for 8B versus 21.71 for 1B), a fixed absolute gain represents a smaller relative gain. The paper should also express the 2.5× claim as a relative or percentage improvement, or as BLEU differences normalized by baseline, and should accompany the absolute comparison with this caveat.
[Section 5.2, Figure 4] The main-text evidence for the statement that 'uniform contamination tends to yield the highest final performance gains' is a single language pair (German-English, 8B, Full, 100 copies). The reader is referred to Appendix C, but the main text should state whether the uniform-window ordering is consistent across language pairs, copy counts, and model sizes, and should quantify the variability shown in Appendix C figures. If the ordering is not consistent, the sentence should be hedged accordingly.
[Section 5.1, Figure 3, Abstract] The quantitative claims in the abstract and main text are not fully consistent: Figure 3 reports improvements on contaminated datasets 'up to 26 BLEU points' larger than on non-contaminated sets, the abstract quotes 'up to 30 BLEU points', and Section 5.2 reports transient spikes 'up to 60 BLEU points'. The paper should make clear which condition produces each number, and whether '30' refers to raw inflation on WMT'23 or to an improvement difference between WMT'23 and WMT'24.

minor comments (6)

[Abstract and Section 7] The abstract states 42 contamination conditions, while the conclusion states 46; please make the count consistent and provide a breakdown of how conditions are counted.
[Section 3.2] The phrase 'casual language modelling' should be 'causal language modelling'.
[Table 3] The column header 'Data phantomphantompha' appears to be a formatting artifact; it should read 'Data' or 'Data Source'.
[Section 5.5 and Appendix D] The language name is spelled inconsistently as 'Achenese', 'Achinese', and 'Acehnese'; please standardize the spelling.
[Appendix E, Tables 6-11] The table captions say 'with Confidence Intervals', but the tables show only point estimates; please either add the intervals or remove the phrase from the captions.
[Section 6] The limitation discussion mentions checkpoint-level variance and single initialization, but it does not explicitly note that checkpoint-branching means no contamination from step zero; adding this to the limitations would address the main interpretive concern directly.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the contamination-inflation claims are measured outcomes from a controlled comparison, not derived from or equivalent to the paper's inputs.

full rationale

This is a controlled empirical study, not a derivation, so none of the paper's central claims reduces to its inputs by construction. The primary finding—that contaminating pre-training data with both source and target of WMT'23 test pairs inflates BLEU, with larger inflation for the 8B model—is directly measured by comparing a decontaminated baseline with checkpoint branches that continue training on a mixture where test examples are inserted. The n-gram decontamination step (Section 3.1, Appendix G) defines which test examples had prior overlap, but the reported BLEU differences are experimental outcomes, not consequences of that definition. No parameter is fitted to the reported inflations, and no 'prediction' is derived from a fitted model. Self-citations (Singh et al. 2024 for the longest-match procedure, Briakou et al. 2023 for parallel-data utility, Juraska et al. 2023 and Freitag et al. 2023 for MetricX) support methodology or metric choice but are not load-bearing for the contamination-effect claim; even if those citations were removed, the controlled comparison would stand. The paper itself flags a genuine interpretation caveat in Section 5.2: later contamination has a bigger end-of-training footprint, and the checkpoint-branching design only inserts contamination from 30% of training onward (Section 3.4, Table 2), so headline magnitudes such as 'up to 30 BLEU' and the '2.5x' ratio aggregate over temporal windows of varying realism. That is a limitation relevant to generalization, not circularity. No step reduces to its own inputs, so the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims are measurement claims rather than derivations. The free parameters listed are experimental design choices and thresholds that influence the quantitative results, but the qualitative direction of the findings (full contamination inflates scores more than partial, larger models inflate more) is not fitted to the outcome; it is directly observed in the controlled comparisons.

free parameters (4)

Decontamination overlap threshold = 0.7
Chosen from Chowdhery et al. (2022) to label test examples as contaminated if the longest matching n-gram covers >70% of source or target tokens. This threshold determines which examples are removed from the test sets and shapes all downstream measurements.
N-gram length for contamination search = 8
The paper searches for matches using 8-gram sub-word tokens. The choice affects sensitivity of contamination detection and thus which examples are considered contaminated.
Data mixture ratios = 60% Dolma, 35% Madlad, 5% parallel
The pre-training mixture is downsampled to 325B tokens with these ratios; the authors keep the same data for both model sizes to isolate scale. These ratios influence baseline translation ability and therefore contamination impact.
Contamination temporal windows = Early at 30%, Middle at 60%, Late at 90%, Uniform between 30-90% of training
The set of contamination insertion points is chosen by the authors; the results on temporal dependence are defined relative to these windows and may not capture contamination before 30% of training.

assumptions (4)

domain assumption The 8-gram overlap search with a 0.7 threshold correctly identifies all meaningful training-test contamination.
The paper removes test examples whose longest matching subsequence exceeds 70% of tokens, following Chowdhery et al. (2022). If this catches too little, the baseline is contaminated; if too much, the test set is biased. No validation of the threshold is provided for the multiple languages.
domain assumption Checkpoint-branching from a 30%-trained baseline approximates the effect of contamination present from the start of pre-training.
All contamination is introduced after at least 30% of training has completed, by continuing from a baseline checkpoint. This may not capture interactions between early training dynamics and contamination, though the paper argues it reduces variance.
domain assumption BLEU and MetricX score changes are meaningful indicators of translation quality for the language pairs studied.
The paper relies on BLEU as the main metric and MetricX as a learned metric. If these metrics mis-rank translations for low-resource or non-English languages, the measured inflation could be an artifact.
domain assumption WMT'24 test sets are valid non-contaminated counterparts to WMT'23 for measuring generalization.
The paper uses WMT'24 for five language pairs to distinguish contamination inflation from true capability gains. This assumes WMT'24 is a comparable and clean benchmark, which the authors decontaminate but do not otherwise control.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact on Machine Translation." pith.science (2026). https://pith.science/paper/V6G5IIYU

@misc{pith2026250118771,
  author       = {Pith},
  title        = {Pith review of: Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact on Machine Translation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V6G5IIYU}},
  note         = {Machine review of arXiv:2501.18771}
}

read the original abstract

Data contamination -- the accidental consumption of evaluation examples within the pre-training data -- can undermine the validity of evaluation benchmarks. In this paper, we present a rigorous analysis of the effects of contamination on language models at 1B and 8B scales on the machine translation task. Starting from a carefully decontaminated train-test split, we systematically introduce contamination at various stages, scales, and data formats to isolate its effect and measure its impact on performance metrics. Our experiments reveal that contamination with both source and target substantially inflates BLEU scores, and this inflation is 2.5 times larger (up to 30 BLEU points) for 8B compared to 1B models. In contrast, source-only and target-only contamination generally produce smaller, less consistent over-estimations. Finally, we study how the temporal distribution and frequency of contaminated samples influence performance over-estimation across languages with varying degrees of data resources.

Figures

Figures reproduced from arXiv: 2501.18771 by the authors.

**Figure 1.** Large-scale contamination analysis setup: We decontaminate our train-test splits and train a baseline model. Then, insert test data into the pre-training data and train a contaminated model branching out from the baseline checkpoint. Finally, we compare the relative performance of the contaminated and the baseline model on contaminated and non-contaminated data. • Contaminating source-target MT pairs inflates perfo… view at source ↗

**Figure 2.** Box plot of BLEU differences of contaminated vs. uncontaminated models across WMT’23 language-pairs, for 1B (left) and 8B (rights) model sizes. Contaminating paired source-target instances (full) consistently inflates translation performance across languages, with larger effects on the 8B model. Source-only and target-only contamination does not inflate performance consistently. hover around 9 BLEU points, while for… view at source ↗

**Figure 4.** BLEU score throughout training for German to English in WMT’23 for the 8B model, Full contamination and 100 Copies. Earlier contamination causes larger performance peaks, while later contamination causes lower spikes but higher eventual performance gaps. Uniform contamination tends to yield the highest final performance gains and no sharp peaks. within extended pre-training or fine-tuning settings exaggerates its … view at source ↗

Figures from the paper (17 more)

**Figure 3.** Figure 3: Box plot of BLEU improvement differences of contaminated vs. uncontaminated models between WMT’23 - WMT’24. Contaminating source-targe examples yields higher performance “improvements” on contaminated vs. non-contaminated datasets. datasets are up to 26 BLEU points la…

**Figure 6.** Figure 6: compares these two settings with the prompted format (Source and Target, Prompted also named Full) and the case where we only contaminate the target text which we add as an additional reference point. Comparing with these baselines, we see that both the Split and Batch…

**Figure 7.** Figure 7: illustrates the percentage BLEU score improvements grouped by out and into English translation directions (En→X and X→En, respectively). As shown, contamination has a more significant impact on the En→X translation direction compared to X→En, for all Full contaminati…

**Figure 8.** Figure 8: Box plot of absolute MetricX (lower is better) improvements for all 10 WMT’23 Language Pairs for 1B and 8B Model. Notice the scales of the X-axis is different for different model sizes. The methods on the Y-axis are sorted based on the mean improvement for the 1B model…

**Figure 9.** Figure 9: MetricX (lower is better) score throughout training for German to English in WMT’23 for the 8B model, Full contamination and 100 Copies [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]

**Figure 11.** Figure 11: Average percentage MetricX score improvement for two performance groups. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]

**Figure 10.** Figure 10: Percent improvement for different contamination methods for increasing number of copies. Dotted lines are the percentage improvements per language pair in WMT’23. The solid lines are the mean improvement per method [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]

**Figure 12.** Figure 12: The performance improvement right after the contamination and performance improvement at the end of training for WMT23 language pairs. We show that our observations in Section 5.2 hold true in general for all WMT’23 langauge pairs and number of copies. 15 [PITH_FULL_…

**Figure 13.** Figure 13: The performance improvement right after the contamination and performance improvement at the end of training for WMT24 language pairs. In this context we can read this plot in the context of adding high-quality data into the pre-training mixture to see how it improves…

**Figure 14.** Figure 14: BLEU scores through training for 100 Copies of Full contamination 18 [PITH_FULL_IMAGE:figures/full_fig_p018_14.png]

**Figure 15.** Figure 15: BLEU scores through training for 10 Copies of Full contamination 19 [PITH_FULL_IMAGE:figures/full_fig_p019_15.png]

**Figure 16.** Figure 16: BLEU scores through training for 1 Copies of Full contamination 20 [PITH_FULL_IMAGE:figures/full_fig_p020_16.png]

**Figure 17.** Figure 17: WMT’24 BLEU scores through training for 100 Copies of Full contamination [PITH_FULL_IMAGE:figures/full_fig_p022_17.png]

**Figure 18.** Figure 18: WMT’24 BLEU scores through training for 10 Copies of Full contamination 22 [PITH_FULL_IMAGE:figures/full_fig_p022_18.png]

**Figure 19.** Figure 19: WMT’24 BLEU scores through training for 1 Copies of Full contamination 23 [PITH_FULL_IMAGE:figures/full_fig_p023_19.png]

**Figure 20.** Figure 20: Riverplot of clean, contaminated and different forms of contamination for the threshold 0.7. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_20.png]

**Figure 21.** Figure 21: Histogram of contamination scores. Here the scores are calculates as max(ssource, starget) where ssource is the percentage of overlapping tokens with the longest contamination that contains any tokens from the source field and starget similarly for target. 26 [PITH_F…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 4 canonical work pages

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
[2]

M., Pombal, J., Guerreiro, N

Alves, D. M., Pombal, J., Guerreiro, N. M., Martins, P. H., Alves, J., Farajian, A., Peters, B., Rei, R., Fernandes, P., Agrawal, S., Colombo, P., de Souza, J. G. C., and Martins, A. F. T. Tower: An open multilingual large language model for translation-related tasks, 2024. URL https://arxiv.org/abs/2402.17733

arXiv 2024
[3]

Briakou, E., Cherry, C., and Foster, G. F. Searching for needles in a haystack: On the role of incidental bilingualism in palm’s translation capability. In Annual Meeting of the Association for Computational Linguistics, 2023. URL https://api.semanticscholar.org/CorpusID:258740723

work page 2023
[4]

Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer, N., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B., Pope, R., Bradbury, J., Austin, J., Isard, M., Gur-Ari, G., Yin, P., Duke, T., Levska...

arXiv 2022
[5]

Results of WMT 23 metrics shared task: Metrics might be guilty but references are not innocent

Freitag, M., Mathur, N., Lo, C.-k., Avramidis, E., Rei, R., Thompson, B., Kocmi, T., Blain, F., Deutsch, D., Stewart, C., Zerva, C., Castilho, S., Lavie, A., and Foster, G. Results of WMT 23 metrics shared task: Metrics might be guilty but references are not innocent. In Koehn, P., Haddow, B., Kocmi, T., and Monz, C. (eds.), Proceedings of the Eighth Conf...

work page doi:10.18653/v1/2023.wmt-1.5 2023
[6]

The F lores-101 evaluation benchmark for low-resource and multilingual machine translation

Goyal, N., Gao, C., Chaudhary, V., Chen, P.-J., Wenzek, G., Ju, D., Krishnan, S., Ranzato, M., Guzm \'a n, F., and Fan, A. The F lores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computational Linguistics, 10: 0 522--538, 2022. doi:10.1162/tacl_a_00474. URL https://aclanthology.org/20...

doi:10.1162/tacl_a_00474 2022
[7]

Z., Zhong, M., Schaeffer, R., Ouyang, S., Han, J., and Koyejo, S

Jiang, M., Liu, K. Z., Zhong, M., Schaeffer, R., Ouyang, S., Han, J., and Koyejo, S. Investigating data contamination for pre-training language models, 2024. URL https://arxiv.org/abs/2401.06059

arXiv 2024
[8]

M etric X -23: The G oogle submission to the WMT 2023 metrics shared task

Juraska, J., Finkelstein, M., Deutsch, D., Siddhant, A., Mirzazadeh, M., and Freitag, M. M etric X -23: The G oogle submission to the WMT 2023 metrics shared task. In Koehn, P., Haddow, B., Kocmi, T., and Monz, C. (eds.), Proceedings of the Eighth Conference on Machine Translation, pp.\ 756--767, Singapore, December 2023. Association for Computational Lin...

doi:10.18653/v1/2023.wmt-1.63 2023

Show all 23 references

[9]

Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014. URL https://api.semanticscholar.org/CorpusID:6628106

2014 arXiv
[10]

Findings of the 2023 conference on machine translation ( WMT 23): LLM s are here but not quite there yet

Kocmi, T., Avramidis, E., Bawden, R., Bojar, O., Dvorkovich, A., Federmann, C., Fishel, M., Freitag, M., Gowda, T., Grundkiewicz, R., Haddow, B., Koehn, P., Marie, B., Monz, C., Morishita, M., Murray, K., Nagata, M., Nakazawa, T., Popel, M., Popovi \'c , M., and Shmatova, M. F...

2023
[11]

Findings of the WMT 24 general machine translation shared task: The LLM era is here but MT is not solved yet

Kocmi, T., Avramidis, E., Bawden, R., Bojar, O., Dvorkovich, A., Federmann, C., Fishel, M., Freitag, M., Gowda, T., Grundkiewicz, R., Haddow, B., Karpinska, M., Koehn, P., Marie, B., Monz, C., Murray, K., Nagata, M., Popel, M., Popovi \'c , M., Shmatova, M., Steingr \' msson, ...

2024
[12]

Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing

Kudo, T. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226, 2018

2018 arXiv
[13]

Madlad-400: a multilingual and document-level large audited dataset

Kudugunta, S., Caswell, I., Zhang, B., Garcia, X., Xin, D., Kusupati, A., Stella, R., Bapna, A., and Firat, O. Madlad-400: a multilingual and document-level large audited dataset. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIP...

2024
[14]

and Schwartz, R

Magar, I. and Schwartz, R. Data contamination: From memorization to exploitation, 2022. URL https://arxiv.org/abs/2203.08242

2022 arXiv
[15]

Oren, Y., Meister, N., Chatterji, N., Ladhak, F., and Hashimoto, T. B. Proving test set contamination in black box language models, 2023. URL https://arxiv.org/abs/2310.17623

2023 arXiv
[16]

B leu: a method for automatic evaluation of machine translation

Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. B leu: a method for automatic evaluation of machine translation. In Isabelle, P., Charniak, E., and Lin, D. (eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp.\ 311--318, Philad...

2002
[17]

Sainz, O., García-Ferrero, I., Jacovi, A., Campos, J. A., Elazar, Y., Agirre, E., Goldberg, Y., Chen, W.-L., Chim, J., Choshen, L., D'Amico-Wong, L., Dell, M., Fan, R.-Z., Golchin, S., Li, Y., Liu, P., Pahwa, B., Prabhu, A., Sharma, S., Silcock, E., Solonko, K., Stap, D., Surd...

2024 arXiv
[18]

Detecting pretraining data from large language models, 2024

Shi, W., Ajith, A., Xia, M., Huang, Y., Liu, D., Blevins, T., Chen, D., and Zettlemoyer, L. Detecting pretraining data from large language models, 2024. URL https://arxiv.org/abs/2310.16789

2024 arXiv
[19]

K., Kocyigit, M

Singh, A. K., Kocyigit, M. Y., Poulton, A., Esiobu, D., Lomeli, M., Szilvasy, G., and Hupkes, D. Evaluation data contamination in llms: how do we measure it and (when) does it matter?, 2024. URL https://arxiv.org/abs/2411.03923

2024 arXiv
[20]

Dolma: an open corpus of three trillion tokens for language model pretraining research

Soldaini, L., Kinney, R., Bhagia, A., Schwenk, D., Atkinson, D., Authur, R., Bogin, B., Chandu, K., Dumas, J., Elazar, Y., Hofmann, V., Jha, A., Kumar, S., Lucy, L., Lyu, X., Lambert, N., Magnusson, I., Morrison, J., Muennighoff, N., Naik, A., Nam, C., Peters, M., Ravichander,...

2024
[21]

Attention is all you need

Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017

2017
[22]

E., and Stoica, I

Yang, S., Chiang, W.-L., Zheng, L., Gonzalez, J. E., and Stoica, I. Rethinking benchmark and contamination for language models with rephrased samples, 2023. URL https://arxiv.org/abs/2311.04850

2023 arXiv
[23]

X., Chen, X., Lin, Y., Wen, J.-R., and Han, J

Zhou, K., Zhu, Y., Chen, Z., Chen, W., Zhao, W. X., Chen, X., Lin, Y., Wen, J.-R., and Han, J. Don't make your llm an evaluation benchmark cheater, 2023. URL https://arxiv.org/abs/2311.01964

2023 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

[2] [2]

M., Pombal, J., Guerreiro, N

Alves, D. M., Pombal, J., Guerreiro, N. M., Martins, P. H., Alves, J., Farajian, A., Peters, B., Rei, R., Fernandes, P., Agrawal, S., Colombo, P., de Souza, J. G. C., and Martins, A. F. T. Tower: An open multilingual large language model for translation-related tasks, 2024. URL https://arxiv.org/abs/2402.17733

arXiv 2024

[3] [3]

Briakou, E., Cherry, C., and Foster, G. F. Searching for needles in a haystack: On the role of incidental bilingualism in palm’s translation capability. In Annual Meeting of the Association for Computational Linguistics, 2023. URL https://api.semanticscholar.org/CorpusID:258740723

work page 2023

[4] [4]

Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer, N., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B., Pope, R., Bradbury, J., Austin, J., Isard, M., Gur-Ari, G., Yin, P., Duke, T., Levska...

arXiv 2022

[5] [5]

Results of WMT 23 metrics shared task: Metrics might be guilty but references are not innocent

Freitag, M., Mathur, N., Lo, C.-k., Avramidis, E., Rei, R., Thompson, B., Kocmi, T., Blain, F., Deutsch, D., Stewart, C., Zerva, C., Castilho, S., Lavie, A., and Foster, G. Results of WMT 23 metrics shared task: Metrics might be guilty but references are not innocent. In Koehn, P., Haddow, B., Kocmi, T., and Monz, C. (eds.), Proceedings of the Eighth Conf...

work page doi:10.18653/v1/2023.wmt-1.5 2023

[6] [6]

The F lores-101 evaluation benchmark for low-resource and multilingual machine translation

Goyal, N., Gao, C., Chaudhary, V., Chen, P.-J., Wenzek, G., Ju, D., Krishnan, S., Ranzato, M., Guzm \'a n, F., and Fan, A. The F lores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computational Linguistics, 10: 0 522--538, 2022. doi:10.1162/tacl_a_00474. URL https://aclanthology.org/20...

doi:10.1162/tacl_a_00474 2022

[7] [7]

Z., Zhong, M., Schaeffer, R., Ouyang, S., Han, J., and Koyejo, S

Jiang, M., Liu, K. Z., Zhong, M., Schaeffer, R., Ouyang, S., Han, J., and Koyejo, S. Investigating data contamination for pre-training language models, 2024. URL https://arxiv.org/abs/2401.06059

arXiv 2024

[8] [8]

M etric X -23: The G oogle submission to the WMT 2023 metrics shared task

Juraska, J., Finkelstein, M., Deutsch, D., Siddhant, A., Mirzazadeh, M., and Freitag, M. M etric X -23: The G oogle submission to the WMT 2023 metrics shared task. In Koehn, P., Haddow, B., Kocmi, T., and Monz, C. (eds.), Proceedings of the Eighth Conference on Machine Translation, pp.\ 756--767, Singapore, December 2023. Association for Computational Lin...

doi:10.18653/v1/2023.wmt-1.63 2023