Pith. sign in

REVIEW 3 major objections 4 minor 54 references

Zero Gap Is Not Restoration: Stratified Per-Question Probability Evaluation and Step-wise Mitigation of Benchmark Contamination

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Zero aggregate performance gap is not proof of restoration; stratified per-question probabilities overturn prior mitigation verdicts, and a step-wise decoding cap wins.

desk verdict A clean metric contribution that deserves review, with empirical claims that rest on a strong synthetic contamination setup. read the letter →

arxiv 2608.07341 v1 pith:AFK7QWKY submitted 2026-08-07 cs.CL cs.AI

Ruijie Hou , Yueyang Jiao , Zhao Wang , Yingming Li This is my paper · ORCID
classification cs.CLcs.AI
keywords datacontaminationbenchmarkleakageevaluationmetricssolveprobabilityper-questiongapdecodinginterventionmemorizationsuppressionRailCap
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Benchmark leakage inflates scores through memorization, and mitigation strategies that suppress memorization are judged by how close their aggregate accuracy comes to the clean model's. The paper argues that this G-AP metric is structurally wrong: a 0/1 readout cannot represent per-question performance, and averaging before differencing lets over- and under-suppression cancel, so a zero gap can coexist with zero restoration. It replaces the readout with each question's solve probability, differences per question, and stratifies by the clean model's solve probability to block trivial all-fail strategies, giving SA-PPG. On the mitigation side it proposes RailCap, which caps the greedy-trajectory token's logit to the runner-up whenever sampling falls back onto the greedy path, so suppression is decided online step by step. The payoff claimed is concrete: prior strategies' restoration is substantially overestimated, and RailCap achieves the lowest SA-PPG across six model-by-domain settings.

What carries the argument

The load-bearing object is the per-question probability gap, $\Delta_s(q) = r^{\mathrm{prob}}_{M_{\mathrm{co}}^s}(q) - r^{\mathrm{prob}}_{M_{\mathrm{cl}}}(q)$, where $r^{\mathrm{prob}}$ is the solve probability estimated from repeated sampling. A-PPG averages $|\Delta_s(q)|$ over questions, and SA-PPG first averages inside bins defined by the clean model's solve probability, then across bins, so a strategy cannot score well by pushing every question to the clean model's most frequent probability value. The other load-bearing mechanism is RailCap's cap: once the last $n$ generated tokens match a window of the contaminated model's greedy trajectory, the trajectory successor's logit is replaced by $\min(\ell_t[x], v_t^{(2)})$, the runner-up's logit, so the clean model's next likely choice becomes samplable and suppression accumulates across steps. This turns contamination handling from a pre-hoc estimate into online, per-step supervision of the greedy fall-back signal.

What would settle it

Take a model whose pretraining corpus is public and verify that a held-out benchmark like GSM8K or a paraphrased variant actually appears in its training data, then apply Identity, TED, LNE-blocking, shortcut-neuron patching, and RailCap and compute SA-PPG with the same $m=50$, $T=0.7$, $B=50$ protocol. If any estimate-based strategy beats RailCap on that naturally contaminated model, the paper's claim that restoration is 'substantially overestimated' and that RailCap is best would be falsified. A cheaper check is to test whether the greedy fall-back signal of Figure 1 appears on naturally contaminated questions at all.

Watch

Extended reading notes

Core claim

The paper's central claim is that 'zero gap is not restoration.' Under the prevailing G-AP metric, LNE-blocking looks near-perfect (0.0235 on Llama-2/GSM8K) while RailCap looks worse (0.0728), yet under SA-PPG the order reverses: RailCap is best (0.1914) and LNE-blocking (0.2932) is barely better than no intervention (0.3261). The same responses, the same models, and the same strategies change verdict when discrete averaged readouts are replaced by per-question solve probabilities and stratified aggregation. The paper also claims that existing strategies—TED, LNE-blocking, shortcut-neuron patching—are estimate-then-operate methods whose correctness is limited by a one-shot contamination estimate, whereas RailCap's step-wise greedy-trajectory cap needs no estimate and attains the lowest SA-PPG in all six settings tested. The takeaway, stated in the paper's own terms, is that restoration should be judged question by question and intervention should be supervised during generation.

Load-bearing premise

The load-bearing premise is that contaminating a model by additional fine-tuning on the benchmark's own test questions faithfully represents the contamination a model acquires during large-scale pretraining; if real memorization differs in scale or form, the measured overestimation and RailCap's advantage may not transfer.

Editorial extensions

If this is right

  • Every prior mitigation result reported as a G-AP gap needs to be re-checked with per-question probability gaps; a small G-AP is no longer evidence that a strategy restored the model.
  • Mitigation strategies should be designed around online signals such as fall-back onto the greedy trajectory instead of one-shot estimates of where contamination lies.
  • Under SA-PPG, an all-fail strategy is exposed as worse than no intervention, so the trivial-strategy loophole in aggregate metrics is closed.
  • In the six settings tested across three models and both verbatim and paraphrased contamination, RailCap attains the lowest SA-PPG, with LNE-blocking performing worse than identity on the paraphrased domain.
  • Paraphrased contamination is harder for estimate-based strategies, and RailCap's step-wise supervision is the only method that does not lose ground there.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I would test whether the greedy fall-back cap transfers to natural contamination by taking a model whose public pretraining corpus provably contains a benchmark's test split and running the same six-way comparison; if a one-shot estimator wins there, the synthetic fine-tuning setup is the reason RailCap leads.
  • SA-PPG's stratification depends on choosing a number of bins and on having a trustworthy clean reference; a natural next step is a sensitivity analysis over bin counts and a reference-free approximation for models with no clean counterpart.
  • The same per-question probability-gap logic could evaluate decoding interventions beyond contamination, such as detoxification or privacy filters, where the goal is also to match a reference behavior question by question rather than on average.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper addresses the evaluation and mitigation of benchmark contamination in language models. It argues that the prevailing metric, G-AP (Gap of Aggregate Performance), is flawed because it uses discrete per-question readouts and averages before differencing, so over- and under-suppression can cancel out. The authors propose A-PPG (Aggregate of Per-question Probability Gaps) and its stratified variant SA-PPG (Stratified Aggregate of Per-question Probability Gaps), which estimate each question's solve probability by sampling, difference it against a clean model per question, and aggregate within groups defined by the clean model's solve probability. They also propose RailCap, a decoding-time mitigation strategy that caps the probability of the next greedy-trajectory token to the runner-up whenever a sample falls back onto the greedy trajectory. The experimental section simulates contamination by further LoRA fine-tuning on test-split questions, and claims that SA-PPG reveals prior strategies' restoration to be substantially overestimated and that RailCap attains the lowest SA-PPG in all six tested settings. The mathematical definitions and the zero-set argument for A-PPG are correct, and the decomposition in Table 2 is instructive, but the empirical generalization rests on a single synthetic contamination regime and on point estimates without uncertainty quantification.

Significance. The paper's theoretical contribution is solid and clearly presented: the definitions of A-PPG and SA-PPG are self-consistent, the inequality G-APP ≤ A-PPG is proven, and the zero-set argument in Eq. (6) correctly identifies why aggregate gaps can vanish without per-question restoration. The decomposition into under- and over-suppression in Table 2 is a useful diagnostic, and the All-Zero example convincingly illustrates the failure of equal weighting. RailCap is a novel decoding-time strategy that is well motivated by the two observations in Figure 1. If the empirical claims held beyond the controlled simulation, the paper would materially improve how contamination mitigation is evaluated. At present, however, the quantitative conclusions are established only for a synthetic contamination regime, and the ranking claims lack uncertainty quantification; therefore the significance is conditional on additional evidence.

major comments (3)
  1. [§5.1, Tables 1 and 3] The empirical generalization that G-AP substantially overestimates prior restoration and that RailCap attains the lowest SA-PPG rests entirely on a synthetic contamination simulation: Mco is obtained by five-epoch LoRA fine-tuning of Mcl on 660 test-split questions in the 8-shot CoT format mixed with 1,840 OpenOrca examples. This is a strong, homogeneous contamination regime, and Figure 1 demonstrates the greedy-fallback signal for Llama-2 on GSM8K only. No naturally contaminated model, no duplication-frequency sweep, and no variation of contamination strength is reported. Since the magnitudes of the overestimation (Table 1: LNE-blocking 0.0235 vs. SA-PPG 0.2932) and the six-way ranking (Table 3) may depend on this regime, the paper should either restrict its claims to the controlled setting or add experiments varying contamination strength and repetition frequency, preferably including a naturally contaminated model. Without such evidence, the central empirical conclusions are not established beyond the synthetic setup.
  2. [§5.2.1, Table 1] The sentence 'changing only the metric overturns the verdict entirely' overstates what Table 1 shows: the G-AP column uses a single 0/1 observation per question, while the SA-PPG column uses m=50 samples, so the readout estimator and the aggregation change simultaneously. Moreover, the caption's phrase 'Readings of the same responses' is inconsistent with the protocol in §5.1, where the G-AP readout uses one sample and SA-PPG uses fifty. The aggregation-only comparison in Table 2 (G-APP 0.0207 vs. SA-PPG 0.2932 for LNE-blocking) is the correct evidence for the rank-reversal claim, and the text should be revised to present Table 2 as the primary support for that claim.
  3. [§5.3.1, Tables 1–3] The strategy-ranking claims are reported as point estimates with no uncertainty quantification. All SA-PPG values are derived from m=50 samples per question in a single run, and no bootstrap confidence intervals, multiple seeds, or significance tests are provided. Since some reported differences are small (e.g., Pythia × GSM8K: RailCap 0.1648 vs. Shortcut 0.1708), the reader cannot determine whether the six-way win is robust. The authors should report error bars or repeated-seed intervals for the metric values in Tables 1–3 before drawing strong comparative conclusions.
minor comments (4)
  1. [§3.3] The zero-set property of SA-PPG is stated for the true solve probability rprob, but in practice this quantity is estimated from m=50 samples; the paper should note that finite-sample SA-PPG is a noisy estimator and that the exact per-question restoration guarantee holds only in the population limit.
  2. [§4, Eq. (14)] The n-gram index H is built as a dictionary from a trailing n-gram to its successor; if the same n-gram appears more than once on a greedy trajectory with different successors, the later entry overwrites the earlier one. The paper should state how such repeated windows are handled, since this affects the cap applied at those steps.
  3. [§5.1] The choices B=50 for SA-PPG and T=0.7 for sampling are not justified or varied; a sensitivity analysis for B and m, or at least a brief discussion, would strengthen the practical recommendations of the metric.
  4. [Figure 1] In panels (b) and (d), the y-axis label 'share' is not defined; the caption should state whether the shares are computed over forking steps or over questions, and clarify the denominator for each rank category.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SA-PPG and RailCap are independently defined, and the paper's only self-citation is a baseline that it explicitly falsifies.

full rationale

SA-PPG is defined in Eqs. (3)-(5) and (12)-(13) as a per-question probability gap aggregated within bins of the clean model's solve probability, while RailCap is defined in Eqs. (14)-(15) as an online decoding rule that caps a trajectory token to the runner-up whenever an n-gram window of the contaminated model's own greedy trajectory recurs. Neither definition invokes the other. The claim that G-AP overestimates restoration is supported by the algebraic decomposition in Eqs. (8)-(11): since G-APP = |Delta+ - Delta-| while A-PPG = Delta+ + Delta-, cancellation is a mathematical property of averaging before differencing, not an input relabeled as a result. The paper's self-citation of prior work, LNE-blocking (Hou et al., 2025), is used as a baseline and is explicitly shown to be misleading under SA-PPG, which is the opposite of load-bearing self-support. The RailCap n-gram threshold is tuned and reported as an ablation (Table 4) with a robustness range, not disguised as a prediction. The main weakness is external validity: contamination is simulated by 5-epoch LoRA fine-tuning on 660 test-split questions (Section 5.1), so natural contamination may behave differently; that is a correctness concern, not a definitional circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claims rest on a small number of hand-set evaluation parameters (n, B, m, T) and on the fidelity of the synthetic contamination setup. No new physical or conceptual entities are postulated; the only invented constructs are the metric SA-PPG and the algorithm RailCap, which are methods rather than entities.

free parameters (4)
  • n-gram threshold n = 4
    n-gram threshold for RailCap's fall-back trigger; selected as best on Llama-2 GSM8K ablation (Table 4) and used for all main results.
  • number of strata B = 50
    Number of equal-width bins in SA-PPG (Eq. 13); chosen without principled justification; results may depend on it.
  • sampling count m = 50
    Samples per question for solve-probability estimates; setting chosen for the experiments; no sensitivity or error bars reported.
  • sampling temperature T = 0.7
    Sampling temperature for all strategies; adopted from prior protocols; all methods compared at same value.
assumptions (5)
  • domain assumption Fine-tuning on the test split (LoRA, 5 epochs) simulates real pretraining contamination.
    Section 5.1; no validation against naturally contaminated models.
  • domain assumption A clean model of the same lineage (trained without test leakage) is available and is the reference for genuine capability.
    SA-PPG is defined relative to Mcl; in practice such a model may not exist.
  • domain assumption The greedy trajectory of the contaminated model is the memorized path; falling back into it indicates memorization.
    Figure 1 observation for Llama-2 GSM8K; used as RailCap trigger for all models.
  • domain assumption The clean model's selected token at a divergence step is frequently the contaminated model's runner-up token.
    Figure 1b,d; motivates capping to the runner-up; not verified per model.
  • standard math Sample proportion of correct responses estimates the true per-question solve probability (law of large numbers).
    Used in Eq. 3 and the estimator rhat; assumed large enough m.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Zero Gap Is Not Restoration: Stratified Per-Question Probability Evaluation and Step-wise Mitigation of Benchmark Contamination." pith.science (2026). https://pith.science/paper/AFK7QWKY

@misc{pith2026260807341,
  author       = {Pith},
  title        = {Pith review of: Zero Gap Is Not Restoration: Stratified Per-Question Probability Evaluation and Step-wise Mitigation of Benchmark Contamination},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AFK7QWKY}},
  note         = {Machine review of arXiv:2608.07341}
}
read the original abstract

Test data from public benchmarks inevitably leaks into pretraining corpora, inflating evaluation scores once memorized. \textbf{Contamination mitigation evaluation} intervenes in the decoding process to suppress memorization and restore a contaminated model's genuine capability, but its prevailing metric, the \textbf{G-AP} (\textbf{G}ap of \textbf{A}ggregate \textbf{P}erformance), is flawed. Discrete correct/incorrect readouts cannot characterize per-question performance, averaging before differencing lets over- and under-suppression cancel out, and uniform per-question weighting invites strategies to push solve probabilities onto the clean model's high-frequency values. We propose \textbf{SA-PPG} (\textbf{S}tratified \textbf{A}ggregate of \textbf{P}er-question \textbf{P}robability \textbf{G}aps): estimate each question's solve probability by sampling, difference it against the clean model per question, and aggregate within groups defined by the clean model's solve probability. Existing mitigation strategies first estimate where contamination lies and then operate on the estimate, so they are only as correct as the estimate. \textbf{RailCap} instead judges contamination during generation: whenever a sample falls back onto the greedy trajectory, the next trajectory token is capped to the runner-up, accumulating suppression until the response distribution becomes sufficiently dispersed. Across multiple contaminated models and benchmarks, SA-PPG reveals that prior strategies' restoration is substantially overestimated, while RailCap attains the lowest SA-PPG.

Figures

Figures reproduced from arXiv: 2608.07341 by the authors.

Figure 1
Figure 1. The contaminated model’s generation behaviour on Llama-2 (GSM8K, 1,319 questions, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Per-question readouts of two independent sampling batches drawn from the same clean model [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 21 canonical work pages

  1. [1]

    LNE -Blocking: An Efficient Framework for Contamination Mitigation Evaluation on Large Language Models

    Hou, Ruijie and Jiao, Yueyang and Hu, Hanxu and Li, Yingming and Lam, Wai and Zhang, Huajian and Lu, Hongyuan. LNE -Blocking: An Efficient Framework for Contamination Mitigation Evaluation on Large Language Models. Findings of the Association for Computational Linguistics: EMNLP 2025. 2025. doi:10.18653/v1/2025.findings-emnlp.188

    work page doi:10.18653/v1/2025.findings-emnlp.188 2025

  2. [2]

    Establishing Trustworthy LLM Evaluation via Shortcut Neuron Analysis

    Zhu, Kejian and Tu, Shangqing and Jin, Zhuoran and Hou, Lei and Li, Juanzi and Zhao, Jun. Establishing Trustworthy LLM Evaluation via Shortcut Neuron Analysis. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025. doi:10.18653/v1/2025.acl-long.192

    work page doi:10.18653/v1/2025.acl-long.192 2025

  3. [3]

    The Thirteenth International Conference on Learning Representations , year=

    Training on the Test Task Confounds Evaluation and Emergence , author=. The Thirteenth International Conference on Learning Representations , year=

    work page

  4. [4]

    DeepSeek-V4 : Towards Highly Efficient Million-Token Context Intelligence

    DeepSeek-AI. DeepSeek-V4 : Towards Highly Efficient Million-Token Context Intelligence. arXiv preprint arXiv:2606.19348. 2026

    arXiv 2026

  5. [5]

    Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling

    Biderman, Stella and Schoelkopf, Hailey and Anthony, Quentin Gregory and Bradley, Herbie and O'Brien, Kyle and Hallahan, Eric and Khan, Mohammad Aflah and Purohit, Shivanshu and Prashanth, Usvsn Sai and Raff, Edward and Skowron, Aviya and Sutawika, Lintang and Van Der Wal, Oskar. Pythia: A Suite for Analyzing Large Language Models Across Training and Scal...

    work page 2023

  6. [6]

    OpenOrca : An Open Dataset of GPT Augmented FLAN Reasoning Traces

    Lian, Wing and Goodson, Bleys and Pentland, Eugene and Cook, Austin and Vong, Chanvichet and Teknium. OpenOrca : An Open Dataset of GPT Augmented FLAN Reasoning Traces. 2023

    work page 2023

  7. [7]

    Gemma 4 Technical Report

    Gemma Team. Gemma 4 Technical Report. arXiv preprint arXiv:2607.02770. 2026

    arXiv 2026

  8. [8]

    and Zhou, Denny

    Wei, Jason and Wang, Xuezhi and Schuurmans, Dale and Bosma, Maarten and Ichter, Brian and Xia, Fei and Chi, Ed and Le, Quoc V. and Zhou, Denny. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Advances in Neural Information Processing Systems. 2022

    work page 2022

Show all 54 references
  1. [9]

    and Le, Quoc V

    Wei, Jason and Wang, Xuezhi and Schuurmans, Dale and Bosma, Maarten and Ichter, Brian and Xia, Fei and Chi, Ed H. and Le, Quoc V. and Zhou, Denny , year =. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models , booktitle =

  2. [10]

    2023 , month = may, journal =

    Chain-of-Dictionary Prompting Elicits Translation in Large Language Models , author =. 2023 , month = may, journal =. doi:10.48550/arXiv.2305.06575 , adsnote =. arXiv , keywords =:2305.06575 , primaryclass =

    2023 doi arXiv

  3. [11]

    2024 , month = feb, journal =

    Chain-of-Thought Reasoning without Prompting , author =. 2024 , month = feb, journal =. doi:10.48550/arXiv.2402.10200 , adsnote =. arXiv , keywords =:2402.10200 , primaryclass =

    2024 doi arXiv

  4. [12]

    Leak, Cheat, Repeat:

    Balloccu, Simone and Schmidtov. Leak, Cheat, Repeat:. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1:. 2024 , month = mar, pages =. doi:10.18653/v1/2024.eacl-long.5 , abstract =

    2024 doi

  5. [13]

    Large Scale Kernel Machines , author =

    Scaling Learning Algorithms towards. Large Scale Kernel Machines , author =. 2007 , publisher =

    2007

  6. [14]

    2021 , journal =

    Evaluating Large Language Models Trained on Code , author =. 2021 , journal =. arXiv , keywords =:2107.03374 , doi =

    2021 arXiv

  7. [15]

    Findings of the Association for Computational Linguistics:

    Zhu, Wenhong and Hao, Hongkun and He, Zhiwei and Song, Yun-Ze and Yueyang, Jiao and Zhang, Yumeng and Hu, Hanxu and Wei, Yiran and Wang, Rui and Lu, Hongyuan , editor =. Findings of the Association for Computational Linguistics:. 2024 , month = jun, pages =. doi:10.18653/v1/20...

    2024 doi

  8. [16]

    2021 , journal =

    Training Verifiers to Solve Math Word Problems , author =. 2021 , journal =. 2110.14168 , archiveprefix =

    2021 arXiv

  9. [17]

    Unveiling the Spectrum of Data Contamination in Language Model: A Survey from Detection to Remediation , booktitle =

    Deng, Chunyuan and Zhao, Yilun and Heng, Yuzhao and Li, Yitong and Cao, Jiannan and Tang, Xiangru and Cohan, Arman , editor =. Unveiling the Spectrum of Data Contamination in Language Model: A Survey from Detection to Remediation , booktitle =. 2024 , month = aug, pages =. doi...

    2024 doi

  10. [18]

    Pretraining

    Zhang, Weichao and Zhang, Ruqing and Guo, Jiafeng and. Pretraining. Proceedings of the 2024. 2024 , month = nov, pages =. doi:10.18653/v1/2024.emnlp-main.300 , urldate =

    2024 doi

  11. [19]

    Generalization or Memorization:

    Dong, Yihong and Jiang, Xue and Liu, Huanyu and Jin, Zhi and Gu, Bin and Yang, Mengfei and Li, Ge , editor =. Generalization or Memorization:. Findings of the Association for Computational Linguistics. 2024 , month = aug, pages =. doi:10.18653/v1/2024.findings-acl.716 , abstract =

    2024 doi

  12. [20]

    2024 , journal =

    The Llama 3 Herd of Models , author =. 2024 , journal =. 2407.21783 , archiveprefix =

    2024 arXiv

  13. [21]

    2023 , journal =

    Benchmarking Foundation Models with Language-Model-as-an-Examiner , author =. 2023 , journal =

    2023

  14. [22]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1:

    Yu, Zhuohao and Gao, Chang and Yao, Wenjin and Wang, Yidong and Ye, Wei and Wang, Jindong and Xie, Xing and Zhang, Yue and Zhang, Shikun , editor =. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1:. 2024 , month = aug, pages =....

    2024 doi

  15. [23]

    Treeeval:

    Li, Xiang and Lan, Yunshi and Yang, Chao , year =. Treeeval:. Proceedings of the

  16. [24]

    and Dodge, Jesse , year =

    Elazar, Yanai and Bhagia, Akshita and Magnusson, Ian Helgi and Ravichander, Abhilasha and Schwenk, Dustin and Suhr, Alane and Walsh, Evan Pete and Groeneveld, Dirk and Soldaini, Luca and Singh, Sameer and Hajishirzi, Hannaneh and Smith, Noah A. and Dodge, Jesse , year =. What'...

  17. [25]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations) , address=

    LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations) , address=. 2024 , url=

    2024

  18. [26]

    The Twelfth International Conference on Learning Representations , author =

    Time Travel in. The Twelfth International Conference on Learning Representations , author =. 2024 , keywords =

    2024

  19. [27]

    2016 , volume =

    Deep Learning , author =. 2016 , volume =

    2016

  20. [28]

    2020 , journal =

    Language Models Are Few-Shot Learners , author =. 2020 , journal =. 2005.14165 , archiveprefix =

    2020 arXiv

  21. [29]

    2023 , eprint =

    Textbooks Are All You Need , author =. 2023 , eprint =

    2023

  22. [30]

    Privacy Risks of General-Purpose Language Models , booktitle =

    Pan, Xudong and Zhang, Mi and Ji, Shouling and Yang, Min , year =. Privacy Risks of General-Purpose Language Models , booktitle =

  23. [31]

    2023 , journal =

    Don't Make Your Llm an Evaluation Benchmark Cheater , author =. 2023 , journal =. 2311.01964 , archiveprefix =

    2023 arXiv

  24. [32]

    Stop Uploading Test Data in Plain Text:

    Jacovi, Alon and Caciularu, Avi and Goldman, Omer and Goldberg, Yoav , year =. Stop Uploading Test Data in Plain Text:. arXiv preprint arXiv:2305.10160 , eprint =

    arXiv

  25. [33]

    Documenting Large Webtext Corpora:

    Dodge, Jesse and Sap, Maarten and Marasovi. Documenting Large Webtext Corpora:. 2021 , journal =. 2104.08758 , archiveprefix =

    2021 arXiv

  26. [34]

    2006 , journal =

    A Fast Learning Algorithm for Deep Belief Nets , author =. 2006 , journal =

    2006

  27. [35]

    Chain-of-Symbol Prompting for Spatial Reasoning in Large Language Models , booktitle =

    Hu, Hanxu and Lu, Hongyuan and Zhang, Huajian and Song, Yun-Ze and Lam, Wai and Zhang, Yue , year =. Chain-of-Symbol Prompting for Spatial Reasoning in Large Language Models , booktitle =

  28. [36]

    Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1:

    Li, Peng and Sun, Tianxiang and Tang, Qiong and Yan, Hang and Wu, Yuanbin and Huang, Xuanjing and Qiu, Xipeng , editor =. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1:. 2023 , month = jul, pages =. doi:10.18653/v1/2023.acl-l...

    2023 doi

  29. [37]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1:

    Li, Qintong and Cui, Leyang and Zhao, Xueliang and Kong, Lingpeng and Bi, Wei , editor =. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1:. 2024 , month = aug, pages =. doi:10.18653/v1/2024.acl-long.163 , abstract =

    2024 doi

  30. [38]

    2023 , journal =

    Starcoder: May the Source Be with You! , author =. 2023 , journal =. 2305.06161 , archiveprefix =

    2023 arXiv

  31. [39]

    arXiv preprint arXiv:2402.13125 , eprint =

    Li, Xiang and Lan, Yunshi and Yang, Chao , year =. arXiv preprint arXiv:2402.13125 , eprint =

    arXiv

  32. [40]

    2004 , month = jul, pages =

    Text Summarization Branches Out , author =. 2004 , month = jul, pages =

    2004

  33. [41]

    2023 , month = may, pages =

    Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics , author =. 2023 , month = may, pages =. doi:10.18653/v1/2023.eacl-main.5 , abstract =

    2023 doi

  34. [42]

    Data Contamination:

    Magar, Inbal and Schwartz, Roy , editor =. Data Contamination:. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2:. 2022 , month = may, pages =. doi:10.18653/v1/2022.acl-short.18 , abstract =

    2022 doi

  35. [43]

    Detecting Pretraining Data from Large Language Models , booktitle =

    Shi, Weijia and Ajith, Anirudh and Xia, Mengzhou and Huang, Yangsibo and Liu, Daogao and Blevins, Terra and Chen, Danqi and Zettlemoyer, Luke , year =. Detecting Pretraining Data from Large Language Models , booktitle =

  36. [44]

    Chain-of-Thought Reasoning without Prompting , booktitle =

    Wang, Xuezhi and Zhou, Denny , editor =. Chain-of-Thought Reasoning without Prompting , booktitle =. 2024 , volume =

    2024

  37. [45]

    Codegen:

    Nijkamp, Erik and Pang, Bo and Hayashi, Hiroaki and Tu, Lifu and Wang, Huan and Zhou, Yingbo and Savarese, Silvio and Xiong, Caiming , year =. Codegen:. arXiv preprint arXiv:2203.13474 , eprint =

    arXiv

  38. [46]

    Estimating Contamination via Perplexity:

    Li, Yucheng , year =. Estimating Contamination via Perplexity:. arXiv preprint arXiv:2309.10677 , eprint =

    arXiv

  39. [47]

    Code Llama:

    Roziere, Baptiste and Gehring, Jonas and Gloeckle, Fabian and Sootla, Sten and Gat, Itai and Tan, Xiaoqing Ellen and Adi, Yossi and Liu, Jingyu and Sauvestre, Romain and Remez, Tal and others , year =. Code Llama:. arXiv preprint arXiv:2308.12950 , eprint =

    arXiv

  40. [48]

    Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics:

    Takeshita, Sotaro and Green, Tommaso and Reinig, Ines and Eckert, Kai and Ponzetto, Simone , editor =. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics:. 2024 , month = jun, pages =. doi:10.18653/v1/2024.naacl-lo...

    2024 doi

  41. [49]

    Llama 2:

    Touvron, Hugo and Martin, Louis and Stone, Kevin and Albert, Peter and Almahairi, Amjad and Babaei, Yasmine and Bashlykov, Nikolay and Batra, Soumya and Bhargava, Prajjwal and Bhosale, Shruti and others , year =. Llama 2:. arXiv preprint arXiv:2307.09288 , eprint =

    arXiv

  42. [50]

    Towards Understanding Chain-of-Thought Prompting:

    Wang, Boshi and Min, Sewon and Deng, Xiang and Shen, Jiaming and Wu, You and Zettlemoyer, Luke and Sun, Huan , editor =. Towards Understanding Chain-of-Thought Prompting:. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1:. 2023 ...

    2023 doi

  43. [51]

    Data Contamination Calibration for Black-Box

    Ye, Wentao and Hu, Jiaqi and Li, Liyao and Wang, Haobo and Chen, Gang and Zhao, Junbo , editor =. Data Contamination Calibration for Black-Box. Findings of the Association for Computational Linguistics. 2024 , month = aug, pages =. doi:10.18653/v1/2024.findings-acl.644 , abstract =

    2024 doi

  44. [52]

    Self-Edit:

    Zhang, Kechi and Li, Zhuo and Li, Jia and Li, Ge and Jin, Zhi , editor =. Self-Edit:. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1:. 2023 , month = jul, pages =. doi:10.18653/v1/2023.acl-long.45 , abstract =

    2023 doi

  45. [53]

    2024 , journal =

    A Careful Examination of Large Language Model Performance on Grade School Arithmetic , author =. 2024 , journal =. 2405.00332 , archiveprefix =

    2024 arXiv

  46. [54]

    Multilingual Machine Translation with Large Language Models:

    Zhu, Wenhao and Liu, Hongyi and Dong, Qingxiu and Xu, Jingjing and Huang, Shujian and Kong, Lingpeng and Chen, Jiajun and Li, Lei , editor =. Multilingual Machine Translation with Large Language Models:. Findings of the Association for Computational Linguistics:. 2024 , month ...

    2024 doi

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.