Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Low-Perplexity LLM-Generated Sequences and Where To Find Them

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper reports that only around 40% of low-perplexity 6-token windows generated by Pythia 6.9B have at least one exact match in The Pile, and classifies the rest and the matched subsets by match frequency.

desk verdict Real empirical finding on low-perplexity windows, but the headline match rate is conditional on a disclosed tokenizer mismatch; deserves review with a same-tokenizer sensitivity check. read the letter →

arxiv 2507.01844 v1 pith:JWNG7EKD submitted 2025-07-02 cs.CL cs.LG

Arthur Wuhrmann , Anastasiia Kucherenko , Andrei Kucharavy This is my paper
classification cs.CLcs.LG
keywords low-perplexitysequencestrainingdataattributionmemorizationverbatimrecallInfini-gramPythiaThePileexactn-grammatching
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether the stretches of text an LLM generates with very high confidence are verbatim copies of its training data. The authors measure this on Pythia 6.9B over four specialised topics, extracting 6-token windows whose every token has conditional probability $p \ge 0.9$ and counting exact matches in The Pile via Infini-gram. They find that only about 40% of such low-perplexity windows have any exact match, and the rest appear to be synthetically coherent text the model produced without copying. Matched windows split by match frequency into memorization, segmental replication, and frequently encountered text, with about 20% traceable to few enough documents for manual review. The paper thus provides a quantitative, category-level answer to when low perplexity signals memorization rather than fluent synthesis.

What carries the argument

The central object is the low-perplexity window: a contiguous 6-token span in which every token has conditional probability at least 0.9 ($\log_2 P \le 0.152$). The pipeline slides a stride-1 window over low-perplexity sequences extracted from model outputs, then queries Infini-gram's trillion-token n-gram index of The Pile for exact occurrences of each window. The number of matching documents $c$ for each window is the quantity that drives the taxonomy of synthetic coherence, memorization, segmental replication, and frequently encountered text.

What would settle it

Re-run the matching pipeline with Infini-gram re-indexed using the Pythia tokenizer (or an equivalent tokenizer-matched exact-match oracle) on the same generated windows. If the share of windows with at least one exact match rises from roughly 40% toward the 90%+ range, the claim that most low-perplexity spans cannot be mapped to the corpus would be an artifact of tokenizer mismatch rather than a property of generation.

Watch

Extended reading notes

Core claim

The paper's central empirical discovery is that low perplexity is not a reliable marker of verbatim training-data recall. Measuring 6-token windows with per-token probability $p \ge 0.9$ in generations by Pythia 6.9B, only 40% of windows have at least one exact n-gram match in The Pile ($N_{c>0}$), with topic-level match rates from 25% (nuclear physics) to 67% (drugs). The remaining ~60% of low-perplexity windows are coherent text the model produced with no exact corpus match. Among matched windows, the number of matching documents $c$ separates three behaviours: memorization ($0<c<5$), segmental replication ($5\le c<50$), and frequently encountered text ($c>50$), roughly 20% of all windows falling in the manually reviewable first two categories.

Load-bearing premise

The whole measurement rests on Infini-gram's exact-match counts being a faithful oracle for whether a 6-token span actually occurs in The Pile; because Pythia and Infini-gram use different tokenizers, some verbatim spans that are present in the data may never align, which would inflate the 'no match' fraction.

Editorial extensions

If this is right

  • Low perplexity alone is not evidence of memorization: roughly 60% of high-confidence windows have no exact corpus match and are classified as synthetic coherence.
  • About 20% of low-perplexity windows fall into memorization ($0<c<5$) and segmental replication ($5\le c<50$), a subset small enough for manual document review.
  • Topic representation in the training data drives match rates, from 25% for nuclear physics to 67% for drugs, reflecting the Pile's uneven topical coverage.
  • Model size and temperature change the number of low-perplexity windows but leave the share with at least one match roughly constant, so the categorical pattern is stable across those settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Correcting the tokenizer mismatch with a Pythia-indexed Infini-gram could shrink the synthetic-coherence bucket; the paper's own limitation section indicates such spans may fail to align despite being present in the training data.
  • The arbitrary thresholds of 5 and 50 matches could be replaced by data-driven clustering of the match-count distribution to produce sharper boundaries between the three matched categories.
  • The pipeline could serve as a cheap pre-filter for training-data attribution: windows with a small number of matches are the ones worth expensive influence-based tracing, since they pinpoint few reviewable documents.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces an open-source pipeline that extracts low-perplexity token sequences from Pythia-6.9B generations on four specialized topics (cryptography, drugs, genetics, nuclear physics), converts them into 6-token sliding windows, and queries Infini-gram for exact matches in The Pile. The central empirical claim, stated in Section 3.1 and Table 2, is that only about 40% of low-perplexity windows have at least one exact match in the training data, and the paper interprets the complementary portion as evidence that many low-perplexity spans cannot be mapped back to the corpus. Matched windows are then partitioned by match count into four categories: synthetic coherence (c=0), memorization (0<c<5), segmental replication (5<=c<50), and frequently encountered text (c>50). The paper also reports scaling and temperature experiments in Section 3.3. The main conclusion is that exact-match tracing of low-perplexity sequences is incomplete and that the observed match distribution supports a nuanced view of memorization, but the headline measurement is conditional on the tokenizer mismatch and the absence of a control baseline.

Significance. If the central measurement survives the tokenizer-mismatch and baseline checks, the paper would be a useful empirical contribution to training-data attribution and memorization research. Its strengths are the open-source pipeline, the multi-topic design, the model-scaling and temperature sweeps, and the explicit categorical framework for interpreting match counts. The paper is also honest in listing its limitations, including the tokenizer mismatch in Section 5. The claim is an empirical observation rather than a theoretical law, and its value depends on the reliability of exact-match search as a proxy for presence in the training data; that reliability is exactly what the main concern targets.

major comments (3)
  1. [Section 3.1, Table 2, and Section 5] The central measurement in Table 2 is made with Infini-gram's LLaMA-2 tokenizer on windows generated by Pythia, which uses a different tokenizer. The authors acknowledge this mismatch in Section 5, but because BPE tokenization is context-dependent, a 6-token Pythia window that is present verbatim in The Pile may not align to any contiguous LLaMA-2 n-gram stored in Infini-gram's index, producing systematic false negatives. This can only lower the observed Nc>0 rate, so the headline 'only 40% of low-perplexity windows have at least one exact match' is potentially an artifact of tokenization rather than a property of low-perplexity generation. The manuscript needs a same-tokenizer sensitivity check, for example by indexing the relevant Pile subset with the Pythia tokenizer or by re-tokenizing matched windows, and a quantitative estimate of the mismatch rate, before the 'cannot be mapped' conclusion can be accepted.
  2. [Section 3.1, Table 2, and Section 5] There is no control condition comparing the match rate of low-perplexity windows with high-perplexity or randomly selected windows of the same length. Without such a baseline, the statement that low-perplexity spans are surprisingly unmappable is not supported: the observed 25-67% match rates could reflect the general sparsity of exact 6-gram matches in a trillion-token corpus under a mismatched tokenizer rather than a distinctive property of low-perplexity generation. I request a control experiment, for example sampling windows from the same outputs with high standalone perplexity or sampling random contiguous spans from held-out text, and reporting the same Nc>0 statistics for those controls.
  3. [Section 3.1 and Section 5] Because the prompts are quotes extracted from the Wikipedia portion of The Pile, a substantial fraction of low-perplexity windows are prompt repetitions (Nrep/N ranges from 7.9% to 32% in Table 2), and these repetitions are retained in the subsequent analyses. This inflates the overall Nc>0 rate and the memorization/segmental-replication categories in a way that is not intrinsic to low-perplexity generation but instead reflects re-generation of the provided context. The main statistics should be reported both with and without windows that overlap the prompt, so readers can separate genuine recall from re-sampling of the supplied prompt.
minor comments (5)
  1. [Table 1] The table header contains the typo 'Crypt2ography'; it should read 'Cryptography'.
  2. [Tables 4 and 5] The column header 'N>0/N' is inconsistent with the notation 'Nc>0' used in Table 2 and the text; the same quantity should be named uniformly throughout.
  3. [Section 3.1] The text contains the typo 'occurances' for 'occurrences'.
  4. [Section 2] The justification that a 6-token window is 'long enough to avoid random matches' would benefit from a null-model estimate or a citation; as written, the choice of window size is asserted rather than tested.
  5. [Section 3.2] The thresholds c=5 and c=50 are acknowledged as arbitrary, but the paper does not provide a sensitivity analysis for these cutoffs; a brief robustness check would make the cross-topic comparison in Table 3 more convincing.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an empirical measurement pipeline with no fitted parameters, no self-citation chain, and no prediction that reduces to its own inputs.

full rationale

The paper does not claim to derive a prediction from first principles. Its central claim (Section 3.1, Table 2) is the measured frequency 40% (Nc>0) with which low-perplexity 6-token windows generated by Pythia-6.9B have an exact n-gram match in The Pile as indexed by Infini-gram. That is an observation, not a quantity forced by construction: the low-perplexity threshold (log2 P <= 0.152) is a hand-set inclusion criterion, and the c thresholds (5 and 50) defining Memorization, Segmental replication, and Frequently encountered text are stated to be arbitrary and adjustable, so the four categories are descriptive bins over match counts rather than outputs derived from the thresholds. No parameter is fitted to a subset of data and then reported as a prediction; no external result is invoked through self-citation; the cited tools and corpora (Pythia, The Pile, Infini-gram, OlmoTrace) are independent external resources. The acknowledged tokenizer mismatch between Pythia-generated windows and the LLaMA-2-tokenized Infini-gram index is a potential validity threat to the 'cannot be mapped' claim because it can only suppress matches, but it is disclosed in Section 5 and is not a circularity: the measurement does not assume the conclusion. The prompt-repetition bias (Appendix C, Nrep/N) is also a validity concern, and if anything it biases match rates upward, not toward the headline. Under the review rules, such correctness concerns belong in a validity pass, not in a circularity score. No circular step was found.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the choice of perplexity threshold, window size, and match-count cutoffs, plus the fidelity of Infini-gram's tokenization to Pythia's. No new physical or conceptual entities are postulated.

free parameters (4)
  • perplexity threshold (log2(P) <= 0.152, probability >= 0.9) = 0.152
    Defines what counts as a low-perplexity token; changing it changes the set of windows and the match rates.
  • window size = 6 tokens
    Sliding window length; the match rate is sensitive to this choice. The paper justifies it but does not test alternatives.
  • match count category thresholds = 5 and 50
    Arbitrary boundaries between memorization, segmental replication, and frequent text; the paper acknowledges they are adjustable.
  • generation parameters = topk=20, topp=0.8, T=0.7
    Chosen to balance meaningful output and degeneration; variants are tested in Section 3.3.
assumptions (3)
  • domain assumption Pythia-6.9B was trained on The Pile
    Model and dataset are public; this grounds the matching corpus.
  • domain assumption Infini-gram's index over The Pile is complete for the queries used
    The paper relies on Infini-gram for exact-match counts; any gap or tokenizer mismatch affects the counts.
  • ad hoc to paper A token with probability >= 0.9 is a meaningful low-perplexity unit
    Threshold chosen by the authors; not derived from theory or prior benchmark.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Low-Perplexity LLM-Generated Sequences and Where To Find Them." pith.science (2026). https://pith.science/paper/JWNG7EKD

@misc{pith2026250701844,
  author       = {Pith},
  title        = {Pith review of: Low-Perplexity LLM-Generated Sequences and Where To Find Them},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JWNG7EKD}},
  note         = {Machine review of arXiv:2507.01844}
}
read the original abstract

As Large Language Models (LLMs) become increasingly widespread, understanding how specific training data shapes their outputs is crucial for transparency, accountability, privacy, and fairness. To explore how LLMs leverage and replicate their training data, we introduce a systematic approach centered on analyzing low-perplexity sequences - high-probability text spans generated by the model. Our pipeline reliably extracts such long sequences across diverse topics while avoiding degeneration, then traces them back to their sources in the training data. Surprisingly, we find that a substantial portion of these low-perplexity spans cannot be mapped to the corpus. For those that do match, we quantify the distribution of occurrences across source documents, highlighting the scope and nature of verbatim recall and paving a way toward better understanding of how LLMs training data impacts their behavior.

Figures

Figures reproduced from arXiv: 2507.01844 by the authors.

Figure 1
Figure 1. Visualization of a generated subsequence that [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Boxplots comparing the number of matches of [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Illustration of the low-perplexity sequences, [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Example of the perplexities of an output that [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Quantifying the Agreement Between Data-Influence and Data-Similarity to Understand LLM Behavior

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    Data-similarity and data-influence produce significantly overlapping rankings of training documents for LLM outputs, with asymmetry allowing a favorable cost-accuracy trade-off.

Reference graph

Works this paper leans on

33 extracted references · 10 canonical work pages · cited by 1 Pith paper

  1. [1]

    Ekin Akyurek, Tolga Bolukbasi, Frederick Liu, Binbin Xiong, Ian Tenney, Jacob Andreas, and Kelvin Guu. 2022. https://doi.org/10.18653/v1/2022.findings-emnlp.180 Towards tracing knowledge in language models back to the training data . In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2429--2446, Abu Dhabi, United Arab Emirates...

    doi:10.18653/v1/2022.findings-emnlp.180 2022

  2. [2]

    Ali Al-Kaswan, Maliheh Izadi, and Arie van Deursen. 2024. https://doi.org/10.1145/3597503.3639133 Traces of memorisation in large language models for code . In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ICSE ’24, page 1–12. ACM

    arXiv 2024

  3. [3]

    Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell

    Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. https://doi.org/10.1145/3442188.3445922 On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT '21, page 610–623, New York, NY, USA. Association for Computing...

    arXiv 2021

  4. [4]

    Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023. https://arxiv.org/abs/2304.01373 Pythia: A suite for analyzing large language models across training and scaling . Preprint...

    arXiv 2023

  5. [5]

    Hannah Brown, Katherine Lee, Fatemehsadat Mireshghallah, Reza Shokri, and Florian Tramèr. 2022. https://arxiv.org/abs/2202.05520 What does it mean for a language model to preserve privacy? Preprint, arXiv:2202.05520

    arXiv 2022

  6. [6]

    Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramèr, Borja Balle, Daphne Ippolito, and Eric Wallace. 2023 a . https://arxiv.org/abs/2301.13188 Extracting training data from diffusion models . Preprint, arXiv:2301.13188

    arXiv 2023

  7. [7]

    Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2023 b . https://arxiv.org/abs/2202.07646 Quantifying memorization across neural language models . Preprint, arXiv:2202.07646

    arXiv 2023

  8. [8]

    Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. https://arxiv.org/abs/2012.07805 Extracting training data from large language models . Preprint, arXiv:2012.07805

    arXiv 2021

Show all 33 references
  1. [9]

    Chang, Dheeraj Rajagopal, Tolga Bolukbasi, Lucas Dixon, and Ian Tenney

    Tyler A. Chang, Dheeraj Rajagopal, Tolga Bolukbasi, Lucas Dixon, and Ian Tenney. 2024. https://arxiv.org/abs/2410.17413 Scalable influence and fact tracing for large language model pretraining . Preprint, arXiv:2410.17413

    2024 arXiv

  2. [10]

    Deric Cheng, Juhan Bae, Justin Bullock, and David Kristofferson. 2025. https://arxiv.org/abs/2501.12642 Training data attribution (tda): Examining its adoption & use cases . Preprint, arXiv:2501.12642

    2025 arXiv

  3. [11]

    Hiroyuki Deguchi, Go Kamoda, Yusuke Matsushita, Chihiro Taguchi, Kohei Suenaga, Masaki Waga, and Sho Yokoi. 2025. https://arxiv.org/abs/2503.03703 Softmatcha: A soft and fast pattern matcher for billion-scale corpus searches . Preprint, arXiv:2503.03703

    2025 arXiv

  4. [12]

    Vitaly Feldman and Chiyuan Zhang. 2020. What neural networks memorize and why: discovering the long tail via influence estimation. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA. Curran Associates Inc

    2020

  5. [13]

    Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie-Yan Liu. 2019. https://arxiv.org/abs/1907.12009 Representation degeneration problem in training natural language generation models . Preprint, arXiv:1907.12009

    2019 arXiv

  6. [14]

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. https://arxiv.org/abs/2101.00027 The pile: An 800gb dataset of diverse text for language modeling . P...

    2020 arXiv

  7. [15]

    Smith, and Luke Zettlemoyer

    Hila Gonen, Srini Iyer, Terra Blevins, Noah A. Smith, and Luke Zettlemoyer. 2024. https://arxiv.org/abs/2212.04037 Demystifying prompts in language models via perplexity estimation . Preprint, arXiv:2212.04037

    2024 arXiv

  8. [16]

    Clinton Gormley and Zachary Tong. 2015. https://www.oreilly.com/library/view/elasticsearch-the-definitive/9781449358532/ Elasticsearch: The Definitive Guide . O'Reilly Media

    2015

  9. [17]

    Kelvin Guu, Albert Webson, Ellie Pavlick, Lucas Dixon, Ian Tenney, and Tolga Bolukbasi. 2023. https://arxiv.org/abs/2303.08114 Simfluence: Modeling the influence of individual training examples by simulating training runs . Preprint, arXiv:2303.08114

    2023 arXiv

  10. [18]

    Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. https://arxiv.org/abs/1911.00172 Generalization through memorization: Nearest neighbor language models . Preprint, arXiv:1911.00172

    2020 arXiv

  11. [19]

    Yongchan Kwon, Eric Wu, Kevin Wu, and James Zou. 2023. https://doi.org/10.48550/ARXIV.2310.00902 Datainf: Efficiently estimating data influence in lora-tuned llms and diffusion models . CoRR, abs/2310.00902

    2023 doi arXiv

  12. [20]

    Weixin Liang, Yaohui Zhang, Zhengxuan Wu, Haley Lepp, Wenlong Ji, Xuandong Zhao, Hancheng Cao, Sheng Liu, Siyu He, Zhi Huang, Diyi Yang, Christopher Potts, Christopher D Manning, and James Y. Zou. 2024. https://arxiv.org/abs/2404.01268 Mapping the increasing use of llms in sci...

    2024 arXiv

  13. [21]

    Jiacheng Liu, Taylor Blanton, Yanai Elazar, Sewon Min, YenSung Chen, Arnavi Chheda-Kothary, Huy Tran, Byron Bischoff, Eric Marsh, Michael Schmitz, Cassidy Trier, Aaron Sarnat, Jenna James, Jon Borchardt, Bailey Kuehl, Evie Cheng, Karen Farley, Sruthi Sreeram, Taira Anderson, a...

    2025 arXiv

  14. [22]

    Jiacheng Liu, Sewon Min, Luke Zettlemoyer, Yejin Choi, and Hannaneh Hajishirzi. 2025 b . https://arxiv.org/abs/2401.17377 Infini-gram: Scaling unbounded n-gram language models to a trillion tokens . Preprint, arXiv:2401.17377

    2025 arXiv

  15. [23]

    Thomas McCoy, Paul Smolensky, Tal Linzen, Jianfeng Gao, and Asli Celikyilmaz

    R. Thomas McCoy, Paul Smolensky, Tal Linzen, Jianfeng Gao, and Asli Celikyilmaz. 2023. https://doi.org/10.1162/tacl_a_00567 How much do language models copy from their training data? evaluating linguistic novelty in text generation using RAVEN . Transactions of the Association...

    2023 doi

  16. [24]

    Smith, and Yanai Elazar

    William Merrill, Noah A. Smith, and Yanai Elazar. 2024. https://arxiv.org/abs/2406.13069 Evaluating n -gram novelty of language models using rusty-dawg . Preprint, arXiv:2406.13069

    2024 arXiv

  17. [25]

    Monology. 2021. Pile uncopyrighted. https://huggingface.co/datasets/monology/pile-uncopyrighted. Accessed: May 17, 2025

    2021

  18. [26]

    Yijun Pan, Taiwei Shi, Jieyu Zhao, and Jiaqi Ma. 2025. https://doi.org/10.48550/arXiv.2502.11411 Detecting and filtering unsafe training data via data attribution

    2025 doi

  19. [27]

    Low-Carbon Power. 2024. https://lowcarbonpower.org/region/Switzerland Carbon intensity of electricity in switzerland . Accessed: May 17, 2025

    2024

  20. [28]

    Choquette-Choo, Jacob Ray Fuehne, Stella Biderman, Tracy Ke, Katherine Lee, and Naomi Saphra

    USVSN Sai Prashanth, Alvin Deng, Kyle O'Brien, Jyothir S V, Mohammad Aflah Khan, Jaydeep Borkar, Christopher A. Choquette-Choo, Jacob Ray Fuehne, Stella Biderman, Tracy Ke, Katherine Lee, and Naomi Saphra. 2025. https://arxiv.org/abs/2406.17746 Recite, reconstruct, recollect: ...

    2025 arXiv

  21. [29]

    MIT Technology Review. 2024. https://www.technologyreview.com/2024/03/04/1089403/large-language-models-amazing-but-nobody-knows-why/ Large language models can do jaw-dropping things. but nobody knows exactly why. Accessed: 2025-05-18

    2024

  22. [30]

    Kangxi Wu, Liang Pang, Huawei Shen, and Xueqi Cheng. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.782 Enhancing training data attribution for large language models with fitting error consideration . In Proceedings of the 2024 Conference on Empirical Methods in Natural Lan...

    2024 doi

  23. [31]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. https://arxiv.org/abs/2306.05685 Judging llm-as-a-judge with mt-bench and chatbot arena . P...

    2023 arXiv

  24. [32]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  25. [33]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.