Pith. sign in

REVIEW 3 major objections 4 minor 80 references

SoftMatcha 2: A Fast and Soft Pattern Matcher for Trillion-Scale Corpora

T0 review · 3 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that semantic search over a trillion-token corpus can run in well under 0.3 seconds without the search space exploding in query length.

desk verdict Engineering result is real and worth referee time; the theoretical guarantee is weaker than advertised and should be re-scoped. read the letter →

arxiv 2602.10908 v2 pith:VHTOSQO4 submitted 2026-02-11 cs.CL cs.LGstat.ML

Masataka Yoneda , Yusuke Matsushita , Go Kamoda , Kohei Suenaga , Takuya Akiba , Masaki Waga , Sho Yokoi This is my paper
classification cs.CLcs.LGstat.ML MSC 68P2068W32
keywords corpussearchsuffixarraysemanticsimilarityapproximatestringmatchingbenchmarkcontaminationtrillion-tokencorporadynamicpruningn-gramstatistics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to make semantically flexible corpus search practical at trillion-token scale. It claims that by inverting the usual direction—enumerating candidate word patterns similar to the query and checking each against a fast exact-match index—search that allows word substitution, insertion, and deletion can complete in roughly a tenth to three-tenths of a second on a 1.4-trillion-token corpus. The speed comes from two mechanisms: a disk-aware suffix-array design that needs one disk access per exact lookup, and iterative corpus-aware pruning that uses the power-law distribution of n-grams to keep candidate sets from exploding. The authors also claim this soft matching catches benchmark-contamination cases, such as questions with altered numbers, that exact matching misses, and they provide a theoretical argument that the number of lookups stays bounded rather than growing exponentially in query length.

What carries the argument

Two mechanisms carry the method. The disk-aware, staged suffix array is a two-level index: a RAM-resident sparse table narrows the binary search to a small disk block, so each exact string match costs one random disk access; run-length compression shrinks the index to 21.6 TB for the full corpus. Dynamic corpus-aware pruning is the anti-explosion mechanism: the algorithm expands the query prefix, generates all words within a similarity threshold, filters to corpus-occurring prefixes via the fast lookup, and uses two extra techniques—precomputed RAM tables for common 2-grams and 3-grams, and direct enumeration when a prefix is rare. The similarity itself is a smooth minimum of per-word cosine

What would settle it

Run the released implementation on the full 1.4-trillion-token index with queries built from deliberately polysemous high-frequency words (e.g., "get make set run") at a low similarity threshold, and count exact lookups per prefix; if the per-step candidate count rises steadily with query length instead of staying flat, Theorem 1's hypothesis fails. Alternatively, plot the distribution of |W_w| across the vocabulary: if high-frequency words average thousands of embedding-near alternatives, the constant-order assumption is false.

Watch

Extended reading notes

Core claim

SoftMatcha 2 is a search algorithm that, given a short token query, returns phrases in a corpus that are similar to the query under a word-embedding-based similarity measure, allowing substitutions as well as inserted and deleted tokens. Rather than scanning documents, it generates candidate token sequences prefix by prefix, keeps only those that actually occur in the corpus—checked by exact lookup in a suffix-array index—and prunes the rest. The central claim is that this generate-and-filter loop tolerates semantic variation at trillion-token scale: p95 latency is 278 ms for soft search and 0.34 ms for exact lookup on FineWeb-Edu, with essentially flat scaling from 1B to 1.4T tokens. It fur

Load-bearing premise

The efficiency guarantee rests on the assumption that the average number of semantically similar candidate words per surviving prefix stays roughly constant—Hypothesis 2 in Appendix J—rather than growing with vocabulary size; the authors concede in the conclusion that the theoretical analysis presupposes statistical regularities of natural language, and they also note the method is word-level and cannot catch multi-word paraphrases like "U.S." vs "United States".

Editorial extensions

If this is right

  • Exact string search on a 1.4-trillion-token corpus takes 0.34 ms at p95, about 33 times faster than the previous suffix-array engine, and the soft-search index builds faster than the exact-only index.
  • Soft search p95 latency is 278 ms on the full English corpus, stays under 400 ms for Japanese and Chinese corpora, and remains almost flat as corpus size grows from 1B to 1.4T tokens.
  • Queries and results no longer need the same token count: insertions and deletions are handled with penalties tuned by word informativeness, enabling matches like "importance of machine learning" against "importance of the machine learning".
  • The paper argues that the number of exact lookups is approximately constant in query length for fixed threshold and moderate insertion-deletion penalty, and grows like |C|^{1/δ} in corpus size under Zipf n-gram statistics.
  • Soft matching identifies 36 benchmark items as contaminated that exact matching misses, and 29 of 36 are confirmed by manual inspection as semantic contamination or template leakage.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the paper's own limitation—multi-word paraphrases like "U.S." vs "United States" are invisible—suggests a natural extension: composing word vectors into phrase representations, though the authors note this would enlarge the substitution candidate set.
  • Editorial inference: the observed growth of 2.27x in lookups per token under full pruning is still exponential, just with a small base; a testable prediction is that repetitive domains such as code or boilerplate legal text will show a larger base and higher latency than narrative prose.
  • Editorial inference: the template-leakage examples imply a cheap contamination audit: run soft search on perturbed versions of benchmark questions with numbers and entities substituted, and the method automatically surfaces near-matches that exact tools miss.
  • Editorial inference: because exact lookup is roughly 33 times faster than the previous engine, the same disk-aware suffix-array design could accelerate exact n-gram mining and retrieval tasks beyond the soft-search application presented.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents SoftMatcha 2, a system for soft (semantic) full-text search over trillion-token corpora. It combines a disk-aware staged suffix array (one random disk access per exact lookup) with dynamic corpus-aware pruning (iterative, k-gram, last-bits) to enumerate candidate variants with substitution/insertion/deletion. On FineWeb-Edu (1.4T tokens) it reports 0.34 ms p95 exact lookup and 278 ms p95 soft search, substantially faster than infini-gram, infini-gram mini, and SoftMatcha. The paper also provides an informal theoretical analysis (Theorems 1-2) claiming that total exact lookups are O(1) in query length under a hypothesis about natural-language occurrence statistics, and O(|C|^{1/δ}) in corpus size under Zipf's law. As an application, it detects 36 benchmark-contamination cases missed by exact matching, 29 confirmed by manual inspection.

Significance. If the empirical results and the system's design are reproducible, this is a valuable engineering contribution to large-scale corpus search and contamination auditing. The latency measurement methodology is careful (cache cleared before each query, p95/median, baseline failures reported). The 36-case contamination study with manual verification is a concrete, useful demonstration. However, the theoretical claims as stated are not established: the two hypotheses on which Theorems 1-2 rest are close to the conclusions and are not empirically validated, and the observed per-token growth of 2.27× in lookups (Figure 5) under the actual adaptive protocol shows that the exponential growth is only mitigated, not provably eliminated. The contribution is therefore primarily empirical and systems-oriented; the theory needs substantial revision or removal.

major comments (3)
  1. [Appendix J, Hypothesis 1 and proof of Theorem 1 (Informal)] The formal statement of Hypothesis 1 is E[|Li|] ≈ O(E[|Si||C*_i|/|V|^i]). The proof of Theorem 1 proceeds by bounding |C*_i| ≤ |C| and using E[|Si|/|V|^i]=O(r^i) to get E[|Li|]=O(r^i). This is only valid if the occurrence ratio for similar patterns is comparable to the uniform ratio |C*_i|/|V^i|. For natural-language queries, S_i is not a uniform sample of V^i; it is biased toward grammatical/collocational sequences, so |Li|/|Si| can be orders of magnitude larger than the uniform ratio. The 'We believe...' justification (p.5) is not empirical. Figure 5 shows 2.27× per-token growth with all pruning, which is hard to reconcile with the O(1) total-lookup claim. The authors should directly measure |Li|/|Si| and |C*_i|/|V^i| as a function of i on their corpus to validate Hypothesis 1, or remove the formal theorem.
  2. [Appendix J, Hypothesis 2 and Lemma 2] Hypothesis 2 states E[|L'_i|] ≈ O(E[|L_{i-1}|]), i.e., the branching factor |W_w| is constant-order. This is structurally the same as the conclusion of Theorem 1. Lemma 2 then simply sums this assumption over i. The justification (p.25) 'we assume that |W_w| on average is approximately up to constant order' is an assumption, not a derivation. Without a mechanism (e.g., embedding dimension, threshold α, corpus redundancy) that yields this bound, the result is circular. Since both Theorems 1 and 2 depend on Lemma 2, the theoretical contribution is currently an assumption rather than a theorem.
  3. [§5.3, Figure 5 and Theorem 1 (Informal)] The theorem applies to fixed α and moderate γ, as the paper notes in footnote 5. The deployed system uses adaptive α relaxing until K results, and Figure 5 shows 2.27× per-token growth under that protocol. The abstract and conclusion claim the method 'suppresses exponential growth' and 'mitigates exponential growth' without qualifying that the theoretical guarantee is for a different protocol. The empirical 0.3 s latency is an impressive engineering result, but the theoretical explanation does not cover the measured behavior. The authors should either extend the analysis to adaptive α or explicitly scope the claim.
minor comments (4)
  1. [Abstract] Typo: 'handlingsemanticvariations' appears to be a missing-space error.
  2. [§3.1] The smooth-min similarity formula is hard to parse; the parenthesization should be clarified for readability.
  3. [Appendix J] The use of '≈' and 'O' inside expectations without rigorous constants makes the theorems informal. If the goal is a theoretical guarantee, precise statements with explicit constants are needed.
  4. [§6] The contamination thresholds (0.6 and 0.8) and the parameters β, γ' are chosen on a single corpus; a sensitivity analysis would strengthen the claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: theorems are conditional on explicit statistical hypotheses; latency and contamination results are measured or verified, not derived from fitted constants.

full rationale

The central efficiency claim (Theorem 1, Informal, §4) is explicitly conditional on Hypothesis 1 and Hypothesis 2, which are stated as assumptions rather than derived from the conclusion. Hypothesis 2 bounds per-step candidate growth, and Hypothesis 1 supplies the exponential decay of |L_i|; the O(1) total-lookup conclusion follows from their conjunction, so it is not identical to either premise. Calibrated constants (β=10^4, γ′≈21.70, the 0.45 output floor, and the 0.6/0.8 contamination thresholds) are disclosed fitting or choice parameters, not predictions, and the headline latencies are direct measurements on FineWeb-Edu. Citations to the authors' prior SoftMatcha and Zipfian-whitening work provide context and reusable tools, but they are not used as a uniqueness theorem or as the justification for the theoretical claim. The main weakness—Hypothesis 1 is supported only by 'We believe that this hypothesis is valid from a linguistic perspective' and is not empirically validated—is a robustness/correctness concern, not a circular-dependency concern. Consequently, no step reduces by construction to its own inputs.

Assumptions & free parameters 9 free parameters · 6 assumptions · 0 invented entities

The headline latency is measured, not derived, so fitted constants (β, γ', thresholds) affect quality rather than the time bound. The theoretical efficiency claim rests on two stated hypotheses about natural-language statistics (Hypotheses 1-2 in §4 and Appendix J) and an unstated independence assumption in Lemma 3; the most fragile step is Hypothesis 2, whose justification assumes the per-step candidate set is constant-order, close to the bound being proven.

free parameters (9)
  • β (smooth-min temperature) = β = 10^4
    Chosen in experiments for the smooth-minimum similarity in §3.1; no sensitivity analysis.
  • γ' (insertion/deletion penalty scale) = γ'≈21.70 for FineWeb-Edu
    Calibrated so that the scaling factor equals 1/e when m=5 and the 50th-lowest-norm word is inserted (§3.1, footnote 4); corpus-specific.
  • Similarity floor for output = 0.45
    Minimum reported similarity in soft-search experiments (§5); results are shown only above this value.
  • Top-K output count = K=20
    Number of matching patterns returned per query (§5); latency grows with K (Figure 12).
  • Contamination thresholds = sim ≥ 0.6; η ≥ 0.8
    Soft-match similarity and aggregate contamination fraction used in §6; chosen without independent justification and produce the 36-sample volume.
  • Max query length L = L=12 in experiments; queries ≤10 tokens
    The staged index assumes short queries (Appendix H); 8 subword-tokenizer queries failed by exceeding the limit (Appendix G).
  • Staged-index block size B = B between 128 and 256
    Chosen so B/2 fits the OS page size to achieve one disk access per lookup (Appendix H).
  • Last-bits pruning trigger = ≤50 occurrences
    Threshold below which the algorithm enumerates occurrences directly instead of appending suffixes (Appendix I.2).
  • k-gram pruning frequency ranks = 2-grams in top 100K words; 3-gram rank-sum ≤10K
    Conditions for precomputed existence tables kept in RAM (Appendix I.1); hand-set.
assumptions (6)
  • domain assumption Hypothesis 1: E[|Li|] ≈ O(E[|Si||C*_i|/|V|^i]) — corpus occurrence ratio among query-similar word sequences is about the same as among all V^i sequences.
    Assumed in §4 with the comment 'We believe that this hypothesis is valid from a linguistic perspective'; it is the bridge that converts query similarity to corpus occurrence statistics.
  • ad hoc to paper Hypothesis 2: E[|L'_i|] ≈ O(E[|L_{i-1}|]) — the per-step candidate set stays within a constant factor of the previous step's surviving set.
    Appendix J justifies this by 'we assume that |W_w| on average is approximately up to constant order' — an assumption close to the bounded-search theorem it is used to prove.
  • domain assumption Zipf's law for n-gram occurrence probabilities with exponent δ>1 (δ≈1.5 empirically).
    Used in Theorem 2 and Lemma 3; cited to Bellina & Servedio (2025) and Silva & Cunha (2020), not established in this paper.
  • domain assumption Independence of n-gram occurrences in Lemma 3's expectation computation.
    The formula E[Occurs(w)] = 1 - (1 - pk^{-δ})^|C| treats corpus positions as independent trials; this probabilistic assumption is unstated.
  • domain assumption Cosine similarity of word embeddings approximates semantic similarity for substitution/insertion/deletion scoring.
    Whole method premise (§3.1); quality, not latency, depends on it.
  • domain assumption Tokenizer-embedding compatibility across languages and corpora.
    Appendix B admits 'the tokenizers and embeddings we used are not completely compatible, but empirically they are compatible in most of the cases'; the multilingual results depend on this.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SoftMatcha 2: A Fast and Soft Pattern Matcher for Trillion-Scale Corpora." pith.science (2026). https://pith.science/paper/VHTOSQO4

@misc{pith2026260210908,
  author       = {Pith},
  title        = {Pith review of: SoftMatcha 2: A Fast and Soft Pattern Matcher for Trillion-Scale Corpora},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VHTOSQO4}},
  note         = {Machine review of arXiv:2602.10908}
}
read the original abstract

We present SoftMatcha 2, an ultra-fast and flexible search algorithm that enables search over trillion-scale natural language corpora in under 0.3 seconds while allowing semantic variations in the form of substitution, insertion, and deletion. Our approach employs string matching based on suffix arrays that scales well with corpus size, and represents words as vectors, which underpin its semantic flexibility. To mitigate the combinatorial explosion induced by the semantic relaxation of queries, our method is built on two key algorithmic ideas: dynamic corpus-aware pruning and fast exact lookup enabled by a disk-aware design. We theoretically analyze the efficiency of the proposed method, indicating that it can mitigate exponential growth in the search space. Empirically, on FineWeb-Edu (Lozhkov et al., 2024) (1.4T tokens), it attains substantially lower search latency than existing methods: infini-gram (Liu et al., 2024), infini-gram mini (Xu et al., 2025), and SoftMatcha (Deguchi et al., 2025). As a practical application, our method uncovers benchmark contamination in training corpora that existing approaches miss, and it also benefits information retrieval and paraphrase detection. We also provide an online demo of fast, soft search across corpora in seven languages.

Figures

Figures reproduced from arXiv: 2602.10908 by the authors.

Figure 1
Figure 1. An example of search in SoftMatcha 2, which performs a soft search for trillion-scale corpora within 0.3 seconds, including word substitution, insertion, and deletion. including deep neural networks (Fukushima, 1980; Hinton et al., 2006; Vaswani et al., 2017), gradient descent-based optimization (Rumelhart et al., 1986; Bottou, 2010; Kingma and Ba, 2014), and self-supervised learning (Bengio et al., 2013; GutmannMic… view at source ↗
Figure 2
Figure 2. The sketch of our soft searching algorithm, when the query is “olympics gold medal”. Without iterative pruning, we must search the gray-striped zone as well as the blue zone. 3.3 Dynamic Corpus-Aware Pruning of Search Space To reduce the exponential growth of the search space in soft pattern matching, we employ iterative pruning, which drastically reduces the number of candidate patterns by ex￾ploiting the power-law… view at source ↗
Figure 3
Figure 3. The p95 (95th-percentile) latency of exact search for FineWeb-Edu dataset (1.4T tokens). infini-gram mini had reached timeout of index construction for larger corpora, and an error occurred for smaller corpora. 100M 10B 1T 100ms 1s 10s 100s Soft, P95, EN 100M 10B 1T Soft, P95, JA 10M 1B 100B Soft, P95, ZH Corpus Size (#tokens) Search Time SoftMatcha SoftMatcha (Disk) Ours [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: The p95 (95th-percentile) latency of soft search for EN (FineWeb-Edu, 1.4T tokens), JA (C4 Japanese, 169B tokens), and ZH (C4 Chinese, 38.3B tokens). SoftMatcha had reached memory limit, timeout of index construction, or errors for larger corpora. spelling. In the seco…
Figure 5
Figure 5. Figure 5: The number of exact string matching lookups with and without enabling the pruning techniques over the FineWeb-Edu dataset (1.4T-token dataset and 436M-token subsampled dataset). No data is displayed if a timeout (10 sec.) occurred. 5.5 Ablation Study We evaluated the e…
Figure 6
Figure 6. Figure 6: The number of problems flagged as dirty, including the breakdown of contamination. The numbers inside the brackets are the number of samples per dataset. For a dataset of more than 400 samples, we randomly subsampled to 400 samples. 6 Application: Contamination Detec￾t…
Figure 7
Figure 7. Figure 7: Screenshot of a search for “olympics gold medalist”, showing example matches. In this example, the demo returns the top 20 patterns in 55 ms [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Screenshot of a search for “ai may replace fifty percent of jobs”. Using extended search mode, the demo returns 40 patterns with similarity ≥ 0.27. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: The zero-hits ratio (i.e., the percentage of test data where no match was found) for FineWeb-Edu dataset on soft search and exact search, out of 400 English queries. with 5.8% (23/400) for exact search.10 E Additional Quantitative Evalua￾tions Median Latency. Figures 1…
Figure 11
Figure 11. Figure 11: The median latency on soft search for EN (FineWeb-Edu, 1.4T tokens), JA (C4 Japanese, 169B tokens), and ZH (C4 Chinese, 38.3B tokens) datasets. SoftMatcha reached memory limit, timeout of index construction, or other errors (only for ZH) for larger corpora. 5 10 20 40…
Figure 12
Figure 12. Figure 12: Latency when outputting top K patterns with different similarity thresholds (0.45 or 0.20), for FineWeb-Edu dataset. minimum similarity of 0.2, we achieved a median latency of 127.20 ms and a p95 latency of 5697.39 ms. Disk Usage. As described in §3.2, our implementat…
Figure 14
Figure 14. Figure 14: The sketch of our fast suffix array for the text “pattern match for a trillion scale corpus”, with L = 2 and B = 3. to reduce the number of random disk accesses. Let L be the maximum query length (in our experiments, we used L = 12). The first step is to record a sort…
Figure 15
Figure 15. Figure 15: The improvement on p95 and p99 latencies when we applied pruning for FineWeb-Edu dataset. Note that if we do not even apply the iterative pruning, both p95 and p99 latencies were 10+ sec (timeout) for all of the corpus sizes. Proof. By induction over i. Basics. We use…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

80 extracted references · 7 canonical work pages

  1. [1]

    Adjeroh, T

    D. Adjeroh, T. Bell, and A. Mukherjee. Exact and approximate pattern matching, pages 187--263. Springer, Boston, MA, 2008. ISBN 978-0-387-78909-5. doi:10.1007/978-0-387-78909-5_7. URL https://doi.org/10.1007/978-0-387-78909-5_7

    work page doi:10.1007/978-0-387-78909-5_7 2008

  2. [2]

    Akyurek, T

    E. Akyurek, T. Bolukbasi, F. Liu, B. Xiong, I. Tenney, J. Andreas, and K. Guu. Towards tracing knowledge in language models back to the training data . In Findings of the Association for Computational Linguistics: EMNLP 2022 , pages 2429--2446. Association for Computational Linguistics, Dec. 2022. doi:10.18653/v1/2022.findings-emnlp.180. URL http://dx.doi...

    doi:10.18653/v1/2022.findings-emnlp.180 2022

  3. [3]

    Andoni and P

    A. Andoni and P. Indyk. Near-optimal hashing algorithms for approximate nearest neighbor in high dimensions . Commun. ACM, 51 0 (1): 0 117--122, 1 Jan. 2008. doi:10.1145/1327452.1327494. URL http://dx.doi.org/10.1145/1327452.1327494

    arXiv 2008

  4. [4]

    Baeza-Yates and G

    R. Baeza-Yates and G. Navarro. Block addressing indices for approximate text retrieval. page 1–8, 1997. doi:10.1145/266714.266719. URL https://doi.org/10.1145/266714.266719

    arXiv 1997

  5. [5]

    Bellina and V

    A. Bellina and V. D. P. Servedio. Cognitive limits shape language statistics, 2025. URL https://arxiv.org/abs/2503.17512

    arXiv 2025

  6. [6]

    Bengio, A

    Y. Bengio, A. Courville, and P. Vincent. Representation learning: a review and new perspectives . IEEE Trans. Pattern Anal. Mach. Intell., 35 0 (8): 0 1798--1828, 1 Aug. 2013. doi:10.1109/TPAMI.2013.50. URL http://dx.doi.org/10.1109/TPAMI.2013.50

    doi:10.1109/tpami.2013.50 2013

  7. [7]

    DWDS -- Digitales W \"o rterbuch der deutschen Sprache

    Berlin-Brandenburgische Akademie der Wissenschaften . DWDS -- Digitales W \"o rterbuch der deutschen Sprache. Das Wortauskunftssystem zur deutschen Sprache in Geschichte und Gegenwart . https://www.dwds.de/, 2026. abgerufen am 28.01.2026

    2026

  8. [8]

    L. Bottou. Large-scale machine learning with stochastic gradient descent . In Proceedings of COMPSTAT'2010 , pages 177--186. Physica-Verlag HD, 2010. doi:10.1007/978-3-7908-2604-3\_16. URL http://dx.doi.org/10.1007/978-3-7908-2604-3_16

    work page doi:10.1007/978-3-7908-2604-3 2010

Show all 80 references
  1. [9]

    S. Ceri, A. Bozzon, M. Brambilla, E. Della Valle, P. Fraternali, and S. Quarteroni. An Introduction to Information Retrieval . In Web Information Retrieval , pages 3--11. Springer Berlin Heidelberg, 2013. doi:10.1007/978-3-642-39314-3\_1. URL http://dx.doi.org/10.5555/1394399

    2013 doi

  2. [10]

    Chang, J

    F. Chang, J. Dean, S. Ghemawat, W. C. Hsieh, D. A. Wallach, M. Burrows, T. Chandra, A. Fikes, and R. E. Gruber. Bigtable: A distributed storage system for structured data. ACM Transactions on Computer Systems (TOCS), 26 0 (2): 0 1--26, 2008

    2008

  3. [11]

    M. Chen. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  4. [12]

    Clark, I

    P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  5. [13]

    Cobbe, V

    K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    2021 arXiv

  6. [14]

    Deguchi, G

    H. Deguchi, G. Kamoda, Y. Matsushita, C. Taguchi, K. Suenaga, M. Waga, and S. Yokoi. SoftMatcha: A Soft and Fast Pattern Matcher for Billion-Scale Corpus Searches . In The Thirteenth International Conference on Learning Representations , 4 Mar. 2025. URL https://openreview.net...

    2025

  7. [15]

    C. Deng, Y. Zhao, X. Tang, M. Gerstein, and A. Cohan. Benchmark Probing: Investigating Data Leakage in Large Language Models . In NeurIPS 2023 Workshop on Backdoors in Deep Learning - The Good, the Bad, and the Ugly , 28 Oct. 2023. URL https://openreview.net/forum?id=a34bgvner1

    2023

  8. [16]

    Devlin, M.-W

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding . In Proceedings of the 2019 Conference of the North , pages 4171--4186. Association for Computational Linguistics, 2019. doi:10.18653/v1/n19-1423...

    2019 doi

  9. [17]

    Dodge, M

    J. Dodge, M. Sap, A. Marasović, W. Agnew, G. Ilharco, D. Groeneveld, M. Mitchell, and M. Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , ...

    2021 doi

  10. [18]

    Douze, A

    M. Douze, A. Guzhva, C. Deng, J. Johnson, G. Szilvasy, P.-E. Mazaré, M. Lomeli, L. Hosseini, and H. Jégou. The Faiss library . arXiv [cs.LG], 23 Oct. 2025. doi:10.48550/arXiv.2401.08281. URL http://arxiv.org/abs/2401.08281

    2025 doi arXiv

  11. [19]

    Ferragina and G

    P. Ferragina and G. Manzini. Indexing compressed text. J. ACM , 52 0 (4): 0 552--581, 2005. doi:10.1145/1082036.1082039. URL https://doi.org/10.1145/1082036.1082039

    2005

  12. [20]

    Fukushima

    K. Fukushima. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position . Biological Cybernetics, 36: 0 193--202, 1980. URL https://api.semanticscholar.org/CorpusID:206775608

    1980

  13. [21]

    T. Gao, X. Yao, and D. Chen. SimCSE: Simple Contrastive Learning of Sentence Embeddings . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 6894--6910. Association for Computational Linguistics, Nov. 2021. doi:10.18653/v1/2021.em...

    2021 doi

  14. [22]

    Kamath, J

    Gemma Team , A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ramé, M. Rivière, L. Rouillard, T. Mesnard, G. Cideron, J.-B. Grill, S. Ramos, E. Yvinec, M. Casbon, E. Pot, I. Penchev, G. Liu, F. Visin, K. Kenealy, L. Beyer, X. Zhai, A. Ts...

    2025 doi arXiv

  15. [23]

    Gokaslan, V

    A. Gokaslan, V. Cohen, E. Pavlick, and S. Tellex. OpenWebText Corpus . http://Skylion007.github.io/OpenWebTextCorpus, 2019

    2019

  16. [24]

    Gormley and Z

    C. Gormley and Z. Tong. Elasticsearch: The Definitive Guide . O'Reilly Media, 30 Jan. 2015

    2015

  17. [25]

    Grave, P

    E. Grave, P. Bojanowski, P. Gupta, A. Joulin, and T. Mikolov. Learning Word Vectors for 157 Languages . In N. Calzolari, K. Choukri, C. Cieri, T. Declerck, S. Goggi, K. Hasida, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, S. Piperidis, and T. Tokunaga, ed...

    2018

  18. [26]

    Z. Guo, M. Schlichtkrull, and A. Vlachos. A survey on automated fact-checking . Trans. Assoc. Comput. Linguist., 10: 0 178--206, 9 Feb. 2022. doi:10.1162/tacl\_a\_00454. URL http://dx.doi.org/10.1162/tacl_a_00454

    2022 doi

  19. [27]

    GutmannMichael and HyvärinenAapo

    U. GutmannMichael and HyvärinenAapo . Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics . J. Mach. Learn. Res., 1 Feb. 2012. doi:10.5555/2188385.2188396. URL http://dx.doi.org/10.5555/2188385.2188396

    2012

  20. [28]

    Hendrycks, C

    D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  21. [29]

    Hendrycks, C

    D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  22. [30]

    G. E. Hinton, S. Osindero, and Y.-W. Teh. A fast learning algorithm for deep belief nets . Neural Comput., 18 0 (7): 0 1527--1554, July 2006. doi:10.1162/neco.2006.18.7.1527. URL http://dx.doi.org/10.1162/neco.2006.18.7.1527

    2006 doi

  23. [31]

    Indyk and R

    P. Indyk and R. Motwani. Approximate nearest neighbors: towards removing the curse of dimensionality . In Proceedings of the thirtieth annual ACM symposium on Theory of computing - STOC '98 , pages 604--613. ACM Press, 1998. doi:10.1145/276698.276876. URL http://dx.doi.org/10....

    1998

  24. [32]

    Ippolito, F

    D. Ippolito, F. Tramer, M. Nasr, C. Zhang, M. Jagielski, K. Lee, C. Choquette Choo, and N. Carlini. Preventing generation of verbatim memorization in language models gives a false sense of privacy . In Proceedings of the 16th International Natural Language Generation Conferenc...

    2023 doi

  25. [33]

    Kaplan, S

    J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models . arXiv [cs.LG], 22 Jan. 2020. doi:10.48550/arXiv.2001.08361. URL http://arxiv.org/abs/2001.08361

    2020 doi arXiv

  26. [34]

    Karpukhin, B

    V. Karpukhin, B. Oguz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W.-T. Yih. Dense passage retrieval for open-domain question answering . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 6769--6781. Association for...

    2020 doi

  27. [35]

    B. W. Kernighan and M. M. Douglas. UNIX Programmer's Manual (Seventh Edition) . Bell Telephone Laboratories, 1979

    1979

  28. [36]

    D. P. Kingma and J. Ba. Adam: A method for stochastic optimization . arXiv [cs.LG], 22 Dec. 2014. doi:10.48550/arXiv.1412.6980. URL http://arxiv.org/abs/1412.6980

    2014 doi arXiv

  29. [37]

    D. Knight. Corpus linguistics: Methods, theory and practice by Tony McEnery and Andrew Hardie . In Yearbook of Corpus Linguistics and Pragmatics 2013 , volume 1, pages 275--277. Springer Netherlands, 9 Apr. 2013. doi:10.1007/978-94-007-6250-3\_13. URL http://dx.doi.org/10.1007...

    2013 doi

  30. [38]

    Koehn, H

    P. Koehn, H. Hoang, A. Birch, C. Callison-Burch, M. Federico, N. Bertoldi, B. Cowan, W. Shen, C. Moran, R. Zens, C. Dyer, O. Bojar, A. Constantin, and E. Herbst. M oses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the A...

    2007

  31. [39]

    T. Kudo, H. Kazawa, and Y. Matsumoto. Japanese morphological analysis with conditional random fields. In Proceedings of the 10th Annual Meeting of the Association for Natural Language Processing, pages 592--595, 2004. (in Japanese)

    2004

  32. [40]

    Kučera and W

    H. Kučera and W. N. Francis. Computational Analysis of Present-Day American English . Brown University Press, 1967

    1967

  33. [41]

    Lewis, E

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-T. Yih, T. Rocktäschel, S. Riedel, and D. Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks . arXiv [cs.CL], 22 May 2020. URL http://arxiv.org/abs/2005.11401

    2020 arXiv

  34. [42]

    BROWN Corpus

    Lexical Computing . BROWN Corpus . https://www.sketchengine.eu/brown-corpus/, 26 May 2015

    2015

  35. [43]

    J. Liu, S. Min, L. Zettlemoyer, Y. Choi, and H. Hajishirzi. Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens . In First Conference on Language Modeling , 26 Aug. 2024. URL https://openreview.net/forum?id=u2vAyMeLMm

    2024

  36. [44]

    Lozhkov, L

    A. Lozhkov, L. Ben Allal, L. von Werra, and T. Wolf. FineWeb-Edu: the Finest Collection of Educational Content , 2024. URL http://dx.doi.org/10.57967/hf/2497

    2024 doi

  37. [45]

    H. P. Luhn. Key word‐in‐context index for technical literature (kwic index) . Am. Doc., 11 0 (4): 0 288--295, 1 Oct. 1960. doi:10.1002/asi.5090110403. URL http://dx.doi.org/10.1002/asi.5090110403

    1960 doi

  38. [46]

    Y. A. Malkov and D. A. Yashunin. Efficient and robust approximate nearest neighbor search using hierarchical Navigable Small World graphs . IEEE Trans. Pattern Anal. Mach. Intell., 42 0 (4): 0 824--836, 1 Apr. 2020. doi:10.1109/TPAMI.2018.2889473. URL http://dx.doi.org/10.1109...

    2020

  39. [47]

    Manber and G

    U. Manber and G. Myers. Suffix arrays: A new method for on-line string searches. In Proceedings of the First Annual ACM-SIAM Symposium on Discrete Algorithms, SODA '90, page 319–327, USA, 1990. Society for Industrial and Applied Mathematics. ISBN 0898712513

    1990

  40. [48]

    Manning and H

    C. Manning and H. Schuetze. Foundations of statistical natural language processing . Foundations of Statistical Natural Language Processing. MIT Press, 28 May 1999

    1999

  41. [49]

    The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation

    Meta . The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation . https://ai.meta.com/blog/llama-4-multimodal-intelligence/, 25 Apr. 2025

    2025

  42. [50]

    Mikolov, K

    T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations in vector space . arXiv [cs.CL], 16 Jan. 2013. URL http://arxiv.org/abs/1301.3781

    2013 arXiv

  43. [51]

    NINJAL . 中納言 . https://chunagon.ninjal.ac.jp/, 2011

    2011

  44. [52]

    Oyama, S

    M. Oyama, S. Yokoi, and H. Shimodaira. Norm of word embedding encodes information gain . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 2108--2130. Association for Computational Linguistics, Dec. 2023. doi:10.18653/v1/2023.emn...

    2023 doi

  45. [53]

    Parker, D

    R. Parker, D. Graff, J. Kong, K. Chen, and K. Maeda. English Gigaword Fifth Edition . Web Download, 2011. URL http://dx.doi.org/10.35111/wk4f-qt80

    2011 doi

  46. [54]

    Penedo, H

    G. Penedo, H. Kydl \' c ek, A. Lozhkov, M. Mitchell, C. A. Raffel, L. Von Werra, T. Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 37: 0 30811--30849, 2024 a

    2024

  47. [55]

    Penedo, H

    G. Penedo, H. Kydlíček, L. B. Allal, A. Lozhkov, M. Mitchell, C. Raffel, L. V. Werra, and T. Wolf. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale . In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track ...

    2024

  48. [56]

    Pennington, R

    J. Pennington, R. Socher, and C. D. Manning. GloVe: Global Vectors for Word Representation . In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 1532--1543, Oct. 2014. doi:10.3115/v1/D14-1162. URL https://aclanthology.org/D...

    2014 doi

  49. [57]

    Radford, J

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language Models are Unsupervised Multitask Learners . OpenAI blog, 1 0 (8): 0 9, 2019. URL https://insightcivic.s3.us-east-1.amazonaws.com/language-models.pdf

    2019

  50. [58]

    Raffel, N

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer . arXiv [cs.LG], 23 Oct. 2019. URL http://arxiv.org/abs/1910.10683

    2019 arXiv

  51. [59]

    Raffel, N

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 0 (140): 0 1--67, 2020

    2020

  52. [60]

    Reimers and I

    N. Reimers and I. Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT-networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pag...

    2019 doi

  53. [61]

    D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024

    2024

  54. [62]

    S. E. Robertson, S. Walker, S. Jones, M. Hancock-Beaulieu, and M. Gatford. Okapi at TREC-3 . In D. K. Harman, editor, Proceedings of The Third Text REtrieval Conference, TREC 1994, Gaithersburg, Maryland, USA, November 2-4, 1994 , volume 500-225 of NIST Special Publication, pa...

    1994

  55. [63]

    D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning representations by back-propagating errors . Nature, 323 0 (6088): 0 533--536, Oct. 1986. doi:10.1038/323533a0. URL http://dx.doi.org/10.1038/323533a0

    1986 doi

  56. [64]

    Salton and C

    G. Salton and C. Buckley. Term-weighting approaches in automatic text retrieval . Inf. Process. Manag., 24 0 (5): 0 513--523, 1 Jan. 1988. doi:10.1016/0306-4573(88)90021-0. URL http://dx.doi.org/10.1016/0306-4573(88)90021-0

    1988 doi

  57. [65]

    J. F. Silva and J. C. Cunha. An empirical model for n-gram frequency distribution in large corpora. In Advances in Knowledge Discovery and Data Mining: 24th Pacific-Asia Conference, PAKDD 2020, Singapore, May 11–14, 2020, Proceedings, Part II, page 840–851, Berlin, Heidelberg,...

    2020 doi

  58. [66]

    Singh, A

    A. Singh, A. Fry, A. Perelman, A. Tart, A. Ganesh, A. El-Kishky, A. McLaughlin, A. Low, A. J. Ostrow, A. Ananthram, A. Nathan, A. Luo, A. Helyar, A. Madry, A. Efremov, A. Spyra, A. Baker-Whitcomb, A. Beutel, A. Karpenko, A. Makelov, A. Neitz, A. Wei, A. Barr, A. Kirchmeyer, A....

    2025 doi arXiv

  59. [67]

    Soldaini, R

    L. Soldaini, R. Kinney, A. Bhagia, D. Schwenk, D. Atkinson, R. Authur, B. Bogin, K. Chandu, J. Dumas, Y. Elazar, V. Hofmann, A. H. Jha, S. Kumar, L. Lucy, X. Lyu, N. Lambert, I. Magnusson, J. Morrison, N. Muennighoff, A. Naik, C. Nam, M. E. Peters, A. Ravichander, K. Richardso...

    2024

  60. [68]

    Y. Sun, Y. Gai, L. Chen, A. Ravichander, Y. Choi, N. Dziri, and D. Song. Why and How LLMs Hallucinate: Connecting the Dots with Subsequence Associations . In The Thirty-ninth Annual Conference on Neural Information Processing Systems , 29 Oct. 2025. URL https://openreview.net/...

    2025

  61. [69]

    Sutskever, O

    I. Sutskever, O. Vinyals, and Q. V. Le. Sequence to Sequence Learning with Neural Networks . In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Pr...

    2014

  62. [70]

    International components for unicode

    The ICU Project . International components for unicode. https://icu.unicode.org/, 2024

    2024

  63. [71]

    Touvron, L

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  64. [72]

    D. C. van Leijenhorst and T. P. van der Weide. A formal derivation of heaps' law. Inf. Sci., 170 0 (2-4): 0 263--272, 2005. doi:10.1016/J.INS.2004.03.006. URL https://doi.org/10.1016/j.ins.2004.03.006

    2005 doi

  65. [73]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, . Kaiser, and I. Polosukhin. Attention is All you Need . Advances in Neural Information Processing Systems, 30, 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1...

    2017

  66. [74]

    R. A. Wagner and M. J. Fischer. The string-to-string correction problem. J. ACM , 21 0 (1): 0 168--173, 1974. doi:10.1145/321796.321811. URL https://doi.org/10.1145/321796.321811

    1974

  67. [75]

    Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. Advances in Neural Information Processing Systems, 37: 0 95266--95290, 2024

    2024

  68. [76]

    Wu and U

    S. Wu and U. Manber. Agrep--a fast approximate pattern-matching tool. In Usenix Winter 1992 Technical Conference, pages 153--162, 1992

    1992

  69. [77]

    H. Xu, J. Liu, Y. Choi, N. A. Smith, and H. Hajishirzi. Infini-gram mini: Exact n-gram Search at the Internet Scale with FM-Index . In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages 24955--24980. Association for Computational Lin...

    2025 doi

  70. [78]

    R. Xu, Z. Wang, R.-Z. Fan, and P. Liu. Benchmarking benchmark leakage in Large Language Models . arXiv [cs.CL], 29 Apr. 2024. doi:10.48550/arXiv.2404.18824. URL http://arxiv.org/abs/2404.18824

    2024 doi arXiv

  71. [79]

    Yokoi, H

    S. Yokoi, H. Bao, H. Kurita, and H. Shimodaira. Zipfian Whitening . In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 6 Nov. 2024. URL https://openreview.net/forum?id=pASJxzMJb7

    2024

  72. [80]

    Zobel and A

    J. Zobel and A. Moffat. Inverted files for text search engines. ACM Comput. Surv. , 38 0 (2): 0 6, 2006. doi:10.1145/1132956.1132959. URL https://doi.org/10.1145/1132956.1132959

    2006

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.