Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Minerva: A Programmable Memory Test Benchmark for Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read At only 4k tokens, language models that ace memory search fail at editing, counting, and state tracking.

desk verdict A genuinely broader memory benchmark whose headline pattern (search good, state tracking bad) is credible, but the near-zero quantity-state scores may partly be output-format artifacts. read the letter →

arxiv 2502.03358 v2 pith:MMUZEXYL submitted 2025-02-05 cs.CL

Menglin Xia , Victor Ruehle , Saravan Rajmohan , Reza Shokri This is my paper · ORCID
classification cs.CL
keywords memorybenchmarkcontextutilizationlargelanguagemodelsatomictestscompositeprogrammablestatefulprocessingneedle-in-a-haystack
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that current LLM memory evaluation over-relies on search tasks like needle-in-a-haystack and thereby misses most of what models do with context. It presents Minerva, a programmable benchmark that generates randomized test cases for atomic capabilities — searching, recalling and editing, matching and comparing, spotting differences, computing over sets and lists, and stateful tracking — plus composite tasks that combine them. Across nine models at 4k tokens, models score high on search but show wide and task-specific gaps, with near-zero performance for several open models on stateful processing and large drops on composite tasks. The authors' central point is that strong retrieval does not imply usable context memory, so benchmarks must dissect capabilities to guide training.

What carries the argument

The Minerva framework consists of parametric test templates for each atomic capability (string search, key-value and batch search, snapshot, replace-all, overwrite positions, functional updates, compare positions, find duplicates, count, check association, compare two lists, identify the odd group, patch the difference, group membership, group association, iterate, and quantity and set state tracking), each with configurable hyperparameters and randomly sampled dictionary words. It also defines composite tests — processing data blocks and composite-state tracking (theory of mind) — that require coordinating several atomic abilities. The framework isolates capabilities by generating fresh randomized instances and scoring them with exact match, ROUGE-L, or Jaccard similarity, enabling adjacent tasks that differ in one requirement to pinpoint where a model fails.

What would settle it

Rerun the Snapshot, Iterate, Replace-all, and Functional-updates tasks using normalized exact-match scoring (lowercased, whitespace-collapsed, punctuation-stripped) and with a second prompt that explicitly specifies the exact output format; if scores rise sharply, then format compliance, not memory capability, explains part of the reported gaps.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that at just 4k tokens — a context length where many models are assumed near perfect — performance varies strongly by capability category rather than by model scale. Search tasks are comparatively easy; tasks requiring edits, counting, comparisons, partition awareness, and sequential state updates degrade sharply, and composite tasks that combine atomic abilities produce uniformly low scores. The authors present this as evidence that simple retrieval performance is not a proxy for context processing and that search-only benchmarks give an incomplete picture of model memory.

Load-bearing premise

The load-bearing premise is that each atomic test isolates a single memory capability, so a low score reflects a missing capability rather than confounds such as instruction comprehension, output formatting, or tokenization of random dictionary words.

Editorial extensions

If this is right

  • Search-only evaluations (e.g., NIAH-style tests) overstate memory ability: at 4k tokens many models score near perfect on search while failing on other memory tasks.
  • Capability gaps appear well before long contexts, so failures at 4k mean context length alone is not the bottleneck for non-search memory use.
  • Model scale does not explain the pattern: an 8B model can outperform larger models on some comparative tasks, and even the strongest models show uneven performance across categories.
  • Composite tasks drop all models, including GPT-4-turbo and GPT-4o, indicating that integrating multiple memory capabilities is a distinct and largely unsolved challenge.
  • Programmable generation of fresh test cases and hyperparameter sweeps reduces the risks of static benchmarks, including overfitting and memorization of specific test instances.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to test whether the reported gaps shrink under chain-of-thought prompting or instruction fine-tuning; the paper's own prompt-variation results suggest instruction comprehension is not the main cause, but they do not rule out reasoning-time strategies.
  • Because ROUGE-L is used for exact-recall tasks like Snapshot, Iterate, and Replace-all, some low scores may reflect output-format deviations rather than memory failure; a normalized exact-match rescoring would separate these two failure modes.
  • The atomic test design could be ported to agentic or tool-using settings where 'memory' is a structured state rather than flat text, to see whether the same capability hierarchy holds when models can offload storage.
  • The theory-of-mind composite task approximates multi-agent belief tracking, so its results could be cross-checked against dedicated social reasoning benchmarks to see whether the bottleneck is state tracking or perspective-taking.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces Minerva, a programmable benchmark for evaluating LLM context/memory utilization. It defines a set of atomic task families (search, recall and edit, match and compare, spot the differences, compute on sets and lists, stateful processing) and two composite families (processing data blocks and composite-state tracking), generates randomized instances from parameterized templates, and evaluates nine black-box and open-source models on a fixed snapshot of 1110 examples, mostly at 4k tokens (with stateful tasks at roughly 1.5k tokens). The central finding is that models perform relatively well on search tasks but show large and systematic gaps on other memory capabilities, especially state tracking, and that composite tasks are substantially harder. The paper also reports ablations with gibberish context and prompt variations, and an error-pattern analysis for subsequence search.

Significance. If the results hold, this is a useful benchmark contribution: it goes beyond NIAH-style retrieval tests, provides a programmable generator with fresh random instances, and offers a more interpretable capability decomposition than static end-to-end benchmarks. The gibberish-context and prompt-variation ablations are good construct-validity checks for the tasks they cover, and the subsequence-search false-positive/false-negative analysis is a nice example of actionable diagnostics. The qualitative claim that strong retrieval performance does not imply strong context utilization is important and likely robust. However, the most dramatic quantitative claims, particularly the near-zero state-tracking scores, are not yet cleanly attributable to memory deficits because of exact-match scoring and the absence of error analysis.

major comments (4)
  1. [§3.3, Table 5; Appendix A (Stateful Processing)] The exact-match scoring of the Quantity state task makes a zero score ambiguous between a genuine state-tracking failure and an output-format deviation: the task requires the answer to appear after the literal prefix 'FINAL ANSWER:' and is scored by exact match (Appendix B). Table 5 reports 0.0 for five of the nine models on this task, and no error analysis is provided to separate 'correct total with different phrasing or prefix' from 'incorrect total.' The paper itself acknowledges in §3.4 that models 'tend to generate longer answers' in the set-state task, showing that response format is a real confound. Because the stateful category provides the starkest evidence for the central claim, please add a tolerant parser or a few-shot/format-controlled variant and report error breakdowns, or the near-zero scores should be described as format-sensitive rather than as pure memory deficits.
  2. [§3.1; Appendix B] Several load-bearing results rest on very small samples and are reported without confidence intervals: Quantity state uses 10 examples, Set state 40, Snapshot 10, and the length sweeps in Table 7 report no sample sizes. With n=10, a reported 0.0 has a 95% CI extending to roughly 0.31, so fine-grained distinctions such as 'GPT-4-turbo 0.8 vs. Cohere 0.58' in Table 5, or the near-zero stateful scores, are not statistically meaningful as presented. Please report confidence intervals or bootstrap estimates in the main tables, or increase the number of generated instances for the small-sample cells.
  3. [Abstract; §3.1; §3.2] The headline claim that models show disparities 'even at a context length of 4k tokens' is not accurate for the Stateful Processing category, which uses around 1.5k tokens (200/100 steps per §3.1) and is included in the overall summary of Figure 1. Since the stateful tasks drive much of the observed gap, please either qualify the 4k statement per category or run those tasks at 4k tokens so the headline reflects the experimental conditions.
  4. [§2; Table 3; Tables 8 and 12] The interpretability claim that the atomic tests isolate single capabilities is only weakly validated for most categories. The gibberish-context ablation (Table 3) covers only Snapshot and String Search, and the prompt-variation ablations (Tables 8 and 12) cover a handful of tasks, not Compare Two Lists, Patch the Difference, the composite tests, or the Set/Quantity variants. Without per-task error analysis or additional controls, low scores cannot be cleanly attributed to the intended memory capability rather than to instruction ambiguity, tokenization of random dictionary words, or output formatting. Please add at least a per-category error analysis or state this limitation explicitly for each category.
minor comments (6)
  1. [Appendix C; Tables 3, 4, 11] Tables label the metric simply 'ROUGE-L' without saying whether ROUGE-L F1 or ROUGE-L recall is used, although Appendix C defines both. Because recall ignores extra tokens while F1 penalizes them, the chosen variant affects scores on tasks like Snapshot and Replace All; please state the variant in each table caption.
  2. [Table 4] The column header 'Iterate (last)' and the parenthetical comparison labels are unclear: §3.3 describes an ablation that asks for first words instead of last, but Table 4 does not report that baseline, and the parenthetical in the last column appears self-referential. Please clarify what each parenthetical value is compared against and add the first-word baseline.
  3. [Appendix A (Theory of Mind)] In the Theory of Mind template, 'Agent B starts with the following words:' appears twice; the third occurrence should presumably refer to Agent C. Please fix the template so the three agents are distinct.
  4. [Tables 2 and 5] Model naming is inconsistent ('cohere-command-rplus' in Table 2, 'cohere' in Table 5), and Table 2 prints '-0.00' for zero differences; please use one canonical name and '0.00' for zero values.
  5. [Appendix D] The second prompt variant for the Check association task is given only as 'Given the context', which appears truncated; please provide the complete variant used for the results in Table 12.
  6. [§3.1] Please report exact model versions and API snapshot dates (e.g., gpt-4o-2024-05-13 rather than 'gpt-4o') and the same decoding settings for all models, since API models change over time and this affects reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: Minerva is an operational benchmark with no fitted parameters and no self-citation load-bearing steps.

full rationale

The paper does not derive its claims from a fitted model or from prior results by the same authors. Its central claim—that models perform well on simple search tasks but exhibit significant disparities across context-utilization capabilities at 4k tokens—comes directly from evaluating nine models on 1110 randomly generated test cases, with each test scored by exact match, ROUGE-L, or Jaccard similarity. Capability categories are defined operationally through the test templates, and the paper explicitly hedges that atomic tests isolate capabilities only 'to the extent that isolating such capabilities is possible.' This is a standard benchmark-construction assumption, not a circular derivation: a low score is interpreted as evidence about a capability, but the benchmark does not define a capability as whatever the test measures in a way that makes the conclusion true by definition. No parameter is fitted to the data and then renamed a prediction, and no uniqueness theorem or prior work by these authors is invoked as load-bearing evidence. Potential concerns about exact-match scoring on Stateful Processing conflating format non-compliance with memory failure are measurement-validity issues, not circularity; they do not reduce any claimed result to its own inputs. The evaluation is self-contained against external benchmarks and the paper's conclusions rest on direct measurement, so no significant circularity is present.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim does not rest on fitted parameters or derived constants. The listed free parameters are experimental design choices that shape the measurements but are not optimized to force a conclusion. The axioms are standard benchmark assumptions about synthetic data and metric validity.

free parameters (4)
  • context_length = 4,000 tokens for main results; up to 32,000 in ablations
    Hand-chosen as the fixed test length; the claim of failure at short context depends on this value.
  • quantity_state_operation_steps = 200
    Number of add/subtract operations used in the quantity state test; small sample size and specific step count affect the near-zero scores.
  • set_state_operation_steps = 100
    Number of draw/discard operations in the set state test, controlling context length around 1.5k tokens.
  • sample_counts_per_parameter_setting = 5 or 10 per measured cell
    The measured accuracies, for example n=10 for quantity state and n=40 for set state, have wide confidence intervals; results could shift with more samples.
assumptions (3)
  • domain assumption Uniform random sampling of words from an English dictionary yields contexts that are representative for probing memory abilities.
    The generated contexts are synthetic; the validity of the benchmark for drawing conclusions about real-world memory use depends on this.
  • domain assumption Exact match, ROUGE-L, and Jaccard similarity are valid scoring rules for the intended capabilities.
    For example, ROUGE-L is used for tasks that require verbatim reproduction of edited text, but it also penalizes formatting deviations, so it may not cleanly separate content errors from format errors.
  • domain assumption Temperature-0 decoding makes single-run evaluation deterministic and adequate for model comparison.
    The paper sets temperature 0 and top p 1 for all models, but does not discuss how sensitive results are to sampling; other seeds or snapshots might change numbers.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Minerva: A Programmable Memory Test Benchmark for Language Models." pith.science (2026). https://pith.science/paper/MMUZEXYL

@misc{pith2026250203358,
  author       = {Pith},
  title        = {Pith review of: Minerva: A Programmable Memory Test Benchmark for Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MMUZEXYL}},
  note         = {Machine review of arXiv:2502.03358}
}
read the original abstract

How effectively can LLM-based AI assistants utilize their memory (context) to perform various tasks? Traditional data benchmarks, which are often manually crafted, suffer from several limitations: they are static, susceptible to overfitting, difficult to interpret, and lack actionable insights--failing to pinpoint the specific capabilities a model lacks when it does not pass a test. In this paper, we present a framework for automatically generating a comprehensive set of tests to evaluate models' abilities to use their memory effectively. Our framework extends the range of capability tests beyond the commonly explored (passkey, key-value, needle in the haystack) search, a dominant focus in the literature. Specifically, we evaluate models on atomic tasks such as searching, recalling, editing, matching, comparing information in context memory, performing basic operations when inputs are structured into distinct blocks, and maintaining state while operating on memory, simulating real-world data. Additionally, we design composite tests to investigate the models' ability to perform more complex, integrated tasks. Our benchmark enables an interpretable, detailed assessment of memory capabilities of LLMs.

Figures

Figures reproduced from arXiv: 2502.03358 by the authors.

Figure 1
Figure 1. Overall performance of nine models on a snapshot within 4k context length of Minerva. Models Word Subsequence Key-value Batch gpt-4-turbo 0.94 0.94 (-0.00) 1.00 1.00 (-0.00) gpt-4o 1.00 0.82 (-0.18) 1.00 1.00 (-0.00) gpt-4o-mini 0.98 0.64 (-0.34) 1.00 0.96 (-0.04) cohere-command-rplus 1.00 0.85 (-0.15) 0.98 0.87 (-0.11) mistral-7b 0.78 0.80 (+0.02) 0.92 0.47 (-0.45) phi-3-small 0.94 0.84 (-0.10) 0.94 0.77 (-0.17) ph… view at source ↗
Figure 2
Figure 2. Analysis on String Search (with subsequence) across increasing subsequence lengths. This figure examines the behavior of models on positive samples (where the subsequence is present) and negative samples (where the subsequence is absent). that these models frequently deviated from instructions over longer sequences, suggesting difficulties in maintaining con￾sistent rule applications over extended contexts. Addition… view at source ↗
Figure 3
Figure 3. Results for the Recall and Edit tasks. Compare positions Find duplicates Count Check association 0 0.2 0.4 0.6 0.8 1 gpt-4-turbo gpt-4o gpt-4o-mini cohere Compare positions Find duplicates Count Check association 0 0.2 0.4 0.6 0.8 1 mistral-7b phi-3-small phi-3-medium gemma-2-9b llama-3.1-8b [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Results for Spot the Differences tasks. Spot the Differences As shown in [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Effect of context length (number of operation steps) on performance in the quantity state (left) and set state (right) tasks. String Search task, where the former requires identifying which list a word belongs to rather than simply determining its presence, the perform…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SCOPE: Stochastic and Counterbiased Option Placement for Evaluating Large Language Models

    cs.CL 2025-07 reject novelty 4.0 of 10

    SCOPE estimates a model's position bias with nonsense prompts, puts correct answers in disliked slots, and spreads similar distractors apart to cap lucky guessing.

Reference graph

Works this paper leans on

34 extracted references · 10 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    L -eval: Instituting standardized evaluation for long context language models

    An, C., Gong, S., Zhong, M., Zhao, X., Li, M., Zhang, J., Kong, L., and Qiu, X. L -eval: Instituting standardized evaluation for long context language models. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 14388--14411, Bangkok, Thail...

    doi:10.18653/v1/2024.acl-long.776 2024

  3. [3]

    Introducing the next generation of claude

    Anthropic. Introducing the next generation of claude. https://www.anthropic.com/news/claude-3-family, 2024. Accessed: 2024-03-27

    work page 2024

  4. [4]

    L ong B ench: A bilingual, multitask benchmark for long context understanding

    Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., Dong, Y., Tang, J., and Li, J. L ong B ench: A bilingual, multitask benchmark for long context understanding. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lo...

    doi:10.18653/v1/2024.acl-long.172 2024

  5. [5]

    Titans: Learning to memorize at test time, 2024

    Behrouz, A., Zhong, P., and Mirrokni, V. Titans: Learning to memorize at test time, 2024. URL https://arxiv.org/abs/2501.00663

    arXiv 2024

  6. [6]

    Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Ch...

    2021

  7. [7]

    Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URL https://arxiv.org/abs/1803.05457

    arXiv 2018

  8. [8]

    Training verifiers to solve math word problems

    Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

Show all 34 references
  1. [9]

    and Parrish, J

    Crannell, C. and Parrish, J. A comparison of immediate memory span for digits, letters, and words. The Journal of Psychology, 44 0 (2): 0 319--327, 1957

    1957

  2. [10]

    L., Zhang, C., Xu, Y., Shang, N., Xu, J., Yang, F., and Yang, M

    Ding, Y., Zhang, L. L., Zhang, C., Xu, Y., Shang, N., Xu, J., Yang, F., and Yang, M. Longrope: Extending llm context window beyond 2 million tokens, 2024

    2024

  3. [11]

    Samsum corpus: A human-annotated dialogue dataset for abstractive summarization

    Gliwa, B., Mochol, I., Biesek, M., and Wawer, A. Samsum corpus: A human-annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization. Association for Computational Linguistics, 2019. doi:10.18653/v1/d19-5409. UR...

    2019 doi

  4. [12]

    Measuring massive multitask language understanding

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021 a . URL https://openreview.net/forum?id=d7KBjmI3GmQ

    2021

  5. [13]

    Measuring mathematical problem solving with the math dataset

    Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021 b

    2021

  6. [14]

    RULER : What s the real context size of your long-context language models? In First Conference on Language Modeling, 2024

    Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., and Ginsburg, B. RULER : What s the real context size of your long-context language models? In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=kIoBbc76Sy

    2024

  7. [15]

    \'E tude comparative de la distribution florale dans une portion des alpes et des jura

    Jaccard, P. \'E tude comparative de la distribution florale dans une portion des alpes et des jura. Bull Soc Vaudoise Sci Nat, 37: 0 547--579, 1901

    1901

  8. [16]

    Needle in a haystack - pressure testing llms

    Kamradt, G. Needle in a haystack - pressure testing llms. GitHub, 2023. URL https://github.com/gkamradt/LLMTest_NeedleInAHaystack/tree/main

    2023

  9. [17]

    L., Roebuck-Spencer, T., Short, P., Kabat, M., and Wilken, J

    Kane, R. L., Roebuck-Spencer, T., Short, P., Kabat, M., and Wilken, J. Identifying and monitoring cognitive deficits in clinical populations using automated neuropsychological assessment metrics (anam) tests. Archives of Clinical Neuropsychology, 22 0 (Suppl\_1): 0 S115--S126, 2007

    2007

  10. [18]

    Natural questions: a benchmark for question answering research

    Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7: 0 453--...

    2019

  11. [19]

    Needlebench: Can llms do retrieval and reasoning in 1 million context window?, 2024

    Li, M., Zhang, S., Liu, Y., and Chen, K. Needlebench: Can llms do retrieval and reasoning in 1 million context window?, 2024. URL https://arxiv.org/abs/2407.11963

    2024

  12. [20]

    ROUGE : A package for automatic evaluation of summaries

    Lin, C.-Y. ROUGE : A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp.\ 74--81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https://aclanthology.org/W04-1013/

    2004

  13. [21]

    F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P

    Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12: 0 157--173, 2024. doi:10.1162/tacl_a_00638. URL https://acla...

    2024 doi

  14. [22]

    P., Santorini, B., and Marcinkiewicz, M

    Marcus, M. P., Santorini, B., and Marcinkiewicz, M. A. Building a large annotated corpus of E nglish: The P enn T reebank. Computational Linguistics, 19 0 (2): 0 313--330, 1993. URL https://aclanthology.org/J93-2004/

    1993

  15. [23]

    and Jaggi, M

    Mohtashami, A. and Jaggi, M. Random-access infinite context length for transformers. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=7eHn64wOVy

    2023

  16. [24]

    S., Phillips, N

    Nasreddine, Z. S., Phillips, N. A., B \'e dirian, V., Charbonneau, S., Whitehead, V., Collin, I., Cummings, J. L., and Chertkow, H. The montreal cognitive assessment, moca: a brief screening tool for mild cognitive impairment. Journal of the American Geriatrics Society, 53 0 (...

    2005

  17. [25]

    Counting-stars: A multi-evidence, position-aware, and scalable benchmark for evaluating long-context large language models, 2024

    Song, M., Zheng, M., and Luo, X. Counting-stars: A multi-evidence, position-aware, and scalable benchmark for evaluating long-context large language models, 2024. URL https://arxiv.org/abs/2403.11802

    2024 arXiv

  18. [26]

    W., Chowdhery, A., Le, Q

    Suzgun, M., Scales, N., Sch \"a rli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D., , and Wei, J. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022

    arXiv

  19. [27]

    N., Cowan, N., Hitch, G

    Towse, J. N., Cowan, N., Hitch, G. J., and Horton, N. J. The recall of information from working memory: Insights from behavioural and chronometric perspectives. Experimental Psychology, 55 0 (6): 0 371--383, 2008

    2008

  20. [28]

    Voyager: An open-ended embodied agent with large language models

    Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., and Anandkumar, A. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv: Arxiv-2305.16291, 2023

    2023 arXiv

  21. [29]

    H., White, R

    Wu, Q., Bansal, G., Zhang, J., Wu, Y., Li, B., Zhu, E., Jiang, L., Zhang, X., Zhang, S., Liu, J., Awadallah, A. H., White, R. W., Burger, D., and Wang, C. Autogen: Enabling next-gen llm applications via multi-agent conversation, 2023. URL https://arxiv.org/abs/2308.08155

    2023 arXiv

  22. [30]

    S., Hu, Z., and Lee, R

    Wu, Y., Hee, M. S., Hu, Z., and Lee, R. K.-W. Longgenbench: Benchmarking long-form generation in long context llms. arXiv preprint arXiv:2409.02076, 2024

    2024 arXiv

  23. [31]

    A., Oguz, B., Khabsa, M., Fang, H., Mehdad, Y., Narang, S., Malik, K., Fan, A., Bhosale, S., Edunov, S., Lewis, M., Wang, S., and Ma, H

    Xiong, W., Liu, J., Molybog, I., Zhang, H., Bhargava, P., Hou, R., Martin, L., Rungta, R., Sankararaman, K. A., Oguz, B., Khabsa, M., Fang, H., Mehdad, Y., Narang, S., Malik, K., Fan, A., Bhosale, S., Edunov, S., Lewis, M., Wang, S., and Ma, H. Effective long-context scaling o...

    2023 arXiv

  24. [32]

    Inftybench: Extending long context evaluation beyond 100 K tokens

    Zhang, X., Chen, Y., Hu, S., Xu, Z., Chen, J., Hao, M., Han, X., Thai, Z., Wang, S., Liu, Z., and Sun, M. Inftybench: Extending long context evaluation beyond 100 K tokens. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Associ...

    2024 doi

  25. [33]

    E., and Stoica, I

    Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging LLM -as-a-judge with MT -bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Da...

    2023

  26. [34]

    Agieval: A human-centric benchmark for evaluating foundation models, 2023

    Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., and Duan, N. Agieval: A human-centric benchmark for evaluating foundation models, 2023. URL https://arxiv.org/abs/2304.06364

    2023 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.