Pith. sign in

REVIEW 4 major objections 6 minor 50 references

Dynamic Chunking and Selection for Reading Comprehension of Ultra-Long Context in Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a two-stage pipeline—semantic-boundary chunking plus a question-aware relevance classifier—lets small LLMs answer questions from contexts up to 256k tokens more accurately than streaming, eviction, or fixed-chunk…

desk verdict Useful compression pipeline with broad empirical coverage, but the chunk-selection mechanism is not actually validated and the headline numbers are partly fitted to the test sets. read the letter →

arxiv 2506.00773 v2 pith:PHJFLQVL submitted 2025-06-01 cs.CL

Boheng Sheng , Jiacheng Yao , Meicong Zhang , Guoxiu He This is my paper
classification cs.CL
keywords long-contextreadingcomprehensiondynamicchunkingquestion-awarechunkselectionlargelanguagemodelsquestionansweringsemanticsimilaritymulti-hopQAcontextcompression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Long-context reading comprehension fails not only because context windows are short, but because fixed-length splitting cuts through sentences and separates the evidence an answer needs. The paper proposes Dynamic Chunking and Selection (DCS): first split a document at places where adjacent sentences are semantically dissimilar, keeping each chunk coherent; then use a small question-aware classifier to keep only the chunks that matter for the question, and feed those to the LLM in original order. DCS is training-light, since only the classifier is trained, and it lets 7-8B instruction-tuned models work with inputs far beyond their native windows. Across twelve single- and multi-hop QA datasets, DCS reports consistent gains over streaming, eviction, and fixed-chunk baselines, with the smallest degradation when context length grows from 16k to 256k tokens.

What carries the argument

The central object is the chunk: a variable-length unit of text that preserves sentence integrity. Dynamic chunking builds these units by splitting the document on punctuation into sentences, merging each sentence with its immediate neighbors, encoding the merged segments with a sentence embedding model, and computing cosine similarity between adjacent encoded sentences; the lowest-similarity positions become candidate boundaries, subject to a percentile threshold $\alpha$, and chunks are then merged again to stay under a length cap $l$. The second object is a question-aware classifier: a three-layer MLP whose input is a distilled feature vector of six pieces—the hidden states of the first and last context and question boundary tokens plus attention-pooled context and question representations taken from the backbone LLM's final layer. Trained with binary cross-entropy on answerable versus unanswerable context-question pairs, the classifier gives each chunk a relevance score $T_i$; the top $\lfloor m/\alpha_c \rfloor$ chunks are concatenated in their original order with the question into the LLM's context window.

What would settle it

A concrete falsifier: on a held-out domain such as legal or medical documents, mark the single chunk that contains the gold answer, let the classifier rank all chunks, and compute whether the evidence chunk appears in the selected top-k. If the selection recall is no better than random for multi-hop questions whose evidence is spread across four or more chunks, the paper's robustness claim is overturned.

Watch

Extended reading notes

Core claim

The central claim is that the semantic structure of a document is the right unit for compressing long contexts before question answering. DCS first recovers that structure by encoding sentences and cutting where cosine similarity between neighboring sentences is low, so coherent units—complete sentences and their supporting context—stay together. It then trains a three-layer classifier on question-aware features distilled from the LLM's own final-layer hidden states and attention scores; at inference the classifier scores every chunk for the given question and only the top chunks are passed to the LLM. The paper reports that this configuration beats all compared baselines on most of twelve QA datasets: on the largest 8B backbone it raises average single-hop score from 27.60 to 35.50 and multi-hop score from 24.22 to 29.07, and it holds performance as inputs scale to 256k tokens.

Load-bearing premise

The load-bearing premise is that a classifier trained on a few open-domain QA datasets can rank chunks by relevance well enough for every new document, question type, and context length; if its relevance concept fails to transfer to a new domain or to multi-hop evidence spread across many chunks, the selected top-k will omit the answer.

Editorial extensions

If this is right

  • Contexts up to 256k tokens can be answered by models whose native window is 4k-8k tokens, with less performance loss than streaming or eviction baselines.
  • Multi-hop questions benefit disproportionately, because keeping each supporting fact inside an intact chunk makes it possible for the LLM to combine evidence rather than receive a sentence fragment.
  • The method adds almost no training cost: only the small MLP classifier is trained, so it can be dropped onto new backbone LLMs without fine-tuning them.
  • Ablations show both halves earn their keep: dynamic chunking beats fixed-length chunking, and the learned classifier beats cosine-similarity selection.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the classifier's relevance scores could be reused as a learned reranker in retrieval pipelines, turning a document-level QA method into a chunk-level retriever without additional training.
  • Editorial inference: because DCS preserves the original order of kept chunks, part of its gain may come from moving evidence out of the middle of long inputs; a position-shuffling experiment would separate this effect from pure evidence selection.
  • Editorial inference: the dynamic chunker is unsupervised and model-agnostic, so it may transfer to other long-text tasks such as summarization or code completion, though the paper evaluates only question answering.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Dynamic Chunking and Selection (DCS), a training-light method for long-context reading comprehension. DCS first splits the input into variable-length chunks by computing sentence embeddings and cutting at low-similarity boundaries, then trains a small question-aware classifier on top of features extracted from a frozen LLM (boundary hidden states and attention maps) to score each chunk with respect to the question. The top-scoring chunks are concatenated and fed to the LLM within its context window. The authors evaluate DCS against StreamingLLM, LM-Infinite, InfLLM, and MoICE on 12 QA datasets (single- and multi-hop) using Llama-3-8B-Instruct, Mistral-7B-Instruct, and Vicuna-7B, including versions up to 256k tokens, and report consistent average improvements as well as an ablation and latency analysis.

Significance. If the reported results hold, DCS is a simple and reproducible baseline that extends LLM reading comprehension to very long inputs without architectural changes or heavy training, and the release of code and data is valuable. The dynamic chunking idea is plausible and the comparison across three backbones is broader than in many long-context papers. However, the significance is currently limited by the lack of validation that the learned classifier actually selects evidence chunks, the selection of key hyperparameters on the test benchmarks, and underspecified inference protocols for out-of-window inputs; these issues must be addressed before the claims can be fully trusted.

major comments (4)
  1. [§3.2, Eqs. (8)–(23), Table 4] The question-aware classifier is trained on whole-context question-answerability pairs, where the positive label indicates that the full context contains an answer somewhere, but at inference it is applied to individual chunks (each chunk paired with the question). The manuscript provides no chunk-level validation (e.g., recall of the oracle evidence chunks or an oracle-top-k experiment), so it is not established that the classifier learns to localize evidence rather than to detect topical relevance. Table 4 shows that the trained classifier is actually worse than cosine-similarity selection on Llama-3 single-hop QA (32.90 vs. 33.07) and only slightly better on average (26.85 vs. 25.84). Since the paper's central claim is that DCS 'trains a question-aware classifier to select sensitive chunks,' this lack of direct evidence weakens the attribution of the reported gains to the proposed selection mechanism. Please report chunk-level retrieval metrics and/or retrain the classifier with chunk-level supervision and show consistent gains across all three backbones.
  2. [§5.4.1, Table 2] The hyperparameters l and α are chosen by evaluating on the same six datasets (NarrativeQA, HotpotQA, 2WikiMQA, MFQA_en, Qasper, Musique) that are part of the main evaluation in Table 1. Selecting hyperparameters on the test benchmarks means that the headline improvements in Table 1 are partly the result of fitting to the evaluation sets. Please use a separate validation split (or a small development set disjoint from the final benchmarks) and report the resulting performance; alternatively, show that the reported gains are stable under a wide range of l and α chosen without access to the test labels.
  3. [§4.2, Figure 3] The protocol for evaluating inputs longer than the backbone's context window is not specified. The manuscript states that Llama-3-8B-Instruct and Mistral-7B-Instruct have a maximum length of 8K and Vicuna-7B has 4K, but the long-context experiments use inputs of 16k–256k tokens. It is not described how the 'Origin' baseline (or the StreamingLLM/LM-Infinite/InfLLM/MoICE baselines) is applied to such inputs—e.g., whether the context is truncated, split, or processed with the baseline's own mechanism. Without this information, the comparison against 'Origin' is not a well-defined baseline, and the claim in §5.3 that 'baselines suffer from significant performance deterioration' is not supported by Figure 3, which plots only 'Origin' and DCS rather than the named baselines. Please specify the exact inference protocol for all methods on out-of-window inputs and include the actual baseline curves (or tables) for the 64k–256k region.
  4. [Table 1 caption and §5] The caption of Table 1 states that 'The t-test proves that the improvement is statistically significant (p<0.05),' but no details are given about the test procedure, the unit of analysis, the number of samples, or whether the test is paired. Similarly, the average improvements in Tables 4 and 5 are reported without error bars or significance testing. Please provide a precise description of the statistical test and, ideally, report confidence intervals or per-dataset significance levels.
minor comments (6)
  1. [Eq. (23)] The selection criterion uses ⌊m/α⌋, but the compression ratio is defined as αc in Eq. (22); please use consistent notation.
  2. [Table 1] The entry '44.7921.25' appears to be two numbers (44.79 and 21.25) merged without a space; please fix the formatting.
  3. [Figures 10–12] 'Ispossitive' should be 'Is positive' in the example annotations.
  4. [§3.2] The negative sampling strategy S is described only as generating 'semantically uncorrelated pairs'; please specify how these pairs are generated (e.g., random pairing within the same corpus) and whether any filtering is applied.
  5. [§4.2] The main text lists MoICE as a baseline, but the main table only reports Llama-3 results; please clarify in the text where the MoICE results (Mistral and Vicuna) are presented and why the main table omits them.
  6. [§5.4.4] The complexity statement O(nl) should also account for the cost of encoding each chunk with the backbone LLM when computing classifier features; please state the total computational cost more carefully.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; DCS is an empirical pipeline whose claims are evaluated against external benchmarks.

full rationale

The paper does not claim to derive QA performance from first principles; every substantive claim is an empirical result (Tables 1, 9, 10) comparing DCS against baselines on external benchmarks. The two modules are defined by their own operational procedures: dynamic chunking splits at low cross-sentence embedding similarity (Eqs. 2-6), and the question-aware classifier is trained with binary cross-entropy on context-question pairs from AdversarialQA/CoQA/SQuAD (Eq. 19) and then applied to chunk-question pairs (Eqs. 20-23). Neither equation makes the output equal to an input by construction: the classifier is a learned model, not an identity or an interpolant of the target answers, and no target-dataset answer is used to fit the classifier. The only potential concern is that l=512 and alpha=60 are chosen by sweeping on six evaluation datasets (Table 2), which is test-set selection rather than circularity; it would affect generalization claims, not the logical independence of the method from its inputs. There are no load-bearing self-citations, no imported uniqueness theorems, and no renamed known results presented as predictions. The train/serve mismatch between whole-context positives and chunk-level inference is a possible validity concern, but it is an empirical gap, not a definitional identity. Under the rule that findings of circularity require quoting an equation-level reduction, none exists here.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method relies on standard embeddings and attention features rather than newly invented entities. The key implicit assumptions are about the transferability of the relevance classifier and the sufficiency of cosine-similarity chunking; no free parameters beyond the explicitly tuned hyperparameters are introduced.

free parameters (3)
  • alpha (percentile threshold) = 60 (Llama3, Mistral), 65 (Vicuna)
    Controls how many dissimilarity boundaries are used for dynamic chunking; selected by evaluating on the benchmark test datasets (Section 5.4.1, Table 2).
  • l (target chunk size) = 512 for all models
    Target maximum chunk length in tokens; selected via test-set hyperparameter search (Section 5.4.1, Table 2).
  • l_T (target context length) = 7.5k (Llama3), 7k (Mistral), 3.5k (Vicuna)
    Target compressed context length; set manually to fit each model's context window, no sensitivity analysis reported.
assumptions (4)
  • domain assumption Sentence-BERT embeddings accurately reflect semantic relatedness of adjacent sentences for chunk boundary detection.
    Dynamic chunking uses cosine distance between embeddings to pick boundaries (Section 3.1, Eq. 2-4). If these embeddings misjudge semantic similarity, chunks will break at wrong places.
  • domain assumption The question-aware classifier trained on AdversarialQA/CoQA/SQuAD generalizes to the evaluation datasets and context lengths up to 256k tokens.
    Chunk selection in all experiments uses this classifier (Section 3.2); robustness is only checked across training datasets (Section 5.4.3), not across domains.
  • domain assumption Retaining the top-k chunks with the highest relevance scores preserves all information needed to answer the question.
    The final input is the top-k selected chunks (Eq. 23). If relevant evidence is ranked below the cutoff, the LLM cannot recover it.
  • domain assumption Attention and hidden-state features from the backbone LLM are sufficient for chunk-question relevance classification.
    The classifier input is a 6xd feature matrix from boundary tokens and attention pooling (Eq. 18). No external features are used.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dynamic Chunking and Selection for Reading Comprehension of Ultra-Long Context in Large Language Models." pith.science (2026). https://pith.science/paper/PHJFLQVL

@misc{pith2026250600773,
  author       = {Pith},
  title        = {Pith review of: Dynamic Chunking and Selection for Reading Comprehension of Ultra-Long Context in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PHJFLQVL}},
  note         = {Machine review of arXiv:2506.00773}
}
read the original abstract

Large language models (LLMs) often struggle to accurately read and comprehend extremely long texts. Current methods for improvement typically rely on splitting long contexts into fixed-length chunks. However, fixed truncation risks separating semantically relevant content, leading to ambiguity and compromising accurate understanding. To overcome this limitation, we propose a straightforward approach for dynamically separating and selecting chunks of long context, facilitating a more streamlined input for LLMs. In particular, we compute semantic similarities between adjacent sentences, using lower similarities to adaptively divide long contexts into variable-length chunks. We further train a question-aware classifier to select sensitive chunks that are critical for answering specific questions. Experimental results on both single-hop and multi-hop question-answering benchmarks show that the proposed approach consistently outperforms strong baselines. Notably, it maintains robustness across a wide range of input lengths, handling sequences of up to 256k tokens. Our datasets and code are available at the following link: https://github.com/ECNU-Text-Computing/DCS

Figures

Figures reproduced from arXiv: 2506.00773 by the authors.

Figure 1
Figure 1. A failure case of a fixed-length chunking [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The overall structure of the proposed DCS. It includes two small modules to compress the input to help [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Comparison of time spent by different ap [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figures from the paper (6 more)
Figure 3
Figure 3. Figure 3: Results on longer datasets. performance across most datasets. Based on these findings, we establish l = 512 and α=60 as the model’s default hyperparameters. This configu￾ration not only delivers optimal performance as measured by the evaluation metrics but also ex￾hibi…
Figure 6
Figure 6. Figure 6: Prompts of Hotpotqa, 2wikimqa, and Musique. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: Prompts of Loogle_SD, Multifieldqa_en, and [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 9
Figure 9. Figure 9: An example of question-aware classifier input [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 12
Figure 12. Figure 12: An example of context-question pairs of Squad 20 [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]
Figure 11
Figure 11. Figure 11: An example of context-question pairs of CoQA Context: Beyoncé Giselle Knowles-Carter (/bijnse/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. ... earned five Grammy Awards and featured the Bill￾board Hot 100 numbe…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

50 extracted references · 3 canonical work pages

  1. [1]

    AI@Meta. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783

    arXiv 2024

  2. [2]

    Chenxin An, Fei Huang, Jun Zhang, Shansan Gong, Xipeng Qiu, Chang Zhou, and Lingpeng Kong. 2024. https://arxiv.org/abs/2402.17463 Training-free long-context scaling of large language models . Preprint, arXiv:2402.17463

    arXiv 2024

  3. [3]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

    arXiv 2023

  4. [4]

    Yu Bai, Xiyuan Zou, Heyan Huang, Sanxing Chen, Marc-Antoine Rondeau, Yang Gao, and Jackie Chi Kit Cheung. 2024 a . https://arxiv.org/abs/2406.12018 Citrus: Chunked instruction-aware state eviction for long sequence modeling . Preprint, arXiv:2406.12018

    work page Pith review arXiv 2024

  5. [5]

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024 b . https://doi.org/10.18653/v1/2024.acl-long.172 L ong B ench: A bilingual, multitask benchmark for long context understanding . In Proceedings of the 62nd Annual Meeting of the Association f...

    doi:10.18653/v1/2024.acl-long.172 2024

  6. [6]

    Max Bartolo, Alastair Roberts, Johannes Welbl, Sebastian Riedel, and Pontus Stenetorp. 2020. https://doi.org/10.1162/tacl\_a\_00338 Beat the ai: Investigating adversarial human annotation for reading comprehension . Transactions of the Association for Computational Linguistics, 8:662--678

    doi:10.1162/tacl 2020

  7. [7]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. https://arxiv.org/abs/2004.05150 Longformer: The long-document transformer . Preprint, arXiv:2004.05150

    arXiv 2020

  8. [8]

    Amanda Bertsch, Uri Alon, Graham Neubig, and Matthew R. Gormley. 2023. https://arxiv.org/abs/2305.01625 Unlimiformer: Long-range transformers with unlimited length input . Preprint, arXiv:2305.01625

    arXiv 2023

Show all 50 references
  1. [9]

    Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595

    2023 arXiv

  2. [10]

    Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. 2024. https://arxiv.org/abs/2309.12307 Longlora: Efficient fine-tuning of long-context large language models . Preprint, arXiv:2309.12307

    2024 arXiv

  3. [11]

    Smith, and Matt Gardner

    Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. https://doi.org/10.18653/v1/2021.naacl-main.365 A dataset of information-seeking questions and answers anchored in research papers . In Proceedings of the 2021 Conference of the North Amer...

    2021 doi

  4. [12]

    Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. 2024. Lm-infinite: Zero-shot extreme length generalization for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Lingui...

    2024

  5. [13]

    Zifan He, Zongyue Qin, Neha Prakriya, Yizhou Sun, and Jason Cong. 2024. https://arxiv.org/abs/2405.06067 Hmt: Hierarchical memory transformer for long context language processing . Preprint, arXiv:2405.06067

    2024 arXiv

  6. [14]

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. https://doi.org/10.18653/v1/2020.coling-main.580 Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps . In Proceedings of the 28th International Conference on Computational Li...

    2020 doi

  7. [15]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. 2021. https://arxiv.org/abs/2106.09685 Lora: Low-rank adaptation of large language models . CoRR, abs/2106.09685

    2021 arXiv

  8. [16]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  9. [17]

    Greg Kamradt. 2023. Semantic splitting - embedding walk based chunking. [Online]. https://retrieval-tutorials.vercel.app/document-loaders/text-splitting

    2023

  10. [18]

    Tom\'a s Ko cisk\'y, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, G\'abor Melis, and Edward Grefenstette. 2018. https://doi.org/10.1162/tacl_a_00023 The N arrative QA reading comprehension challenge . Transactions of the Association for Computational Lingui...

    2018 doi

  11. [19]

    Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. 2023. Loogle: Can long-context language models understand long contexts? arXiv preprint arXiv:2311.04939

    2023 arXiv

  12. [20]

    Hongzhan Lin, Ang Lv, Yuhan Chen, Chen Zhu, Yang Song, Hengshu Zhu, and Rui Yan. 2024. https://arxiv.org/abs/2406.19598 Mixture of in-context experts enhance llms' long context awareness . Preprint, arXiv:2406.19598

    2024 arXiv

  13. [21]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023. https://arxiv.org/abs/2307.03172 Lost in the middle: How language models use long contexts . Preprint, arXiv:2307.03172

    2023 arXiv

  14. [22]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. https://doi.org/10.1162/tacl_a_00638 Lost in the middle: How language models use long contexts . Transactions of the Association for Computational Linguistics, 12...

    2024 doi

  15. [23]

    Yinquan Lu, Wenhao Zhu, Lei Li, Yu Qiao, and Fei Yuan. 2024. https://arxiv.org/abs/2407.05975 Llamax: Scaling linguistic horizons of llm by enhancing translation capabilities beyond 100 languages . Preprint, arXiv:2407.05975

    2024 arXiv

  16. [24]

    Xuezhe Ma, Xiaomeng Yang, Wenhan Xiong, Beidi Chen, Lili Yu, Hao Zhang, Jonathan May, Luke Zettlemoyer, Omer Levy, and Chunting Zhou. 2024. https://arxiv.org/abs/2404.08801 Megalodon: Efficient llm pretraining and inference with unlimited context length . Preprint, arXiv:2404.08801

    2024 arXiv

  17. [25]

    OpenAI. 2024. https://arxiv.org/abs/2303.08774 Gpt-4 technical report . Preprint, arXiv:2303.08774

    2024 arXiv

  18. [26]

    Matanel Oren, Michael Hassid, Nir Yarden, Yossi Adi, and Roy Schwartz. 2024. https://arxiv.org/abs/2401.06104 Transformers are multi-state rnns . Preprint, arXiv:2401.06104

    2024 arXiv

  19. [27]

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2023. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071

    2023 arXiv

  20. [28]

    Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. https://arxiv.org/abs/1806.03822 Know what you don't know: Unanswerable questions for squad . Preprint, arXiv:1806.03822

    2018 arXiv

  21. [29]

    Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. https://arxiv.org/abs/1808.07042 Coqa: A conversational question answering challenge . Preprint, arXiv:1808.07042

    2019 arXiv

  22. [30]

    Nils Reimers and Iryna Gurevych. 2019. https://arxiv.org/abs/1908.10084 Sentence-bert: Sentence embeddings using siamese bert-networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics

    2019 arXiv

  23. [31]

    Vinay Samuel, Houda Aynaou, Arijit Chowdhury, Karthik Venkat Ramanan, and Aman Chadha. 2024. https://doi.org/10.18653/v1/2024.acl-srw.36 Can LLM s augment low-resource reading comprehension datasets? opportunities and challenges . In Proceedings of the 62nd Annual Meeting of t...

    2024 doi

  24. [32]

    Woomin Song, Seunghyuk Oh, Sangwoo Mo, Jaehyung Kim, Sukmin Yun, Jung-Woo Ha, and Jinwoo Shin. 2024. Hierarchical context merging: Better long context understanding for pre-trained llms. arXiv preprint arXiv:2404.10308

    2024 arXiv

  25. [33]

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063

    2024

  26. [34]

    Derek Tam, Anisha Mascarenhas, Shiyue Zhang, Sarah Kwan, Mohit Bansal, and Colin Raffel. 2023. https://doi.org/10.18653/v1/2023.findings-acl.322 Evaluating the factual consistency of large language models through news summarization . In Findings of the Association for Computat...

    2023 doi

  27. [35]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023 a . https://arxiv.org/abs/2302.13971 Lla...

    2023 arXiv

  28. [36]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  29. [37]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. https://doi.org/10.1162/tacl_a_00475 ♫ M u S i Q ue: Multihop questions via single-hop question composition . Transactions of the Association for Computational Linguistics, 10:539--554

    2022 doi

  30. [38]

    Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. https://arxiv.org/abs/2002.10957 Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers . CoRR, abs/2002.10957

    2020 arXiv

  31. [39]

    Xindi Wang, Mahsa Salmani, Parsa Omidi, Xiangyu Ren, Mehdi Rezagholizadeh, and Armaghan Eshaghi. 2024. https://arxiv.org/abs/2402.02244 Beyond the limits: A survey of techniques to extend the context length in large language models . Preprint, arXiv:2402.02244

    2024 arXiv

  32. [40]

    Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. 2024. Infllm: Training-free long-context extrapolation for llms with an efficient context memory. In The Thirty-eighth Annual Conference on Neural Information Processi...

    2024

  33. [41]

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2023. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453

    2023 arXiv

  34. [42]

    Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. 2024. https://arxiv.org/abs/2401.08417 Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation . Preprint, arXiv:2...

    2024 arXiv

  35. [43]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. https://doi.org/10.18653/v1/D18-1259 H otpot QA : A dataset for diverse, explainable multi-hop question answering . In Proceedings of the 2018 Conference...

    2018 doi

  36. [44]

    Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. 2020. https://arxiv.org/abs/2002.04326 Reclor: A reading comprehension dataset requiring logical reasoning . Preprint, arXiv:2002.04326

    2020 arXiv

  37. [45]

    Tao Yuan, Xuefei Ning, Dong Zhou, Zhijie Yang, Shiyao Li, Minghui Zhuang, Zheyue Tan, Zhuyu Yao, Dahua Lin, Boxun Li, Guohao Dai, Shengen Yan, and Yu Wang. 2024. https://arxiv.org/abs/2402.05136 Lv-eval: A balanced long-context benchmark with 5 length levels up to 256k . Prepr...

    2024

  38. [46]

    Hashimoto

    Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B. Hashimoto. 2024. https://doi.org/10.1162/tacl_a_00632 Benchmarking large language models for news summarization . Transactions of the Association for Computational Linguistics, 12:39--57

    2024 doi

  39. [47]

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. 2023. https://arxiv.org/abs/2306.14048 H _2 o: Heavy-hitter oracle for efficient generative inference of lar...

    2023 arXiv

  40. [48]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. https://arxiv.org/abs/2306.05685 Judging llm-as-a-judge with mt-bench and chatbot arena . P...

    2023 arXiv

  41. [49]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  42. [50]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.