Pith. sign in

REVIEW 4 major objections 6 minor 47 references

LCIRC: A Recurrent Compression Approach for Efficient Long-form Context and Query Dependent Modeling in LLMs

T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Long-form contexts beyond a model's length limit can be compressed recurrently and injected back via gated cross-attention, letting a frozen 4K-context LLM answer 128K-context questions.

desk verdict A clean long-context retrofit with a hidden order-blind compressor that reviewers should probe first. read the letter →

arxiv 2502.06139 v2 pith:PXY5HQEE submitted 2025-02-10 cs.CL

Sumin An , Junyoung Sung , Wonpyo Park , Chanjun Park , Paul Hongsuck Seo This is my paper · ORCID
classification cs.CL
keywords long-contextlanguagemodelsrecurrentcompressionquery-dependentmodelingPerceivergatedcross-attentiontruncatedbackpropagationthroughtimelong-formquestionansweringcontextwindowextension
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a pretrained large language model can handle inputs far beyond its positional-embedding limit without retraining the base model. The proposed method, LCIRC, splits the over-length context into segments, recurrently compresses each segment into 64 latent vectors with a Perceiver module, and injects the compressed representation back into the frozen transformer with gated cross-attention. Adding a query-conditioned gating step, QD-LCIRC, is claimed to retain the information most relevant to a user question. The reported results show about 308% relative improvement over Llama-2-7B on InfiniteBench and about 90% on LongBench, along with near-flat perplexity as context grows from 4K to 128K tokens. A sympathetic reader would care because the approach promises long-context competence at a small fraction of the quadratic attention cost.

What carries the argument

The central object is a Perceiver-based recurrent compressor paired with gated cross-attention injection. The Perceiver module uses a small set of learnable query vectors (length $K=64$) and cross-attention to fold each segment's token embeddings into a fixed-size latent summary; the summary from segment $i-1$ becomes the query for segment $i$, implementing recurrence. The compressed summaries are then fed as keys and values into a Gated Cross-Attention (GCA) block inserted into each transformer layer, with tanh gates initialized to zero so the pretrained network is untouched at the start. A third piece, Selective State BPTT, randomly samples a subset of timesteps for gradient computation and routes gradients through the direct connection from $\mathbf{h}$, which the paper argues is needed to learn long-term query-dependent compression. This mechanism carries the whole argument: compression determines what survives, and injection determines what the frozen LLM can attend to.

What would settle it

Take a long document, put a target fact only in the very first segment, and ask a question whose answer is that fact; then repeat with the same fact placed only in the last segment. If QD-LCIRC answers the second version much more accurately than the first, the recurrent compression is discarding early-segment information even when it is relevant, which would contradict the claim that the compressed representation retains essential context.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that a recurrent, learned compression channel is enough to carry the task-relevant portion of an arbitrarily long context into a frozen LLM. The truncated prefix $x_C$ is cut into segments $\mathbf{s}_1,\dots,\mathbf{s}_S$, and the compressor iterates $\mathbf{h}^{(i)} = \operatorname{Perceiver}(\mathbf{h}^{(i-1)}, \mathbf{s}_i)$, so each step's 64 latent vectors summarize the current segment together with everything seen before. Those vectors are concatenated into $\mathbf{h}$ and injected at every transformer layer through a gated cross-attention block whose gates start at zero, preserving the pretrained model's behavior on short inputs. Query-dependent modeling replaces the query vectors with a query-conditioned version at each step, letting compression discard content the question does not need. The empirical claim is that this design, trained only on the added components, beats AutoCompressor and an extended full-attention baseline on long-form QA while cutting inference FLOPs by about 99% at 128K tokens.

Load-bearing premise

The load-bearing premise is that 64 learnable latent vectors per segment, carried recurrently, can preserve enough of every prior segment for whichever question is asked later; if a needed fact is lost in compression, the cross-attention injection has nothing to recover.

Editorial extensions

If this is right

  • A 4K-context base model can answer questions from contexts of hundreds of thousands of tokens, because the recurrent compressor has no explicit length limit and the injection happens through cross-attention.
  • Inference cost stays roughly constant as context grows: LCIRC uses about 120 TFLOPs at 128K tokens versus about 10,739 for an extended full-attention model, a ~99% reduction.
  • Adding query-conditioned gating improves QA average scores beyond the same model without query conditioning, by about 308% relative to the base Llama on InfiniteBench and 90% on LongBench.
  • The base model's short-context perplexity is preserved (5.472 at 4K), so the method can be added to an existing LLM without degrading what it already does.
  • The training scheme (Selective State BPTT) outperforms truncated BPTT on all three long-context benchmarks, supporting the claim that direct gradient paths into early timesteps matter.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implicit testable extension is to vary the latent budget $K$: if 64 vectors per segment are the bottleneck, increasing $K$ should help mainly on tasks where the answer depends on scattered facts, while hurting efficiency; the paper's own preliminary experiments on a smaller model found no gain from $K=256$, suggesting diminishing returns.
  • Because the paper trains query dependency only on QA pairs, open-ended generation or retrieval tasks where the relevant information is not known in advance may not benefit, and the query gate could even discard useful context; this follows from the paper's stated limitation to QA.
  • The fixed-size recurrent summary imposes an information funnel per segment; connecting LCIRC to retrieval-style selection that routes only query-relevant segments through the compressor could reduce the lossiness the authors acknowledge.
  • The method's training cost is substantial despite cheap inference, so deployment in resource-constrained settings would need the compressor to be reused across many tasks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes LCIRC, a method that extends a frozen pretrained LLM's context window by recurrently compressing earlier context segments into K=64 latent vectors per segment using a Perceiver module, then injecting all compressed vectors into the LLM through gated cross-attention layers. A query-dependent variant, QD-LCIRC, conditions the compression queries on the user's question via an additional gated cross-attention block. Training uses truncated or selectively sampled BPTT on long-form text from FineWeb-Edu and query-answer data from FineWeb-LQA. Experiments report perplexity on FineWeb-Edu, performance on InfiniteBench, LongBench, and L-Eval, and TFLOP comparisons. The authors claim QD-LCIRC consistently outperforms Llama-2-7B, ExtendedFA, and AutoCompressor, with relative gains of about 308% on InfiniteBench and 90% on LongBench, while drastically reducing inference complexity.

Significance. If the empirical results are reliable, the paper offers a practical and efficient way to extend an LLM's context without retraining the base model, and the query-dependent extension is a plausible mechanism for focusing compression on task-relevant content. The architecture is clearly described, the efficiency comparison in Table 3 is useful, and the authors are candid about limitations such as training cost and English-only evaluation. However, the paper currently lacks code, error bars, and statistical tests; the 815K-token claim in Section 5.2 is unsupported; and the Perceiver compressor has no positional encoding, making its output permutation-invariant within each segment. These issues leave the central claims in need of additional support before the results can be fully trusted.

major comments (4)
  1. [Section 3.2.1, Eq. (3)] The recurrent compressor uses the Perceiver with token embeddings si as input features and no positional encoding. Because Llama-2's embedding table is position-independent (position is injected only later by RoPE in the LLM layers), the cross-attention output h(i) is a weighted sum over the segment's token embeddings and is permutation-invariant within each segment. Consequently, for the truncated context xC (which for N=128K is 124K of the 128K tokens), word order and syntactic structure are not represented in the compressed features. This is in direct tension with the claim in Section 3.2.1 that the method retains essential information from long-form contexts, since many long-context tasks require order-sensitive reasoning. Please either add a positional encoding to the Perceiver input or queries, or provide an ablation (e.g., shuffling tokens within each segment and showing no performance change) that justifies order-insensitive compression for the reported benchmarks.
  2. [Table 2 and Section 5.4] Table 2 shows LCIRC and QD-LCIRC produce exactly the same perplexity at 64K and 128K tokens (5.312 and 5.298, respectively). The text in Section 5.4 states the model 'maintains this improved performance even as the context further lengthens,' but identical values indicate that no additional predictive information is being extracted from tokens between 64K and 128K. This is not necessarily a flaw, but the claim of scalable long-context modeling needs either longer-context experiments, variance estimates, or an analysis of when compression capacity saturates. As reported, the table provides no evidence of benefit beyond 64K.
  3. [Section 5.2] Section 5.2 states that LCIRC 'imposes no explicit length limit' and 'allowing us to process sequences up to 815K tokens in length.' No experiment or calculation in the paper uses 815K tokens; Table 3 stops at 128K. If this is an architectural upper bound, please show the derivation (e.g., based on segment count and memory); if it is an empirical result, report the experiment. As stated, the claim is unsupported.
  4. [Section 5.3 and Tables 4-6] The empirical comparisons are reported as point estimates without error bars, statistical tests, or multiple seeds. Given the large claimed improvements (308% relative on InfiniteBench), it is important to know whether these differences are stable. In addition, no code is provided, and Section 5.3 does not specify training steps, segment length R, Perceiver depth, or the number of segments used at inference. Without these details, the central benchmark claims cannot be independently verified. Please report run-to-run variance and release code or detailed hyperparameters.
minor comments (6)
  1. [Section 3.2.4] The phrase 'the our recurrent compression mechanism' should be corrected to 'our recurrent compression mechanism.'
  2. [Section 4 and Figure 3] The method is referred to as 'Random Selective BPTT' in the text but 'Selective State BPTT' in Figure 3 and Table 6; please unify the terminology.
  3. [Section 5.4] The sentence 'all other models exhibit improved perplexity scores by utilizing additional context compared to their scores with N=4K' is contradicted by AutoCompressor's 64K perplexity (6.188), which is higher than its 4K score (6.127); the sentence should be corrected to match the data.
  4. [Table 3] For AutoCompressor at 128K tokens, the table uses a dash with no explanation; the caption or text should state explicitly that AutoCompressor cannot process 128K tokens.
  5. [Section 5.1] The generation details for FineWeb-LQA (prompt template, filtering criteria, number of QA pairs, and any quality checks) are missing, which makes it difficult to assess the query-conditioned training data and potential distribution shift from the benchmark questions.
  6. [Section 5.3] The implementation details should specify the number of training steps, the segment length R used for segmentation, the number of Perceiver layers, and the learning rate schedule for the fine-tuning stage; these are needed for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: LCIRC is an empirical architecture with external benchmark evaluation, not a derivation that reduces to its inputs.

full rationale

The paper's claims are empirical and architectural, not derivational. The compression equations (2)-(5) define the mechanism (Perceiver recurrent compression and gated cross-attention injection) rather than deriving a predicted value from fitted data; the training objectives in Eqs. (6) and (7) are standard next-token NLL losses. The headline results are measured on external benchmarks (InfiniteBench, LongBench, L-Eval) after training on FineWeb-Edu and FineWeb-LQA, so no benchmark score is an input to the method's construction. The K=64 latent count is explicitly disclosed as a hyperparameter chosen in preliminary experiments on OPT-2.7B, not presented as a prediction or fitted to the reported test sets. The only potentially self-referential citation (An et al. 2024) is used for the FineWeb-LQA data construction recipe and belongs to a different author group (Shengnan An et al.), so it is not a self-citation; in any event it is not load-bearing for the central improvement claim. The noted absence of positional encoding in the Perceiver compressor is a possible model-capacity limitation, not a circular step: it concerns whether the architecture retains order information, not whether any equation reduces to its own inputs. Accordingly, no circularity steps are identified.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The method introduces no new physical entities. It relies on known neural architectures (Perceiver, gated cross-attention) and on training data. The main hand-chosen quantity is the compression width K; the other listed hyperparameters are standard training choices.

free parameters (3)
  • Latent query length K = 64
    Length of h(0) and each compressed segment feature h(i). Chosen via preliminary experiments on OPT-2.7B; controls compression capacity (Section 5.3).
  • Selective BPTT random selection count T = 8
    Number of random timesteps selected for gradient computation in Selective State BPTT (Section 4 and Table 6).
  • Learning rates = 5e-5 for LCIRC, 2e-5 for QD-LCIRC
    Separate learning rates for base and query-dependent training (Section 5.3).
assumptions (3)
  • domain assumption Gated cross-attention with zero-initialized scalars preserves the frozen LLM's behavior at initialization and does not corrupt pretrained capabilities.
    Invoked in Eq. (5) and Optimization Strategy (Section 3.2.3); if this fails, short-context performance would degrade, which Table 2 shows roughly preserved.
  • domain assumption A Perceiver with K=64 latent queries can compress the information in each text segment sufficiently for the final tasks.
    Core capacity assumption in Section 3.2.1; the entire method relies on lossy compression retaining task-relevant details.
  • domain assumption FineWeb-LQA, auto-generated with Llama-3.1-70B-Instruct, provides a valid training distribution for query-dependent long-context compression.
    Section 5.1; if the synthetic QA distribution does not match real queries, the query-dependent gains may not transfer.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LCIRC: A Recurrent Compression Approach for Efficient Long-form Context and Query Dependent Modeling in LLMs." pith.science (2026). https://pith.science/paper/PXY5HQEE

@misc{pith2026250206139,
  author       = {Pith},
  title        = {Pith review of: LCIRC: A Recurrent Compression Approach for Efficient Long-form Context and Query Dependent Modeling in LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PXY5HQEE}},
  note         = {Machine review of arXiv:2502.06139}
}
read the original abstract

While large language models (LLMs) excel in generating coherent and contextually rich outputs, their capacity to efficiently handle long-form contexts is limited by fixed-length position embeddings. Additionally, the computational cost of processing long sequences increases quadratically, making it challenging to extend context length. To address these challenges, we propose Long-form Context Injection with Recurrent Compression (LCIRC), a method that enables the efficient processing long-form sequences beyond the model's length limit through recurrent compression without retraining the entire model. We further introduce query dependent context modeling, which selectively compresses query-relevant information, ensuring that the model retains the most pertinent content. Our empirical results demonstrate that Query Dependent LCIRC (QD-LCIRC) significantly improves LLM's ability to manage extended contexts, making it well-suited for tasks that require both comprehensive context understanding and query relevance.

Figures

Figures reproduced from arXiv: 2502.06139 by the authors.

Figure 1
Figure 1. The overall process of the proposed Long-form Context Injection with Recurrent Compression (LCIRC). LCIRC comprises two components: Recurrent Context Compression (left) and Compressed Context Injection (right). In the i-th step of Recurrent Context Compression, the previously compressed features h (i−1) and the segment embeddings si are fed into the Perceiver module as query and input features, respectively. The com… view at source ↗
Figure 2
Figure 2. Comparison of the recurrent context com￾pression module with and without query dependent modeling. In addition to the regular context compres￾sion module (left), we add additional cross attention module (blue box) to inject query information into the compressed feature h (i−1) (right). features h¨(i−1) are computed through the same pro￾cess in Eq. (5) with h (i−1) and equery. The query dependent compressed feature h… view at source ↗
Figure 3
Figure 3. Comparisons of the proposed Selective State BPTT with vanilla and truncated BPTT. Green boxes represent timesteps where gradients are computed in BPTT whereas the light green ones indicate the timesteps without gradient computation. Finally, dotted red lines illustrate the gradient flows. (a) Vanilla BPTT computes the full gradients through the entire timesteps in recurrence but is computationally infeasible with a … view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 11 canonical work pages

  1. [1]

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716--23736

    2022

  2. [2]

    Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2023. L-eval: Instituting standardized evaluation for long context language models. arXiv preprint arXiv:2307.11088

    arXiv 2023

  3. [3]

    Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, and Jian-Guang Lou. 2024. Make your llm fully utilize the context. arXiv preprint arXiv:2404.16811

    arXiv 2024

  4. [4]

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. 2023. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508

    arXiv 2023

  5. [5]

    Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150

    arXiv 2020

  6. [6]

    Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev. 2022. Recurrent memory transformer. Advances in Neural Information Processing Systems, 35:11079--11091

    2022

  7. [7]

    Pu-Chin Chen, Henry Tsai, Srinadh Bhojanapalli, Hyung Won Chung, Yin-Wen Chang, and Chun-Sung Ferng. 2021. A simple and effective positional encoding for transformers. arXiv preprint arXiv:2104.08698

    work page Pith review arXiv 2021

  8. [8]

    Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595

    arXiv 2023

Show all 47 references
  1. [9]

    Yinpeng Chen, DeLesley Hutchins, Aren Jansen, Andrey Zhmoginov, David Racz, and Jesper Andersen. 2024. Melodi: Exploring memory compression for long contexts. arXiv preprint arXiv:2410.03156

    2024 arXiv

  2. [10]

    Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. 2023. Adapting language models to compress contexts. arXiv preprint arXiv:2305.14788

    2023 arXiv

  3. [11]

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509

    2019 arXiv

  4. [12]

    Zihang Dai. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860

    2019 arXiv

  5. [13]

    Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. Eli5: Long form question answering. arXiv preprint arXiv:1907.09190

    2019 arXiv

  6. [14]

    Quentin Fournier, Ga \'e tan Marceau Caron, and Daniel Aloise. 2023. A practical survey on faster and lighter transformers. ACM Computing Surveys, 55(14s):1--40

    2023

  7. [15]

    Giorgio Franceschelli, Claudia Cevenini, and Mirco Musolesi. 2024. Training foundation models as data compression: On information, model weights and copyright law. arXiv preprint arXiv:2407.13493

    2024 arXiv

  8. [16]

    Tao Ge, Jing Hu, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. 2023. In-context autoencoder for context compression in a large language model. arXiv preprint arXiv:2307.06945

    2023 arXiv

  9. [17]

    Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. 2021 a . Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2107.14795

    2021 arXiv

  10. [18]

    Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. 2021 b . Perceiver: General perception with iterative attention. In International conference on machine learning, pages 4651--4664. PMLR

    2021

  11. [19]

    Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2024. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention. arXiv preprint...

    2024 arXiv

  12. [20]

    Diederik P Kingma. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980

    2014 arXiv

  13. [21]

    Nikita Kitaev, ukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451

    2020 arXiv

  14. [22]

    Huan Yee Koh, Jiaxin Ju, Ming Liu, and Shirui Pan. 2022. An empirical survey on long document summarization: Datasets, models, and metrics. ACM computing surveys, 55(8):1--35

    2022

  15. [23]

    Brian Lester, Jaehoon Lee, Alex Alemi, Jeffrey Pennington, Adam Roberts, Jascha Sohl-Dickstein, and Noah Constant. 2024. Training llms over neurally compressed text. arXiv preprint arXiv:2404.03626

    2024 arXiv

  16. [24]

    Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. 2024. Long-context llms struggle with long in-context learning. arXiv preprint arXiv:2404.02060

    2024 arXiv

  17. [25]

    Xianming Li, Zongxi Li, Xiaotian Luo, Haoran Xie, Xing Lee, Yingbin Zhao, Fu Lee Wang, and Qing Li. 2023. Recurrent attention networks for long-text modeling. arXiv preprint arXiv:2306.06843

    2023 arXiv

  18. [26]

    Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu. 2022. A survey of transformers. AI open, 3:111--132

    2022

  19. [27]

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157--173

    2024

  20. [28]

    Yinghan Long, Sayeed Shafayet Chowdhury, and Kaushik Roy. 2023. Segmented recurrent transformer: An efficient sequence-to-sequence model. arXiv preprint arXiv:2305.16340

    2023 arXiv

  21. [29]

    Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. 2024. https://doi.org/10.57967/hf/2497 Fineweb-edu

    2024 doi

  22. [30]

    Jesse Mu, Xiang Li, and Noah Goodman. 2024. Learning to compress prompts with gist tokens. Advances in Neural Information Processing Systems, 36

    2024

  23. [31]

    Thomas Mulc and Jennifer L Steele. 2024. Compressing search with language models. arXiv preprint arXiv:2407.00085

    2024 arXiv

  24. [32]

    Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. 2023. A comprehensive overview of large language models. arXiv preprint arXiv:2307.06435

    2023 arXiv

  25. [33]

    Piotr Nawrot, Adrian a\' n cucki, Marcin Chochowski, David Tarjan, and Edoardo Ponti. 2024. Dynamic memory compression: Retrofitting LLM s for accelerated inference. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine L...

    2024

  26. [34]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 3...

    2022

  27. [35]

    Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap. 2019. Compressive transformers for long-range sequence modelling. arXiv preprint arXiv:1911.05507

    2019 arXiv

  28. [36]

    Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. 2021. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics, 9:53--68

    2021

  29. [37]

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063

    2024

  30. [38]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  31. [39]

    Chandra Shekhara Kaushik Valmeekam, Krishna Narayanan, Dileep Kalathil, Jean-Francois Chamberland, and Srinivas Shakkottai. 2023. Llmzip: Lossless text compression using large language models. arXiv preprint arXiv:2306.04050

    2023 arXiv

  32. [40]

    Xindi Wang, Mahsa Salmani, Parsa Omidi, Xiangyu Ren, Mehdi Rezagholizadeh, and Armaghan Eshaghi. 2024. Beyond the limits: A survey of techniques to extend the context length in large language models. arXiv preprint arXiv:2402.02244

    2024 arXiv

  33. [41]

    Qingyang Wu, Zhenzhong Lan, Kun Qian, Jing Gu, Alborz Geramifard, and Zhou Yu. 2020. Memformer: A memory-augmented transformer for sequence modeling. arXiv preprint arXiv:2010.06891

    2020 arXiv

  34. [42]

    Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283--17297

    2020

  35. [43]

    Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, et al. 2024 a . bench: Extending long context evaluation beyond 100k tokens. In Proceedings of the 62nd Annual Meeting of the Association for Computational Lin...

    2024

  36. [44]

    Yichi Zhang, Bofei Gao, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, Wen Xiao, et al. 2024 b . Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling. arXiv preprint arXiv:2406.02069

    2024 arXiv

  37. [45]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223

    2023 arXiv

  38. [46]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  39. [47]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.