Pith. sign in

REVIEW 5 major objections 6 minor 36 references

ECHO-LLaMA: Efficient Caching for High-Performance LLaMA Training

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ECHO-LLaMA claims one shared KV cache lifts LLaMA training throughput by up to 77% while keeping or improving loss.

desk verdict The incremental adaptation recipe for converting pretrained LLaMA models into flexible shared-KV architectures is a real extension of YOCO, but the headline efficiency numbers are not backed by the experiments as reported. read the letter →

arxiv 2505.17331 v2 pith:3LEOHIE7 submitted 2025-05-22 cs.LG cs.CL

Maryam Dialameh , Rezaul Karim , Hossein Rajabzadeh , Omar Mohamed Awad , Hyock Ju Kwon , Boxing Chen , Walid Ahmed , Yang Liu This is my paper · ORCID
classification cs.LGcs.CL
keywords sharedKVcacheLLaMAefficiencycross-attentionincrementaladaptationtrainingthroughputmodelFLOPsutilizationpretrainedconversionmemory
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ECHO-LLaMA sets out to show that a pretrained LLaMA model does not need to compute fresh key and value vectors at every layer. It converts a chosen set of upper layers into cross-attention layers that all reuse a single pair of key-value matrices computed once from the middle of the network. The conversion is done gradually, layer by layer on a small token budget, followed by a short fine-tune, so an existing checkpoint is adapted rather than retrained. If the claims hold, the architecture trains substantially faster, with up to 77% higher tokens-per-second, up to 16% higher model FLOPs utilization, and up to 14% lower loss at equal tokens, while matching or slightly beating the baseline on language benchmarks. This matters because most efficiency work targets inference, whereas ECHO-LLaMA targets the much larger cost of training and fine-tuning.

What carries the argument

The load-bearing object is the shared global KV cache: a single pair of key and value matrices computed once at layer $N$ by global projections followed by RMSNorm, then reused as the attention memory for all converted cross-decoder layers. This replaces per-layer key/value projection and caching in the upper layers with one shared memory, so the KV cache memory ratio versus a standard LLaMA becomes $(1-p) + 1/L$, which approaches $1-p$ for deep models, where $p$ is the fraction of shared layers. The other mechanism is the layer-wise incremental adaptation schedule, which converts layers from the top down with small per-stage token budgets to protect pretrained knowledge; the paper argues that freezing all but the converted layer at each stage mitigates catastrophic forgetting.

What would settle it

Rerun the 4-billion-token comparison with identical global batch size, sequence length, device count, and total wall-clock budget for both a LLaMA baseline and its ECHO-LLaMA conversion, and record final validation loss and tokens per second; if ECHO does not show both lower loss at equal tokens and higher throughput, the headline claims are measurement artifacts rather than architectural gains.

Watch

Extended reading notes

Core claim

The central claim is that a LLaMA-style transformer can be reorganized so that only the first $N$ layers keep full self-attention, and every layer above $N$ computes cross-attention against one global KV pair derived from the output of layer $N$. The shared pair is obtained as $K_{\text{shared}} = \mathrm{RMSNorm}(W^K_{\text{global}} X_N)$ and $V_{\text{shared}} = \mathrm{RMSNorm}(W^V_{\text{global}} X_N)$, so the upper layers drop their own key and value projections and reuse cached activations. The paper further claims that a pretrained checkpoint can be moved into this shape by an incremental adaptation schedule: convert the last layer first, train about 150 steps on roughly 4 million tokens with other layers frozen, then repeat downward through the chosen range, and finish with one epoch on about 4 billion tokens. The experiments report up to 77% higher training throughput, up to 16% higher model FLOPs utilization, and up to 14% lower loss at equal tokens relative to LLaMA baselines, with about 7% faster test-time generation on the 1.1B model at 50% sharing and roughly unchanged benchmark accuracy. The flexibility of choosing $N$, for example sharing 25% or 50% of layers, is presented as an advantage over the fixed half-layer sharing of prior YOCO-style designs.

Load-bearing premise

The efficiency figures compare ECHO and baseline on an equal footing, meaning the same effective training budget and comparable hardware settings for both, so the reported throughput, MFU, and loss differences reflect the architecture and not the measurement setup.

Editorial extensions

If this is right

  • Existing LLaMA checkpoints can be converted to the shared-KV architecture with roughly 4 billion tokens of adaptation, rather than full pretraining, making efficiency upgrades available to already-trained models.
  • At 25% sharing the adapted TinyLLaMA matches or slightly exceeds baseline accuracy, while at 50% sharing it stays within about half a point on zero-shot benchmarks and generates about 7% faster, so the same checkpoint can be tuned for speed or accuracy.
  • KV memory falls by almost half at 50% sharing, with a measured 2.19x reduction at 16k sequence length, which directly extends the feasible context length on fixed hardware.
  • The claimed gains reach 77% training throughput and 16% model FLOPs utilization on the tested NPU hardware, while on the tested GPUs the gains are smaller, meaning the benefit is hardware-dependent.
  • Pretraining from scratch with the ECHO structure also yields equal or lower training loss across 125M to 7B parameter scales, suggesting the shared-KV design does not sacrifice optimization quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the loss improvements are real, the shared KV cache may act as a mild regularizer or parameter-sharing inductive bias; one could test this by checking whether the advantage persists when both models train on far more than 4 billion tokens or on out-of-distribution held-out text.
  • The paper's own discussion implies the speedup shrinks as models grow because MLP compute dominates, so the practical sweet spot may be mid-size models, and a systematic sweep of speedup versus model size could map where conversion stops paying off.
  • A direct comparison against YOCO under matched training budgets would isolate whether the gains come from the flexible sharing fraction or from the incremental adaptation recipe, since the paper could not run one for lack of public YOCO checkpoints.
  • The same incremental conversion recipe could in principle be applied to other transformer families that use RMSNorm and rotary embeddings, which the paper itself tries on Qwen models, so the mechanism is a general efficiency retrofit rather than a LLaMA-specific trick.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes ECHO-LLaMA, a modification of LLaMA in which a configurable subset of later layers (cross-decoders) share a single global KV cache computed from the output of an intermediate layer, while the remaining layers keep standard self-attention. Pretrained LLaMA checkpoints are converted into this architecture by an incremental, layer-by-layer adaptation procedure followed by a final fine-tuning phase. The authors report lower training loss, higher tokens-per-second throughput, higher MFU, and reduced memory usage, with headline claims of up to 77% higher training throughput, up to 16% higher MFU, up to 14% lower loss, and about 7% higher inference throughput.

Significance. If the efficiency and quality claims were properly supported, the work would be practically valuable: it offers a way to convert existing pretrained LLaMA checkpoints into cheaper architectures without full retraining, and it relaxes YOCO's rigid half-layer KV sharing. The incremental adaptation idea is coherent, and the paper includes useful ablations (incremental versus full-stage conversion) and generalization experiments to Qwen and LLaMA3.2 models. However, the central quantitative claims are not currently supported by the presented measurements. Batch sizes differ systematically between ECHO and baseline runs, the MFU calculation uses a modified script whose FLOP accounting is unspecified, and multiple table cells are marked 'Loss Scale Error.' These issues affect the headline efficiency numbers directly, so the significance of the contribution cannot be assessed until they are resolved.

major comments (5)
  1. [§3, Tables 2 and 3] The efficiency comparison is not commensurable. In Table 2, ECHO models use systematically larger batch sizes than baselines (e.g., LLaMA-3B: baseline Bs=20 vs ECHO Bs=30 on 4 and 8 NPUs; LLaMA-7B: baseline Bs=10 vs ECHO Bs=18 on 8 NPUs). MFU and tokens/sec are batch-size-sensitive, so the higher MFU and throughput could be due to the larger batch rather than the architecture. Table 3, despite its caption saying 'Batch size (BS) ... are reported,' does not list any batch sizes. The caption 'Final training loss values are reported under an equal training budget' is not operationalized: no definition of the budget is given, and the statement in §3 that each model is pretrained on 4B tokens through 1000 training steps is arithmetically inconsistent with the listed batch sizes at seq_len=2048 (e.g., Bs=50 over 1000 steps is approximately 102M tokens). The authors must run matched-batch-size comparisons, state the exact token budget and step count for every configuration, and justify that any batch-size differences are part of the method's intended benefit rather than an uncontrolled variable.
  2. [§3, TinyLLaMA MFU paragraph] The MFU computation is not reproducible. The paper states that the LLaMA-Factory MFU script was 'modified' and that 'updating the theoretical FLOPs based on your computing devices' was needed, but it does not specify the FLOP formula used for ECHO-LLaMA versus the baseline. If the denominator still counts full self-attention FLOPs for all layers, then ECHO-LLaMA's reduced FLOP count would automatically raise its MFU without any improvement in hardware utilization. The authors should provide the exact FLOP equations for both the baseline and ECHO architectures, including how the shared KV and cross-attention layers are counted.
  3. [§2, Eq. (6)] The cross-attention definition is ambiguous or incorrect. Eq. (6) writes Cross-Attn(X) = softmax(X_N W_Q (K_shared)^T / sqrt(d_k)) V_shared, using X_N for the query regardless of the layer index. If this is literal, every cross-decoder layer would attend from the same input X_N, which does not match the description in Figure 1 that each cross-decoder layer computes cross-attention with its own query from the shared KV. If X_N is a typo for X_l, the equation should be corrected. This is central to the architecture definition, so the intended formula must be stated precisely.
  4. [§3, Scaling Law Validation and Figure 2] The paper conflates two training setups. The scaling-law subsection and Figure 2 caption say 'Each model is pretrained from scratch on 4B tokens through 1000 steps,' but the main method (Section 2 and Table 1) adapts an existing pretrained TinyLLaMA checkpoint, and Appendix B says ECHO was 'pretraining from scratch.' These are different training budgets and different claims. The paper should clearly separate the from-scratch experiments from the adaptation experiments and give the token count, step count, and batch size for each.
  5. [Tables 2 and 3] Several entries are marked 'Loss Scale Error' (ECHO-LLaMA-7B on 4 NPUs in Table 2; TinyLLaMA baseline and ECHO on 8 GPUs in Table 3) with no explanation. These missing values mean the claimed up-to-14% loss reduction is not fully supported by the tables. The authors should either provide the missing loss values (or explain the loss-scale issue and how it was resolved) or explicitly list which configurations are excluded from the loss comparisons.
minor comments (6)
  1. [Throughout] There is an inconsistent use of 'TinyLLaMA' and 'TinyLlama' (including in the reference to the Hugging Face model); please standardize.
  2. [§2, text near Eq. (6)] The text has several typos and grammatical issues, e.g., 'Let's Xl−1' and 'share KV'; these should be fixed.
  3. [Table 1] The last column reports throughput improvement but gives only percentage values, not absolute tokens/sec; reporting absolute throughput for each row would be more informative.
  4. [§3, Ablation Study] The ablation for selecting S (Figure 4) uses the same loss curves that motivate the final S=150 choice; because the hyperparameter is selected on the basis of the training loss shown, the 'justification' is partly circular. Please state whether this is training loss or held-out loss and acknowledge this selection dependency.
  5. [Appendix E] Appendix E duplicates the text of Section 3's 'Efficiency Comparison' paragraph almost verbatim; please remove the duplication.
  6. [Conclusion] The conclusion says 'training speeds up to 50% faster,' while the abstract claims up to 77% higher token-per-second throughput; these headline numbers should be reconciled.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported efficiency and loss gains are empirical comparisons, not derived from fitted inputs or load-bearing self-citations.

full rationale

This paper's central claims—higher tokens/sec, higher MFU, lower loss, and faster inference—are empirical measurements comparing ECHO-LLaMA against LLaMA baselines on standard benchmarks and hardware. The architecture is defined by explicit equations (1)-(6), where self-attention in converted layers is replaced by cross-attention over a shared KV computed from layer N; these equations do not presuppose the outcome. No fitted parameter is renamed as a prediction: the adaptation hyperparameters (S≈150, T_stage≈4M, T_final≈4B) are selected from an ablation showing diminishing returns, and the downstream loss and benchmark comparisons are then measured, not derived from that ablation. The self-citations to EchoAtt and GQKVA are background context and are not load-bearing justifications for the reported gains; no uniqueness theorem or ansatz is imported from prior work. The efficiency comparison does have potential measurement-validity concerns—batch sizes differ between ECHO and baseline in Tables 2 and 3, and the MFU theoretical-FLOP update is unspecified—but unequal batch sizes and undisclosed FLOP formulas are threats to commensurability, not to the logical circularity of the derivation. The Limitations section appropriately notes that broader generalization remains unexplored. Accordingly, no circular step can be exhibited by quote-and-reduction, so the circularity score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

No new physical entities are introduced. The free parameters are architecture and training-strategy choices that are tuned by hand or by ablation rather than derived from first principles. The main assumptions concern the validity of the adaptation recipe and the correctness of the MFU measurement.

free parameters (5)
  • N or p, fraction of shared-KV layers = 25% or 50% of layers
    Chosen by hand as two configurations; no principled derivation. It controls the architecture and therefore all speed and accuracy trade-offs.
  • S, training steps per adaptation stage = 150
    Selected from an ablation study (Figure 4) showing diminishing returns after 150 steps. This is a tuned hyperparameter, not derived.
  • T_stage, token budget per adaptation stage = approximately 4M tokens
    Hand-chosen; no sensitivity analysis across values is provided.
  • T_final, final fine-tuning token budget = approximately 4B tokens
    Hand-chosen; the loss-comparison claims depend on this budget being equal across models, which is not the case due to differing batch sizes.
  • Batch size per device configuration = varies, e.g. batch size 10 vs 18 for 7B on 8 NPUs
    Not held constant between baseline and ECHO models, confounding the loss and throughput comparisons in Tables 2 and 3.
assumptions (4)
  • domain assumption Incremental layer-wise fine-tuning can convert a pretrained LLaMA into a shared-KV architecture without catastrophic forgetting.
    The adaptation strategy assumes 150 gradient steps per layer preserve pretrained knowledge. This is referenced to Kirkpatrick et al. but not independently verified here.
  • domain assumption Cross-attention with a shared global KV pair is expressively sufficient to preserve language modeling performance.
    The architecture's validity depends on later layers working with only a global KV computed from layer N. YOCO provides prior evidence, but this is assumed for pretrained checkpoints.
  • domain assumption The modified LLaMA-Factory MFU script correctly estimates model FLOPs for ECHO across GPU and NPU.
    Section 3 states the script needed modifications including updating theoretical FLOPs, but the exact updated formula is not given.
  • ad hoc to paper Equation (6) either uses X_N as the query for every cross-attention layer, or it is a typo for X_l.
    If Eq. (6) is literal, later layer outputs depend only on the middle layer output, which is not how cross-decoder layers are described. The paper does not clarify which reading is correct.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ECHO-LLaMA: Efficient Caching for High-Performance LLaMA Training." pith.science (2026). https://pith.science/paper/3LEOHIE7

@misc{pith2026250517331,
  author       = {Pith},
  title        = {Pith review of: ECHO-LLaMA: Efficient Caching for High-Performance LLaMA Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3LEOHIE7}},
  note         = {Machine review of arXiv:2505.17331}
}
read the original abstract

This paper introduces ECHO-LLaMA, an efficient LLaMA architecture designed to improve both the training speed and inference throughput of LLaMA architectures while maintaining its learning capacity. ECHO-LLaMA transforms LLaMA models into shared KV caching across certain layers, significantly reducing KV computational complexity while maintaining or improving language performance. Experimental results demonstrate that ECHO-LLaMA achieves up to 77\% higher token-per-second throughput during training, up to 16\% higher Model FLOPs Utilization (MFU), and up to 14\% lower loss when trained on an equal number of tokens. Furthermore, on the 1.1B model, ECHO-LLaMA delivers approximately 7\% higher test-time throughput compared to the baseline. By introducing a computationally efficient adaptation mechanism, ECHO-LLaMA offers a scalable and cost-effective solution for pretraining and finetuning large language models, enabling faster and more resource-efficient training without compromising performance.

Figures

Figures reproduced from arXiv: 2505.17331 by the authors.

Figure 1
Figure 1. ECHO-LLaMA: The architecture uses shared [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Comparison between of training throughput (Tokens/sec) and final loss for ECHO-LLaMA models and [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Scaling law diagram comparing training loss [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Training Loss vs. Training Steps for Differ [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: GPU memory usage comparison between the baseline TinyLLaMA and ECHO-TinyLLaMA across different sequence lengths. ECHO-TinyLLaMA consis￾tently consumes nearly half the memory compared to TinyLLaMA, as indicated by the annotated reduction ratios. ciently transform a pret…
Figure 6
Figure 6. Figure 6: Comparison of training throughput (Tokens/sec) and final loss for ECHO and non-ECHO models on [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Training results for the LLaMA-125M LLM on GPU (V100) and NPU (910B). The columns represent [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Training results for TinyLLaMA-1.1B LLM on GPU (V100) and NPU (910B). The columns represent (1) [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Training results for LLaMA-3B on GPU (V100) and NPU (910B). The columns represent (1) Train Loss [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Training results for LLaMA-7B on GPU (V100) and NPU (910B). The columns represent (1) Train [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: LLaMA-7B: Comparing Incremental Sharing over Full-Stage Sharing for shared-KV adaptation [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: LLaMA-7B: Comparing Incremental Sharing over Full-Stage Sharing for shared-KV adaptation [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 8 canonical work pages

  1. [1]

    https://huggingface.co/TinyLlama/TinyLlama_v1.1

    T iny L lama/ T iny L lama\_v1.1 · H ugging F ace --- huggingface.co. https://huggingface.co/TinyLlama/TinyLlama_v1.1. [Accessed 27-01-2025]

    work page 2025

  2. [2]

    Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant Nair, Ilya Soloveychik, and Purushotham Kamath. 2024. Keyformer: Kv cache reduction through key tokens selection for efficient generative inference. Proceedings of Machine Learning and Systems, 6:114--127

    2024

  3. [3]

    Walid Ahmed, Habib Hajimolahoseini, Austin Wen, and Yang Liu. 2023. Speeding up resnet architecture with layers targeted low rank decomposition. arXiv preprint arXiv:2309.12412

    work page Pith review arXiv 2023

  4. [4]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732

    arXiv 2021

  5. [5]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

    arXiv 2023

  6. [6]

    Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150

    arXiv 2020

  7. [7]

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1--45

    2024

  8. [8]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    arXiv 2021

Show all 36 references
  1. [9]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1--113

    2023

  2. [10]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    2021 arXiv

  3. [11]

    Stefan Elfwing, Eiji Uchibe, and Kenji Doya. 2018. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural networks, 107:3--11

    2018

  4. [12]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024 doi

  5. [13]

    Habib Hajimolahoseini, Walid Ahmed, and Yang Liu. 2023. Training acceleration of low-rank decomposed networks using sequential freezing and rank quantization. arXiv preprint arXiv:2309.03824

    2023 arXiv

  6. [14]

    Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2021 a . Aligning ai with shared human values. Proceedings of the International Conference on Learning Representations (ICLR)

    2021

  7. [15]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021 b . Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR)

    2021

  8. [16]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021 c . Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874

    2021 arXiv

  9. [17]

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556

    2022 arXiv

  10. [18]

    Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. 2023. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint a...

    2023 arXiv

  11. [19]

    Farnoosh Javadi, Walid Ahmed, Habib Hajimolahoseini, Foozhan Ataiefard, Mohammad Hassanpour, Saina Asani, Austin Wen, Omar Mohamed Awad, Kangling Liu, and Yang Liu. 2023. Gqkva: Efficient pre-training of transformers by grouping queries, keys, and values. arXiv preprint arXiv:...

    2023 arXiv

  12. [20]

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of...

    2017

  13. [21]

    Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. 2024. \ InfiniGen \ : Efficient generative inference of large language models with dynamic \ KV \ cache management. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 155--172

    2024

  14. [22]

    Bingli Liao and Danilo Vasconcellos Vargas. 2024. Beyond kv caching: Shared attention for efficient llms. arXiv preprint arXiv:2407.12866

    2024 arXiv

  15. [23]

    Spyros Makridakis, Fotios Petropoulos, and Yanfei Kang. 2023. Large language models: Their success and impact. Forecasting, 5(3):536--549

    2023

  16. [24]

    Silvia Milano, Joshua A McGrane, and Sabina Leonelli. 2023. Large language models challenge the future of higher education. Nature Machine Intelligence, 5(4):333--334

    2023

  17. [25]

    Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. 2023. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems, 5:606--624

    2023

  18. [26]

    Hossein Rajabzadeh, Aref Jafari, Aman Sharma, Benyamin Jami, Hyock Ju Kwon, Ali Ghodsi, Boxing Chen, and Mehdi Rezagholizadeh. 2024. Echoatt: Attend, copy, then adjust for more efficient large language models. arXiv preprint arXiv:2409.14595

    2024

  19. [27]

    Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, and Furu Wei. 2024. You only cache once: Decoder-decoder architectures for language models. arXiv preprint arXiv:2405.05254

    2024 arXiv

  20. [28]

    Hanlin Tang, Yang Lin, Jing Lin, Qingsen Han, Shikuan Hong, Yiwu Yao, and Gongyi Wang. 2024. Razorattention: Efficient kv cache compression through retrieval heads. arXiv preprint arXiv:2407.15891

    2024 arXiv

  21. [29]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  22. [30]

    Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. 2023. Gated linear attention transformers with hardware-efficient training. arXiv preprint arXiv:2312.06635

    2023 arXiv

  23. [31]

    Biao Zhang and Rico Sennrich. 2019. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32

    2019

  24. [32]

    Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B Hashimoto. 2024. Benchmarking large language models for news summarization. Transactions of the Association for Computational Linguistics, 12:39--57

    2024

  25. [33]

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher R \'e , Clark Barrett, et al. 2023. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Proces...

    2023

  26. [34]

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. 2024. http://arxiv.org/abs/2403.13372 Llamafactory: Unified efficient fine-tuning of 100+ language models . In Proceedings of the 62nd Annual Meeting of the Association for Compu...

    2024 arXiv

  27. [35]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  28. [36]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.