Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

LASP-2: Rethinking Sequence Parallelism for Linear Attention and Its Hybrid

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that sequence-parallel linear attention can be reorganized around a single AllGather of chunk memory states, giving 15.2% and 36.6% training-speed gains over LASP and Ring Attention at 2048K sequence length.

desk verdict A real algorithmic improvement for sequence-parallel linear attention, but the paper's own cost model overstates the traffic reduction and the abstract conflicts with its own OOM table. read the letter →

arxiv 2502.07563 v1 pith:A6ZVQGDG submitted 2025-02-11 cs.LG cs.AIcs.CL

Weigao Sun , Disen Lan , Yiran Zhong , Xiaoye Qu , Yu Cheng This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords sequenceparallelismlinearattentionAllGathercollectivememorystateslongtraininghybridmodelsdistributed
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces LASP-2, a sequence-parallel training scheme for linear attention transformers on very long inputs. Because each attention chunk can be summarized by a $d\times d$ memory state $M_t=K_t^\top V_t$, one AllGather can place all chunk states on every device, after which prefix sums and a local multiplication produce the output. The paper argues this replaces the chained ring-style sends of the earlier LASP method with a single well-optimized collective, raising both communication and computation parallelism, and reports 15.2% higher throughput than LASP and 36.6% higher than Ring Attention at sequence length 2048K across 64 GPUs. A hybrid extension, LASP-2H, applies the same AllGather pattern to standard attention layers so mixed linear/softmax models share one SP design. If the claim holds, multi-million-token training of linear and hybrid transformers becomes faster while communication overhead stays independent of sequence length.

What carries the argument

The central object is the memory state $M_t = K_t^\top V_t \in \mathbb{R}^{d\times d}$, computed per chunk from that chunk's keys and values, plus its all-gathered prefix sums. Because linear attention satisfies $O = Q(K^\top V) = QM$, each chunk's contribution to every later position is fully summarized by its $d\times d$ state; LASP-2 uses one AllGather to collect all these state matrices, then prefix/suffix sums to build $M_{1:t-1}$ for the inter-chunk output and a masked intra-chunk term for causality. This object is what makes the communication volume independent of sequence length and what lets one collective replace a ring of sends.

What would settle it

Run LASP-2 and LASP-1 on the same cluster and sequence length, profile only the communication phases, and record both wall-clock time and per-device bytes; if the AllGather transfers $(W-1)BHd^2$ bytes per device (same as the ring) and the total communication time scales with bytes rather than with step count, then the claimed $W-1$ traffic reduction is not real and the measured speedup must be attributed to launch overhead and overlap.

Watch

Extended reading notes

Core claim

The paper's central claim is that in LASP-2 the forward pass needs exactly one AllGather of the memory states $[M_t]_1^T$, each of shape $d\times d$, followed by a sum or prefix-sum to get $M_{1:T}$; the backward pass needs one more AllGather on the gradients $dM_t$. With causal masking, the computation is decomposed into a parallel intra-chunk term $O_{t,\mathrm{intra}}=[(Q_tK_t^\top)\odot\Psi]V_t$ and an inter-chunk term $O_{t,\mathrm{inter}}=Q_tM_{1:t-1}$ built from gathered states. Thus the paper's step-counting model gives two communication steps per iteration instead of $2(W-1)$ for LASP-1, and the per-step traffic $BHd^2$ is independent of sequence length. LASP-2H transfers the same single-collective principle to standard attention by AllGathering $K_t,V_t$, and experiments on Linear-Llama3 up to 2048K tokens support the efficiency claim.

Load-bearing premise

The load-bearing premise is that replacing the ring of point-to-point sends with one AllGather is genuinely cheaper in wall-clock time; the paper's cost model counts communication steps rather than bytes, and an AllGather of all $W$ states moves the same per-device bytes as the ring it replaces.

Editorial extensions

If this is right

  • At sequence length 2048K across 64 A100 GPUs, LASP-2 reports 15.2% higher throughput than LASP and 36.6% higher than Ring Attention.
  • Communication per step stays at $BHd^2$ bytes regardless of sequence length, so the relative cost shrinks as sequences get longer.
  • Each iteration uses only two AllGather collectives (forward and backward), versus $2(W-1)$ point-to-point hops for LASP-1.
  • For hybrid models, LASP-2H gives both linear and standard attention layers the same AllGather-based design, gathering $K_t,V_t$ for standard layers and $M_t$ for linear layers.
  • With 128 GPUs, the method trains a Linear-Llama3-1B model on a 2048K-token sequence without running out of memory, keeping per-GPU usage around 57.8 GB.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The Section 3.4 cost model counts steps, not bytes; an AllGather of all $W$ states moves $(W-1)BHd^2$ bytes per device, the same volume as LASP-1's ring, so the real advantage is likely fewer launches, lower latency, and easier overlap rather than reduced data movement.
  • The same chunk-state-then-AllGather pattern should transfer to other linear RNNs with matrix-valued additive states, such as SSM-style recurrences, since their recurrence has the same sum-of-states form; the paper does not make this claim.
  • LASP-2H's AllGather-based standard attention trades higher latency for mask flexibility, suggesting it will combine naturally with variable-length packing and document masks, which ring-based strategies handle less easily.
  • On very fast interconnects the throughput gap over ring methods should narrow because collective latency dominates less; a direct byte-level comparison would quantify how much of the reported speedup is overlap versus raw traffic.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. LASP-2 proposes a new sequence-parallelism scheme for training linear-attention and hybrid linear/standard-attention transformers. Each rank computes its chunk-local memory state M_t = K_t^T V_t (shape [B,H,d,d]), performs a single AllGather to replicate all W states, and then computes its output Q_t M_{1:T} locally; the masked autoregressive variant separates intra-chunk (left-product with causal mask) and inter-chunk (gathered prefix-sum) computations, and the backward passes use an analogous AllGather on gradients dM_t. The paper claims that this reorganization reduces communication traffic by a factor of W-1 relative to the authors' earlier LASP-1, improves computation and communication parallelism, and reports throughput gains of 15.2% over LASP-1 and 36.6% over Ring Attention at sequence length 2048K on 64 A100 GPUs using a Linear-Llama3-1B model, together with scalability measurements up to 4096K on 128 GPUs and convergence results on several linear-attention variants. An extension, LASP-2H, applies the same allgather-based design to standard attention layers for hybrid models.

Significance. The algorithmic idea is clean and directly derived from the associativity of the right-product form of linear attention; there are no fitted parameters, the forward/backward algorithms are fully specified, code is released, and the paper reports concrete per-configuration throughput and memory numbers (Table 6) that are falsifiable. The single-AllGather design has genuine operational virtues over the ring-style P2P exchange in LASP-1: one collective launch instead of W-1 send/receive pairs, removal of the serial inter-chunk dependency so that all ranks compute their inter-chunk outputs concurrently, and straightforward overlap with intra-chunk computation in the masked case. If the measured speedups survive a corrected communication-accounting analysis, this is a practically useful, low-risk improvement for very-long-sequence training. The significance is substantially tempered, however, by the finding that the paper's headline traffic-reduction argument (Section 3.4) is a step-counting artifact: byte-for-byte, the AllGather moves the same volume as LASP-1's ring.

major comments (3)
  1. [Section 3.4] Section 3.4, communication-traffic formulas (2(W-1)IBHd^2 for LASP-1 vs 2IBHd^2 for LASP-2): the model counts each AllGather as one message of BHd^2 bytes, but an AllGather that replicates all W memory states moves (W-1)*BHd^2 bytes into each rank and the same volume out of each rank; including forward and backward passes this is 4(W-1)*BHd^2 per rank per iteration, exactly the volume of LASP-1's 2(W-1) send/receive steps. The claim that LASP-2 reduces communication traffic by a factor of W-1 is therefore a step-counting artifact and is not supported by byte accounting. The genuine benefits are fewer collective launches, the removal of LASP-1's serial inter-chunk dependency, and overlap of the AllGather with intra-chunk computation; Sections 1 and 3.3 and the abstract should be revised to attribute the measured speedups to these effects rather than to reduced traffic.
  2. [Abstract / Section 4.2 / Table 6] The headline result, training speed improvements of 15.2% over LASP and 36.6% over Ring Attention at sequence length 2048K across 64 GPUs, is contradicted by Table 6, which reports OOM for LASP-2 at 2048K with 64 GPUs, and Figure 3's caption states that the speed comparisons used 64 GPUs with batch size 1. The configuration that produced the 2048K speedup numbers (for example a different batch size, a different SP size, or an error in Table 6) must be identified and reported consistently before the abstract's central claim can be verified.
  3. [Section 4.1] The baselines Megatron-SP and Ring Attention are evaluated on linear-attention layers without the right-product kernel trick, keeping each method's original standard-attention computational manner. The reported 36.6% gain over Ring Attention at 2048K therefore conflates the quadratic-to-linear compute saving with the sequence-parallelism communication advantage. The comparison against LASP-1 is the fair right-product-aware baseline; the Ring Attention comparison should either be re-run with a right-product-first linear kernel inside the ring communication pattern or explicitly re-scoped, and the paper's central empirical claim should rest on the LASP-1 comparison.
minor comments (5)
  1. [Algorithm 3 (Appendix A.1)] Algorithm 3, line 5: the notation 'dM1:T = Sum([dM]^T_{t+1})' reads as a suffix sum over chunks t+1 through T, but the unmasked backward pass requires dM_{1:T} to be the sum of dM_i over all chunks i = 1..T; please correct the subscript and state explicitly how this quantity differs from the SuffixSum used in Algorithm 4.
  2. [Section 4.1] Typos and grammar: 'Ring Attentoin' (Section 4.1), 'decices' (Section 3.3), 'lenghth' (Appendix A.5.3), 'striked a balance' (Section 4.4), and 'each memory state will has' (Section 3.4) should be corrected.
  3. [Table 6] Table 6 reports identical per-GPU memory (25.6 GB) for sequence lengths 2K through 16K at every fixed GPU count; please clarify whether this plateau reflects a memory floor from model weights and optimizer states, since otherwise the activation-memory scaling shown in Figure 4 is not visible at these settings.
  4. [Figures 1 and 2] Figure 1's caption ('Colored chunks represent inter-chunks') does not make clear which regions are intra-chunk versus inter-chunk, and Figure 2's AG/RS and AG/No labels are too small to read at print resolution; both figures should be redrawn for clarity.
  5. [Section 3.1] The statement that the communication cost for the all-gather operation is independent of the sequence or chunk length is true per message, but the total gathered volume is W*BHd^2 and grows with the SP size; the abstract's phrasing that the communicated tensors' sizes are independent of the sequence length should be qualified so that readers do not infer constant communication as the device count grows.

Circularity Check

0 steps flagged · score 1.0 of 10

LASP-2 is not circular: its AllGather workflow is forced by the associative right-product form of linear attention, and the headline speedups are measured against external baselines; the Sec. 3.4 'W−1 traffic reduction' is a step-counting artifact (a correctness caveat), not a circular derivation.

full rationale

LASP-2's core algorithm is derived from the algebraic identity O_t = Q_t * sum_t (K_t^T V_t). After each rank forms M_t = K_t^T V_t, the identity forces the only needed global datum to be the sum (or, with causal masking, the prefix sums) of these d x d states, so the single AllGather workflow follows from the linear-attention definition rather than from the result being claimed. No parameter is fitted and no benchmark number is generated from the cost model; the reported 15.2% over LASP-1 and 36.6% over Ring Attention are measured on 64 A100 GPUs against independent external baselines. LASP-1 is the authors' prior method, but it is used as a concrete algorithmic baseline with pseudocode provided in the appendix, not as an unverified uniqueness or ansatz justification. The causal-mask decomposition is inherited from prior work (including the same authors' LASP-1), but it is independently checkable and does not smuggle the LASP-2 conclusion. The one caveat is Section 3.4, Theoretical Cost Analysis: the theoretical claim that LASP-2 cuts communication traffic by a factor of W-1 counts one AllGather as a single BHd^2 step, whereas a W-way AllGather of W memory states moves (W-1)BHd^2 per rank in each direction, the same byte volume as LASP-1's ring. That is a correctness risk in the cost model, not a circularity, because the measured speedups stand independently of that equation. No circular steps are present.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

This is an engineering paper, so the ledger is light. No free parameters are fitted to data; the method's hyperparameters (chunk size, SP size) are chosen by system constraints, not by fitting. The main assumptions are the standard associativity of linear attention and two methodological choices in the evaluation: the step-counting communication cost model in Section 3.4 and the decision to run baseline SP methods without the right-product kernel trick in Section 4.1. No invented entities are introduced.

assumptions (3)
  • standard math Associativity of matrix multiplication lets (QK^T)V = Q(K^T V) for the unnormalized linear attention kernel.
    Invoked in Eq. (3) and Section 3.1; it is the reason M_t = K_t^T V_t is the only state that must be exchanged.
  • ad hoc to paper A single AllGather of all W memory states can be modeled as one communication step with volume BHd^2, independent of W.
    Section 3.4 uses this step-counting model to claim LASP-2 traffic is 2IBHd^2 versus LASP-1's 2(W-1)IBHd^2; a byte-level model would count (W-1) moves per device for both methods.
  • domain assumption Throughput differences against Ring Attention and Megatron-SP are attributable to SP communication design even though those baselines do not use the right-product kernel trick.
    Section 4.1 explicitly disables the right-product trick for baselines, so the comparison conflates communication strategy with attention-computation complexity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LASP-2: Rethinking Sequence Parallelism for Linear Attention and Its Hybrid." pith.science (2026). https://pith.science/paper/A6ZVQGDG

@misc{pith2026250207563,
  author       = {Pith},
  title        = {Pith review of: LASP-2: Rethinking Sequence Parallelism for Linear Attention and Its Hybrid},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/A6ZVQGDG}},
  note         = {Machine review of arXiv:2502.07563}
}
read the original abstract

Linear sequence modeling approaches, such as linear attention, provide advantages like linear-time training and constant-memory inference over sequence lengths. However, existing sequence parallelism (SP) methods are either not optimized for the right-product-first feature of linear attention or use a ring-style communication strategy, which results in lower computation parallelism, limits their scalability for longer sequences in distributed systems. In this paper, we introduce LASP-2, a new SP method to enhance both communication and computation parallelism when training linear attention transformer models with very-long input sequences. Compared to previous work LASP, LASP-2 rethinks the minimal communication requirement for SP on linear attention layers, reorganizes the whole communication-computation workflow of LASP. In this way, only one single AllGather collective communication is needed on intermediate memory states, whose sizes are independent of the sequence length, leading to significant improvements of both communication and computation parallelism, as well as their overlap. Additionally, we extend LASP-2 to LASP-2H by applying similar communication redesign to standard attention modules, offering an efficient SP solution for hybrid models that blend linear and standard attention layers. Our evaluation on a Linear-Llama3 model, a variant of Llama3 with linear attention replacing standard attention, demonstrates the effectiveness of LASP-2 and LASP-2H. Specifically, LASP-2 achieves training speed improvements of 15.2% over LASP and 36.6% over Ring Attention, with a sequence length of 2048K across 64 GPUs. The Code is released as a part of: https://github.com/OpenSparseLLMs/Linear-MoE.

Figures

Figures reproduced from arXiv: 2502.07563 by the authors.

Figure 1
Figure 1. Computation Decomposition in LASP-2 with masking. Colored chunks represent inter-chunks. 3.2. LASP-2 with Masking In autoregressive tasks, the mask matrix Ψ ∈ {−∞, 1} N×N is typically a lower triangular matrix, where Ψij = 1 for i ≥ j and Ψij = −∞ when i < j. This structure enforces a causal constraint during computation. Specifically, when calculating O = Softmax(QK⊤ ⊙Ψ)V, it becomes impossible to leverage the asso… view at source ↗
Figure 2
Figure 2. Visualization of LASP-2H on Linear Attention and Standard Attention hybrid model. We exemplify LASP￾2H on the hybrid layers of linear attention and standard attention modules with both TP and SP (both have a dimension of 2). The communication operations colored in yellow and green are for TP and SP, respectively. AG/RS: all-gather in forward and reduce-scatter in backward, and vice versa. AG/No: all-gather in forwar… view at source ↗
Figure 3
Figure 3. Speed Comparison (tokens/s). Experiments were carried out on a pure Linear-Llama3-1B model, utilizing the basic linear attention module. A total of 64 A100 GPUs were employed, and the SP size T was also set to 64. To accommodate very-long sequence lengths, such as 2048K, the batch size was kept fixed at 1 throughout this experiment. performance advantage became increasingly prominent as sequence lengths grew longer.… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Scalability Results. Experiments were conducted on a pure Linear-Llama3-1B model using the Basic Linear Attention module. SP size T was always equal to number of GPUs. Batch size was fixed as 1 to accommodate very-long sequence lengths, e.g., 2048K. The sign "×" with a…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Training-Memory Regression in MLA Sequence Parallelism: Why Megatron-Core Forbids Absorption, and LAGA -- a Communication-Efficient Fix

    cs.DC 2026-07 conditional novelty 6.0 of 10

    LAGA replaces MLA training's per-head K/V all-to-all with a latent all-gather and local up-projection, matching explicit-form memory while cutting collective communication ~1.98x.

Reference graph

Works this paper leans on

37 extracted references · 8 canonical work pages · cited by 1 Pith paper

  1. [1]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774,

    arXiv

  2. [3]

    Aksenov, Y ., Balagansky, N., Vaina, S. M. L. C., Shaposh- nikov, B., Gorbatovski, A., and Gavrilov, D. Linear transformers with learnable kernel functions are better in- context models. arXiv preprint arXiv:2402.10644,

    arXiv

  3. [5]

    Searching for Needles in a Haystack: On the Role of Incidental Bilingualism in PaLM's Translation Capability

    Briakou, E., Cherry, C., and Foster, G. Searching for needles in a haystack: On the role of incidental bilin- gualism in palm’s translation capability. arXiv preprint arXiv:2305.10266,

    work page Pith review arXiv

  4. [6]

    Flashattention-2: Faster attention with bet- ter parallelism and work partitioning

    Dao, T. Flashattention-2: Faster attention with bet- ter parallelism and work partitioning. arXiv preprint arXiv:2307.08691,

    arXiv

  5. [7]

    and Gu, A

    Dao, T. and Gu, A. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060,

    arXiv

  6. [9]

    The Llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  7. [10]

    and Dao, T

    Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752,

    arXiv

  8. [11]

    Measuring mas- sive multitask language understanding

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint arXiv:2009.03300,

    arXiv 2009

Show all 37 references
  1. [12]

    M., and Malach, E

    Jelassi, S., Brandfonbrener, D., Kakade, S. M., and Malach, E. Repeat after me: Transformers are bet- ter than state space models at copying. arXiv preprint arXiv:2402.01032,

    arXiv

  2. [14]

    Minimax-01: Scaling foundation models with lightning attention

    Li, A., Gong, B., Yang, B., Shan, B., Liu, C., Zhu, C., Zhang, C., Guo, C., Chen, D., Li, D., et al. Minimax-01: Scaling foundation models with lightning attention. arXiv preprint arXiv:2501.08313,

    arXiv

  3. [15]

    Jamba: A hybrid transformer-mamba language model

    Lieber, O., Lenz, B., Bata, H., Cohen, G., Osin, J., Dalmedi- gos, I., Safahi, E., Meirom, S., Belinkov, Y ., Shalev- Shwartz, S., et al. Jamba: A hybrid transformer-mamba language model. arXiv preprint arXiv:2403.19887,

    arXiv

  4. [16]

    10 LASP-2 Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Biderman, S., Cao, H., Cheng, X., Chung, M., Der- czynski, L., Du, X., Grella, M., Gv, K., He, X., Hou, H., Kazienko, P., Kocon, J., Kong, J., Koptyra, B., Lau, H., Lin, J., Mantri, K. S. I., Mom, F., Sa...

    2023

  5. [17]

    doi: 10.18653/v1/2023.findings-emnlp

    Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-emnlp

    2023 doi

  6. [19]

    R., Vasu, P

    Pouransari, H., Li, C.-L., Chang, J.-H. R., Vasu, P. K. A., Koc, C., Shankar, V ., and Tuzel, O. Dataset decomposi- tion: Faster llm training with variable sequence length curriculum. arXiv preprint arXiv:2405.13226,

    arXiv

  7. [20]

    TransNormerLLM: A faster and better large language model with improved transnormer

    Qin, Z., Li, D., Sun, W., Sun, W., Shen, X., Han, X., Wei, Y ., Lv, B., Luo, X., Qiao, Y ., et al. TransNormerLLM: A faster and better large language model with improved transnormer. 2023a. Qin, Z., Li, D., Sun, W., Sun, W., Shen, X., Han, X., Wei, Y ., Lv, B., Yuan, F., Luo, ...

    arXiv

  8. [21]

    Samba: Simple hybrid state space models for efficient unlimited context language modeling

    Ren, L., Liu, Y ., Lu, Y ., Shen, Y ., Liang, C., and Chen, W. Samba: Simple hybrid state space models for efficient unlimited context language modeling. arXiv preprint arXiv:2406.07522,

    arXiv

  9. [23]

    Scaling laws for linear complexity language models

    Shen, X., Li, D., Leng, R., Qin, Z., Sun, W., and Zhong, Y . Scaling laws for linear complexity language models. arXiv preprint arXiv:2406.16690,

    arXiv

  10. [24]

    Megatron-LM: Training multi- billion parameter language models using model paral- lelism

    Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-LM: Training multi- billion parameter language models using model paral- lelism. arXiv preprint arXiv:1909.08053,

    1909 arXiv

  11. [25]

    Sun, W., Qin, Z., Li, D., Shen, X., Qiao, Y ., and Zhong, Y

    URL https://huggingface.co/ datasets/cerebras/SlimPajama-627B. Sun, W., Qin, Z., Li, D., Shen, X., Qiao, Y ., and Zhong, Y . Linear attention sequence parallelism.arXiv preprint arXiv:2404.02882, 2024a. Sun, W., Qin, Z., Sun, W., Li, S., Li, D., Shen, X., Qiao, Y ., and Zhong,...

    arXiv

  12. [27]

    LongVILA: Scal- ing long-context visual language models for long videos

    Xue, F., Chen, Y ., Li, D., Hu, Q., Zhu, L., Li, X., Fang, Y ., Tang, H., Yang, S., Liu, Z., et al. LongVILA: Scal- ing long-context visual language models for long videos. arXiv preprint arXiv:2408.10188,

    arXiv

  13. [28]

    Gated linear attention transformers with hardware-efficient train- ing

    Yang, S., Wang, B., Shen, Y ., Panda, R., and Kim, Y . Gated linear attention transformers with hardware-efficient train- ing. arXiv preprint arXiv:2312.06635,

    arXiv

  14. [29]

    Par- allelizing linear transformers with the delta rule over se- quence length

    Yang, S., Wang, B., Zhang, Y ., Shen, Y ., and Kim, Y . Par- allelizing linear transformers with the delta rule over se- quence length. arXiv preprint arXiv:2406.06484,

    arXiv

  15. [30]

    Boosting distributed training performance of the un- padded bert model

    Zeng, J., Li, M., Wu, Z., Liu, J., Liu, Y ., Yu, D., and Ma, Y . Boosting distributed training performance of the un- padded bert model. arXiv preprint arXiv:2208.08124 ,

    arXiv

  16. [31]

    ByteTransformer: A high- performance transformer boosted for variable-length in- puts

    Zhai, Y ., Jiang, C., Wang, L., Jia, X., Zhang, S., Chen, Z., Liu, X., and Zhu, Y . ByteTransformer: A high- performance transformer boosted for variable-length in- puts. In 2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS) , pp. 344–355. IEEE,

    2023

  17. [33]

    Pytorch FSDP: experiences on scaling fully sharded data parallel

    Zhao, Y ., Gu, A., Varma, R., Luo, L., Huang, C.-C., Xu, M., Wright, L., Shojanazeri, H., Ott, M., Shleifer, S., et al. Pytorch FSDP: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.11277,

    arXiv

  18. [34]

    As these techniques are variants of data parallelism, they integrate seamlessly with LASP

    offer methods for distributing model states such as optimizer states, gradients, and model parameters across all GPUs in the distributed system. As these techniques are variants of data parallelism, they integrate seamlessly with LASP. Their primary objective of minimizing the...

    2022

  19. [35]

    LASP-2 can manage variable sequence lengths efficiently by treating the entire batch as a single long sequence, streamlining the process without requiring padding

    or packing multiple sequences into a single batch and adjusting the attention mask accordingly (Ding et al., 2024; Pouransari et al., 2024). LASP-2 can manage variable sequence lengths efficiently by treating the entire batch as a single long sequence, streamlining the process...

    2024

  20. [37]

    Split Size of Gathering 2048 512 128 32 Number of Splits 1 4 16 64 Throughput 486183 486166 486169 486158 A.5.4

    is used. Split Size of Gathering 2048 512 128 32 Number of Splits 1 4 16 64 Throughput 486183 486166 486169 486158 A.5.4. Q UANTITATIVE SCALABILITY RESULTS See Table 6 in next page. 16 LASP-2 Table 6: Quantitative Scalability Results of LASP-2 on Throughput (tokens/sec) and Me...

  21. [936]

    findings-emnlp.936

    URL https://aclanthology.org/2023. findings-emnlp.936. Peng, B., Goldstein, D., Anthony, Q., Albalak, A., Al- caide, E., Biderman, S., Cheah, E., Ferdinan, T., Hou, H., Kazienko, P., et al. Eagle and Finch: RWKV with matrix- valued states and dynamic recurrence. arXiv preprint...

    2023 arXiv

  22. [2017]

    An empirical study of mamba- based language models

    Waleffe, R., Byeon, W., Riach, D., Norick, B., Kor- thikanti, V ., Dao, T., Gu, A., Hatamizadeh, A., Singh, S., Narayanan, D., et al. An empirical study of mamba- based language models. arXiv preprint arXiv:2406.07887,

    arXiv

  23. [2019]

    B., Zhou, P., and Fu, G

    Zhang, Y ., Yang, S., Zhu, R., Zhang, Y ., Cui, L., Wang, Y ., Wang, B., Freda Shi, Bailin Wang, W. B., Zhou, P., and Fu, G. Gated slot attention for efficient linear-time sequence modeling. arXiv preprint arXiv:2409.07146 ,

    arXiv

  24. [2020]

    Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980,

    arXiv

  25. [2021]

    Flashattention-3: Fast and accurate atten- tion with asynchrony and low-precision

    Shah, J., Bikshandi, G., Zhang, Y ., Thakkar, V ., Ramani, P., and Dao, T. Flashattention-3: Fast and accurate atten- tion with asynchrony and low-precision. arXiv preprint arXiv:2407.08608,

    arXiv

  26. [2022]

    Fewer truncations improve language modeling

    Ding, H., Wang, Z., Paolini, G., Kumar, V ., Deoras, A., Roth, D., and Soatto, S. Fewer truncations improve language modeling. arXiv preprint arXiv:2404.10830,

    arXiv

  27. [2023]

    GQA: Training generalized multi-query transformer models from multi-head check- points

    Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y ., Lebrón, F., and Sanghai, S. GQA: Training generalized multi-query transformer models from multi-head check- points. arXiv preprint arXiv:2305.13245,

    arXiv

  28. [2024]

    Z., Rudra, A., and Ré, C

    Arora, S., Eyuboglu, S., Zhang, M., Timalsina, A., Alberti, S., Dylan Zinsley, J. Z., Rudra, A., and Ré, C. Sim- ple linear attention language models balance the recall- throughput tradeoff. arXiv preprint arXiv:2402.18668 ,

    arXiv

  29. [2048]

    L" denotes linear Transformer layers and

    As the results shown in Table 3, LASP-2 with Basic Linear Attention is able to reach an approximate convergence performance with Ring Attention on the standard attention based model. Table 3: Convergence Performance on Bidirectional Language Modeling Task.Both training and val...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.