Pith. sign in

REVIEW 2 major objections 5 minor 45 references

LV-XAttn: Distributed Cross-Attention for Long Visual Inputs in Multimodal Large Language Models

T0 review · 2 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By rotating only small query blocks instead of large visual key-value blocks across GPUs, LV-XAttn makes distributed cross-attention compute-bound, delivering up to 45.85x faster attention and up to 10.62x faster end-to-end MLLM iteration…

desk verdict A genuinely new twist on Ring Attention for MLLM cross-attention with long visual inputs; solid forward-pass systems paper whose training-speedup claim needs a gradient check before it fully lands. read the letter →

arxiv 2502.02406 v3 pith:4RYX6ACM submitted 2025-02-04 cs.CV cs.AIcs.DCcs.LG

Tzu-Tao Chang , Shivaram Venkataraman This is my paper · ORCID
classification cs.CVcs.AIcs.DCcs.LG
keywords distributedcross-attentionmultimodallargelanguagemodelssequenceparallelismRingAttentionlongvideounderstandingcommunicationoverheadactivationrecomputationexact
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LV-XAttn claims that the cross-attention bottleneck in multimodal LLMs with long visual inputs is communication, not compute, and that the communication can be nearly eliminated by exploiting a structural asymmetry: the text-query sequence is far shorter than the visual key-value sequence ($S_Q \ll S_{KV}$). The method keeps large key-value blocks stationary on each GPU and rotates only the small query, output, and softmax-stat blocks around the ring, reducing per-round transfer volume from $O(S_{KV})$ to $O(S_Q)$ and hiding it under the attention FLOPs. On Llama 3-V, mPLUG-Owl3, and OpenFlamingo with long-video inputs, the paper reports up to 45.85x cross-attention speedup and up to 10.62x end-to-end iteration speedup over Ring Attention, with less than 0.42% overhead versus a no-communication baseline. A companion activation-recomputation trick, storing one shared copy of the visual tokens and recomputing keys and values in the backward pass, extends processable visual context by about 1.6x at under 8% iteration-time overhead. If correct, this removes the main reason long-video MLLM training and inference slow down when spread across GPUs.

What carries the argument

The load-bearing mechanism is the ring rotation pattern inverted: instead of circulating large key-value blocks (Ring Attention), LV-XAttn fixes each worker's KV block locally and circulates the small query block $Q_i$, the partial output $O_i$, and the softmax statistics $L_i$, rescaling partial outputs as in FlashAttention. The work is carried by the per-round runtime identity $\max\left(f\left(\frac{S_Q d}{n}, \frac{S_{KV} d}{n}\right), \operatorname{comm}\left(\frac{2 S_Q d}{n} + \frac{S_Q}{n}\right)\right)$ for LV-XAttn versus $\max\left(f\left(\frac{S_Q d}{n}, \frac{S_{KV} d}{n}\right), \operatorname{comm}\left(\frac{2 S_{KV} d}{n}\right)\right)$ for Ring Attention: when $S_{KV} \gg S_Q$ the first expression collapses to the compute term while the second stays communication-dominated. The secondary mechanism is MLLM-specific activation recomputation: because all cross-attention layers share the same visual features $y$, only one copy of $y$ is stored and the key-value tensors are recomputed in the backward pass, cutting per-layer KV storage.

What would settle it

Run the same cross-attention workload at two query lengths, e.g., $S_Q = 1024$ and $S_Q = 2048$ with $S_{KV}$ fixed at $1{,}200{,}000$ on 16 A100s: the paper's runtime model predicts LV-XAttn's cross-attention time should nearly double while Ring Attention's stays flat. If LV-XAttn's time stays flat, or Ring Attention's time doubles, the communication is not fully hidden and the core mechanism fails; equivalently, on a fast interconnect where Ring Attention also becomes compute-bound, the two methods should show identical attention times.

Watch

Extended reading notes

Core claim

The paper's central claim is that exact, sequence-parallel cross-attention can be made communication-free in the long-visual-input regime by reversing which tensors travel. In Ring Attention, every worker holds a block of queries and must see all key-value blocks, so large KV blocks circulate and the operation is communication-bound: cross-attention can consume up to 88% of an iteration even though it is about 3% of the parameters. LV-XAttn observes that in MLLM long-video workloads the query sequence is tiny relative to the key-value sequence (for Llama 3-V on Video-MME, $S_Q = 5{,}514$ versus $S_{KV} = 15{,}279{,}944$), so it pins each worker's key-value block in place and rotates the small query, output, and softmax-statistics blocks around the ring, cutting communication volume to 0.04% of Ring Attention's. With computation-communication overlap, the transfers are fully hidden and attention becomes compute-bound; the paper derives a closed-form forward-pass speedup of $\frac{2 S_Q}{n}\cdot\frac{\mathrm{GPU\,FLOPS}}{\mathrm{Net\,BW}}$ over Ring Attention and verifies it across three model families and two cluster configurations.

Load-bearing premise

The speedup rests on the per-round query/output/softmax transfers being fully hidden under the local attention computation, which holds only when the attention FLOPs take longer than the communication, a hardware- and workload-dependent condition rather than a mathematical guarantee.

Editorial extensions

If this is right

  • Distributed cross-attention on long visual inputs can run at compute-bound speed while remaining exact: LV-XAttn's output matches the standard scaled dot-product attention implementation exactly.
  • The speedup is largest when the per-worker query block is small and the interconnect is slow relative to GPU FLOPs, the same conditions that make Ring Attention slowest, so the method bears directly on long-video training and inference.
  • For self-attention, where query and key-value lengths are equal, Ring Attention remains the better choice; the paper therefore applies Ring Attention to the LM blocks and reserves LV-XAttn for cross-attention layers.
  • The activation-recomputation scheme raises the maximum visual input length by roughly 1.6x at under 8% iteration-time overhead by storing a single shared copy of the visual tokens rather than per-layer key-value activations.
  • The scheme can be applied to hybrid concatenation-plus-cross-attention architectures, extending the communication savings to models that mix both designs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A per-layer scheduler could decide between LV-XAttn and Ring Attention by plugging the per-worker query size, key-value size, GPU FLOPs, and network bandwidth into the paper's runtime expressions; the paper does not build such a selector, but its own analysis supplies the decision rule.
  • The same keep-large-local/rotate-small pattern should transfer to concatenation-based MLLMs during prefill, where visual tokens form a long fixed prefix and the text side is short; the paper does not test this, but the runtime model predicts the same communication collapse whenever $S_Q \ll S_{KV}$.
  • Because the speedup formula is linear in the per-worker query size, batching several text prompts into one step would enlarge the query block and shrink the advantage; the reported numbers are for batch size 1, which is the standard setting for long-video workloads.
  • The claim that communication is fully hidden could be stress-tested by timing each worker's block computation in isolation on the same GPUs, without any ring transfers, and comparing against the distributed LV-XAttn runtime.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes LV-XAttn, a distributed exact cross-attention mechanism for multimodal LLMs with large visual inputs. The key observation is that in long-video workloads the query sequence is much shorter than the key/value sequence, so LV-XAttn keeps the large KV blocks local to each worker and rotates smaller query, output, and softmax-statistics blocks around a ring, in contrast to Ring Attention, which rotates the large KV blocks. The paper gives a parameter-free FLOP/bandwidth runtime model, an activation-recomputation technique that exploits the fact that visual features are shared across cross-attention layers, and an implementation in PyTorch/Triton. Evaluations on Llama 3-V, mPLUG-Owl3, and OpenFlamingo across A100 and A30 clusters report up to 45.85x cross-attention speedup and up to 10.62x end-to-end iteration speedup over Ring Attention, with less than 0.42% overhead relative to a no-communication baseline.

Significance. If the results hold, LV-XAttn addresses a real and increasingly important bottleneck: cross-attention with long visual inputs is memory-prohibitive on a single GPU and communication-heavy under existing sequence-parallel schemes. The algorithmic insight is simple and likely reusable, the theoretical model in Table 1 and Figure 4 is parameter-free and matches the qualitative scaling behavior in the benchmarks, the implementation is released, and the evaluation covers multiple models and cluster configurations. The main weakness is that correctness is validated only for the forward pass, while the headline per-iteration training speedups include the backward pass; the backward algorithm for the distributed online-softmax rescaling is not specified and no gradient check is reported. This is a missing-support issue that is fixable within the scope of the manuscript, and it should be resolved before the training-speedup claims are accepted.

major comments (2)
  1. [§4.1, Algorithm 1, Tables 3–4] The correctness validation in §4.1 only checks the forward output against PyTorch's scaled dot-product attention, while the headline end-to-end iteration speedups in Tables 3–4 include the backward pass, as Figure 2's 'BWD CA' bars and Table 1's backward row make explicit. The distributed backward pass for the online-softmax rescaling in Algorithm 1 is not specified: it is not described how partial dQ contributions from each KV-owning worker are accumulated across the ring, nor how the per-row softmax statistics L are handled under global rescaling in the backward direction. No gradient check is reported. Because the paper's stated goal covers training as well as inference, the iteration-time numbers currently measure an unvalidated training loop. I ask the authors to (i) specify the backward algorithm or point to the relevant code path, and (ii) report a numerical gradient check comparing LV-XAttn's gradients with a reference implementation (e.g., PyTorch autograd of the exact attention formula) for at least one model and one distributed configuration, reporting the maximum relative error.
  2. [Eq. (1), Table 1, Appendix A] The speedup analysis in Eq. (1) and Table 1 assumes that LV-XAttn's per-round runtime is the max of a compute time and a communication time, i.e., that the small query/output/softmax transfer is fully hidden by local attention FLOPs. The paper acknowledges in Appendix A and Figure 7 that this condition fails when SKV is not much larger than SQ or when the interconnect is fast relative to GPU FLOPs; for example, OpenFlamingo-9b reaches only 1.04x end-to-end on the A30 cluster (Table 4). The abstract's claim of speedups for a 'wide range of models' is therefore conditional on a hardware- and workload-dependent regime. I recommend that the paper state this condition as a formal caveat in the abstract or introduction and report, for each cluster, the measured overlap efficiency (e.g., the fraction of LV-XAttn cross-attention time spent in communication) so that readers can see where the top-left-quadrant assumption holds.
minor comments (5)
  1. [§4.1 and Figure 5] The reported runtimes are averages over 5 trials with no variance or standard deviation; given that some speedup numbers are close to 1 (e.g., OpenFlamingo-9b at 1.04x), reporting the spread across trials would strengthen the comparison.
  2. [Figure 5] The 'no communication' baseline is derived by scaling a single-GPU run by the number of workers; this assumes perfect linear scaling and no inter-worker synchronization overhead. Please state this assumption explicitly and, if possible, report a stronger baseline such as an idealized compute-only lower bound.
  3. [Table 1] The speedup row in Table 1 is difficult to parse: the fraction as typeset appears to give the inverse of the ratio one obtains from the two runtime rows above it. Please rewrite the speedup as a single unambiguous expression, and clarify whether d is the total hidden dimension or the per-head dimension (Eq. (1) uses d as the total hidden dimension, while the h·d terms in Table 1 suggest d is per-head).
  4. [§4.3 and Table 6] The memory comparison against DeepSpeed-Ulysses does not appear to apply the proposed activation-recomputation technique to the DeepSpeed-Ulysses baseline; the text says the comparison is made 'without activation recomputation.' This makes the 'more than 4x longer inputs' claim somewhat uneven, and the paper should state this explicitly in the table discussion.
  5. [Typos] There are several minor text errors: 'wall-lock' should be 'wall-clock' in the captions of Tables 5 and 6, 'xOpenFlamingo' appears in §4.1, and 'LLama-3V' appears in the Figure 4 caption.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the claimed speedups are measured against external baselines and the theoretical runtime analysis is a parameter-free model whose inputs are hardware specifications and problem sizes.

full rationale

The paper's central claims are empirical runtime speedups of LV-XAttn over Ring Attention and DeepSpeed-Ulysses on real cluster configurations, with wall-clock measurements reported in Tables 3-6. These comparisons are against external distributed-attention baselines, not against quantities derived from LV-XAttn itself. The theoretical analysis in Section 3.1 and Table 1 is a first-principles FLOP and bandwidth model: it takes hardware parameters (GPU FLOPS, network bandwidth) and problem sizes (SQ, SKV, n, h, d) as inputs, computes the expected communication and computation times for Ring Attention and LV-XAttn, and derives a speedup ratio. None of these parameters are fitted to the reported speedups, and the model is separately checked against measurements in Figure 5 and Figure 7. The correctness claim for the forward pass is checked against PyTorch's scaled dot-product attention implementation, which is an external reference. No load-bearing step invokes a self-citation or an unverified uniqueness theorem. The activation recomputation technique is an observation about MLLM architecture plus a measured memory/runtime tradeoff, not a circular construction. The main weakness identified by the skeptic is that backward-pass gradients are not explicitly validated, and the distributed backward protocol is underspecified; that is a missing-support or correctness-risk concern, not a circularity, because it does not make any prediction equivalent to an input or fit. Overall, the paper's derivation chain is self-contained against external baselines and hardware specs, so no significant circularity is present.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim depends on no fitted free parameters and introduces no invented entities. The speedup model uses hardware constants (GPU FLOPS, network bandwidth) and benchmark-chosen sequence lengths, none of which are tuned to make the reported speedups hold. The axioms are workload and architecture assumptions, all explicitly stated or standard.

assumptions (4)
  • domain assumption Query blocks are much smaller than key-value blocks in target MLLM workloads (SQ << SKV).
    Used to justify keeping KV local and rotating Q. Supported by Video-MME statistics in Section 3.1 (average SQ = 5,514 vs SKV = 15,279,944 for Llama 3-V). Appendix A shows the method loses this advantage as SQ approaches SKV.
  • domain assumption Communication of small query, output, and softmax blocks is fully hidden by local attention computation.
    Equation 1's max() resolves to the compute term for the target regime; Figure 5 empirically supports this on 6 A100s for OpenFlamingo-3b, but the condition is hardware- and workload-dependent. If FLOPs per byte is small, the communication term dominates.
  • domain assumption Visual features are identical across all cross-attention layers in the evaluated MLLMs.
    Section 3.2 states that visual features y are fed unchanged into each cross-attention layer, so one shared copy can be used to recompute K and V in the backward pass. This holds for Flamingo-style architectures but not for models that update visual features between layers.
  • standard math FlashAttention's online softmax rescaling computes exact attention.
    Algorithm 1 relies on the rescaling of partial O and L statistics; exactness is established in Dao et al. (2022) and the paper validates against PyTorch SDPA in Section 4.1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LV-XAttn: Distributed Cross-Attention for Long Visual Inputs in Multimodal Large Language Models." pith.science (2026). https://pith.science/paper/4RYX6ACM

@misc{pith2026250202406,
  author       = {Pith},
  title        = {Pith review of: LV-XAttn: Distributed Cross-Attention for Long Visual Inputs in Multimodal Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4RYX6ACM}},
  note         = {Machine review of arXiv:2502.02406}
}
abstract

Cross-attention is commonly adopted in multimodal large language models (MLLMs) for integrating visual information into the language backbone. However, in applications with large visual inputs, such as video understanding, processing a large number of visual tokens in cross-attention layers leads to high memory demands and often necessitates distributed computation across multiple GPUs. Existing distributed attention mechanisms face significant communication overheads, making cross-attention layers a critical bottleneck for efficient training and inference of MLLMs. To address this, we propose LV-XAttn, a distributed, exact cross-attention mechanism with minimal communication overhead. We observe that in applications involving large visual inputs, the size of the query block is typically much smaller than that of the key-value blocks. Thus, in LV-XAttn we keep the large key-value blocks locally on each GPU and exchange smaller query blocks across GPUs. We also introduce an efficient activation recomputation technique to support longer visual context. We theoretically analyze the communication benefits of LV-XAttn and show that it can achieve speedups for a wide range of models. Our evaluations with Llama 3-V, mPLUG-Owl3 and OpenFlamingo models find that LV-XAttn achieves up to 10.62$\times$ end-to-end speedup compared to existing approaches.

Figures

Figures reproduced from arXiv: 2502.02406 by the authors.

Figure 1
Figure 1. MLLM with cross-attention. incurs less than 0.42% overhead compared to the theoretical no-communication baseline. LV-XAttn is available at https://github.com/uw-mad-dash/LV-XAttn. 2. Background Cross-attention Cross-attention (Vaswani et al., 2017) is a variant of self-attention to model interactions between different sequences. The input to cross-attention consists of two sequences x ∈ R SQ×dembed and y ∈ R SKV ×de… view at source ↗
Figure 2
Figure 2. Runtime breakdown for a single iteration of Llama 3-V, mPLUG-Owl3-7b, and OpenFlamingo-3b using Ring Attention and LV-XAttn on 16 A100 GPUs. LV-XAttn reduces the time spent on cross-attention computation by 96%, 93%, and 53% for the three models, respectively, compared to Ring Attention. “FWD Vision” refers to the forward pass through the vision encoder and the projection layer; “FWD CA” and “BWD CA” refer to the fo… view at source ↗
Figure 3
Figure 3. LV-XAttn with 4 workers. We partition the KV blocks and each worker stores their respective large key-value blocks Ki, Vi. We also partition the query (Qi), output (Oi), and softmax statistics (mi and li omitted in the figure). The query and output are rotated among workers to compute the attention. Algorithm 1 LV-XAttn Forward Pass for Worker i Input: data Qi , Ki , Vi Initialize Oi , Li ← 0 for round = 0 to n − 1 … view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The theoretical speedup of LV-XAttn over Ring Attention for cross-attention on a 4-node cluster. Each node is equipped with 4 A100 GPUs, and nodes are interconnected by a 25 GB/s network. The markers represent processing a 2,386-second video and a 3,128-word text promp…
Figure 5
Figure 5. Figure 5: Ablation study on the effect of overlapping communi￾cation and computation with 6 A100 40GB GPUs. The frame count is set to 2048 per worker. Since processing the same total number of frames on a single GPU is not feasible due to memory constraints, the “no communicatio…
Figure 6
Figure 6. Figure 6: Ablation study on the effect of activation recomputation for cross-attention layers with 3 A30 24GB GPUs. Text length is set to 2K and 8K for mPLUG-Owl-7b and OpenFlamingo-3b, respectively. note that LV-XAttn can also be applied to such architecture to address communic…
Figure 7
Figure 7. Figure 7: The theoretical speedup of LV-XAttn over Ring Attention for on a cluster with 4 nodes, each equipped with 4 A100 GPUs. The uncolored region indicates a speedup of less than 1, meaning LV-XAttn performs slower than Ring Attention. Top-left and bottom-left quadrant repre…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 20 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Flamingo: a visual language model for few-shot learning

    Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J., Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., Binkowski, M., Barreira, R., Vinyals, O., Zisserman, A., and Simonyan, K. Flaming...

    work page 2022

  3. [3]

    W., Ilharco, G., Wortsman, M., and Schmidt, L

    Awadalla, A., Gao, I., Gardner, J., Hessel, J., Hanafy, Y., Zhu, W., Marathe, K., Bitton, Y., Gadre, S., Sagawa, S., Jitsev, J., Kornblith, S., Koh, P. W., Ilharco, G., Wortsman, M., and Schmidt, L. Openflamingo: An open-source framework for training large autoregressive vision-language models, 2023. URL https://arxiv.org/abs/2308.01390

    arXiv 2023

  4. [4]

    E., and Cohan, A

    Beltagy, I., Peters, M. E., and Cohan, A. Longformer: The long-document transformer. arXiv:2004.05150, 2020

    arXiv 2004

  5. [5]

    Bertsch, A., Alon, U., Neubig, G., and Gormley, M. R. Unlimiformer: Long-range transformers with unlimited length input. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=lJWUJWLCJo

    work page 2023

  6. [6]

    Striped attention: Faster ring attention for causal transformers

    Brandon, W., Nrusimha, A., Qian, K., Ankner, Z., Jin, T., Song, Z., and Ragan-Kelley, J. Striped attention: Faster ring attention for causal transformers. arXiv preprint arXiv:2311.09431, 2023

    arXiv 2023

  7. [7]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 24185--24198, 2024

    2024

  8. [8]

    Adapting language models to compress contexts

    Chevalier, A., Wettig, A., Ajith, A., and Chen, D. Adapting language models to compress contexts. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https://openreview.net/forum?id=kp1U6wBPXq

    work page 2023

Show all 45 references
  1. [9]

    M., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J

    Choromanski, K. M., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J. Q., Mohiuddin, A., Kaiser, L., Belanger, D. B., Colwell, L. J., and Weller, A. Rethinking attention with performers. In International Conference on Learning Representations...

    2021

  2. [10]

    Nvlm: Open frontier-class multimodal llms

    Dai, W., Lee, N., Wang, B., Yang, Z., Liu, Z., Barker, J., Rintamaki, T., Shoeybi, M., Catanzaro, B., and Ping, W. Nvlm: Open frontier-class multimodal llms. arXiv preprint, 2024

    2024

  3. [11]

    Flash A ttention-2: Faster attention with better parallelism and work partitioning

    Dao, T. Flash A ttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR), 2024

    2024

  4. [12]

    Y., Ermon, S., Rudra, A., and R \'e , C

    Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and R \'e , C. Flash A ttention: Fast and memory-efficient exact attention with IO -awareness. In Advances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  5. [13]

    S., Monga, R., Chen, K., Devin, M., Le, Q

    Dean, J., Corrado, G. S., Monga, R., Chen, K., Devin, M., Le, Q. V., Mao, M. Z., Ranzato, M., Senior, A., Tucker, P., Yang, K., and Ng, A. Y. Large scale distributed deep networks. In Proceedings of the 26th International Conference on Neural Information Processing Systems - V...

    2012

  6. [14]

    Longnet: Scaling transformers to 1,000,000,000 tokens, 2023

    Ding, J., Ma, S., Dong, L., Zhang, X., Huang, S., Wang, W., Zheng, N., and Wei, F. Longnet: Scaling transformers to 1,000,000,000 tokens, 2023. URL https://arxiv.org/abs/2307.02486

    2023 arXiv

  7. [15]

    The design and operation of CloudLab

    Duplyakin, D., Ricci, R., Maricq, A., Wong, G., Duerig, J., Eide, E., Stoller, L., Hibler, M., Johnson, D., Webb, K., Akella, A., Wang, K., Ricart, G., Landweber, L., Elliott, C., Zink, M., Cecchet, E., Kar, S., and Mishra, P. The design and operation of CloudLab . In Proceedi...

    2019

  8. [16]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Fu, C., Dai, Y., Luo, Y., Li, L., Ren, S., Zhang, R., Wang, Z., Zhou, C., Shen, Y., Zhang, M., et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024

    2024 arXiv

  9. [17]

    Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., Yang, A., and et al, A. F. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783

    2024 arXiv

  10. [18]

    Llava-uhd: An lmm perceiving any aspect ratio and high-resolution images

    Guo, Z., Xu, R., Yao, Y., Cui, J., Ni, Z., Ge, C., Chua, T.-S., Liu, Z., and Huang, G. Llava-uhd: An lmm perceiving any aspect ratio and high-resolution images. In Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceedings, P...

    2024 doi

  11. [19]

    K., Jia, M., Cao, X., Shah, A., Shrivastava, A., and Lim, S.-N

    He, B., Li, H., Jang, Y. K., Jia, M., Cao, X., Shah, A., Shrivastava, A., and Lim, S.-N. Ma-lmm: Memory-augmented large multimodal model for long-term video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  12. [20]

    M., Ho, N., Yang, X., Nagarajan, T., Torresani, L., and Bertasius, G

    Islam, M. M., Ho, N., Yang, X., Nagarajan, T., Torresani, L., and Bertasius, G. Video recap: Recursive captioning of hour-long videos. arXiv preprint arXiv:2402.13250, 2024

    2024 arXiv

  13. [21]

    A., Tanaka, M., Zhang, C., Zhang, M., Aminadabi, R

    Jacobs, S. A., Tanaka, M., Zhang, C., Zhang, M., Aminadabi, R. Y., Song, S. L., Rajbhandari, S., and He, Y. System optimizations for enabling training of extreme long sequence transformer models. In Proceedings of the 43rd ACM Symposium on Principles of Distributed Computing, ...

    2024

  14. [22]

    Reformer: The efficient transformer

    Kitaev, N., Kaiser, L., and Levskaya, A. Reformer: The efficient transformer. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=rkgNKkHtvB

    2020

  15. [23]

    A., Casper, J., Lym, S., McAfee, L., Andersch, M., Shoeybi, M., and Catanzaro, B

    Korthikanti, V. A., Casper, J., Lym, S., McAfee, L., Andersch, M., Shoeybi, M., and Catanzaro, B. Reducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5: 0 341--353, 2023 a

    2023

  16. [24]

    A., Casper, J., Lym, S., McAfee, L., Andersch, M., Shoeybi, M., and Catanzaro, B

    Korthikanti, V. A., Casper, J., Lym, S., McAfee, L., Andersch, M., Shoeybi, M., and Catanzaro, B. Reducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5: 0 341--353, 2023 b

    2023

  17. [25]

    M., Kiela, D., Cord, M., and Sanh, V

    Lauren c on, H., Saulnier, L., Tronchon, L., Bekman, S., Singh, A., Lozhkov, A., Wang, T., Karamcheti, S., Rush, A. M., Kiela, D., Cord, M., and Sanh, V. Obelics: an open web-scale filtered dataset of interleaved image-text documents. In Proceedings of the 37th International C...

    2024

  18. [26]

    Mimic-it: Multi-modal in-context instruction tuning

    Li, B., Zhang, Y., Chen, L., Wang, J., Pu, F., Yang, J., Li, C., and Liu, Z. Mimic-it: Multi-modal in-context instruction tuning. CoRR, abs/2306.05425, 2023 a . URL https://doi.org/10.48550/arXiv.2306.05425

    2023 doi arXiv

  19. [27]

    P., Ma, X., Stoica, I., Gonzalez, J

    Li, D., Shao, R., Xie, A., Xing, E. P., Ma, X., Stoica, I., Gonzalez, J. E., and Zhang, H. DISTFLASHATTN : Distributed memory-efficient attention for long-context LLM s training. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=pUEDkZyPDl

    2024

  20. [28]

    Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models

    Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023 b

    2023

  21. [29]

    Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning, 2023

    2023

  22. [30]

    Ringattention with blockwise transformers for near-infinite context

    Liu, H., Zaharia, M., and Abbeel, P. Ringattention with blockwise transformers for near-infinite context. In The Twelfth International Conference on Learning Representations, 2024 a . URL https://openreview.net/forum?id=WsRHpHH4s0

    2024

  23. [31]

    Nvila: Efficient frontier visual language models, 2024 b

    Liu, Z., Zhu, L., Shi, B., Zhang, Z., Lou, Y., Yang, S., Xi, H., Cao, S., Gu, Y., Li, D., Li, X., Fang, Y., Chen, Y., Hsieh, C.-Y., Huang, D.-A., Cheng, A.-C., Nath, V., Hu, J., Liu, S., Krishna, R., Xu, D., Wang, X., Molchanov, P., Kautz, J., Yin, H., Han, S., and Lu, Y. Nvil...

    2024 arXiv

  24. [32]

    Leave no context behind: Efficient infinite context transformers with infini-attention, 2024

    Munkhdalai, T., Faruqui, M., and Gopal, S. Leave no context behind: Efficient infinite context transformers with infini-attention, 2024. URL https://arxiv.org/abs/2404.07143

    2024 arXiv

  25. [33]

    R., Ganger, G

    Narayanan, D., Harlap, A., Phanishayee, A., Seshadri, V., Devanur, N. R., Ganger, G. R., Gibbons, P. B., and Zaharia, M. Pipedream: generalized pipeline parallelism for dnn training. In Proceedings of the 27th ACM Symposium on Operating Systems Principles, SOSP '19, pp.\ 1–15,...

    2019

  26. [34]

    Momentor: advancing video large language model with fine-grained temporal reasoning

    Qian, L., Li, J., Wu, Y., Ye, Y., Fei, H., Chua, T.-S., Zhuang, Y., and Tang, S. Momentor: advancing video large language model with fine-grained temporal reasoning. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024

    2024

  27. [35]

    ModServe : Scalable and resource-efficient large multimodal model serving, 2025

    Qiu, H., Biswas, A., Zhao, Z., Mohan, J., Khare, A., Choukse, E., Íñigo Goiri, Zhang, Z., Shen, H., Bansal, C., Ramjee, R., and Fonseca, R. ModServe : Scalable and resource-efficient large multimodal model serving, 2025. URL https://arxiv.org/abs/2502.00937

    2025

  28. [36]

    Zero: memory optimizations toward training trillion parameter models

    Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. Zero: memory optimizations toward training trillion parameter models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC '20. IEEE Press, 2020. ISBN 9781728199986

    2020

  29. [37]

    Megatron-lm: Training multi-billion parameter language models using model parallelism

    Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019

    1909 arXiv

  30. [38]

    Repository-level prompt generation for large language models of code

    Shrivastava, D., Larochelle, H., and Tarlow, D. Repository-level prompt generation for large language models of code. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Machine Learnin...

    2023

  31. [39]

    PEARL : Prompting large language models to plan and execute actions over long documents

    Sun, S., Liu, Y., Wang, S., Iter, D., Zhu, C., and Iyyer, M. PEARL : Prompting large language models to plan and execute actions over long documents. In Graham, Y. and Purver, M. (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computat...

    2024

  32. [40]

    T., and Cox, D

    Tillet, P., Kung, H. T., and Cox, D. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, pp.\ 10–19, New York, NY, USA, 2019....

    2019

  33. [41]

    N., Kaiser, L., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, pp.\ 6000–6010, Red Hook, NY, USA, 201...

    2017

  34. [42]

    mplug-owl3: Towards long image-sequence understanding in multi-modal large language models, 2024

    Ye, J., Xu, H., Liu, H., Hu, A., Yan, M., Qian, Q., Zhang, J., Huang, F., and Zhou, J. mplug-owl3: Towards long image-sequence understanding in multi-modal large language models, 2024. URL https://arxiv.org/abs/2408.04840

    2024 arXiv

  35. [43]

    Big bird: transformers for longer sequences

    Zaheer, M., Guruganesh, G., Dubey, A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., and Ahmed, A. Big bird: transformers for longer sequences. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS...

    2020

  36. [44]

    R epo C oder: Repository-level code completion through iterative retrieval and generation

    Zhang, F., Chen, B., Zhang, Y., Keung, J., Liu, J., Zan, D., Mao, Y., Lou, J.-G., and Chen, W. R epo C oder: Repository-level code completion through iterative retrieval and generation. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empiri...

    2023 doi

  37. [45]

    Mini GPT -4: Enhancing vision-language understanding with advanced large language models

    Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Mini GPT -4: Enhancing vision-language understanding with advanced large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=1tZbq88f27

    2024

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.