Pith Number

pith:7ZKTROEY

pith:2024:7ZKTROEYPUT3OHQRT22M4P2O2E

not attested not anchored not stored refs resolved

World Model on Million-Length Video And Language With Blockwise RingAttention

Hao Liu, Matei Zaharia, Pieter Abbeel, Wilson Yan

7B parameter models process video and language sequences exceeding 1 million tokens.

arxiv:2402.08268 v4 · 2024-02-13 · cs.LG

Open paper page JSON Open Graph Bundle Merged state Verified badge What is a Pith Number?

Add to your LaTeX paper

\usepackage{pith}
\pithnumber{7ZKTROEYPUT3OHQRT22M4P2O2E}

Prints a linked badge after your title and injects PDF metadata. Compiles on arXiv. Learn more · Embed verified badge

Record completeness

1 Bitcoin timestamp

2 Internet Archive

3 Author claim open · sign in to claim

4 Citations open

5 Replications open

✓ Portable graph bundle live · download bundle · merged state

The bundle contains the canonical record plus signed events. A mirror can host it anywhere and recompute the same current state with the deterministic merge algorithm.

Claims

C1strongest claim

We open-source a family of 7B parameter models capable of processing long text documents and videos exceeding 1M tokens, setting new benchmarks in language retrieval and new capabilities in long video understanding.

C2weakest assumption

That the Blockwise RingAttention mechanism combined with progressive context extension from 4K to 1M tokens enables effective utilization of the full context length without prohibitive computational costs or performance loss.

C3one line summary

Presents open-source 7B models for million-token video and language understanding via Blockwise RingAttention, setting new benchmarks in retrieval and long video tasks.

References

36 extracted · 36 resolved · 23 Pith anchors

[1] Jointly training large autoregressive multimodal models

[2] OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models · arXiv:2308.01390

[3] Longformer: The Long-Document Transformer 2004 · arXiv:2004.05150

[4] Striped attention: Faster ring attention for causal transformers

[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al 1901

Formal links

3 machine-checked theorem links

Cited by

29 papers in Pith

LLaVA-Octopus: Unlocking Instruction-Driven Adaptive Projector Fusion for Video Understanding

Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces

Lance: Unified Multimodal Modeling by Multi-Task Synergy

FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision

WinTok: A Win-Win Hybrid Tokenizer via Decomposing Visual Understanding and Generation with Transferable Tokens

Receipt and verification

First computed	2026-05-17T23:38:48.813238Z
Builder	pith-number-builder-2026-05-17-v1
Signature	Pith Ed25519 (`pith-v1-2026-05`) · public key
Schema	pith-number/v1.0

Canonical hash

fe5538b8987d27b71e119eb4ce3f4ed1270d9ee4b9302811473a31b6c09ebabf

Aliases

arxiv: 2402.08268 · arxiv_version: 2402.08268v4 · doi: 10.48550/arxiv.2402.08268 · pith_short_12: 7ZKTROEYPUT3 · pith_short_16: 7ZKTROEYPUT3OHQR · pith_short_8: 7ZKTROEY

Agent API

Resolver JSON Graph JSON Events JSON Schema Signing key

Verify this Pith Number yourself

curl -sH 'Accept: application/ld+json' https://pith.science/pith/7ZKTROEYPUT3OHQRT22M4P2O2E \
  | jq -c '.canonical_record' \
  | python3 -c "import sys,json,hashlib; b=json.dumps(json.loads(sys.stdin.read()), sort_keys=True, separators=(',',':'), ensure_ascii=False).encode(); print(hashlib.sha256(b).hexdigest())"
# expect: fe5538b8987d27b71e119eb4ce3f4ed1270d9ee4b9302811473a31b6c09ebabf

Canonical record JSON

{
  "metadata": {
    "abstract_canon_sha256": "dca7c76a40ebd7467d053db24b9fac5090178337ce8421c47f2ee989a5b189d7",
    "cross_cats_sorted": [],
    "license": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
    "primary_cat": "cs.LG",
    "submitted_at": "2024-02-13T07:47:36Z",
    "title_canon_sha256": "a17f2d3c39ca5c2a6aec7c5170eff5fd2cf874427dfa86450ed3bb835872d23b"
  },
  "schema_version": "1.0",
  "source": {
    "id": "2402.08268",
    "kind": "arxiv",
    "version": 4
  }
}