{"paper":{"title":"World Model on Million-Length Video And Language With Blockwise RingAttention","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"7B parameter models process video and language sequences exceeding 1 million tokens.","cross_cats":[],"primary_cat":"cs.LG","authors_text":"Hao Liu, Matei Zaharia, Pieter Abbeel, Wilson Yan","submitted_at":"2024-02-13T07:47:36Z","abstract_excerpt":"Enabling long-context understanding remains a key challenge in scaling existing sequence models -- a crucial component in developing generally intelligent models that can process and operate over long temporal horizons that potentially consist of millions of tokens. In this paper, we aim to address these challenges by providing a comprehensive exploration of the full development process for producing 1M context language models and video-language models, setting new benchmarks in language retrieval and new capabilities in long video understanding. We detail our long context data curation proces"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"We open-source a family of 7B parameter models capable of processing long text documents and videos exceeding 1M tokens, setting new benchmarks in language retrieval and new capabilities in long video understanding.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"That the Blockwise RingAttention mechanism combined with progressive context extension from 4K to 1M tokens enables effective utilization of the full context length without prohibitive computational costs or performance loss.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"Presents open-source 7B models for million-token video and language understanding via Blockwise RingAttention, setting new benchmarks in retrieval and long video tasks.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"7B parameter models process video and language sequences exceeding 1 million tokens.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"aeecb379bdff3ba1a5bb08a27b1c3b63f69bae0ac603dd068a30ecf1c010ac14"},"source":{"id":"2402.08268","kind":"arxiv","version":4},"verdict":{"id":"6374161b-ebcd-483d-98e6-c6c767d8879d","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-16T06:32:36.859746Z","strongest_claim":"We open-source a family of 7B parameter models capable of processing long text documents and videos exceeding 1M tokens, setting new benchmarks in language retrieval and new capabilities in long video understanding.","one_line_summary":"Presents open-source 7B models for million-token video and language understanding via Blockwise RingAttention, setting new benchmarks in retrieval and long video tasks.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"That the Blockwise RingAttention mechanism combined with progressive context extension from 4K to 1M tokens enables effective utilization of the full context length without prohibitive computational costs or performance loss.","pith_extraction_headline":"7B parameter models process video and language sequences exceeding 1 million tokens."},"references":{"count":36,"sample":[{"doi":"","year":null,"title":"Jointly training large autoregressive multimodal models","work_id":"d6fca016-5cb9-4bbe-8e83-c1333222097e","ref_index":1,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":null,"title":"OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models","work_id":"87bfa84a-e663-4165-806f-93ef439d88d0","ref_index":2,"cited_arxiv_id":"2308.01390","is_internal_anchor":true},{"doi":"","year":2004,"title":"Longformer: The Long-Document Transformer","work_id":"abea7a44-6668-4de7-aab6-f53a6e5aa088","ref_index":3,"cited_arxiv_id":"2004.05150","is_internal_anchor":true},{"doi":"","year":null,"title":"Striped attention: Faster ring attention for causal transformers","work_id":"7cc4e141-1418-4a62-bc6a-4e3270436eec","ref_index":4,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":1901,"title":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al","work_id":"0cfb0982-a2c1-413b-87eb-f784fefa6518","ref_index":5,"cited_arxiv_id":"","is_internal_anchor":false}],"resolved_work":36,"snapshot_sha256":"6704dbb0e5dc8e38c64853e13334e7ffa68523f39882096af3ae1bb67c779095","internal_anchors":23},"formal_canon":{"evidence_count":3,"snapshot_sha256":"73287facd96eb55d6898c0a4d98d798e125534b6d1fff6125b0d2c85f006dedf"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"}