{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2023:XCXLFWSANHUSBZ2RU7WQZVMB2J","short_pith_number":"pith:XCXLFWSA","schema_version":"1.0","canonical_sha256":"b8aeb2da4069e920e751a7ed0cd581d277d482abcbec9b58570126d1319f18bf","source":{"kind":"arxiv","id":"2312.14125","version":4},"attestation_state":"computed","paper":{"title":"VideoPoet: A Large Language Model for Zero-Shot Video Generation","license":"http://creativecommons.org/licenses/by/4.0/","headline":"A decoder-only transformer trained on multimodal data generates high-fidelity videos zero-shot from text, images, and audio.","cross_cats":["cs.AI"],"primary_cat":"cs.CV","authors_text":"Agrim Gupta, Alonso Martinez, Anja Hauth, Bryan Seybold, Dan Kondratyuk, David A. Ross, David Hendon, David Minnen, Grant Schindler, Hartwig Adam, Hassan Akbari, Huisheng Wang, Irfan Essa, Jimmy Yan, Jonathan Huang, Jos\\'e Lezama, Josh Dillon, Kihyuk Sohn, Krishna Somandepalli, Lijun Yu, Lu Jiang, Meera Hahn, Mikhail Sirotenko, Ming-Chang Chiu, Ming-Hsuan Yang, Rachel Hornung, Vighnesh Birodkar, Xiuye Gu, Xuan Yang, Yair Alon, Yong Cheng","submitted_at":"2023-12-21T18:46:41Z","abstract_excerpt":"We present VideoPoet, a language model capable of synthesizing high-quality video, with matching audio, from a large variety of conditioning signals. VideoPoet employs a decoder-only transformer architecture that processes multimodal inputs -- including images, videos, text, and audio. The training protocol follows that of Large Language Models (LLMs), consisting of two stages: pretraining and task-specific adaptation. During pretraining, VideoPoet incorporates a mixture of multimodal generative objectives within an autoregressive Transformer framework. The pretrained LLM serves as a foundatio"},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":true,"formal_links_present":false},"canonical_record":{"source":{"id":"2312.14125","kind":"arxiv","version":4},"metadata":{"license":"http://creativecommons.org/licenses/by/4.0/","primary_cat":"cs.CV","submitted_at":"2023-12-21T18:46:41Z","cross_cats_sorted":["cs.AI"],"title_canon_sha256":"18b0d90fb9392b704e8d07576d72bd1dfbee6398ddf50fe6abad23281ad20228","abstract_canon_sha256":"544d649ec5a1c6c28686e7b7a0beb204295ee44d4bde2e405b1197873c44127f"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-05-17T23:38:50.672877Z","signature_b64":"g9cBJSGJITYhXXePuLJUBMLYOw2tn9gA2U3F+Uo32KkWWufZFKtYsnN/1u4nE5Z7RJVajM7kF+EARteh4VmCCQ==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"b8aeb2da4069e920e751a7ed0cd581d277d482abcbec9b58570126d1319f18bf","last_reissued_at":"2026-05-17T23:38:50.672005Z","signature_status":"signed_v1","first_computed_at":"2026-05-17T23:38:50.672005Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"VideoPoet: A Large Language Model for Zero-Shot Video Generation","license":"http://creativecommons.org/licenses/by/4.0/","headline":"A decoder-only transformer trained on multimodal data generates high-fidelity videos zero-shot from text, images, and audio.","cross_cats":["cs.AI"],"primary_cat":"cs.CV","authors_text":"Agrim Gupta, Alonso Martinez, Anja Hauth, Bryan Seybold, Dan Kondratyuk, David A. Ross, David Hendon, David Minnen, Grant Schindler, Hartwig Adam, Hassan Akbari, Huisheng Wang, Irfan Essa, Jimmy Yan, Jonathan Huang, Jos\\'e Lezama, Josh Dillon, Kihyuk Sohn, Krishna Somandepalli, Lijun Yu, Lu Jiang, Meera Hahn, Mikhail Sirotenko, Ming-Chang Chiu, Ming-Hsuan Yang, Rachel Hornung, Vighnesh Birodkar, Xiuye Gu, Xuan Yang, Yair Alon, Yong Cheng","submitted_at":"2023-12-21T18:46:41Z","abstract_excerpt":"We present VideoPoet, a language model capable of synthesizing high-quality video, with matching audio, from a large variety of conditioning signals. VideoPoet employs a decoder-only transformer architecture that processes multimodal inputs -- including images, videos, text, and audio. The training protocol follows that of Large Language Models (LLMs), consisting of two stages: pretraining and task-specific adaptation. During pretraining, VideoPoet incorporates a mixture of multimodal generative objectives within an autoregressive Transformer framework. The pretrained LLM serves as a foundatio"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"VideoPoet employs a decoder-only transformer architecture that processes multimodal inputs -- including images, videos, text, and audio -- and demonstrates state-of-the-art capabilities in zero-shot video generation, specifically highlighting VideoPoet's ability to generate high-fidelity motions.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"The assumption that a mixture of multimodal generative objectives trained autoregressively on a decoder-only transformer will transfer effectively to zero-shot video generation without requiring substantial additional video-specific inductive biases or post-training.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"VideoPoet is a large language model that performs zero-shot video generation with audio from diverse multimodal conditioning signals.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"A decoder-only transformer trained on multimodal data generates high-fidelity videos zero-shot from text, images, and audio.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"a15f98adaa3ec8f737e33d7350aec97e9be28a7d090d4bf25983cb7d7e7f9f13"},"source":{"id":"2312.14125","kind":"arxiv","version":4},"verdict":{"id":"0470c117-63d1-4cce-8ae5-5f929abddc25","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-15T17:46:28.066693Z","strongest_claim":"VideoPoet employs a decoder-only transformer architecture that processes multimodal inputs -- including images, videos, text, and audio -- and demonstrates state-of-the-art capabilities in zero-shot video generation, specifically highlighting VideoPoet's ability to generate high-fidelity motions.","one_line_summary":"VideoPoet is a large language model that performs zero-shot video generation with audio from diverse multimodal conditioning signals.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"The assumption that a mixture of multimodal generative objectives trained autoregressively on a decoder-only transformer will transfer effectively to zero-shot video generation without requiring substantial additional video-specific inductive biases or post-training.","pith_extraction_headline":"A decoder-only transformer trained on multimodal data generates high-fidelity videos zero-shot from text, images, and audio."},"references":{"count":55,"sample":[{"doi":"","year":null,"title":"MusicLM: Generating Music From Text","work_id":"15e6566e-1c36-468f-966e-823248cbf87f","ref_index":1,"cited_arxiv_id":"2301.11325","is_internal_anchor":true},{"doi":"","year":null,"title":"Alternating gradient descent and mixture-of- experts for integrated multimodal perception","work_id":"c82d289d-9341-4a52-8917-f972c5d5da2a","ref_index":2,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":null,"title":"PaLM 2 Technical Report","work_id":"905ee9a7-ea61-4a94-bd62-2600cbe3e315","ref_index":3,"cited_arxiv_id":"2305.10403","is_internal_anchor":true},{"doi":"","year":null,"title":"Lumiere: A space-time diffusion model for video generation","work_id":"8a0a0735-d82c-4090-a039-697d06ccc3f0","ref_index":4,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":null,"title":"Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets","work_id":"4f68eada-27e3-437a-a2fe-6e4ca524d0d3","ref_index":5,"cited_arxiv_id":"2311.15127","is_internal_anchor":true}],"resolved_work":55,"snapshot_sha256":"7da2eb9360b22bfbeb58e0bddb321f751c71359f68ccd8c56f9b73111db36026","internal_anchors":29},"formal_canon":{"evidence_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2312.14125","created_at":"2026-05-17T23:38:50.672125+00:00"},{"alias_kind":"arxiv_version","alias_value":"2312.14125v4","created_at":"2026-05-17T23:38:50.672125+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2312.14125","created_at":"2026-05-17T23:38:50.672125+00:00"},{"alias_kind":"pith_short_12","alias_value":"XCXLFWSANHUS","created_at":"2026-05-18T12:33:37.589309+00:00"},{"alias_kind":"pith_short_16","alias_value":"XCXLFWSANHUSBZ2R","created_at":"2026-05-18T12:33:37.589309+00:00"},{"alias_kind":"pith_short_8","alias_value":"XCXLFWSA","created_at":"2026-05-18T12:33:37.589309+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":42,"internal_anchor_count":42,"sample":[{"citing_arxiv_id":"2504.18576","citing_title":"DriVerse: Navigation World Model for Driving Simulation via Multimodal Trajectory Prompting and Motion Alignment","ref_index":40,"is_internal_anchor":true},{"citing_arxiv_id":"2505.14202","citing_title":"MSDformer: Multi-scale Discrete Transformer For Time Series Generation","ref_index":35,"is_internal_anchor":true},{"citing_arxiv_id":"2602.02214","citing_title":"Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation","ref_index":19,"is_internal_anchor":true},{"citing_arxiv_id":"2602.02214","citing_title":"Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation","ref_index":19,"is_internal_anchor":true},{"citing_arxiv_id":"2603.04727","citing_title":"Are Multimodal LLMs Ready for Surveillance? A Reality Check on Zero-Shot Anomaly Detection in the Wild","ref_index":41,"is_internal_anchor":true},{"citing_arxiv_id":"2605.18678","citing_title":"Lance: Unified Multimodal Modeling by Multi-Task Synergy","ref_index":52,"is_internal_anchor":true},{"citing_arxiv_id":"2605.15824","citing_title":"FashionChameleon: Towards Real-Time and Interactive Human-Garment Video Customization","ref_index":35,"is_internal_anchor":true},{"citing_arxiv_id":"2605.16003","citing_title":"Echo-Forcing: A Scene Memory Framework for Interactive Long Video Generation","ref_index":4,"is_internal_anchor":true},{"citing_arxiv_id":"2605.15585","citing_title":"See Before You Code: Learning Visual Priors for Spatially Aware Educational Animation Generation","ref_index":5,"is_internal_anchor":true},{"citing_arxiv_id":"2605.16713","citing_title":"GeoWorld-VLM: Geometry from World Models for Vision-Language Models","ref_index":21,"is_internal_anchor":true},{"citing_arxiv_id":"2605.18678","citing_title":"Lance: Unified Multimodal Modeling by Multi-Task Synergy","ref_index":51,"is_internal_anchor":true},{"citing_arxiv_id":"2605.18052","citing_title":"Efficient 3D Content Reconstruction and Generation","ref_index":120,"is_internal_anchor":true},{"citing_arxiv_id":"2406.03520","citing_title":"VideoPhy: Evaluating Physical Commonsense for Video Generation","ref_index":46,"is_internal_anchor":true},{"citing_arxiv_id":"2605.15391","citing_title":"PanoWorld: Geometry-Consistent Panoramic Video World Modeling","ref_index":12,"is_internal_anchor":true},{"citing_arxiv_id":"2507.13942","citing_title":"Frozen Forecasting: A Unified Evaluation","ref_index":19,"is_internal_anchor":true},{"citing_arxiv_id":"2509.12052","citing_title":"FluentAvatar: Flicker-Free Talking-Head Animation via Phoneme-Guided Autoregressive Modeling","ref_index":14,"is_internal_anchor":true},{"citing_arxiv_id":"2509.23980","citing_title":"Towards Redundancy Reduction in Diffusion Models for Efficient Video Super-Resolution","ref_index":4,"is_internal_anchor":true},{"citing_arxiv_id":"2412.14169","citing_title":"Autoregressive Video Generation without Vector Quantization","ref_index":13,"is_internal_anchor":true},{"citing_arxiv_id":"2512.00336","citing_title":"MVAD: A Benchmark Dataset for Multimodal AI-Generated Video-Audio Detection","ref_index":28,"is_internal_anchor":true},{"citing_arxiv_id":"2503.19325","citing_title":"Long-Context Autoregressive Video Modeling with Next-Frame Prediction","ref_index":18,"is_internal_anchor":true},{"citing_arxiv_id":"2509.25161","citing_title":"Rolling Forcing: Autoregressive Long Video Diffusion in Real Time","ref_index":75,"is_internal_anchor":true},{"citing_arxiv_id":"2602.07775","citing_title":"Rolling Sink: Bridging Limited-Horizon Training and Open-Ended Testing in Autoregressive Video Diffusion","ref_index":48,"is_internal_anchor":true},{"citing_arxiv_id":"2503.22020","citing_title":"CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models","ref_index":31,"is_internal_anchor":true},{"citing_arxiv_id":"2503.12605","citing_title":"Multimodal Chain-of-Thought Reasoning: A Comprehensive Survey","ref_index":209,"is_internal_anchor":true},{"citing_arxiv_id":"2505.15809","citing_title":"MMaDA: Multimodal Large Diffusion Language Models","ref_index":99,"is_internal_anchor":true}]},"formal_canon":{"evidence_count":0,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/XCXLFWSANHUSBZ2RU7WQZVMB2J","json":"https://pith.science/pith/XCXLFWSANHUSBZ2RU7WQZVMB2J.json","graph_json":"https://pith.science/api/pith-number/XCXLFWSANHUSBZ2RU7WQZVMB2J/graph.json","events_json":"https://pith.science/api/pith-number/XCXLFWSANHUSBZ2RU7WQZVMB2J/events.json","paper":"https://pith.science/paper/XCXLFWSA"},"agent_actions":{"view_html":"https://pith.science/pith/XCXLFWSANHUSBZ2RU7WQZVMB2J","download_json":"https://pith.science/pith/XCXLFWSANHUSBZ2RU7WQZVMB2J.json","view_paper":"https://pith.science/paper/XCXLFWSA","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2312.14125&json=true","fetch_graph":"https://pith.science/api/pith-number/XCXLFWSANHUSBZ2RU7WQZVMB2J/graph.json","fetch_events":"https://pith.science/api/pith-number/XCXLFWSANHUSBZ2RU7WQZVMB2J/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/XCXLFWSANHUSBZ2RU7WQZVMB2J/action/timestamp_anchor","attest_storage":"https://pith.science/pith/XCXLFWSANHUSBZ2RU7WQZVMB2J/action/storage_attestation","attest_author":"https://pith.science/pith/XCXLFWSANHUSBZ2RU7WQZVMB2J/action/author_attestation","sign_citation":"https://pith.science/pith/XCXLFWSANHUSBZ2RU7WQZVMB2J/action/citation_signature","submit_replication":"https://pith.science/pith/XCXLFWSANHUSBZ2RU7WQZVMB2J/action/replication_record"}},"created_at":"2026-05-17T23:38:50.672125+00:00","updated_at":"2026-05-17T23:38:50.672125+00:00"}