{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2024:VIEYAONDM5LUTKLMMNKATGG3N7","short_pith_number":"pith:VIEYAOND","schema_version":"1.0","canonical_sha256":"aa098039a3675749a96c63540998db6fc6907ba0875170782140cef6079be0de","source":{"kind":"arxiv","id":"2404.12390","version":4},"attestation_state":"computed","paper":{"title":"BLINK: Multimodal Large Language Models Can See but Not Perceive","license":"http://creativecommons.org/licenses/by-nc-sa/4.0/","headline":"Multimodal LLMs like GPT-4V reach only 51% accuracy on visual perception tasks that humans solve at 96%.","cross_cats":["cs.AI","cs.CL"],"primary_cat":"cs.CV","authors_text":"Bangzheng Li, Dan Roth, Haoyu Wang, Noah A. Smith, Ranjay Krishna, Wei-Chiu Ma, Xingyu Fu, Xudong Lin, Yu Feng, Yushi Hu","submitted_at":"2024-04-18T17:59:54Z","abstract_excerpt":"We introduce Blink, a new benchmark for multimodal language models (LLMs) that focuses on core visual perception abilities not found in other evaluations. Most of the Blink tasks can be solved by humans \"within a blink\" (e.g., relative depth estimation, visual correspondence, forensics detection, and multi-view reasoning). However, we find these perception-demanding tasks cast significant challenges for current multimodal LLMs because they resist mediation through natural language. Blink reformats 14 classic computer vision tasks into 3,807 multiple-choice questions, paired with single or mult"},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":true,"formal_links_present":true},"canonical_record":{"source":{"id":"2404.12390","kind":"arxiv","version":4},"metadata":{"license":"http://creativecommons.org/licenses/by-nc-sa/4.0/","primary_cat":"cs.CV","submitted_at":"2024-04-18T17:59:54Z","cross_cats_sorted":["cs.AI","cs.CL"],"title_canon_sha256":"4d8fd9e1fea6457fae3bc1f04cdd373d055d3fb0b8cdf6f80054724814cfc882","abstract_canon_sha256":"dd25bcb3e35202474023a787b0b9d122840766b9a54178a832f88e9f180d9e66"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-05-17T23:38:50.298491Z","signature_b64":"eqzLJzOtimHKDjaODKdcnBkdv5u3lrlkcWSoKFU0UHn202N2RdZrXAWToiUAjJ/k3S7augmwop5OGA7QIVtPAQ==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"aa098039a3675749a96c63540998db6fc6907ba0875170782140cef6079be0de","last_reissued_at":"2026-05-17T23:38:50.297986Z","signature_status":"signed_v1","first_computed_at":"2026-05-17T23:38:50.297986Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"BLINK: Multimodal Large Language Models Can See but Not Perceive","license":"http://creativecommons.org/licenses/by-nc-sa/4.0/","headline":"Multimodal LLMs like GPT-4V reach only 51% accuracy on visual perception tasks that humans solve at 96%.","cross_cats":["cs.AI","cs.CL"],"primary_cat":"cs.CV","authors_text":"Bangzheng Li, Dan Roth, Haoyu Wang, Noah A. Smith, Ranjay Krishna, Wei-Chiu Ma, Xingyu Fu, Xudong Lin, Yu Feng, Yushi Hu","submitted_at":"2024-04-18T17:59:54Z","abstract_excerpt":"We introduce Blink, a new benchmark for multimodal language models (LLMs) that focuses on core visual perception abilities not found in other evaluations. Most of the Blink tasks can be solved by humans \"within a blink\" (e.g., relative depth estimation, visual correspondence, forensics detection, and multi-view reasoning). However, we find these perception-demanding tasks cast significant challenges for current multimodal LLMs because they resist mediation through natural language. Blink reformats 14 classic computer vision tasks into 3,807 multiple-choice questions, paired with single or mult"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"even the best-performing GPT-4V and Gemini achieve accuracies of 51.26% and 45.72%, only 13.17% and 7.63% higher than random guessing, indicating that such perception abilities have not emerged yet in recent multimodal LLMs","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"That the selected tasks genuinely require visual perception that cannot be solved through language patterns or statistical shortcuts in the training data.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"BLINK benchmark shows multimodal LLMs reach only 45-51 percent accuracy on core visual perception tasks where humans achieve 95 percent, indicating these abilities have not emerged.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"Multimodal LLMs like GPT-4V reach only 51% accuracy on visual perception tasks that humans solve at 96%.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"2917149b3d2dde9ce6f177db5480221f03644168e304eef6bdde5e78bf6798a4"},"source":{"id":"2404.12390","kind":"arxiv","version":4},"verdict":{"id":"8e751e7c-b924-40d4-95b3-a023e85b974f","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-15T20:14:26.616449Z","strongest_claim":"even the best-performing GPT-4V and Gemini achieve accuracies of 51.26% and 45.72%, only 13.17% and 7.63% higher than random guessing, indicating that such perception abilities have not emerged yet in recent multimodal LLMs","one_line_summary":"BLINK benchmark shows multimodal LLMs reach only 45-51 percent accuracy on core visual perception tasks where humans achieve 95 percent, indicating these abilities have not emerged.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"That the selected tasks genuinely require visual perception that cannot be solved through language patterns or statistical shortcuts in the training data.","pith_extraction_headline":"Multimodal LLMs like GPT-4V reach only 51% accuracy on visual perception tasks that humans solve at 96%."},"references":{"count":90,"sample":[{"doi":"","year":2024,"title":"Introducing the next generation of claude.https://www.anthropic.com/news/ claude-3-family (March 2024) 11, 12, 23, 24","work_id":"f8bea833-ebe2-4366-aa34-0ae3433f6dc7","ref_index":1,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2019,"title":"In: AAAI (2019) 10","work_id":"365c08d1-df9d-4bc5-859b-42eb662a253d","ref_index":2,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2022,"title":"Advances in Neural Information Processing Systems35, 23716–23736 (2022) 2, 4, 22","work_id":"88ba30a1-83ea-4142-9669-25cd7d6dffa5","ref_index":3,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2015,"title":"In: Proceedings of the IEEE international conference on computer vision","work_id":"a8727b08-59c4-43f9-b90a-a33fa63ffb0b","ref_index":4,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2023,"title":"OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models","work_id":"87bfa84a-e663-4165-806f-93ef439d88d0","ref_index":5,"cited_arxiv_id":"2308.01390","is_internal_anchor":true}],"resolved_work":90,"snapshot_sha256":"0ff96068e0e13eefed46d8bbcc5691567336e60abbd40ed9842bf8fa961171ee","internal_anchors":20},"formal_canon":{"evidence_count":2,"snapshot_sha256":"b1aea25256854ddbeeb2d9df367d438391cb3695d8baf28238790b470a4e8fb0"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2404.12390","created_at":"2026-05-17T23:38:50.298083+00:00"},{"alias_kind":"arxiv_version","alias_value":"2404.12390v4","created_at":"2026-05-17T23:38:50.298083+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2404.12390","created_at":"2026-05-17T23:38:50.298083+00:00"},{"alias_kind":"pith_short_12","alias_value":"VIEYAONDM5LU","created_at":"2026-05-18T12:33:37.589309+00:00"},{"alias_kind":"pith_short_16","alias_value":"VIEYAONDM5LUTKLM","created_at":"2026-05-18T12:33:37.589309+00:00"},{"alias_kind":"pith_short_8","alias_value":"VIEYAOND","created_at":"2026-05-18T12:33:37.589309+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":33,"internal_anchor_count":33,"sample":[{"citing_arxiv_id":"2503.19786","citing_title":"Gemma 3 Technical Report","ref_index":18,"is_internal_anchor":true},{"citing_arxiv_id":"2505.23678","citing_title":"Grounded Reinforcement Learning for Visual Reasoning","ref_index":13,"is_internal_anchor":true},{"citing_arxiv_id":"2605.20278","citing_title":"ClaimDiff-RL: Fine-Grained Caption Reinforcement Learning through Visual Claim Comparison","ref_index":8,"is_internal_anchor":true},{"citing_arxiv_id":"2605.16403","citing_title":"When Vision Speaks for Sound","ref_index":18,"is_internal_anchor":true},{"citing_arxiv_id":"2605.18445","citing_title":"What's Holding Back Latent Visual Reasoning?","ref_index":6,"is_internal_anchor":true},{"citing_arxiv_id":"2605.18714","citing_title":"Semantic Generative Tuning for Unified Multimodal Models","ref_index":16,"is_internal_anchor":true},{"citing_arxiv_id":"2605.18746","citing_title":"ESI-Bench: Towards Embodied Spatial Intelligence that Closes the Perception-Action Loop","ref_index":7,"is_internal_anchor":true},{"citing_arxiv_id":"2507.01955","citing_title":"How Well Does GPT-4o Understand Vision? Evaluating Multimodal Foundation Models on Standard Computer Vision Tasks","ref_index":21,"is_internal_anchor":true},{"citing_arxiv_id":"2511.10946","citing_title":"Abstract 3D Perception for Spatial Intelligence in Vision-Language Models","ref_index":11,"is_internal_anchor":true},{"citing_arxiv_id":"2412.14164","citing_title":"MetaMorph: Multimodal Understanding and Generation via Instruction Tuning","ref_index":258,"is_internal_anchor":true},{"citing_arxiv_id":"2406.09411","citing_title":"MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding","ref_index":17,"is_internal_anchor":true},{"citing_arxiv_id":"2406.16860","citing_title":"Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs","ref_index":42,"is_internal_anchor":true},{"citing_arxiv_id":"2601.09536","citing_title":"Omni-R1: Towards the Unified Generative Paradigm for Multimodal Reasoning","ref_index":31,"is_internal_anchor":true},{"citing_arxiv_id":"2408.13257","citing_title":"MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?","ref_index":20,"is_internal_anchor":true},{"citing_arxiv_id":"2605.11405","citing_title":"20/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alone","ref_index":15,"is_internal_anchor":true},{"citing_arxiv_id":"2409.02813","citing_title":"MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark","ref_index":12,"is_internal_anchor":true},{"citing_arxiv_id":"2604.00799","citing_title":"Multimodal Language Models Cannot Spot Spatial Inconsistencies","ref_index":13,"is_internal_anchor":true},{"citing_arxiv_id":"2406.09414","citing_title":"Depth Anything V2","ref_index":21,"is_internal_anchor":true},{"citing_arxiv_id":"2605.11405","citing_title":"20/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alone","ref_index":15,"is_internal_anchor":true},{"citing_arxiv_id":"2605.09883","citing_title":"The Cartesian Shortcut: Re-evaluate Vision Reasoning in Polar Coordinate Space","ref_index":13,"is_internal_anchor":true},{"citing_arxiv_id":"2503.01743","citing_title":"Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs","ref_index":19,"is_internal_anchor":true},{"citing_arxiv_id":"2604.24583","citing_title":"Improving Vision-language Models with Perception-centric Process Reward Models","ref_index":11,"is_internal_anchor":true},{"citing_arxiv_id":"2605.02735","citing_title":"Visual Latents Know More Than They Say: Unsilencing Latent Reasoning in MLLMs","ref_index":8,"is_internal_anchor":true},{"citing_arxiv_id":"2604.20012","citing_title":"EmbodiedMidtrain: Bridging the Gap between Vision-Language Models and Vision-Language-Action Models via Mid-training","ref_index":7,"is_internal_anchor":true},{"citing_arxiv_id":"2605.04075","citing_title":"RetentiveKV: State-Space Memory for Uncertainty-Aware Multimodal KV Cache Eviction","ref_index":6,"is_internal_anchor":true}]},"formal_canon":{"evidence_count":2,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/VIEYAONDM5LUTKLMMNKATGG3N7","json":"https://pith.science/pith/VIEYAONDM5LUTKLMMNKATGG3N7.json","graph_json":"https://pith.science/api/pith-number/VIEYAONDM5LUTKLMMNKATGG3N7/graph.json","events_json":"https://pith.science/api/pith-number/VIEYAONDM5LUTKLMMNKATGG3N7/events.json","paper":"https://pith.science/paper/VIEYAOND"},"agent_actions":{"view_html":"https://pith.science/pith/VIEYAONDM5LUTKLMMNKATGG3N7","download_json":"https://pith.science/pith/VIEYAONDM5LUTKLMMNKATGG3N7.json","view_paper":"https://pith.science/paper/VIEYAOND","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2404.12390&json=true","fetch_graph":"https://pith.science/api/pith-number/VIEYAONDM5LUTKLMMNKATGG3N7/graph.json","fetch_events":"https://pith.science/api/pith-number/VIEYAONDM5LUTKLMMNKATGG3N7/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/VIEYAONDM5LUTKLMMNKATGG3N7/action/timestamp_anchor","attest_storage":"https://pith.science/pith/VIEYAONDM5LUTKLMMNKATGG3N7/action/storage_attestation","attest_author":"https://pith.science/pith/VIEYAONDM5LUTKLMMNKATGG3N7/action/author_attestation","sign_citation":"https://pith.science/pith/VIEYAONDM5LUTKLMMNKATGG3N7/action/citation_signature","submit_replication":"https://pith.science/pith/VIEYAONDM5LUTKLMMNKATGG3N7/action/replication_record"}},"created_at":"2026-05-17T23:38:50.298083+00:00","updated_at":"2026-05-17T23:38:50.298083+00:00"}