Pith. sign in

REVIEW 9 cited by

Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2401.10529 v2 pith:AEDRHEWW submitted 2024-01-19 cs.CV cs.AIcs.CLcs.LG

Xiyao Wang , Yuhang Zhou , Xiaoyu Liu , Hongjin Lu , Yuancheng Xu , Feihong He , Jaehong Yoon , Taixi Lu
show 4 more authors
Gedas Bertasius Mohit Bansal Huaxiu Yao Furong Huang
This is my paper
classification cs.CVcs.AIcs.CLcs.LG
keywords imagemementosmllmsreasoningsequenceshallucinationsbehavioralbehaviors
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Multimodal Large Language Models (MLLMs) have demonstrated proficiency in handling a variety of visual-language tasks. However, current MLLM benchmarks are predominantly designed to evaluate reasoning based on static information about a single image, and the ability of modern MLLMs to extrapolate from image sequences, which is essential for understanding our ever-changing world, has been less investigated. To address this challenge, this paper introduces Mementos, a new benchmark designed to assess MLLMs' sequential image reasoning abilities. Mementos features 4,761 diverse image sequences with varying lengths. We also employ a GPT-4 assisted method to evaluate MLLM reasoning performance. Through a careful evaluation of nine recent MLLMs on Mementos, including GPT-4V and Gemini, we find that they struggle to accurately describe dynamic information about given image sequences, often leading to hallucinations/misrepresentations of objects and their corresponding behaviors. Our quantitative analysis and case studies identify three key factors impacting MLLMs' sequential image reasoning: the correlation between object and behavioral hallucinations, the influence of cooccurring behaviors, and the compounding impact of behavioral hallucinations. Our dataset is available at https://github.com/umd-huang-lab/Mementos.

Discussion (0). Sign in to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Beyond Retrieval: Analytic Memory for Multimodal Agents

    cs.AI 2026-07 conditional novelty 6.0 of 10

    ADAMM induces queryable analytic tables from multimodal interaction histories and combines them with semantic retrieval, improving benchmark accuracy by up to 11.3 points over memory baselines.

  2. What makes Reasoning Models Different? Follow the Reasoning Leader for Efficient Decoding

    cs.CL 2025-06 conditional novelty 6.0 of 10

    FoReaL-Decoding lets a strong reasoning model generate the first few tokens of each sentence and a weaker model complete the sentence, cutting theoretical FLOPs by 30-55% while retaining 86-100% of accuracy on four ma...

  3. Retrieval Visual Contrastive Decoding to Mitigate Object Hallucinations in Large Vision-Language Models

    cs.CV 2025-05 conditional novelty 6.0 of 10

    RVCD uses YOLO detections and retrieved single-concept AI images to adjust LVLM logits at decode time, cutting CHAIR hallucination rates by roughly half versus prior contrastive decoding baselines.

  4. Mono-InternVL-1.5: Towards Cheaper and Faster Monolithic Multimodal Large Language Models

    cs.CV 2025-07 conditional novelty 5.5 of 10

    A monolithic multimodal LLM that cuts pre-training data by 58% and first-token latency by up to 69% while matching or beating its predecessor on 15 benchmarks.

  5. Towards Mitigating Hallucinations in Large Vision-Language Models by Refining Textual Embeddings

    cs.CV 2025-11 conditional novelty 5.0 of 10

    Injecting an average-pooled visual embedding into every text token improves hallucination-benchmark scores of Video-LLaVA by small single-digit amounts.

  6. CAT: Causal Attention Tuning For Injecting Fine-grained Causal Knowledge into Large Language Models

    cs.CL 2025-09 conditional novelty 5.0 of 10

    A training method that injects token-level causal labels into attention improves out-of-distribution accuracy on a synthetic benchmark and slightly on math/reasoning tasks.

  7. Dual-Stage Value-Guided Inference with Margin-Based Reward Adjustment for Fast and Faithful VLM Captioning

    cs.CV 2025-06 reject novelty 5.0 of 10

    A two-stage, value-guided decoding strategy with a margin-based reward adjustment is claimed to yield more faithful, detailed VLM captions at about a quarter of VisVM's inference cost.

  8. SAILViT: Towards Robust and Generalizable Visual Backbones for MLLMs via Gradual Feature Refinement

    cs.CV 2025-07 conditional novelty 4.0 of 10

    A three-stage coarse-to-fine training recipe for vision backbones produces consistent benchmark gains for lightweight multimodal LLMs.

  9. Mitigating Behavioral Hallucination in Multimodal Large Language Models for Sequential Images

    cs.AI 2025-06 reject novelty 4.0 of 10

    SHE lowers behavioral hallucination scores by about 10 percent by detecting low visual-textual similarity and projecting out the hallucinated direction in embedding space.

Pith tools