REVIEW 7 cited by
Order Matters: Exploring Order Sensitivity in Multimodal Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Multimodal Large Language Models (MLLMs) utilize multimodal contexts consisting of text, images, or videos to solve various multimodal tasks. However, we find that changing the order of multimodal input can cause the model's performance to fluctuate between advanced performance and random guessing. This phenomenon exists in both single-modality (text-only or image-only) and mixed-modality (image-text-pair) contexts. Furthermore, we demonstrate that popular MLLMs pay special attention to certain multimodal context positions, particularly the beginning and end. Leveraging this special attention, we place key video frames and important image/text content in special positions within the context and submit them to the MLLM for inference. This method results in average performance gains of 14.7% for video-caption matching and 17.8% for visual question answering tasks. Additionally, we propose a new metric, Position-Invariant Accuracy (PIA), to address order bias in MLLM evaluation. Our research findings contribute to a better understanding of Multi-Modal In-Context Learning (MMICL) and provide practical strategies for enhancing MLLM performance without increasing computational costs.
Forward citations
Cited by 7 Pith papers
-
Same Evidence, Different Answer: Auditing Order Sensitivity in Multimodal Large Language Models
All 18 audited MLLMs exhibit order sensitivity with per-facet flip rates of 24-50%, exceeding same-order decoder noise.
-
Lost at the End: Primacy Bias in Multimodal Retrieval-Augmented Question Answering
Multimodal KB-VQA exhibits a primacy bias where gold passages at prompt start outperform those at the end by 16-26 points, flipping the text-only lost-in-the-middle pattern.
-
Towards Order Fairness: Mitigating LLMs Order Sensitivity through Dual Group Advantage Optimization
DGAO uses reinforcement learning to optimize LLMs for both accuracy and order stability by balancing intra-group accuracy advantages and inter-group stability advantages.
-
Lost at the End: Primacy Bias in Multimodal Retrieval-Augmented Question Answering
In multimodal KB-VQA, gold evidence at the first prompt slot beats gold at the last by 16–26 points, flipping the classic U-shaped lost-in-the-middle pattern into primacy bias.
-
Logit-Attention Divergence: Mitigating Position Bias in Multi-Image Retrieval via Attention-Guided Calibration
A training-free attention-guided debiasing framework mitigates position bias in MLLM multi-image retrieval by exploiting the observed mismatch between biased logits and aligned attention maps, yielding over 40% accura...
-
Detecting Differences Is Not Understanding Structure: Large Language Models Fail at Graph Isomorphism
LLMs succeed at graph isomorphism detection but fail to recognize isomorphic graphs under node label permutation, indicating pattern exploitation over topological understanding.
-
MetaRA: Metamorphic Robustness Assessment for Multimodal Large Language Model-based Visual Question Answering Systems
MetaRA applies metamorphic testing to VQA tasks and shows that MLLM models exhibit sensitivity to linguistic perturbations and superficial visual cues not detected by conventional accuracy benchmarks.
Discussion (0). Sign in to comment.