Pith. sign in

REVIEW 3 major objections 5 minor 87 references

EgoTextVQA: Towards Egocentric Scene-Text Aware Video Question Answering

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A new benchmark for egocentric scene-text video QA shows every tested model answers under 34 percent of questions.

desk verdict A solid, well-built egocentric scene-text VideoQA benchmark; the headline numbers are believable but the single-reference GT and GPT-4o-based evaluation deserve scrutiny before the benchmark is treated as ground truth. read the letter →

arxiv 2502.07411 v2 pith:DNQJQOXW submitted 2025-02-11 cs.CV cs.MM

Sheng Zhou , Junbin Xiao , Qingyun Li , Yicong Li , Xun Yang , Dan Guo , Meng Wang , Tat-Seng Chua
show 1 more author
Angela Yao
This is my paper · ORCID
classification cs.CVcs.MM
keywords egocentricvideoquestionansweringscene-textunderstandingQAbenchmarkmultimodallargelanguagemodelsreal-timeassistancetemporalgroundingopticalcharacterrecognitiondrivingandhousekeepingactivities
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces EgoTextVQA, a benchmark meant to measure whether AI assistants can answer real questions that require reading scene text in first-person video. It contains about 7,000 timestamped questions over 1,500 egocentric videos of driving and household activities, with questions phrased the way a person would ask, not pointing at the text. The central finding is that ten strong multimodal models all perform poorly, with the best, Gemini 1.5 Pro, reaching about 33% accuracy on both splits. The authors argue this exposes a large gap between scene-text QA on clean images and the messy, temporally grounded, intention-driven setting of real egocentric assistance. They also report that human annotators score only about 43% outdoors and 28% indoors, which they interpret as evidence of genuine difficulty, while acknowledging answer diversity is a limitation of the dataset.

What carries the argument

The load-bearing object is the EgoTextVQA dataset itself, built through a pipeline that filters text-rich first-person videos with a scene-text detector, generates candidate QA pairs with a multimodal LLM from uniformly sampled frames, and then passes them through five stages of manual review that delete about 70% of the generated questions and revise the rest. A distinctive design choice is the per-question timestamp: answers must be derivable only from video content before the timestamp, which forces temporal grounding rather than whole-video look-up. The accompanying evaluation protocol uses GPT-4o mini to compare predicted answers with ground truth for semantic match, yielding an accuracy percentage and a 0-5 score that the paper shows correlates with human judgments.

What would settle it

Re-annotate a random sample of EgoTextVQA questions with several independent annotators who do not see the ground truth; if many questions receive multiple defensible answers, the reported model accuracies partly measure annotation ambiguity. Alternatively, give models the exact question-relevant frames and compare to uniform sampling; if accuracy does not consistently rise, the temporal-grounding narrative weakens.

Watch

Extended reading notes

Core claim

EgoTextVQA is presented as the first VideoQA testbed designed for egocentric scene-text aware QA assistance in real-world scenarios. The dataset pairs 7,064 questions with 1,507 videos drawn from outdoor driving and indoor house-keeping; each question carries a timestamp and models can only use video content before that timestamp, simulating real-time assistance. Across ten MLLMs, all models struggle: the best result is Gemini 1.5 Pro at 33.4% outdoor and 34.4% indoor accuracy, while open-source models range from 7.0% to 28.2%. The paper's analyses attribute the difficulty to four factors: intention reasoning, multi-frame temporal grounding, inferring user state, and egocentric dynamic scene-text recognition. Heuristic experiments show that adding high-resolution frames, scene-text super-resolution, and auxiliary OCR text each help, with GPT-4o reaching 52.9% outdoor when video, question, and OCR text are all provided; even then, this is far from reliable assistance.

Load-bearing premise

The load-bearing premise is that the human-annotated ground-truth answers are correct and unambiguous enough to score models against, even though human annotators themselves score only about 43% outdoors and 28% indoors and the authors note that answer diversity lowers human performance.

Editorial extensions

If this is right

  • If the benchmark is valid, current multimodal models are not usable for egocentric scene-text QA assistance, since even the best model answers only about one question in three.
  • Temporal grounding is a major bottleneck: feeding models the frames used to generate the question improves some models by up to 7-13 points, while a single frame at the question timestamp hurts most models.
  • Resolution and explicit scene-text input are the most reliable levers: adding OCR text alongside video lifts GPT-4o from 30.3% to 52.9% on EgoTextVQA-Outdoor, and scene-text super-resolution helps on the outdoor split.
  • Real-time QA is substantially harder than offline QA: the best accuracy on the real-time subset drops to 20.2%, suggesting live assistance needs different methods.
  • Because humans also score low, especially indoors (27.7%), the ceiling for this task is not yet clear; the authors connect this to scene-text difficulty, external knowledge, and answer diversity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next step is to measure inter-annotator agreement on the ground-truth answers; if many questions admit multiple defensible answers, the current accuracy numbers conflate model capability with annotation ambiguity.
  • The benchmark could be extended to other egocentric activities, such as shopping, repair, or navigation in unfamiliar buildings, to see whether the observed gap generalizes beyond driving and housekeeping.
  • A testable consequence of the paper's results is that a model combining explicit OCR with temporal grounding and high-resolution key-frame selection should beat both video-only and OCR-only baselines by a large margin; the paper's heuristic experiments are consistent with that but do not yet provide a trained system achieving it.
  • The authors' finding that humans also err suggests that future versions may want to allow multiple acceptable answers or partial credit, rather than a single ground truth, before interpreting model-versus-human gaps.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces EgoTextVQA, a new benchmark for egocentric scene-text aware video question answering, constructed from RoadTextVQA and EgoSchema videos using GPT-4o-generated QA pairs followed by extensive manual filtering. The dataset contains 7,064 QA pairs over 1,507 videos, split into outdoor driving and indoor housekeeping subsets, with timestamps for real-time QA. The authors evaluate 10 multimodal LLMs and find that all struggle, with the best model (Gemini 1.5 Pro) achieving approximately 33% accuracy on the outdoor set. They further analyze factors such as frame sampling, temporal grounding, resolution, and OCR input through heuristic experiments. A human study shows that human performance is also low (43% outdoor, 28% indoor), which the paper attributes partly to scene-text difficulty and partly to answer diversity.

Significance. If the benchmark's ground truth is reliable, this is a valuable contribution: it addresses a practical gap in egocentric QA assistance, covers both outdoor and indoor scenarios, provides fine-grained categories and real-time timestamps, and includes a broad evaluation of 10 models. The paper is transparent about its construction pipeline and releases the dataset, and the finding that all state-of-the-art MLLMs perform poorly is likely to stimulate research on scene-text aware egocentric reasoning. The strongest aspects are the breadth of evaluation and the heuristic analyses (e.g., effects of resolution, OCR input). However, the paper's central quantitative claim depends on the reliability and uniqueness of the ground-truth answers, which is not established and is even questioned by the authors' own human study.

major comments (3)
  1. [Section 4.2, Tables 3-4] The paper's own human study shows that human annotators achieve only 43.1% accuracy on EgoTextVQA-Outdoor and 27.7% on EgoTextVQA-Indoor, and the text attributes part of this to "answer diversity" and calls it "a limitation of our dataset." With a single ground-truth answer per question, a model that produces a semantically correct but non-canonical answer is marked incorrect, so the reported accuracies conflate model capability with label ambiguity. The paper should quantify GT reliability, for example by collecting multiple independent human answers on a sample and reporting the proportion of questions with multiple acceptable answers, or by evaluating models with a "match any acceptable answer" protocol.
  2. [Appendix C] The validation of the GPT-4o-mini evaluator (Pearson 0.80/0.87, Cohen's Kappa 0.77 on 100 QA pairs) measures agreement between the evaluator and humans when scoring model outputs against the same existing GT; it does not establish that the GT is complete or unique. Moreover, the evaluator is from the same model family as the question generator (GPT-4o vs GPT-4o-mini), which may introduce a systematic bias toward accepting GPT-4o-style phrasing. The authors should validate the evaluator on questions with multiple manually collected references and report per-category agreement.
  3. [Section 3.1] The manual filtering pipeline removed about 70% of auto-generated QAs and revised another 30%, but no inter-annotator agreement or quality-control statistics are reported for these five annotation stages. Since the dataset's defining claim is that questions "reflect real user needs" and are "scene-text aware," the consistency of these judgments is load-bearing for benchmark validity. Please report agreement metrics (e.g., Cohen's Kappa) for the filtering decisions and provide a breakdown of revision types per stage.
minor comments (5)
  1. [Section 3.1] The final video counts do not add up clearly: the text says 933 videos from Ego4D and 700 from RoadTextVQA (1633 total), but the dataset contains 1,507 videos; please specify how many videos are removed in each filtering stage.
  2. [Figure 5] The caption "Accuracy Score" is ambiguous; please clarify whether it reports Accuracy (%) and Score (0–5) separately or a single combined value.
  3. [Section 4.2] The statement "Intriguingly, we find that humans perform even worse than the closed-source models" appears in the EgoTextVQA-Indoor discussion, but Table 3 shows that humans outperform all models on EgoTextVQA-Outdoor; please clarify the scope of this sentence.
  4. [Table 2] The average video length of 101.7s would be more informative if reported separately for the two sub-datasets, since they are drawn from very different sources.
  5. [Throughout] Use of "GPT-4o mini" and "GPT-4o-mini" is inconsistent; please standardize the notation.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: EgoTextVQA is an empirical benchmark whose central claim is grounded in external model evaluations and human validation, with no derivation chain that reduces to its inputs.

full rationale

This is a benchmark-construction paper, not a derivation paper, so the standard circularity patterns (self-definitional reduction, fitted input called prediction, self-citation chain, uniqueness imported from authors, ansatz smuggled via citation, renaming known result) do not apply in a load-bearing way. The central claim that current MLLMs struggle on egocentric scene-text QA assistance is supported by direct evaluations of ten external models against ground-truth answers, which is an empirical measurement rather than a derivation from the dataset's construction. The authors do use GPT-4o to generate initial QA pairs and GPT-4o-mini to score model answers, which introduces a potential same-family bias, but they validate the evaluator against human annotators on 100 sampled QA pairs, reporting Pearson correlations of 0.80 and 0.87 and Cohen's Kappa of 0.77 (Appendix C). This independent human grounding means the evaluation is not circular by construction. The paper explicitly acknowledges a data-quality limitation: answer diversity leads to relatively lower human performance and the authors state they plan to enrich GT answers in the future (Section 4.2). That is a validity concern about the benchmark's ground truth, not a circularity concern; it does not make the measured model deficiencies equivalent to the dataset's construction choices. Self-citations appear only in related-work and grounding contexts and are not invoked as the basis for the paper's central empirical findings. The heuristic investigations (e.g., adding OCR text, increasing resolution) are exploratory and their improvements are measured against the same external models, so they are not fitted parameters disguised as predictions. Overall, no specific step reduces to its own input; the honest finding is no significant circularity, with the minor same-family evaluator note being a reproducibility caveat rather than a circularity defect.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on dataset construction choices and evaluation assumptions rather than on derived equations; the main risks are the use of hand-set thresholds, LLM-based generation and evaluation, and unvalidated ground-truth uniqueness.

free parameters (3)
  • Scene-text frame ratio thresholds = 5% (Ego4D), 15% (RoadTextVQA)
    Hand-set thresholds in Section 3.1 determine which videos are selected; they shape dataset difficulty and text density.
  • Frames per segment for QA generation = 3
    Section 3.1 uses uniform sampling of 3 frames from each of 5 segments; this affects temporal coverage and question generation.
  • Evaluation frame counts per model = 16 to 128 depending on model
    Tables 3 and 4 use different numbers of frames per model, chosen to fit context limits; cross-model comparisons may be confounded by this choice.
assumptions (3)
  • domain assumption GPT-4o-generated questions, after manual filtering, reflect real user needs in egocentric assistance
    Section 3.1: QA generation via GPT-4o with prompts; no real-user elicitation is performed. The paper's framing as 'real user needs' rests on this.
  • domain assumption GPT-4o-mini evaluation aligns with human judgment across the full benchmark
    Appendix C validates on 100 QA pairs (Pearson 0.80/0.87, Kappa 0.77), then applies to all 7,064 pairs without further human checks.
  • domain assumption Manual annotation yields unambiguous, correct ground-truth answers
    Section 3.1 describes five-stage human review, yet human accuracy on the same questions is only 43% (outdoor) and 27% (indoor), and the paper acknowledges rich answer diversity as a limitation (Section 4.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of EgoTextVQA: Towards Egocentric Scene-Text Aware Video Question Answering." pith.science (2026). https://pith.science/paper/DNQJQOXW

@misc{pith2026250207411,
  author       = {Pith},
  title        = {Pith review of: EgoTextVQA: Towards Egocentric Scene-Text Aware Video Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DNQJQOXW}},
  note         = {Machine review of arXiv:2502.07411}
}
read the original abstract

We introduce EgoTextVQA, a novel and rigorously constructed benchmark for egocentric QA assistance involving scene text. EgoTextVQA contains 1.5K ego-view videos and 7K scene-text aware questions that reflect real user needs in outdoor driving and indoor house-keeping activities. The questions are designed to elicit identification and reasoning on scene text in an egocentric and dynamic environment. With EgoTextVQA, we comprehensively evaluate 10 prominent multimodal large language models. Currently, all models struggle, and the best results (Gemini 1.5 Pro) are around 33\% accuracy, highlighting the severe deficiency of these techniques in egocentric QA assistance. Our further investigations suggest that precise temporal grounding and multi-frame reasoning, along with high resolution and auxiliary scene-text inputs, are key for better performance. With thorough analyses and heuristic suggestions, we hope EgoTextVQA can serve as a solid testbed for research in egocentric scene-text QA assistance. Our dataset is released at: https://github.com/zhousheng97/EgoTextVQA.

Figures

Figures reproduced from arXiv: 2502.07411 by the authors.

Figure 1
Figure 1. Our EgoTextVQA aims for QA assistance involving scene text from an ego-perspective mainly in outdoor driving (EgoTextVQA [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Examples of EgoTextVQA. Scene text plays pivotal role in understanding and answering the questions which reflect real user [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Distribution of QAs and OCR numbers [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: Performance of MLLMs on the real-time QA subset of [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 4
Figure 4. Figure 4: Result Visualization. accuracy and a score of 2.0, making it the SOTA model on EgoTextVQA-Outdoor. However, it still trails human per￾formance by ∼10%, indicating that even advanced MLLMs struggle with scene text perception in complex, dynamic, egocentric environments.…
Figure 6
Figure 6. Figure 6: Effects with different numbers of frame inputs sampled [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Higher resolution (1920×1080, 1280×720) QA Frames generally improve performance on EgoTextVQA-Outdoor. source models that can accept frames with resolutions of up to 1920×1080 and 1280×720 [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Manual participation on EgoTextVQA creation. [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Result visualization on EgoTextVQA [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

87 extracted references · 55 canonical work pages

  1. [1]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966 , 2023. 12

    arXiv 2023

  2. [2]

    Where did i leave my keys? - episodic-memory-based question answering on egocentric videos

    Leonard B ¨armann and Alex Waibel. Where did i leave my keys? - episodic-memory-based question answering on egocentric videos. In CVPR Workshops, pages 1560–1568,

  3. [3]

    Scene text visual question answering

    Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marc ¸al Rusinol, Ernest Valveny, CV Jawahar, and Dimos- thenis Karatzas. Scene text visual question answering. In ICCV, pages 4291–4301, 2019. 2, 3, 4, 5, 12

    2019

  4. [4]

    Nougat: Neural optical understanding for academic documents

    Lukas Blecher, Guillem Cucurull, Thomas Scialom, and Robert Stojnic. Nougat: Neural optical understanding for academic documents. arXiv preprint arXiv:2308.13418 ,

    arXiv

  5. [5]

    Internlm2 technical report

    Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internlm2 technical report. arXiv preprint arXiv:2403.17297, 2024. 12

    arXiv 2024

  6. [6]

    Sharegpt4video: Improving video understand- ing and generation with better captions

    Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. Sharegpt4video: Improving video understand- ing and generation with better captions. arXiv preprint arXiv:2406.04325, 2024. 2, 3, 5, 6, 12

    arXiv 2024

  7. [7]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024. 3, 5, 6, 7, 8, 12

    arXiv 2024

  8. [8]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. In CVPR, pages 24185–24198, 2024. 12, 13, 14

    2024

Show all 87 references
  1. [9]

    Vide- gothink: Assessing egocentric video understanding capabili- ties for embodied ai.arXiv preprint arXiv:2410.11623, 2024

    Sijie Cheng, Kechen Fang, Yangyang Yu, Sicheng Zhou, Bo- hao Li, Ye Tian, Tingguang Li, Lei Han, and Yang Liu. Vide- gothink: Assessing egocentric video understanding capabili- ties for embodied ai.arXiv preprint arXiv:2410.11623, 2024. 1

    arXiv

  2. [10]

    Egothink: Evalu- ating first-person perspective thinking capability of vision- language models

    Sijie Cheng, Zhicheng Guo, Jingwen Wu, Kechen Fang, Peng Li, Huaping Liu, and Yang Liu. Egothink: Evalu- ating first-person perspective thinking capability of vision- language models. In CVPR, pages 14291–14302, 2024. 4, 16

    2024

  3. [11]

    Grounded question-answering in long egocentric videos

    Shangzhe Di and Weidi Xie. Grounded question-answering in long egocentric videos. In CVPR, pages 12934–12943,

  4. [12]

    Egovqa-an egocentric video question answer- ing benchmark dataset

    Chenyou Fan. Egovqa-an egocentric video question answer- ing benchmark dataset. In ICCV Workshop, 2019. 1

    2019

  5. [13]

    Video-mme: The first-ever compre- hensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever compre- hensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024. 3

    2024 arXiv

  6. [14]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In CVPR, pages 18995–19012, 2022. 1, 3

    2022

  7. [15]

    Context-aware graph inference with knowledge distillation for visual dialog.IEEE TPAMI, 44(10):6056–6073, 2021

    Dan Guo, Hui Wang, and Meng Wang. Context-aware graph inference with knowledge distillation for visual dialog.IEEE TPAMI, 44(10):6056–6073, 2021. 1

    2021

  8. [16]

    Vizwiz grand challenge: Answering visual questions from blind people

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In CVPR, pages 3608–3617, 2018. 2, 5

    2018

  9. [17]

    Gomatching: A simple baseline for video text spotting via long and short term matching

    Haibin He, Maoyuan Ye, Jing Zhang, Juhua Liu, and Dacheng Tao. Gomatching: A simple baseline for video text spotting via long and short term matching. arXiv preprint arXiv:2401.07080, 2024. 3

    2024 arXiv

  10. [18]

    Cogvlm2: Visual language mod- els for image and video understanding

    Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Jun- hui Ji, Zhao Xue, et al. Cogvlm2: Visual language mod- els for image and video understanding. arXiv preprint arXiv:2408.16500, 2024. 2, 3, 5, 6, 12

    2024 arXiv

  11. [19]

    Understanding video scenes through text: Insights from text-based video question answering

    Soumya Jahagirdar, Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Understanding video scenes through text: Insights from text-based video question answering. InICCV, pages 4646–4650, 2023. 2

    2023

  12. [20]

    Egotaskqa: Understanding human tasks in egocentric videos

    Baoxiong Jia, Ting Lei, Song-Chun Zhu, and Siyuan Huang. Egotaskqa: Understanding human tasks in egocentric videos. In NeurIPS, pages 3343–3360, 2022. 1

    2022

  13. [21]

    Llava-next: Stronger llms supercharge multimodal capa- bilities in the wild, 2024

    Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. Llava-next: Stronger llms supercharge multimodal capa- bilities in the wild, 2024. 12

    2024

  14. [22]

    Pp-ocrv3: More attempts for the im- provement of ultra lightweight ocr system

    Chenxia Li, Weiwei Liu, Ruoyu Guo, Xiaoting Yin, Kaitao Jiang, Yongkun Du, Yuning Du, Lingfeng Zhu, Baohua Lai, Xiaoguang Hu, et al. Pp-ocrv3: More attempts for the im- provement of ultra lightweight ocr system. arXiv preprint arXiv:2206.03001, 2022. 12

    2022 arXiv

  15. [23]

    Flex- attention for efficient high-resolution vision-language mod- els

    Junyan Li, Delin Chen, Tianle Cai, Peihao Chen, Yining Hong, Zhenfang Chen, Yikang Shen, and Chuang Gan. Flex- attention for efficient high-resolution vision-language mod- els. In ECCV, 2025. 3

    2025

  16. [24]

    Mvbench: A comprehensive multi-modal video understand- ing benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understand- ing benchmark. In CVPR, pages 22195–22206, 2024. 3

    2024

  17. [25]

    Invariant grounding for video question answering

    Yicong Li, Xiang Wang, Junbin Xiao, Wei Ji, and Tat-Seng Chua. Invariant grounding for video question answering. In CVPR, pages 2928–2937, 2022. 2

    2022

  18. [26]

    Transformer-empowered invariant grounding for video question answering

    Yicong Li, Xiang Wang, Junbin Xiao, Wei Ji, and Tat- Seng Chua. Transformer-empowered invariant grounding for video question answering. IEEE TPAMI, 2023. 1

    2023

  19. [27]

    Vila: On pre-training for visual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Moham- mad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In CVPR, pages 26689–26699, 2024. 3, 5, 6, 12

    2024

  20. [28]

    Egocentric video-language pretraining

    Kevin Qinghong Lin, Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Z Xu, Difei Gao, Rong-Cheng Tu, Wen- zhe Zhao, Weijie Kong, et al. Egocentric video-language pretraining. NeurIPS, 35:7575–7586, 2022. 1

    2022

  21. [29]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 12

    2024

  22. [30]

    Ocrbench: On the hidden mystery of ocr in large multimodal models, 2024

    Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xucheng Yin, Cheng lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: On the hidden mystery of ocr in large multimodal models, 2024. 2, 3

    2024

  23. [31]

    Feast your eyes: Mixture-of- resolution adaptation for multimodal large language models

    Gen Luo, Yiyi Zhou, Yuxin Zhang, Xiawu Zheng, Xi- aoshuai Sun, and Rongrong Ji. Feast your eyes: Mixture-of- resolution adaptation for multimodal large language models. arXiv preprint arXiv:2403.03003, 2024. 3

    2024 arXiv

  24. [32]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fa- had Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023. 5

    2023 arXiv

  25. [33]

    Egoschema: A diagnostic benchmark for very long- form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long- form video language understanding. In NeurIPS, pages 46212–46244, 2023. 1, 3, 4

    2023

  26. [34]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In WACV, pages 2200–2209, 2021. 3

    2021

  27. [35]

    Infographicvqa

    Minesh Mathew, Viraj Bagal, Rub `en Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. In WACV, pages 1697–1706, 2022

    2022

  28. [36]

    Ocr-vqa: Visual question answering by reading text in images

    Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In ICDAR, pages 947–952, 2019. 2, 3, 12

    2019

  29. [37]

    Gpt-4o system card

    OpenAI. Gpt-4o system card. 2024. 5, 6, 7, 8, 12, 14, 16

    2024

  30. [38]

    Gpt-4o mini: advancing cost-efficient intelligence

    OpenAI. Gpt-4o mini: advancing cost-efficient intelligence

  31. [39]

    Egovlpv2: Egocentric video-language pre-training with fusion in the backbone

    Shraman Pramanick, Yale Song, Sayan Nag, Kevin Qinghong Lin, Hardik Shah, Mike Zheng Shou, Rama Chellappa, and Pengchuan Zhang. Egovlpv2: Egocentric video-language pre-training with fusion in the backbone. In ICCV, pages 5285–5297, 2023. 1

    2023

  32. [40]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, pages 8748–8763, 2021. 12

    2021

  33. [41]

    Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrit- twieser, et al. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv...

    2024 arXiv

  34. [42]

    Laion-400m: Open dataset of clip-filtered 400 million image-text pairs

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021. 12

    2021 arXiv

  35. [43]

    Annotating objects and relations in user- generated videos

    Xindi Shang, Donglin Di, Junbin Xiao, Yu Cao, Xun Yang, and Tat-Seng Chua. Annotating objects and relations in user- generated videos. In Proceedings of the 2019 on Interna- tional Conference on Multimedia Retrieval, pages 279–287,

    2019

  36. [44]

    Glu variants improve transformer

    Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020. 12

    2002 arXiv

  37. [45]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR, pages 8317–8326, 2019. 2, 3, 4, 5, 12

    2019

  38. [46]

    Eva-clip: Improved training techniques for clip at scale

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389, 2023. 12

    2023 arXiv

  39. [47]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 12

    2023 arXiv

  40. [48]

    Reading between the lanes: Text videoqa on the road

    George Tom, Minesh Mathew, Sergi Garcia-Bordils, Dimos- thenis Karatzas, and CV Jawahar. Reading between the lanes: Text videoqa on the road. In ICDAR, pages 137–154,

  41. [49]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 12

    2023 arXiv

  42. [50]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 2, 3, 5, 6, 7, 8, 12, 13, 14

    2024 arXiv

  43. [51]

    Cogvlm: Visual expert for pretrained language models

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079, 2023. 3, 12

    2023 arXiv

  44. [52]

    On the general value of ev- idence, and bilingual scene-text visual question answering

    Xinyu Wang, Yuliang Liu, Chunhua Shen, Chun Chet Ng, Canjie Luo, Lianwen Jin, Chee Seng Chan, Anton van den Hengel, and Liangwei Wang. On the general value of ev- idence, and bilingual scene-text visual question answering. In CVPR, pages 10126–10135, 2020. 2, 4

    2020

  45. [53]

    Assistq: Affordance-centric question-driven task completion for ego- centric assistant

    Benita Wong, Joya Chen, You Wu, Stan Weixian Lei, Dongxing Mao, Difei Gao, and Mike Zheng Shou. Assistq: Affordance-centric question-driven task completion for ego- centric assistant. In ECCV, pages 485–501. Springer, 2022. 1, 4

    2022

  46. [54]

    Next-qa: Next phase of question-answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In CVPR, pages 9777–9786, 2021. 3

    2021

  47. [55]

    Videoqa in the era of llms: An empirical study

    Junbin Xiao, Nanxin Huang, Hangyu Qin, Dongyang Li, Yi- cong Li, Fengbin Zhu, Zhulin Tao, Jianxing Yu, Liang Lin, Tat-Seng Chua, and Angela Yao. Videoqa in the era of llms: An empirical study. arXiv preprint arXiv:2408.04223, 2024. 1

    2024 arXiv

  48. [56]

    Deconfounded video moment retrieval with causal intervention

    Xun Yang, Fuli Feng, Wei Ji, Meng Wang, and Tat-Seng Chua. Deconfounded video moment retrieval with causal intervention. In Proceedings of the 44th international ACM SIGIR conference on research and development in informa- tion retrieval, pages 1–10, 2021. 3

    2021

  49. [57]

    Video moment retrieval with cross-modal neural architecture search

    Xun Yang, Shanshan Wang, Jian Dong, Jianfeng Dong, Meng Wang, and Tat-Seng Chua. Video moment retrieval with cross-modal neural architecture search. IEEE Transac- tions on Image Processing, 31:1204–1216, 2022. 3

    2022

  50. [58]

    Robust video question answer- ing via contrastive cross-modality representation learning

    Xun Yang, Jianming Zeng, Dan Guo, Shanshan Wang, Jian- feng Dong, and Meng Wang. Robust video question answer- ing via contrastive cross-modality representation learning. Science China Information Sciences , 67(10):202104, 2024. 1

    2024

  51. [59]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024. 2, 3, 5, 6, 7, 8, 12

    2024 arXiv

  52. [60]

    Mm-ego: To- wards building egocentric multimodal llms

    Hanrong Ye, Haotian Zhang, Erik Daxberger, Lin Chen, Zongyu Lin, Yanghao Li, Bowen Zhang, Haoxuan You, Dan Xu, Zhe Gan, Jiasen Lu, and Yinfei Yang. Mm-ego: To- wards building egocentric multimodal llms. arXiv preprint arXiv:2410.07177, 2024. 1, 4, 5

    arXiv

  53. [61]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, pages 11975–11986, 2023. 12

    2023

  54. [62]

    Multi-factor adaptive vision selec- tion for egocentric video question answering

    Haoyu Zhang, Meng Liu, Zixin Liu, Xuemeng Song, Yaowei Wang, and Liqiang Nie. Multi-factor adaptive vision selec- tion for egocentric video question answering. InICML, 2024. 1

    2024

  55. [63]

    Llava-read: Enhancing read- ing ability of multimodal language models

    Ruiyi Zhang, Yufan Zhou, Jian Chen, Jiuxiang Gu, Changyou Chen, and Tong Sun. Llava-read: Enhancing read- ing ability of multimodal language models. arXiv preprint arXiv:2407.19185, 2024. 3

    2024 arXiv

  56. [64]

    Llava- next: A strong zero-shot video understanding model, 2024

    Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava- next: A strong zero-shot video understanding model, 2024. 3, 5, 6, 7, 8, 12, 13

    2024

  57. [65]

    Diffusion-based blind text image super-resolution

    Yuzhe Zhang, Jiawei Zhang, Hao Li, Zhouxia Wang, Luwei Hou, Dongqing Zou, and Liheng Bian. Diffusion-based blind text image super-resolution. In CVPR, pages 25827–25836,

  58. [66]

    Beyond llava-hd: Diving into high-resolution large multimodal models

    Yi-Fan Zhang, Qingsong Wen, Chaoyou Fu, Xue Wang, Zhang Zhang, Liang Wang, and Rong Jin. Beyond llava-hd: Diving into high-resolution large multimodal models. arXiv preprint arXiv:2406.08487, 2024. 3

    2024 arXiv

  59. [67]

    Towards video text visual question answering: Benchmark and baseline

    Minyi Zhao, Bingjia Li, Jie Wang, Wanqing Li, Wenjing Zhou, Lan Zhang, Shijie Xuyang, Zhihang Yu, Xinkun Yu, Guangze Li, et al. Towards video text visual question answering: Benchmark and baseline. In NeurIPS, pages 35549–35562, 2022. 2, 4

    2022

  60. [68]

    Exploring sparse spatial relation in graph inference for text- based vqa

    Sheng Zhou, Dan Guo, Jia Li, Xun Yang, and Meng Wang. Exploring sparse spatial relation in graph inference for text- based vqa. IEEE TIP, 32:5060–5074, 2023. 2

    2023

  61. [69]

    blue dart

    Sheng Zhou, Junbin Xiao, Xun Yang, Peipei Song, Dan Guo, Angela Yao, Meng Wang, and Tat-Seng Chua. Scene-text grounding for text-based video question answering. arXiv preprint arXiv:2409.14319, 2024. 2 EgoTextVQA: Towards Egocentric Scene-Text Aware Video Question Answering Su...

    2024 arXiv

  62. [70]

    I” should be used appropriately. Requirement 5: The questions should be of moderate length. When announcing the question please label each question as “Question 1, 2, 3: {question}

    and the prompt in Table 16. Table 12. Judgments of human and GPT-4o mini. Method GPT-4o [37] Gemini 1.5 Pro [41] Accuracy Score Accuracy Score Human 36.0 1.9 47.3 2.5 GPT-4o mini [38] 34.0 1.8 42.0 2.3 D. Model Prompts Table 13 provides the prompts used by GPT-4o for ques- tio...

  63. [71]

    For example:

    Location: Questions about a place or location. For example:

  64. [72]

    Where is the gas station?

  65. [73]

    Which stores can I find on the right side of the road at this intersection?

  66. [74]

    For example:

    Direction: Questions related to navigation, driving direction, and turns. For example:

  67. [75]

    Is the next road a left or right turn?

  68. [76]

    If I want to go to Cava, on which side of the street should I look for it?

  69. [77]

    Where should trucks go according to the signs?

  70. [78]

    For example:

    Description: Questions that focus on scene text such as road signs, price labels, and billboards. For example:

  71. [79]

    What does the sign on the side of the road say?

  72. [80]

    What is the name of the center on the left side of the road?

  73. [81]

    What is the name of the street to my right?

  74. [82]

    For example:

    Intention Reasoning: Questions about behavioral activities involving drivers or passengers to solve personal needs. For example:

  75. [83]

    Where do I need to go to solve my financial problems?

  76. [84]

    Is there a place nearby where I can shop for appliances and electronics?

  77. [85]

    For example:

    Others: Composite questions that involve multiple different or the same types of the above, such as asking about both description and location. For example:

  78. [86]

    What event is being advertised on the bus, and where is it taking place?

  79. [87]

    Unanswerable

    What is the contact number for the leadspace building, and what service might they provide? Table 15. Prompts for MLLM inference on EgoTextVQA. Model General Prompts GPT-4o Based on the following images from a video, please briefly answer the following question with a single w...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.