Pith. sign in

REVIEW 3 major objections 3 minor 2 cited by

HCQA-1.5 @ Ego4D EgoSchema Challenge 2025

T0 review · 3 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A three-model vote with a confidence filter reaches 77% accuracy on the EgoSchema blind test.

desk verdict A legitimate third-place challenge report with a plausible 77% on EgoSchema; the confidence-routing trick is real but under-analyzed, and the paper is too thin for a top venue but deserves a serious workshop referee. read the letter →

arxiv 2505.20644 v1 pith:44DXYPYU submitted 2025-05-27 cs.CV cs.AI

Haoyu Zhang , Yisen Feng , Qiaohui Chu , Meng Liu , Weili Guan , Yaowei Wang , Liqiang Nie This is my paper · ORCID
classification cs.CVcs.AI
keywords egocentricvideoquestionansweringEgoSchemalong-formunderstandingmulti-sourceaggregationconfidence-basedfilteringfine-grainedreasoningvision-languagemodelslargelanguage
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper reports a two-stage extension of the HCQA pipeline for egocentric video question answering and claims 77% accuracy on the EgoSchema blind test set, a gain over its prior 75% result and enough for third place in the CVPR 2025 Ego4D EgoSchema Challenge. Instead of trusting one large language model with the final decision, it lets three models (Gemini-1.5-Pro, GPT-4.1, and Qwen2.5) each propose an answer with a confidence score. Answers scored above 4 on a 1–5 scale are kept; the remaining cases are re-examined by a visual module and by a text-reasoning module, and the higher-confidence result becomes the final answer. A sympathetic reader would care because the method improves reliability on long, ambiguous first-person videos without redesigning the underlying architecture.

What carries the argument

The load-bearing mechanism is a two-stage decision pipeline. Stage 1 is multi-source aggregation: three LLMs generate candidate answers with self-reported confidence scores, and any answer with confidence above 4 (on a 1–5 scale) is kept as final. Stage 2 is fine-grained reasoning for the leftover cases: Qwen2.5-VL-72B re-reads 45 uniformly sampled frames, while DeepSeek-R1 reasons over all accumulated captions, summaries, and candidate answers; the more confident of the two refined outputs wins. The confidence score is therefore the router that decides which samples need deeper work.

What would settle it

A calibration check: for each of the three aggregation models, compare the accuracy of answers rated 4 versus 5 on a held-out sample; if ratings are not monotonic with accuracy, or if the best cutoff differs by model, the fixed 4-point threshold is not doing the claimed routing work.

Watch

Extended reading notes

Core claim

The paper's central claim is that answer reliability in long-form egocentric video QA can be improved by replacing the single final LLM with a multi-source ensemble plus a confidence-based router. On the EgoSchema blind test, the full pipeline scores 77.3% in the authors' ablation and appears as 77% on the public leaderboard, outperforming GPT-4o (72%), Gemini-1.5-Pro (71%), and most competing teams while ranking third. The gain comes in two steps: high-confidence predictions from the ensemble are emitted directly, and low-confidence predictions are sent to Qwen2.5-VL-72B for vision re-analysis and to DeepSeek-R1 for text-based reasoning, with the higher-confidence output selected.

Load-bearing premise

The load-bearing premise is that the confidence scores each language model reports on a 1–5 scale are meaningful and comparable, so a fixed cutoff of 4 correctly separates answers that can be trusted from answers that need more reasoning.

Editorial extensions

If this is right

  • If the 77% result holds, a relatively cheap two-stage ensemble closes part of the gap between general VLMs (around 71–76%) and the top challenge submissions without new training.
  • The ablation implies each ensemble member contributes: GPT-4.1 alone gives 76.1%, Qwen2.5 74.8%, Gemini-1.5-Pro 71.0%, and the confidence-based integration reaches 76.3%.
  • Low-confidence samples are recoverable: either visual reasoning (76.8%) or thought-based reasoning (76.6%) alone nearly matches the full pipeline, whose combined score is 77.3%.
  • The failed case in Section 3.3 indicates that surface cues such as dice and cards can mislead the router into overconfident wrong answers, so the remaining bottleneck is understanding social interaction and intent, not object recognition.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The fixed threshold of 4 is never calibrated; a natural test is to vary the cutoff on a held-out split and measure whether the 77% is sensitive to it.
  • Confidence scores from different LLMs are treated as comparable, which is a strong assumption; an alternative routing rule based on model agreement or per-model calibration could behave differently.
  • Because the vision module samples only 45 frames from a three-minute video, long-range temporal reasoning is delegated to the text summaries; an adaptive frame-selection strategy could be a direct extension.
  • The reported failure on the board-game example suggests that joint-action understanding is a promising next target; forcing the model to first state who does what with whom might reduce such errors.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The manuscript reports HCQA-1.5, the authors' third-place entry in the Ego4D EgoSchema Challenge at CVPR 2025. The method extends their prior HCQA pipeline by replacing the single final-stage LLM with a multi-source aggregation of Gemini-1.5-Pro, GPT-4.1, and Qwen2.5, and by adding a confidence-based filter: answers with confidence higher than 4 on a 1-5 scale are output directly, while low-confidence cases are sent to a fine-grained reasoning stage that combines vision-based reasoning (Qwen2.5-VL-72B on 45 uniformly sampled frames) and thought-based reasoning (DeepSeek-R1 over textual captions, summaries, and prior predictions). The paper reports 77% accuracy on the EgoSchema blind test set, third place on the leaderboard, and an ablation (Table 2) showing stage-wise accuracies of 0.763 after integration, 0.768 with Qwen2.5-VL-72B, 0.766 with DeepSeek-R1, and 0.773 for the full pipeline. A single failed example is analyzed in Figure 2.

Significance. If the result holds, the paper provides a simple and effective demonstration that ensembling multiple LLMs with confidence-based selective reasoning improves long-form egocentric video QA, outperforming GPT-4o and Gemini-1.5-Pro on a public blind benchmark. The strengths are the external evaluation on over 5,000 questions and the clear ablation. The main weakness is that confidence scores are used as a load-bearing routing signal without any calibration or sensitivity analysis, and the reported incremental gains are small. These issues do not invalidate the empirical result, but they need to be addressed for the claim that each component contributes positively.

major comments (3)
  1. [§2.1, §2.2] The confidence-based routing is the load-bearing component of the method, but the paper provides no evidence that the confidence scores produced by Gemini-1.5-Pro, GPT-4.1, and Qwen2.5 are comparable or calibrated. Section 2.1 states that predictions with confidence higher than 4 on a 1-5 scale are 'considered reliable,' and the aggregation and final selection both use the highest confidence, yet no distribution of confidence scores, no threshold sweep, and no analysis of tie-breaking are reported. Because the two-stage design and all of the reported gains depend on this threshold, the central claim would be substantially stronger with a sensitivity analysis showing that the 77% result is stable across reasonable thresholds (e.g., 3, 4, 5) and that the filtering does not discard correct high-confidence answers or promote incorrect ones.
  2. [Table 2, §3.2] The ablation table shows small differences between adjacent configurations: integration improves over the best single model by 0.2 points (0.763 vs. 0.761), and the full pipeline improves over the better stage-2 model by 0.5 points (0.773 vs. 0.768). No error bars, repeated runs, or significance tests are reported, and with roughly 5,000 test questions a 0.2-point difference is within the range of sampling noise if any stochastic component is involved. The sentence 'each stage makes a positive contribution' (end of §3.2) is therefore not supported with the current evidence; reporting variance across multiple runs or at least a bootstrap interval would make the ablation claim convincing.
  3. [§2.2] The vision-based reasoning stage extracts 45 frames uniformly from the video and inputs them to Qwen2.5-VL-72B 'with default settings.' The number 45 is not justified and no ablation over frame count (e.g., 16, 32, 64) is provided. Since this stage is one of the two fine-grained reasoning strategies used for low-confidence samples, the choice of frame sampling could affect which answers are revised and the overall accuracy. Please clarify why 45 frames was selected and report sensitivity to this parameter, or state that it was not tuned.
minor comments (3)
  1. [§2.1, References] Section 2.1 cites 'Qwen2.5 [17]' but reference [17] is the Qwen3 technical report; the citation should be corrected to the Qwen2.5 technical report or the text should be updated.
  2. [§3.1, Table 2] The headline accuracy is 77% while Table 2 reports 0.773; please state explicitly that 77% is a rounded value or explain the discrepancy.
  3. [Table 1] The team name 'iLearn2.0' appears in Table 1 without introduction; please define it in the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the reported 77% accuracy is measured against the external EgoSchema blind test set, and no fitted parameter or self-citation is used to derive the result.

full rationale

The paper's central claim is an empirical benchmark result: HCQA-1.5 achieves 77% accuracy on the EgoSchema blind test set and ranks third in the CVPR 2025 challenge. This accuracy is evaluated externally against human-curated ground-truth answers, so it cannot be manufactured by construction. The method extends the authors' own HCQA framework, but HCQA is used as a baseline component whose outputs are fed into multi-source aggregation; the final numbers are not derived from any fitted parameter renamed as a prediction, and no equation in the paper reduces the target result to its inputs. The confidence-based routing (threshold 4 on a 1-5 scale) is an uncalibrated design assumption that could affect robustness, but that is a correctness/empirical-validity concern, not circular reasoning: the confidence scores are model outputs, not parameters fitted to the test set. Self-citations appear (HCQA [24], prior egocentric work) but none is load-bearing in the sense of justifying the 77% figure; the result stands on the external leaderboard and ablations. Therefore no circular step meeting the required quote-and-reduction standard can be identified.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central accuracy claim rests on the external benchmark, on the reliability of model confidence scores, and on the sufficiency of the extracted text and frames. The paper introduces no new entities. The main free parameters are the confidence threshold and the frame count, both chosen by hand.

free parameters (2)
  • confidence threshold = >4 on a 1-5 scale
    Used in Section 2.1 to decide which answers are accepted from the multi-source aggregation stage. No sensitivity analysis is provided, and the threshold is presented as a fixed design choice.
  • number of uniformly sampled frames = 45
    Used in Section 2.2 for the vision-based reasoning stage. Chosen without ablation or justification; different sampling rates could change which visual evidence the model sees.
assumptions (4)
  • domain assumption EgoSchema blind test labels are correct and the evaluation protocol is valid.
    The reported accuracy is a point estimate on this external benchmark; the paper does not independently verify labels.
  • domain assumption LLM-reported confidence scores from different models are comparable and meaningful.
    Stage 1 filtering in Section 2.1 assumes a score above 4 from any of the three models indicates a reliable answer; no calibration or cross-model normalization is described.
  • domain assumption The extracted captions and summaries from HCQA contain enough information for the reasoning stage.
    The thought-based reasoning stage in Section 2.2 uses only HCQA's text outputs, so errors in those summaries propagate to the final answer.
  • ad hoc to paper Uniform sampling of 45 frames is sufficient to capture all visual evidence needed to answer the question.
    The 45-frame choice in Section 2.2 is a paper-specific design decision with no supporting analysis; short but important moments could be missed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HCQA-1.5 @ Ego4D EgoSchema Challenge 2025." pith.science (2026). https://pith.science/paper/44DXYPYU

@misc{pith2026250520644,
  author       = {Pith},
  title        = {Pith review of: HCQA-1.5 @ Ego4D EgoSchema Challenge 2025},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/44DXYPYU}},
  note         = {Machine review of arXiv:2505.20644}
}
read the original abstract

In this report, we present the method that achieves third place for Ego4D EgoSchema Challenge in CVPR 2025. To improve the reliability of answer prediction in egocentric video question answering, we propose an effective extension to the previously proposed HCQA framework. Our approach introduces a multi-source aggregation strategy to generate diverse predictions, followed by a confidence-based filtering mechanism that selects high-confidence answers directly. For low-confidence cases, we incorporate a fine-grained reasoning module that performs additional visual and contextual analysis to refine the predictions. Evaluated on the EgoSchema blind test set, our method achieves 77% accuracy on over 5,000 human-curated multiple-choice questions, outperforming last year's winning solution and the majority of participating teams. Our code will be added at https://github.com/Hyu-Zhang/HCQA.

Figures

Figures reproduced from arXiv: 2505.20644 by the authors.

Figure 1
Figure 1. An illustration of two-stage decision-making process. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. One failed example of our framework on EgoSchema subset. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. OSGNet @ Ego4D Episodic Memory Challenge 2025

    cs.CV 2025-06 conditional novelty 4.0 of 10

    OSGNet, an early-fusion grounding model, wins all three Ego4D Episodic Memory Challenge tracks by converting localization tasks into retrieval problems.

  2. Technical Report for Ego4D Long-Term Action Anticipation Challenge 2025

    cs.CV 2025-06 conditional novelty 4.0 of 10

    A three-stage pipeline using the EgoVideo-V encoder, a verb-noun co-occurrence reranker, SAM2 hand-object features, and a fine-tuned Llama 2 model took first place in the Ego4D 2025 long-term action anticipation challenge.

Reference graph

Works this paper leans on

26 extracted references · 3 canonical work pages · cited by 2 Pith papers

  1. [1]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025. 2

    arXiv 2025

  2. [2]

    Zero-shot video question answering with pro- cedural programs

    Rohan Choudhury, Koichiro Niinuma, Kris M Kitani, and L´aszl´o A Jeni. Zero-shot video question answering with pro- cedural programs. arXiv preprint arXiv:2312.00937, 2023. 2

    arXiv 2023

  3. [3]

    Objectnlq@ ego4d episodic memory challenge 2024

    Yisen Feng, Haoyu Zhang, Yuquan Xie, Zaijing Li, Meng Liu, and Liqiang Nie. Objectnlq@ ego4d episodic memory challenge 2024. arXiv preprint arXiv:2406.15778, 2024. 1

    arXiv 2024

  4. [4]

    Object-shot enhanced grounding network for egocentric video

    Yisen Feng, Haoyu Zhang, Meng Liu, Weili Guan, and Liqiang Nie. Object-shot enhanced grounding network for egocentric video. arXiv preprint arXiv:2505.04270, 2025. 1

    arXiv 2025

  5. [5]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18995–19012, 2022. 1

    2022

  6. [6]

    Bi-directional het- erogeneous graph hashing towards efficient outfit recom- mendation

    Weili Guan, Xuemeng Song, Haoyu Zhang, Meng Liu, Chung-Hsing Yeh, and Xiaojun Chang. Bi-directional het- erogeneous graph hashing towards efficient outfit recom- mendation. In Proceedings of the 30th ACM international conference on multimedia, pages 268–276, 2022. 1

    2022

  7. [7]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. 2

    arXiv 2025

  8. [8]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perel- man, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Weli- hinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 2

    arXiv 2024

Show all 26 references
  1. [9]

    Egoschema: A diagnostic benchmark for very long- form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long- form video language understanding. Advances in Neural In- formation Processing Systems, 36:46212–46244, 2023. 1

    2023

  2. [10]

    Egoschema: A diagnostic benchmark for very long- form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long- form video language understanding. Advances in Neural In- formation Processing Systems, 36, 2024. 1

    2024

  3. [11]

    A simple recipe for contrastively pre-training video-first en- coders beyond 16 frames

    Pinelopi Papalampidi, Skanda Koppula, Shreya Pathak, Justin Chiu, Joe Heyward, Viorica Patraucean, Jiajun Shen, Antoine Miech, Andrew Zisserman, and Aida Nematzdeh. A simple recipe for contrastively pre-training video-first en- coders beyond 16 frames. arXiv preprint arXiv:2312.07395,

    arXiv

  4. [12]

    Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrit- twieser, et al. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv...

    2024 arXiv

  5. [13]

    Videoagent: Long-form video understand- ing with large language model as agent

    Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. Videoagent: Long-form video understand- ing with large language model as agent. arXiv preprint arXiv:2403.10517, 2024. 2

    2024 arXiv

  6. [14]

    Lifelongmem- ory: Leveraging llms for answering queries in egocentric videos

    Ying Wang, Yanlai Yang, and Mengye Ren. Lifelongmem- ory: Leveraging llms for answering queries in egocentric videos. arXiv preprint arXiv:2312.05269, 2023. 2

    2023 arXiv

  7. [15]

    Internvideo2: Scaling video foundation mod- els for multimodal video understanding

    Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, et al. Internvideo2: Scaling video foundation mod- els for multimodal video understanding. arXiv preprint arXiv:2403.15377, 2024. 2

    2024 arXiv

  8. [16]

    Time: Temporal-sensitive multi-dimensional instruc- tion tuning and benchmarking for video-llms

    Yunxiao Wang, Meng Liu, Rui Shao, Haoyu Zhang, Bin Wen, Fan Yang, Tingting Gao, Di Zhang, and Liqiang Nie. Time: Temporal-sensitive multi-dimensional instruc- tion tuning and benchmarking for video-llms. arXiv preprint arXiv:2503.09994, 2025. 1

    2025 arXiv

  9. [17]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. 2

    2025 arXiv

  10. [18]

    mplug-owl: Modularization empowers large language models with multimodality

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178, 2023. 2

    2023 arXiv

  11. [19]

    A sim- ple llm framework for long-range video question-answering

    Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu, Mohit Bansal, and Gedas Bertasius. A sim- ple llm framework for long-range video question-answering. arXiv preprint arXiv:2312.17235, 2023. 2

    2023 arXiv

  12. [20]

    Multimodal dialog system: Rela- tional graph-based context-aware question understanding

    Haoyu Zhang, Meng Liu, Zan Gao, Xiaoqiang Lei, Yinglong Wang, and Liqiang Nie. Multimodal dialog system: Rela- tional graph-based context-aware question understanding. In Proceedings of the 29th ACM international conference on multimedia, pages 695–703, 2021. 1

    2021

  13. [21]

    Attribute-guided collab- orative learning for partial person re-identification

    Haoyu Zhang, Meng Liu, Yuhong Li, Ming Yan, Zan Gao, Xiaojun Chang, and Liqiang Nie. Attribute-guided collab- orative learning for partial person re-identification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(12):14144–14160, 2023. 1

    2023

  14. [22]

    Uncovering hidden connections: Iterative tracking and reasoning for video-grounded dialog

    Haoyu Zhang, Meng Liu, Yaowei Wang, Da Cao, Weili Guan, and Liqiang Nie. Uncovering hidden connections: Iterative tracking and reasoning for video-grounded dialog. arXiv preprint arXiv:2310.07259, 2023. 1

    2023 arXiv

  15. [23]

    Multi-factor adaptive vision selec- tion for egocentric video question answering

    Haoyu Zhang, Meng Liu, Zixin Liu, Xuemeng Song, Yaowei Wang, and Liqiang Nie. Multi-factor adaptive vision selec- tion for egocentric video question answering. InProceedings of the 41st International Conference on Machine Learning , pages 59310–59328. PMLR, 2024. 1

    2024

  16. [24]

    Hcqa@ ego4d egoschema challenge

    Haoyu Zhang, Yuquan Xie, Yisen Feng, Zaijing Li, Meng Liu, and Liqiang Nie. Hcqa@ ego4d egoschema challenge

  17. [25]

    Exo2ego: Exocentric knowledge guided mllm for egocentric video understanding

    Haoyu Zhang, Qiaohui Chu, Meng Liu, Yunxiao Wang, Bin Wen, Fan Yang, Tingting Gao, Di Zhang, Yaowei Wang, and Liqiang Nie. Exo2ego: Exocentric knowledge guided mllm for egocentric video understanding. arXiv preprint arXiv:2503.09143, 2025. 1

    2025

  18. [2024]

    arXiv preprint arXiv:2406.15771, 2024. 1

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.