Pith. sign in

REVIEW 4 major objections 5 minor 29 references

Perception Test 2024: Challenge Summary and a Novel Hour-Long VideoQA Benchmark

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper introduces 1h-walk VQA, a benchmark for hour-long video question answering, and reports that even the best model is far below human accuracy.

desk verdict New hour-long video QA benchmark with a real human-model gap, but the long-context claim is unvalidated and the tables have errors. read the letter →

arxiv 2411.19941 v1 pith:IKDLYWCT submitted 2024-11-29 cs.CV cs.CLcs.LG

Joseph Heyward , João Carreira , Dima Damen , Andrew Zisserman , Viorica Pătrăucean This is my paper · ORCID
classification cs.CVcs.CLcs.LG
keywords perceptionevaluationhour-longvideounderstandingquestionansweringlongtemporalcontextmultimodalbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper’s central contribution is 1h-walk VQA, a small benchmark for hour-long video question answering built from ten un-narrated city-tour videos. It adds 70 manually curated five-way questions that are meant to require reasoning over video and audio across temporally separated moments, with many questions spanning more than thirty minutes. The motivation is that existing video-QA benchmarks can often be answered from a single frame or a short clip, so they do not test long-range temporal understanding. In the first challenge run on this benchmark, the best model scored 44.07% top-1 accuracy while a zero-shot human baseline scored 99.64%, leaving a wide gap to close.

What carries the argument

The load-bearing object is the 1h-walk VQA benchmark itself: ten hour-long, un-narrated city-walk videos paired with 70 five-way multiple-choice questions. The curation process is what makes it target long context: raters first proposed questions that needed one, two, or more separated video segments or audio, and a second annotation round used event detection to localise repeated events such as bridge crossings, stair climbs, and clock-tower views before writing questions around those timestamps. A final manual review kept only questions that cannot be answered from a single frame or a very short clip.

What would settle it

Give a capable video-QA model the same 1h-walk VQA questions under two conditions: once with the full hour-long video and once with only a short clip around the answer-relevant frames. If the short-clip condition matches or nearly matches full-video accuracy, or if full-video accuracy is no better than chance for human raters seeing only short clips, the benchmark is not measuring hour-long understanding.

Watch

Extended reading notes

Core claim

The paper claims that 1h-walk VQA measures hour-long understanding because its questions were selected through iterative annotation and manual review to require more than one video segment, or both video and audio, to answer. The benchmark contains 70 question-answer pairs over the ten Walking Tours videos, split into 3 validation videos with 11 questions and 7 test videos with 59 questions, and is deliberately zero-shot with no training data provided. The paper’s reported results show that the strongest submitted system, a sliding-window chain-of-thought method, still falls far short of human accuracy, which it takes as evidence that hour-long video reasoning remains an open problem.

Load-bearing premise

The benchmark’s value depends on the manual-review judgment that no selected question can be answered from a single frame or a short clip, and the paper provides no quantitative verification of that property.

Editorial extensions

If this is right

  • The best 2024 hour-long-VQA submission reached 44.07% versus 99.64% for human raters, so the benchmark leaves substantial headroom.
  • Because the benchmark is zero-shot by design, strong performance has to come from pretraining and prompting rather than fine-tuning on its 11 validation questions.
  • The same event-detector-then-question pipeline used to create 1h-walk VQA could be re-run on other long videos to build larger hour-long benchmarks.
  • Across all seven tracks, the 2024 winners improved over the 2023 winners, with grounded video QA showing the largest jump.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: the no-short-clip property is the one assumption most worth stress-testing; an input-length ablation could shrink the benchmark’s claimed difficulty if many questions are solvable locally.
  • My inference: the event-detection pipeline could be automated to scale beyond 70 curated questions, replacing expensive human annotation with detector-guided template questions.
  • My inference: with only ten source videos, per-city and per-walk idiosyncrasies may dominate results, so the benchmark’s numbers should be read as provisional evidence about hour-long understanding generally.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper reports on the Second Perception Test challenge, organized as a workshop at ECCV 2024, summarizing the setup, baselines, and results of seven tracks: object tracking, point tracking, temporal action localization, temporal sound localization, multiple-choice video QA, grounded video QA, and the newly introduced hour-long video QA track. The main novel contribution is 1h-walk VQA, a small benchmark built from the Walking Tours dataset, consisting of 70 manually curated five-way question-answer pairs over ten hour-long city tour videos, with a 3-video/11-question validation split and a 7-video/59-question test split. The paper describes the two-stage annotation procedure, reports human and random baselines, and presents the top two submissions per track, highlighting that the winning hour-long video QA system reaches 44% accuracy versus a human baseline of 99.64%.

Significance. The proposed benchmark addresses a real gap in video QA evaluation, since most existing datasets permit answers from very short clips or single frames; a public benchmark that genuinely requires hour-long temporal reasoning would be a valuable community resource. The challenge results provide a useful snapshot of progress across multiple video understanding tasks, including a new long-video track, and the paper makes its question list available in the appendix. However, the central claim that 1h-walk VQA measures hour-long temporal reasoning is not yet convincingly established, and the statistical and tabular inconsistencies reduce the reliability of the reported results. If the long-context property is later verified with a short-clip control and the reporting errors are corrected, the benchmark could serve as a seed for larger-scale efforts, as the discussion itself acknowledges.

major comments (4)
  1. [Section 2] The paper asserts that questions were manually reviewed and selected because they 'cannot be answered from a single frame or a very short clip,' but it provides no quantitative check of this property, no inter-annotator agreement, and no details of the selection protocol. The appendix contains questions that appear to violate the stated criterion: Q1 ('How many statue figures were there above the gate seen just before the DIESEL fashion store?') can be answered from a single frame, and Q41 ('When passing by Caffé Nero, what sound can be heard?') from a short audio clip. A short-clip control or an analysis showing the temporal spread of the evidence needed for each question is required to support the load-bearing claim that 1h-walk VQA measures hour-long understanding.
  2. [Section 4.7, Table 9] The human baseline of 99.64% is not consistent with the described annotation protocol. The text says each question was answered by 10 participants and each participant received 27 questions; with 70 questions this gives 700 total answers in the simplest reading, and an accuracy of 99.64% corresponds to about 0.36 errors, which is not an integer. The paper should report the exact total number of human answers and the number of errors, or revise the protocol description to match the reported accuracy.
  3. [Tables 2 and 9] There are clear reporting errors in the results tables. In Table 2, the 'Best' entry (NJUST-THU) is reported with IoU 0.734, which is lower than the 'Runner-up' entry (FAUgeddaboudit) with IoU 0.813; as ranked, the best should have the highest score. In Table 9, the metric column is labeled 'HOTA' although Section 4.7 explicitly defines the metric for this track as top-1 accuracy. These errors call into question the reliability of the reported challenge results and must be corrected.
  4. [Section 2] Even if a question is not answerable from a single frame or a very short clip, it does not follow that the question requires hour-long temporal reasoning. Several questions, such as Q13 (two appearances of a clocktower), Q15 (comparing two marketplaces at timestamps 33:20 and 51:00), and Q16 (time between two bridge crossings), can be answered by retrieving two localized short clips rather than integrating information across the entire hour. The paper should provide evidence—for example, an analysis of the temporal distance between relevant segments, or an evaluation comparing full-video access with access only to short windows—that the questions indeed require long-range video understanding.
minor comments (5)
  1. [Figure 2] The caption states 'Average video length in our newly-proposed 1h-walk VQA benchmark compared to existing benchmarks,' but the figure does not show axis labels or units; please add them so the comparison is interpretable.
  2. [Section 4.7] The description of the human baseline is under-specified: 'Each participant received 27 questions' does not make clear how the 70 questions were distributed across participants or how the total number of answers was derived; please provide the exact counts.
  3. [Table 1] The 'Train' row uses dashes for both columns; consider stating explicitly that no training data or training questions are provided, which is already implied by the zero-shot evaluation intent.
  4. [Section 2] The appendix lists the unique questions, but the paper does not provide a direct pointer to the actual benchmark files (e.g., question JSON, annotations, or a dataset URL); adding this would make the benchmark fully reproducible and usable by the community.
  5. [References] Some references have formatting issues, such as the trailing period in the URL of the 'A simple recipe' entry and inconsistent page ranges; please standardize according to the journal style.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: this is a benchmark/challenge report whose quantities are external measurements (human annotations, human baselines, leaderboard runs), not fitted inputs re-presented as predictions.

full rationale

This paper is a challenge summary and benchmark introduction, not a derivation chain: no quantity is fitted from a subset of data and then re-predicted, and no result is defined in terms of another result it claims to derive. The 1h-walk VQA benchmark consists of 70 human-curated 5-way QA pairs over Walking Tours videos, the human baseline (99.64%) is an independent measurement obtained from 10 participants per question, and the leaderboard accuracies are external model evaluations; none of these reduces by construction to an input of the paper. The paper cites the authors' own Perception Test benchmark (Pătrăucean et al., 2023) for the dataset, task definitions, and human baseline used in the other tracks, but that is a dataset reference with independent external content, not a self-citation that carries the argument or forbids alternatives. The Section 2 claim that questions were selected so that they 'cannot be answered from a single frame or a very short clip' is a benchmark-validity assertion about the annotation process, not a fitted parameter renamed as a prediction; even if that property were unverified or questionable, that would be a correctness/validity risk about whether the benchmark measures hour-long reasoning, not a circularity of the paper's derivation. Overall, there is no load-bearing circular step, and no self-citation chain is used to force a conclusion.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The benchmark rests on the assumption that the curated questions genuinely require hour-long temporal reasoning. No new physical or mathematical entities are introduced; the only 'invention' is the question set itself, which is a data artifact rather than a theoretical entity.

assumptions (2)
  • domain assumption Manual review selects questions that cannot be answered from a single frame or a very short clip.
    The paper states this in Section 2 but offers no quantitative verification; this is the key assumption making the benchmark measure long-context reasoning.
  • domain assumption The 1h-walk videos have no narration, preventing language shortcuts.
    Section 2 notes the videos are not narrated, which is assumed to ensure questions require watching the video rather than relying on a spoken commentary.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Perception Test 2024: Challenge Summary and a Novel Hour-Long VideoQA Benchmark." pith.science (2026). https://pith.science/paper/IKDLYWCT

@misc{pith2026241119941,
  author       = {Pith},
  title        = {Pith review of: Perception Test 2024: Challenge Summary and a Novel Hour-Long VideoQA Benchmark},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IKDLYWCT}},
  note         = {Machine review of arXiv:2411.19941}
}
read the original abstract

Following the successful 2023 edition, we organised the Second Perception Test challenge as a half-day workshop alongside the IEEE/CVF European Conference on Computer Vision (ECCV) 2024, with the goal of benchmarking state-of-the-art video models and measuring the progress since last year using the Perception Test benchmark. This year, the challenge had seven tracks (up from six last year) and covered low-level and high-level tasks, with language and non-language interfaces, across video, audio, and text modalities; the additional track covered hour-long video understanding and introduced a novel video QA benchmark 1h-walk VQA. Overall, the tasks in the different tracks were: object tracking, point tracking, temporal action localisation, temporal sound localisation, multiple-choice video question-answering, grounded video question-answering, and hour-long video question-answering. We summarise in this report the challenge tasks and results, and introduce in detail the novel hour-long video QA benchmark 1h-walk VQA.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 19 canonical work pages

  1. [1]

    Alayrac, A

    J.-B. Alayrac, A. Recasens, R. Schneider, R. Arandjelovi \'c , J. Ramapuram, J. De Fauw, L. Smaira, S. Dieleman, and A. Zisserman. Self-supervised multimodal versatile networks. Advances in Neural Information Processing Systems, 33: 0 25--37, 2020

    work page 2020

  2. [2]

    Alayrac, J

    J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, R. Ring, E. Rutherford, S. Cabi, T. Han, Z. Gong, S. Samangooei, M. Monteiro, J. Menick, S. Borgeaud, A. Brock, A. Nematzadeh, S. Sharifzadeh, M. Binkowski, R. Barreira, O. Vinyals, A. Zisserman, and K. Simonyan. Flamingo: a visual language model...

    work page 2022

  3. [3]

    Alwassel, S

    H. Alwassel, S. Giancola, and B. Ghanem. TSP : Temporally-sensitive pretraining of video encoders for localization tasks. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 3173--3183, 2021

    work page 2021

  4. [4]

    S. Chen, Y. Wu, C. Wang, S. Liu, D. Tompkins, Z. Chen, W. Che, X. Yu, and F. Wei. BEAT s: Audio pre-training with acoustic tokenizers. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 51...

    work page 2023

  5. [5]

    S. Cho, J. Huang, J. Nam, H. An, S. Kim, and J.-Y. Lee. Local all-pair correspondence for point tracking. In ECCV2024, 2024

    work page 2024

  6. [6]

    Damen, H

    D. Damen, H. Doughty, G. M. Farinella, A. Furnari, J. Ma, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, and M. Wray. Rescaling egocentric vision: Collection, pipeline and challenges for EPIC-KITCHENS -100. International Journal of Computer Vision (IJCV), 130: 0 33–55, 2022

    work page 2022

  7. [7]

    Doersch, A

    C. Doersch, A. Gupta, L. Markeeva, A. R. Continente, L. Smaira, Y. Aytar, J. Carreira, A. Zisserman, and Y. Yang. TAP -vid: A benchmark for tracking any point in a video. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL https://openreview.net/forum?id=Zmosb2KfzYd

    work page 2022

  8. [8]

    Y. Gong, A. Rouditchenko, A. H. Liu, D. Harwath, L. Karlinsky, H. Kuehne, and J. R. Glass. Contrastive audio-visual masked autoencoder. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=QPtMRyk5rb

    work page 2023

Show all 29 references
  1. [9]

    Kamath, M

    A. Kamath, M. Singh, Y. LeCun, I. Misra, G. Synnaeve, and N. Carion. Mdetr--modulated detection for end-to-end multi-modal understanding. arXiv preprint arXiv:2104.12763, 2021

    2021 arXiv

  2. [10]

    B. Li, Y. Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang, Y. Li, Z. Liu, and C. Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  3. [11]

    K. Li, Y. Wang, Y. Li, Y. Wang, Y. He, L. Wang, and Y. Qiao. Unmasked teacher: Towards training-efficient video foundation models, 2023

    2023

  4. [12]

    L. Lin, H. Fan, Z. Zhang, Y. Wang, Y. Xu, and H. Ling. Tracking meets lora: Faster training, larger model, stronger performance. In ECCV, 2024

    2024

  5. [13]

    S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, J. Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In ECCV, 2024

    2024

  6. [14]

    Y. Liu, S. Li, Y. Wu, C. W. Chen, Y. Shan, and X. Qie. Umt: Unified multi-modal transformers for joint video moment retrieval and highlight detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3042--3051, 2022

    2022

  7. [15]

    Luiten, A

    J. Luiten, A. Osep, P. Dendorfer, P. Torr, A. Geiger, L. Leal-Taix \'e , and B. Leibe. Hota: A higher order metric for evaluating multi-object tracking. International Journal of Computer Vision, pages 1--31, 2020

    2020

  8. [16]

    Minderer, A

    M. Minderer, A. A. Gritsenko, and N. Houlsby. Scaling open-vocabulary object detection. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=mQPNcBWjGc

    2023

  9. [17]

    Gpt-4v(ision) system card

    OpenAI. Gpt-4v(ision) system card. 2023. URL https://api.semanticscholar.org/CorpusID:263218031

    2023

  10. [18]

    Papalampidi, S

    P. Papalampidi, S. Koppula, S. Pathak, J. Chiu, J. Heyward, V. Patraucean, J. Shen, A. Miech, A. Zisserman, and A. Nematzdeh. A simple recipe for contrastively pre-training video-first encoders beyond 16 frames. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recogniti...

    2024

  11. [19]

    Pătrăucean, L

    V. Pătrăucean, L. Smaira, A. Gupta, A. R. Continente, L. Markeeva, D. Banarse, S. Koppula, J. Heyward, M. Malinowski, Y. Yang, C. Doersch, T. Matejovicova, Y. Sulsky, A. Miech, A. Frechette, H. Klimczak, R. Koster, J. Zhang, S. Winkler, Y. Aytar, S. Osindero, D. Damen, A. Ziss...

    2023

  12. [20]

    N. Ravi, V. Gabeur, Y.-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. R \"a dle, C. Rolland, L. Gustafson, E. Mintun, J. Pan, K. V. Alwala, N. Carion, C.-Y. Wu, R. Girshick, P. Doll \'a r, and C. Feichtenhofer. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:24...

    2024 arXiv

  13. [21]

    G. Team. Gemini: A family of highly capable multimodal models, 2024 a . URL https://arxiv.org/abs/2312.11805

    2024 arXiv

  14. [22]

    L. . Team. The llama 3 herd of models, 2024 b . URL https://arxiv.org/abs/2407.21783

    2024 arXiv

  15. [23]

    R. Team. Reka core, flash, and edge: A series of powerful multimodal language models, 2024 c . URL https://arxiv.org/abs/2404.12387

    2024 arXiv

  16. [24]

    Z. Tong, Y. Song, J. Wang, and L. Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35...

    2022

  17. [25]

    Venkataramanan, M

    S. Venkataramanan, M. N. Rizve, J. Carreira, Y. M. Asano, and Y. Avrithis. Is imagenet worth 1 video? learning strong image encoders from 1 long unlabelled video. In International Conference on Learning Representations, 2024

    2024

  18. [26]

    L. Wang, B. Huang, Z. Zhao, Z. Tong, Y. He, Y. Wang, Y. Wang, and Y. Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14549--14560, June 2023

    2023

  19. [27]

    P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, Y. Fan, K. Dang, M. Du, X. Ren, R. Men, D. Liu, C. Zhou, J. Zhou, and J. Lin. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  20. [28]

    S. Yu, J. Cho, P. Yadav, and M. Bansal. Self-chained image-language model for video localization and question answering. arXiv preprint arXiv:2305.06988, 2023

    2023 arXiv

  21. [29]

    Zhang, J

    C. Zhang, J. Wu, and Y. Li. Actionformer: Localizing moments of actions with transformers. In European Conference on Computer Vision, 2022

    2022

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.