REVIEW 4 major objections 4 minor 38 references
ScaleLong: A Multi-Timescale Benchmark for Long Video Understanding
T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read ScaleLong embeds clip, shot, event, and story questions in the same videos and reports that 23 multimodal models score higher at the two ends of the timescale range than in the middle.
desk verdict ScaleLong is a genuinely useful benchmark resource, but the U-shaped accuracy finding is not yet established as a temporal-processing effect; the paper needs per-scale difficulty controls and more careful reporting. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the intra-video multi-timescale question design: each video receives 4–8 questions spanning four defined temporal levels, with clip covering a few seconds, shot covering 4–15 seconds within one continuous shot, event covering 16 seconds to 10 minutes across shots, and story covering more than 10 minutes of whole-video narrative. Because questions at all four levels share the same video content, any accuracy gap between levels is attributed to temporal granularity rather than to content differences. A second supporting mechanism is the token-budget ablation, which holds total visual tokens constant while varying the split between frame count and resolution, isolating temporal versus spatial contributions to each timescale.
What would settle it
A reader could re-annotate a sample of ScaleLong videos and compare per-scale accuracy of human annotators or a strong reference model; if the same U-shape appears for humans, the dip is a property of the questions rather than of MLLM temporal processing.
Extended reading notes
Core claim
The paper introduces ScaleLong, claiming it is the first benchmark whose questions are embedded at four hierarchical temporal scales within the same video, so that timescale ability is measured against identical content rather than being confounded by different videos. On 269 videos averaging 86 minutes, each annotated with 4–8 questions (at least one per scale), the paper evaluates 23 multimodal models and finds a consistent U-shaped performance curve: accuracy is higher at the clip and story levels and lower at the shot and event levels. The paper also claims that increased visual token capacity consistently improves reasoning across all timescales, and that under a fixed token budget a balanced allocation between frame count and resolution works best.
Load-bearing premise
The load-bearing premise is that the paper's hierarchy labels are correct and that questions at different timescales are equally hard except for the timescale they test; the paper does not calibrate difficulty across scales or report annotator agreement on labels.
Editorial extensions
If this is right
- Model developers can use ScaleLong to detect where a video model loses temporal coherence, and improvements aimed at shot- and event-level reasoning should raise overall benchmark scores.
- The U-shaped pattern appears across all 23 open- and closed-source models tested, so it is not an artifact of a single architecture or training recipe.
- Under a fixed visual-token budget, the ablation results imply that neither extreme frame count nor extreme resolution is optimal; balanced allocations such as 32×240p for clip-level tasks and 16×360p for story-level tasks perform best.
- Counting problems are the weakest task type across nearly all models, indicating that quantitative reasoning over long videos is a separate bottleneck from timescale handling.
- Error analysis shows models frequently accept missing-information and spatial-replacement distractors, pointing to insensitivity to evidential completeness and spatial layout in long-video contexts.
Reading between the lines
- The paper does not report inter-annotator agreement on the four hierarchy labels or per-scale difficulty calibration, so a plausible extension is to re-annotate a sample and test whether the U-shape survives when question difficulty is balanced across scales.
- A testable follow-up would use ScaleLong's within-content design to train or fine-tune models specifically on mid-range temporal reasoning and then measure whether story-level performance improves, which would suggest transfer along the temporal hierarchy.
- The token-budget results leave open the question of whether even finer-grained sampling schedules, such as more frames per second at scene boundaries, could push shot- and event-level accuracy closer to the clip and story ends of the curve.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces ScaleLong, a long-video QA benchmark containing 269 videos (average 86 minutes) and 1,747 manually annotated questions, with each video containing questions at four hierarchical timescales (clip, shot, event, story) and five task types. The authors evaluate 23 multimodal LLMs and report a U-shaped accuracy curve across the four timescales, present ablations on frame count, resolution, and token allocation, and analyze error rates by distractor type. The central claim is that the within-content, multi-timescale design disentangles temporal-granularity ability from content variability, enabling a direct comparison of model performance across timescales on identical videos.
Significance. If the validity concerns are addressed, ScaleLong would be a useful contribution: the intra-video multi-timescale design is a genuine novelty relative to benchmarks that scatter scale-specific questions across different videos, and the manual annotation pipeline with two-round quality control and ten distractor types is a strength. The public release of code and data, together with evaluation of 23 models, makes the benchmark immediately usable. However, the main empirical finding - the U-shaped performance curve - and the interpretation that it reflects temporal processing ability rather than question-design confounds are not yet established at the manuscript's current level of evidence.
major comments (4)
- [2.2, 2.4, A.2] The central claim that ScaleLong provides a disentangled measure of timescale ability requires that the four timescale question sets differ only in temporal granularity, not in intrinsic difficulty. The manuscript does not establish this. Section 2.4.2 states that hierarchy labels are validated but reports no inter-annotator agreement, and Appendix A.2 instructs annotators to maintain balanced task types without reporting the resulting per-timescale distribution of task types or distractor types. Since Table 2 shows large task-type accuracy gaps (e.g., GPT-4o: Object Recognition 62.6 vs. Counting Problems 36.0), a disproportionate share of harder task types at Shot or Event would produce the observed dip as a question-design artifact. Please report per-timescale distributions of task types and distractor types, provide human accuracy or a difficulty-calibration analysis per scale, and ideally include a matched-pair analysis that controls for task type.
- [Table 2, Section 4.2] The statement that the U-shaped trend 'holds consistently across all open-source and closed-source models' is contradicted by the data in Table 2. LLaVA-Mini has its lowest accuracy at Story (25.2) rather than at an intermediate scale, and LLaVA-Video-72B has Event accuracy (53.2) above Story (51.8). Qwen2.5-VL-7B shows a nearly monotonic decline from Clip to Event (52.8/50.0/49.5) with only a small Story rebound (52.7). The conclusion should be reformulated as a central tendency or a statistical claim (e.g., the proportion of models exhibiting the pattern, or a formal test of quadratic trend) rather than as a universal property of all evaluated models.
- [4.2, 4.3, Figure 3] No uncertainty quantification is provided for any reported accuracy. With 1,747 QA pairs split across four scales and many per-scale differences of only a few points (e.g., InternVL2.5-26B: 60.2/50.1/48.5/56.8), binomial noise could change the shape of the curve; bootstrap confidence intervals or a paired significance test (e.g., McNemar) is needed before the U-shape is asserted. In addition, Figure 3(a) shows Event accuracy peaking at 64 frames and declining at 128 frames, and Figure 3(b) shows Clip accuracy dipping from 360p to 480p, so the abstract's claim that 'increased visual token capacity consistently enhances reasoning across all timescales' is stronger than the plotted data support.
- [Table 1, Section 2.1] The paper never states the exact number of questions per timescale. The design description says 4-8 questions per video with at least one per scale, and Table 1 reports 1,747 total QA pairs, but the distribution across Clip, Shot, Event, and Story is not given. An unbalanced distribution would affect both the aggregate accuracy values and the interpretation of the U-shape, so the per-scale question counts should be reported explicitly.
minor comments (4)
- [Section 1, Appendix A, Appendix B] There are several typos: 'at last one per scale' should be 'at least one per scale' in Section 1; the Appendix A heading reads 'Annotation T utorial'; Appendix B begins with 'LoneVideoBenchmark' instead of 'ScaleLong'; and Section B.2 contains 'accurity' instead of 'accuracy'.
- [Figure 1] The caption of Figure 1 refers to 'LongVideoBench' rather than ScaleLong; please verify whether the figure content and caption are consistent with the correct benchmark.
- [Table 2] The task-type column is labeled 'II' in Table 2, but the task is defined as 'Information Summary (IS)' in Section 2.3; the label should be corrected for consistency.
- [Section 4.3.2] The token-allocation ablation reports 'the average accuracy of three models' without showing per-model results; including a model-level breakdown would make the finding more interpretable.
Circularity Check
No circularity: ScaleLong reports external MLLM measurements from a benchmark, with no fitted-parameter predictions and no self-referential derivation chain.
full rationale
This is an empirical benchmark paper, not a derivation. The central claim is that 23 MLLMs exhibit a U-shaped accuracy curve across four annotated timescales (Section 4.2, Table 2). That finding is a measurement of external models on held-out questions, not an output of the benchmark's own construction. The timescale labels defined in Section 2.2 are used both to annotate questions and to report per-scale accuracies, but this is standard benchmark design: the definition of the measurement axes does not determine the measured accuracies, which come from the evaluated MLLMs. No parameter is fitted to a subset of data and then renamed a prediction, and no cited prior work by the same authors is invoked as load-bearing justification. The paper does not contain equations that reduce a claimed result to its inputs. The strongest caveat is that the U-shape may be confounded by per-scale question difficulty or task-type mix (Section 2.3 and Table 2 show large task-type gaps, and Section 2.4 reports no inter-annotator agreement or difficulty calibration); that is a validity threat, not circularity. Accordingly, the appropriate finding is no significant circularity, score 0.
Assumptions & free parameters
assumptions (3)
- domain assumption The four timescale classes (clip, shot, event, story) are well-defined and annotators can reliably assign them.
- domain assumption The quality control ensures questions cannot be answered from commonsense or external knowledge alone.
- domain assumption The selected 269 videos are representative of long-video content.
Cite this review
Pith. "Pith review of ScaleLong: A Multi-Timescale Benchmark for Long Video Understanding." pith.science (2026). https://pith.science/paper/LQBMYWVH
@misc{pith2026250523922,
author = {Pith},
title = {Pith review of: ScaleLong: A Multi-Timescale Benchmark for Long Video Understanding},
year = {2026},
howpublished = {\url{https://pith.science/paper/LQBMYWVH}},
note = {Machine review of arXiv:2505.23922}
}
read the original abstract
Although long-video understanding demands that models capture hierarchical temporal information -- from clip (seconds) and shot (tens of seconds) to event (minutes) and story (hours) -- existing benchmarks either neglect this multi-scale design or scatter scale-specific questions across different videos, preventing direct comparison of model performance across timescales on the same content. To address this, we introduce ScaleLong, the first benchmark to disentangle these factors by embedding questions targeting four hierarchical timescales -- clip (seconds), shot (tens of seconds), event (minutes), and story (hours) -- all within the same video content. This within-content multi-timescale questioning design enables direct comparison of model performance across timescales on identical videos. ScaleLong features 269 long videos (avg.\ 86\,min) from 5 main categories and 36 sub-categories, with 4--8 carefully designed questions, including at least one question for each timescale. Evaluating 23 MLLMs reveals a U-shaped performance curve, with higher accuracy at the shortest and longest timescales and a dip at intermediate levels. Furthermore, ablation studies show that increased visual token capacity consistently enhances reasoning across all timescales. ScaleLong offers a fine-grained, multi-timescale benchmark for advancing MLLM capabilities in long-video understanding. The code and dataset are available https://github.com/multimodal-art-projection/ScaleLong.
Figures
Reference graph
Works this paper leans on
-
[1]
S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025
arXiv 2025
-
[2]
K. Chandrasegaran, A. Gupta, L. M. Hadzic, T. Kota, J. He, C. Eyzaguirre, Z. Durante, M. Li, J. Wu, and F.-F. Li. Hourvideo: 1-hour video-language understanding. Advances in Neural Information Processing Systems, 37: 0 53168--53197, 2024
work page 2024
-
[3]
Z. Chen, W. Wang, Y. Cao, Y. Liu, Z. Gao, E. Cui, J. Zhu, S. Ye, H. Tian, Z. Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024 a
arXiv 2024
-
[4]
Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 24185--24198, 2024 b
work page 2024
- [5]
- [6]
-
[7]
C. Fu, Y. Dai, Y. Luo, L. Li, S. Ren, R. Zhang, Z. Wang, C. Zhou, Y. Shen, M. Zhang, P. Chen, Y. Li, S. Lin, S. Zhao, K. Li, T. Xu, X. Zheng, E. Chen, R. Ji, and X. Sun. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis, 2024 a
work page 2024
-
[8]
C. Fu, Y. Dai, Y. Luo, L. Li, S. Ren, R. Zhang, Z. Wang, C. Zhou, Y. Shen, M. Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024 b
arXiv 2024
Show all 38 references
-
[9]
J. Guo, T. Zheng, Y. Bai, B. Li, Y. Wang, K. Zhu, Y. Li, G. Neubig, W. Chen, and X. Yue. Mammoth-vl: Eliciting multimodal reasoning with instruction tuning at scale. arXiv preprint arXiv:2412.05237, 2024
2024 arXiv
-
[10]
B. Li, Y. Zhang, L. Chen, J. Wang, J. Yang, and Z. Liu. Otter: A multi-modal model with in-context instruction tuning, 2023. URL https://arxiv.org/abs/2305.03726
2023 arXiv
-
[11]
B. Li, Y. Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang, Y. Li, Z. Liu, and C. Li. Llava-onevision: Easy visual task transfer, 2024 a
2024
-
[12]
K. Li, Y. Wang, Y. He, Y. Li, Y. Wang, Y. Liu, Z. Wang, J. Xu, G. Chen, P. Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195--22206, 2024 b
2024
-
[13]
Y. Li, X. Chen, B. Hu, L. Wang, H. Shi, and M. Zhang. Videovista: A versatile benchmark for video understanding and reasoning. arXiv preprint arXiv:2406.11303, 2024 c
2024 arXiv
-
[14]
Y. Li, X. Chen, B. Hu, L. Wang, H. Shi, and M. Zhang. Videovista: A versatile benchmark for video understanding and reasoning, 2024 d
2024
-
[15]
Y. Li, C. Wang, and J. Jia. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pages 323--340. Springer, 2024 e
2024
-
[16]
H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. Advances in neural information processing systems, 36: 0 34892--34916, 2023 a
2023
-
[17]
H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning, 2023 b . URL https://arxiv.org/abs/2304.08485
2023 arXiv
-
[18]
D. Ma, Y. Zhang, J. Ren, J. Guo, Y. Yao, Z. Wei, Z. Yang, Z. Peng, B. Feng, J. Ma, et al. Iv-bench: A benchmark for image-grounded video perception and reasoning in multimodal llms. arXiv preprint arXiv:2504.15415, 2025
2025 arXiv
-
[19]
Mangalam, R
K. Mangalam, R. Akshulakov, and J. Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems, 36: 0 46212--46244, 2023
2023
-
[20]
Gpt-4o, 2024
OpenAi . Gpt-4o, 2024
2024
-
[21]
Rawal, K
R. Rawal, K. Saifullah, M. Farr \'e , R. Basri, D. Jacobs, G. Somepalli, and T. Goldstein. Cinepile: A long video question answering dataset and benchmark. arXiv preprint arXiv:2405.08813, 2024
2024 arXiv
-
[22]
X. Tan, Y. Luo, Y. Ye, F. Liu, and Z. Cai. Allvb: All-in-one long video understanding benchmark. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 7211--7219, 2025
2025
-
[23]
G. Team, P. Georgiev, V. I. Lei, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan, S. Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024
2024 arXiv
-
[24]
K. Team, A. Du, B. Yin, B. Xing, B. Qu, B. Wang, C. Chen, C. Zhang, C. Du, C. Wei, et al. Kimi-vl technical report. arXiv preprint arXiv:2504.07491, 2025
2025 arXiv
-
[25]
W. Wang, Z. He, W. Hong, Y. Cheng, X. Zhang, J. Qi, X. Gu, S. Huang, B. Xu, Y. Dong, et al. Lvbench: An extreme long video understanding benchmark. arXiv preprint arXiv:2406.08035, 2024
2024 arXiv
- [26]
- [27]
-
[28]
H. Wu, D. Li, B. Chen, and J. Li. Longvideobench: A benchmark for long-context interleaved video-language understanding, 2024. URL https://arxiv.org/abs/2407.15754
2024 arXiv
-
[29]
J. Xiao, X. Shang, A. Yao, and T.-S. Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9777--9786, 2021
2021
-
[30]
J. Ye, H. Xu, H. Liu, A. Hu, M. Yan, Q. Qian, J. Zhang, F. Huang, and J. Zhou. mplug-owl3: Towards long image-sequence understanding in multi-modal large language models. arXiv preprint arXiv:2408.04840, 2024
2024 arXiv
-
[31]
Q. Ye, H. Xu, G. Xu, J. Ye, M. Yan, Y. Zhou, J. Wang, A. Hu, P. Shi, Y. Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178, 2023
2023 arXiv
-
[32]
Zhang, X
H. Zhang, X. Li, and L. Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858, 2023 a
2023 arXiv
-
[33]
Zhang, Y
H. Zhang, Y. Liu, L. Dong, Y. Huang, Z.-H. Ling, Y. Wang, L. Wang, and Y. Qiao. Movqa: A benchmark of versatile question-answering for long-form movie understanding. arXiv preprint arXiv:2312.04817, 2023 b
2023 arXiv
-
[34]
Zhang, Y
H. Zhang, Y. Wang, Y. Tang, Y. Liu, J. Feng, J. Dai, and X. Jin. Flash-vstream: Memory-based real-time understanding for long video streams. arXiv preprint arXiv:2406.08085, 2024
2024 arXiv
-
[35]
J. Zhao, B. Sun, X. Chen, X. Wei, and Q. Hou. Llava-octopus: Unlocking instruction-driven adaptive projector fusion for video understanding. arXiv preprint arXiv:2501.05067, 2025
2025 arXiv
-
[36]
J. Zhou, Y. Shu, B. Zhao, B. Wu, S. Xiao, X. Yang, Y. Xiong, B. Zhang, T. Huang, and Z. Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04264, 2024
2024 arXiv
-
[37]
J. Zhu, W. Wang, Z. Chen, Z. Liu, S. Ye, L. Gu, Y. Duan, H. Tian, W. Su, J. Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479, 2025
2025 arXiv
-
[38]
H. Zou, T. Luo, G. Xie, F. Lv, G. Wang, J. Chen, Z. Wang, H. Zhang, H. Zhang, et al. Hlv-1k: A large-scale hour-long video benchmark for time-specific long video understanding. arXiv preprint arXiv:2501.01645, 2025
2025 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.