REVIEW 4 major objections 7 minor 1 cited by
A Benchmark for Crime Surveillance Video Analysis with Large Models
T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read UCVL, the first MLLM benchmark for crime surveillance video, shows large models miss anomalies—and finetuning on its data largely fixes that.
desk verdict A useful first MLLM benchmark for crime-surveillance video analysis, but its reliability claim depends on LLM-generated ground truth and scorer agreement that the paper has not yet validated. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism carrying the argument is the benchmark construction and evaluation pipeline: UCF-Crime's crime labels are combined with UCF-Crime Annotation's segment-level human descriptions, and Qwen2-72B is prompted to produce a coherent video summary with start/end times and to generate five multiple-choice questions per video. From these, six QA types are posed to MLLMs in a unified format. Open-ended description answers are graded by GPT-4o using detailed scoring guidelines, while objective answers use pattern matching, IoU, and top-3 accuracy; a weighted total score combines all six tasks. This pipeline converts conventional anomaly tasks into a form MLLMs can answer directly, which is what makes the benchmark usable without task-specific adaptation.
What would settle it
Take a random sample of UCVL test videos, have independent human annotators answer the same six question types using the original UCA segment descriptions, and compare their answers to the Qwen2-72B-generated ground truth. If agreement is substantially below human-human agreement on the same videos, the benchmark's reliability claim is falsified.
Extended reading notes
Core claim
The central claim is that UCVL is the first MLLM benchmark designed specifically for multi-task anomaly analysis in crime surveillance video, and that evaluation results show MLLMs suffer from a cognitive blindness to anomalous events despite strong general capabilities. On the benchmark, the 7B LLaVA-OneVision model scores only 44.1 total, while the 26B InternVL2 reaches 58.7; after finetuning on UCVL's training split, LLaVA-UCVL-7B reaches 63.8, surpassing the 40B InternVL2 score of 57.8. The authors interpret the consistent trends across frame counts and the clear performance gap between general and finetuned models as demonstrating that the benchmark reliably measures anomaly perception and that the training data is of high quality.
Load-bearing premise
The ground truth answers that every UCVL score is measured against are produced by an AI model (Qwen2-72B) summarizing and expanding human annotations, with no human verification reported; if those summaries drift from what actually happens in a video, the entire benchmark's reliability claim falls apart.
Editorial extensions
If this is right
- Researchers get a standardized six-task benchmark to compare MLLMs on anomaly perception in surveillance video, with scores that are consistent across different frame-sampling counts.
- Model developers can use UCVL's training split to specialize MLLMs for security applications; one epoch of finetuning lifted a 7B model above a 40B model.
- The observed anomaly blindness gives a concrete target for training-data curation: filtering out NSFW-like content may be leaving models systematically blind to abnormal events.
- Finetuned models close part of the gap, but the 0.5B model overfits and loses general performance, suggesting dataset mixing with general instruction data is needed for small models.
Reading between the lines
- If the LLM-generated ground truth is reliable, the same summarization-plus-QA pipeline could be applied to other safety-critical video domains (e.g., traffic incidents, industrial accidents) to produce MLLM benchmarks cheaply from existing spatiotemporal annotations.
- The use of GPT-4o for open-ended scoring, without human correlation analysis in the paper, leaves open the question of whether score differences of a few points are meaningful; a human-judgment validation study would settle that.
- The low temporal grounding scores across most models suggest UCVL could double as a diagnostic for a specific weakness: MLLMs can recognize an anomaly but cannot place it in time, which is exactly what investigators need.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. UCVL is a proposed benchmark for evaluating multimodal large language models (MLLMs) on crime surveillance video analysis. It reuses UCF-Crime labels and UCF-Crime Annotation segment captions, reorganizes them with Qwen2-72B-generated video summaries, and derives six question-answer tasks: anomaly detection (TF), anomaly classification (AC), temporal grounding (TG), multiple-choice questions (MCQ), event description (ED), and anomaly description (AD). Scoring combines pattern matching for objective questions with GPT-4o-based evaluation for open-ended descriptions. The authors benchmark eight open-source MLLMs, report a weighted Total score, and fine-tune LLaVA-OneVision 0.5B and 7B on the UCVL training split, reporting substantial Total improvements, which they interpret as evidence of the benchmark's reliability and the training data's quality.
Significance. The paper addresses a genuine gap: existing video benchmarks do not focus on anomaly perception in surveillance footage, and MLLMs may be blindly insensitive to violent or criminal content. The six-task design is clear, the Total formula is explicit, the frame-ablation study is a reasonable sanity check, and the per-category heatmap is informative. The promise to release the dataset, evaluation code, and checkpoints is a strength for reproducibility. However, the benchmark's validity hinges on two unvalidated components: Qwen2-72B-generated summaries and MCQs used as ground truth, and GPT-4o scoring of open-ended responses. The paper gives qualitative assurance but no quantitative human agreement. If those components are independently validated, UCVL would be a useful contribution; without them, the reliability claim in the Abstract and the data-quality claim in Section IV-C are not yet supported.
major comments (4)
- [§III-B, §III-C, Abstract] The Abstract claims the results "demonstrate the reliability" of the benchmark, but the ground truth for ED, AD, and TG comes from Qwen2-72B-generated summaries, and the MCQs are generated from those same summaries. The only validation mentioned is a qualitative comparison "with human annotations" in Appendix A, and no quantitative results of that comparison appear in the submitted text. If a summary hallucinates an action, misassigns a timestamp, or contains a biased distractor, the error propagates directly into ED, AD, TG, and MCQ scores. Please provide numeric human validation: for example, event-level precision and recall between summaries and UCA segment annotations, timestamp IoU, and human verification of the correctness of MCQ answers and the plausibility of distractors.
- [§III-D, Abstract] Open-ended ED and AD responses are scored by GPT-4o, and the Abstract calls this "accurate assessment," but no human agreement study is reported. A benchmark's reliability claim requires at least a sample-based comparison between GPT-4o scores and human ratings, reported as correlation, pairwise agreement, or an equivalent measure. Without this, the ED and AD scores, which contribute 0.30 to the Total, rest on an unverified judge. Please add such a validation and specify the exact evaluation prompt, temperature, and number of trials used.
- [§IV-C and Table IV] The fine-tuning results are interpreted as validating "our data's high quality," but the training text (Qwen2-72B summaries and MCQs), the evaluation references (the same summaries), and the judge for open-ended answers (GPT-4o) are all LLM-generated. The reported Total gain from 44.1 to 63.8 for LLaVA-UCVL-7B could therefore partly reflect the model learning to imitate Qwen2's phrasing, which GPT-4o may preferentially score, rather than an improved factual understanding of anomalies. To support the data-quality conclusion, please add an independent evaluation, such as human scoring of a sample of open-ended answers, evaluation on a hold-out set with human-authored references, or a control fine-tuning experiment on generic instruction data of the same size.
- [Abstract, §III-A, Tables II and III] The Abstract states that UCVL includes 1,829 videos, while Section III-A and Table II state 1,699 videos, and the split in Table III (1030 + 369 + 300) also sums to 1,699. This is an inconsistency that must be corrected. It also affects the description of the test set: if the released dataset actually contains 1,829 videos, the evaluation procedure on 300 test videos needs to be stated precisely so that results are reproducible.
minor comments (7)
- [Abstract] The phrase "the reliability of this bench" should read "the reliability of this benchmark."
- [Table IV] The header contains "ED, ED, AD" with a duplicated "ED"; one of the two ED entries should be removed or replaced with the intended task name.
- [Throughout] The text frequently writes "LLaV A" with a space; it should be "LLaVA" for consistency with the cited model name.
- [§III-A and §III-B] Section III-A says Qwen2-72B and GPT-4o are used to summarize descriptions and generate QAs, while Section III-B states that Qwen2-72B performs the summarization via few-shot prompting. Please clarify the role of GPT-4o in the generation pipeline.
- [Fig. 2 caption] The caption says "evaluate ten models," but the Abstract says "eight prevailing MLLMs"; the two additional fine-tuned models should be explicitly identified to avoid confusion.
- [§IV-B and Fig. 3] The per-category results in the heatmap likely have small sample sizes per cell; please report the number of test videos per category and add confidence intervals or significance tests before drawing conclusions about which crime types are easier or harder.
- [Eq. (1)] The weights in the Total formula are stated without justification; please provide a rationale for the chosen weights or a sensitivity analysis showing the main conclusions are not driven by the exact values.
Circularity Check
UCVL's reliability and data-quality claims are partly self-referential because the open-ended and MCQ ground truths are Qwen2-72B outputs scored by GPT-4o, while the independent UCC/UCA anchors keep the benchmark from being definitionally void.
-
self definitional
[Section III-B (Video Summarization), Section III-C (Question-Answer Generation), Section III-D (Evaluation Metrics)]
"We employ Qwen2-72B to perform the process via few-shot prompting. ... The summaries serve as the ground truth for both questions, which are concise and meaningful for depicting the anomaly event. ... We use GPT-4o for comparing open-ended text responses."
The ground truth for ED and AD is Qwen2-72B's summary of UCA annotations, and MCQs are generated from those same summaries; the correct answers and timestamps are therefore LLM outputs rather than independently verified facts. GPT-4o then scores a model's open-ended response against this same LLM-generated summary. A high score can mean agreement with Qwen2-72B's paraphrase, not accurate perception of the video. Hence the Abstract's claim that 'the results demonstrate the reliability of this bench' rests on a reference set that was defined by the generator itself; human validation is only asserted in Appendix A and not quantified in the available text.
-
other
[Abstract and Section IV-C (Finetuning Results)]
"Moreover, we finetune LLaVA-OneVision on UCVL's training set. The improvement validates our data's high quality for video anomaly analysis. ... The 7B model achieves substantial improvements across all tasks, outperforming InternVL2-40B."
LLaVA-UCVL is trained on UCVL's Qwen2-generated training QAs and evaluated on UCVL's test split, whose ED/AD references are Qwen2-generated summaries and whose MCQs are Qwen2-generated questions with LLM-chosen correct options. The reported gain (Total 63.8 vs 44.1) can therefore reflect learning to imitate Qwen2's summary style and distractor preferences, which GPT-4o may reward, rather than improved factual anomaly understanding. Using that gain to 'validate data quality' is circular validation: the train and test targets are produced by the same LLM pipeline, so the benchmark cannot independently certify the facts in its own ground truth.
full rationale
UCVL is not entirely circular: TF, AC, and TG draw on UCC crime labels and UCA human segment timestamps, so those tasks have external anchors, and the benchmark construction from existing datasets is a legitimate contribution. No load-bearing self-citation was found; the same-author citation [15] only motivates the data-filtering explanation and is not used to prove benchmark validity. However, the reliability and data-quality conclusions are partly self-referential because the open-ended and MCQ ground truths are Qwen2-72B outputs and the open-ended judge is GPT-4o. The paper's own Appendix A human comparison is referenced but not included or quantified, so the loop is not broken in the available text. The fine-tuning improvement is evaluated on that same LLM-generated test set, so the 'high quality' claim is not independently confirmed. These two constructed-loop steps justify a moderate circularity score rather than zero; the independent UCC/UCA anchors keep the benchmark from being definitionally void.
Assumptions & free parameters
free parameters (3)
- Total score weights =
w_TF=0.15, w_AC=0.10, w_ED=0.15, w_AD=0.15, w_TG=0.20, w_MCQ=0.25
- Uniform frame sampling count =
32 for main results; 8, 16, 64 in ablation
- Fine-tuning hyperparameters =
LR 3e-6 and 1e-6, batch sizes 32 and 16, 1 epoch
assumptions (4)
- domain assumption UCF-Crime labels and UCF-Crime Annotation segment descriptions are correct and complete.
- ad hoc to paper Qwen2-72B-generated summaries and MCQs are sufficiently accurate to serve as ground truth.
- ad hoc to paper GPT-4o open-ended scoring reflects human semantic judgments.
- domain assumption Evaluated MLLMs have not memorized UCF-Crime or UCVL content during pretraining.
Cite this review
Pith. "Pith review of A Benchmark for Crime Surveillance Video Analysis with Large Models." pith.science (2026). https://pith.science/paper/CSPRI7WP
@misc{pith2026250209325,
author = {Pith},
title = {Pith review of: A Benchmark for Crime Surveillance Video Analysis with Large Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/CSPRI7WP}},
note = {Machine review of arXiv:2502.09325}
}
read the original abstract
Anomaly analysis in surveillance videos is a crucial topic in computer vision. In recent years, multimodal large language models (MLLMs) have outperformed task-specific models in various domains. Although MLLMs are particularly versatile, their abilities to understand anomalous concepts and details are insufficiently studied because of the outdated benchmarks of this field not providing MLLM-style QAs and efficient algorithms to assess the model's open-ended text responses. To fill this gap, we propose a benchmark for crime surveillance video analysis with large models denoted as UCVL, including 1,829 videos and reorganized annotations from the UCF-Crime and UCF-Crime Annotation datasets. We design six types of questions and generate diverse QA pairs. Then we develop detailed instructions and use OpenAI's GPT-4o for accurate assessment. We benchmark eight prevailing MLLMs ranging from 0.5B to 40B parameters, and the results demonstrate the reliability of this bench. Moreover, we finetune LLaVA-OneVision on UCVL's training set. The improvement validates our data's high quality for video anomaly analysis.
Figures
Forward citations
Cited by 1 Pith paper
-
The Evolution of Video Anomaly Detection: A Unified Framework from DNN to MLLM
The paper organizes VAD methods into a five-dimension framework spanning task objective, modality, input, architecture, and optimization, with emphasis on MLLM/LLM-era work.
Reference graph
Works this paper leans on
-
[1]
Real-world anomaly detection in surveillance videos,
Waqas Sultani, Chen Chen, and Mubarak Shah, “Real-world anomaly detection in surveillance videos,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2018
work page 2018
-
[2]
End-to-end dense video captioning with parallel decoding,
Teng Wang, Ruimao Zhang, Zhichao Lu, Feng Zheng, Ran Cheng, and Ping Luo, “End-to-end dense video captioning with parallel decoding,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 6847–6857
work page 2021
-
[3]
Batchnorm-based weakly supervised video anomaly detection,
Yixuan Zhou, Yi Qu, Xing Xu, Fumin Shen, Jingkuan Song, and Heng Tao Shen, “Batchnorm-based weakly supervised video anomaly detection,” IEEE Transactions on Circuits and Systems for Video Technology, 2024
work page 2024
-
[4]
Negative sample matters: A renaissance of metric learning for temporal grounding,
Zhenzhi Wang, Limin Wang, Tao Wu, Tianhao Li, and Gangshan Wu, “Negative sample matters: A renaissance of metric learning for temporal grounding,” in Proceedings of the AAAI Conference on Artificial Intelligence, 2022, vol. 36, pp. 2613–2623
work page 2022
-
[5]
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al., “Qwen technical report,” arXiv preprint arXiv:2309.16609 , 2023
arXiv 2023
-
[6]
Llava- onevision: Easy visual task transfer,
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al., “Llava- onevision: Easy visual task transfer,” arXiv preprint arXiv:2408.03326 , 2024
arXiv 2024
-
[7]
Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks,
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al., “Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 24185–24198
work page 2024
-
[8]
Qwen2- vl: Enhancing vision-language model’s perception of the world at any resolution,
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al., “Qwen2- vl: Enhancing vision-language model’s perception of the world at any resolution,” arXiv preprint arXiv:2409.12191 , 2024
arXiv 2024
Show all 23 references
-
[9]
Learning transferable visual models from natural language supervision,
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever, “Learning transferable visual models from natural language supervision,” in International C...
2021
-
[10]
Mm-vet: Evaluating large multimodal models for integrated capabilities,
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang, “Mm-vet: Evaluating large multimodal models for integrated capabilities,” arXiv preprint arXiv:2308.02490, 2023
2023 arXiv
-
[11]
Mmbench: Is your multi-modal model an all-around player?,
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, and Songyang Zhang, “Mmbench: Is your multi-modal model an all-around player?,” arXiv:2307.06281, 2023
2023 arXiv
-
[12]
Mmbench-video: A long-form multi- shot benchmark for holistic video understanding,
Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen, “Mmbench-video: A long-form multi- shot benchmark for holistic video understanding,” arXiv preprint arXiv:2406.14515, 2024
2024 arXiv
-
[13]
Mvbench: A comprehen- sive multi-modal video understanding benchmark,
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al., “Mvbench: A comprehen- sive multi-modal video understanding benchmark,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp....
2024
-
[14]
Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis,
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Ren- rui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al., “Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis,” arXiv preprint arXiv:2405.21075, 2024
2024 arXiv
-
[15]
Taisu: A 166m large-scale high-quality dataset for chinese vision-language pre- training,
Yulong Liu, Guibo Zhu, Bin Zhu, Qi Song, Guojing Ge, Haoran Chen, GuanHui Qiao, Ru Peng, Lingxiang Wu, and Jinqiao Wang, “Taisu: A 166m large-scale high-quality dataset for chinese vision-language pre- training,” Advances in Neural Information Processing Systems , vol. 35, pp....
2022
-
[16]
Not only look, but also listen: Learning multimodal violence detection under weak supervision,
Peng Wu, Jing Liu, Yujia Shi, Yujia Sun, Fangtao Shao, Zhaoyang Wu, and Zhiwei Yang, “Not only look, but also listen: Learning multimodal violence detection under weak supervision,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Procee...
2020
-
[17]
Towards surveillance video-and-language understanding: New dataset baselines and challenges,
Tongtong Yuan, Xuange Zhang, Kun Liu, Bo Liu, Chen Chen, Jian Jin, and Zhenzhen Jiao, “Towards surveillance video-and-language understanding: New dataset baselines and challenges,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. ...
2024
-
[18]
Gpt-4 technical report,
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023
2023 arXiv
-
[19]
Quo vadis, action recognition? a new model and the kinetics dataset,
Joao Carreira and Andrew Zisserman, “Quo vadis, action recognition? a new model and the kinetics dataset,” in proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2017, pp. 6299–6308
2017
-
[20]
Learning spatiotemporal features with 3d convolu- tional networks,
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri, “Learning spatiotemporal features with 3d convolu- tional networks,” in Proceedings of the IEEE international conference on computer vision , 2015, pp. 4489–4497
2015
-
[21]
Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,
V Sanh, “Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,” arXiv preprint arXiv:1910.01108 , 2019
1910 arXiv
-
[22]
Bleu: a method for automatic evaluation of machine translation,
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceed- ings of the 40th annual meeting of the Association for Computational Linguistics, 2002, pp. 311–318
2002
-
[23]
Making the v in vqa matter: Elevating the role of image understanding in visual question answering,
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh, “Making the v in vqa matter: Elevating the role of image understanding in visual question answering,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 6904– 6913
2017
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.