REVIEW 3 major objections 5 minor 45 references
Efficient Vision Language Model Fine-tuning for Text-based Person Anomaly Search
T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read The paper claims that a test-time similarity re-ranking step, SCA, raises X-VLM's Recall@1 from 77.86 to 80.54 on the PAB test set, and that fine-tuning on only 10,000 image-text pairs pushes the full pipeline to 85.49.
desk verdict A fourth-place challenge write-up with a credible fine-tuning result but an under-specified SCA module and inconsistent ablation numbers; not a research contribution as written. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Similarity Coverage Analysis (SCA) is the paper's central mechanism: a test-time re-ranking rule over grouped results. For a set of similar text descriptions, each with a ranked list of retrieved images and a similarity score per image, SCA looks for the same image appearing in multiple queries' results. For a shared result $j$, it computes the score difference $\Delta s = s_{(1,j)} - s_{(2,j)}$ between two queries and replaces the occurrence with the lower score by the next result in that query's list. It is doing the work of a label-free post-filter: it assumes the shared result is correct for one query and that the query with lower similarity to that image is the one whose retrieval should be discarded.
What would settle it
Run SCA in reverse on the PAB test set, replacing the higher-similarity duplicate instead of the lower-similarity one; if reversed re-ranking also improves Recall@1, the direction of SCA's replacement rule is not what drives the gain. A directly targeted check is whether, among pairs of similar queries that share a top-1 image but have different ground truths, the lower-similarity query's match is ever the correct one more often than chance.
Extended reading notes
Core claim
On its own terms, the paper's central discovery is that retrieval failures caused by similar text descriptions can be partly repaired at inference time by comparing confidence scores across queries. When two similar query descriptions return the same image but only one can be correct, SCA keeps the higher-similarity match and replaces the lower-similarity match with the next candidate from that query's ranked list. The paper reports that this rule raises X-VLM's Recall@1 from 77.86 to 80.54 on the PAB test set, and that combining X-VLM fine-tuned on 10,000 image-text pairs with the full pipeline yields 85.49 Recall@1, which placed the entry fourth in the 2025 challenge.
Load-bearing premise
The load-bearing premise is that when two similar queries retrieve the same image, the query with the lower similarity score is the one that got the wrong answer, so its result can safely be replaced without seeing any labels.
Editorial extensions
If this is right
- Fine-tuning on a carefully chosen small subset of 10,000 pairs can beat fine-tuning on the full synthetic training set for this task, so data selection may matter more than data volume.
- The SCA re-ranking step is a training-free post-process that can be applied to any retrieval model that outputs similarity scores, not only X-VLM.
- Because SCA acts on groups of similar queries, it converts a single-query retrieval problem into a problem of reasoning over query neighborhoods.
- The reported results identify X-VLM as a stronger zero-shot baseline (77.86 R@1) than EVA-CLIP (60.01) and LLM2CLIP (73.61) on PAB, so future work on this task can start from it.
Reading between the lines
- SCA's core assumption suggests a testable generalization: group queries by embedding similarity rather than by hand and use group-level score calibration, which the paper does not implement.
- The finding that 10,000 pairs beat 1 million pairs hints that the synthetic training data contains redundant or noisy pairs; a diversity-based sampler could push the same result further.
- If SCA is applied to tasks beyond anomaly search, the direction of the replacement rule would likely need to be recalibrated per dataset, since score distributions differ across models and galleries.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper, authored by the HFUT-LMC team, presents a solution to the WWW 2025 Text-based Person Anomaly Search (TPAS) challenge. The authors fine-tune the X-VLM vision-language model on the PAB dataset and propose a post-processing strategy called Similarity Coverage Analysis (SCA), which aims to improve retrieval when multiple text queries are similar. The manuscript reports a Recall@1 of 85.49 on the PAB test set (fourth place) and claims that SCA improves Recall@1 of zero-shot X-VLM from 77.86 to 80.54 (Table 4). Additional ablations compare baseline models and training-data volumes. The paper is structured as a short challenge-report, with the main novelty being SCA.
Significance. If the claims were fully supported, the paper would offer two modestly useful practical insights: that fine-tuning on a small, carefully chosen subset of a large synthetic dataset can outperform fine-tuning on the full dataset, and that a simple re-ranking heuristic over similar text queries can improve retrieval accuracy. The challenge results are also useful as a reference point for the TPAS benchmark. However, the significance is limited because SCA is evaluated only on a single test set, the method is not compared against existing re-ranking or post-processing techniques, and no code or detailed algorithm is provided. The paper also does not provide any theoretical analysis or machine-checkable artifacts, so the contribution rests entirely on the empirical claims, which are currently not fully substantiated.
major comments (3)
- [Section 3.3, Eq. (9)-(10)] The SCA method is underspecified and therefore not reproducible. Eq. (10) defines Δs only for two queries (i=1 and i=2), yet the text states that 'two or more similar text descriptions m' are handled. The manuscript does not state how similar text queries are automatically grouped at test time, how many queries form a group, how the 'same answer' across ranked lists is identified, how ties in Δs are broken, or how the replacement rule 'j with j+1' operates when m > 2 or when j is the last retrieved item. Without these details, the reported +2.68 Recall@1 gain in Table 4 cannot be independently computed or verified.
- [Section 4.3-4.4, Tables 1-3] The experimental tables are internally inconsistent, which undermines the central empirical claim. Table 2 reports fine-tuned X-VLM with Recall@1 = 79.98, but Table 3 reports Recall@1 = 83.52 when training on ALL data, and Table 1 gives the full pipeline Recall@1 = 85.49. The paper does not explain these discrepancies, such as which training subset is used in Table 2 or whether SCA is applied in that row. Furthermore, Table 4 evaluates SCA only on the zero-shot X-VLM baseline (77.86), not on the fine-tuned model, so the contribution of SCA to the headline 85.49 is unknown.
- [Section 4.4, 'Training Data'] The training-data volume is selected based on test-set performance, which constitutes a form of test-set selection bias. The text states that when 10,000 pairs are used for fine-tuning, 'the effect is best, and the Recall@1 index even reaches 85.49,' and Table 3 is used to justify this choice. No validation split is used, and all hyperparameter or data-subset decisions appear to be made on the PAB test set. As a result, the reported 85.49 overstates expected performance on unseen data, and the comparison in Table 3 is not a valid ablation of data volume.
minor comments (5)
- [Section 3.2, Eq. (5)-(6)] The notation for pairwise similarity is inconsistent: Eq. (4) defines cosine similarity between F_T and F_P, but Eq. (5) uses sim(P,T) and sim(P,T_i), and Eq. (6) uses sim(T,P), without clarifying whether these are the same function or a different compatibility score. This makes the contrastive loss formulation ambiguous.
- [Table 3] The row labels '1W', '2W', '3W', '4W' are not defined; the text refers to '10,000 pairs,' but the units in the table are unclear. Please define the abbreviations and state the exact number of image-text pairs for each row.
- [Section 4.4, 'Baseline Model Selection'] The model name 'LLM-CLIP' in the text is inconsistent with 'LLM2CLIP' in Table 2 and in the references. The same name should be used throughout.
- [Figure 1(b) caption] The caption is circular and difficult to follow: 'Replace the answer with the lower confidence score by using the answer from the group where the confidence score is lower than the current score.' Please rephrase to state the actual decision rule clearly.
- [References] Several references are incomplete or informal (e.g., reference [36] lists only an arXiv ID without publication venue). The authors should ensure all entries are fully specified, including page numbers where available.
Circularity Check
The headline 85.49 R@1 is selected on the PAB test set (Table 3); the same test metric is then reported as the method's result, making the headline a test-set-fitted statistic rather than an independent prediction.
-
fitted input called prediction
[Section 4.4 'Training Data', Table 3]
"As shown in Table 3, we can see that when only 10,000 pairs of image-text data pairs are used for fine-tuning, the effect is best, and the Recall@1 index even reaches 85.49. This result not only proves the importance of the right amount of data for model tuning but also shows that this choice is more effective than the strategy of fine-tuning with the full amount of data."
The training-data volume (1W vs 2W vs 3W vs 4W vs ALL) is selected by comparing Recall@1 on the PAB test set, and the same test-set Recall@1 (85.49) is then reported as the final performance of the solution (Table 1 and abstract). Because the selection criterion and the reported metric are identical, the headline 85.49 is the maximum of the test metric over the hyperparameter grid, not an unbiased evaluation of a pre-specified model. The result is thus a fitted test statistic presented as a prediction; the evaluation loop reduces to the selection criterion.
full rationale
The paper's derivation chain is largely self-contained: X-VLM, PAB, EVA-CLIP, and LLM2CLIP are external, and the SCA re-ranking rule in Eqs. (9)-(10) is an algorithmic heuristic rather than a quantity defined in terms of its own output. There are no load-bearing self-citations or imported uniqueness theorems. However, the central reported number, 85.49 R@1, is circular in the evaluation sense: Table 3 uses the PAB test set to choose the training-data volume ('when only 10,000 pairs ... the effect is best'), and the same test-set R@1 is then presented as the method's achievement. The 85.49 is therefore an optimistically selected maximum rather than an independent prediction. The SCA ablation in Table 4 is also measured on the same test set and is under-specified (grouping of similar queries, tie-breaking, and the exact replacement rule are not given), which prevents independent reproduction; this is a reproducibility and validation problem rather than a definitional circularity. The internal tables are also inconsistent (fine-tuned X-VLM is 79.98 in Table 2 but 83.52 with ALL data in Table 3), further complicating the derivation but not constituting circularity by itself. The challenge leaderboard (Table 1) and the cross-model comparison (Table 2) give some external anchoring, but they do not validate the ablation claims. Score 6 reflects the partial circularity of the headline result, while noting that the SCA mechanism itself is not definitionally circular.
Assumptions & free parameters
free parameters (3)
- Training subset size =
10,000 image-text pairs
- SCA parameters =
Not disclosed
- Fine-tuning hyperparameters =
lr=3e-5, batch=16, epochs=10, weight decay=0.01, lr multiplier=2
assumptions (4)
- domain assumption Cosine similarity between text and image embeddings is a valid matching score for TPAS.
- domain assumption A static image can indicate abnormal behavior such as falling or being hit.
- ad hoc to paper If two similar text queries retrieve the same image, the query with the lower confidence score is the incorrect one.
- ad hoc to paper Similar text descriptions can be grouped automatically at test time without ground-truth labels.
Cite this review
Pith. "Pith review of Efficient Vision Language Model Fine-tuning for Text-based Person Anomaly Search." pith.science (2026). https://pith.science/paper/MSGKGYBA
@misc{pith2026250203230,
author = {Pith},
title = {Pith review of: Efficient Vision Language Model Fine-tuning for Text-based Person Anomaly Search},
year = {2026},
howpublished = {\url{https://pith.science/paper/MSGKGYBA}},
note = {Machine review of arXiv:2502.03230}
}
read the original abstract
This paper presents the HFUT-LMC team's solution to the WWW 2025 challenge on Text-based Person Anomaly Search (TPAS). The primary objective of this challenge is to accurately identify pedestrians exhibiting either normal or abnormal behavior within a large library of pedestrian images. Unlike traditional video analysis tasks, TPAS significantly emphasizes understanding and interpreting the subtle relationships between text descriptions and visual data. The complexity of this task lies in the model's need to not only match individuals to text descriptions in massive image datasets but also accurately differentiate between search results when faced with similar descriptions. To overcome these challenges, we introduce the Similarity Coverage Analysis (SCA) strategy to address the recognition difficulty caused by similar text descriptions. This strategy effectively enhances the model's capacity to manage subtle differences, thus improving both the accuracy and reliability of the search. Our proposed solution demonstrated excellent performance in this challenge.
Figures
Reference graph
Works this paper leans on
-
[1]
Andra Acsintoae, Andrei Florescu, Mariana-Iuliana Georgescu, Tudor Mare, Paul Sumedrea, Radu Tudor Ionescu, Fahad Shahbaz Khan, and Mubarak Shah. 2022. Ubnormal: New Benchmark for Supervised Open-Set Video Anomaly Detection. In Computer Vision and Pattern Recognition. 20143–20153
work page 2022
-
[2]
Yang Bai, Jingyao Wang, Min Cao, Chen Chen, Ziqiang Cao, Liqiang Nie, and Min Zhang. 2023. Text-based Person Search without Parallel Image-Text Data. In ACM International Conference on Multimedia. 757–767
work page 2023
-
[3]
Min Cao, Yang Bai, Ziyin Zeng, Mang Ye, and Min Zhang. 2024. An Empirical Study of Clip for Text-based Person Search. In AAAI Conference on Artificial Intelligence. 465–473
work page 2024
-
[4]
Fei-Long Chen, Du-Zhen Zhang, Ming-Lun Han, Xiu-Yi Chen, Jing Shi, Shuang Xu, and Bo Xu. 2023. Vlp: A Survey on Vision-Language Pre-training. Machine Intelligence Research (2023), 38–56
work page 2023
-
[5]
Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang, Chenguang Zhu, Pengchuan Zhang, Lu Yuan, Nanyun Peng, et al . 2022. An Empirical Study of Training End-to-End Vision-and-Language Transformers. In Computer Vision and Pattern Recognition. 18166–18176
work page 2022
-
[6]
Xinyang Feng, Dongjin Song, Yuncong Chen, Zhengzhang Chen, Jingchao Ni, and Haifeng Chen. 2021. Convolutional Transformer Based Dual Discriminator Gen- erative Adversarial Networks for Video Anomaly Detection. InACM International Conference on Multimedia. 5546–5554
work page 2021
-
[7]
Alessandro Flaborea, Luca Collorone, Guido Maria D’Amely Di Melendugno, Stefano D’Arrigo, Bardh Prenkaj, and Fabio Galasso. 2023. Multimodal Motion Conditioned Diffusion Model for Skeleton-Based Video Anomaly Detection. In International Conference on Computer Vision. 10318–10329
work page 2023
-
[8]
Or Hirschorn and Shai Avidan. 2023. Normalizing Flows for Human Pose Anom- aly Detection. In International Conference on Computer Vision. 13545–13554
work page 2023
Show all 45 references
-
[9]
Weiquan Huang, Aoqi Wu, Yifan Yang, Xufang Luo, Yuqing Yang, Liang Hu, Qi Dai, Xiyang Dai, Dongdong Chen, Chong Luo, and Lili Qiu. 2024. LLM2CLIP: Powerful Language Model Unlock Richer Visual Representation. arXiv:2411.04997 [cs.CV] https://arxiv.org/abs/2411.04997
2024
-
[10]
Huaizu Jiang, Ishan Misra, Marcus Rohrbach, Erik Learned-Miller, and Xinlei Chen. 2020. In Defense of Grid Features for Visual Question Answering. In Computer Vision and Pattern Recognition. 10267–10276
2020
-
[11]
Xintong Jiang, Yaxiong Wang, Mengjian Li, Yujiao Wu, Bingwen Hu, and Xuem- ing Qian. 2024. CaLa: Complementary Association Learning for Augmenting Comoposed Image Retrieval. InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information...
2024
-
[12]
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021. Align before Fuse: Vision and Language Representation Learning with Momentum Distillation. In Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y . D...
2021
-
[13]
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021. Align before Fuse: Vision and Language Representation Learning with Momentum Distillation. In Neural Information Processing Systems. 9694–9705
2021
-
[14]
Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu
-
[15]
Shuang Li, Tong Xiao, Hongsheng Li, Bolei Zhou, Dayu Yue, and Xiaogang Wang
-
[16]
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao
-
[17]
Xin Lin, Li Zhu, Shuyu Yang, and Yaxiong Wang. 2023. Diff attention: A novel attention scheme for person re-identification. Computer Vision and Image Understanding 228 (2023), 103623
2023
-
[18]
Yuting Ma, Shengeng Tang, Xiaohua Xu, and Lechao Cheng. 2024. Modality Alignment Meets Federated Broadcasting. arXiv preprint arXiv:2411.15837 (2024)
2024 arXiv
-
[19]
In European Conference on Computer Vision
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks. In European Conference on Computer Vision. 121–137
-
[20]
Jicheol Park, Dongwon Kim, Boseung Jeong, and Suha Kwak. 2025. PLOT: Text- based Person Search with Part Slot Attention for Corresponding Part Discovery. In European Conference on Computer Vision. 474–490
2025
-
[21]
Fei Shen, Xiangbo Shu, Xiaoyu Du, and Jinhui Tang. 2023. Pedestrian-specific Bipartite-aware Similarity Learning for Text-based Person Retrieval. In ACM International Conference on Multimedia. 8922–8931
2023
-
[22]
Kai Niu, Yan Huang, Wanli Ouyang, and Liang Wang. 2020. Improving Description-Based Person Re-Identification by Multi-Granularity Image-Text Alignments. IEEE Transactions on Image Processing(2020), 5542–5556
2020
-
[23]
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. 2023. Eva-Clip: Improved Training Techniques for Clip at Scale. arXiv preprint arXiv:2303.15389 (2023)
2023 arXiv
-
[24]
Shengeng Tang, Dan Guo, Richang Hong, and Meng Wang. 2022. Graph-based multimodal sequential embedding for sign language translation. IEEE Transac- tions on Multimedia 24 (2022), 4433–4445
2022
-
[25]
Peipei Song, Dan Guo, Xun Yang, Shengeng Tang, and Meng Wang. 2024. Emo- tional Video Captioning With Vision-Based Emotion Interpretation Network. IEEE Transactions on Image Processing33 (2024), 1122–1135
2024
-
[26]
Shengeng Tang, Richang Hong, Dan Guo, and Meng Wang. 2022. Gloss semantic- enhanced network with online back-translation for sign language production. In Proceedings of the 30th ACM International Conference on Multimedia . 5630– 5638
2022
-
[27]
Shengeng Tang, Feng Xue, Jingjing Wu, Shuo Wang, and Richang Hong. 2024. Gloss-driven Conditional Diffusion Models for Sign Language Production. ACM Transactions on Multimedia Computing, Communications and Applications (2024)
2024
-
[28]
Shengeng Tang, Jiayi He, Dan Guo, Yanyan Wei, Feng Li, and Richang Hong
-
[29]
Yaxiong Wang, Hao Yang, Xiuxiu Bai, Xueming Qian, Lin Ma, Jing Lu, Biao Li, and Xin Fan. 2020. PFAN++: Bi-directional image-text retrieval with position focused attention network. IEEE Transactions on Multimedia 23 (2020), 3362– 3376
2020
-
[30]
Yaxiong Wang, Hao Yang, Xueming Qian, Lin Ma, Jing Lu, Biao Li, and Xin Fan
-
[31]
Yanyan Wei, Yilin Zhang, Kun Li, Fei Wang, Shengeng Tang, and Zhao Zhang
-
[32]
Yaxiong Wang, Yunzhong Hou, Shuyu Yang, Zhedong Zheng, Zhun Zhong, and Liang Zheng. 2025. MORE’25 Multimedia Object Re-ID: Advancements, Chal- lenges, and Opportunities. In ACM Web Conference Workshop
2025
-
[33]
Jingjing Wu, Yunkai Zhang, Xi Zhou, Shengeng Tang, and Yanyan Wei. 2024. Comprehensive Survey on Person Identification: Queries, Methods, and Datasets. In Proceedings of the 1st ICMR Workshop on Multimedia Object Re-Identification. 1–6
2024
-
[34]
Shuyu Yang, Yaxiong Wang, Li Zhu, and Zhedong Zheng. 2024. Beyond Walking: A Large-Scale Image-Text Benchmark for Text-based Person Anomaly Search. arXiv preprint arXiv:2411.17776 (2024)
2024 arXiv
-
[35]
M Zaigham Zaheer, Arif Mahmood, M Haris Khan, Mattia Segu, Fisher Yu, and Seung-Ik Lee. 2022. Generative Cooperative Learning for Unsupervised Video Anomaly Detection. In Computer Vision and Pattern Recognition. 14744–14754
2022
-
[36]
Yan Zeng, Xinsong Zhang, and Hang Li. 2022. Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts. arXiv:2111.08276 [cs.CL]
2022 arXiv
-
[37]
Computer Vision and Image Understanding 250 (2025), 104222
Leveraging vision-language prompts for real-world image restoration and enhancement. Computer Vision and Image Understanding 250 (2025), 104222
2025
-
[38]
Jingjing Wu, Richang Hong, and Shengeng Tang. 2024. Intermediary-Generated Bridge Network for RGB-D Cross-Modal Re-Identification. ACM Transactions on Intelligent Systems and Technology15, 6 (2024), 1–25
2024
-
[39]
Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. 2020. Unified Vision-Language Pre-training for Image Captioning and Vqa. In AAAI Conference on Artificial Intelligence. 13041–13049
2020
-
[43]
Shengyu Zhang, Tan Jiang, Tan Wang, Kun Kuang, Zhou Zhao, Jianke Zhu, Jin Yu, Hongxia Yang, and Fei Wu. 2020. Devlbert: Learning Deconfounded Visio- Linguistic Representations. In ACM International Conference on Multimedia . 4373–4382
2020
-
[44]
Ying Zhang and Huchuan Lu. 2018. Deep Cross-Modal Projection Learning for Image-Text Matching. In European Conference on Computer Vision. 686–701
2018
-
[2017]
In Computer Vision and Pattern Recognition
Person Search with Natural Language Description. In Computer Vision and Pattern Recognition. 1970–1979
1970
-
[2019]
In Proceedings of the 28th International Joint Conference on Artificial Intelligence
Position focused attention network for image-text matching. In Proceedings of the 28th International Joint Conference on Artificial Intelligence. 3792–3798
-
[2020]
arXiv preprint arXiv:2005.00200 (2020)
Hero: Hierarchical Encoder for Video+ Language Omni-Representation Pre-training. arXiv preprint arXiv:2005.00200 (2020)
2020 arXiv
-
[2025]
In Proceedings of the AAAI Conference on Artificial Intelligence
Sign-IDD: Iconicity Disentangled Diffusion for Sign Language Production. In Proceedings of the AAAI Conference on Artificial Intelligence
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.