REVIEW 4 major objections 4 minor 38 references
Multispectral Pedestrian Detection with Sparsely Annotated Label
T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read SAMPD claims that multispectral similarity between pseudo-labels and ground-truth boxes, plus adaptive retrieval augmentation, lets pedestrian detectors match nearly fully-supervised accuracy even when 30% of labels are removed.
desk verdict A well-engineered SAOD framework for multispectral pedestrian detection whose claimed gains rely on a synthetic deterministic small-box-removal benchmark that deserves more scrutiny. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the per-modality pseudo-label quality score $w_k = \frac{1}{N}\sum_{i=1}^{N}\max_{j} d(l^{k(s)}_{PL_i}, l^{k(s)}_{GT_j})$, where $d$ is cosine similarity between global-average-pooled features of student-model pseudo-label boxes and ground-truth boxes, computed separately for visible, thermal, and fusion paths. This score both reweights detection losses and, through thresholds $\tau_1 = 0.9$ and $\tau_2 = 0.7$, sorts pseudo-labels into positive, negative, and uncertain categories for the positive pseudo-label guiding loss. The second mechanism is APRA, which retrieves pedestrian patches whose brightness matches the input image, pastes them into low-saliency regions where pedestrians are plausible, and dynamically converts high-quality pseudo-labels into ground truth. Together these turn pseudo-labels from noise into a curriculum for the student detector.
What would settle it
Re-run the same comparisons on a dataset with naturally sparse labels, or on KAIST/LLVIP with annotations removed according to occlusion or random patterns; if the MR advantage over Calibrated Teacher narrows or reverses (for example, if random removal makes SAMPD's 8.56 vs 10.47 gap shrink below measurement noise), the size-biased protocol is doing the work.
Extended reading notes
Core claim
On its own terms, the paper claims that sparse annotation is not a data-quantity problem but a label-quality problem, and that multispectral cues can repair it. SAMPD uses a teacher–student setup in which the teacher produces pseudo-labels for missing pedestrians; the student, trained on visible, thermal, and fused features, computes for each pseudo-label a cosine similarity to the nearest ground-truth feature. That single similarity score drives three mechanisms: it weights the per-modality detection loss (MPAW), it separates pseudo-labels into positive, negative, and uncertain groups for a guiding contrastive loss (PPE), and, when high, it promotes the pseudo-label into ground truth (APRA). The reported outcome is that at 30% removal SAMPD reaches 8.56 MR on KAIST and 7.65 on LLVIP, beating the best prior SAOD method by roughly 2 MR points, and at 0% removal it improves the fully labeled baseline from 7.58 to 6.50 MR on KAIST.
Load-bearing premise
The whole evaluation runs on a synthetic protocol that removes small bounding boxes with higher probability than large ones, so the reported gains assume real annotation gaps follow that size-based pattern rather than, say, random or occlusion-driven omissions.
Editorial extensions
If this is right
- At 30% label removal, SAMPD reports near-fully-supervised accuracy (8.56 MR on KAIST against 7.58 with all labels), meaning sparse annotations may cost little if pseudo-labels are quality-filtered by modality similarity.
- The dynamic ground-truth refinement grows the annotation set during training (e.g., from 40,107 to 47,369 boxes on KAIST), closing gaps that otherwise make the visible and thermal streams disagree.
- The framework transfers to other backbones and detectors: supplementary results show gains with ResNet backbones and with MLPD and ProbEn as base detectors, not just the SSD+VGG16 used in the main tables.
- The method also improves the fully annotated setting (KAIST MR from 7.58 to 6.50; LLVIP from 6.01 to 5.70), which the paper interprets as evidence that real-world 'complete' labels still contain missing pedestrians.
Reading between the lines
- Because the synthetic protocol deletes small boxes preferentially, the reported margins likely overstate the benefit if real annotators miss pedestrians mainly because of occlusion, motion blur, or low contrast; testing on occlusion-sorted removal would separate those causes.
- The cosine-similarity quality score is a generic idea: any detector with a feature backbone could weight pseudo-labels by their feature proximity to labeled objects, so the MPAW/PPE mechanism may transfer to single-modality SAOD tasks, not just multispectral pedestrians.
- The paper's own limitation section notes APRA operates at image level using global brightness; a feature-level retrieval that matches individual pedestrian appearance rather than scene brightness could further reduce domain artifacts, and could be tested by replacing the brightness query with a per-patch appearance embedding.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes SAMPD, a teacher-student framework for multispectral pedestrian detection under sparse annotation. It introduces three modules: MPAW, which weights per-modality detection losses by the cosine similarity between pseudo-label and ground-truth features; PPE, a contrastive loss that pulls high-quality pseudo-labels together and pushes them away from low-quality ones; and APRA, which augments training images with retrieved pedestrian patches and dynamically converts high-confidence pseudo-labels into ground-truth labels. The method is evaluated on KAIST and LLVIP with synthetically removed annotations (30%, 50%, 70%), reporting consistent MR reductions over prior SAOD methods, plus ablations and fully-annotated (0% removal) results.
Significance. If the reported results hold, SAMPD would be a useful contribution to sparsely annotated multispectral pedestrian detection, a practical setting where annotation is costly and incomplete. The paper's strengths include its clearly motivated modules, code release, consistent gains across two datasets and three removal ratios, and additional generalization experiments with different backbones and advanced MPD models (supplementary Tables 2 and 8). The fully-annotated results (Table 6) are also a nice bonus, showing the method can help even when labels are nominally complete. However, the central claim rests entirely on a synthetic annotation-removal protocol whose realism is not established, and the evaluation lacks statistical replication, which tempers the significance.
major comments (4)
- [Supplementary Algorithm 1; Supplementary Table 5] The sparse-annotation benchmark is the load-bearing element of the paper, but as written it is a deterministic procedure: Algorithm 1 repeatedly removes the single smallest remaining box in each image via ARGMAX(PV), not a stochastic size-weighted deletion. The main text describes 'increasing the probability of removing bounding-box annotations with smaller widths,' which contradicts the deterministic algorithm. More importantly, the supplementary comparison between random and size-based removal (Supplementary Table 5) is run only for the Supervised baseline, not for SAMPD or Calibrated Teacher, so it does not establish that the reported margins over baselines survive under a different missingness mechanism. Since APRA explicitly re-introduces pedestrian patches and MPAW/PPE re-weight pseudo-labels, the method may be disproportionately well-matched to the 'all small boxes missing' protocol. I request experiments under random removal for all compared methods, or a real-world sparsely annotated dataset, to support the general claim of improved performance in sparsely annotated environments.
- [Tables 1 and 2; Implementation Details] All reported numbers appear to come from a single training run, with no error bars, multiple seeds, or statistical tests. The improvements over the best prior method are modest in several cells (e.g., 8.56 vs. 10.47 MR at 30% removal on KAIST; 7.65 vs. 9.41 on LLVIP), and without variance estimates it is unclear whether these differences are meaningful. The abstract's word 'significantly' is not supported by any statistical evidence. Please report means and standard deviations over at least three seeds, or justify why single-run comparisons are standard for this task.
- [Supplementary Table 6; PPE module thresholds] The PPE thresholds τ1 = 0.9 and τ2 = 0.7 were selected by searching over several values and choosing the one that yields the best MR on the test set. This is a form of test-set overfitting, and it creates an unfair comparison with baselines whose hyperparameters were not tuned on this benchmark. The authors should either split a validation set for threshold selection or acknowledge that the reported gains may be optimistic. The threshold search also does not include error bars for each configuration, so the selection could be driven by noise.
- [Eq. (7) and surrounding text] The description of the PPE loss is ambiguous about which model is updated. The text says the PG loss 'helps train the teacher model to generate higher-quality pseudo-labels,' but the loss uses feature vectors explicitly labeled as coming from the student model (l_{PL}^{(s)}, l_{GT}^{(s)}). If the teacher is an exponential moving average of the student, this should be stated; if the teacher has its own parameters, the gradient path for L_PG is unclear given Eq. (8) sums it into the total loss. This ambiguity affects reproducibility of the core method.
minor comments (4)
- [Abstract] There is a typo: 'approches' should be 'approaches.'
- [Ablation Study (main text, after Table 3)] The sentence 'the multispectral pedestrian-aware adaptive weight WAL in MPAW module' appears to contain a typo: 'WAL' is not defined anywhere; it likely should be 'wk' or 'MPAW.'
- [Eq. (8)] The sentence 'Lsum_det denotes the detection loss using.' is incomplete; it should probably say 'using the MPAW-weighted combination in Eq. (3).'
- [Implementation Details] The number of random seeds, validation split usage, and exact data augmentation details are not reported; this makes it harder to assess the stability and reproducibility of the results.
Circularity Check
No significant circularity: the reported gains are measured on held-out test sets against external baselines, and the training-time self-labeling modules do not define the evaluation outcome by construction.
full rationale
The paper makes no first-principles derivation whose output equals an input; its central claim is an empirical MR/AP comparison on held-out KAIST and LLVIP test sets against external SAOD baselines under a shared sparse-annotation protocol. MPAW, PPE, and APRA all operate during training: Eq. (1) and Eq. (4) use cosine similarity between pseudo-label and current ground-truth features to reweight losses or refine labels, and APRA converts high-confidence pseudo-labels into ground truth. This is a standard teacher-student self-training bootstrap for SAOD, not a reduction of the test-time prediction to the training signal; the evaluated annotations are not fed back into training. The synthetic sparse-annotation protocol in supplementary Algorithm 1 is borrowed from SparseDet (Suri et al. 2023); whether it faithfully models real annotator omissions is a question of benchmark validity and correctness risk, not circularity. Several self-citations appear for the MR evaluation metric and the base detection loss, but these are conventional and non-load-bearing for the claimed improvement. No Eq. X = Eq. Y by construction, no fitted parameter renamed as a prediction, and no load-bearing uniqueness argument was found.
Assumptions & free parameters
free parameters (5)
- tau1 (PPE positive threshold) =
0.9
- tau2 (PPE negative threshold) =
0.7
- tau (PG temperature) =
0.1
- lambda1, lambda2 =
1, 1
- m (number of retrieved patches) =
1
assumptions (5)
- domain assumption Teacher-student pseudo-labeling improves learning from sparse annotations
- domain assumption Cosine similarity between pseudo-label features and ground-truth features measures pseudo-label quality
- domain assumption Brightness difference is a valid proxy for scene-specific pedestrian appearance
- domain assumption Low-saliency regions are suitable for inserting pedestrian patches
- ad hoc to paper Removing annotations with higher probability for small boxes faithfully models real-world sparse annotations
Cite this review
Pith. "Pith review of Multispectral Pedestrian Detection with Sparsely Annotated Label." pith.science (2026). https://pith.science/paper/OJU5Q4SV
@misc{pith2026250102640,
author = {Pith},
title = {Pith review of: Multispectral Pedestrian Detection with Sparsely Annotated Label},
year = {2026},
howpublished = {\url{https://pith.science/paper/OJU5Q4SV}},
note = {Machine review of arXiv:2501.02640}
}
read the original abstract
Although existing Sparsely Annotated Object Detection (SAOD) approches have made progress in handling sparsely annotated environments in multispectral domain, where only some pedestrians are annotated, they still have the following limitations: (i) they lack considerations for improving the quality of pseudo-labels for missing annotations, and (ii) they rely on fixed ground truth annotations, which leads to learning only a limited range of pedestrian visual appearances in the multispectral domain. To address these issues, we propose a novel framework called Sparsely Annotated Multispectral Pedestrian Detection (SAMPD). For limitation (i), we introduce Multispectral Pedestrian-aware Adaptive Weight (MPAW) and Positive Pseudo-label Enhancement (PPE) module. Utilizing multispectral knowledge, these modules ensure the generation of high-quality pseudo-labels and enable effective learning by increasing weights for high-quality pseudo-labels based on modality characteristics. To address limitation (ii), we propose an Adaptive Pedestrian Retrieval Augmentation (APRA) module, which adaptively incorporates pedestrian patches from ground-truth and dynamically integrates high-quality pseudo-labels with the ground-truth, facilitating a more diverse learning pool of pedestrians. Extensive experimental results demonstrate that our SAMPD significantly enhances performance in sparsely annotated environments within the multispectral domain.
Figures
Figures from the paper (11 more)
Reference graph
Works this paper leans on
-
[1]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Chen, N.; Xie, J.; Nie, J.; Cao, J.; Shao, Z.; and Pang, Y. 2023. Attentive alignment network for multispectral pedestrian detection. In Proceedings of the 31st ACM international conference on multimedia, 3787--3795
work page 2023
-
[4]
Chen, Y.-T.; Shi, J.; Ye, Z.; Mertz, C.; Ramanan, D.; and Kong, S. 2022. Multimodal object detection via probabilistic ensembling. In European Conference on Computer Vision, 139--158. Springer
work page 2022
-
[5]
Dasgupta, K.; Das, A.; Das, S.; Bhattacharya, U.; and Yogamani, S. 2022. Spatio-Contextual Deep Network-Based Multimodal Pedestrian Detection for Autonomous Driving. IEEE Transactions on Intelligent Transportation Systems, 23(9): 15940--15950
work page 2022
-
[6]
Guan, D.; Cao, Y.; Yang, J.; Cao, Y.; and Yang, M. Y. 2019. Fusion of multispectral data through illumination-aware deep neural networks for pedestrian detection. Information Fusion, 50: 148--157
work page 2019
-
[7]
Hu, Y.; Zhang, N.; and Weng, L. 2023. Retrieve the Visible Feature to Improve Thermal Pedestrian Detection Using Discrepancy Preserving Memory Network. In 2023 IEEE International Conference on Image Processing (ICIP), 1125--1129
work page 2023
-
[8]
Hwang, S.; Park, J.; Kim, N.; Choi, Y.; and So Kweon, I. 2015. Multispectral pedestrian detection: Benchmark dataset and baseline. In Proceedings of the IEEE conference on computer vision and pattern recognition, 1037--1045
work page 2015
Show all 38 references
-
[9]
Jia, X.; Zhu, C.; Li, M.; Tang, W.; and Zhou, W. 2021. LLVIP: A visible-infrared paired dataset for low-light vision. In Proceedings of the IEEE/CVF international conference on computer vision, 3496--3504
2021
-
[10]
Kiefer, J.; and Wolfowitz, J. 1952. Stochastic estimation of the maximum of a regression function. The Annals of Mathematical Statistics, 462--466
1952
-
[11]
Kim, J.; Kim, H.; Kim, T.; Kim, N.; and Choi, Y. 2021. MLPD: Multi-label pedestrian detector in multispectral domain. IEEE Robotics and Automation Letters, 6(4): 7846--7853
2021
-
[12]
U.; Park, S.; and Ro, Y
Kim, J. U.; Park, S.; and Ro, Y. M. 2021 a . Robust small-scale pedestrian detection with cued recall via memory learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 3050--3059
2021
-
[13]
U.; Park, S.; and Ro, Y
Kim, J. U.; Park, S.; and Ro, Y. M. 2021 b . Uncertainty-guided cross-modal learning for robust multispectral pedestrian detection. IEEE Transactions on Circuits and Systems for Video Technology, 32(3): 1510--1523
2021
-
[14]
U.; Park, S.; and Ro, Y
Kim, J. U.; Park, S.; and Ro, Y. M. 2022. Towards versatile pedestrian detector with multisensory-matching and multispectral recalling memory. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36(1), 1157--1165
2022
-
[15]
U.; and Ro, Y
Kim, J. U.; and Ro, Y. M. 2023. Similarity Relation Preserving Cross-Modal Learning for Multispectral Pedestrian Detection Against Adversarial Attacks. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1--5. IEEE
2023
-
[16]
G.; and Ro, Y
Kim, T.; Shin, S.; Yu, Y.; Kim, H. G.; and Ro, Y. M. 2024. Causal Mode Multiplexer: A Novel Framework for Unbiased Multispectral Pedestrian Detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 26784--26793
2024
-
[17]
Lewis, P.; Perez, E.; Piktus, A.; Petroni, F.; Karpukhin, V.; Goyal, N.; Küttler, H.; Lewis, M.; tau Yih, W.; Rocktäschel, T.; Riedel, S.; and Kiela, D. 2021. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:2005.11401
2021 arXiv
-
[18]
Li, C.; Song, D.; Tong, R.; and Tang, M. 2019. Illumination-aware faster R-CNN for robust multispectral pedestrian detection. Pattern Recognition, 85: 161--171
2019
-
[19]
Li, S.; Liu, J.; Shen, W.; Sun, J.; and Tan, C. 2023. Robust Teacher: Self-correcting pseudo-label-guided semi-supervised learning for object detection. Computer Vision and Image Understanding, 235: 103788
2023
-
[20]
Liu, W.; Anguelov, D.; Erhan, D.; Szegedy, C.; Reed, S.; Fu, C.-Y.; and Berg, A. C. 2016. Ssd: Single shot multibox detector. In Computer Vision--ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11--14, 2016, Proceedings, Part I 14, 21--37. Springer
2016
-
[21]
Liu, Y.; Hu, C.; Zhao, B.; Huang, Y.; and Zhang, X. 2024. Region-Based Illumination-Temperature Awareness and Cross-Modality Enhancement for Multispectral Pedestrian Detection. IEEE Transactions on Intelligent Vehicles
2024
-
[22]
Niitani, Y.; Akiba, T.; Kerola, T.; Ogawa, T.; Sano, S.; and Suzuki, S. 2019. Sampling Techniques for Large-Scale Object Detection from Sparsely Annotated Objects. arXiv:1811.10862
2019 arXiv
-
[23]
H.; Kim, J
Park, S.; Choi, D. H.; Kim, J. U.; and Ro, Y. M. 2022. Robust thermal infrared pedestrian detection by associating visible pedestrian knowledge. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 4468--4472. IEEE
2022
-
[24]
Paszke, A.; Gross, S.; Chintala, S.; Chanan, G.; Yang, E.; DeVito, Z.; Lin, Z.; Desmaison, A.; Antiga, L.; and Lerer, A. 2017. Automatic differentiation in PyTorch. NeurIPS-W
2017
-
[25]
Ramos, R.; Elliott, D.; and Martins, B. 2023. Retrieval-augmented Image Captioning. arXiv:2302.08268
2023 arXiv
-
[26]
Sarto, S.; Cornia, M.; Baraldi, L.; and Cucchiara, R. 2022. Retrieval-Augmented Transformer for Image Captioning. arXiv:2207.13162
2022 arXiv
-
[27]
Simonyan, K.; and Zisserman, A. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv:1409.1556
2015 arXiv
-
[28]
S.; Chellappa, R.; and Shrivastava, A
Suri, S.; Rambhatla, S. S.; Chellappa, R.; and Shrivastava, A. 2023. SparseDet: Improving Sparsely Annotated Object Detection with Pseudo-positive Mining. arXiv:2201.04620
2023 arXiv
-
[29]
Wang, H.; Liu, L.; Zhang, B.; Zhang, J.; Zhang, W.; Gan, Z.; Wang, Y.; Wang, C.; and Wang, H. 2023 a . Calibrated Teacher for Sparsely Annotated Object Detection. arXiv:2303.07582
2023 arXiv
-
[30]
Wang, T.; Yang, T.; Cao, J.; and Zhang, X. 2021. Co-mining: Self-Supervised Learning for Sparsely Annotated Object Detection. arXiv:2012.01950
2021 arXiv
-
[31]
Wang, X.; Yang, X.; Zhang, S.; Li, Y.; Feng, L.; Fang, S.; Lyu, C.; Chen, K.; and Zhang, W. 2023 b . Consistent-Teacher: Towards Reducing Inconsistent Pseudo-targets in Semi-supervised Object Detection. arXiv:2209.01589
2023 arXiv
-
[32]
M.; Cholakkal, H.; Nie, J.; Cao, J.; Laaksonen, J.; and Khan, F
Xie, J.; Anwer, R. M.; Cholakkal, H.; Nie, J.; Cao, J.; Laaksonen, J.; and Khan, F. S. 2022. Learning a dynamic cross-modal network for multispectral pedestrian detection. In Proceedings of the 30th ACM International Conference on Multimedia, 4043--4052
2022
-
[33]
Xie, Q.; Cheng, T.-Y.; Zhong, J.-X.; Zhou, K.; Markham, A.; and Trigoni, N. 2024. Beyond Fusion: Modality Hallucination-based Multispectral Fusion for Pedestrian Detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 655--664
2024
-
[34]
Xu, X.; Zhan, W.; Zhu, D.; Jiang, Y.; Chen, Y.; and Guo, J. 2023. Contour Information-Guided Multi-Scale Feature Detection Method for Visible-Infrared Pedestrian Detection. Entropy, 25(7)
2023
-
[35]
Zhang, H.; Chen, F.; Shen, Z.; Hao, Q.; Zhu, C.; and Savvides, M. 2020. Solving Missing-Annotation Object Detection with Background Recalibration Loss. arXiv:2002.05274
2020 arXiv
-
[36]
Zhang, L.; Zhu, X.; Chen, X.; Yang, X.; Lei, Z.; and Liu, Z. 2019. Weakly aligned cross-modal learning for multispectral pedestrian detection. In Proceedings of the IEEE/CVF international conference on computer vision, 5127--5137
2019
-
[37]
Zhou, K.; Chen, L.; and Cao, X. 2020. Improving multispectral pedestrian detection by addressing modality imbalance problems. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part XVIII 16, 787--803. Springer
2020
-
[38]
Zhu, H.; Wu, H.; Wang, X.; He, D.; Liu, Z.; and Pan, X. 2023. DPACFuse: Dual-Branch Progressive Learning for Infrared and Visible Image Fusion with Complementary Self-Attention and Convolution. Sensors, 23(16): 7205
2023
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.