REVIEW 2 major objections 5 minor 45 references
Are you In or Out (of gallery)? Wisdom from the Same-Identity Crowd
T0 review · 2 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read The paper claims that the ranks of additional enrolled images of the rank-one match—rather than a similarity-score threshold—can tell whether a probe is in the gallery, reaching up to 98.58% accuracy on mugshot-quality probes.
desk verdict Genuinely novel rank-vector feature for in/out-of-gallery detection; results are consistent and useful, but the missing identity-disjoint split must be clarified before the headline accuracy is credible. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the rank query vector: for a probe search, take the rank-one identity and record the ranks of up to three additional enrolled images of that identity. For an in-gallery probe these ranks are concentrated near the top, while for an out-of-gallery probe they scatter, because the false rank-one identity was selected by incidental similarity. A small multi-layer perceptron with feature-permutation augmentation learns the non-linear boundary between these two rank patterns, using $R_{\mathrm{din}}=3$ ranks as input in all experiments.
What would settle it
Take a probe known to be out-of-gallery whose highest-scoring false identity has several enrolled photos that look similar to the probe (same expression, beard, lighting). If those additional photos consistently rank near the top, and a classifier trained on this data cannot beat chance, the premise fails. Concretely: measure the rank of the second enrolled image of the rank-one identity for known out-of-gallery probes; if a large fraction of false hits have additional images inside the top ranks, the signal disappears.
Extended reading notes
Core claim
The paper's central claim is that a probe is in-gallery precisely when the additional enrolled images of the rank-one identity keep ranking near the top: for a true match these images cluster at low ranks, whereas for a false rank-one match they drift far down because that identity was selected by incidental similarity. The authors build a small nonlinear classifier on the ranks of up to three additional images, trained on paired searches with and without the probe's own images, and report that this classifies in-gallery vs. out-of-gallery probes with 98.58% accuracy on original MORPH probes using AdaFace, stays above 86% for blurred, downsampled, turbulent, and sunglasses probes with modern
Load-bearing premise
The method assumes that a false top match is picked because of some incidental similarity, so the other photos of that wrong identity will rank far down the list; if those other photos also rank near the top, the classifier has no signal to work with.
Editorial extensions
If this is right
- Open-set face identification can be performed without hand-tuned score thresholds; the rank pattern of the top identity's extra photos supplies the decision signal.
- With margin-based-loss matchers such as AdaFace and TransFace, the method keeps high accuracy on degraded probes, so it can be used in surveillance-style settings where blur and low resolution are common; older embeddings like FaceNet are not sufficient.
- Accuracy stays within a few points across the four demographic groups tested, so the approach does not introduce a large demographic disparity in this In/Out decision.
- In-gallery and out-of-gallery training data can be generated automatically from any multi-image-per-identity gallery by running each probe twice, once with and once without its own images in the gallery.
- The method needs only about four enrolled images per identity; variable enrollment sizes can be handled by sampling to a fixed rank-vector size and averaging repeated samples.
Reading between the lines
- An untested extension is combining the rank-pattern classifier with a traditional score threshold; thresholding and rank patterns may be complementary, and a fused decision could raise precision in near-boundary cases.
- The contrast between FaceNet and margin-based matchers suggests a diagnostic: the method's accuracy should track how tightly the embedding loss packs same-identity images in rank space; one could measure rank displacement across models trained with increasing angular margins.
- The same idea—using the ranks of the top match's other enrolled samples—should transfer to other multi-sample biometrics or to re-identification, but the paper only demonstrates faces; a test on fingerprints or person re-identification would show whether the signal is general.
- A cautious deployment reading: the classifier gives a continuous confidence score, so it could be used to flag low-confidence hits for human review rather than as a binary arrest trigger; the paper reports classification accuracy, not downstream decision policy.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a threshold-free method for deciding whether the rank-one identity returned by a one-to-many face search is actually in the gallery or is an out-of-gallery false match. The signal is the vector of ranks of additional enrolled images of the rank-one identity: for a true in-gallery match these additional images tend to rank close behind rank-one, while for a false match they are displaced. The authors train a small MLP on this rank vector, using synthetic in-gallery/out-of-gallery training data constructed from MORPH and ND-MFAD by including or excluding the probe identity's own images from the gallery. Experiments with FaceNet, ArcFace, AdaFace, and TransFace under original, blurred, downsampled, turbulence-distorted, and sunglasses-wearing probes report accuracies up to 98.58%, with strong performance for margin-based modern matchers and degraded performance for FaceNet. The method is compared favorably against score thresholding, mean/median classifiers, and gallery-image fusion.
Significance. If the results are valid, the paper offers a genuinely new signal for open-set face identification that does not require score threshold calibration and leverages the multi-image enrollment already common in operational galleries. Strengths include evaluation on two datasets with different acquisition conditions, four matchers spanning pre-margin and margin-based architectures, an explicit degradation analysis, a demographic breakdown, and comparison to several baselines. The core weakness is that the evaluation protocol does not guarantee the training and test sets are identity-disjoint, which is essential for supporting the claim that the classifier generalizes to genuinely unknown probe identities. The paper is clearly written and the idea is well motivated, but the headline accuracy figures must be re-derived under an identity-disjoint split before the central claim can be accepted.
major comments (2)
- [Section 5, 'Train-Test Rank Data Curation'] The split into 'disjoint 80-20 training and testing sets' is not specified as identity-disjoint. Each probe identity contributes an In-gallery sample (its images in the gallery) and an Out-of-gallery sample (its images removed). If the same identity appears in both training and testing galleries, the MLP may learn identity-specific rank patterns rather than a generalizable In/Out signal, and the reported accuracies (e.g., 98.58% in Table 1) may not reflect performance on unseen identities. This directly affects the central claim about 'unknown probe image'. Please re-run the evaluation with an identity-disjoint split (all samples of a given identity restricted to either train or test) and report the resulting accuracies, or explicitly justify why sample-level disjointness is sufficient.
- [Section 6, Tables 1 and 2] Only a single accuracy value is reported per condition and demographic group, with no variance, confidence intervals, or number of runs. The paper concludes that accuracy is 'similar across demographics' based on differences of a few percentage points (e.g., 98.58 vs 97.24 for original probes, AdaFace). Without variance estimates or repeated-seed statistics, these differences are not interpretable. Please report mean and standard deviation across cross-validation folds or multiple random seeds, and state the number of test samples per group so the reader can assess the precision of the demographic comparison.
minor comments (5)
- [Section 5, 'Rank Cardinality for Effective Learning'] The argument that a two-dimensional output requires at least three input dimensions to avoid 'trivial mappings' is handwavy. Figure 5 is cited as the empirical justification, but the figure caption does not describe the experimental setup or whether the plateau is statistically robust. Please clarify what is plotted and add error bars.
- [Section 5, 'Data Augmentation for Robust Classification'] The feature permutation augmentation is reasonable if the three additional ranks are exchangeable, but this should be stated explicitly. If the enrolled images have a fixed order (e.g., chronological), permuting would destroy a potentially meaningful signal.
- [Section 8, 'Handling Variable Image Enrollment Conditions'] The paper claims the method 'naturally generalizes' to variable enrollment sizes, but all experiments use exactly four enrolled images. The proposed per-enrollment-size classifiers and repeated-sampling procedure are not evaluated. This is a practical claim that needs at least a small experiment or should be phrased as future work.
- [Author affiliation, title page] Typo: 'Florida Insitute of Technology' should be 'Florida Institute of Technology'.
- [General] No code or model weights are provided. Given the simplicity of the classifier, releasing code would aid reproducibility.
Circularity Check
No significant circularity: rank features and ground-truth labels are independent, and the classifier is a standard supervised model validated on held-out data.
full rationale
The paper's derivation chain is: (i) rank features are direct outputs of the face matcher for the rank-one identity's additional enrolled images; (ii) In/OOG labels are ground-truth identity membership of the probe relative to the gallery; (iii) an MLP is trained on these features with labels from a disjoint 80-20 split and 10-fold CV. There is no equation in which a fitted constant is renamed a prediction, no target quantity defined in terms of the feature, and no uniqueness theorem invoked from the authors' prior work. The central hypothesis (Section 4) is explicitly stated as a hypothesis and supported by empirical rank-distribution plots (Figure 3), not derived from a self-citation. The self-citations ([9], [29], [42]) are used for experimental parameters (blur sigma, same-demographic sampling, MFAD data) and are not load-bearing for the central claim. The method is benchmarked against thresholding, statistical classifiers, and CAFace fusion, providing external comparisons. The only notable caveat—the lack of explicit identity-disjoint train/test splitting—is a potential evaluation-validity issue, not circularity. Thus no circular step is exhibited.
Assumptions & free parameters
free parameters (2)
- Rdin (rank vector dimension) =
3
- Gallery enrollment size =
4 images per identity
assumptions (4)
- domain assumption Additional enrolled images of the rank-one identity are ranked in the 1-to-many search and these ranks are informative for In/Out classification.
- domain assumption In the In-gallery scenario the rank-one identity is almost always correct for modern matchers.
- domain assumption Same-demographic gallery sampling reflects real-world false arrest scenarios.
- domain assumption Deployment galleries have multiple images per identity.
Cite this review
Pith. "Pith review of Are you In or Out (of gallery)? Wisdom from the Same-Identity Crowd." pith.science (2026). https://pith.science/paper/BULV6TN4
@misc{pith2026250806357,
author = {Pith},
title = {Pith review of: Are you In or Out (of gallery)? Wisdom from the Same-Identity Crowd},
year = {2026},
howpublished = {\url{https://pith.science/paper/BULV6TN4}},
note = {Machine review of arXiv:2508.06357}
}
read the original abstract
A central problem in one-to-many facial identification is that the person in the probe image may or may not have enrolled image(s) in the gallery; that is, may be In-gallery or Out-of-gallery. Past approaches to detect when a rank-one result is Out-of-gallery have mostly focused on finding a suitable threshold on the similarity score. We take a new approach, using the additional enrolled images of the identity with the rank-one result to predict if the rank-one result is In-gallery / Out-of-gallery. Given a gallery of identities and images, we generate In-gallery and Out-of-gallery training data by extracting the ranks of additional enrolled images corresponding to the rank-one identity. We then train a classifier to utilize this feature vector to predict whether a rank-one result is In-gallery or Out-of-gallery. Using two different datasets and four different matchers, we present experimental results showing that our approach is viable for mugshot quality probe images, and also, importantly, for probes degraded by blur, reduced resolution, atmospheric turbulence and sunglasses. We also analyze results across demographic groups, and show that In-gallery / Out-of-gallery classification accuracy is similar across demographics. Our approach has the potential to provide an objective estimate of whether a one-to-many facial identification is Out-of-gallery, and thereby to reduce false positive identifications, wrongful arrests, and wasted investigative time. Interestingly, comparing the results of older deep CNN-based face matchers with newer ones suggests that the effectiveness of our Out-of-gallery detection approach emerges only with matchers trained using advanced margin-based loss functions.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
https://github.com/davidsandberg/ facenet/tree/master
facenet. https://github.com/davidsandberg/ facenet/tree/master. 4
-
[2]
https:// github.com/deepinsight/insightface/tree/ master/model_zoo
Insightface: 2d and 3d face analysis project. https:// github.com/deepinsight/insightface/tree/ master/model_zoo. 4
-
[3]
Morph dataset. https : / / uncw . edu / research / innovation/commercialization/technology- portfolio/morph. 3
-
[4]
https://github.com/DanJun6737/ TransFace
Transface. https://github.com/DanJun6737/ TransFace. 4
-
[5]
img2pose: Face alignment and detection via 6dof, face pose estimation
V ´ıtor Albiero, Xingyu Chen, Xi Yin, Guan Pang, and Tal Hassner. img2pose: Face alignment and detection via 6dof, face pose estimation. InComputer Vision and Pattern Recog- nition (CVPR), 2021. 4
work page 2021
-
[6]
Towards open world recognition
Abhijit Bendale and Terrance E Boult. Towards open world recognition. In Computer Vision and Pattern Recognition (CVPR), pages 1893–1902, 2015. 3
work page 1902
-
[7]
Towards open set deep networks
Abhijit Bendale and Terrance E Boult. Towards open set deep networks. In Computer Vision and Pattern Recognition (CVPR), pages 1563–1572, 2016. 2, 3
work page 2016
-
[8]
Unconstrained face recognition: Identifying a person of interest from a media collection
Lacey Best-Rowden, Hu Han, Charles Otto, Brendan F Klare, and Anil K Jain. Unconstrained face recognition: Identifying a person of interest from a media collection. In Transactions on Information, Forensics and Security (TIFS), pages 2144–2157. IEEE, 2014. 5
work page 2014
Show all 45 references
-
[9]
Impact of blur and resolution on demo- graphic disparities in 1-to-many facial identification
Aman Bhatta, Gabriella Pangelinan, Michael C King, and Kevin W Bowyer. Impact of blur and resolution on demo- graphic disparities in 1-to-many facial identification. InWin- ter Conference on Applications of Computer Vision Work- shops (WACVW), pages 412–420, 2024. 7
2024
-
[10]
Deep hetero- geneous feature fusion for template-based face recognition
Navaneeth Bodla, Jingxiao Zheng, Hongyu Xu, Jun-Cheng Chen, Carlos Castillo, and Rama Chellappa. Deep hetero- geneous feature fusion for template-based face recognition. In Winter Conference on Applications of Computer Vision (WACV), pages 586–595. IEEE, 2017. 3
2017
-
[11]
Operational open-set recognition and post- max refinement
Steve Cruz, Ryan Rabinowitz, Manuel G ¨unther, and Ter- rance E Boult. Operational open-set recognition and post- max refinement. In European Conference on Computer Vi- sion (ECCV), pages 475–492. Springer, 2024. 3
2024
-
[12]
Transface: Calibrating trans- former training for face recognition from a data-centric per- spective
Jun Dan, Yang Liu, Haoyu Xie, Jiankang Deng, Haoran Xie, Xuansong Xie, and Baigui Sun. Transface: Calibrating trans- former training for face recognition from a data-centric per- spective. In International Conference on Computer Vision (ICCV), pages 20642–20653, 2023. 4
2023
-
[13]
Arcface: Additive angular margin loss for deep face recognition
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In Computer Vision and Pattern Recogni- tion (CVPR), 2019. 4
2019
-
[14]
Demographic bias in biometrics: A survey on an emerging challenge
Pawel Drozdowski, Christian Rathgeb, Antitza Dantcheva, Naser Damer, and Christoph Busch. Demographic bias in biometrics: A survey on an emerging challenge. In Transac- tions on Technology and Society (TTS), pages 89–103, 2020. 3
2020
-
[15]
Open-set face recognition-based visitor interface sys- tem
Hazım Kemal Ekenel, Lorant Szasz-Toth, and Rainer Stiefel- hagen. Open-set face recognition-based visitor interface sys- tem. In International conference on computer vision systems, pages 43–52. Springer, 2009. 5
2009
-
[16]
A false facial recognition match sent this innocent black man to jail
John General and Jon Sarlin. A false facial recognition match sent this innocent black man to jail. CNN Business,
-
[17]
Re- cent advances in open set recognition: A survey
Chuanxing Geng, Sheng-Jun Huang, and Songcan Chen. Re- cent advances in open set recognition: A survey. In Trans- actions on Pattern Analysis and Machine Intelligence (T- PAMI), pages 3976–3994. IEEE, 2021. 3
2021
-
[18]
Face recog- nition vendor test (FRVT) part 2: Identification
Patrick Grother, Mei Ngan, and Kayee Hanaoka. Face recog- nition vendor test (FRVT) part 2: Identification. NISTIR, 8271, 2019. 2, 3, 4, 8
2019
-
[19]
Toward open-set face recognition
Manuel G ¨unther, Steven A Cruz, Ethan M Rudd, and Ter- rance E Boult. Toward open-set face recognition. In Com- puter Vision and Pattern Recognition Workshops (CVPRW),
-
[20]
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Computer Vision and Pattern Recognition (CVPR) , pages 770–778,
-
[21]
T. Hoggins. ‘racist and sexist’ facial recogni- tion cameras could lead to false arrests, 2019. https://www.telegraph.co.uk/technology/2019/12/20/racist- sexist-facial-recognition-cameras-could-lead-false-arrests/. 1, 2, 4
2019
-
[22]
Proxyfu- sion: Face feature aggregation through sparse experts
Bhavin Jawade, Alexander Stone, Deen Dayal Mohan, Xiao Wang, Srirangaraj Setlur, and Venu Govindaraju. Proxyfu- sion: Face feature aggregation through sparse experts. InAd- vances in Neural Information Processing Systems (NeurIPS), pages 70130–70147, 2025. 3
2025
-
[23]
Cluster and aggregate: Face recognition with large probe set
Minchul Kim, Feng Liu, Anil K Jain, and Xiaoming Liu. Cluster and aggregate: Face recognition with large probe set. In Advances in Neural Information Processing Systems (NeurIPS), pages 36054–36066, 2022. 3, 8
2022
-
[24]
Klare et al
B. Klare et al. Pushing the frontiers of unconstrained face detection and recognition. iarpa janus benchmark a (ijb-a). In CVPR, 2015. 3
2015
-
[25]
Pushing the frontiers of uncon- strained face detection and recognition: Iarpa janus bench- mark a
Brendan F Klare, Ben Klein, Emma Taborsky, Austin Blan- ton, Jordan Cheney, Kristen Allen, Patrick Grother, Alan Mah, and Anil K Jain. Pushing the frontiers of uncon- strained face detection and recognition: Iarpa janus bench- mark a. In Computer Vision and Pattern Recognition...
1931
-
[26]
Open set face recognition us- ing transduction
Fayin Li and Harry Wechsler. Open set face recognition us- ing transduction. In Transactions on Pattern Analysis and Machine Intelligence (T-PAMI) , pages 1686–1697. IEEE,
-
[27]
A bench- mark study of large-scale unconstrained face recognition
Shengcai Liao, Zhen Lei, Dong Yi, and Stan Z Li. A bench- mark study of large-scale unconstrained face recognition. In International Joint Conference on Biometrics (IJCB), pages 1–8. IEEE, 2014. 1, 5
2014
-
[28]
Accel- erating atmospheric turbulence simulation via learned phase- to-space transform
Zhiyuan Mao, Nicholas Chimitt, and Stanley H Chan. Accel- erating atmospheric turbulence simulation via learned phase- to-space transform. In International Conference on Com- puter Vision (ICCV), pages 14759–14768, 2021. 3 9
2021
-
[29]
Analyzing the impact of de- mographic and operational variables on 1-to-many face id search
Gabriella Pangelinan, Aman Bhatta, Haiyu Wu, Michael C King, and Kevin W Bowyer. Analyzing the impact of de- mographic and operational variables on 1-to-many face id search. In Transactions on Technology and Society (TTS) . IEEE, 2024. 3, 4
2024
-
[30]
Face Recog- nition Vendor Test (FRVT) Part 3: Demographic Effects
Mei Ngan Patrick Grother and Kayee Hanaoka. Face Recog- nition Vendor Test (FRVT) Part 3: Demographic Effects. In NIST IR 8280, 2019. 2
2019
-
[31]
Open set fingerprint spoof detection across novel fabrication materi- als
Ajita Rattani, Walter J Scheirer, and Arun Ross. Open set fingerprint spoof detection across novel fabrication materi- als. In Transactions on Information, Forensics and Security (TIFS), pages 2434–2444. IEEE, 2015. 2
2015
-
[32]
Williams v
ACLU Reports. Williams v. city of detroit. https:// www.aclu.org/cases/williams- v- city- of- detroit - face - recognition - false - arrest,
-
[33]
Morph: A longitudinal image database of normal adult age-progression
Karl Ricanek and Tamirat Tesafaye. Morph: A longitudinal image database of normal adult age-progression. In Auto- matic Face and Gesture Recognition (AFGR), 2006. 3
2006
-
[34]
On the effect of atmo- spheric turbulence in the feature space of deep face recogni- tion
Wes Robbins and Terrance E Boult. On the effect of atmo- spheric turbulence in the feature space of deep face recogni- tion. In Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1618–1626, 2022. 3
2022
-
[35]
Meta-recognition: The theory and prac- tice of recognition score analysis
Walter J Scheirer, Anderson Rocha, Ross J Micheals, and Terrance E Boult. Meta-recognition: The theory and prac- tice of recognition score analysis. InTransactions on Pattern Analysis and Machine Intelligence (T-PAMI) , pages 1689–
-
[36]
To- ward open set recognition
Walter J Scheirer, Lalit P Jain, and Terrance E Boult. To- ward open set recognition. In Transactions on Pattern Anal- ysis and Machine Intelligence (T-PAMI), pages 1757–1772. IEEE, 2013. 2, 3
2013
-
[37]
Prob- ability models for open set recognition
Walter J Scheirer, Lalit P Jain, and Terrance E Boult. Prob- ability models for open set recognition. In Transactions on Pattern Analysis and Machine Intelligence (T-PAMI), pages 2317–2324. IEEE, 2014. 3
2014
-
[38]
Facenet: A unified embedding for face recognition and clus- tering
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clus- tering. In Computer Vision and Pattern Recognition (CVPR),
-
[39]
Probabilistic face embeddings
Yichun Shi and Anil K Jain. Probabilistic face embeddings. In Computer Vision and Pattern Recognition (CVPR), pages 6902–6911, 2019. 3
2019
-
[40]
Video-based face recognition on real-world data
Johannes Stallkamp, Hazim K Ekenel, and Rainer Stiefelha- gen. Video-based face recognition on real-world data. In International Conference on Computer Vision (ICCV), pages 1–8. IEEE, 2007. 5
2007
-
[41]
Inception-v4, inception-resnet and the im- pact of residual connections on learning
Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander Alemi. Inception-v4, inception-resnet and the im- pact of residual connections on learning. InAAAI Conference on Artificial Intelligence (AAAI), 2017. 4
2017
-
[42]
Impact of sunglasses on one-to-many facial identi- fication accuracy
Sicong Tian, Haiyu Wu, Michael C King, and Kevin W Bowyer. Impact of sunglasses on one-to-many facial identi- fication accuracy. In International Conference on Automatic Face and Gesture Recognition (FG), 2025. 3
2025
-
[43]
Open-set face recognition with maximal entropy and objectosphere loss
Rafael Henrique Vareto, Yu Linghu, Terrance Edward Boult, William Robson Schwartz, and Manuel G ¨unther. Open-set face recognition with maximal entropy and objectosphere loss. Image and Vision Computing, 141:104862, 2024. 3
2024
-
[44]
Neural aggregation network for video face recognition
Jiaolong Yang, Peiran Ren, Dongqing Zhang, Dong Chen, Fang Wen, Hongdong Li, and Gang Hua. Neural aggregation network for video face recognition. In Winter Conference on Applications of Computer Vision (WACV), pages 4362–4371,
-
[2021]
https://www.cnn.com/2021/04/29/tech/nijeer-parks- facial-recognition-police-arrest/index.html. 2
2021
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.