REVIEW 3 major objections 5 minor 16 references
SLAPBench, the first benchmark for MLLM-based four-finger SLAP fingerprint verification, shows prompting governs collapse while model capability governs discrimination.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 23:05 UTC pith:YNF2GRXG
load-bearing objection A useful first SLAP MLLM baseline with a real protocol confound that touches every similarity-scoring AUC, not just the perfect Qwen3 result. the 3 major comments →
SLAPBench: Benchmarking Multimodal Large Language Models for Four-Finger SLAP Fingerprint Verification
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
On SLAPBench, the first SLAP-specific MLLM verification benchmark, the authors establish that prompting format governs whether verification collapses while model capability governs whether discrimination can be recovered. Under binary 'same person / different people' prompts, five of ten configurations collapse to near-100% false accept rate—all four open-source models under task-description prompting, and Gemma-3-12B even under zero-shot—while Claude Opus 4.8 resists collapse under both prompts and achieves the strongest binary result (FAR 20.2%). A continuous similarity-scoring prompt (0–100) eliminates collapse across the open-source models and reveals a wide capability spread, from Claud
What carries the argument
The central mechanism is the pair-construction protocol plus the prompt design. SLAPBench builds 7,832 verification pairs from NIST SD302b: 176 mated pairs, each formed by pairing a subject's native 1000 PPI SLAP capture with its 500 PPI downsampled twin, and 7,656 non-mated pairs between different subjects at 500 PPI. Three prompting strategies are compared—zero-shot, task-description, and continuous similarity scoring (an integer 0–100 confidence). The similarity-scoring prompt is the device that breaks positive-bias collapse by asking for calibrated confidence instead of a binary commit; the matched-resolution control, which rebuilds non-mated pairs as 1000-vs-500 PPI so both classes shar
Load-bearing premise
SD302b provides only one SLAP capture per finger position, so every 'same person' pair in the benchmark is the same scan shown at two resolutions; within this dataset, a model can never be proved to be matching identity rather than duplicated images.
What would settle it
A repeat-capture SLAP dataset: build mated pairs from two independent live-scan impressions of the same subject at the same resolution. If Qwen3-VL-8B's AUC stays at 1.000, the perfect score reflects genuine identity discrimination; if it drops, the SLAPBench perfect score was near-duplicate detection and the single-capture mated-pair design was the cue being read.
If this is right
- Binary forced-choice prompting is currently unreliable for MLLM SLAP verification: most open-source models answer 'same person' on nearly every pair, and adding a natural-language task description makes collapse worse rather than better.
- Continuous similarity scoring removes collapse in all open-source models tested, but does not confer discrimination—architecture sets the ceiling, with results ranging from functional to inverted to compressed.
- A sufficiently capable model can overcome collapse even in the binary format, so collapse is a capability threshold, not a fixed property of the question format.
- Qwen3-VL-8B's perfect separation (AUC 1.000) should not be taken as operational verification capability; within SD302b the near-duplicate cue cannot be distinguished from identity matching.
- Early evidence suggests demographic disparity grows as discrimination weakens, though subgroup sizes limit this to an initial fairness probe.
Where Pith is reading between the lines
- Editorial inference: if near-duplicate detection is what produced Qwen3-VL-8B's perfect score, then any SLAP-style benchmark built from single-capture databases will systematically overstate MLLM verification ability; a repeat-capture dataset is the decisive next test.
- Editorial inference: the finding that task-description text increases collapse likely generalizes beyond SLAP and beyond fingerprints—naturalistic domain context can act as a negative prior for any biometric comparison task where the model already has a 'same person' bias.
- Editorial inference: InternVL3-8B's inverted scores suggest some MLLMs encode texture contrast rather than identity; if that signal is stable, reversing the score direction could turn an apparent failure into usable discrimination—but only if the inversion is consistent across datasets.
- Editorial inference: the similarity-scoring prompt is a cheap, model-agnostic probe that could be applied to existing single-finger fingerprint benchmarks to test whether the same collapse/unlock pattern holds there, or to the SLAPBench extension tasks (finger counting, localization, quality) proposed in the paper.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces SLAPBench, described as the first benchmark for evaluating multimodal large language models (MLLMs) on four-finger SLAP fingerprint verification. It builds a fixed pair manifest from NIST SD302b with 88 clean subjects, yielding 176 mated pairs formed by cross-resolution (1000 vs. 500 PPI) versions of the same capture and 7,656 non-mated same-resolution pairs. Five models are evaluated under zero-shot, task-description, and similarity-scoring prompts. The authors report that task-description prompting collapses all open-source models to near-100% FAR, while Claude Opus 4.8 resists collapse. Similarity scoring removes collapse and exposes a performance ranking: Qwen3-VL-8B achieves AUC=1.000 (interpreted as a near-duplicate diagnostic), Claude Opus 4.8 AUC=0.953, Gemma-3-12B AUC=0.837, InternVL3-8B AUC=0.590, and Qwen2.5-VL-7B AUC=0.567. A demographic fairness probe is included. The paper is explicitly careful about the Qwen3 result, states that near-duplicate detection cannot be excluded, and releases code, prompts, and raw scores for reproducibility.
Significance. If the claims were fully established, SLAPBench would be a useful and timely benchmark: it is reproducible (fixed manifest, verbatim prompts, released result CSVs), it systematically documents positive-bias collapse on SLAP data, and it offers a cautionary protocol audit showing how a benchmark can manufacture a perfect score. The collapse findings and the prompt-format dependence are valuable and are not undermined by the dataset's mated-pair construction. However, the central interpretive claim—that similarity scoring reveals 'model capability governs discrimination'—is not fully supported because every mated pair is the same capture rendered twice. The near-duplicate cue is not limited to Qwen3-VL-8B; it potentially inflates all similarity-scoring AUCs, including those of Claude and Gemma. Without a non-duplicate mated-pair control or a classical matcher baseline, the paper's capability ranking remains a measurement of separation on a synthetic near-duplicate protocol rather than of fingerprint verification capability.
major comments (3)
- [§4.1, §5.2, §6.5] The mated pairs are, by construction, one capture rendered at 1000 and 500 PPI, while non-mated pairs are different captures. The matched-resolution control in §6.3 removes only the resolution asymmetry and is run only for Qwen3-VL-8B; the same-capture near-duplicate signal remains for every model. Therefore the AUCs for Claude Opus 4.8 (0.953) and Gemma-3-12B (0.837) in Table 1 may reflect low-level duplicate-image detection rather than identity verification. The conclusion that 'model capability governs discrimination' is not established without a classical matcher baseline or a repeat-capture dataset to calibrate the effect of near-duplication. The paper's limitation statement in §6.5 correctly says the single-capture design leaves near-duplicate detection in place, but it applies that caveat primarily to Qwen3-VL-8B; the same caveat applies to the other models.
- [§6.3] The matched-resolution control is reported only for Qwen3-VL-8B and only for FRGP 13. The paper explicitly acknowledges that near-duplicate detection 'cannot be excluded within SD302b.' Since this confound is structural to the dataset, the 'functional' interpretation of Claude and Gemma in §5.2 is not supported unless the same control is applied to those models—and even then, the control cannot remove the same-capture correspondence. The central claim should be reframed as separation on a synthetic near-duplicate protocol, with identity-verification capability left open. As written, the §7 conclusion overreaches.
- [§5.2, Table 1] No uncertainty quantification is reported for AUC or EER. With only 176 mated pairs and discrete 0–100 scores, the apparent ordering (Claude 0.953 vs. Gemma 0.837 vs. Qwen2.5 0.567) may be within sampling noise. This matters especially for the low-AUC models, where the score distributions are highly compressed or inverted. The authors should provide bootstrap confidence intervals for AUC/EER, or at least exact binomial intervals, before interpreting the ranking as 'model capability.'
minor comments (5)
- [Table 1] Collapsed configurations are marked by red/green cell colors. Please add textual markers (e.g., asterisks or bold) so the table is interpretable without color.
- [§6.3] The sentence 'no other model (not even Claude at AUC=0.953) reproduces the result' is presented as evidence against memorization. This is not conclusive, since different prompts can change the use of memorized information. Consider softening or removing.
- [§5.3, Table S1] Subgroup AUCs are computed on different non-mated distributions (because a non-mated pair is counted only when both subjects share the attribute). The authors do caution about small groups, but the comparability caveat could be stated directly in the main text.
- [§4.1] The matched-resolution control is described in prose. Reporting full metrics for all models in a supplementary table would make the control easier to assess.
- [References] Some references are dated 2026 and several are arXiv preprints; verify that all bibliographic details (especially FPBench and FaceXBench) are stable or update to published versions if available.
Circularity Check
No circularity: SLAPBench reports empirical MLLM measurements; the acknowledged near-duplicate mated-pair issue is a validity threat, not a definitional reduction.
full rationale
SLAPBench is an evaluation study with no fitted parameters, no learned model, and no derivation that assumes its own conclusion. The reported AUC/EER values are empirical outputs from frozen MLLMs on fixed image pairs; the mated/non-mated labels come from NIST SD302b subject metadata, not from the model outputs. The mated-pair construction is indeed problematic: because SD302b has one capture per finger position, every mated pair is the same capture at 1000 and 500 PPI, making mated pairs near-duplicates (§4.1, §6.3). The authors explicitly acknowledge this and do not claim capability for Qwen3-VL-8B's AUC=1.000, writing that 'near-duplicate detection remains and cannot be excluded within SD302b' (§6.3, §6.5). The matched-resolution control is run only for Qwen3-VL-8B, so the other models' AUCs are not fully controlled and the sentence 'the ranking is not an artefact of resolution mismatch' (§4.1) is under-supported. However, under-support and confounding are not circularity: the numbers are not forced by the benchmark definition, and no fitted input is renamed as a prediction. The only self-citation, ref [7], is a peripheral segmentation citation and is not load-bearing. Thus no circular step can be exhibited, and the appropriate score is 0.
Axiom & Free-Parameter Ledger
axioms (4)
- domain assumption NIST SD302b identity labels are correct and participants' identities are unique across captures.
- domain assumption A mated pair formed from one capture at 1000 PPI and its 500 PPI downsampled twin is a valid proxy for an operational mated pair.
- domain assumption Resizing to 448x448 with grayscale/contrast preprocessing preserves identity-relevant ridge detail for all models.
- domain assumption Model outputs can be interpreted as comparable confidence scores on a shared 0-100 scale after parsing and clamping.
read the original abstract
Four-finger SLAP fingerprints are flat live-scan impressions of the index, middle, ring, and little fingers of one hand, used for identity verification in border control and law enforcement. No benchmark has evaluated whether multimodal large language models (MLLMs) can verify identity from SLAP images. We introduce SLAPBench, the first benchmark for MLLM-based four-finger SLAP fingerprint verification, built from NIST SD302b with 7,832 pairs (176 mated, 7,656 non-mated). We evaluate four open-source MLLMs (InternVL3-8B, Qwen2.5-VL-7B, Qwen3-VL-8B, Gemma-3-12B) and the proprietary Claude Opus 4.8 under zero-shot, task-description, and similarity-scoring prompts. Prompting governs verification behavior. Task-description prompting collapses all four open-source models to near-100% False Accept Rate (FAR), and Gemma-3-12B collapses under zero-shot as well; Claude Opus 4.8 alone resists collapse under both binary prompts, giving the best binary result (FAR = 20.2%). Similarity scoring removes collapse across the open-source models and exposes wide capability gaps: Claude reaches AUC = 0.953 and Gemma-3-12B 0.837, while InternVL3-8B is inverted (AUC = 0.590) and Qwen2.5-VL-7B near random (0.567). Qwen3-VL-8B attains perfect separation (AUC = 1.000), which we treat as a diagnostic rather than as capability: SD302b holds one SLAP capture per finger position, so mated pairs are cross-resolution. A matched-resolution control leaves the perfect score intact, ruling out the resolution shortcut; what cannot be excluded within SD302b is near-duplicate detection, since a mated pair is one capture rendered twice. A fairness probe over gender, race, and age suggests disparity grows as discrimination weakens. SLAPBench establishes the first SLAP-specific MLLM baseline and shows that prompting governs collapse while model capability governs discrimination.
Figures
Reference graph
Works this paper leans on
-
[1]
anthropic
Anthropic: The Claude model family (2025),https : / / www . anthropic . com / claude, claude Opus 4.8, accessed via the Anthropic API
2025
-
[2]
Duan, H., Fang, X., Yang, J., Zhao, X., Qiao, Y., Li, M., Agarwal, A., Chen, Z., Chen, L., Liu, Y., Ma, Y., Sun, H., Zhang, Y., Lu, S., Wong, T.H., Wang, W., Zhou, P., Li, X., Fu, C., Cui, J., Chen, J., Song, E., Mao, S., Ding, S., Liang, T., Zhang, Z., Dong, X., Zang, Y., Zhang, P., Wang, J., Lin, D., Chen, K.: Vlmevalkit: An open-source toolkit for eval...
Pith/arXiv arXiv 2025
-
[3]
Engelsma, J.J., Cao, K., Jain, A.K.: Learning a fixed-length fingerprint represen- tation. IEEE Transactions on Pattern Analysis and Machine Intelligence43(6), 1981–1997 (2021).https://doi.org/10.1109/TPAMI.2019.2956523
arXiv 1981
-
[4]
Gavas, E., Banerjee, S., Hegde, C., Memon, N.: Fpbench: A comprehensive bench- mark of multimodal large language models for fingerprint analysis (2026),https: //arxiv.org/abs/2512.18073
Pith/arXiv arXiv 2026
-
[5]
Official model release and documen- tation
Gemma Team: Gemma 3 technical report (2025),https://huggingface.co/ google/gemma-3-12b-it, google DeepMind. Official model release and documen- tation. SLAPBench: MLLMs for Four-Finger SLAP Verification 15
2025
-
[6]
Maltoni, D., Maio, D., Jain, A.K., Feng, J.: Handbook of Fingerprint Recognition. Springer, Cham, 3rd edn. (2022).https://doi.org/10.1007/978-3-030-83624- 5,https://doi.org/10.1007/978-3-030-83624-5
-
[7]
Murshed, M.G.S., Bahmani, K., Schuckers, S., Hussain, F.: Deep age-invariant fingerprint segmentation system. IEEE Transactions on Biometrics, Behavior, and Identity Science7(3), 313–330 (2025).https://doi.org/10.1109/TBIOM.2024. 3506926
-
[8]
Narayan, K., VS, V., Patel, V.M.: Facexbench: Evaluating multimodal llms on face understanding (2026),https://arxiv.org/abs/2501.10360
arXiv 2026
-
[9]
National Institute of Standards and Technology: NIST Special Database 302: Nail to nail fingerprint challenge dry run dataset. Tech. rep., National Institute of Standards and Technology (2018),https://www.nist.gov/itl/iad/btg/nist- special-database-302, accessed: 2026-05-24
2018
-
[10]
neurotechnology.com/verifinger.html
Neurotechnology: VeriFinger fingerprint recognition SDK (2024),https://www. neurotechnology.com/verifinger.html
2024
-
[11]
Qwen Team: Qwen2.5-vl (Jan 2025),https://qwenlm.github.io/blog/qwen2.5- vl/, official model release page for Qwen2.5-VL
2025
-
[12]
Official model release and documentation
Qwen Team: Qwen3-vl technical report (2025),https://huggingface.co/Qwen/ Qwen3-VL-8B-Instruct, alibaba Group. Official model release and documentation
2025
-
[13]
Shahreza, H.O., George, A., Marcel, S.: Evaluating multimodal large language models for heterogeneous face recognition (2026),https://arxiv.org/abs/2601. 15406
2026
-
[14]
Shahreza, H.O., Marcel, S.: Benchmarking multimodal large language models for face recognition (2025),https://arxiv.org/abs/2510.14866
arXiv 2025
-
[15]
In: IEEE International Conference on Biometrics: Theory, Applications and Systems (BTAS)
Tang, Y., Gao, F., Feng, J., Liu, Y.: FingerNet: An unified deep network for fin- gerprint minutiae extraction. In: IEEE International Conference on Biometrics: Theory, Applications and Systems (BTAS). pp. 1–7 (2017).https://doi.org/10. 1109/BTAS.2017.8272679
arXiv 2017
-
[16]
Zhu, J., Wang, W., Chen, Z., Liu, Z., Ye, S., Gu, L., Tian, H., Duan, Y., Su, W., Shao, J., et al.: Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models (2025),https://arxiv.org/abs/2504.10479 16 B. Pyakurel and M. G. S. Murshed A Supporting Figures This appendix collects the supporting figures referenced in the m...
Pith/arXiv arXiv 2025
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.