Pith. sign in

REVIEW 3 major objections 5 minor 16 references

SLAPBench, the first benchmark for MLLM-based four-finger SLAP fingerprint verification, shows prompting governs collapse while model capability governs discrimination.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · deepseek-v4-flash

2026-08-01 23:05 UTC pith:YNF2GRXG

load-bearing objection A useful first SLAP MLLM baseline with a real protocol confound that touches every similarity-scoring AUC, not just the perfect Qwen3 result. the 3 major comments →

arxiv 2607.15517 v1 pith:YNF2GRXG submitted 2026-07-17 cs.CV cs.AI

SLAPBench: Benchmarking Multimodal Large Language Models for Four-Finger SLAP Fingerprint Verification

classification cs.CV cs.AI
keywords SLAP fingerprintfingerprint verificationmultimodal large language modelsbiometric benchmarkpositive-bias collapsesimilarity scoringNIST SD302bverification protocol
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces SLAPBench, the first benchmark for evaluating multimodal large language models on four-finger SLAP fingerprint verification, built from NIST SD302b with 7,832 pairs. Its central result is that binary forced-choice prompting—especially with a task-description preamble—collapses most open-source models into accepting nearly every pair as the same person, while Claude Opus 4.8 resists collapse. Switching to a continuous 0–100 similarity-scoring prompt removes collapse and exposes wide capability gaps: Claude reaches AUC 0.953 and Gemma-3-12B 0.837, while InternVL3-8B is inverted (0.590) and Qwen2.5-VL-7B near random (0.567). The perfect AUC 1.000 of Qwen3-VL-8B is deliberately read as a protocol diagnostic, not a capability claim, because mated pairs in SD302b are one capture rendered at two resolutions, leaving near-duplicate detection possible. Why it matters: prompt format, not just model power, determines whether MLLM verification behavior is meaningful, and near-perfect scores in biometric benchmarks warrant a protocol audit before belief.

Core claim

On SLAPBench, the first SLAP-specific MLLM verification benchmark, the authors establish that prompting format governs whether verification collapses while model capability governs whether discrimination can be recovered. Under binary 'same person / different people' prompts, five of ten configurations collapse to near-100% false accept rate—all four open-source models under task-description prompting, and Gemma-3-12B even under zero-shot—while Claude Opus 4.8 resists collapse under both prompts and achieves the strongest binary result (FAR 20.2%). A continuous similarity-scoring prompt (0–100) eliminates collapse across the open-source models and reveals a wide capability spread, from Claud

What carries the argument

The central mechanism is the pair-construction protocol plus the prompt design. SLAPBench builds 7,832 verification pairs from NIST SD302b: 176 mated pairs, each formed by pairing a subject's native 1000 PPI SLAP capture with its 500 PPI downsampled twin, and 7,656 non-mated pairs between different subjects at 500 PPI. Three prompting strategies are compared—zero-shot, task-description, and continuous similarity scoring (an integer 0–100 confidence). The similarity-scoring prompt is the device that breaks positive-bias collapse by asking for calibrated confidence instead of a binary commit; the matched-resolution control, which rebuilds non-mated pairs as 1000-vs-500 PPI so both classes shar

Load-bearing premise

SD302b provides only one SLAP capture per finger position, so every 'same person' pair in the benchmark is the same scan shown at two resolutions; within this dataset, a model can never be proved to be matching identity rather than duplicated images.

What would settle it

A repeat-capture SLAP dataset: build mated pairs from two independent live-scan impressions of the same subject at the same resolution. If Qwen3-VL-8B's AUC stays at 1.000, the perfect score reflects genuine identity discrimination; if it drops, the SLAPBench perfect score was near-duplicate detection and the single-capture mated-pair design was the cue being read.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Binary forced-choice prompting is currently unreliable for MLLM SLAP verification: most open-source models answer 'same person' on nearly every pair, and adding a natural-language task description makes collapse worse rather than better.
  • Continuous similarity scoring removes collapse in all open-source models tested, but does not confer discrimination—architecture sets the ceiling, with results ranging from functional to inverted to compressed.
  • A sufficiently capable model can overcome collapse even in the binary format, so collapse is a capability threshold, not a fixed property of the question format.
  • Qwen3-VL-8B's perfect separation (AUC 1.000) should not be taken as operational verification capability; within SD302b the near-duplicate cue cannot be distinguished from identity matching.
  • Early evidence suggests demographic disparity grows as discrimination weakens, though subgroup sizes limit this to an initial fairness probe.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if near-duplicate detection is what produced Qwen3-VL-8B's perfect score, then any SLAP-style benchmark built from single-capture databases will systematically overstate MLLM verification ability; a repeat-capture dataset is the decisive next test.
  • Editorial inference: the finding that task-description text increases collapse likely generalizes beyond SLAP and beyond fingerprints—naturalistic domain context can act as a negative prior for any biometric comparison task where the model already has a 'same person' bias.
  • Editorial inference: InternVL3-8B's inverted scores suggest some MLLMs encode texture contrast rather than identity; if that signal is stable, reversing the score direction could turn an apparent failure into usable discrimination—but only if the inversion is consistent across datasets.
  • Editorial inference: the similarity-scoring prompt is a cheap, model-agnostic probe that could be applied to existing single-finger fingerprint benchmarks to test whether the same collapse/unlock pattern holds there, or to the SLAPBench extension tasks (finger counting, localization, quality) proposed in the paper.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces SLAPBench, described as the first benchmark for evaluating multimodal large language models (MLLMs) on four-finger SLAP fingerprint verification. It builds a fixed pair manifest from NIST SD302b with 88 clean subjects, yielding 176 mated pairs formed by cross-resolution (1000 vs. 500 PPI) versions of the same capture and 7,656 non-mated same-resolution pairs. Five models are evaluated under zero-shot, task-description, and similarity-scoring prompts. The authors report that task-description prompting collapses all open-source models to near-100% FAR, while Claude Opus 4.8 resists collapse. Similarity scoring removes collapse and exposes a performance ranking: Qwen3-VL-8B achieves AUC=1.000 (interpreted as a near-duplicate diagnostic), Claude Opus 4.8 AUC=0.953, Gemma-3-12B AUC=0.837, InternVL3-8B AUC=0.590, and Qwen2.5-VL-7B AUC=0.567. A demographic fairness probe is included. The paper is explicitly careful about the Qwen3 result, states that near-duplicate detection cannot be excluded, and releases code, prompts, and raw scores for reproducibility.

Significance. If the claims were fully established, SLAPBench would be a useful and timely benchmark: it is reproducible (fixed manifest, verbatim prompts, released result CSVs), it systematically documents positive-bias collapse on SLAP data, and it offers a cautionary protocol audit showing how a benchmark can manufacture a perfect score. The collapse findings and the prompt-format dependence are valuable and are not undermined by the dataset's mated-pair construction. However, the central interpretive claim—that similarity scoring reveals 'model capability governs discrimination'—is not fully supported because every mated pair is the same capture rendered twice. The near-duplicate cue is not limited to Qwen3-VL-8B; it potentially inflates all similarity-scoring AUCs, including those of Claude and Gemma. Without a non-duplicate mated-pair control or a classical matcher baseline, the paper's capability ranking remains a measurement of separation on a synthetic near-duplicate protocol rather than of fingerprint verification capability.

major comments (3)
  1. [§4.1, §5.2, §6.5] The mated pairs are, by construction, one capture rendered at 1000 and 500 PPI, while non-mated pairs are different captures. The matched-resolution control in §6.3 removes only the resolution asymmetry and is run only for Qwen3-VL-8B; the same-capture near-duplicate signal remains for every model. Therefore the AUCs for Claude Opus 4.8 (0.953) and Gemma-3-12B (0.837) in Table 1 may reflect low-level duplicate-image detection rather than identity verification. The conclusion that 'model capability governs discrimination' is not established without a classical matcher baseline or a repeat-capture dataset to calibrate the effect of near-duplication. The paper's limitation statement in §6.5 correctly says the single-capture design leaves near-duplicate detection in place, but it applies that caveat primarily to Qwen3-VL-8B; the same caveat applies to the other models.
  2. [§6.3] The matched-resolution control is reported only for Qwen3-VL-8B and only for FRGP 13. The paper explicitly acknowledges that near-duplicate detection 'cannot be excluded within SD302b.' Since this confound is structural to the dataset, the 'functional' interpretation of Claude and Gemma in §5.2 is not supported unless the same control is applied to those models—and even then, the control cannot remove the same-capture correspondence. The central claim should be reframed as separation on a synthetic near-duplicate protocol, with identity-verification capability left open. As written, the §7 conclusion overreaches.
  3. [§5.2, Table 1] No uncertainty quantification is reported for AUC or EER. With only 176 mated pairs and discrete 0–100 scores, the apparent ordering (Claude 0.953 vs. Gemma 0.837 vs. Qwen2.5 0.567) may be within sampling noise. This matters especially for the low-AUC models, where the score distributions are highly compressed or inverted. The authors should provide bootstrap confidence intervals for AUC/EER, or at least exact binomial intervals, before interpreting the ranking as 'model capability.'
minor comments (5)
  1. [Table 1] Collapsed configurations are marked by red/green cell colors. Please add textual markers (e.g., asterisks or bold) so the table is interpretable without color.
  2. [§6.3] The sentence 'no other model (not even Claude at AUC=0.953) reproduces the result' is presented as evidence against memorization. This is not conclusive, since different prompts can change the use of memorized information. Consider softening or removing.
  3. [§5.3, Table S1] Subgroup AUCs are computed on different non-mated distributions (because a non-mated pair is counted only when both subjects share the attribute). The authors do caution about small groups, but the comparability caveat could be stated directly in the main text.
  4. [§4.1] The matched-resolution control is described in prose. Reporting full metrics for all models in a supplementary table would make the control easier to assess.
  5. [References] Some references are dated 2026 and several are arXiv preprints; verify that all bibliographic details (especially FPBench and FaceXBench) are stable or update to published versions if available.

Circularity Check

0 steps flagged

No circularity: SLAPBench reports empirical MLLM measurements; the acknowledged near-duplicate mated-pair issue is a validity threat, not a definitional reduction.

full rationale

SLAPBench is an evaluation study with no fitted parameters, no learned model, and no derivation that assumes its own conclusion. The reported AUC/EER values are empirical outputs from frozen MLLMs on fixed image pairs; the mated/non-mated labels come from NIST SD302b subject metadata, not from the model outputs. The mated-pair construction is indeed problematic: because SD302b has one capture per finger position, every mated pair is the same capture at 1000 and 500 PPI, making mated pairs near-duplicates (§4.1, §6.3). The authors explicitly acknowledge this and do not claim capability for Qwen3-VL-8B's AUC=1.000, writing that 'near-duplicate detection remains and cannot be excluded within SD302b' (§6.3, §6.5). The matched-resolution control is run only for Qwen3-VL-8B, so the other models' AUCs are not fully controlled and the sentence 'the ranking is not an artefact of resolution mismatch' (§4.1) is under-supported. However, under-support and confounding are not circularity: the numbers are not forced by the benchmark definition, and no fitted input is renamed as a prediction. The only self-citation, ref [7], is a peripheral segmentation citation and is not load-bearing. Thus no circular step can be exhibited, and the appropriate score is 0.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The central claims rest on NIST labels, the same-capture mated-pair proxy, and the assumption that prompt/score outputs are comparable across models. No invented entities or fitted parameters are introduced; the main structural risk is the near-duplicate mated class.

axioms (4)
  • domain assumption NIST SD302b identity labels are correct and participants' identities are unique across captures.
    The benchmark's mated/non-mated ground truth comes entirely from dataset metadata (§3.1).
  • domain assumption A mated pair formed from one capture at 1000 PPI and its 500 PPI downsampled twin is a valid proxy for an operational mated pair.
    Needed to construct the 176 genuine pairs; the authors explicitly note this cannot be verified within SD302b (§4.1, §6.3).
  • domain assumption Resizing to 448x448 with grayscale/contrast preprocessing preserves identity-relevant ridge detail for all models.
    All inputs pass through identical preprocessing (§4.3); if downsampling destroyed fine ridge cues, scores would reflect image-level statistics rather than identity.
  • domain assumption Model outputs can be interpreted as comparable confidence scores on a shared 0-100 scale after parsing and clamping.
    Integer outputs from different models are treated as comparable confidence; cross-model calibration is not established (§4.4).

pith-pipeline@v1.3.0-alltime-deepseek · 13749 in / 13103 out tokens · 136686 ms · 2026-08-01T23:05:00.959497+00:00 · methodology

0 comments
read the original abstract

Four-finger SLAP fingerprints are flat live-scan impressions of the index, middle, ring, and little fingers of one hand, used for identity verification in border control and law enforcement. No benchmark has evaluated whether multimodal large language models (MLLMs) can verify identity from SLAP images. We introduce SLAPBench, the first benchmark for MLLM-based four-finger SLAP fingerprint verification, built from NIST SD302b with 7,832 pairs (176 mated, 7,656 non-mated). We evaluate four open-source MLLMs (InternVL3-8B, Qwen2.5-VL-7B, Qwen3-VL-8B, Gemma-3-12B) and the proprietary Claude Opus 4.8 under zero-shot, task-description, and similarity-scoring prompts. Prompting governs verification behavior. Task-description prompting collapses all four open-source models to near-100% False Accept Rate (FAR), and Gemma-3-12B collapses under zero-shot as well; Claude Opus 4.8 alone resists collapse under both binary prompts, giving the best binary result (FAR = 20.2%). Similarity scoring removes collapse across the open-source models and exposes wide capability gaps: Claude reaches AUC = 0.953 and Gemma-3-12B 0.837, while InternVL3-8B is inverted (AUC = 0.590) and Qwen2.5-VL-7B near random (0.567). Qwen3-VL-8B attains perfect separation (AUC = 1.000), which we treat as a diagnostic rather than as capability: SD302b holds one SLAP capture per finger position, so mated pairs are cross-resolution. A matched-resolution control leaves the perfect score intact, ruling out the resolution shortcut; what cannot be excluded within SD302b is near-duplicate detection, since a mated pair is one capture rendered twice. A fairness probe over gender, race, and age suggests disparity grows as discrimination weakens. SLAPBench establishes the first SLAP-specific MLLM baseline and shows that prompting governs collapse while model capability governs discrimination.

Figures

Figures reproduced from arXiv: 2607.15517 by Bibesh Pyakurel, M. G. Sarwar Murshed.

Figure 1
Figure 1. Figure 1: Score distributions under the similarity-scoring prompt (7,832 pairs). Bars give the fraction of pairs within each class, normalized separately to offset the 176:7,656 mated-to-non-mated imbalance; dashed lines mark per-class means; each panel is an￾notated with AUC, EER and mean separation ∆. Scores are discrete, so no contin￾uous density is fitted. The first three models discriminate; InternVL3-8B is inv… view at source ↗
Figure 2
Figure 2. Figure 2: ROC curves under the similarity-scoring prompt (7,832 pairs) for the four open-source models and the proprietary Claude Opus 4.8. The inset magnifies the low￾FAR region (FAR ≤ 0.05). The dotted vertical line marks FAR = 0.1%. Qwen3-VL-8B (AUC = 1.000) achieves perfect discrimination, Claude Opus 4.8 (AUC = 0.953) and Gemma-3-12B (AUC = 0.837) are clearly functional, while InternVL3-8B and Qwen2.5- VL-7B pe… view at source ↗
Figure 3
Figure 3. Figure 3: Per-subgroup AUC by gender and race for the three discriminating models (age bands and exact values in Table S1). Qwen3-VL-8B is uniformly perfect; Claude Opus 4.8 varies by ≤1.5 AUC points across gender; Gemma-3-12B shows the largest disparities. The Black group is small (ni = 272), so Gemma’s high value there should be read with caution. 5.3 Demographic Fairness FaceRecBench reports that the best face-ve… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

16 extracted references · 2 canonical work pages

  1. [1]

    anthropic

    Anthropic: The Claude model family (2025),https : / / www . anthropic . com / claude, claude Opus 4.8, accessed via the Anthropic API

  2. [2]

    Duan, H., Fang, X., Yang, J., Zhao, X., Qiao, Y., Li, M., Agarwal, A., Chen, Z., Chen, L., Liu, Y., Ma, Y., Sun, H., Zhang, Y., Lu, S., Wong, T.H., Wang, W., Zhou, P., Li, X., Fu, C., Cui, J., Chen, J., Song, E., Mao, S., Ding, S., Liang, T., Zhang, Z., Dong, X., Zang, Y., Zhang, P., Wang, J., Lin, D., Chen, K.: Vlmevalkit: An open-source toolkit for eval...

  3. [3]

    IEEE Transactions on Pattern Analysis and Machine Intelligence43(6), 1981–1997 (2021).https://doi.org/10.1109/TPAMI.2019.2956523

    Engelsma, J.J., Cao, K., Jain, A.K.: Learning a fixed-length fingerprint represen- tation. IEEE Transactions on Pattern Analysis and Machine Intelligence43(6), 1981–1997 (2021).https://doi.org/10.1109/TPAMI.2019.2956523

  4. [4]

    Gavas, E., Banerjee, S., Hegde, C., Memon, N.: Fpbench: A comprehensive bench- mark of multimodal large language models for fingerprint analysis (2026),https: //arxiv.org/abs/2512.18073

  5. [5]

    Official model release and documen- tation

    Gemma Team: Gemma 3 technical report (2025),https://huggingface.co/ google/gemma-3-12b-it, google DeepMind. Official model release and documen- tation. SLAPBench: MLLMs for Four-Finger SLAP Verification 15

  6. [6]

    Springer, Cham, 3rd edn

    Maltoni, D., Maio, D., Jain, A.K., Feng, J.: Handbook of Fingerprint Recognition. Springer, Cham, 3rd edn. (2022).https://doi.org/10.1007/978-3-030-83624- 5,https://doi.org/10.1007/978-3-030-83624-5

  7. [7]

    IEEE Transactions on Biometrics, Behavior, and Identity Science7(3), 313–330 (2025).https://doi.org/10.1109/TBIOM.2024

    Murshed, M.G.S., Bahmani, K., Schuckers, S., Hussain, F.: Deep age-invariant fingerprint segmentation system. IEEE Transactions on Biometrics, Behavior, and Identity Science7(3), 313–330 (2025).https://doi.org/10.1109/TBIOM.2024. 3506926

  8. [8]

    Narayan, K., VS, V., Patel, V.M.: Facexbench: Evaluating multimodal llms on face understanding (2026),https://arxiv.org/abs/2501.10360

  9. [9]

    National Institute of Standards and Technology: NIST Special Database 302: Nail to nail fingerprint challenge dry run dataset. Tech. rep., National Institute of Standards and Technology (2018),https://www.nist.gov/itl/iad/btg/nist- special-database-302, accessed: 2026-05-24

  10. [10]

    neurotechnology.com/verifinger.html

    Neurotechnology: VeriFinger fingerprint recognition SDK (2024),https://www. neurotechnology.com/verifinger.html

  11. [11]

    Qwen Team: Qwen2.5-vl (Jan 2025),https://qwenlm.github.io/blog/qwen2.5- vl/, official model release page for Qwen2.5-VL

  12. [12]

    Official model release and documentation

    Qwen Team: Qwen3-vl technical report (2025),https://huggingface.co/Qwen/ Qwen3-VL-8B-Instruct, alibaba Group. Official model release and documentation

  13. [13]

    Shahreza, H.O., George, A., Marcel, S.: Evaluating multimodal large language models for heterogeneous face recognition (2026),https://arxiv.org/abs/2601. 15406

  14. [14]

    Shahreza, H.O., Marcel, S.: Benchmarking multimodal large language models for face recognition (2025),https://arxiv.org/abs/2510.14866

  15. [15]

    In: IEEE International Conference on Biometrics: Theory, Applications and Systems (BTAS)

    Tang, Y., Gao, F., Feng, J., Liu, Y.: FingerNet: An unified deep network for fin- gerprint minutiae extraction. In: IEEE International Conference on Biometrics: Theory, Applications and Systems (BTAS). pp. 1–7 (2017).https://doi.org/10. 1109/BTAS.2017.8272679

  16. [16]

    other,” and “no answer

    Zhu, J., Wang, W., Chen, Z., Liu, Z., Ye, S., Gu, L., Tian, H., Duan, Y., Su, W., Shao, J., et al.: Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models (2025),https://arxiv.org/abs/2504.10479 16 B. Pyakurel and M. G. S. Murshed A Supporting Figures This appendix collects the supporting figures referenced in the m...