Pith. sign in

REVIEW 4 major objections 5 minor 23 references

Transparency Techniques for Neural Networks trained on Writer Identification and Writer Verification

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Pixel-wise saliency maps outperform point-specific saliency maps for explaining writer-identification and writer-verification networks, and can support forensic handwriting analysis.

desk verdict First application of two saliency methods to writer ID/verification, but the deletion-score comparison that drives the headline is compromised by a same-image protocol that favors the pixel-level maps. read the letter →

arxiv 2506.16331 v1 pith:LPSSYWI7 submitted 2025-06-19 cs.CV

Viktoria Pundy , Marco Peer , Florian Kleber This is my paper · ORCID
classification cs.CV
keywords writeridentificationverificationsaliencymapsmodeltransparencyforensichandwritinganalysisdeletionandinsertionmetricsdeepmetriclearningretrieval
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Neural networks identify and verify handwriting authors well, but what they look at inside a page of text stays opaque. This paper is the first to apply two transparency techniques to writer-identification and writer-verification networks: pixel-wise saliency maps and point-specific saliency maps. Using deletion and insertion scores on three handwriting datasets, it finds that pixel-wise maps outperform point-specific maps and highlight the same character forms that forensic experts consider during authorship analysis. The point-specific maps allocate highlights less consistently and are judged not suitable for supporting the analysis process.

What carries the argument

The load-bearing objects are two gradient-based saliency constructions. The pixel-wise map is $s(I) = \partial d(x_I, x_{\text{base}})/\partial I$, the gradient of the cosine-similarity distance between the embedding of the input image and a white blank-paper base image, averaged over four random white-pixel masks in a smooth-grad style. The point-specific map starts from rewriting cosine similarity between a query image $q$ and a retrieval image $r$ as a sum over all location pairs of dot products of last-layer feature maps, so each pair of points $(i,j)$ and $(x,y)$ receives saliency $\sum_k A^q_{i,j,k} A^r_{x,y,k}$; an overall map sums these contributions. The evaluation machinery is the deletion/insertion protocol: iteratively remove or insert the map's most significant black pixels, track cosine similarity against a random baseline, and summarize with AUC scores.

What would settle it

Run the deletion and insertion protocol with several different cosine-similarity thresholds and with genuine retrieval pairs (two different pages, sometimes by the same writer, sometimes by different writers) instead of the same image; if the pixel-level map's deletion-score advantage over point-specific maps disappears or reverses under any of these settings, the paper's central quantitative claim does not survive. A second check: match saliency peaks against labeled character occurrences on a larger sample; if the repeated 'y' and 'f' patterns do not appear, the qualitative support claim weakens.

Watch

Extended reading notes

Core claim

The paper claims that for embedding-based writer-identification and writer-verification networks, pixel-wise saliency maps derived from the gradient of cosine similarity between the input embedding and a white base image are more faithful than point-specific saliency maps derived from decomposing cosine similarity over feature-map locations. Faithfulness is measured by deletion and insertion AUC scores: deleting the most-highlighted pixels should lower similarity faster than random deletion, and inserting them should raise it faster than random insertion. Across ResNet18, ResNet20, and ResNet50 backbones and the CVL, Firemaker, and ICDAR2013 datasets, the pixel-level maps achieve substantially higher deletion scores than the point-specific maps, while insertion scores are high for both. Qualitatively, pixel-level maps show repeated highlighting patterns across occurrences of the same character, whereas point-specific maps often place peak highlights on the wrong character or spread highlights over many characters, making them less suitable for forensic use.

Load-bearing premise

The load-bearing premise is that the deletion/insertion metric, which uses the same image as query and retrieved image and zeroes similarities below an unspecified threshold, correctly measures how well a saliency map explains the network's decisions.

Editorial extensions

If this is right

  • If pixel-wise maps are faithful, forensic experts can use them to locate writer-specific character forms without reading model internals.
  • Deletion and insertion scores can serve as a standard quantitative check for saliency methods in document-analysis networks.
  • Deletion scores depend strongly on network architecture, so transparency quality is a property of the network and loss as much as of the explanation method.
  • Repeated highlighting of the same characters suggests these networks learn allographic letter-shape features, which could inform better writer-identification training.
  • Point-specific maps may become interpretable only with character-level or tighter image crops, a direction the paper identifies for future work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial: Because the paper uses the same image as query and retrieved image, the deletion/insertion metric measures self-consistency rather than retrieval relevance; testing with genuine retrieval pairs would clarify whether pixel-level maps also explain real identification decisions.
  • Editorial: The cosine-similarity threshold used to zero scores is never stated, so a simple threshold sweep would show whether the reported pixel-level advantage is robust.
  • Editorial: The paper hints that single-character inputs could improve point-specific maps; a quantitative comparison of highlight placement on cropped characters would turn that suggestion into a testable claim.
  • Editorial: The point-to-image maps' tendency to highlight similarly shaped letter parts suggests the networks encode sub-character grapheme fragments, which could become a vocabulary for forensic comparison if confirmed on larger samples.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper applies two transparency techniques to neural networks trained for writer identification and writer verification: pixel-level saliency maps based on gradients of the embedding cosine distance to a blank white image, and point-specific saliency maps based on feature-map inner products that highlight similar image regions. The methods are evaluated on the CVL, Firemaker, and ICDAR2013 datasets with deletion and insertion AUC scores adapted from Hu et al., and qualitatively by inspecting whether highlighted regions repeat across occurrences of the same character. The authors report that pixel-wise saliency maps achieve higher deletion scores than point-specific maps, and conclude that pixel-wise maps outperform point-specific maps and are suitable for supporting forensic handwriting analysis.

Significance. This is the first application of these two transparency techniques to writer identification and verification, and the topic is relevant for forensic document analysis. The quantitative comparison in Table 2 covers multiple architectures and datasets, and the qualitative analysis examines character-level patterns in a domain where such evidence could be useful. If the evaluation were sound, the finding that pixel-level gradients localize writer-specific features more reliably than point-specific self-similarity maps would be a useful contribution. However, the current evaluation protocol has structural asymmetries and missing parameters that undermine the headline claim, so the significance is conditional on a reworked evaluation.

major comments (4)
  1. [Sec. 3.2 and Table 2] The deletion/insertion protocol sets 'the query and retrieved image are the same.' This makes the point-specific overall saliency map a self-similarity map: with q=r, Eq. (2) highlights pixels that are similar to other pixels in the same image. Deleting such pixels removes repeated patterns, but the global embedding may change little because other instances of the pattern remain, so low deletion scores for the point-specific maps are partly predetermined. In contrast, the pixel-level map is the gradient of the cosine distance to a blank white base image (Sec. 2.1), so its top pixels are by construction the pixels whose deletion moves the embedding away from the original image. The comparison in Table 2 therefore does not establish that pixel-wise maps are intrinsically better; it may reflect a protocol artifact. I recommend re-running the evaluation with query and retrieved images being different test images, or otherwise grounding both methods in the same retrieval task.
  2. [Sec. 3.2] The text says 'The value is set to zero if it falls below this threshold,' but the threshold is never specified. Because the AUC values in Table 2 depend on this threshold, the quantitative results are not reproducible, and the ranking of methods could change with different thresholds. The authors should state the exact threshold used and report sensitivity of the deletion/insertion scores to this parameter.
  3. [Sec. 3.4 and Table 2] The insertion scores are near ceiling for both methods: for the WI networks, all values in Table 2 are above 89%, and for the WV networks all but two are above 90%. This makes insertion scores non-discriminating, so the claim that pixel-wise maps outperform point-specific maps rests entirely on the deletion scores. Given the protocol issue in Sec. 3.2, the deletion-score advantage is not sufficient evidence. A metric that is not saturated, or an explicit argument for why the saturated insertion scores do not matter, is needed.
  4. [Sec. 3.5] The qualitative evaluation is based on only two pages from the CVL dataset, one with successful retrieval and one with failed retrieval. The comparison to 'the areas forensic experts consider' is informal: no forensic expert annotations, no defined scoring rule, and no inter-rater protocol are provided. The claim that the pixel-wise maps are 'suitable for the support of forensic experts' is therefore not supported by the presented evidence. A systematic study with expert-provided ground truth or a predefined highlight-matching criterion would be required.
minor comments (5)
  1. [Sec. 2.1] The number of random masked variants, n=4, is fixed without any sensitivity analysis or justification. Since SmoothGrad results can depend on n, the robustness of the saliency maps to this choice should be discussed.
  2. [Sec. 2.1] The choice of a white base image is motivated as representing an empty writing surface, but the networks were trained on snippets containing handwriting. The effect of using a blank image as the base for gradient computation is not analyzed; a short discussion or ablation would be helpful.
  3. [Sec. 2.2] The definition of the overall saliency map is ambiguous: after Eq. (2), the summation over (i,j) with a fixed (x,y) is not clearly distinguished from a summation over all spatial locations. Please define the variables and index ranges explicitly.
  4. [Sec. 3.2] The description of the deletion and insertion procedure says only black pixels are altered, but it is not clear how partial deletions or insertions are applied per iteration. A precise algorithmic description would improve reproducibility.
  5. [Sec. 3.4] The text states that for WI networks the insertion scores are high with 'all but one value being above 90%,' which is correct for Table 2, but the corresponding sentence for WV networks should be checked: the ResNet20 CVL insertion score is 84.3%, which is the second exception, not the only one.

Circularity Check

1 steps flagged · score 6.0 of 10

Deletion-score comparison is partly self-referential: the pixel-wise map is the gradient of the same cosine similarity that the metric measures, so the reported superiority is substantially a self-consistency artifact.

  1. self definitional [Section 2.1 (Pixel-Wise Saliency Maps) and Section 3.2 (Metrics)]
    "For the generation of the saliency map, the following gradients are calculated [20] as s(I) = ∂d(xI, xbase)/∂I, where d is a distance function for two embedding vectors. The Cosine Similarity is used as distance function. ... For this work, the query and retrieved image are the same. Iteratively, the pixels of the retrieved image are altered based on the saliency map. ... The similarity between the images is then calculated using the Cosine Similarity."

    The pixel-wise map is the gradient of the cosine distance to the white base image. The deletion score removes the highest-gradient pixels, setting them to white, and measures the drop in cosine similarity between the altered image and the original. To first order, deleting the pixels with the largest gradient is exactly the perturbation that this map was constructed to flag, so a high deletion score is a self-consistency check rather than an external validation. The point-specific overall map, by contrast, is a decomposition of the self-similarity S(q,r) with q=r (Eq. 2); it is not constructed to optimize the deletion objective, so its systematically lower aucd values are partly predetermined by the choice of metric.

full rationale

The central quantitative comparison in this paper is not fully independent: the deletion metric and the pixel-wise saliency map share the same cosine-similarity objective, so the reported superiority is substantially a self-consistency result. I do not find any load-bearing self-citation circularity: the CVL dataset reference [17] is a data resource, not an argument that forces the paper's conclusion, and the method citations to Kobs, Zhu, and Hu are prior external work. The random-deletion baseline and the qualitative comparison of repeated character highlights provide some independent content, but the qualitative analysis is informal and the quantitative comparison remains biased by construction. The unspecified cosine threshold in Sec 3.2 is a reproducibility problem, not a circularity. Overall, the claim that pixel-wise maps outperform point-specific maps is partially circular because the evaluation protocol is constructed from the pixel-wise method's objective; however, the paper does contain independent qualitative observations, so the circularity is partial rather than total.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities and fits no scientific constants; it does rely on hand-chosen evaluation parameters and on the validity of two existing explanation methods plus the adapted deletion and insertion protocol for binarized handwriting.

free parameters (4)
  • number of random masked variants (n) = 4
    Smooth-Grad style averaging uses n=4 masked input variants (Sec 2.1); this hand-picked value controls saliency map noise and affects every subsequent score.
  • cosine similarity zeroing threshold = not specified
    In Sec 3.2 the authors state 'The value is set to zero if it falls below this threshold' without giving the threshold; this directly shapes the AUC curves in Table 2 and prevents exact replication.
  • snippet size = 400x400 px (200x200 px for ICDAR2013)
    Input snippets are sampled at a hand-chosen size (Sec 3.1), which determines the resolution of the saliency maps and the granularity of the deletion and insertion measurement.
  • minimum black pixel fraction for snippets = 2%
    Training snippets containing less than 2% black pixels are excluded (Sec 3.1), a hand-set threshold that may bias the saliency evaluation toward text-dense regions.
assumptions (4)
  • domain assumption The trained WI and WV models produce embeddings whose cosine similarity is a meaningful measure of writer identity.
    The networks are trained with triplet and contrastive losses using cosine similarity (Sec 3.1); the entire transparency analysis inherits this assumption.
  • domain assumption Gradient-based saliency methods (Kobs et al. [20], Zhu et al. [16]) applied with cosine similarity validly localize the features used by the network.
    The two techniques are adopted without an independent validation for the WI and WV setting beyond the deletion and insertion metrics and the informal qualitative inspection (Sec 2.1, 2.2).
  • ad hoc to paper A white base image represents an empty writing surface for the pixel-wise saliency calculation.
    The authors replace the black base image proposed in Kobs et al. with a white image because handwriting is black on white (Sec 2.1); this choice is justified only by intuition and is not parameter-swept.
  • ad hoc to paper The deletion and insertion protocol with query equal to retrieved image and an unstated similarity threshold measures saliency quality.
    The protocol is adapted from Hu et al. [14] to binarized handwriting (Sec 3.2), but no external ground truth or expert labels are used to validate that fast similarity drop reflects correct highlighting.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Transparency Techniques for Neural Networks trained on Writer Identification and Writer Verification." pith.science (2026). https://pith.science/paper/LPSSYWI7

@misc{pith2026250616331,
  author       = {Pith},
  title        = {Pith review of: Transparency Techniques for Neural Networks trained on Writer Identification and Writer Verification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LPSSYWI7}},
  note         = {Machine review of arXiv:2506.16331}
}
read the original abstract

Neural Networks are the state of the art for many tasks in the computer vision domain, including Writer Identification (WI) and Writer Verification (WV). The transparency of these "black box" systems is important for improvements of performance and reliability. For this work, two transparency techniques are applied to neural networks trained on WI and WV for the first time in this domain. The first technique provides pixel-level saliency maps, while the point-specific saliency maps of the second technique provide information on similarities between two images. The transparency techniques are evaluated using deletion and insertion score metrics. The goal is to support forensic experts with information on similarities in handwritten text and to explore the characteristics selected by a neural network for the identification process. For the qualitative evaluation, the highlights of the maps are compared to the areas forensic experts consider during the identification process. The evaluation results show that the pixel-wise saliency maps outperform the point-specific saliency maps and are suitable for the support of forensic experts.

Figures

Figures reproduced from arXiv: 2506.16331 by the authors.

Figure 1
Figure 1. Example of deletion and insertion scores calculated for the given saliency map. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. The first three lines of the pages taken from the test set of the CVL dataset [ [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Examples for the pixel-wise saliency maps for the handwritten text of the first page. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Reoccurring highlighting patterns for the occurrences of the character ”y”. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Occurrences of the character ”f”. The highlights for the first page display a pattern, while [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Examples for the point-specific saliency maps for the handwritten text of the first page. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 8
Figure 8. Figure 8: Point-specific saliency maps for the first author with a selected point. The selected points [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 10
Figure 10. Figure 10: Point-specific saliency maps for the second author with a selected point. The selected [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 23 canonical work pages

  1. [1]

    Evaluating the Visualization of What a Deep Neural Network Has Learned

    Wojciech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus-Robert Müller. Evaluating the Visualization of What a Deep Neural Network Has Learned. IEEE Transactions on Neural Networks and Learning Systems, 28(11):2660–2673, 2017

    work page 2017

  2. [2]

    A Survey on Neural Network Interpretability

    Yu Zhang, Peter Tiˇno, Aleš Leonardis, and Ke Tang. A Survey on Neural Network Interpretability. IEEE Transactions on Emerging Topics in Computational Intelligence, 5(5):726–742, 2021

    work page 2021

  3. [3]

    Nattkemper

    Martin Zurowietz and Tim W. Nattkemper. An Interactive Visualization for Feature Localization in Deep Neural Networks. Frontiers in Artificial Intelligence, 3:49, 2020

    work page 2020

  4. [4]

    Deep TEN: Texture Encoding Network

    Hang Zhang, Jia Xue, and Kristin Dana. Deep TEN: Texture Encoding Network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 708–717, 2017

    work page 2017

  5. [5]

    Writer Identification and Verification

    Lambert Schomaker. Writer Identification and Verification. InAdvances in Biometrics: Sensors, Algorithms and Systems, pages 247–264. Springer, 2008

    work page 2008

  6. [6]

    Text-Independent Writer Identification via CNN Features and Joint Bayesian

    Youbao Tang and Xiangqian Wu. Text-Independent Writer Identification via CNN Features and Joint Bayesian. In 2016 15th International Conference on Frontiers in Handwriting Recognition (ICFHR), pages 566–571. IEEE, 2016

    work page 2016

  7. [7]

    Towards End-to-End Deep Learning-based Writer Identification

    Zhenghua Wang, Andreas Maier, and Vincent Christlein. Towards End-to-End Deep Learning-based Writer Identification. INFORMATIK 2020, 2021

    work page 2020

  8. [8]

    Chaudhuri, and Michael Blumenstein

    Chandranath Adak, Bidyut B. Chaudhuri, and Michael Blumenstein. An Empirical Study on Writer Identification and Verification From Intra-Variable Individual Handwriting. IEEE Access, 7:24738–24758, 2019

    work page 2019

Show all 23 references
  1. [9]

    Learning Features for Writer Retrieval and Identifica- tion using Triplet CNNs

    Manuel Keglevic, Stefan Fiel, and Robert Sablatnig. Learning Features for Writer Retrieval and Identifica- tion using Triplet CNNs. In 2018 16th International Conference on Frontiers in Handwriting Recognition (ICFHR), pages 211–216. IEEE, 2018

    2018

  2. [10]

    Text-Independent Writer Identification and Verification Using Textural and Allographic Features

    Marius Bulacu and Lambert Schomaker. Text-Independent Writer Identification and Verification Using Textural and Allographic Features. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29 (4):701–717, 2007

    2007

  3. [11]

    A Survey of Methods for Explaining Black Box Models

    Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. A Survey of Methods for Explaining Black Box Models. ACM computing surveys (CSUR), 51(5):1–42, 2018

    2018

  4. [12]

    Towards Explainable Artificial Intelligence

    Wojciech Samek and Klaus-Robert Müller. Towards Explainable Artificial Intelligence. Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, pages 5–22, 2019

    2019

  5. [13]

    Ruderman

    Daniel L. Ruderman. The statistics of natural images. Network: Computation in Neural Systems, 5(4):517, 1994

    1994

  6. [14]

    X-MIR: EXplainable Medical Image Retrieval

    Brian Hu, Bhavan Vasu, and Anthony Hoogs. X-MIR: EXplainable Medical Image Retrieval. In Proceed- ings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 440–450, 2022

    2022

  7. [15]

    Towards Visually Explaining Similarity Models

    Meng Zheng, Srikrishna Karanam, Terrence Chen, Richard J Radke, and Ziyan Wu. Towards Visually Explaining Similarity Models. arXiv preprint arXiv:2008.06035, 2020

    2008 arXiv

  8. [16]

    Visual Explanation for Deep Metric Learning

    Sijie Zhu, Taojiannan Yang, and Chen Chen. Visual Explanation for Deep Metric Learning. IEEE Transactions on Image Processing, 30:7593–7607, 2021

    2021

  9. [17]

    CVL-Database: An Off-line Database for Writer Retrieval, Writer Identification and Word Spotting

    Florian Kleber, Stefan Fiel, Markus Diem, and Robert Sablatnig. CVL-Database: An Off-line Database for Writer Retrieval, Writer Identification and Word Spotting. In 2013 12th International Conference on Document Analysis and Recognition (ICDAR), pages 560–564. IEEE, 2013

    2013

  10. [18]

    Writer Identification Using Edge-Based Direc- tional Features

    Marius Bulacu, Lambert Schomaker, and Louis Vuurpijl. Writer Identification Using Edge-Based Direc- tional Features. In ICDAR ’03: Proceedings of the 7th International Conference on Document Analysis and Recognition, pages 937–941. IEEE Computer Society, 2003. ISBN 0-7695-1960-1

    2003

  11. [19]

    ICDAR 2013 Com- petition on Writer Identification

    Georgios Louloudis, Basilios Gatos, Nikolaos Stamatopoulos, and A Papandreou. ICDAR 2013 Com- petition on Writer Identification. In 2013 12th International Conference on Document Analysis and Recognition (ICDAR), pages 1397–1401. IEEE, 2013

    2013

  12. [20]

    Konstantin Kobs, Michael Steininger, Andrzej Dulny, and Andreas Hotho. Do Different Deep Metric Learning Losses Lead to Similar Learned Features? In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10644–10654, 2021

    2021

  13. [21]

    Writer identification and writer retrieval based on NetVLAD with Re-ranking

    Shervin Rasoulzadeh and Bagher BabaAli. Writer identification and writer retrieval based on NetVLAD with Re-ranking. IET Biometrics, 11(1):10–22, 2022

    2022

  14. [22]

    Ghosh, Josep Lladós, and Umapada Pal

    Sounak Dey, Anjan Dutta, Juan Ignacio Toledo, Suman K. Ghosh, Josep Lladós, and Umapada Pal. SigNet: Convolutional Siamese Network for Writer Independent Offline Signature Verification. arXiv preprint arXiv:1707.02131, 2017

    2017 arXiv

  15. [23]

    Handwriting Analysis with Focus on Writer Identification and Writer Retrieval

    Vincent Christlein. Handwriting Analysis with Focus on Writer Identification and Writer Retrieval. Ph.D. dissertation, Friedrich-Alexander-Universitaet Erlangen-Nuernberg (Germany), 2019

    2019

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.