REVIEW 4 major objections 4 minor 34 references
Anonymization of Documents for Law Enforcement with Machine Learning
T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read A hybrid system that adds one reference redaction to automatic detectors redacts PII on scanned documents more accurately than either component alone.
desk verdict A useful applied paper whose central redaction idea is plausible, but the headline end-to-end claim is unsupported because the evaluation never exercises the retrieval step. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a redaction-matching pipeline that fuses detector boxes with reference boxes. A DINOv2 model trained from scratch on 320,257 scanned documents retrieves the nearest reference document by cosine similarity; A-KAZE keypoints matched by Hamming distance and aligned with RANSAC supply an affine transform that maps reference redaction boxes onto the target scan; and per-class matching rules adjust widths so that, for text, only the true PII span is masked rather than the full detected text line. The reference box tells the system what needs redacting, and the detector box tells it where each element actually sits in the target document.
What would settle it
Run the full pipeline on a held-out document model: take a scan of a new model, retrieve its reference from the database with one expert redaction, and measure mHIoU on ground-truth PII boxes; if retrieval selects a wrong model or the affine alignment fails on a shifted, cropped, or low-quality scan, the 0.741 advantage over the automatic baseline will not reproduce.
Extended reading notes
Core claim
On a hand-annotated evaluation set of 206 scanned documents across six document types and seven countries, the paper shows that using transformed redaction boxes from a reference document to filter and adjust automatically detected bounding boxes yields better redaction quality than either baseline. The best results come from merging detections with reference boxes differently per content type: text boxes inherit the left edge and width from the best-matching detected text region while the reference decides which text is PII; face, MRZ, and barcode boxes are matched and adjusted similarly; signatures are transferred from the reference alone because pre-trained signature detectors fail on this domain. The overall improvement is largest on text, where automatic detection alone redacts nearly all text and the copy-reference baseline ignores text length, while the proposed method reaches 0.527 mAP on text versus 0.019 and 0.153 for the baselines.
Load-bearing premise
The load-bearing premise is that a correctly anonymized reference document of the exact same model will always be available and will be found by the retrieval model; the paper reports perfect retrieval on 54 types but does not state that those types were held out from training, and the redaction evaluation avoids the retrieval step by choosing the reference randomly.
Editorial extensions
If this is right
- A single expert-anonymized reference document can be reused to redact every other scan of the same document model, reducing per-document manual work to verification.
- Adding a new document type does not require retraining the retrieval component, because matching is instance-based rather than tied to a fixed set of classes.
- The selective text masking avoids the over-redaction of purely automatic methods, which mask all detected text and therefore destroy non-PII content.
- The reported retrieval accuracy of 1.0 over 54 document models suggests the reference can be selected automatically from image content alone, without document metadata.
- For content types where detectors are weak, such as signatures, the affine-aligned reference boxes still provide a usable fallback.
Reading between the lines
- Because the redaction evaluation picks the reference randomly from ground-truth candidates of the correct type, the reported 0.741 mHIoU measures the redaction stage under perfect retrieval; end-to-end accuracy with retrieval in the loop remains a testable open question.
- A natural stress test is to feed the system a document model that was absent from retrieval training and measure whether retrieval remains correct, which would reveal whether the reported perfect retrieval is a general layout-level skill or a memorization of familiar types.
- The same matching logic could transfer to other structured documents such as invoices, forms, and medical records, where one expert-redacted template could drive redaction across entire batches if the retrieval model has enough examples of each layout.
- The reference document itself contains the same PII categories, so the database of reference documents must be protected as sensitive data even while it enables anonymization of the larger collection.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a two-stage pipeline for anonymizing scanned law-enforcement documents. A self-supervised DINOv2 model, trained from scratch on 320,257 scans from a state criminal investigation office, is used for instance retrieval to find a manually anonymized reference document of the same model; then a set of object detectors (YuNet for faces, PP-OCRv3 for text, a custom YOLO model for barcodes/stamps, and a morphological MRZ detector) predicts sensitive regions, which are combined with the reference redactions via affine alignment and class-specific matching rules. The method is evaluated on a hand-annotated dataset of 206 scans across six document types, with baselines consisting of automatic detection alone and naive copying of the reference redactions. The authors report that the proposed method achieves mHIoU 0.741 versus 0.571 and 0.486 for the two baselines, and mAP 0.445 versus 0.245 and 0.166. The code and data are proprietary and not released.
Significance. If the claimed results hold for a realistically deployed pipeline, the work would be a useful practical contribution to document anonymization, particularly because it explicitly targets downstream forensic usability by minimizing redacted areas rather than blanket-redacting entire scans. The manuscript has several genuine strengths: the evaluation is conducted on a domain-specific, expert-annotated dataset; the authors compare against two well-chosen baselines that isolate the contribution of each pipeline component; and they use both mHIoU and mAP so that localization quality and detection quality are separately visible. However, the central claim is stronger than the evidence: the headline comparison in Tables III–V evaluates the redaction-matching stage with an oracle-selected reference, while the retrieval component—the one element that distinguishes the proposed system from a purely manual-reference approach—is evaluated separately with no stated held-out split. The practical promise of 'only one anonymized example' depends on retrieval working on unseen scans, and that is precisely the part of the pipeline that is not end-to-end validated.
major comments (4)
- [V.A] The instance retrieval evaluation reports an accuracy of 1.0 across 54 document types, but the manuscript never states that those document models, or the individual scans used for the queries, were excluded from the 320,257 scans used to train DINOv2 from scratch. If the evaluated document models (or the same scans) were part of the training set, perfect retrieval accuracy is expected from memorization and says nothing about generalization to a new, unseen scan of a known document type. This is load-bearing because the deployment scenario requires retrieving the correct reference for documents that the database must process at inference time, not for documents already used in training.
- [V.C] The redaction evaluation sidesteps the retrieval step: the paper states that 'we randomly select one document from all candidates as a reference,' where the candidates are the manually annotated ground-truth documents of the tested type. Thus Tables III, IV, and V measure only the combination of automatic detection with a correct reference, not the end-to-end retrieval-plus-redaction pipeline that the abstract and introduction claim to outperform the baselines. A retrieval failure could send a wrong document type into the matching stage, and the reported margins (mHIoU 0.741 versus 0.571 and 0.486) could shrink or invert. An end-to-end evaluation using the actual retrieved reference, not an oracle-chosen one, is required to support the headline claim.
- [V.C] The reported mHIoU and mAP values are point estimates without any measure of variance. Because the reference document is randomly selected from the candidate set of the same type, and because the annotated evaluation set is small (206 images), the results in Tables III–V could depend substantially on that single random draw. The authors should repeat the reference selection many times (or report per-candidate statistics) and provide means and standard deviations, so the reader can judge the stability of the 0.741 versus 0.571/0.486 comparison.
- [V.C, Table V] The German bank transfer form row is an exception to the headline claim: the proposed method achieves mHIoU 0.569, below the Copy Reference baseline's 0.673, and the text-class mAP drops from 0.281 to 0.096. The authors acknowledge the OCR failure in this case, but the abstract's unqualified statement that the approach 'outperforms both' baselines is too strong. The limitation is real and should be either incorporated into the abstract-level claim or accompanied by an analysis of which document properties cause the proposed method to underperform its copy-reference baseline.
minor comments (4)
- [V.C] In the discussion following Table V, the phrase 'the ground truth baseline only transfers the bounding boxes' appears to refer to the Copy Reference baseline, not the ground truth; this terminology is confusing and should be corrected.
- [II] The dataset name 'Flicker logo dataset' should be 'Flickr logo dataset,' and the author name 'Van Rooji' should be 'Van Rooij' for consistency with the cited reference.
- [IV.A.2] Equation (1) uses x_ref and x_pred without defining the exact coordinate convention (e.g., top-left x-coordinate of the bounding box); a short definition or a figure would remove ambiguity.
- [IV.A.1] The face-image rule 'add any (transformed) reference bounding box that does not overlap with any of the predicted bounding boxes' does not specify an IoU threshold for 'overlap'; a precise threshold would make the method reproducible.
Circularity Check
No significant circularity: the redaction predictions combine external detectors with a separate reference document, and the reported gains are not forced by construction.
full rationale
The paper's derivation chain is not circular. The proposed redaction prediction is a combination of independent object detectors and a reference document's redactions; the reference redactions are not the target document's ground truth. The evaluation in Section V.C randomly selects a reference from ground-truth candidates to ablate reference selection, but that is an oracle setup, not a circular derivation; the mHIoU improvement over the 'Copy Reference' baseline comes from detection-based refinement and is not forced by definition. The instance retrieval accuracy of 1.0 in Section V.A is reported on 54 document types without an explicit held-out split, which is a data-leakage or generalization risk rather than a circularity; the DinoV2 encoder is an external component and its retrieval accuracy is not defined in terms of the redaction metric. The only self-citation, reference [18], appears in related work on printer identification and is not load-bearing for the proposed method. The manuscript also explicitly acknowledges that a human must verify redactions and that models are not fine-tuned, which are limitations, not circular self-support. Consequently, no step reduces to its own input by construction.
Assumptions & free parameters
free parameters (2)
- IoU matching threshold for text redaction =
0.1
- RANSAC parameters for affine transform
assumptions (5)
- domain assumption Document layout is consistent within each document model, so one reference's redaction boxes can be transferred to other scans.
- domain assumption A-KAZE keypoint matching plus RANSAC yields an affine transform that accurately aligns reference and target scans.
- domain assumption Pre-trained YuNet face detector and PP-OCRv3 text detector perform adequately on law enforcement document scans.
- domain assumption The DinoV2 instance retrieval model, trained from scratch on the same office's 320,257 scans, generalizes to the document types used in evaluation.
- domain assumption Expert annotations of redaction ground truth are complete and correct.
Cite this review
Pith. "Pith review of Anonymization of Documents for Law Enforcement with Machine Learning." pith.science (2026). https://pith.science/paper/WF3337TQ
@misc{pith2026250107334,
author = {Pith},
title = {Pith review of: Anonymization of Documents for Law Enforcement with Machine Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/WF3337TQ}},
note = {Machine review of arXiv:2501.07334}
}
read the original abstract
The steadily increasing utilization of data-driven methods and approaches in areas that handle sensitive personal information such as in law enforcement mandates an ever increasing effort in these institutions to comply with data protection guidelines. In this work, we present a system for automatically anonymizing images of scanned documents, reducing manual effort while ensuring data protection compliance. Our method considers the viability of further forensic processing after anonymization by minimizing automatically redacted areas by combining automatic detection of sensitive regions with knowledge from a manually anonymized reference document. Using a self-supervised image model for instance retrieval of the reference document, our approach requires only one anonymized example to efficiently redact all documents of the same type, significantly reducing processing time. We show that our approach outperforms both a purely automatic redaction system and also a naive copy-paste scheme of the reference anonymization to other documents on a hand-crafted dataset of ground truth redactions.
Figures
Reference graph
Works this paper leans on
-
[1]
Available: http://data.europa.eu/eli/reg/2016/679/oj/eng
“Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation) (Text with EEA relevance).” [Online]. Available: http://data.europa.eu/eli/...
work page 2016
-
[2]
California Consumer Privacy Act of 2018
“California Consumer Privacy Act of 2018.” [Online]. Avail- able: https://leginfo.legislature.ca.gov/faces/codes displayText.xhtml? division=3.&part=4.&lawCode=CIV&title=1.81.5
work page 2018
-
[3]
Privacy Preserving by Removing Sensitive Data from Documents with Fully Convolutional Networks,
M. Korytkowski, J. Nowak, R. Scherer, and W. Wei, “Privacy Preserving by Removing Sensitive Data from Documents with Fully Convolutional Networks,” in Artificial Intelligence and Soft Computing , L. Rutkowski, R. Scherer, M. Korytkowski, W. Pedrycz, R. Tadeusiewicz, and J. M. Zurada, Eds. Cham: Springer International Publishing, 2023, pp. 277– 285
work page 2023
-
[4]
D. Biesner, R. Ramamurthy, R. Stenzel, M. L ¨ubbering, L. Hillebrand, A. Ladi, M. Pielka, R. Loitz, C. Bauckhage, and R. Sifa, “Anonymization of German financial documents using neural network-based language models with contextual word representations,” International Journal of Data Science and Analytics , vol. 13, no. 2, pp. 151–161, Mar. 2022
work page 2022
-
[5]
R. Juez-Hernandez, L. Quijano-S ´anchez, F. Liberatore, and J. G ´omez, “AGORA: An intelligent system for the anonymization, information extraction and automatic mapping of sensitive documents,” Applied Soft Computing, vol. 145, p. 110540, Sep. 2023
work page 2023
-
[6]
Document anonymization for border guards and immigration services,
H. Bouma, R. Pruim, A. V . Rooijen, J.-M. ten Hove, J. van Mil, and B. Kromhout, “Document anonymization for border guards and immigration services,” in Counterterrorism, Crime Fighting, Forensics, and Surveillance Technologies IV , vol. 11542. SPIE, Sep. 2020, pp. 68–75
work page 2020
-
[7]
Federated tool for anonymization and annotation in image data,
S. Van Rooij, H. Bouma, J. Van Mil, and J.-M. Ten Hove, “Federated tool for anonymization and annotation in image data,” in Counterterrorism, Crime Fighting, Forensics, and Surveillance Technologies VI, H. Bouma, R. J. Stokes, Y . Yitzhaky, and R. Prabhu, Eds. Berlin, Germany: SPIE, Oct. 2022, p. 13
work page 2022
-
[8]
Ultralyt- ics/yolov5: V7.0 - YOLOv5 SOTA Realtime Instance Segmentation,
G. Jocher, Ayush Chaurasia, A. Stoken, J. Borovec, NanoCode012, Yonghye Kwon, Kalen Michael, TaoXie, Jiacong Fang, Imyhxy, Lorna, Zeng Yifu, C. Wong, Abhiram V, D. Montes, Zhiqiang Wang, C. Fati, Jebastin Nadar, Laughing, UnglvKitDe, V . Sonck, Tkianai, YxNONG, P. Skalski, A. Hogan, Dhruv Nair, M. Strobel, and M. Jain, “Ultralyt- ics/yolov5: V7.0 - YOLOv5...
work page 2022
Show all 34 references
-
[9]
Scalable logo recognition in real-world images,
S. Romberg, L. G. Pueyo, R. Lienhart, and R. Van Zwol, “Scalable logo recognition in real-world images,” in Proceedings of the 1st ACM International Conference on Multimedia Retrieval . Trento Italy: ACM, Apr. 2011, pp. 1–8
2011
-
[10]
Automatic Anonymization of Printed-Text Document Images,
´A. S ´anchez, J. F. V ´elez, J. S ´anchez, and A. B. Moreno, “Automatic Anonymization of Printed-Text Document Images,” in Image and Signal Processing, A. Mansouri, A. El Moataz, F. Nouboud, and D. Mammass, Eds. Cham: Springer International Publishing, 2018, pp. 145–152
2018
-
[11]
Applications of Machine Learning in Digital Forensics,
S. Qadir and B. Noor, “Applications of Machine Learning in Digital Forensics,” in 2021 International Conference on Digital Futures and Transformative Technologies (ICoDT2), May 2021, pp. 1–8
2021
-
[12]
Digital forensics supported by machine learning for the detection of online sexual predatory chats,
C. H. Ngejane, J. H. P. Eloff, T. J. Sefara, and V . N. Marivate, “Digital forensics supported by machine learning for the detection of online sexual predatory chats,” Forensic Science International: Digital Investigation, vol. 36, p. 301109, Mar. 2021
2021
-
[13]
Deep Learning for Person Re-Identification: A Survey and Outlook,
M. Ye, J. Shen, G. Lin, T. Xiang, L. Shao, and S. C. H. Hoi, “Deep Learning for Person Re-Identification: A Survey and Outlook,” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 44, no. 6, pp. 2872–2893, Jun. 2022
2022
-
[14]
Spatial-Temporal Person Re-Identification,
G. Wang, J. Lai, P. Huang, and X. Xie, “Spatial-Temporal Person Re-Identification,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, no. 01, pp. 8933–8940, Jul. 2019
2019
-
[15]
How im- portant are faces for person re-identification?
J. Dietlmeier, J. Antony, K. McGuinness, and N. E. O’Connor, “How im- portant are faces for person re-identification?” in 2020 25th International Conference on Pattern Recognition (ICPR) , Jan. 2021, pp. 6912–6919
2020
-
[16]
Printer Identification Methods Using Global and Local Feature-Based Deep Learning,
S.-H. Lee and H.-Y . Lee, “Printer Identification Methods Using Global and Local Feature-Based Deep Learning,” KIPS Transactions on Soft- ware and Data Engineering , vol. 8, no. 1, pp. 37–44, 2019
2019
-
[17]
Printer source identification of quick response codes using residual attention network and smartphones,
Z. Guo, S. Wang, Z. Zheng, and K. Sun, “Printer source identification of quick response codes using residual attention network and smartphones,” Engineering Applications of Artificial Intelligence , vol. 131, p. 107822, May 2024
2024
-
[18]
Clas- sification of Inkjet Printers based on Droplet Statistics,
P. Takenaka, M. Eberhardinger, D. Grießhaber, and J. Maucher, “Clas- sification of Inkjet Printers based on Droplet Statistics,” in 2024 Inter- national Joint Conference on Neural Networks (IJCNN) , Jun. 2024, pp. 1–7
2024
-
[19]
Deep Learning for Instance Retrieval: A Survey,
W. Chen, Y . Liu, W. Wang, E. M. Bakker, T. Georgiou, P. Fieguth, L. Liu, and M. S. Lew, “Deep Learning for Instance Retrieval: A Survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 45, no. 6, pp. 7270–7292, Jun. 2023
2023
-
[20]
DINOv2: Learning Robust Vi- sual Features without Supervision,
M. Oquab, T. Darcet, T. Moutakanni, H. V . V o, M. Szafraniec, V . Khali- dov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, R. Howes, P.-Y . Huang, S.-W. Li, I. Misra, M. Rabbat, V . Sharma, G. Synnaeve, H. Xu, H. Jegou, J. Mairal, P. Labatu...
2023
-
[21]
An Image is Worth 16x16 Words: Trans- formers for Image Recognition at Scale,
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An Image is Worth 16x16 Words: Trans- formers for Image Recognition at Scale,” in International Conference on Learni...
2020
-
[22]
Fast Explicit Diffusion for Accelerated Features in Nonlinear Scale Spaces,
P. Alcantarilla, J. Nuevo, and A. Bartoli, “Fast Explicit Diffusion for Accelerated Features in Nonlinear Scale Spaces,” in Procedings of the British Machine Vision Conference 2013 . Bristol: British Machine Vision Association, 2013, pp. 13.1–13.11
2013
-
[23]
Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography,
M. A. Fischler and R. C. Bolles, “Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography,” Commun. ACM , vol. 24, no. 6, pp. 381–395, Jun. 1981
1981
-
[24]
YuNet: A Tiny Millisecond-level Face Detector,
W. Wu, H. Peng, and S. Yu, “YuNet: A Tiny Millisecond-level Face Detector,” Machine Intelligence Research , vol. 20, no. 5, pp. 656–665, Oct. 2023
2023
-
[25]
PP-OCRv3: More Attempts for the Improvement of Ultra Lightweight OCR System,
C. Li, W. Liu, R. Guo, X. Yin, K. Jiang, Y . Du, Y . Du, L. Zhu, B. Lai, X. Hu, D. Yu, and Y . Ma, “PP-OCRv3: More Attempts for the Improvement of Ultra Lightweight OCR System,” 2022
2022
-
[26]
PP-OCRv2: Bag of Tricks for Ultra Lightweight OCR System,
Y . Du, C. Li, R. Guo, C. Cui, W. Liu, J. Zhou, B. Lu, Y . Yang, Q. Liu, X. Hu, D. Yu, and Y . Ma, “PP-OCRv2: Bag of Tricks for Ultra Lightweight OCR System,” Oct. 2021
2021
-
[27]
Detecting machine-readable zones in passport images,
A. Rosebrock, “Detecting machine-readable zones in passport images,” Nov. 2015
2015
-
[28]
Optimal Filters for Extended Optical Flow,
H. Scharr, “Optimal Filters for Extended Optical Flow,” in Complex Motion, B. J ¨ahne, R. Mester, E. Barth, and H. Scharr, Eds. Berlin, Heidelberg: Springer, 2007, pp. 14–29
2007
-
[29]
Konstantint/PassportEye,
K. Tretyakov, “Konstantint/PassportEye,” Aug. 2024
2024
-
[30]
Ultralytics Signature Detection Dataset,
Ultralytics, “Ultralytics Signature Detection Dataset,” https://docs.ultralytics.com/datasets/detect/signature
-
[31]
Offline Sig- nature Verification on Real-World Documents,
D. Engin, A. Kantarci, S. Arslan, and H. K. Ekenel, “Offline Sig- nature Verification on Real-World Documents,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020, pp. 808–809
2020
-
[32]
Stepan-coder/HandWritenSignatureDetection,
St.Dev.Lab, “Stepan-coder/HandWritenSignatureDetection,” Sep. 2024
2024
-
[33]
Anomalydino: Boosting patch-based few-shot anomaly detection with dinov2,
S. Damm, M. Laszkiewicz, J. Lederer, and A. Fischer, “Anomalydino: Boosting patch-based few-shot anomaly detection with dinov2,” 2024. [Online]. Available: https://arxiv.org/abs/2405.14529
2024 arXiv
-
[34]
General purpose image encoder dinov2 for medical image registration,
X. Song, X. Xu, and P. Yan, “General purpose image encoder dinov2 for medical image registration,” 2024. [Online]. Available: https://arxiv.org/abs/2402.15687
2024 arXiv
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.