{"id":"0857cbea-f537-4805-9bc8-5ab133674359","arxiv_id":"2501.07334","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A hybrid redaction system aligns one expert-anonymized reference document with automatic detections to redact other documents of the same type.","lead":"This paper describes a system that automatically hides personal information on scanned documents by combining object detection with a single already-anonymized example of the same document type. A smart generalist would read it to see how semi-automatic redaction can save work while keeping personal data hidden.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's headline claim is about the full retrieval-plus-redaction system, but Tables III–V evaluate redaction with an oracle-selected reference, and the 1.0 retrieval accuracy is reported without any held-out split. This leaves the end-to-end claim unsupported.","rationale":"The reader's weakest assumption identifies the same load-bearing issue: the retrieval model is evaluated separately from the redaction pipeline, and the redaction tables assume a correct reference. I agree with that assessment. The paper is an honest engineering report on a proprietary dataset, and the comparison between the proposed redaction combination and the two baselines is plausible for the 206 annotated documents. However, the abstract and introduction claim a system that retrieves a reference automatically and then redacts; the evaluation never measures that full system. The absence of a held-out split for the retrieval model makes the 1.0 accuracy non-interpretable as evidence of generalization. This is an addressable weakness, not a fatal flaw, so the CONDITIONAL verdict remains appropriate: the authors should either provide held-out retrieval numbers or, preferably, an end-to-end evaluation where the reference comes from retrieval rather than from ground-truth candidates. The requested test would settle whether the central claim holds: if retrieval accuracy on held-out document models is actually high and end-to-end mHIoU stays near 0.74, the claim is strengthened; if not, the claimed advantage over baselines may be an artifact of oracle reference selection.","tokens_in":10549,"tokens_out":2871,"duration_ms":30777,"concrete_test":"Hold out all scans of a subset of document models (e.g., 10 of the 54) from the 320,257-scan DinoV2 training set. Then (a) measure top-1 retrieval accuracy for queries drawn from those held-out models against a reference database containing only other held-out scans; and (b) rerun the Table III evaluation using the retrieval-selected reference instead of an oracle-selected candidate, reporting mHIoU and mAP with confidence intervals across repeated reference choices. If held-out retrieval accuracy is not near 1.0, or if end-to-end mHIoU drops below the Copy Reference or Automatic Detection baselines, the central system-level claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the joint reference-plus-detection approach outperforms both baselines, where the reference is obtained automatically by instance retrieval. Yet the redaction evaluation never includes that retrieval step. Section V.C states: \"we randomly select one document from all candidates as a reference,\" i.e., the reference is chosen from ground-truth candidates of the correct type. Thus Tables III, IV, and V measure only the redaction-combination stage under an oracle, not the complete pipeline described in the abstract and introduction.\n\nThe retrieval evaluation in Section V.A reports accuracy 1.0 on 54 document types, but does not state whether those types were held out from the 320,257 scans used to train DinoV2 from scratch. If the same document models (perhaps even the same scans) were in the training set, accuracy 1.0 is expected and says nothing about generalization to a new document model. The system's practical promise is that one anonymized reference suffices for unseen documents of the same type; that promise depends on the retrieval model finding the correct reference without human help.\n\nBecause the redaction results in Table III (mHIoU 0.741 vs. 0.571 and 0.486) are computed with a correct reference, a retrieval failure could send the wrong reference into the matching stage, and the reported margin could shrink or invert. This is the most load-bearing unverified assumption: the one component that distinguishes the proposed system from the baselines is exactly the component excluded from the end-to-end redaction evaluation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-stage pipeline for anonymizing scanned law-enforcement documents. A self-supervised DINOv2 model, trained from scratch on 320,257 scans from a state criminal investigation office, is used for instance retrieval to find a manually anonymized reference document of the same model; then a set of object detectors (YuNet for faces, PP-OCRv3 for text, a custom YOLO model for barcodes/stamps, and a morphological MRZ detector) predicts sensitive regions, which are combined with the reference redactions via affine alignment and class-specific matching rules. The method is evaluated on a hand-annotated dataset of 206 scans across six document types, with baselines consisting of automatic detection alone and naive copying of the reference redactions. The authors report that the proposed method achieves mHIoU 0.741 versus 0.571 and 0.486 for the two baselines, and mAP 0.445 versus 0.245 and 0.166. The code and data are proprietary and not released.","tokens_in":10816,"tokens_out":2800,"duration_ms":29207,"significance":"If the claimed results hold for a realistically deployed pipeline, the work would be a useful practical contribution to document anonymization, particularly because it explicitly targets downstream forensic usability by minimizing redacted areas rather than blanket-redacting entire scans. The manuscript has several genuine strengths: the evaluation is conducted on a domain-specific, expert-annotated dataset; the authors compare against two well-chosen baselines that isolate the contribution of each pipeline component; and they use both mHIoU and mAP so that localization quality and detection quality are separately visible. However, the central claim is stronger than the evidence: the headline comparison in Tables III–V evaluates the redaction-matching stage with an oracle-selected reference, while the retrieval component—the one element that distinguishes the proposed system from a purely manual-reference approach—is evaluated separately with no stated held-out split. The practical promise of 'only one anonymized example' depends on retrieval working on unseen scans, and that is precisely the part of the pipeline that is not end-to-end validated.","major_comments":[{"comment":"The instance retrieval evaluation reports an accuracy of 1.0 across 54 document types, but the manuscript never states that those document models, or the individual scans used for the queries, were excluded from the 320,257 scans used to train DINOv2 from scratch. If the evaluated document models (or the same scans) were part of the training set, perfect retrieval accuracy is expected from memorization and says nothing about generalization to a new, unseen scan of a known document type. This is load-bearing because the deployment scenario requires retrieving the correct reference for documents that the database must process at inference time, not for documents already used in training.","section":"V.A"},{"comment":"The redaction evaluation sidesteps the retrieval step: the paper states that 'we randomly select one document from all candidates as a reference,' where the candidates are the manually annotated ground-truth documents of the tested type. Thus Tables III, IV, and V measure only the combination of automatic detection with a correct reference, not the end-to-end retrieval-plus-redaction pipeline that the abstract and introduction claim to outperform the baselines. A retrieval failure could send a wrong document type into the matching stage, and the reported margins (mHIoU 0.741 versus 0.571 and 0.486) could shrink or invert. An end-to-end evaluation using the actual retrieved reference, not an oracle-chosen one, is required to support the headline claim.","section":"V.C"},{"comment":"The reported mHIoU and mAP values are point estimates without any measure of variance. Because the reference document is randomly selected from the candidate set of the same type, and because the annotated evaluation set is small (206 images), the results in Tables III–V could depend substantially on that single random draw. The authors should repeat the reference selection many times (or report per-candidate statistics) and provide means and standard deviations, so the reader can judge the stability of the 0.741 versus 0.571/0.486 comparison.","section":"V.C"},{"comment":"The German bank transfer form row is an exception to the headline claim: the proposed method achieves mHIoU 0.569, below the Copy Reference baseline's 0.673, and the text-class mAP drops from 0.281 to 0.096. The authors acknowledge the OCR failure in this case, but the abstract's unqualified statement that the approach 'outperforms both' baselines is too strong. The limitation is real and should be either incorporated into the abstract-level claim or accompanied by an analysis of which document properties cause the proposed method to underperform its copy-reference baseline.","section":"V.C, Table V"}],"minor_comments":[{"comment":"In the discussion following Table V, the phrase 'the ground truth baseline only transfers the bounding boxes' appears to refer to the Copy Reference baseline, not the ground truth; this terminology is confusing and should be corrected.","section":"V.C"},{"comment":"The dataset name 'Flicker logo dataset' should be 'Flickr logo dataset,' and the author name 'Van Rooji' should be 'Van Rooij' for consistency with the cited reference.","section":"II"},{"comment":"Equation (1) uses x_ref and x_pred without defining the exact coordinate convention (e.g., top-left x-coordinate of the bounding box); a short definition or a figure would remove ambiguity.","section":"IV.A.2"},{"comment":"The face-image rule 'add any (transformed) reference bounding box that does not overlap with any of the predicted bounding boxes' does not specify an IoU threshold for 'overlap'; a precise threshold would make the method reproducible.","section":"IV.A.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reports no open-source code or data, which limits reproducibility but is understandable given the proprietary law-enforcement setting. The main concern is not the use of proprietary data per se but whether the evaluation protocol supports the end-to-end claim; the missing held-out retrieval evaluation and the oracle-based reference selection are both fixable within the manuscript's scope by re-running the experiments and reporting error bars. If the authors cannot access a held-out set of document models for retrieval, they should at least clearly report which of the 54 evaluated types were seen during DINOv2 training and temper the abstract's claim accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a practical engineering contribution, not a scientific breakthrough. The core idea—use one expert-redacted reference document to correct and refine automatic detection boxes—is sensible, and the paper evaluates it on proprietary data with a per-class and per-document-type breakdown that shows where it helps (text) and where it doesn't (signatures). The limitation section is honest, including the bank transfer form where the method underperforms and the recommendation that a human still verifies redactions.\n\nThe main problem is the mismatch between what the abstract promises and what the evaluation actually measures. The redaction results in Tables III–V are computed with a reference document randomly selected from ground-truth candidates of the correct type, so the instance retrieval component is completely bypassed. That means the headline claim—that the full retrieval-plus-redaction system outperforms the baselines—is not supported. The retrieval evaluation in Section V.A reports 1.0 accuracy on 54 document types but never states whether those types were held out from the 320,257 scans used to train DINOv2 from scratch. Without that, 1.0 is expected and tells you little about generalization to a new document model. This matters because a retrieval failure feeds the wrong reference into the redaction stage, and the reported margin could shrink or invert.\n\nThere are also smaller gaps. No error bars or variance are reported for the redaction metrics, even though the reference is randomly selected. The claim that the system significantly reduces processing time has no runtime measurements. And the proprietary dataset and code mean the results cannot be independently reproduced.\n\nNone of this kills the redaction-combination result in Table III. Given a correct reference, the matching and affine alignment clearly outperform both baselines by a large margin on this dataset. That is a legitimate contribution, and the paper deserves credit for reporting the failure case on the German bank transfer form rather than hiding it.\n\nThe paper will be useful to people building redaction tools for structured documents and to privacy/forensics researchers looking for practical integration ideas. It deserves a serious referee, but the revision should either evaluate the full pipeline including retrieval or clearly limit the claim to redaction given a known reference. Also add a held-out split for retrieval and error bars for the redaction averages. I would send it to peer review with major revision expected.","headline":"A useful applied paper whose central redaction idea is plausible, but the headline end-to-end claim is unsupported because the evaluation never exercises the retrieval step.","tokens_in":11359,"tokens_out":1883,"would_cite":false,"duration_ms":19253,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A hybrid system that adds one reference redaction to automatic detectors redacts PII on scanned documents more accurately than either component alone.","keywords":["document anonymization","PII redaction","instance retrieval","DINOv2","object detection","law enforcement documents","affine transformation","data protection"],"falsifier":"Run the full pipeline on a held-out document model: take a scan of a new model, retrieve its reference from the database with one expert redaction, and measure mHIoU on ground-truth PII boxes; if retrieval selects a wrong model or the affine alignment fails on a shifted, cropped, or low-quality scan, the 0.741 advantage over the automatic baseline will not reproduce.","tokens_in":10337,"feed_emoji":"🔏","tokens_out":6333,"duration_ms":60557,"temperature":0.7,"pith_summary":"The paper argues that law-enforcement document anonymization can be made both safer and less labor-intensive by combining two sources of information: automatic detectors that locate faces, text, barcodes, and machine-readable zones, and a single manually anonymized reference document of the same type. Its central claim is that this joint reference-plus-detection scheme produces redactions that align more closely with expert ground truth than either component alone, with a mean Hungarian IoU of 0.741 versus 0.486 for automatic detection and 0.571 for a naive copy of the reference mask. The authors also report that a self-supervised retrieval model can find the right reference document by image similarity, removing the need for manual document classification or metadata. If the claim holds, institutions that must comply with data-protection rules could anonymize large collections of scanned documents from one expert example, while preserving more of the document for forensic use.","feed_headline":"Reference-guided redaction beats automatic-only masking on scanned IDs","feed_subtitle":"A hybrid pipeline uses one expert-anonymized sample plus detectors, cutting manual effort while keeping redactions tight.","key_machinery":"The load-bearing mechanism is a redaction-matching pipeline that fuses detector boxes with reference boxes. A DINOv2 model trained from scratch on 320,257 scanned documents retrieves the nearest reference document by cosine similarity; A-KAZE keypoints matched by Hamming distance and aligned with RANSAC supply an affine transform that maps reference redaction boxes onto the target scan; and per-class matching rules adjust widths so that, for text, only the true PII span is masked rather than the full detected text line. The reference box tells the system what needs redacting, and the detector box tells it where each element actually sits in the target document.","core_discovery":"On a hand-annotated evaluation set of 206 scanned documents across six document types and seven countries, the paper shows that using transformed redaction boxes from a reference document to filter and adjust automatically detected bounding boxes yields better redaction quality than either baseline. The best results come from merging detections with reference boxes differently per content type: text boxes inherit the left edge and width from the best-matching detected text region while the reference decides which text is PII; face, MRZ, and barcode boxes are matched and adjusted similarly; signatures are transferred from the reference alone because pre-trained signature detectors fail on this domain. The overall improvement is largest on text, where automatic detection alone redacts nearly all text and the copy-reference baseline ignores text length, while the proposed method reaches 0.527 mAP on text versus 0.019 and 0.153 for the baselines.","pith_inferences":["Because the redaction evaluation picks the reference randomly from ground-truth candidates of the correct type, the reported 0.741 mHIoU measures the redaction stage under perfect retrieval; end-to-end accuracy with retrieval in the loop remains a testable open question.","A natural stress test is to feed the system a document model that was absent from retrieval training and measure whether retrieval remains correct, which would reveal whether the reported perfect retrieval is a general layout-level skill or a memorization of familiar types.","The same matching logic could transfer to other structured documents such as invoices, forms, and medical records, where one expert-redacted template could drive redaction across entire batches if the retrieval model has enough examples of each layout.","The reference document itself contains the same PII categories, so the database of reference documents must be protected as sensitive data even while it enables anonymization of the larger collection."],"forward_implications":["A single expert-anonymized reference document can be reused to redact every other scan of the same document model, reducing per-document manual work to verification.","Adding a new document type does not require retraining the retrieval component, because matching is instance-based rather than tied to a fixed set of classes.","The selective text masking avoids the over-redaction of purely automatic methods, which mask all detected text and therefore destroy non-PII content.","The reported retrieval accuracy of 1.0 over 54 document models suggests the reference can be selected automatically from image content alone, without document metadata.","For content types where detectors are weak, such as signatures, the affine-aligned reference boxes still provide a usable fallback."],"supporting_citations":[{"why":"Supplies the self-supervised DINOv2 visual features, trained from scratch on 320,257 scanned documents, used for instance retrieval of the reference document.","marker":"[20]"},{"why":"Provides the YOLO architecture on which the custom barcode and stamp detector is trained to predict barcode bounding boxes.","marker":"[8]"},{"why":"Provides the PP-OCRv3 text detection stage used to locate text regions in the target document before matching against reference redactions.","marker":"[25]"},{"why":"Supplies the pre-trained YuNet face detector used to locate face images, with reference boxes compensating for missed faces such as changeable laser images.","marker":"[24]"},{"why":"Provides the A-KAZE keypoint detection that finds corresponding points between the target scan and the reference document.","marker":"[22]"},{"why":"Provides the RANSAC algorithm used to compute a robust affine transformation from the matched keypoints.","marker":"[23]"},{"why":"Supplies the PassportEye implementation of the MRZ detection algorithm based on morphological closing and the Scharr gradient.","marker":"[29]"},{"why":"Frames instance retrieval as matching document model, country, and type, which motivates the category-level retrieval evaluation.","marker":"[19]"}],"fun_headline_variants":["Hybrid redaction with one reference beats automatic masking","Reference-guided redaction tops auto-only and copy-paste baselines","One anonymized example transforms redaction accuracy on scans","Better PII redaction: use a reference, not just auto-detection","Reference boxes refine detections to redact PII on documents"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a correctly anonymized reference document of the exact same model will always be available and will be found by the retrieval model; the paper reports perfect retrieval on 54 types but does not state that those types were held out from training, and the redaction evaluation avoids the retrieval step by choosing the reference randomly.","fun_headline_variants_meta":{"raw":{"variants":["Hybrid redaction with one reference beats automatic masking","Reference-guided redaction tops auto-only and copy-paste baselines","One anonymized example transforms redaction accuracy on scans","Better PII redaction: use a reference, not just auto-detection","Reference boxes refine detections to redact PII on documents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00027,"raw_usage":{"total_tokens":1587,"prompt_tokens":866,"completion_tokens":721,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":482,"completion_tokens_details":{"reasoning_tokens":635}},"tokens_in":482,"tokens_out":721,"duration_ms":7771,"temperature":1.0,"reasoning_tokens":635,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:43:51.006106+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full pipeline on a held-out document model: take a scan of a new model, retrieve its reference from the database with one expert redaction, and measure mHIoU on ground-truth PII boxes; if retrieval selects a wrong model or the affine alignment fails on a shifted, cropped, or low-quality scan, the 0.741 advantage over the automatic baseline will not reproduce.","supporting_citations":[{"cited_title":"DINOv2: Learning Robust Vi- sual Features without Supervision,","cited_arxiv_id":null,"evidence_quote":"Supplies the self-supervised DINOv2 visual features, trained from scratch on 320,257 scanned documents, used for instance retrieval of the reference document."},{"cited_title":"Ultralyt- ics/yolov5: V7.0 - YOLOv5 SOTA Realtime Instance Segmentation,","cited_arxiv_id":null,"evidence_quote":"Provides the YOLO architecture on which the custom barcode and stamp detector is trained to predict barcode bounding boxes."},{"cited_title":"PP-OCRv3: More Attempts for the Improvement of Ultra Lightweight OCR System,","cited_arxiv_id":null,"evidence_quote":"Provides the PP-OCRv3 text detection stage used to locate text regions in the target document before matching against reference redactions."},{"cited_title":"YuNet: A Tiny Millisecond-level Face Detector,","cited_arxiv_id":null,"evidence_quote":"Supplies the pre-trained YuNet face detector used to locate face images, with reference boxes compensating for missed faces such as changeable laser images."},{"cited_title":"Fast Explicit Diffusion for Accelerated Features in Nonlinear Scale Spaces,","cited_arxiv_id":null,"evidence_quote":"Provides the A-KAZE keypoint detection that finds corresponding points between the target scan and the reference document."},{"cited_title":"Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography,","cited_arxiv_id":null,"evidence_quote":"Provides the RANSAC algorithm used to compute a robust affine transformation from the matched keypoints."},{"cited_title":"Konstantint/PassportEye,","cited_arxiv_id":null,"evidence_quote":"Supplies the PassportEye implementation of the MRZ detection algorithm based on morphological closing and the Scharr gradient."},{"cited_title":"Deep Learning for Instance Retrieval: A Survey,","cited_arxiv_id":null,"evidence_quote":"Frames instance retrieval as matching document model, country, and type, which motivates the category-level retrieval evaluation."}],"review_version":1}