{"id":"ae1bb52f-3da1-41a3-85b7-36c924daa752","arxiv_id":"2504.20657","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A XNAT/DicomEdit pipeline scored 99.61% on the MIDI-B deidentification benchmark after test-set feedback, with remaining failures concentrated in addresses and burned-in pixels.","lead":"This paper describes a DICOM image deidentification workflow built around the XNAT platform and reports its scores on the MIDI-B benchmarking challenge. It shows that a rule-based approach can remove nearly all patient names but struggles with addresses, and that machine-learning text models over-redact useful metadata.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 99.81% headline is an unofficial, answer-key-dependent reclassification, and the paper's own Section 5 reports two missed patient names that contradict the 'complete name redaction' claim; an official re-run is needed before the central claim can be accepted.","rationale":"The report has genuine strengths: it documents a real bug in DicomEdit's handling of private tags with implicit VR (§4.1), shows a large reduction in instance-level errors from preprocessing, and is candid about the brittleness of rules and the unresolved burned-in-PHI problem. Those strengths make a rejection inappropriate. The concern is that the Conclusion's 99.81% figure is presented as the main validated outcome, whereas the paper itself describes it as an internal estimate contingent on answer-key corrections that are not yet official. The separate contradiction between 'complete name redaction' and two missed patient names strengthens the need for an external check rather than weakening the paper's engineering value. Because the authors already disclose most caveats, the existing CONDITIONAL verdict should remain; no further adjustment is needed, but the manuscript should not be upgraded to ACCEPT until the official re-run or an equivalent independent evaluation is available.","tokens_in":12758,"tokens_out":6936,"duration_ms":73199,"concrete_test":"Ask the MIDI-B organizers to finalize and publish the updated answer key resolving all items listed as 'still under discussion'; then run the exact rules-based DicomEdit pipeline (without any further tuning or feedback) once through the official MIDI-B Continuous Benchmarking scorer and report the official <text_removed> false-negative count and total score. If the official score is below 99.81%, or if the two missed patient names from Section 5 are confirmed as <text_removed> errors, the Conclusion's '0.19% genuine failure rate' and 'complete name redaction' claims are unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The Conclusion's headline ('We estimate our overall performance to be 99.81%') rests on the fourth block of Table 1, which the paper itself labels 'our internal estimate — i.e., not officially ratified results,' computed after 'technical discussions' with the MIDI-B organizers about the answer key. The official scores are 97.91% on the original test phase and 99.61% via the Continuous Benchmarking facility after post-submission feedback. The 0.19% 'genuine deidentification failure' rate is therefore not a benchmark measurement: it is a self-selected subset of errors, excluding categories such as <text_retained> over-redaction and DICOM-conformance 'repairs' that the authors do not regard as genuine deidentification failures. Because the answer-key corrections remain under discussion, the headline number can move with the organizers' decisions and is not currently externally ratifiable. The paper is also internally inconsistent on the strongest sub-claim: the Abstract and Conclusion say the rule-based system removed 'all name-related information,' but Section 5 ('PHI that needs to be removed') states that 'only two actual instances of patient names were missed' out of 432k <text_removed> actions. Two missed patient names are not complete redaction and are precisely the residual privacy leaks that matter for the central claim. Finally, the authors themselves describe the rule-based approach as 'inherently brittle' and doubt generalization to a mixed international dataset (Section 5), so the 0.19% figure is at best a benchmark-specific, answer-key-dependent estimate rather than a validated property of the method.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents a DICOM deidentification workflow for the XNAT platform, combining DicomEdit-based metadata scrubbing with optional Microsoft Presidio-based redaction of burned-in text. It reports the authors' participation in the MIDI-B challenge: a test-phase score of 97.91%, improved via the Continuous Benchmarking facility to 99.61% using a purely rule-based pipeline, and 99.54% when ML recognizers (FLAIR+StanfordAIMI or clinical RoBERTa) are added. The authors also discuss deidentification use cases, a DicomEdit private-tag bug, and estimate the 'genuine' failure rate on the MIDI-B test corpus at 0.19%.","tokens_in":13060,"tokens_out":10739,"duration_ms":92059,"significance":"The paper is potentially useful to the imaging-informatics community: it gives a fairly detailed account of a production-grade XNAT/DicomEdit pipeline, documents a real bug in DicomEdit's implicit-VR private-tag handling, and provides a cautious empirical comparison of rule-based and ML-based free-text redaction. The explicit statements about brittleness and generalization limits are welcome. However, the headline validation claims are weakened by internal contradictions and by the reliance on an unratified, self-defined estimate, so the significance as a benchmark-validated methodology is conditional on revision.","major_comments":[{"comment":"The Abstract states that the rule-based approach 'was shown to be capable of removing all name-related information in the test corpus' and the Conclusion repeats 'complete redaction of all patient, staff and hospital name information'. However, Section 5 ('PHI that needs to be removed') reports that 'only two actual instances of patient names were missed' out of 432k <text_removed> actions. These statements are mutually inconsistent. Since missed patient names are exactly the residual privacy leaks that matter, the 'complete' claim must be corrected to 'except two instances' (with an assessment of how many patients are affected), or the abstract and conclusion must be reworded.","section":"Abstract; Section 5; Conclusion"},{"comment":"The headline figure 'We estimate our overall performance to be 99.81%' derives from the fourth block of Table 1, which the paper itself describes as 'our internal estimate — i.e., not officially ratified results' based on 'technical discussions with the organisers' about answer-key changes. At submission, this number is not externally verifiable, excludes <text_retained> over-redaction and DICOM-conformance repairs by construction, and could change with the organizers' decisions. The paper should clearly separate this estimate from the official benchmark scores (97.91% test, 99.61% Continuous Benchmarking), provide the exact definition of 'genuine deidentification failure' and the arithmetic leading to 0.19%, and not present the estimate as the paper's headline performance without these caveats.","section":"Section 4.2; Table 1; Conclusion"},{"comment":"The overall scores (97.91%, 99.61%, 99.54%) are reported without any definition of the underlying metric. The text refers to TCIA 'action' categories (<text_removed>, <text_retained>, etc.) and to 'series-level' aggregation, but does not specify how errors in different categories are weighted, how series-level counts are formed, or how the aggregate percentage is computed. Without this, the reader cannot interpret the reported percentages or compare the rule-based and ML pipelines. Please state the exact scoring formula or cite the challenge's scoring specification with enough detail to reproduce the numbers.","section":"Section 4.1-4.2; Table 1"},{"comment":"The ML experiments that support the claim that ML models are 'over-aggressive' are not reproducible as reported. The paper names the models (FLAIR, StanfordAIMI-deidentifier-base, obi/deid_roberta_i2b2) but does not give hyperparameters, prediction thresholds, text pre-processing, or the 'number of tags that the model was instructed to inspect' that is mentioned as a variable for RoBERTa. No code or configuration files are provided. Please supply full experiment configurations, ideally via a repository, or reduce the strength of the conclusions to a preliminary observation.","section":"Sections 3.3 and 4.2"},{"comment":"The 99.61% and 99.54% scores were obtained through iterative refinement against the MIDI-B test corpus using the Continuous Benchmarking facility and organizer feedback after the original test phase. These numbers are therefore the result of test-set tuning and do not provide an unbiased estimate of expected performance on unseen data. The conclusion's phrase 'validated as part of the MIDI-B challenge' overstates the evidentiary value of these scores. The paper should explicitly acknowledge this non-independence and, if it wishes to make a validation claim, base it on the validation-phase score or on a properly held-out split.","section":"Sections 4.1-4.2; Conclusion"},{"comment":"The text reports that the rules-based approach 'reduces errors in the <text_removed> category to 46, close to a 1% failure rate'. In the same paper, the total number of <text_removed> actions is given as 432k; 46/432,000 is 0.011%, not approximately 1%. Similarly, 372 <text_retained> errors are described as a 0.29% failure rate, which would imply a denominator of about 128k rather than 432k. Please reconcile the denominators and correct the percentages, since this is a quantitative claim about the pipeline's performance.","section":"Section 4.2 (second block of Table 1)"}],"minor_comments":[{"comment":"Define 'protected healthcare information (PHI)' and 'patient identifiable data (PID)' at first use and use them consistently throughout.","section":"Section 1.2"},{"comment":"The list of trigger prepositions ('for', 'by', 'at', 'to', 'on') is presented without justification; add a brief explanation of how the list was derived and its expected precision/recall trade-off.","section":"Section 3.3"},{"comment":"The reduction 'of almost 400k' instance-level warnings is given in parentheses as '49,897 errors in baseline to 6,956'; consider stating these exact numbers in the main text for clarity.","section":"Section 4.2"},{"comment":"Ensure that the four blocks of Table 1 and the column definitions are self-explanatory, especially the connection between the 'fourth block' and notes (1)-(10), so the reader can follow the 'internal estimate' derivation.","section":"Table 1"},{"comment":"When reporting 'only two actual instances of patient names were missed', also report the number of distinct patients affected and the context (e.g., the (0040,A160) tag type), since two instances may represent the same patient or two different patients.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid systems contribution with useful documentation of real-world pitfalls, but the validation claims need substantial revision. I recommend major revision. The authors' transparency about limitations is commendable, but the abstract and conclusion overstate the results; in particular, the internal estimate of 99.81% should not be highlighted until ratified by the challenge organizers. The arithmetic inconsistency between 46/432k errors and a 'close to 1%' failure rate should be caught during revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things before engaging with this paper. First, it is a genuinely useful engineering report: it gives a detailed, honest account of deidentifying DICOM data inside the XNAT ecosystem, and it surfaces a real DicomEdit bug that can break downstream reading of private tags in implicit-VR files. The bug analysis is solid, and the proposed fix is sensible. Second, the headline number you keep seeing — 99.81% — is not an official benchmark result. It is the authors' internal estimate after ongoing technical discussions with the MIDI-B organizers, based on answer-key corrections not yet incorporated into the official scoring. The paper itself labels that block 'not officially ratified results.' So the number can move, and it should not be cited as the measured performance.\n\nWhat is actually new: this appears to be the first published evaluation of a XNAT/DicomEdit pipeline on the MIDI-B benchmark, and the implicit-VR private tag bug has not been reported before. That bug alone is worth a citable note for anyone using DicomEdit with such files.\n\nThe soft spots are in proportion. The abstract and conclusion claim complete redaction of all name-related information, but Section 5 states that two actual patient names were missed out of 432k removals. Two misses are small, but they contradict the word 'complete.' Also, the improvement from the original 97.91% to 99.61% came through the Continuous Benchmarking facility — that is iterative tuning against the test corpus, so the later scores are not independent measurements. The ML experiments are framed as a brief scoping survey, and they are, with limited configuration detail. The authors themselves call their rule-based approach 'inherently brittle' and doubt it generalizes to a mixed international dataset. That is the right level of humility, but it undercuts any reading of 0.19% as a genuine failure rate for real-world deidentification.\n\nDespite that, the paper holds up as an engineering report. The benchmark data are real, the bug report is concrete, and the authors are transparent about what is and is not official. I would send it to peer review, with a request that the authors qualify the 99.81% as an unratified internal estimate and align the abstract's wording with their own two missed names. For readers who maintain or use XNAT/DicomEdit pipelines, this is worth a look. I don't think it changes practice outside that niche, but it is a solid, honest contribution within it.","headline":"Useful engineering report with a real bug discovery, but the 99.81% headline is an unratified internal estimate and the 'all names' claim is contradicted by the paper's own results.","tokens_in":13515,"tokens_out":3752,"would_cite":false,"duration_ms":30525,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A rule-based XNAT/DicomEdit pipeline removes essentially all metadata-level patient identifiers from DICOM images, with an estimated 0.19% genuine failure rate on the MIDI-B benchmark.","keywords":["DICOM deidentification","XNAT","MIDI-B benchmark","protected health information","burned-in PHI","Clean Descriptors","medical image privacy","DICOM metadata scrubbing"],"falsifier":"Run the final corrected script on a fresh corpus of DICOM files into which a known set of addresses and nested report names has been inserted, and have two independent human reviewers compare the outputs against ground truth: if residual identifiers exceed 0.19% of the inserted PHI, the central estimate fails. The same test should include at least one instance of a patient name inside a deeply nested (0040,A160) text sequence, the exact case that produced two of the paper's only name-level misses.","tokens_in":12585,"feed_emoji":"🩻","tokens_out":9676,"duration_ms":88640,"temperature":0.7,"pith_summary":"XNAT is a widely used server platform for curating research collections of DICOM medical images, and this paper describes a complete workflow for removing protected health information from those files before sharing. The authors' central claim is that a purely rule-based anonymisation script, built on the DicomEdit library and covering DICOM's Basic Confidentiality Profile plus the Clean Descriptors option, removed every patient, staff and hospital name from the MIDI-B challenge test corpus while keeping scientifically useful text, with an estimated genuine failure rate of 0.19% and an estimated overall performance of 99.81%. The reason this matters is practical: if correct, a single script can be reused across institutions and XNAT servers, so deidentification can happen at the data source rather than at a central hub. The paper also finds that two off-the-shelf machine-learning text recognizers did not help overall, because they over-redacted non-identifying free text, and that the unsolved residue is concentrated in address fields and in identifying information burned into image pixels.","feed_headline":"XNAT pipeline strips 99.81% of patient data from scans","feed_subtitle":"Rule-based scrubbing in XNAT beats the MIDI-B benchmark; the hard leftovers are addresses and burned-in image text.","key_machinery":"The load-bearing object is the DicomEdit anonymisation script that turns DICOM Part 3.15 Section E's Basic Confidentiality Profile, and the Clean Descriptors option, into explicit per-tag operations. Clean Descriptors means free-text tags such as Series Description must be searched and scrubbed, not just emptied; the paper's implementation uses regular expressions for names, dates and digit runs, trigger prepositions such as 'for' and 'by', and cross-checks against the Patient Name and Patient ID tags. Around the script sit a preprocessing step that forces series-level consistency of key tags and a containerised OCR redaction step for burned-in pixel text; a private-tag workaround also fixes a DicomEdit bug with implicit-VR files that had been breaking downstream Python readers.","core_discovery":"On the paper's own terms, the discovery is that a standards-transcribed script can match or nearly match machine-learning approaches on metadata deidentification, and can do so with predictable, auditable behaviour: rules caught 99%+ of removable PHI, all name-bearing strings were redacted, and the residual errors were almost entirely addresses in free-text fields plus two deeply nested name instances in tag (0040,A160). When machine-learning recognizers were appended, they removed a few more address tokens but increased the count of wrongly removed scientific text, so the overall benchmark score fell; the authors therefore frame the next step as targeted address recognition and improved pixel-level redaction rather than general NLP. They estimate the genuine failure rate at 0.19%, after excluding answer-key items still under discussion.","pith_inferences":["The 99.81% figure should not be read as a general privacy guarantee: the MIDI-B corpus uses synthetically inserted PHI and an answer key still under revision, and the authors themselves call the rule set brittle for mixed international data, so real-world rates will depend on how closely a deployment matches the benchmark's assumptions.","Because the only name-level misses were in deeply nested (0040,A160) text sequences, a recursive rule that traverses that sequence and redacts any token matching Patient Name would likely close the name gap entirely; that is a testable one-line extension.","The over-aggressive ML finding suggests a different architecture: let rules delete high-precision tokens (names, dates, ID patterns) and reserve NLP for detecting complete address blocks, then require the block to be redacted as a whole before accepting the model's output."],"forward_implications":["A single maintained anonymisation script can standardise deidentification across many sites, letting each site run the same rules on its own hardware before data leave the hospital.","Residual privacy risk in metadata is concentrated enough that a focused address recognizer, combined with the existing rules, could plausibly push genuine failures well below 0.19%.","Adding general-purpose NLP redaction to a working rules pipeline can lower overall performance through over-redaction, so such models need per-tag retuning and validation before deployment.","The same pipeline transferred to DICOM-SEG data preserved ROI-to-image linkage while replacing UIDs, suggesting the script-based approach extends beyond image series.","Series-level preprocessing that harmonises tags across slices can cut archive errors by roughly 84%, which matters for multi-slice research data received from multiple centres."],"supporting_citations":[{"why":"Defines the MIDI-B challenge corpus and its programme of inserted fake PHI, the benchmark all scores in the paper refer to.","marker":"[Rutherford et al., 2021]"},{"why":"Documents the Presidio text-anonymization toolkit on which the paper's ML recognizers and pixel redaction container are based.","marker":"[Patchipala, 2023]"},{"why":"Extends Presidio-style text sanitization with LLM redaction, cited as part of the basis for the redaction pipeline.","marker":"[Albanese et al., 2023]"},{"why":"Provides the FLAIR named-entity recognizer used in the first ML address-removal experiment.","marker":"[Akbik et al., 2019]"},{"why":"Provides the radiology-domain deidentifier model combined with FLAIR in the first ML experiment.","marker":"[Chambon et al., 2022]"},{"why":"Provides the RoBERTa architecture whose clinical fine-tuned version is the second ML experiment.","marker":"[Liu et al., 2019]"},{"why":"Describes the Tesseract OCR engine used to locate burned-in text in image pixels.","marker":"[Smith, 2007]"}],"fun_headline_variants":["Rule-based XNAT de-identification tops ML for DICOM","XNAT rules strip 99.81% of DICOM patient data","DICOM privacy: rule-based XNAT hits 99.81% scrubbing","Addresses are the last hurdle in XNAT DICOM de-identification","Rule-based scrubbing outperforms ML on DICOM privacy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The MIDI-B challenge corpus, with its inserted fake names and addresses and its answer key, is a valid enough stand-in for real clinical DICOM workloads that the reported 99.5–99.8% scores mean real privacy protection.","fun_headline_variants_meta":{"raw":{"variants":["Rule-based XNAT de-identification tops ML for DICOM","XNAT rules strip 99.81% of DICOM patient data","DICOM privacy: rule-based XNAT hits 99.81% scrubbing","Addresses are the last hurdle in XNAT DICOM de-identification","Rule-based scrubbing outperforms ML on DICOM privacy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000796,"raw_usage":{"total_tokens":3552,"prompt_tokens":1041,"completion_tokens":2511,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":657,"completion_tokens_details":{"reasoning_tokens":2410}},"tokens_in":657,"tokens_out":2511,"duration_ms":17776,"temperature":1.0,"reasoning_tokens":2410,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:23:08.810468+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the final corrected script on a fresh corpus of DICOM files into which a known set of addresses and nested report names has been inserted, and have two independent human reviewers compare the outputs against ground truth: if residual identifiers exceed 0.19% of the inserted PHI, the central estimate fails. The same test should include at least one instance of a patient name inside a deeply nested (0040,A160) text sequence, the exact case that produced two of the paper's only name-level misses.","supporting_citations":[{"cited_title":"Data anonymization in ai and ml engineering: Balancing privacy and model performance using presidio","cited_arxiv_id":null,"evidence_quote":"Documents the Presidio text-anonymization toolkit on which the paper's ML recognizers and pixel redaction container are based."},{"cited_title":"FLAIR : An easy-to-use framework for state-of-the-art NLP","cited_arxiv_id":null,"evidence_quote":"Provides the FLAIR named-entity recognizer used in the first ML address-removal experiment."}],"review_version":1}