Pith. sign in

REVIEW 6 major objections 5 minor 17 references

Image deidentification in the XNAT ecosystem: use cases and solutions

T0 review · 6 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A rule-based XNAT/DicomEdit pipeline removes essentially all metadata-level patient identifiers from DICOM images, with an estimated 0.19% genuine failure rate on the MIDI-B benchmark.

desk verdict Useful engineering report with a real bug discovery, but the 99.81% headline is an unratified internal estimate and the 'all names' claim is contradicted by the paper's own results. read the letter →

arxiv 2504.20657 v1 pith:EQIAPWLH submitted 2025-04-29 cs.CV

Alex Michie , Simon J Doran This is my paper · ORCID
classification cs.CV
keywords DICOMdeidentificationXNATMIDI-Bbenchmarkprotectedhealthinformationburned-inPHICleanDescriptorsmedicalimageprivacymetadatascrubbing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

XNAT is a widely used server platform for curating research collections of DICOM medical images, and this paper describes a complete workflow for removing protected health information from those files before sharing. The authors' central claim is that a purely rule-based anonymisation script, built on the DicomEdit library and covering DICOM's Basic Confidentiality Profile plus the Clean Descriptors option, removed every patient, staff and hospital name from the MIDI-B challenge test corpus while keeping scientifically useful text, with an estimated genuine failure rate of 0.19% and an estimated overall performance of 99.81%. The reason this matters is practical: if correct, a single script can be reused across institutions and XNAT servers, so deidentification can happen at the data source rather than at a central hub. The paper also finds that two off-the-shelf machine-learning text recognizers did not help overall, because they over-redacted non-identifying free text, and that the unsolved residue is concentrated in address fields and in identifying information burned into image pixels.

What carries the argument

The load-bearing object is the DicomEdit anonymisation script that turns DICOM Part 3.15 Section E's Basic Confidentiality Profile, and the Clean Descriptors option, into explicit per-tag operations. Clean Descriptors means free-text tags such as Series Description must be searched and scrubbed, not just emptied; the paper's implementation uses regular expressions for names, dates and digit runs, trigger prepositions such as 'for' and 'by', and cross-checks against the Patient Name and Patient ID tags. Around the script sit a preprocessing step that forces series-level consistency of key tags and a containerised OCR redaction step for burned-in pixel text; a private-tag workaround also fixes a DicomEdit bug with implicit-VR files that had been breaking downstream Python readers.

What would settle it

Run the final corrected script on a fresh corpus of DICOM files into which a known set of addresses and nested report names has been inserted, and have two independent human reviewers compare the outputs against ground truth: if residual identifiers exceed 0.19% of the inserted PHI, the central estimate fails. The same test should include at least one instance of a patient name inside a deeply nested (0040,A160) text sequence, the exact case that produced two of the paper's only name-level misses.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a standards-transcribed script can match or nearly match machine-learning approaches on metadata deidentification, and can do so with predictable, auditable behaviour: rules caught 99%+ of removable PHI, all name-bearing strings were redacted, and the residual errors were almost entirely addresses in free-text fields plus two deeply nested name instances in tag (0040,A160). When machine-learning recognizers were appended, they removed a few more address tokens but increased the count of wrongly removed scientific text, so the overall benchmark score fell; the authors therefore frame the next step as targeted address recognition and improved pixel-level redaction rather than general NLP. They estimate the genuine failure rate at 0.19%, after excluding answer-key items still under discussion.

Load-bearing premise

The MIDI-B challenge corpus, with its inserted fake names and addresses and its answer key, is a valid enough stand-in for real clinical DICOM workloads that the reported 99.5–99.8% scores mean real privacy protection.

Editorial extensions

If this is right

  • A single maintained anonymisation script can standardise deidentification across many sites, letting each site run the same rules on its own hardware before data leave the hospital.
  • Residual privacy risk in metadata is concentrated enough that a focused address recognizer, combined with the existing rules, could plausibly push genuine failures well below 0.19%.
  • Adding general-purpose NLP redaction to a working rules pipeline can lower overall performance through over-redaction, so such models need per-tag retuning and validation before deployment.
  • The same pipeline transferred to DICOM-SEG data preserved ROI-to-image linkage while replacing UIDs, suggesting the script-based approach extends beyond image series.
  • Series-level preprocessing that harmonises tags across slices can cut archive errors by roughly 84%, which matters for multi-slice research data received from multiple centres.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 99.81% figure should not be read as a general privacy guarantee: the MIDI-B corpus uses synthetically inserted PHI and an answer key still under revision, and the authors themselves call the rule set brittle for mixed international data, so real-world rates will depend on how closely a deployment matches the benchmark's assumptions.
  • Because the only name-level misses were in deeply nested (0040,A160) text sequences, a recursive rule that traverses that sequence and redacts any token matching Patient Name would likely close the name gap entirely; that is a testable one-line extension.
  • The over-aggressive ML finding suggests a different architecture: let rules delete high-precision tokens (names, dates, ID patterns) and reserve NLP for detecting complete address blocks, then require the block to be redacted as a whole before accepting the model's output.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 5 minor

Summary. The manuscript presents a DICOM deidentification workflow for the XNAT platform, combining DicomEdit-based metadata scrubbing with optional Microsoft Presidio-based redaction of burned-in text. It reports the authors' participation in the MIDI-B challenge: a test-phase score of 97.91%, improved via the Continuous Benchmarking facility to 99.61% using a purely rule-based pipeline, and 99.54% when ML recognizers (FLAIR+StanfordAIMI or clinical RoBERTa) are added. The authors also discuss deidentification use cases, a DicomEdit private-tag bug, and estimate the 'genuine' failure rate on the MIDI-B test corpus at 0.19%.

Significance. The paper is potentially useful to the imaging-informatics community: it gives a fairly detailed account of a production-grade XNAT/DicomEdit pipeline, documents a real bug in DicomEdit's implicit-VR private-tag handling, and provides a cautious empirical comparison of rule-based and ML-based free-text redaction. The explicit statements about brittleness and generalization limits are welcome. However, the headline validation claims are weakened by internal contradictions and by the reliance on an unratified, self-defined estimate, so the significance as a benchmark-validated methodology is conditional on revision.

major comments (6)
  1. [Abstract; Section 5; Conclusion] The Abstract states that the rule-based approach 'was shown to be capable of removing all name-related information in the test corpus' and the Conclusion repeats 'complete redaction of all patient, staff and hospital name information'. However, Section 5 ('PHI that needs to be removed') reports that 'only two actual instances of patient names were missed' out of 432k <text_removed> actions. These statements are mutually inconsistent. Since missed patient names are exactly the residual privacy leaks that matter, the 'complete' claim must be corrected to 'except two instances' (with an assessment of how many patients are affected), or the abstract and conclusion must be reworded.
  2. [Section 4.2; Table 1; Conclusion] The headline figure 'We estimate our overall performance to be 99.81%' derives from the fourth block of Table 1, which the paper itself describes as 'our internal estimate — i.e., not officially ratified results' based on 'technical discussions with the organisers' about answer-key changes. At submission, this number is not externally verifiable, excludes <text_retained> over-redaction and DICOM-conformance repairs by construction, and could change with the organizers' decisions. The paper should clearly separate this estimate from the official benchmark scores (97.91% test, 99.61% Continuous Benchmarking), provide the exact definition of 'genuine deidentification failure' and the arithmetic leading to 0.19%, and not present the estimate as the paper's headline performance without these caveats.
  3. [Section 4.1-4.2; Table 1] The overall scores (97.91%, 99.61%, 99.54%) are reported without any definition of the underlying metric. The text refers to TCIA 'action' categories (<text_removed>, <text_retained>, etc.) and to 'series-level' aggregation, but does not specify how errors in different categories are weighted, how series-level counts are formed, or how the aggregate percentage is computed. Without this, the reader cannot interpret the reported percentages or compare the rule-based and ML pipelines. Please state the exact scoring formula or cite the challenge's scoring specification with enough detail to reproduce the numbers.
  4. [Sections 3.3 and 4.2] The ML experiments that support the claim that ML models are 'over-aggressive' are not reproducible as reported. The paper names the models (FLAIR, StanfordAIMI-deidentifier-base, obi/deid_roberta_i2b2) but does not give hyperparameters, prediction thresholds, text pre-processing, or the 'number of tags that the model was instructed to inspect' that is mentioned as a variable for RoBERTa. No code or configuration files are provided. Please supply full experiment configurations, ideally via a repository, or reduce the strength of the conclusions to a preliminary observation.
  5. [Sections 4.1-4.2; Conclusion] The 99.61% and 99.54% scores were obtained through iterative refinement against the MIDI-B test corpus using the Continuous Benchmarking facility and organizer feedback after the original test phase. These numbers are therefore the result of test-set tuning and do not provide an unbiased estimate of expected performance on unseen data. The conclusion's phrase 'validated as part of the MIDI-B challenge' overstates the evidentiary value of these scores. The paper should explicitly acknowledge this non-independence and, if it wishes to make a validation claim, base it on the validation-phase score or on a properly held-out split.
  6. [Section 4.2 (second block of Table 1)] The text reports that the rules-based approach 'reduces errors in the <text_removed> category to 46, close to a 1% failure rate'. In the same paper, the total number of <text_removed> actions is given as 432k; 46/432,000 is 0.011%, not approximately 1%. Similarly, 372 <text_retained> errors are described as a 0.29% failure rate, which would imply a denominator of about 128k rather than 432k. Please reconcile the denominators and correct the percentages, since this is a quantitative claim about the pipeline's performance.
minor comments (5)
  1. [Section 1.2] Define 'protected healthcare information (PHI)' and 'patient identifiable data (PID)' at first use and use them consistently throughout.
  2. [Section 3.3] The list of trigger prepositions ('for', 'by', 'at', 'to', 'on') is presented without justification; add a brief explanation of how the list was derived and its expected precision/recall trade-off.
  3. [Section 4.2] The reduction 'of almost 400k' instance-level warnings is given in parentheses as '49,897 errors in baseline to 6,956'; consider stating these exact numbers in the main text for clarity.
  4. [Table 1] Ensure that the four blocks of Table 1 and the column definitions are self-explanatory, especially the connection between the 'fourth block' and notes (1)-(10), so the reader can follow the 'internal estimate' derivation.
  5. [Section 5] When reporting 'only two actual instances of patient names were missed', also report the number of distinct patients affected and the context (e.g., the (0040,A160) tag type), since two instances may represent the same patient or two different patients.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the 99.81% figure is an explicitly labeled internal estimate, not a prediction derived from its own inputs.

full rationale

The paper makes no claim to derive the MIDI-B scores from first principles. Its central numbers (97.91%, 99.61%, 99.54%) are external benchmark results, and the 99.81% figure is explicitly qualified: 'The fourth block of data in Table 1 represents our internal estimate — i.e., not officially ratified results from the MIDI-B Continuous Benchmarking facility' and 'we estimate that our percentage of “genuine” deidentification failures ... currently stands at 0.19%.' These are post-hoc reclassifications after 'technical discussions with the organisers,' which undermines independent validation but is not circular, because no parameter is fitted and then renamed as a prediction. The rule-based scripts were adapted using validation-phase and post-submission feedback, so the later scores are in-sample and should not be read as unbiased estimates; that is a statistical-validity caveat, not a definitional equivalence. There is no load-bearing self-citation: cited works (NCITA, DPUK, etc.) describe infrastructure, not the deidentification result, and no uniqueness theorem is invoked. Internal inconsistencies (Abstract/Conclusion claim 'all name-related information' removed vs. Section 5 reporting 'only two actual instances of patient names were missed') are correctness/consistency issues, not circular derivation. The paper's own admission that the approach is 'inherently brittle' and that performance 'would not generalise to a truly mixed international dataset' further limits the strength of the central claim, but again does not make the derivation circular. Under the specified patterns, no step reduces to its own input by construction.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The performance claims rest on hand-crafted rules tuned to the MIDI-B validation set, on external gold standards (DICOM Part 3.15, TCIA private tag list, MIDI-B answer key), and on the assumption that the benchmark corpus is representative of real-world DICOM data.

free parameters (4)
  • Trigger preposition list = for/by/at/to/on
    Hand-chosen from MIDI-B validation data to delete following text (Section 3.3).
  • Digit group threshold = 9 characters
    Regular expression deletes groups of 9+ characters from {digits, parentheses, dash, x} as possible phone/SSN numbers (Section 3.3).
  • Date/year patterns = yyyymmdd; 20th/21st century years
    Hand-set regexes for date strings and years (Section 3.3).
  • Retained private tag list = TCIA Private Tag Knowledge Base 02-01-2024
    Configuration choice to maximize challenge score by retaining TCIA 'safe' private tags; resulted in DicomEdit implicit-VR bug (Sections 3.2, 4.1).
assumptions (3)
  • domain assumption DICOM standard Part 3.15 Section E (Table E.1-1) defines correct deidentification actions.
    Used throughout to construct anonymisation scripts; errors occur where the standard is ambiguous (X/Z/D actions).
  • domain assumption MIDI-B answer key and TCIA scoring correctly distinguish PHI to remove from scientifically useful text.
    All performance numbers derive from this key; authors dispute parts of it, making the 99.81% estimate uncertain.
  • domain assumption The MIDI-B challenge corpus is representative of general DICOM deidentification workloads.
    Authors note their rules are brittle and may not generalize to non-Anglophone or mixed datasets (Section 5), so generalization is assumed rather than demonstrated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Image deidentification in the XNAT ecosystem: use cases and solutions." pith.science (2026). https://pith.science/paper/EQIAPWLH

@misc{pith2026250420657,
  author       = {Pith},
  title        = {Pith review of: Image deidentification in the XNAT ecosystem: use cases and solutions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EQIAPWLH}},
  note         = {Machine review of arXiv:2504.20657}
}
read the original abstract

XNAT is a server-based data management platform widely used in academia for curating large databases of DICOM images for research projects. We describe in detail a deidentification workflow for DICOM data using facilities in XNAT, together with independent tools in the XNAT "ecosystem". We list different contexts in which deidentification might be needed, based on our prior experience. The starting point for participation in the Medical Image De-Identification Benchmark (MIDI-B) challenge was a set of pre-existing local methodologies, which were adapted during the validation phase of the challenge. Our result in the test phase was 97.91\%, considerably lower than our peers, due largely to an arcane technical incompatibility of our methodology with the challenge's Synapse platform, which prevented us receiving feedback during the validation phase. Post-submission, additional discrepancy reports from the organisers and via the MIDI-B Continuous Benchmarking facility, enabled us to improve this score significantly to 99.61\%. An entirely rule-based approach was shown to be capable of removing all name-related information in the test corpus, but exhibited failures in dealing fully with address data. Initial experiments using published machine-learning models to remove addresses were partially successful but showed the models to be "over-aggressive" on other types of free-text data, leading to a slight overall degradation in performance to 99.54\%. Future development will therefore focus on improving address-recognition capabilities, but also on better removal of identifiable data burned into the image pixels. Several technical aspects relating to the "answer key" are still under discussion with the challenge organisers, but we estimate that our percentage of genuine deidentification failures on the MIDI-B test corpus currently stands at 0.19\%. (Abridged from original for arXiv submission)

Figures

Figures reproduced from arXiv: 2504.20657 by the authors.

Figure 3
Figure 3. shows an overview of the MIDI-B deidentification [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 1
Figure 1. Overview of PHI-removal process for studies im [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 11 canonical work pages

  1. [1]

    FLAIR : An easy-to-use framework for state-of-the-art NLP

    Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. FLAIR : An easy-to-use framework for state-of-the-art NLP . In NAACL 2019, 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations) , pages 54--59, 2019

    work page 2019

  2. [2]

    Text sanitization beyond specific domains: Zero-shot redaction & substitution with large language models

    Federico Albanese, Daniel Ciolek, and Nicolas D'Ippolito. Text sanitization beyond specific domains: Zero-shot redaction & substitution with large language models. arXiv preprint arXiv:2311.10785, 2023

    arXiv 2023

  3. [3]

    The dementias platform uk (dpuk) data portal

    Sarah Bauermeister, Christopher Orton, Simon Thompson, Roger A Barker, Joshua R Bauermeister, Yoav Ben-Shlomo, Carol Brayne, David Burn, Archie Campbell, Catherine Calvin, et al. The dementias platform uk (dpuk) data portal. European journal of epidemiology, 35: 0 601--611, 2020

    work page 2020

  4. [4]

    hide in plain sight

    Pierre J Chambon, Christopher Wu, Jackson M Steinkamp, Jason Adleberg, Tessa S Cook, and Curtis P Langlotz. Automated deidentification of radiology reports combining transformer and “hide in plain sight” rule-based methods . Journal of the American Medical Informatics Association, 11 2022. ISSN 1527-974X. doi:10.1093/jamia/ocac219. URL https://doi.org/10....

    doi:10.1093/jamia/ocac219 2022

  5. [5]

    Image data sharing for biomedical research—meeting hipaa requirements for de-identification

    John B Freymann, Justin S Kirby, John H Perry, David A Clunie, and C Carl Jaffe. Image data sharing for biomedical research—meeting hipaa requirements for de-identification. Journal of digital imaging, 25 0 (1): 0 14--24, 2012

    2012

  6. [6]

    The washington university central neuroimaging data archive

    Jenny Gurney, Timothy Olsen, John Flavin, Mohana Ramaratnam, Kevin Archie, James Ransford, Rick Herrick, Lauren Wallace, Jeanette Cline, Will Horton, et al. The washington university central neuroimaging data archive. Neuroimage, 144: 0 287--293, 2017

    work page 2017

  7. [7]

    Documenting the de-identification process of clinical and imaging data for ai for health imaging projects

    Haridimos Kondylakis, Rocio Catalan, Sara Martinez Alabart, Caroline Barelle, Paschalis Bizopoulos, Maciej Bobowicz, Jonathan Bona, Dimitrios I Fotiadis, Teresa Garcia, Ignacio Gomez, et al. Documenting the de-identification process of clinical and imaging data for ai for health imaging projects. Insights into Imaging, 15 0 (1): 0 130, 2024

    work page 2024

  8. [8]

    Roberta: A robustly optimized bert pretraining approach

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019

    arXiv 1907

Show all 17 references
  1. [9]

    Introduction to the national cancer imaging translational accelerator (ncita): a uk-wide infrastructure for multicentre clinical translation of cancer imaging biomarkers

    MA McAteer, James PB O’Connor, DM Koh, HY Leung, SJ Doran, M Jauregui-Osoro, N Muirhead, C Brew-Graves, ER Plummer, E Sala, et al. Introduction to the national cancer imaging translational accelerator (ncita): a uk-wide infrastructure for multicentre clinical translation of ca...

    2021

  2. [10]

    High performance on-demand de-identification of a petabyte-scale medical imaging data lake

    Joseph Mesterhazy, Garrick Olson, and Somalee Datta. High performance on-demand de-identification of a petabyte-scale medical imaging data lake. arXiv preprint arXiv:2008.01827, 2020

    2008 arXiv

  3. [11]

    Data anonymization in ai and ml engineering: Balancing privacy and model performance using presidio

    S Patchipala. Data anonymization in ai and ml engineering: Balancing privacy and model performance using presidio. Iconic Research and Engineering Journals, 6 0 (10), 2023

    2023

  4. [12]

    An open source toolkit for medical imaging de-identification

    David Rodr \' guez Gonz \'a lez, Trevor Carpenter, Jano I van Hemert, and Joanna Wardlaw. An open source toolkit for medical imaging de-identification. European radiology, 20: 0 1896--1904, 2010

    1904

  5. [13]

    A dicom dataset for evaluation of medical image de-identification

    Michael Rutherford, Seong K Mun, Betty Levine, William Bennett, Kirk Smith, Phil Farmer, Quasar Jarosz, Ulrike Wagner, John Freyman, Geri Blake, et al. A dicom dataset for evaluation of medical image de-identification. Scientific Data, 8 0 (1): 0 183, 2021

    2021

  6. [14]

    A two-stage de-identification process for privacy-preserving medical image analysis

    Arsalan Shahid, Mehran H Bazargani, Paul Banahan, Brian Mac Namee, Tahar Kechadi, Ceara Treacy, Gilbert Regan, and Peter MacMahon. A two-stage de-identification process for privacy-preserving medical image analysis. In Healthcare, volume 10, page 755. MDPI, 2022

    2022

  7. [15]

    An overview of the tesseract ocr engine

    Ray Smith. An overview of the tesseract ocr engine. In Ninth international conference on document analysis and recognition (ICDAR 2007), volume 2, pages 629--633. IEEE, 2007

    2007

  8. [16]

    Industrial-strength natural language processing in python

    spacy.io. Industrial-strength natural language processing in python. URL https://spacy.io. Accessed on 21 March 2025

    2025

  9. [17]

    Tesseract documentation

    Tesseract. Tesseract documentation. URL https://tesseract-ocr.github.io/. Accessed on 21 March 2025

    2025

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.