{"id":"87cd834d-73ec-4bb6-b3bd-f2b961ba2a23","arxiv_id":"2501.09552","paper_version":4,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A modular pipeline with YOLOv11 detecting text, EasyOCR reading it, and GPT-4o classifying PHI outperforms end-to-end GPT-4o and EasyOCR-only setups in accuracy, latency, and cost.","lead":"This paper tests different ways to combine three AI tools, YOLOv11, EasyOCR, and GPT-4o, to spot private patient information written on medical images. It finds that the setup using one tool for each task, finding text, reading it, and deciding if it is private, works best when balancing accuracy, speed, and cost.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central 'optimal setup' claim rests entirely on synthetic overlay data; real burned-in PHI could change the setup ranking. A targeted evaluation on the original MIDI-B organizer burn-ins would test whether the conclusion transfers.","rationale":"I agree with the reader's weakest assumption; it is also the most load-bearing concern for the central claim. Every quantitative comparison supporting the headline recommendation is generated by overlaying text according to the authors' simulation rules, and the paper itself acknowledges the absence of real-world PHI testing. The proposed check is feasible with data already cited by the paper: MIDI-B is public, and isolating the original organizer burn-ins from the newly added viewer overlays tests the simulation rather than merely adding another synthetic dataset. I would not reject the paper: the internal comparisons are extensive, include multiple runs and a holdout, and the limitation is explicitly disclosed. However, the verdict should remain conditional until the external-validity concern is resolved. Other issues (unspecified instance-level matching, post hoc prompt adaptation on MIDI, and partial source overlap between RadPHI-train and RadPHI-test) are addressable and secondary to the synthetic-data concern, because the central recommendation is primarily driven by the simulated distribution.","tokens_in":20159,"tokens_out":6581,"duration_ms":69303,"concrete_test":"Download the original MIDI-B challenge images (Synapse syn53065760), before any DICOM-viewer overlay, and use the challenge's own ground-truth burn-in annotations. Run Setups 1, 2, 3, and 4 on these images with the same prompts and evaluation code, restricting evaluation to the organizer-created pixel burn-ins. Compare case/instance precision-recall and the Setup 1 vs Setup 3 gap. If Setup 1 is not consistently best, or the gap to Setup 3 reverses, the synthetic-overlay simulation is driving the conclusion.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The claim that Setup 1 (YOLOv11 + EasyOCR + GPT-4o) is optimal depends on the evaluation datasets being representative of real burned-in PHI. Both benchmarks are synthetic: RadPHI-test overlays simulated imprints on public images, and MIDI is produced by rendering DICOM tags with MD.ai on MIDI-B validation images, alongside the challenge organizers' own synthetic burn-ins. The authors explicitly state in Section 6 (Limitations) that 'our evaluation does not include testing our proposed approaches on real-world data containing PHI.' Consequently, all setup rankings are measured on a distribution defined by the authors' font/layout/overlay simulator. Real clinical burn-ins often have different contrast, fonts (e.g., scanner-generated text), warping, languages, and partial occlusion, which can change the relative merits of the modules. The margin between Setup 1 and Setup 3 is small at case level (Setup 3 recall 0.9995 vs Setup 1 0.9923; Tables 4, A1, A3), so the choice of Setup 1 is largely driven by token/latency trade-offs on the synthetic distribution. If real imprints make EasyOCR extraction noisier, GPT-4o-based extraction (Setup 3) could become preferable despite higher cost; if they make CRAFT detection more effective, Setup 2 could close the gap. This is the load-bearing external-validity assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes and evaluates a modular pipeline for pixel-level PHI detection in medical images, consisting of text localization, text extraction, and text analysis. The authors compare four configurations built from YOLOv11, EasyOCR, and GPT-4o, plus a Presidio baseline, on two synthetic datasets: RadPHI-test (1,000 images) and MIDI (200 images). They report case-level and (for some setups) instance-level precision/recall, latency, and token costs, and conclude that the setup with dedicated models for each module (YOLOv11 for localization, EasyOCR for extraction, GPT-4o for analysis) is optimal. They also demonstrate that a prompt adaptation improves performance on the MIDI dataset.","tokens_in":20397,"tokens_out":8885,"duration_ms":79870,"significance":"The paper addresses an important and practical problem: automated PHI detection in medical images. The modular analysis is useful for practitioners choosing among OCR and LLM components. Strengths include an extensive evaluation with five runs at temperature zero, the use of an external hold-out (MIDI) distinct from the development set, a comparison to a deterministic baseline, and statistical testing for prompt changes on MIDI. The construction of two benchmark datasets with detailed annotations is a potentially valuable community resource, though the datasets are not currently publicly released. If the conclusions hold across real-world data, the recommendation of a dedicated localization model (YOLOv11) with a lightweight OCR (EasyOCR) and LLM-based analysis is practically actionable. The main open risk is the external validity of the benchmarks, which are entirely synthetic.","major_comments":[{"comment":"The core claim that Setup 1 is the optimal configuration is supported only by evaluations on synthetic overlay data (RadPHI-test, Section 2.4, and MIDI, Section 2.5). The authors explicitly acknowledge in Section 6 that \"our evaluation does not include testing our proposed approaches on real-world data containing PHI.\" Since the ranking between Setup 1 and Setup 3 is close (e.g., case-level recall 0.9923 vs 0.9995 in Tables A1 and A3; case-level precision for patient name 0.9778 vs 1.0000), the choice of Setup 1 is driven largely by latency and token costs as measured on this synthetic distribution. Real burned-in imprints often have different fonts, contrast, warping, and partial occlusion; any of these could reverse the relative merits of EasyOCR versus GPT-4o as the extractor. Please either provide evidence (e.g., a small real-data pilot) that the synthetic distribution is representative, or explicitly restrict the conclusion to \"optimal among the evaluated synthetic benchmarks.\"","section":"§6 (Limitations); §2.4–2.5"},{"comment":"Instance-level evaluation is introduced in Section 4 and Table 3, and instance-level precision/recall are reported in Tables 5 and 7, but the manuscript never defines the matching protocol between predicted bounding boxes and ground-truth boxes. Is an IoU threshold used (and what value)? How are duplicate predictions or one-to-many matches handled? Without this information the instance-level numbers are not reproducible, and the comparison between Setups 1 and 3 at instance level is not fully interpretable. Please specify the matching algorithm precisely.","section":"§4; Table 3"},{"comment":"The MIDI dataset is described as containing both DICOM tag overlays generated by the authors and burn-ins from the original MIDI-B challenge organizers, but it is not stated whether the instance-level ground truth annotations include the organizers' burn-ins or only the authors' overlays. If the latter, then the MIDI evaluation does not actually test on the original MIDI-B burn-ins, which would substantially weaken the claim that MIDI is a \"realistic hold-out dataset.\" Please clarify the annotation scope and, if the burn-ins are included, describe how their ground truth was obtained.","section":"§2.5"},{"comment":"The Discussion states that \"GPT-4o outperforms EasyOCR by having lower word-level and character-level errors,\" but no OCR-only evaluation is presented anywhere in the paper. This claim is load-bearing for the recommendation to use EasyOCR despite its lower OCR accuracy, under the argument that the downstream LLM compensates. Please add the OCR-only evaluation (e.g., character/word error rates on the crop sets) or remove/qualify the claim.","section":"§6 (Discussion, Text extraction)"},{"comment":"There is a discrepancy between the stated number of PHI-containing images in RadPHI-test: Section 2.4 says \"850 images (85%) contain at least one PHI imprint,\" while Table 4 reports 779 images with PHI presence. This affects the denominator of the reported recall values. Please correct the text or the table and verify all derived statistics.","section":"§2.4 vs Table 4"}],"minor_comments":[{"comment":"The identity of \"Worst Setup\" and \"Best Setup\" is only given by color-coding in the table captions, which may not be discernible in grayscale print. Please include the setup numbers explicitly (e.g., \"Setup 1 (YOLOv11 + EasyOCR + GPT-4o)\").","section":"Tables 4 and 5"},{"comment":"The tool name is misspelled as \"Presdio\" in the captions; it should be \"Presidio\".","section":"Tables 4 and 5 captions"},{"comment":"The text states that RadPHI-test comprises \"1,000 images distributed equally across four modalities,\" but the four modalities are not enumerated in the text; please name them (CT, bone scan, X-ray, MRI, as implied by Figure 3).","section":"§2.4"},{"comment":"The datasets are said to be \"available upon request\"; for benchmarking purposes, a public release (e.g., via a repository) would greatly increase the impact and reproducibility of the study.","section":"Data availability statement"},{"comment":"The caption says \"out-of-the-box EasyOCR tries to fit bounding boxes that cover the entire text space,\" but the example shows two separate boxes; please clarify the intended point.","section":"Figure 8 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper is relevant and likely to be of interest to the journal's audience, but the external validity of the central \"optimal setup\" claim is the main risk. The authors should be encouraged to either provide a small real-world validation or substantially temper the claim to the synthetic benchmarks. The instance-level matching protocol and the MIDI annotation scope are fixable with clarifications. The benchmark datasets would be a stronger contribution if released publicly. No concerns about novelty or citation practice."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is the first head-to-head comparison of four ways to assemble a pixel-level PHI detection pipeline: YOLOv11+EasyOCR+GPT-4o, EasyOCR doing its own localization and OCR, GPT-4o reading YOLO crops, and GPT-4o end-to-end, plus a Presidio baseline. Second, the headline result - the fully modular setup wins on accuracy, latency, and token cost - is measured entirely on synthetic or mostly-synthetic data, and the authors say so in the limitations. What is genuinely useful: the benchmark is carefully built. Two datasets with 1,000 and 200 images across several modalities, instance-level annotations, five runs at temperature zero, and a hold-out set (MIDI) that is partially realistic because it includes the challenge organizers' own burn-ins alongside DICOM tag overlays. The finding that swapping EasyOCR for GPT-4o as the OCR stage buys almost no pipeline accuracy at much higher cost is concrete and useful. The end-to-end GPT-4o failure modes (hallucinations, no bounding boxes, pipeline errors) are documented with real examples, and the Presidio comparison grounds the numbers. Soft spots, in order. The instance-level matching protocol is never specified; we are told precision/recall at instance level, but not how predicted boxes are matched to ground truth or what IoU threshold is used. A reproducibility gap. Instance-level evaluation is missing for Setups 2 and 4, so the tightest comparison is only between Setups 1 and 3. The MIDI prompt was modified after seeing hold-out errors, and the reported precision improvement is partly post hoc; the authors disclose this, but it is a caveat. The external-validity worry is real: RadPHI-test is fully synthetic, MIDI is mostly viewer-rendered DICOM overlays, and the ranking could shift on real scanner burn-ins with different fonts, contrast, and occlusion. I do not see circular argument or dishonestly fitted parameters; the free parameters are design choices, not tuned to the conclusion. This paper deserves a serious referee. It is work a practitioner will actually use, and the datasets should be released outright rather than on request. A careful reviewer should ask for the matching protocol, a breakdown of MIDI performance on organizer burn-ins versus overlays, and ideally a small real-PHI test if shared under data-use agreements.","headline":"A solid, practical comparison of PHI-detection pipeline architectures with clear accuracy-cost tradeoffs, but instance-level matching is underspecified and the external-validity base is synthetic.","tokens_in":692,"tokens_out":1412,"would_cite":true,"duration_ms":39375,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The best pixel-level PHI detector splits the job: YOLOv11 for text, EasyOCR for reading, GPT-4o for judgment.","keywords":["protected health information detection","medical image de-identification","optical character recognition","large language models","YOLOv11","GPT-4o","pixel-level PHI","text localization"],"falsifier":"Run the same four setups on a corpus of real clinical images containing genuine burned-in PHI from diverse institutions, scanners, fonts, and overlay artifacts, and compare instance-level recall and precision against the MIDI results. If any setup other than Setup 1 tops the ranking, or if Setup 1's instance-level recall falls far below the 0.9781 reported on MIDI while another setup holds up, the optimality claim is refuted.","tokens_in":19894,"feed_emoji":"🩻","tokens_out":6520,"duration_ms":55884,"temperature":0.7,"pith_summary":"This paper tries to establish that the most reliable way to detect patient-identifying text burned into medical images is to split the job into three specialized stages: a fine-tuned object detector (YOLOv11) finds text regions, an off-the-shelf OCR engine (EasyOCR) reads them, and a large language model (GPT-4o) classifies which strings are protected health information. The authors benchmark this configuration against four alternatives, including letting EasyOCR do detection, letting GPT-4o read image crops, and letting GPT-4o handle the entire task end-to-end. They find the dedicated-module setup achieves near-perfect case-level performance on their synthetic test set and the best balance of precision, recall, latency, and token cost. A sympathetic reader would care because this gives a practical, low-cost recipe for building pixel-level PHI detection today, and because it clarifies when multimodal LLMs are ready to replace traditional OCR pipelines.","feed_headline":"One model per job beats one model for all in PHI detection","feed_subtitle":"A three-module design with dedicated vision and language models outperforms end-to-end LLMs in accuracy, latency, and cost.","key_machinery":"The central mechanism is the three-module decomposition of PHI detection—text localization, text extraction, text analysis—and the assignment of a dedicated model to each stage. The load-bearing identity is the specific orchestration: YOLOv11's single-shot detector supplies tight bounding boxes; EasyOCR converts those crops to strings; GPT-4o, constrained by a structured-output schema and temperature zero, classifies each string against a prompt-defined PHI ontology. The paper's decision rule is that later modules can compensate for earlier errors: a robust language model absorbs minor OCR mistakes, but no amount of language understanding can recover text that was never localized.","core_discovery":"On the paper's own terms, the central discovery is that a modular pipeline—YOLOv11 fine-tuned for text localization, EasyOCR for text extraction, and GPT-4o for text analysis—is the optimal configuration for pixel-level PHI detection. On the RadPHI-test benchmark it reaches case-level precision 1.0000 and recall 0.9995, and on the realistic MIDI hold-out set, after a one-line prompt adaptation that excludes study-related identifiers from the PHI definition, it reaches instance-level precision 0.9721 and recall 0.9781. The authors also show that using GPT-4o for text extraction (Setup 3) barely improves pipeline accuracy over EasyOCR while roughly doubling latency and tripling prompt tokens, and that using GPT-4o end-to-end is feasible but produces more hallucinations and no bounding box coordinates, making it unsuitable for redaction. The finding matters because it identifies which components actually carry the performance and where LLM flexibility provides the most value.","pith_inferences":["If the modular result holds beyond radiology, the same YOLO+OCR+LLM skeleton could detect PHI in clinical documents, pathology images, or scanned forms, with only the prompt ontology changed.","The near-zero false-negative rate at case level suggests the pipeline could serve as a high-recall pre-filter in de-identification workflows, with flagged images sent to human review and unflagged images released automatically.","The token-cost gap between Setup 1 and Setup 3 implies that as small on-premise LLMs improve, the analysis module could be swapped for a local model, preserving accuracy while removing the privacy cost of sending text to a hosted API.","A direct testable extension would be evaluating Setup 1 on non-English imprints; the paper's observed implicit spelling correction is English-centric and may not transfer to other scripts."],"forward_implications":["A modular, mostly off-the-shelf pipeline can already deliver near-perfect case-level PHI detection on synthetic radiology benchmarks, suggesting immediate practical deployment for screening.","Because the LLM tolerates minor OCR errors, teams can use lightweight OCR instead of paying for high-accuracy commercial engines without sacrificing detection quality.","PHI policy changes—for example, deciding that study or image identifiers are not PHI—can be implemented by editing a natural-language prompt, not by re-training or re-writing rules.","End-to-end multimodal LLMs are not yet a drop-in replacement for dedicated vision models, since they lack coordinate outputs and exhibit higher hallucination and pipeline-error rates.","The chosen setup generalizes to unseen modalities (PET, mammography, ultrasound) without retraining the localization model, indicating the architecture is modality-agnostic."],"supporting_citations":[{"why":"Supplies the YOLOv11 object detector that is fine-tuned on RadPHI-train and serves as the text localization module in Setups 1 and 3.","marker":"[27]"},{"why":"Provides EasyOCR, the out-of-the-box OCR engine used for text extraction in Setups 1 and 2.","marker":"[29]"},{"why":"Provides GPT-4o, the language model used for text analysis in all setups and for extraction/end-to-end roles in Setups 3 and 4.","marker":"[20]"},{"why":"Describes the CRAFT text detection model that underlies EasyOCR's built-in localization, which is what Setup 2 relies on instead of a dedicated detector.","marker":"[30]"},{"why":"Source of the MIDI hold-out dataset's DICOM images and synthetic PHI burn-ins, used to validate the optimal setup.","marker":"[25]"},{"why":"Defines the HIPAA PHI categories that the paper uses to build its imprint ontology and evaluation labels.","marker":"[1]"},{"why":"Provides the evidence that LLMs are effective for biomedical NER, motivating GPT-4o as the text analysis module.","marker":"[17]"},{"why":"Supplies Presidio, the deterministic baseline tool the paper compares against to contextualize LLM-based performance.","marker":"[16]"}],"fun_headline_variants":["Modular AI pipeline wins for medical image PHI detection","Dedicated models beat all-in-one for medical PHI detection","Three-module design boosts PHI detection in medical images","Split tasks, not models: best PHI detection in medical images","Why modular AI beats end-to-end LLMs for PHI spotting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that Setup 1 is optimal rests on the assumption that synthetic imprints overlaid on public radiology images faithfully represent real burned-in clinical PHI, since the ranking of setups is measured only on such synthetic data and the paper states it never tested genuine patient data.","fun_headline_variants_meta":{"raw":{"variants":["Modular AI pipeline wins for medical image PHI detection","Dedicated models beat all-in-one for medical PHI detection","Three-module design boosts PHI detection in medical images","Split tasks, not models: best PHI detection in medical images","Why modular AI beats end-to-end LLMs for PHI spotting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001002,"raw_usage":{"total_tokens":4249,"prompt_tokens":963,"completion_tokens":3286,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":3199}},"tokens_in":579,"tokens_out":3286,"duration_ms":22220,"temperature":1.0,"reasoning_tokens":3199,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:53:52.330541+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same four setups on a corpus of real clinical images containing genuine burned-in PHI from diverse institutions, scanners, fonts, and overlay artifacts, and compare instance-level recall and precision against the MIDI results. If any setup other than Setup 1 tops the ranking, or if Setup 1's instance-level recall falls far below the 0.9781 reported on MIDI while another setup holds up, the optimality claim is refuted.","supporting_citations":[{"cited_title":"Available at https://docs.ultralytics.com/models/yolo11","cited_arxiv_id":null,"evidence_quote":"Supplies the YOLOv11 object detector that is fine-tuned on RadPHI-train and serves as the text localization module in Setups 1 and 3."},{"cited_title":"Available at https://github.com/JaidedAI/EasyOCR","cited_arxiv_id":null,"evidence_quote":"Provides EasyOCR, the out-of-the-box OCR engine used for text extraction in Setups 1 and 2."},{"cited_title":"Available at https://openai.com/index/hello-gpt-4o/","cited_arxiv_id":null,"evidence_quote":"Provides GPT-4o, the language model used for text analysis in all setups and for extraction/end-to-end roles in Setups 3 and 4."},{"cited_title":"Available at https://www.synapse.org/Synapse:syn53065760","cited_arxiv_id":null,"evidence_quote":"Source of the MIDI hold-out dataset's DICOM images and synthetic PHI burn-ins, used to validate the optimal setup."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the HIPAA PHI categories that the paper uses to build its imprint ontology and evaluation labels."},{"cited_title":"Presidio, version 2.2.358","cited_arxiv_id":null,"evidence_quote":"Supplies Presidio, the deterministic baseline tool the paper compares against to contextualize LLM-based performance."}],"review_version":1}