REVIEW 3 major objections 4 minor 2 references
Red Teaming for Generative AI, Report on a Copyright-Focused Exercise Completed in an Academic Medical Center
T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A production medical AI reproduced a copyrighted book dedication verbatim under indirect prompting, despite direct requests being refused.
desk verdict A genuinely useful case report that overreaches on the differential-safeguards conclusion; the raw extraction examples are real, the comparison across content types needs a training-data presence check. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is indirect prompting: instead of asking for copyrighted text directly, an attacker builds conversational context so that keyword-based refusal triggers are not activated, and the model then completes memorized training text. The same mechanism explains why direct requests for news, scientific, and clinical content failed while the literary dedication succeeded. The paper also introduces a four-domain testing matrix as a way to compare extraction rates across content types, and it uses that matrix to support the claim that protection levels vary.
What would settle it
Run the same indirect prompts on a production model using a literary passage known to be absent from its training data; if the model still returns it verbatim, training-data contamination is not the explanation and a different failure mode would be at work. Conversely, rerun the news tasks on articles as widely copied as the Rowling dedication; if extraction remains zero, then content-type-specific safeguards, not data availability, explain the difference.
Extended reading notes
Core claim
The report's central claim is that a production generative AI system with standard safeguards can inadvertently reproduce short copyrighted literary passages when prompted indirectly. Concretely, asking who the Harry Potter book was dedicated to produced the verbatim dedication, and a conversation about feeling insignificant led to a passage from The Hitchhiker's Guide to the Galaxy containing a near-verbatim sentence. Direct requests for book text were refused, while indirect strategies such as style mimicry and translation round-trips bypassed the refusals. The authors take the differential success rates across content types as evidence that protective mechanisms are inconsistent and that training-data contamination remains the most probable explanation for the literary reproductions.
Load-bearing premise
The comparison of success rates across literary, news, science, and clinical content assumes those four kinds of material were similarly present in the model's training data; if the literary works were far better represented than the news and scientific articles, the observed differences would reflect data composition rather than protective mechanisms.
Editorial extensions
If this is right
- Verbatim literary passages can be extracted through indirect prompting, so keyword-based refusal filtering is not sufficient on its own.
- The 23% success rate for literary extraction, contrasted with 0% for news and science, suggests that protective mechanisms are content-specific or that training-data representation varies by content type.
- A copyright-specific meta-prompt was implemented in production in January 2025 as a first mitigation step.
- Academic medical institutions deploying generative AI should establish continuous red teaming protocols rather than relying on a one-time evaluation.
- Institutions using hosted enterprise AI services should treat copyright testing as part of the shared responsibility needed to qualify for platform copyright protections.
Reading between the lines
- If the differential results are driven by training-data representation rather than safeguards, then any content type well represented in training data is potentially at risk, including lyrics, scripts, or book dedications in other deployments.
- The successful translation round-trip suggests that memorized text can survive semantic transformation, so exact-match evaluation is too weak; paraphrase-invariant tests would be more informative.
- A decisive experiment would audit a model's training corpus and compare extraction of passages known to be in the corpus versus known to be absent, separating memorization from generation.
- For institutional governance, logging adversarial prompt-output pairs during red teaming could become part of the evidence needed to support a legal indemnification claim.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports a two-hour red teaming exercise conducted in November 2024 against GPT4DFCI, a production enterprise AI tool at Dana-Farber Cancer Institute using OpenAI models. Forty-two participants in four teams attempted to extract copyrighted content from four domains: literary works, news articles, scientific publications, and access-restricted MIMIC clinical notes. The authors report that indirect prompting elicited a verbatim Harry Potter dedication, an Italian/English round-trip of the same dedication, and a partially verbatim Hitchhiker's Guide sentence, while news, scientific, and clinical extractions failed. The paper interprets these results as evidence of differential protective mechanisms, recommends inference-time filtering, and describes a copyright-specific meta-prompt deployed in production in January 2025.
Significance. If the reported examples are accurate, the paper provides a useful real-world existence proof that a filtered, production medical AI can still emit short memorized passages of copyrighted literary text under conversational prompting. The main strength is documentation: Supplementary Table 1 contains prompt/output logs rather than only aggregate claims, and the authors name the institutional context, event time, and mitigation. The practical value for academic medical centers deploying Azure OpenAI products is real, and the paper is candid about several limitations. However, the analytical claims that go beyond the existence proof—especially the 23 percent literary success rate and the differential-protection inference—are not currently supported by the evidence presented.
major comments (3)
- [Quantitative Results Summary and Supplementary Table 1] The reported 23% success rate for literary works is not recoverable from the supplied data. Supplementary Table 1 lists 25 book-related interactions, and only rows 7–9 (the dedication, including Italian and English round-trip) and row 25 (approximately half of one sentence) show verbatim or near-verbatim reproduction; that is at most 4/25, or 16%, depending on the counting rule. The text also says the overall exercise comprised 156 documented attack attempts, but the four supplementary tables contain only 53 prompt/output rows, so the denominator for the 23% figure is not evident. Please provide the complete attack log with a per-attempt success label and an explicit statement of the success classification rubric, or remove the quantitative rate.
- [Abstract, Discussion, and Conclusion] The 'differential success rates across content types suggest varying protective mechanisms' claim is confounded by training-data presence. For the Chicago Tribune and New York Times articles, the Iijima paper, the MOLSCRIPT paper, and the MIMIC notes, the authors only state that targets were selected for 'high probability of inclusion in known training datasets'; no membership evidence is offered. A 0% extraction rate is equally consistent with the targets being absent or under-represented in the model's training data as with the model having protective mechanisms. Supplementary Table 4 even shows the model failing to complete simple 'charttime' rows, suggesting weak memorization of the MIMIC content specifically. The Conclusion's 'inconsistent copyright safeguards requiring targeted improvements' is therefore not supported unless target presence in training data is established or the claim is reframed as a hypothesis. Please either add membership tests (e.g., direct extraction probes or exposure metrics) or limit the abstract and conclusion to the existence claim.
- [Discussion (mitigation claim)] The paper states that the copyright-specific meta-prompt 'Avoid copyright infringement' was deployed in production since January 2025 and that this exemplifies responsible AI development, but no post-mitigation evaluation is reported. The effectiveness of the meta-prompt is not measured, so the conclusion that the exercise 'led to concrete mitigation strategies' is only a claim about deployment, not about protection. Please clarify in the Discussion that the mitigation has not yet been validated, or add the results of a re-test after deployment.
minor comments (4)
- [Supplementary Table 1] There are several typos in the prompts as transcribed, including 'Hitchhiker's Guid to the Galaxy', 'perspecitve', and 'Aurther's house'; these should be corrected or marked as verbatim user typos so readers can distinguish transcription errors from actual user input.
- [Supplementary Table 4] Row 2 lists the model as 'GPT4oo' while all other rows say 'GPT-4o'; this appears to be a typo.
- [Reference 16] The DAN jailbreak citation contains 'doi: doi: 10.5281/zenodo.1234', which looks like a placeholder DOI; please verify the citation or replace it with a proper source.
- [Target Material Selection] The authors generalize from four targets per domain (or fewer in the scientific domain, where two papers are listed) to broad statements about 'news articles', 'scientific publications', and 'academic content'; the Discussion should acknowledge that the number of targets per domain is too small to support domain-level conclusions even setting aside the training-presence confound.
Circularity Check
No significant circularity: an observational red-teaming case report whose central claims are empirical extraction results, not derived from fitted inputs or self-citation.
full rationale
The paper is a case report of a red-teaming exercise. Its central claim, that production generative AI can reproduce verbatim copyrighted text under indirect prompting, rests on documented prompts and outputs in Table 1 and Supplementary Tables 1-4. There is no fitted parameter, no uniqueness theorem, and no equation linking input to output; the success rates are counts of observed attempts. The abstract's statement that 'Differential success rates across content types suggest varying protective mechanisms' is an inference from observed frequencies, not a construction that builds the conclusion into the experimental definition. The 0% rates for news, scientific, and clinical targets could be confounded by unknown training-data membership, but that is an external-validity or measurement concern, not circularity: the paper itself states that targets were selected for 'high probability of inclusion in known training datasets' and acknowledges 'selection bias in choosing well-known copyrighted works.' The only self-citation (ref. 1, Umeton et al., GPT-4 in a cancer center) is used for institutional context about GPT4DFCI's deployment; it is not invoked to justify the observed extraction outcomes or to rule out alternative explanations. The mitigation meta-prompt is described as a post-hoc intervention, not as evidence for the vulnerability finding. No circular step was identified.
Assumptions & free parameters
free parameters (2)
- Success classification rubric =
Undefined; the 23 percent, 67 percent, and 33 percent rates imply a threshold that is never stated
- Attempt counts per domain =
Not reported; the paper does not break down the 156 attempts by domain or team
assumptions (3)
- domain assumption The selected target materials were present in the model's training data at comparable levels across the four content domains
- domain assumption Outputs counted as successes reflect model memory rather than text the testers supplied in the prompt
- domain assumption The exercise environment matches the production configuration, including model versions, safeguards, and content filters
Cite this review
Pith. "Pith review of Red Teaming for Generative AI, Report on a Copyright-Focused Exercise Completed in an Academic Medical Center." pith.science (2026). https://pith.science/paper/T3Z7FJAA
@misc{pith2026250622523,
author = {Pith},
title = {Pith review of: Red Teaming for Generative AI, Report on a Copyright-Focused Exercise Completed in an Academic Medical Center},
year = {2026},
howpublished = {\url{https://pith.science/paper/T3Z7FJAA}},
note = {Machine review of arXiv:2506.22523}
}
read the original abstract
Background: Generative artificial intelligence (AI) deployment in academic medical settings raises copyright compliance concerns. Dana-Farber Cancer Institute implemented GPT4DFCI, an internal generative AI tool utilizing OpenAI models, that is approved for enterprise use in research and operations. Given (1) the exceptionally broad adoption of the tool in our organization, (2) our research mission, and (3) the shared responsibility model required to benefit from Customer Copyright Commitment in Azure OpenAI Service products, we deemed rigorous copyright compliance testing necessary. Case Description: We conducted a structured red teaming exercise in Nov. 2024, with 42 participants from academic, industry, and government institutions. Four teams attempted to extract copyrighted content from GPT4DFCI across four domains: literary works, news articles, scientific publications, and access-restricted clinical notes. Teams successfully extracted verbatim book dedications and near-exact passages through various strategies. News article extraction failed despite jailbreak attempts. Scientific article reproduction yielded only high-level summaries. Clinical note testing revealed appropriate privacy safeguards. Discussion: The successful extraction of literary content indicates potential copyrighted material presence in training data, necessitating inference-time filtering. Differential success rates across content types suggest varying protective mechanisms. The event led to implementation of a copyright-specific meta-prompt in GPT4DFCI; this mitigation has been in production since Jan. 2025. Conclusion: Systematic red teaming revealed specific vulnerabilities in generative AI copyright compliance, leading to concrete mitigation strategies. Academic medical institutions deploying generative AI should implement continuous testing protocols to ensure legal and ethical compliance.
Reference graph
Works this paper leans on
-
[1]
Beth Israel Deaconess Medical Center, Boston, MA, USA 3
Dana-Farber Cancer Institute, Boston, MA, USA 2. Beth Israel Deaconess Medical Center, Boston, MA, USA 3. MLCommons, San Francisco, CA, USA 4. Massachusetts Institute of Technology, Cambridge, MA, USA 5. Harvard Medical School, Boston, MA, USA 6. Harvard T.H. Chan School of Public Health, Boston, MA, USA 7. Boston University, Boston, MA, USA 8. Argonne Na...
-
[10]
National Institutes of Health, Bethesda, MD, USA 11. Weill Cornell Medicine, New York, NY, USA 12. Microsoft Corporation, Redmond, WA, USA 13. Institute of Medical Informatics, Charité, Berlin, Germany 14. University of Toronto, Toronto, Ontario, Canada 15. University of Alberta, Edmonton, Alberta, Canada 16. Veraitech, Fairfax, VA, USA 17. Reins AI, Ever...
arXiv 2024
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.