Reducing malicious use of synthetic media research: Considerations and potential release practices for machine learning
Pith reviewed 2026-05-24 15:47 UTC · model grok-4.3
The pith
The machine learning community could reduce risks of synthetic media misuse by working with experts, building shared norms, and creating support institutions for release decisions.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The machine learning community might benefit from working with subject matter experts to increase understanding of the risk landscape and possible mitigation strategies; building a community and norms around understanding the impacts of ML research, e.g. through regular workshops at major conferences; and establishing institutions and systems to support release practices that would otherwise be onerous and error-prone.
What carries the argument
A set of considerations, analogies from other fields, and recommended community actions for evaluating and managing release of synthetic media research.
If this is right
- Collaboration with subject matter experts produces clearer maps of how synthetic media research can be misused.
- Regular workshops at major conferences help establish shared expectations about research impacts.
- Dedicated institutions lower the practical cost of making careful release decisions.
- These steps together make it easier for researchers to choose release options that account for downstream risks.
Where Pith is reading between the lines
- The same pattern of expert input and institutional support could apply to other dual-use machine learning areas such as autonomous systems or surveillance tools.
- Adoption would likely require changes in conference culture and funding incentives to make participation routine.
- Without pilot testing, it remains open whether the proposed institutions would actually change release behavior at scale.
Load-bearing premise
That structured community processes and institutions for release decisions will meaningfully reduce harms from synthetic media misuse compared to current practices.
What would settle it
A measurable drop in documented cases of malicious synthetic media use after the machine learning community implements expert consultations, regular workshops, and dedicated support institutions for release decisions.
read the original abstract
The aim of this paper is to facilitate nuanced discussion around research norms and practices to mitigate the harmful impacts of advances in machine learning (ML). We focus particularly on the use of ML to create "synthetic media" (e.g. to generate or manipulate audio, video, images, and text), and the question of what publication and release processes around such research might look like, though many of the considerations discussed will apply to ML research more broadly. We are not arguing for any specific approach on when or how research should be distributed, but instead try to lay out some useful tools, analogies, and options for thinking about these issues. We begin with some background on the idea that ML research might be misused in harmful ways, and why advances in synthetic media, in particular, are raising concerns. We then outline in more detail some of the different paths to harm from ML research, before reviewing research risk mitigation strategies in other fields and identifying components that seem most worth emulating in the ML and synthetic media research communities. Next, we outline some important dimensions of disagreement on these issues which risk polarizing conversations. Finally, we conclude with recommendations, suggesting that the machine learning community might benefit from: working with subject matter experts to increase understanding of the risk landscape and possible mitigation strategies; building a community and norms around understanding the impacts of ML research, e.g. through regular workshops at major conferences; and establishing institutions and systems to support release practices that would otherwise be onerous and error-prone.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper argues that the machine learning community could benefit from developing norms and practices to mitigate risks of malicious use of synthetic media research. It reviews background on misuse risks, outlines paths to harm, surveys mitigation strategies from other fields, identifies points of disagreement, and concludes with three high-level recommendations: collaborating with subject-matter experts, building community norms via workshops at major conferences, and establishing institutions to support release decisions. The authors explicitly do not advocate any particular release policy.
Significance. If the suggested community processes and institutions could be shown to reduce misuse harms relative to current ad-hoc practices, the paper would provide a useful starting point for structured discussion in the ML community. However, the manuscript supplies no outcome data, counterfactual analysis, or assessment of transferability from the reviewed analogies, leaving the central premise unsupported.
major comments (2)
- [Conclusion] Conclusion (final paragraph): The claim that the community 'might benefit from' establishing institutions and systems to support release practices rests on the unexamined assumption that such structures will reduce harms compared with existing practices. The paper reviews analogies from other fields but provides no analysis of whether those components transferred successfully or would transfer to ML's open-publication culture and incentive structure.
- [Review of mitigation strategies] Section reviewing research risk mitigation strategies in other fields: The text identifies 'components that seem most worth emulating' but offers no evaluation of effectiveness, failure modes, or boundary conditions under which the reviewed practices succeeded or failed in their original domains.
minor comments (1)
- [Abstract and Conclusion] The abstract and conclusion use nearly identical phrasing for the three recommendations; consolidating or distinguishing them would improve readability.
Simulated Author's Rebuttal
We thank the referee for their review. Our manuscript is framed as an exploratory discussion piece to surface considerations, analogies, and options for the ML community, without empirical claims about harm reduction or transferability of practices. We address the major comments below.
read point-by-point responses
-
Referee: [Conclusion] Conclusion (final paragraph): The claim that the community 'might benefit from' establishing institutions and systems to support release practices rests on the unexamined assumption that such structures will reduce harms compared with existing practices. The paper reviews analogies from other fields but provides no analysis of whether those components transferred successfully or would transfer to ML's open-publication culture and incentive structure.
Authors: The conclusion deliberately employs the tentative language 'might benefit from' to indicate that these are suggestions for community exploration rather than assertions of harm reduction. The paper does not claim or assume that the reviewed components will transfer successfully; it presents them as starting points for discussion. We can revise the final paragraph to state more explicitly that any potential benefits remain hypothetical and would require further community assessment and evidence. revision: partial
-
Referee: [Review of mitigation strategies] Section reviewing research risk mitigation strategies in other fields: The text identifies 'components that seem most worth emulating' but offers no evaluation of effectiveness, failure modes, or boundary conditions under which the reviewed practices succeeded or failed in their original domains.
Authors: The section provides a high-level survey of strategies from other fields to identify potentially relevant components, consistent with the paper's aim of facilitating discussion rather than conducting a systematic evaluation. A full assessment of effectiveness, failure modes, and boundary conditions lies outside the manuscript's scope and would require different expertise and data. No changes are planned for this section. revision: no
Circularity Check
No circularity: normative discussion paper with no derivations, equations, or self-referential predictions
full rationale
The paper contains no equations, parameters, fitted inputs, or derivation chains of any kind. It reviews background on misuse risks, analogies from other fields, dimensions of disagreement, and offers recommendations framed explicitly as 'tools, analogies, and options for thinking about these issues' rather than derived results. No self-citation is used to establish a uniqueness theorem or load-bearing premise that reduces to itself. The central suggestions (expert collaboration, workshops, institutions) are presented as community benefits to consider, without any modeling or claim that they are proven by the paper's own structure. This is a standard non-circular discussion piece.
Axiom & Free-Parameter Ledger
Reference graph
Works this paper leans on
-
[1]
Jess Whittlestone, Rune Nyrup, Anna Alexandrova, Kanta Dihal, and Stephen Cave. Ethical and Societal Impli- cations of Algorithms, Data, and Artificial Intelligence: a Roadmap for Research. London: Nuffield F oundation, 2019
work page 2019
-
[2]
C., Steinhardt, J., Flynn, C., h\' E igeartaigh, S
Miles Brundage, Shahar Avin, Jack Clark, Helen Toner, Pe ter Eckersley, Ben Garfinkel, Allan Dafoe, Paul Scharre, Thomas Zeitzoff, Bobby Filar, et al. The Malicious Use of Artificial Intelligence: Forecasting, Preven- tion, and Mitigation. arXiv preprint arXiv:1802.07228 , 2018
- [3]
-
[4]
Is the political aide viral sex video confession real or a Deepfake? Malay Mail, 2019
Nic Ker. Is the political aide viral sex video confession real or a Deepfake? Malay Mail, 2019
work page 2019
-
[5]
A Style-Based G enerator Architecture for Generative Adversarial Networks
Tero Karras, Samuli Laine, and Timo Aila. A Style-Based G enerator Architecture for Generative Adversarial Networks. In Proceedings of the IEEE Conference on Computer Vision and Pa ttern Recognition , pages 4401– 4410, 2019
work page 2019
-
[6]
Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei A Efros. Everybody Dance Now. arXiv preprint arXiv:1808.07371, 2018
-
[7]
Face2face: Real- time face capture and reenactment of rgb videos
Justus Thies, Michael Zollhofer, Marc Stamminger, Chri stian Theobalt, and Matthias Nießner. Face2face: Real- time face capture and reenactment of rgb videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2387–2395, 2016
work page 2016
-
[8]
Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron
RJ Skerry-Ryan, Eric Battenberg, Ying Xiao, Y uxuan Wang , Daisy Stanton, Joel Shor, Ron J Weiss, Rob Clark, and Rif A Saurous. Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron. arXiv preprint arXiv:1803.09047, 2018. 10 DRAFT - J ULY 2019
work page internal anchor Pith review Pith/arXiv arXiv 2018
-
[9]
Language Models are Unsupervised Multitask Learners
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners. OpenAI Blog, 1(8), 2019
work page 2019
-
[10]
Drew Harwell. Fake-porn videos are being weaponized to harass and humiliate women: ‘Everybody is a potential target’. W ashington Post, December 2018
work page 2018
-
[11]
Fake voices ‘help cyber-crooks steal cash’, J uly 2019
BBC News. Fake voices ‘help cyber-crooks steal cash’, J uly 2019
work page 2019
-
[12]
Experts: Spy used AI-generated face to connect with targets
Raphael Satter. Experts: Spy used AI-generated face to connect with targets. AP News, June 2019
work page 2019
-
[13]
Claire Leibowicz, Steven Adler, and Peter Eckersley. W hen Is it Appropriate to Publish High-Stakes AI Re- search? Partnership on AI blog post, April 2019
work page 2019
-
[14]
Information Hazards: A Typology of Poten tial Harms from Knowledge
Nick Bostrom. Information Hazards: A Typology of Poten tial Harms from Knowledge. Review of Contemporary Philosophy, 10:44–79, 2011
work page 2011
- [15]
-
[16]
World Health Organization. Laboratory Biosafety Manual . Geneva: World Health Organization, 2004
work page 2004
-
[17]
The Purpose, Compos ition, and Function of an Institutional Review Board: Balancing Priorities
Kyle B Enfield and Jonathon D Truwit. The Purpose, Compos ition, and Function of an Institutional Review Board: Balancing Priorities. Respiratory care, 53(10):1330–1336, 2008
work page 2008
-
[18]
Perspectives on Issues in AI Governance, 2019
Google. Perspectives on Issues in AI Governance, 2019
work page 2019
-
[19]
Brent Hecht, L Wilcox, JP Bigham, J Schöning, E Hoque, J E rnst, Y Bisk, L De Russis, L Y arosh, B Anjum, et al. It’s Time to Do Something: Mitigating the Negative Imp acts of Computing Through a Change to the Peer Review Process. ACM Future of Computing Blog , 2018
work page 2018
-
[20]
Why We Released Grover, July 2019
Rowan Zellers. Why We Released Grover, July 2019. W e are grateful to the many researchers across academia, industry, and civil society who provided invaluable feedback. This is meant to be just the start of a conversation, and we are likely to update this document as we learn more. If you have thoughts or feedback, or would like to otherwise contri bute ...
work page 2019
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.