REVIEW 4 major objections 7 minor 16 references
Position: It's Time to Act on the Risk of Efficient Personalized Text Generation
T0 review · 4 major / 7 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read Cheap, locally run fine-tuning of open-weight LLMs already lets attackers write convincingly in a specific person's voice, and this 'text deepfake' risk is largely absent from AI safety discourse.
desk verdict A legitimate position paper on a real and underappreciated risk; the empirical legs are thin and partly self-cited, but the argument doesn't depend on them, and it deserves a serious referee. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is the personalization pipeline: parameter-efficient fine-tuning (low-rank adaptation) of an open-weight base model such as Llama-3-8B, guided by instruction back-translation, in which an LLM writes synthetic prompts for which a person's genuine texts are the desired answers. This pipeline converts a small corpus of someone's writing into a style-matched text generator that runs on consumer hardware, with inference at about 16 tokens per second on a laptop CPU. The MAUVE metric, measuring distributional distance between corpora, is the evaluation device that makes the style match visible, while the human-acquaintance studies and the Copyleaks detector tests are the evidence that the match holds up in practice.
What would settle it
A preregistered study across diverse authors and genres (chat logs, essays, social media) in which acquaintances identify personalized-model outputs at chance rates, or a commercial detector flags them at rates comparable to base-model outputs, would undercut the paper's claim that personalized imitation is a practical, general threat.
Extended reading notes
Core claim
Its central claim is that efficient personalized text generation is already a working tool for impersonation: an individual's style can be captured by fine-tuning a small open-weight LLM on their own writing, and the resulting model produces text that people who know the author cannot reliably distinguish from the genuine article. The paper supports this with the Panza study's human evaluations, in which about 68% of emails from a model fine-tuned on 75 emails were judged credible by an acquaintance, versus 76% for genuine human emails and 36% for the base model, and with the authors' own Copyleaks test, in which personalized outputs scored 0% AI-generated while the same prompts through the un-fine-tuned model scored 100%. From this, the paper argues that textual impersonation differs fundamentally from visual and audio deepfakes: the medium is low-bandwidth, training and inference can be done entirely locally away from centralized auditing, watermarking is easy to evade, and style imitation can be combined with other modalities. It also documents that current legislation and influential policy reports define deepfakes around image, video, and audio, excluding written style, and that four popular LLM interfaces, with one partial exception, supplied useful harmful drafts for all four malicious prompts tested.
Load-bearing premise
The threat assessment depends on the assumption that a small sample of a person's writing—roughly 75 emails from one email domain, per the reproduced study—is enough to train a model that acquaintances and detectors accept as that person.
Editorial extensions
If this is right
- Spear-phishing can be upgraded with a trusted sender's style, making fraudulent requests dramatically more persuasive while costing only consumer-grade compute.
- Statistical AI-text detectors, the dominant defense today, fail on fine-tuned personalized output, so detection alone cannot be the safety net.
- Fine-tuning can break a base model's safety alignment, potentially giving attackers access to functionality that was blocked at release.
- Legal definitions of deepfakes in the EU AI Act and US state laws omit written style, so victims of text impersonation have limited legal recourse.
- Model distributors could propagate access controls and watermark individual weight copies to give forensic accountability for downstream misuse.
Reading between the lines
- Beyond the paper: the same pipeline should transfer from email to chat logs, social-media posts, and forum writing, which would enlarge the vulnerable population well beyond the ENRON-based study.
- Beyond the paper: the MAUVE-style distributional distance used to prove imitation could be inverted into a defense, making stylometric authorship verification a tool for detecting machine-written impersonation.
- Beyond the paper: if written impersonation becomes routine, message-signing protocols and style-based provenance may become standard trust infrastructure for personal correspondence.
- Beyond the paper: the 75-email threshold hints at a measurable scaling law; a corpus-size versus identification-accuracy curve across genres could predict when any given individual becomes practically imitable.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This position paper argues that efficient fine-tuning of open-weight language models makes personalized text generation—credible imitation of a specific person's writing style—practically feasible, and that this creates novel safety risks distinct from image/audio/video deepfakes. The paper presents illustrative outputs from fine-tuned models on ENRON emails (Table 1), reproduces human-study results from the same group's earlier Panza work (Figure 1), reports a two-model test suggesting evasion of a commercial AI-text detector (Section 4), probes five chatbots with four malicious prompts (Table 2), and reviews legal and policy gaps (Section 5). It concludes with research, model-governance, and policy recommendations (Section 6) and replies to anticipated objections (Section 7).
Significance. If the central feasibility claim holds, the paper identifies a genuinely underexplored risk: text impersonation that is cheap, local, hard to audit, and complementary to other deepfake modalities. The paper's strengths include an original (though small) prompt probe, a concrete detector-evasion test, a specific legal-analysis contribution (EU AI Act, FTC rule, California statutes), and an explicit acknowledgment of underlying evidence gaps. The authors are appropriately self-critical about missing datasets and measurement difficulties. However, the load-bearing empirical evidence for 'credible imitation from ~75 emails' and 'evasion of AI-text detectors' is thin, partly self-cited, and not yet independently replicated; the urgency claims therefore currently outrun the evidence base. The paper is a useful position piece that would benefit from either stronger evidence or more carefully qualified claims.
major comments (4)
- [Section 2 / Figure 1] The central feasibility claim—that as few as 75 emails suffice for a model whose output is mistaken for the author by acquaintances—rests entirely on Figure 1, a reproduction of Nicolicioiu et al. (2025), co-authored by two of the present authors and the senior author. The figure reports no confidence intervals, no per-participant variability, and no independent replication or cross-domain validation. Since this is the load-bearing empirical premise of the paper, the manuscript should either add independent experiments with statistical uncertainty and significance tests (ideally outside the ENRON email domain), or explicitly reframe the claim as suggestive evidence from a single corpus rather than an established feasibility result.
- [Section 4] The detector-evasion demonstration uses a single online tool (Copyleaks), two fine-tuned models (Jeff and Kay), and the handful of outputs in Table 1, with no genuine-human-text false-positive control, no repeated sampling, and no distribution of detector scores. The claim that personalized fine-tuning 'evades AI text detection' is therefore not supported at the level stated. I recommend either a systematic evaluation—multiple detectors, several users, matched human and base-model controls, reported error rates and score distributions—or a strictly hedged wording such as 'initial evidence that one commercial detector failed to flag these outputs.'
- [Section 6] The authors concede in Section 6 that no datasets link the same author across modalities and that it is 'difficult to estimate the amount and type of text necessary to build a compelling personalized model.' This concession is in tension with the abstract and introduction, which describe practical feasibility of impersonating specific individuals based on small amounts of text as an established premise. The paper should calibrate the scope of its claims to this acknowledged uncertainty, for example by distinguishing 'possible in one studied corpus' from 'prevalent across populations,' and by using that distinction in the policy recommendations.
- [Section 5 / Table 2] The prompt probe in Table 2 covers four prompts and five models and is presented as evidence that 'safeguards against unsafe messages are largely missing in current models.' The authors state that the prompts are not cherry-picked, but the sample is too small and insufficiently systematic to support the general claim: there is no taxonomy of attack types, no human rating of whether the responses are harmful, no variation of phrasing, and no discussion of how representative the four scenarios are. I recommend either expanding the probe or explicitly limiting the conclusion to the four tested scenarios.
minor comments (7)
- [Section 2] The sentence 'Personalized text generation has also attracted industrial applications. across modalities.' contains a punctuation/capitalization error; 'applications' should be followed by a comma or the fragment should be merged into the previous sentence.
- [Section 3] In the scholastic-dishonesty paragraph, 'whether a text is is AI-generated' contains a duplicated 'is'.
- [Section 3] The text contains a typo: 'Howerver' should be 'However'.
- [Section 5] The phrase 'the International AI Safety Report ... who in Section 2.1.1' should use 'which' instead of 'who' for a report, or be rephrased to avoid the grammatical mismatch.
- [Section 7] In the cryptographic-authentication paragraph, 'adoption of has been relatively slow' is missing a word; it should read 'adoption of these methods has been relatively slow'.
- [Table 2 / Appendix] The main text refers to 'ChatGPT-4o' and 'AI's Claude-3.5 Haiku,' while the appendix labels the same responses as 'GPT-4-Turbo' and 'Claude 3.5 Sonnet'; the model names should be consistent and, where possible, include version and access date information.
- [References] The Copyleaks reference URL is missing the protocol slashes: 'https:www.copyleaks.com' should be 'https://www.copyleaks.com'.
Circularity Check
Central feasibility premise rests on a self-cited Panza human study; no definitional circularity, but load-bearing self-citation.
-
self citation load bearing
[Section 2, Figure 1 and Table 1 caption]
"...conducting a range of human studies, demonstrating that the Panza-generated emails can be attributed to the individuals the models were finetuned on, and that the AI-generated emails are not easily distinguished from genuine ones by the authors' acquaintances, even when the models are finetuned on small samples of the data (75 emails). To illustrate this point, we produced a sample of harmful emails using Llama3-8B-Instruct finetuned using the authors' recipe on two individuals from the ENRON (Cohen, 2015) dataset in Table 1 and we reproduce the authors' data in Figure 1."
The paper's central empirical premise—that personalized LLMs can credibly imitate a person and fool acquaintances—is drawn directly from Nicolicioiu et al. (2025), whose author list includes the present paper's first-two authors and senior author. Figure 1 is explicitly labeled 'Results from Nicolicioiu et al. (2025)', so the key human-evaluation evidence is a restatement of the authors' own prior study rather than an independent replication. The paper provides no independent human study, confidence intervals, or cross-domain validation.
full rationale
This is a position paper rather than a derivation, so most circularity patterns (self-definitional fits, renamed known results, uniqueness theorems) do not apply. The clearest issue is the load-bearing self-citation: the claim that personalized text generation is already practically capable of credible impersonation is supported almost entirely by Nicolicioiu et al. (2025), a study co-authored by two of the current authors and their senior author. The paper even says 'we reproduce the authors' data in Figure 1,' confirming that the key human-study numbers are not new evidence. That is a significant self-citation dependency, but not a definitional circularity, because Panza is a separate empirical study and the current paper also contributes original prompt-based safety probes, a detector-evasion experiment, and a policy review. The Copyleaks result is a weak demonstration (two finetuned models, one tool, no human-text controls), but it is an independent test rather than a circular one. Section 6 also concedes that the authors know of no cross-modal author-text dataset and that it is 'difficult to estimate the amount and type of text necessary to build a compelling personalized model,' which tempers the certainty of the central threat model. Overall, the central claim still has independent content beyond the self-citation, so a score of 4 is appropriate.
Assumptions & free parameters
assumptions (3)
- domain assumption Fine-tuning an open LLM on a user's text yields a model that is stylistically close enough to be mistaken for that user in realistic attack contexts.
- domain assumption A motivated attacker can obtain enough private or public writing samples and consumer hardware to run local fine-tuning.
- domain assumption Text personalization is not covered by current legal definitions of deepfakes, which the paper reads as limited to image, video, and audio.
Cite this review
Pith. "Pith review of Position: It's Time to Act on the Risk of Efficient Personalized Text Generation." pith.science (2026). https://pith.science/paper/XGWJDJVK
@misc{pith2026250206560,
author = {Pith},
title = {Pith review of: Position: It's Time to Act on the Risk of Efficient Personalized Text Generation},
year = {2026},
howpublished = {\url{https://pith.science/paper/XGWJDJVK}},
note = {Machine review of arXiv:2502.06560}
}
read the original abstract
The recent surge in high-quality open-source Generative AI text models (colloquially: LLMs), as well as efficient finetuning techniques, have opened the possibility of creating high-quality personalized models that generate text attuned to a specific individual's needs and are capable of credibly imitating their writing style by refining an open-source model with that person's own data. The technology to create such models is accessible to private individuals, and training and running such models can be done cheaply on consumer-grade hardware. While these advancements are a huge gain for usability and privacy, this position paper argues that the practical feasibility of impersonating specific individuals also introduces novel safety risks. For instance, this technology enables the creation of phishing emails or fraudulent social media accounts, based on small amounts of publicly available text, or by the individuals themselves to escape AI text detection. We further argue that these risks are complementary to - and distinct from - the much-discussed risks of other impersonation attacks such as image, voice, or video deepfakes, and are not adequately addressed by the larger research community, or the current generation of open- and closed-source models.
Figures
Reference graph
Works this paper leans on
-
[2]
Frontier ai regulation: Managing emerging risks to public safety
Anderljung, M., Barnhart, J., Korinek, A., Leung, J., O’Keefe, C., Whittlestone, J., Avin, S., Brundage, M., Bullock, J., Cass-Beggs, D., Chang, B., Collins, T., Fist, T., Hadfield, G., Hayes, A., Ho, L., Hooker, S., Horvitz, E., Kolt, N., Schuett, J., Shavit, Y ., Siddarth, D., Trager, R., and Wolf, K. Frontier ai regulation: Managing emerging risks to p...
-
[4]
Blessing, G., Azeta, A., Misra, S., Osamor, V ., Fernandez-Sanz, L., and Pospelova, V
URL https://www.gov.uk/government/publications/international- ai-safety-report-2025 . Blessing, G., Azeta, A., Misra, S., Osamor, V ., Fernandez-Sanz, L., and Pospelova, V . The emerging threat of ai-driven cyber attacks: A review. Applied Artificial Intelligence,
work page 2025
-
[5]
QLoRA: Efficient finetuning of quantized llms
10 Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. QLoRA: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314,
-
[7]
Gao, I., Liang, P., and Guestrin, C
URL https://www.hepi.ac.uk/2024/02/01/new- hepi-policy-note-finds-more-than-half-of-students-have-used- generative-ai-for-help-on-assessments-but-only-5-likely-to-be- using-ai-to-cheat/ . Gao, I., Liang, P., and Guestrin, C. Model equality testing: Which model is this API serving? In International Conference on Learning Representations (ICLR),
work page 2024
-
[8]
Gunasekar, S., Zhang, Y ., Aneja, J., Mendes, C. C. T., Giorno, A. D., Gopi, S., Javaheripi, M., Kauffmann, P., de Rosa, G., Saarikivi, O., Salim, A., Shah, S., Behl, H. S., Wang, X., Bubeck, S., Eldan, R., Kalai, A. T., Lee, Y . T., and Li, Y . Textbooks are all you need. arXiv preprint arXiv:2306.11644,
-
[9]
J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W
Hu, E. J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685,
-
[11]
Fish-speech: Leverag- ing large language models for advanced multilingual text-to-speech synthesis
Liao, S., Wang, Y ., Li, T., Cheng, Y ., Zhang, R., Zhou, R., and Xing, Y . Fish-speech: Leverag- ing large language models for advanced multilingual text-to-speech synthesis. arXiv preprint arXiv:2411.01156,
-
[12]
Muldoon, C., Ikram, A., and Khan Mirza, Q. A. ”modern stylometry: A review & experimentation with machine learning”. In 2021 8th International Conference on Future Internet of Things and Cloud (FiCloud),
work page 2021
Show all 16 references
-
[13]
Gpt-4 technical report
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,
-
[15]
and Feizi, S
Saha, S. and Feizi, S. Almost ai, almost human: The challenge of detecting ai-polished writing. arXiv preprint arXiv:2502.15666,
-
[16]
Democratizing large language models via personalized parameter-efficient fine-tuning
Tan, Z., Zeng, Q., Tian, Y ., Liu, Z., Yin, B., and Jiang, M. Democratizing large language models via personalized parameter-efficient fine-tuning. arXiv preprint arXiv:2402.04401,
-
[2020]
A., Dernoncourt, F., Deilamsalehy, H., Chen, X., Zhang, R., Agarwal, S., Lipka, N., Nguyen, C
Kumar, I., Viswanathan, S., Yerra, S., Salemi, A., Rossi, R. A., Dernoncourt, F., Deilamsalehy, H., Chen, X., Zhang, R., Agarwal, S., Lipka, N., Nguyen, C. V ., Nguyen, T. H., and Za- mani, H. Longlamp: A benchmark for personalized long-form text generation. arXiv preprint arX...
-
[2021]
Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03693,
Qi, X., Zeng, Y ., Xie, T., Chen, P.-Y ., Jia, R., Mittal, P., and Henderson, P. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03693,
-
[2023]
Gptq: Accurate post-training quantization for generative pre-trained transformers
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323,
-
[2024]
A., et al
Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan, A. A., et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219,
-
[2025]
Accessed: 2025-05-11. Bengio, Y ., Mindermann, S., Privitera, D., Besiroglu, T., Bommasani, R., Casper, S., Choi, Y ., Fox, P., Garfinkel, B., Goldfarb, D., Heidari, H., Ho, A., Kapoor, S., Khalatbari, L., Longpre, S., Manning, S., Mavroudis, V ., Mazeika, M., Michael, J., New...
2025
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.