REVIEW 4 major objections 7 minor 30 references
Optimising ChatGPT for creativity in literary translation: A case study from English into Dutch, Chinese, Catalan and Spanish
T0 review · 4 major / 7 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read Prompting ChatGPT with 'translate creatively' at temperature 1.0 yields the most creative literary outputs among tested settings and beats DeepL in three languages.
desk verdict Useful multilingual replication of the 'creatively' prompt effect, but the abstract's temperature claim is untested and the evaluation rests on a single annotator. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing instrument is the creativity index (CI), computed as $$\text{CI} = \left(\frac{\#\text{CSs}}{\#\text{UCPs}} - \frac{\text{error points}}{\#\text{words in ST}}\right) \times 100.$$ Creative shifts (CSs) are annotated on 54 pre-selected units of creative potential from the source text, with each solution classified as abstraction, concretization, or modification, following Bayer-Hohenwarter's taxonomy. Error points come from a DQF-MQM-style severity scale (neutral, minor, major, critical), and the formula converts two qualitative judgments—novelty and acceptability—into a single number that ranks every configuration. The paper also uses automatic metrics (BLEU, chrF, TER, COMET, COMET-Kiwi), but the creativity index is what carries the central claim.
What would settle it
Re-annotate the same ChatGPT and DeepL outputs with several independent annotators per language and check whether 'Translate the following text into [TG] creatively' at temperature 1.0 still yields the highest creativity index; as a stricter test, recompute the index on a larger, independently sampled set of creative-potential units from the same story and see whether the same configuration wins.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that minimalism in prompting beats informativeness for eliciting creativity from ChatGPT. The configuration 'Translate the following text into [TG] creatively' at temperature 1.0 achieves the highest creativity index among all tested ChatGPT settings in English-to-Spanish, English-to-Dutch, and English-to-Chinese, and it also outscores DeepL in those three directions. In Catalan, the best setting is the even plainer 'Translate the following text into [TG]' at document level, because the creativity prompt and the genre prompt leave the story's invented nicknames untranslated, hurting the score. Temperature 1.0 generally increases both creative shifts and errors relative to 0.0, but the net index favors 1.0 in three languages. Across all languages and settings, the creativity index of the best ChatGPT output remains substantially below the professional human translations used as reference, and this gap is the paper's concluding caution about the model's creative ceiling.
Load-bearing premise
The ranking of configurations rests on a creativity index computed by one annotator per language on just 54 units of creative potential, so the results stand or fall on whether that annotation and unit selection faithfully measure translational creativity.
Editorial extensions
If this is right
- A direct, short request for creativity is a more effective lever than supplying genre and author information when the goal is creative literary output in ChatGPT.
- Raising temperature from 0.0 to 1.0 is net-positive for creativity in Spanish, Dutch, and Chinese, even though it adds errors, so users optimizing for creativity should not default to the lower setting.
- The best ChatGPT output still trails professional human translations on the same index, so the model cannot currently replace a literary translator.
- The optimal granularity is language-dependent: paragraph-level wins for Dutch and Chinese, document-level for Catalan and Spanish, meaning no single 'best context' setting generalizes.
- Standard automatic metrics do not track the human creativity ranking, so creativity evaluation continues to require manual annotation.
Reading between the lines
- If the one-word 'creatively' effect is robust, it may be a cheap, generalizable steering signal for other large language models and literary genres, without any fine-tuning.
- Because the Catalan exception hinges on a handful of untranslated nicknames, the index is sensitive to a small number of units; a different weighting or a different set of units could plausibly reorder the configurations.
- A testable extension is whether the same minimal-prompt advantage holds for non-literary but stylistically marked text (e.g., marketing copy, subtitles), where context prompts might matter more.
- The large gap between machine and human creativity, even under the best prompt, offers a concrete benchmark for measuring future progress in generative translation systems.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports a case study evaluating ChatGPT (gpt-4o-2024-08-06) for literary translation from English into Dutch, Chinese, Catalan, and Spanish, using a short science-fiction story. Six ChatGPT configurations are compared, varying text granularity (paragraph vs. document), temperature (0.0 vs. 1.0), and zero-shot prompting strategy (minimal, genre/author-informed, direct creativity request), alongside DeepL (and for Catalan, Softcatalà and Google Translate) and human translations. Translations are manually annotated for creative shifts and errors, and a creativity index (CI) combines the two. The central claim is that the minimal prompt 'Translate the following text into [TG] creatively' at temperature 1.0 yields the most creative outputs and outperforms DeepL in Spanish, Dutch, and Chinese, although all ChatGPT outputs remain below human translation. The paper also reports automatic-metric correlations, an ANOVA on creative shifts and error points, and a sustainability statement.
Significance. If the central claim held, the paper would provide concrete, practical guidance for eliciting more creative literary translations from ChatGPT, with a cross-linguistic comparison (four target languages) and a clear negative result against human translation. The study is transparent: all code and data are released, the annotation procedure is described in detail, and the authors explicitly acknowledge the exploratory nature of the work. The finding that ChatGPT and other MT systems remain far below human translators in creativity, despite prompt engineering, is valuable and likely robust. However, the positive headline claim about the optimal prompt and temperature is not yet supported by the evidence as analyzed, because of the adaptive experimental design, the single-annotator evaluation without inter-annotator agreement, and the absence of inferential tests on the creativity index itself.
major comments (4)
- [§3.3.2–§3.3.3 and Abstract] The abstract claims that the minimal creativity instruction 'at the temperature of 1.0' outperforms other configurations, but this exact combination was never directly tested. In Phase 2, temperature is varied only with Prompt 1; in Phase 3, prompts are varied at the single temperature already selected per language (1.0 for ES/NL/ZH, 0.0 for CA) using Prompt 1's results from Phase 2. Thus there is no comparison of Prompt 3 at temperature 0.0 versus 1.0, and the specific interaction between Prompt 3 and temperature is untested. This is load-bearing because the abstract's wording implies an optimal jointly tuned configuration, whereas the experiment only shows that, in this adaptive procedure, Prompt 3 happened to be the best among the prompts tested at the temperature inherited from Phase 2. The authors should either run Prompt 3 at both temperatures and report the comparison, or explicitly rephrase the claim as 'the best configuration among those tested was Prompt 3 at the temperature selected in Phase 2' without asserting an optimality for temperature 1.0.
- [§3.4–§3.5] The creativity index (CI) is computed from annotations by one single annotator per language, and no inter-annotator agreement is reported anywhere in the manuscript. This is a serious concern for the ranking of configurations, especially for languages where the CI differences between configurations are small. For example, in Table 4, ENZH Prompt 3 scores 1.03 versus Prompt 2 at -2.48, and in Table 3, ENZH T-1.0 scores -1.48 versus T-0.0 at -5.38. These gaps could easily be overturned by a different annotator's subjective judgments on a few units of creative potential or error severities. Since the CI is the sole criterion for the headline result, the absence of any reliability evidence makes the reported ranking fragile. The authors should provide at least a second annotation for a subset, report Cohen's kappa or a similar measure, and use annotation-based confidence intervals or a sensitivity analysis to show that the main conclusions are stable across plausible annotation noise.
- [§5] The paper's own ANOVA undermines the central claim. The aligned-rank-transform ANOVA on the number of creative shifts (CSs) reports no significant main effect of Modality and no Modality×Language interaction, with only Language reaching significance. Creative shifts are the numerator of the CI and the operational definition of novelty, so this means the data do not demonstrate that prompting strategy affects the creativity component of the index. The significant Modality effect is found for Error Points, which is an acceptability component, not creativity per se. Moreover, the ANOVA treats individual sentences as independent observations even though each configuration is a single system output; the effective sample size is the number of configurations (7–8 per language), not the hundreds of sentences. The authors should either perform a permutation or bootstrap test on the CI differences across configurations, or frame the conclusion as 'differences in the combined index' without implying that the creative-shift component differs significantly across prompts.
- [§3.3 and §4] The adaptive experimental design selects the better granularity in Phase 1, then the better temperature in Phase 2, then the better prompt in Phase 3, all using the same 54 UCPs and 48 sentences for the final evaluation. This means the reported 'best configuration' is the result of maximizing the CI on the evaluation set itself, not on a held-out or confirmatory sample. When combined with the fact that each configuration is run once (a single stochastic output at temperature 1.0), the selection process can capitalize on sampling luck. The paper implicitly acknowledges variability in §6 ('a level of randomization in the output that is quite unpredictable'), but it does not address the statistical consequences for the ranking. The authors should either run multiple repetitions per configuration and report variance, or perform a small confirmatory study on a separate set of sentences, or explicitly label the result as an exploratory, within-sample optimum rather than a validated finding.
minor comments (7)
- [Throughout] There are inconsistent renderings of the model name: 'Chat-GPT' in the abstract and 'ChatGPT' elsewhere; please standardize.
- [§3.1, §3.2, References] The author name 'Vonnegut' is typeset as 'V onnegut' in multiple places, including the reference entry for the primary source text; correct these typos.
- [§3.2] The text says that the 185 UCPs were annotated by 'two experienced translators and researchers' in the prior study, while the current study uses one annotator per language. Please make this distinction explicit to avoid confusion about the source of the UCP list versus the current annotation of the translations.
- [§3.5] The CI formula uses fixed severity weights (Minor=1, Major=5, Critical=15) but no sensitivity analysis is provided. Given the small CI gaps for ZH, a brief sensitivity check (e.g., reweighting or excluding Critical errors) would strengthen the robustness claims.
- [§5] The ANOVA results do not include effect sizes (e.g., partial eta-squared) or any measure of uncertainty for the pairwise comparisons beyond p-values; adding these would help the reader gauge the magnitude of the reported differences.
- [Table 5 and Table 10] The labels 'ENCA-S', 'ENCA-G', and '3d' in Table 10 are not fully defined in the captions; please explain the abbreviations and the meaning of the additional columns in the captions or in the main text.
- [Sustainability statement] The sustainability statement contains a typo, 'GhatGPT' for 'ChatGPT', and the sentence 'To the best of our knowledge, the average CO2 emissions of GhatGPT models is not disclosed' has a number-agreement error; clean this up.
Circularity Check
No circularity: the creativity index is a fixed, pre-existing measurement formula; the ranking is empirical and not forced by construction.
full rationale
The paper's central claim is an empirical ranking of ChatGPT configurations, not a derivation from the creativity index. The CI formula in Section 3.5 (CI = (#CSs/#UCPs - #error points/#words in ST) x 100) is taken unchanged from the authors' prior work (Guerberof-Arenas and Toral 2020, 2022); it has no fitted parameters and does not encode any expectation about which prompt wins. The observed result that Prompt 3 wins in ES, NL and ZH is data-dependent: the same formula makes Prompt 1 win in Catalan, and Prompt 3 could plausibly have scored lower if its extra creative shifts had brought more error points. Reusing the UCP annotations and human references from the same prior project is dataset reuse, not circular reasoning; the ChatGPT outputs are newly generated and annotated separately. The paper's self-citations to Du (2024) are consistency checks, not load-bearing proof. The abstract's phrase 'at the temperature of 1.0' is a design limitation, not a circularity: Phase 2 selected one temperature per language using Prompt 1, and Phase 3 then compared prompts at that fixed temperature, so Prompt 3 at temperature 0.0 was never tested for ES/NL/ZH. That is an untested interaction or confound, not a step that reduces to its inputs. The acknowledged single-annotator limitation concerns measurement reliability, not circularity. The paper's own ANOVA finding of no significant main effect of modality on CSs is a statistical concern, but it is not an instance of a conclusion being equivalent to its inputs by construction.
Assumptions & free parameters
free parameters (1)
- error severity weights (Minor=1, Major=5, Critical=15) =
1, 5, 15
assumptions (4)
- domain assumption The creativity index formula CI = (#CSs / #UCPs - #error points / #words in ST) x 100 is a valid measure of translational creativity.
- domain assumption The 54 selected units of creative potential (UCPs) are representative of the creative challenges in the source text.
- domain assumption The DQF-MQM error severity weights (Minor=1, Major=5, Critical=15) appropriately reflect error impact on acceptability.
- domain assumption A single annotator per language can reliably distinguish creativity across configurations.
Cite this review
Pith. "Pith review of Optimising ChatGPT for creativity in literary translation: A case study from English into Dutch, Chinese, Catalan and Spanish." pith.science (2026). https://pith.science/paper/IAOQHQSZ
@misc{pith2026250418221,
author = {Pith},
title = {Pith review of: Optimising ChatGPT for creativity in literary translation: A case study from English into Dutch, Chinese, Catalan and Spanish},
year = {2026},
howpublished = {\url{https://pith.science/paper/IAOQHQSZ}},
note = {Machine review of arXiv:2504.18221}
}
read the original abstract
This study examines the variability of Chat-GPT machine translation (MT) outputs across six different configurations in four languages,with a focus on creativity in a literary text. We evaluate GPT translations in different text granularity levels, temperature settings and prompting strategies with a Creativity Score formula. We found that prompting ChatGPT with a minimal instruction yields the best creative translations, with "Translate the following text into [TG] creatively" at the temperature of 1.0 outperforming other configurations and DeepL in Spanish, Dutch, and Chinese. Nonetheless, ChatGPT consistently underperforms compared to human translation (HT).
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Gerrit Bayer-Hohenwarter. 2009. Translational creativity: how to measure the unmeasurable, volume 37. Samfundslitteratur Copenhagen
work page 2009
-
[2]
Gerrit Bayer-Hohenwarter. 2011. https://doi.org/10.7202/1008339ar “Creative Shifts” as a Means of Measuring and Promoting Translational Creativity . Meta Journal des traducteurs, 56(3):663--692
-
[3]
Shuxiang Du. 2024. https://arts.studenttheses.ub.rug.nl/35972/ Optimizing Creative Translations through ChatGPT: An analysis of the Creative Potential of Machine Translation in Literary TextsCommunication and Information Studies . Master's thesis, University of Groningen
work page 2024
-
[4]
Margot Fonteyne, Arda Tezcan, and Lieve Macken. 2020. https://aclanthology.org/2020.lrec-1.468/ Literary machine translation under the magnifying glass: Assessing the quality of an nmt-translated detective novel on document level . In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 3790--3798
work page 2020
-
[5]
Yuan Gao, Ruili Wang, and Feng Hou. 2024. https://doi.org/10.1145/3700410.3702123 How to design translation prompts for chatgpt: An empirical study . In Proceedings of the 6th ACM International Conference on Multimedia in Asia Workshops, pages 1--7
arXiv 2024
-
[6]
Ana Guerberof-Arenas and Antonio Toral. 2020. https://doi.org/10.1075/ts.20035.gue The impact of post-editing and machine translation on creativity and reading experience . Translation Spaces, 9(2):255--282
-
[7]
Ana Guerberof-Arenas and Antonio Toral. 2022. https://doi.org/10.1075/ts.21025.gue Creativity in translation . Translation Spaces, 11(2):184--212
-
[8]
Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. https://arxiv.org/abs/2302.09210 How good are gpt models at machine translation? a comprehensive evaluation . arXiv preprint arXiv:2302.09210
arXiv 2023
Show all 30 references
-
[9]
Daphne Ippolito, Reno Kriz, Jo \ a o Sedoc, Maria Kustikova, and Chris Callison-Burch. 2019. https://aclanthology.org/P19-1365/ Comparison of diverse decoding methods from conditional language models . In Proceedings of the 57th Annual Meeting of the Association for Computatio...
2019
-
[10]
Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, Shuming Shi, and Zhaopeng Tu. 2023. https://arxiv.org/abs/2301.08745 Is chatgpt a good translator? yes with gpt-4 as the engine . arXiv preprint arXiv:2301.08745
2023 arXiv
-
[11]
Katikapalli Subramanyam Kalyan. 2023. https://doi.org/10.1016/j.nlp.2023.100048 A survey of GPT-3 family large language models including ChatGPT and GPT-4 . Natural Language Processing Journal, 6:100048
2023
-
[12]
Marzena Karpinska and Mohit Iyyer. 2023. https://aclanthology.org/2023.wmt-1.41/ Large language models effectively leverage document-level context for literary translation, but critical errors persist . In Proceedings of the Eighth Conference on Machine Translation, pages 419--451
2023
-
[13]
Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ond r ej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Marzena Karpinska, Philipp Koehn, Benjamin Marie, Christof Monz, Kenton Murray, Masaaki Nagata, ...
2024 doi
-
[14]
Arle Lommel, Hans Uszkoreit, and Aljoscha Burchardt. 2014. https://doi.org/10.5565/rev/tradumatica.77 Multidimensional Quality Metrics (MQM): A Framework for Declaring and Describing Translation Quality Metrics . Tradumàtica tecnologies de la traducció, (12):455--463
2014 doi
-
[15]
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 Bleu: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02, page 311...
2002
-
[16]
Max Peeperkorn, Tom Kouwenhoven, Dan Brown, and Anna Jordanous. 2024. https://arxiv.org/abs/2405.00492 Is temperature the creativity parameter of large language models? arXiv preprint arXiv:2405.00492
2024 arXiv
-
[17]
Keqin Peng, Liang Ding, Qihuang Zhong, Li Shen, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. 2023. https://arxiv.org/abs/2303.13780 Towards Making the Most of ChatGPT for Machine Translation
2023 arXiv
- [18]
-
[19]
Maja Popovi \'c . 2015. https://doi.org/10.18653/v1/W15-3049 chr F : character n-gram F -score for automatic MT evaluation . In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392--395, Lisbon, Portugal. Association for Computational Linguistics
2015 doi
-
[20]
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. https://aclanthology.org/2020.emnlp-main.213/ Comet: A neural framework for mt evaluation . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685--2702
2020
-
[21]
Ricardo Rei, Marcos Treviso, Nuno M Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, Jos \'e GC de Souza, Taisiya Glushkova, Duarte Alves, Lu \' sa Coheur, et al. 2022. https://aclanthology.org/2022.wmt-1.60/ Cometkiwi: Ist-unbabel 2022 submission for the quality e...
2022
-
[22]
Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. 2006. https://aclanthology.org/2006.amta-papers.25/ A study of translation edit rate with targeted human annotation . In Proceedings of the 7th Conference of the Association for Machine Translation...
2006
-
[23]
Arda Tezcan, V \'e ronique Hoste, and Lieve Macken. 2017. https://doi.org/10.1163/9789004351790_012 Scate taxonomy and corpus of machine translation errors . Trends in E-tools and resources for translators and interpreters, 45:219--244
2017 doi
-
[24]
Translated. 2025. https://translated.com/speed-to-singularity Discover How Close We Are to AI Singularity
2025
-
[25]
Kurt Vonnegut. 1999. Bagombo Snuff Box . Putnam Adult
1999
-
[26]
Longyue Wang, Zefeng Du, Wenxiang Jiao, Chenyang Lyu, Jianhui Pang, Leyang Cui, Kaiqiang Song, Derek Wong, Shuming Shi, and Zhaopeng Tu. 2024. https://doi.org/10.18653/v1/2024.findings-acl.428 Benchmarking and improving long-text translation with large language models . In Fin...
2024 doi
-
[27]
Longyue Wang, Chenyang Lyu, Tianbo Ji, Zhirui Zhang, Dian Yu, Shuming Shi, and Zhaopeng Tu. 2023. https://aclanthology.org/2023.emnlp-main.1036/ Document-level machine translation with large language models . In Proceedings of the 2023 Conference on Empirical Methods in Natura...
2023
-
[28]
Vinnie Wong. 2024. Gen AI’s Environmental Ledger: A Closer Look at the Carbon Footprint of ChatGPT . https://piktochart.com/blog/carbon-footprint-of-chatgpt/. Accessed: 2025/02/07
2024
-
[29]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
-
[30]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.