Pith. sign in

REVIEW 3 major objections 5 minor 37 references

Backtranslation and paraphrasing in the LLM era? Comparing data augmentation methods for emotion classification

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Backtranslation, not GPT prompting, gives the best emotion-classification gains.

desk verdict Useful empirical comparison, but the headline ranking may be a dataset-size artifact rather than a method effect. read the letter →

arxiv 2507.14590 v1 pith:XUYGKI35 submitted 2025-07-19 cs.CL cs.AI

{L}ukasz Radliński , Mateusz Guściora , Jan Kocoń This is my paper
classification cs.CLcs.AI
keywords dataaugmentationbacktranslationparaphrasingemotionclassificationGoEmotionslargelanguagemodelsfew-shotlearningclassimbalance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether older, deterministic data augmentation can keep pace with prompt-based generation by large language models when training data are scarce and class-imbalanced. Using the GoEmotions dataset, the authors augment the five rarest emotion labels by oversampling, paraphrasing with GPT-3.5 and GPT-4, zero- and few-shot generation, and backtranslation through several translation models and services. They report that backtranslation, especially through DeepL, gives the best classification results on two fine-tuned transformers, and that backtranslation and paraphrasing match or beat zero- and few-shot generation. If correct, this means traditional, deterministic augmentation pipelines remain a competitive alternative to generative augmentation for low-resource text classification.

What carries the argument

The central object is a controlled comparison of four augmentation pipelines on the same five minority emotion classes: oversampling with no linguistic diversity, prompt-based paraphrasing, zero- and few-shot generation, and backtranslation through multiple foreign languages. The backtranslation pipeline translates each source sentence into several target languages and back into English, using DeepL, GPT-3.5, GPT-4, GPT-4-turbo, or MarianMT models, and aggregates all round-trip outputs into one augmented dataset. The argument is carried by two quality axes -- lexical diversity, measured by word-count ratio, Jaccard dissimilarity, information entropy, and type-token ratio, and semantic fidelity, measured by cosine similarity and BERTScore -- plus the downstream F1-macro after fine-tuning LaBSE and distilBERT. Backtranslation's mechanism is that imperfect round-trip translation preserves meaning while introducing lexical variation.

What would settle it

Re-run every augmentation setup at least five times with different random seeds, hold the number of added training examples per class fixed across methods, and report F1-macro with confidence intervals; the central claim fails if DeepL backtranslation's lead over zero- and few-shot generation disappears or falls inside the noise bands.

Watch

Extended reading notes

Core claim

The central claim is that backtranslation and paraphrasing can yield comparable or even better results than zero-shot and few-shot generation of examples. In the classification experiments, backtranslation produced the best overall results: DeepL backtranslation raised LaBSE F1-macro from 0.467 to 0.497 and distilBERT F1-macro from 0.458 to 0.494, with F1-macro on the five augmented classes rising from 0.174 to 0.387 on distilBERT, a 121.99 percent increase. Notably, even simple oversampling with duplicated examples improved results and in several setups outperformed prompt-based generation. The best F1-macro scores across all classes were achieved with backtranslation using DeepL, and zero-shot learning with GPT-3.5 also exceeded 0.49; no method showed a significant drop in performance on the non-augmented classes.

Load-bearing premise

The ranking rests on the assumption that the F1-macro gaps between augmentation methods reflect systematic method effects rather than fine-tuning noise, since each setup appears to have been run once without seeds or confidence intervals, and that the augmented datasets are comparable in size and class balance even though those counts are never reported.

Editorial extensions

If this is right

  • Practitioners with scarce or imbalanced text data can choose deterministic backtranslation over costly prompt-based generation without giving up classification performance.
  • Augmenting only the rare classes does not appear to hurt the other classes, since no setup showed more than a small drop in F1-macro on non-augmented classes.
  • Oversampling duplicated examples is a stronger baseline than commonly assumed, beating several GPT-based setups on some models.
  • Augmentation method and prompt configuration matter more than model scale, because GPT-4 did not consistently outperform GPT-3.5.
  • Resource-light choices are viable: distilBERT trailed LaBSE only slightly while needing far less compute, and zero- and few-shot generation used fewer input tokens than paraphrasing or backtranslation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The comparison does not control for the number of generated examples per class, so part of backtranslation's advantage could be a data-quantity effect rather than a method effect; counting and equalizing examples across methods would test this.
  • Because each fine-tuning run appears to be a single run without seeds or confidence intervals, the ranking could change under repeated runs; a multi-seed evaluation would reveal how stable the 0.497 versus 0.492 gap is.
  • Since backtranslation and LLM paraphrasing excel on different quality axes, a natural next test is combining them, for example feeding backtranslated outputs as few-shot examples or filtering prompt-generated samples by backtranslation fidelity.
  • The GoEmotions data come from English Reddit, so the backtranslation advantage may not survive transfer to other languages, registers, or professionally curated datasets; a cross-lingual replication would delimit the claim.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper compares four data-augmentation strategies for low-resource emotion classification on GoEmotions: oversampling, paraphrasing with GPT-3.5/GPT-4, zero-shot and few-shot generation with GPT-3.5/GPT-4, and backtranslation with DeepL, GPT-3.5/GPT-4/GPT-4-turbo, and MarianMT. For each method, the authors generate additional examples for five rare emotion labels, measure the generated text's lexical diversity and semantic fidelity, and fine-tune LaBSE and DistilBERT on the augmented training sets. The central claim is that traditional backtranslation and paraphrasing can match or outperform purely generative zero/few-shot augmentation, with DeepL backtranslation yielding the best overall classification results.

Significance. If the ranking were statistically reliable, the finding would be practically valuable: it suggests that deterministic and comparatively inexpensive backtranslation pipelines can compete with prompt-based LLM generation for low-resource text classification. The paper also has strengths: it uses a standard benchmark, compares a broad set of augmentation configurations, evaluates both text quality and downstream classification, and releases code and prompts publicly. However, as detailed in the major comments, two load-bearing methodological gaps currently prevent the headline ranking from being established: per-method augmented dataset sizes are never reported, and all classification results appear to come from single fine-tuning runs without variance or significance information.

major comments (3)
  1. [Sections 4.2-4.4, 6.2, Table 3] The number of generated examples per method and per class is never reported, so the headline ranking is not identifiable as a method effect. Section 4.4 states that backtranslation aggregates outputs from ten languages 'to reach a sufficient number of generated examples'; the zero/few-shot prompts in Section 4.3 ask for 'N different sentences' without specifying N; and the paraphrasing prompts use uncontrolled targets such as 'as many as possible' and 'equal number of samples from each class'. Table 3 itself shows that oversampling 5x outperforms oversampling 3x on almost every metric (e.g., LaBSE all-class F1-macro 0.484 vs 0.477; distilBERT augmented-class F1-macro 0.307 vs 0.229), demonstrating that augmentation quantity alone drives part of the gain. Please report the final augmented dataset sizes and class counts for every setup and, ideally, include a size-controlled or class-balanced comparison.
  2. [Section 6.2, Table 3] All classification results appear to be single runs with no seeds, standard deviations, or significance tests. The headline differences are small (LaBSE F1-macro: DeepL backtranslation 0.497 vs 0-shot GPT-3.5 0.492 vs GPT-4 backtranslation 0.493), and the text uses 'significant' and 'no significant difference' without any statistical support, even though it also notes 'significant variance in results achieved by particular experimental setups'. Please add repeated runs (at least three to five seeds per configuration) and report means with standard deviations or an appropriate significance test; otherwise the qualitative ranking cannot be distinguished from fine-tuning stochasticity.
  3. [Sections 6.1-6.2] The paper presents lexical diversity and semantic fidelity measurements as an evaluation of generated data quality, but it never links these measurements to the observed classification improvements. For instance, GPT-4-turbo backtranslation has the worst semantic fidelity in Table 2 (cosine similarity 0.1516, BERTScore 0.6892) yet produces competitive classification results in Table 3 (LaBSE 0.482, distilBERT 0.479), while DeepL backtranslation has near-identical word counts, entropy, and TTR ratios (Table 1) but the highest fidelity scores. The claimed relationship between the quality metrics and downstream utility is therefore not demonstrated; please either analyze this relationship explicitly or soften the claim that the quality evaluation supports the classification findings.
minor comments (5)
  1. [Table 3] In the 'BT with gpt-4' distilBERT row, the '%Change (othr Cls)' column appears to contain a stray value '52.02' before '0.07'; please correct this formatting error.
  2. [Section 5.1, Eq. (2)] The text refers to 'the information entropy (Figure 2)', but Figure 2 is the emotion-label distribution; the intended reference is to Equation (2).
  3. [Section 4.2] The paraphrasing prompt configurations are listed as 'Prompt 1', 'Prompt 2 - Nmax', and 'Prompt 2 - Nbal', which is confusing because two distinct configurations share the name 'Prompt 2'; please use unique labels.
  4. [Section 7 and Abstract] Section 7 appropriately states that 'further experiments need to be performed to confirm these findings', but the abstract and conclusions state the backtranslation result more categorically; please align the wording with the acknowledged need for confirmation.
  5. [Section 1, Figure 1] The paper claims a 'comprehensive comparison' and 'systematic' methodology, but several hyperparameters (e.g., the number of paraphrase iterations in Prompt 1, the value of N in zero/few-shot generation, and the exact number of backtranslation outputs per language) are unspecified in the main text; please provide these details in the experimental setup or an appendix.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical benchmark with held-out evaluation, and its central comparison is not defined in terms of its own inputs.

full rationale

This paper is a comparative empirical study rather than a derivation chain. The central claim that backtranslation and paraphrasing can match or beat zero- and few-shot generation is supported by fine-tuning LaBSE and DistilBERT on augmented versions of the external GoEmotions dataset and comparing held-out F1-macro scores (Table 3). No equation defines an output metric in terms of fitted parameters, no parameter is fit to the target result, and no uniqueness theorem is invoked. The self-citations in Related Work (e.g., [12, 14, 15, 16, 17, 18, 24, 30, 31]) are background references on ChatGPT-based augmentation and do not supply the empirical ranking or any load-bearing premise. The concern that augmented dataset sizes per method are unreported and may confound the comparison is a potential validity threat about experimental control, not a circularity: the evaluation is external to the augmentation pipeline and the benchmark is an established public dataset. Therefore no circular step can be exhibited, and the score is 0.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The paper relies on standard supervised learning assumptions and several under-specified experimental choices. The main caveats are the unstated number of generated examples per method, the absence of error bars, and the implicit assumption that generated text retains its label.

free parameters (4)
  • augmentation_count_N
    The number of generated sentences per original sample for zero-shot, few-shot, and paraphrasing prompts is never reported. It directly determines dataset size and likely confounds method comparisons (Sections 4.2, 4.3).
  • oversampling_multiplier = 3x and 5x
    Baseline oversampling duplicates minority-class examples 3 or 5 times, and the choice of multiplier affects the strength of the baseline (Section 4.1).
  • backtranslation_language_set = 10 languages per model
    The set of languages (Russian, Polish, Finnish, Japanese, etc.) determines diversity and quality of backtranslated examples, but no external criterion justifies this specific list (Section 4.4).
  • prompt_configuration = Prompt 1, Prompt 2 Nmax, Prompt 2 Nbal
    Paraphrasing results depend heavily on prompt structure; three configurations are tested, but there is no principled selection or systematic search over prompts (Section 4.2).
assumptions (3)
  • domain assumption Generated examples preserve the intended emotion label.
    The paper measures semantic fidelity but does not validate the labels of generated samples. If some backtranslated or paraphrased examples change meaning or emotion, they would contaminate the training data and skew F1 results (Sections 5.2, 6.2).
  • domain assumption The same train/validation/test split is used across all augmentation methods.
    The paper never describes the data split for GoEmotions. If splits differ between method evaluations, the comparison is invalid (Section 5.3).
  • standard math F1-macro on the five augmented classes is a reliable summary of augmentation benefit.
    F1-macro treats all classes equally but is sensitive to per-class sample counts, and with differing augmentation sizes across methods the metric may not be directly comparable (Section 5.3).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Backtranslation and paraphrasing in the LLM era? Comparing data augmentation methods for emotion classification." pith.science (2026). https://pith.science/paper/XUYGKI35

@misc{pith2026250714590,
  author       = {Pith},
  title        = {Pith review of: Backtranslation and paraphrasing in the LLM era? Comparing data augmentation methods for emotion classification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XUYGKI35}},
  note         = {Machine review of arXiv:2507.14590}
}
read the original abstract

Numerous domain-specific machine learning tasks struggle with data scarcity and class imbalance. This paper systematically explores data augmentation methods for NLP, particularly through large language models like GPT. The purpose of this paper is to examine and evaluate whether traditional methods such as paraphrasing and backtranslation can leverage a new generation of models to achieve comparable performance to purely generative methods. Methods aimed at solving the problem of data scarcity and utilizing ChatGPT were chosen, as well as an exemplary dataset. We conducted a series of experiments comparing four different approaches to data augmentation in multiple experimental setups. We then evaluated the results both in terms of the quality of generated data and its impact on classification performance. The key findings indicate that backtranslation and paraphrasing can yield comparable or even better results than zero and a few-shot generation of examples.

Figures

Figures reproduced from arXiv: 2507.14590 by the authors.

Figure 1
Figure 1. Data augmentation for emotion classification [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Emotion Labels Distribution [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Correlation Heatmap of labels Furthermore, looking at the correlations between individual emotions, Fig￾ure 3, there are visible dependencies between some of the labels. These include a high correlation between annoyance and anger, sadness and grief, remorse and sadness, disappointment and sadness, fear and nervousness, desire and optimism, excitement and joy, as well as confusion and curiosity. This means that many… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Example ZSL, input and output for GPT model [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

37 extracted references · 28 canonical work pages

  1. [1]

    IEEE Transactions on Artificial Intelligence (2021)

    Abonizio, H.Q., Paraiso, E.C., Barbon, S.: Toward text data augmentation for sentiment analysis. IEEE Transactions on Artificial Intelligence (2021)

    work page 2021

  2. [2]

    Natural Language Engineering pp

    Balkus, S.V., Yan, D.: Improving short text classification with augmented data using gpt-3. Natural Language Engineering pp. 1–30 (2022)

    work page 2022

  3. [3]

    Online Social Networks and Media (2021)

    Beddiar, D.R., Jahan, M.S., Oussalah, M.: Data expansion using back translation and paraphrasing for hate speech detection. Online Social Networks and Media (2021)

    work page 2021

  4. [4]

    Advances in Neural Information Processing Systems33, 1877–1901 (2020)

    Brown, T., et al.: Language models are few-shot learners. Advances in Neural Information Processing Systems33, 1877–1901 (2020)

    work page 2020

  5. [5]

    Chai, Y., Xie, H., Qin, J.S.: Text data augmentation for large language models: A comprehensive survey of methods, challenges, and opportunities (2025)

    work page 2025

  6. [6]

    MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification

    Chen, J., Yang, Z., Yang, D.: Mixtext: Linguistically-informed interpolation of hid- den space for semi-supervised text classification. arXiv preprint arXiv:2004.12239 (2020)

    work page Pith review arXiv 2020

  7. [7]

    arXiv preprint arXiv:2302.13007 (2023)

    Dai, H., et al.: Auggpt: Leveraging chatgpt for text data augmentation. arXiv preprint arXiv:2302.13007 (2023)

    arXiv 2023

  8. [8]

    arXiv preprint arXiv:2005.00547 (2020)

    Demszky, D., Movshovitz-Attias, D., Ko, J., Cowen, A., Nemade, G., Ravi, S.: Goemotions: A dataset of fine-grained emotions. arXiv preprint arXiv:2005.00547 (2020)

    arXiv 2020

Show all 37 references
  1. [9]

    arXiv preprint arXiv:1705.00440 (2017)

    Fadaee, M., Bisazza, A., Monz, C.: Data augmentation for low-resource neural machine translation. arXiv preprint arXiv:1705.00440 (2017)

    2017 arXiv

  2. [10]

    Feng, F., Yang, Y., Cer, D., Arivazhagan, N., Wang, W.: Language-agnostic bert sentence embedding (2022)

    2022

  3. [11]

    arXiv preprint arXiv:2010.01794 (2020)

    Feng, S.Y., Gangal, V., Kang, D., Mitamura, T., Hovy, E.: Genaug: Data augmen- tation for finetuning text generators. arXiv preprint arXiv:2010.01794 (2020)

    2020 arXiv

  4. [12]

    Information Fusion114, 102692 (2025)

    Ferdinan,T.,Kocoń,J.:Fortifyingnlpmodelsagainstpoisoningattacks:Thepower of personalized prediction architectures. Information Fusion114, 102692 (2025)

    2025

  5. [13]

    Proceedings of the National Academy of Sciences120(30) (2023)

    Gilardi, F., Alizadeh, M., Kubli, M.: Chatgpt outperforms crowd workers for text- annotation tasks. Proceedings of the National Academy of Sciences120(30) (2023)

    2023

  6. [14]

    Information Fusion94, 43–65 (2023)

    Kazienko, P., et al.: Human-centered neural reasoning for subjective content pro- cessing: Hate speech, emotions, and humor. Information Fusion94, 43–65 (2023)

    2023

  7. [15]

    Electronics 13(12), 2255 (2024)

    Kochanek, M., et al.: Improving training dataset balance with chatgpt prompt engineering. Electronics 13(12), 2255 (2024)

    2024

  8. [16]

    In: 2023 IEEE International Conference on Data Mining Workshops (ICDMW)

    Kocoń, J.: Deep emotions across languages: A novel approach for sentiment prop- agation in multilingual wordnets. In: 2023 IEEE International Conference on Data Mining Workshops (ICDMW). pp. 744–749. IEEE (2023)

    2023

  9. [17]

    Information Fusion (2023) Backtranslation and Paraphrasing in the LLM era? 15

    Kocoń, J., et al.: Chatgpt: Jack of all trades, master of none. Information Fusion (2023) Backtranslation and Paraphrasing in the LLM era? 15

    2023

  10. [18]

    In: International conference on computational science

    Koptyra, B., Ngo, A., Radliński, Ł., Kocoń, J.: Clarin-emo: Training emotion recog- nition models using human annotation and chatgpt. In: International conference on computational science. pp. 365–379. Springer (2023)

    2023

  11. [19]

    arXiv preprint arXiv:2003.02245 (2020)

    Kumar, V., Choudhary, A., Cho, E.: Data augmentation using pre-trained trans- former models. arXiv preprint arXiv:2003.02245 (2020)

    2020 arXiv

  12. [20]

    ACM Transactions on Knowledge Discovery from Data (2023)

    Kwon, S., Lee, Y.: Explainability-based mix-up approach for text data augmenta- tion. ACM Transactions on Knowledge Discovery from Data (2023)

    2023

  13. [21]

    Li, X., et al.: Self-alignment with instruction backtranslation (2024)

    2024

  14. [22]

    arXiv preprint arXiv:2012.02952 (2020)

    Liu, R., et al.: Data boost: Text data augmentation through reinforcement learning guided conditional generation. arXiv preprint arXiv:2012.02952 (2020)

    2020 arXiv

  15. [23]

    arXiv preprint arXiv:2304.13861 (2023)

    Møller,A.G.,Dalsgaard,J.A.,Pera,A.,Aiello,L.M.:Isapromptandafewsamples all you need? using gpt-4 for data augmentation in low-resource classification tasks. arXiv preprint arXiv:2304.13861 (2023)

    2023 arXiv

  16. [24]

    Information Fusion p

    Ngo, A., Kocoń, J.: Integrating personalized and contextual information in fine- grained emotion recognition in text: A multi-source fusion approach with explain- ability. Information Fusion p. 102966 (2025)

    2025

  17. [25]

    Nguyen, T., et al.: Better alignment with instruction back-and-forth translation (2024)

    2024

  18. [26]

    Qi, Y., Peng, H., Wang, X., Xu, B., Hou, L., Li, J.: Constraint back-translation improves complex instruction following of large language models (2024)

    2024

  19. [27]

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners (2019)

    2019

  20. [28]

    Sanh, V., Debut, L., Chaumond, J., Wolf, T.: Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter (2020)

    2020

  21. [29]

    arXiv preprint arXiv:2010.02394 (2020)

    Sun, L., Xia, C., Yin, W., Liang, T., Yu, P.S., He, L.: Mixup-transformer: Dynamic data augmentation for nlp tasks. arXiv preprint arXiv:2010.02394 (2020)

    2020 arXiv

  22. [30]

    In: 2023 IEEE International Conference on Data Mining Workshops (ICDMW)

    Woźniak, S., Kocoń, J.: From big to small without losing it all: Text augmen- tation with chatgpt for efficient sentiment analysis. In: 2023 IEEE International Conference on Data Mining Workshops (ICDMW). pp. 799–808. IEEE (2023)

    2023

  23. [31]

    arXiv preprint arXiv:2402.09269 (2024)

    Woźniak, S., Koptyra, B., Janz, A., Kazienko, P., Kocoń, J.: Personalized large language models. arXiv preprint arXiv:2402.09269 (2024)

    2024 arXiv

  24. [32]

    In: Computational Science–ICCS 2019: 19th International Conference, Faro, Portugal, June 12–14, 2019, Proceedings, Part IV

    Wu, X., Lv, S., Zang, L., Han, J., Hu, S.: Conditional bert contextual augmenta- tion. In: Computational Science–ICCS 2019: 19th International Conference, Faro, Portugal, June 12–14, 2019, Proceedings, Part IV. Springer International Publish- ing (2019)

    2019

  25. [33]

    Advances in Neural Information Processing Systems (2020)

    Xie, Q., Dai, Z., Hovy, E., Luong, T., Le, Q.: Unsupervised data augmentation for consistency training. Advances in Neural Information Processing Systems (2020)

    2020

  26. [34]

    In: 2023 International Joint Conference on Neural Networks (IJCNN)

    Xu, Q., Hong, Y., Chen, J., Yao, J., Zhou, G.: Data augmentation via back- translation for aspect term extraction. In: 2023 International Joint Conference on Neural Networks (IJCNN). pp. 1–8 (2023)

    2023

  27. [35]

    arXiv preprint arXiv:2104.08826 (2021)

    Yoo, K.M., Park, D., Kang, J., Lee, S.W., Park, W.: Gpt3mix: Leveraging large- scale language models for text augmentation. arXiv preprint arXiv:2104.08826 (2021)

    2021 arXiv

  28. [36]

    arXiv preprint arXiv:2202.13047 (2022)

    Zheng, C., et al.: Augesc: Dialogue augmentation with large language models for emotional support conversation. arXiv preprint arXiv:2202.13047 (2022)

    2022 arXiv

  29. [37]

    Zhou, Y., Guo, C., Wang, X., Chang, Y., Wu, Y.: A survey on data augmentation in large model era (2024)

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.