REVIEW 5 major objections 5 minor 33 references
Transforming Delete, Retrieve, Generate Approach for Controlled Text Style Transfer
T0 review · 5 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A Transformer-based delete-and-generate pipeline outperforms prior style-transfer systems on five datasets, and the paper proposes GLEU as a metric that tracks human preferences better than BLEU.
desk verdict A genuinely useful BERT-deletion + GPT-generation combination for non-parallel style transfer, but the headline claim overstates what the paper's own tables show. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing components are three. The Delete Transformer is a pre-trained Transformer text classifier whose per-head, per-layer attention weights onto its [CLS] token serve as importance scores for each input token; the paper selects the single head-layer pair that, after removing the top $\gamma|x|$ tokens, most raises the classifier's confusion about the sentence's style (Eqs. 2 and 5). The Generative Style Transformer (GST) is a decoder-only Transformer language model, pre-trained on a large text corpus, that takes content, with or without retrieved target attributes, and the target style as input and generates the output sentence token by token. The two variants, B-GST (blind to target attributes) and G-GST (guided by retrieved attributes), share this machinery but differ in whether a Retrieve step supplies explicit target attribute tokens. A third component, the GLEU metric, scores outputs by n-gram overlap among source, reference, and generated sentence, letting it penalize wrongly changed content words while rewarding successful style edits.
What would settle it
Run B-GST on the Yelp sentiment task while replacing the chosen head-layer attention scores with random token scores that delete the same fraction of tokens; if human raters do not noticeably prefer the attention-based deletions for content preservation and style match, then the claim that attention identifies style attributes is not doing the work.
Extended reading notes
Core claim
The central discovery is that a Transformer classifier's per-head attention onto its [CLS] token can act as a deletion policy for style transfer. The Delete Transformer identifies style-bearing words by choosing the head-layer pair whose attention, when the highest-attended tokens are removed, most confuses the classifier about the sentence's style; the remaining tokens are treated as content. A decoder-only Transformer language model, pre-trained on a large unlabeled corpus, is then fine-tuned to reconstruct sentences from their own content, or from content plus retrieved or noisy attributes, using the target style as a prompt. The paper reports that in human evaluations on five datasets covering sentiment, gender, and political slant, its blind variant B-GST is preferred over the previous best systems for content preservation, fluency, style match, and overall quality, and that GLEU, an n-gram metric that compares source, reference, and output, correlates better with those human judgments than BLEU does.
Load-bearing premise
The method assumes that the attention weights from one chosen head-layer pair of the style classifier mark style words, not content words; if that fails, the delete step either throws away content or leaves style in place, and the pipeline cannot do both.
Editorial extensions
If this is right
- On the Yelp, Amazon, and Captions datasets, human raters prefer B-GST over the previous Delete-and-Retrieve system for content, fluency, style match, and overall quality.
- On the Political and Gender datasets, human raters prefer B-GST over back-translation for content and fluency.
- Because B-GST keeps output length closely correlated with the source sentence length, the generated rewrites are less prone to the drift seen in earlier systems.
- GLEU scores for B-GST are consistently the highest across datasets, which the paper reads as evidence that the model makes only necessary edits.
- G-GST offers controllability: at inference the user can supply desired target attributes directly, bypassing the Retrieve step, which is useful when the target corpus lacks similar sentences.
Reading between the lines
- If the head-layer attention selection generalizes, the same delete mechanism could be applied to formality, toxicity, or dialect rewriting without retraining a classifier on each new style pair.
- The gap between B-GST and G-GST points to the Retrieve component as the bottleneck; a learned attribute predictor or retrieval over latent attribute vectors could restore controllability without sacrificing quality.
- GLEU's fit with human ratings on these five datasets suggests style-transfer evaluation can follow grammatical-error-correction practice; a systematic correlation study across more models and human-judgement setups could settle which metric should be standard.
- The selected head-layer pair and the deletion-rate parameter are tuned per dataset; a sensitivity analysis of these choices would tell whether the method's gains are robust or concentrated in a narrow operating point.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents a non-parallel text style transfer system built on a Delete-Retrieve-Generate (DRG) framework. The Delete component uses attention weights from a fine-tuned BERT-based classifier (Delete Transformer) to identify and remove style attributes; the Retrieve component selects a similar target-style sentence via TF-IDF; the Generate component is a decoder-only GPT-based Transformer (GST) fine-tuned with a reconstruction objective, with two variants (B-GST and G-GST). The authors report human and automatic evaluations on Yelp, Amazon, Captions, Political, and Gender datasets, claiming state-of-the-art performance, and they propose GLEU as an automatic evaluation metric.
Significance. The idea of using attention from a single head-layer of a Transformer as style-importance scores is novel and potentially useful, and the use of a pre-trained GPT model for generation is a sensible extension of the DRG framework. The authors make code and data available, and they include a human evaluation of the delete mechanism. However, the significance is currently limited by evaluation weaknesses: key claims of style-transfer superiority are not consistently supported by the reported data, and the statistical basis of the human evaluation is not documented.
major comments (5)
- [Abstract and Section 4.3, Table 2] The abstract and Section 4.3 claim that the models 'outperform state-of-art systems across 5 datasets on sentiment, gender and political slant transfer.' Table 2 shows that on CAPTIONS, D&R is preferred over B-GST on target style match (47.75% vs 34%), so B-GST is not superior on the style-transfer axis for that dataset. For POLITICAL and GENDER, Section 4.1 states that annotators only rated content and fluency because target style strength is hard to judge; the only style signal for POLITICAL is the automatic accuracy in Table 5, where B-GST scores 71.2% versus BT's 88.0%. Thus the paper's own evidence contradicts the style-transfer superiority part of the claim, and the claim should be revised or additional style evaluations supplied.
- [Section 4.1, Tables 2 and 3] The human evaluation results in Tables 2 and 3 are reported as pairwise preference percentages with no information about the number of annotators, confidence intervals, or significance tests. For example, the 34% vs 47.75% difference on CAPTIONS style match cannot be assessed for reliability without such information. Please report annotator counts and at least approximate confidence intervals or a significance test for the main comparisons.
- [Abstract and Section 4.3] The abstract states that GLEU 'compare[s] better with human ratings than the predominantly used BLEU score,' but Section 4.3 explicitly says 'a detailed statistical correlation study is left for future work.' This is an unsupported claim; it should be backed by a correlation analysis (e.g., system-level or segment-level Spearman correlation with human judgments) or presented as a hypothesis rather than a finding.
- [Sections 2.1.1 and 2.3.2] The Delete Transformer is used both to select the head-layer pair for deletion (Eq. 5) and, during inference, to re-rank beam outputs by target-style match (Section 2.3.2). This means the automatic style accuracy in Tables 4 and 5 measures the model against the same classifier that selects its outputs, which can inflate style scores. Additionally, the deletion proportion gamma and smoothing parameter lambda are tuned per dataset, but no sensitivity analysis is provided, and the validity of the single head-layer assumption is only tested on 200 sentences. Please include a sensitivity analysis and clarify the consequences of the beam-selection procedure for the automatic style metric.
- [Section 4.2] The FastText style classifiers used for automatic style accuracy have accuracies of 80% on CAPTIONS and 82% on GENDER (Section 4.2). For a binary task, 80% accuracy means the classifier is wrong on 20% of examples, so automatic style accuracy on these datasets is a noisy measure. Since the POLITICAL and GENDER style comparisons rely on this metric, its limitations should be discussed and, where possible, supplemented with human style judgments.
minor comments (5)
- [Throughout] The paper repeatedly uses 'state-of-art' instead of 'state-of-the-art' (e.g., abstract, Section 1, Section 3.2); please fix this typo consistently.
- [Table 6] Table 6 refers to colored attributes, but the printed text does not show colors; use bold or underline to mark attributes.
- [Equation (2)] Equation (2) uses softmax over w to define α_{h,l}(w), which is a distribution over tokens; if this normalized value is used as an importance score, please state this explicitly and clarify that the score is not a raw attention weight.
- [Section 2.2] Section 2.2 states that TF-IDF retrieval gives 'best retrieval results' without presenting a comparison; please add a small table or a quantitative justification for this choice.
- [Section 4.3, Figure 2] Figure 2 is referenced in Section 4.3 but the actual plot is not visible in the manuscript text; please ensure the figure is included and legible.
Circularity Check
No significant circularity: the central claim rests on external human evaluation and independent FastText metrics; no prediction reduces to a fit.
full rationale
Walking the derivation chain, the paper's components are not defined in terms of the claimed outputs. The Delete Transformer assigns token importance scores from BERT attention (Eq. 2), selects a head-layer pair by minimizing classifier confusion on a validation set (Eqs. 4-5), and deletes the top-gamma tokens; gamma and the head-layer pair are hyperparameters selected on validation data, not fitted to the reported test outcomes. Both GST variants are trained with a reconstruction objective (Eqs. 7-8), which is a standard non-parallel training device rather than a way of encoding the target-style transfer result. The headline claim of outperforming state-of-the-art systems is supported by human preference judgments in Tables 2-3 and by automatic metrics from separately trained FastText classifiers and a separate GPT-2 perplexity model in Tables 4-5. The only self-referential element is that the Delete Transformer is also used at inference to re-rank the beam (Section 2.3.2, 'The output beam ... that obtains the highest target-style match score using the Delete Transformer described earlier, is chosen as the output sentence'), but this is not the paper's evidence for its claims; the central comparison uses independent human and FastText evaluations. The skeptical observation that the paper's own tables do not uniformly support the abstract's 'outperform across 5 datasets' claim (e.g., D&R wins on CAPTIONS target style match, and BT has higher automatic style accuracy on POLITICAL) is a correctness or evaluation-interpretation issue, not circularity. No uniqueness theorem is imported from the authors' prior work, no ansatz is smuggled in via self-citation, and no known empirical pattern is merely renamed. Therefore the appropriate finding is no significant circularity.
Assumptions & free parameters
free parameters (4)
- Deletion proportion gamma =
Not reported; tuned per dataset
- Smoothing parameter lambda =
Not reported
- G-GST noise rates =
10% / 5% / 5%
- Beam width and look-left window =
5 / 1
assumptions (6)
- standard math Transformer attention equations (Vaswani et al. 2017) and BERT pretraining assumptions hold
- domain assumption Style attributes are localized to a small subset of words
- ad hoc to paper Attention weights of a single head-layer pair act as reliable style importance scores
- domain assumption Input reduction (Feng et al. 2018) is a valid interpretation method for attention
- domain assumption Target style corpora contain sentences whose retrieved attributes are compatible with source content
- domain assumption Style labels in the datasets are reliable enough for training classifiers and deletion
Cite this review
Pith. "Pith review of Transforming Delete, Retrieve, Generate Approach for Controlled Text Style Transfer." pith.science (2026). https://pith.science/paper/SZCZYLFG
@misc{pith2026190809368,
author = {Pith},
title = {Pith review of: Transforming Delete, Retrieve, Generate Approach for Controlled Text Style Transfer},
year = {2026},
howpublished = {\url{https://pith.science/paper/SZCZYLFG}},
note = {Machine review of arXiv:1908.09368}
}
read the original abstract
Text style transfer is the task of transferring the style of text having certain stylistic attributes, while preserving non-stylistic or content information. In this work we introduce the Generative Style Transformer (GST) - a new approach to rewriting sentences to a target style in the absence of parallel style corpora. GST leverages the power of both, large unsupervised pre-trained language models as well as the Transformer. GST is a part of a larger `Delete Retrieve Generate' framework, in which we also propose a novel method of deleting style attributes from the source sentence by exploiting the inner workings of the Transformer. Our models outperform state-of-art systems across 5 datasets on sentiment, gender and political slant transfer. We also propose the use of the GLEU metric as an automatic metric of evaluation of style transfer, which we found to compare better with human ratings than the predominantly used BLEU score.
Figures
Reference graph
Works this paper leans on
-
[1]
URL: " 'urlintro :=
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. http://dl.acm.org/citation.cfm?id=2969239.2969370 Scheduled sampling for sequence prediction with recurrent neural networks . In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, NIPS'15, pages 1171--1179, Cambridge, MA, USA. MIT Press
-
[4]
Daniel Cer, Yinfei Yang, Sheng yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018. http://arxiv.org/abs/1803.11175 Universal sentence encoder
arXiv 2018
-
[5]
Ning Dai, Jianze Liang, Xipeng Qiu, and Xuanjing Huang. 2019. http://arxiv.org/abs/1905.05621 Style transformer: Unpaired text style transfer without disentangled latent representation
arXiv 2019
-
[6]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805
arXiv 2018
-
[7]
Shi Feng, Eric Wallace, Alvin Grissom II, Mohit Iyyer, Pedro Rodriguez, and Jordan Boyd-Graber. 2018. https://www.aclweb.org/anthology/D18-1407 Pathologies of neural models make interpretations difficult . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3719--3728, Brussels, Belgium. Association for Computa...
work page 2018
-
[8]
Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. 2018. Style transfer in text: Exploration and evaluation. In Thirty-Second AAAI Conference on Artificial Intelligence
2018
Show all 33 references
-
[9]
Amir Globerson and Sam Roweis. 2006. https://doi.org/10.1145/1143844.1143889 Nightmare at test time: Robust learning by feature deletion . In Proceedings of the 23rd International Conference on Machine Learning, ICML '06, pages 353--360, New York, NY, USA. ACM
2006
-
[10]
Hongyu Gong, Suma Bhat, Lingfei Wu, Jinjun Xiong, and Wen mei Hwu. 2019. http://arxiv.org/abs/1903.10671 Reinforcement learning based text style transfer without parallel training corpus
2019 arXiv
-
[11]
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P. Xing. 2017. http://proceedings.mlr.press/v70/hu17e.html Toward controlled generation of text . In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine...
2017
-
[12]
Vineet John, Lili Mou, Hareesh Bahuleyan, and Olga Vechtomova. 2018. Disentangled representation learning for non-parallel text style transfer
2018
-
[13]
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427--431. Ass...
2017
-
[14]
Jiwei Li, Will Monroe, and Daniel Jurafsky. 2016. Understanding neural networks through representation erasure. CoRR, abs/1612.08220
2016 arXiv
-
[15]
Juncen Li, Robin Jia, He He, and Percy Liang. 2018. https://doi.org/10.18653/v1/N18-1169 Delete, retrieve, generate: a simple approach to sentiment and style transfer . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computational Li...
2018 doi
-
[16]
Remi Mir, Bjarke Felbo, Nick Obradovich, and Iyad Rahwan. 2019. http://arxiv.org/abs/1904.02295 Evaluating style transfer for text
2019 arXiv
-
[17]
Courtney Napoles, Keisuke Sakaguchi, Matt Post, and Joel Tetreault. 2015. https://doi.org/10.3115/v1/P15-2097 Ground truth for grammatical error correction metrics . In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th Internat...
2015 doi
-
[18]
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2001. Bleu: a method for automatic evaluation of machine translation. In ACL
2001
-
[19]
Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, and Alan W Black. 2018. Style transfer through back-translation. In Proc. ACL
2018
-
[20]
Improving language understanding by generative pre-training
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training
-
[21]
Sudha Rao and Joel Tetreault. 2018. https://doi.org/10.18653/v1/N18-1012 Dear sir or madam, may I introduce the GYAFC dataset: Corpus, benchmarks and metrics for formality style transfer . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association ...
2018 doi
-
[22]
Sravana Reddy and Kevin Knight. 2016. https://doi.org/10.18653/v1/W16-5603 Obfuscating gender in social media writing . In Proceedings of the First Workshop on NLP and Computational Social Science , pages 17--26, Austin, Texas. Association for Computational Linguistics
2016 doi
-
[23]
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. In Advances in neural information processing systems, pages 6830--6841
2017
-
[24]
Sandeep Subramanian, Guillaume Lample, Eric Michael Smith, Ludovic Denoyer, Marc'Aurelio Ranzato, and Y-Lan Boureau. 2018. http://arxiv.org/abs/1811.00552 Multiple-attribute text style transfer
2018 arXiv
-
[25]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf Attention is all you need . In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fe...
2017
-
[26]
Jesse Vig. 2019. http://arxiv.org/abs/1904.02679 Visualizing attention in transformer-based language representation models
2019 arXiv
-
[27]
Rob Voigt, David Jurgens, Vinodkumar Prabhakaran, Dan Jurafsky, and Yulia Tsvetkov. 2018. RtGender : A corpus for studying differential responses to gender. In Proc. LREC
2018
-
[28]
Williams and David Zipser
Ronald J. Williams and David Zipser. 1989. A learning algorithm for continually running fully recurrent neural networks
1989
-
[29]
Jingjing Xu, Xu SUN, Qi Zeng, Xiaodong Zhang, Xuancheng Ren, Houfeng Wang, and Wenjie Li. 2018. https://www.aclweb.org/anthology/P18-1090 Unpaired sentiment-to-sentiment translation: A cycled reinforcement learning approach . In Proceedings of the 56th Annual Meeting of the As...
2018
-
[30]
Cheng Yang, Maosong Sun, Xiaoyuan Yi, and Wenhao Li. 2018. https://www.aclweb.org/anthology/D18-1430 Stylistic C hinese poetry generation via unsupervised style disentanglement . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3...
2018
-
[31]
Ye Zhang, Nan Ding, and Radu Soricut. 2018 a . https://doi.org/10.18653/v1/N18-1138 SHAPED : Shared-private encoder-decoder for text style adaptation . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human ...
2018 doi
-
[32]
Zhirui Zhang, Shuo Ren, Shujie Liu, Jianyong Wang, Peng Chen, Mu Li, Ming Zhou, and Enhong Chen. 2018 b . Style transfer as unsupervised machine translation. CoRR, abs/1808.07894
2018 arXiv
-
[33]
Ganbin Zhou, Ping Luo, Rongyu Cao, Fen Lin, Bo Chen, and Qing He. 2017. Mechanism-aware neural machine for dialogue response generation. In Thirty-First AAAI Conference on Artificial Intelligence
2017
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.