{"id":"4e610315-e17b-482e-be0a-06602616c35d","arxiv_id":"1908.09368","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A delete-retrieve-generate style transfer system using BERT attention for deleting style words and a GPT-based transformer for generation outperforms prior systems on five non-parallel style transfer datasets.","lead":"The authors build a three-step sentence rewriting system: first, a language model finds and removes the words that carry the original style; then relevant words for the target style are retrieved; finally, a text generator writes a new sentence in the target style. Human raters preferred the new system over prior state-of-the-art models on five datasets for sentiment, gender, and political slant transfer.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline claim is undercut by the paper's own tables: on CAPTIONS, D&R beats B-GST on human style match (47.75% vs 34%), and on POLITICAL, B-GST's automatic style accuracy is below BT (71.2 vs 88.0) with no human style judgement.","rationale":"The reader's weakest assumption concerns whether the Delete Transformer's selected head-layer attention weights correctly isolate style attributes. That is a plausible technical fragility, but the more decisive problem is that the central empirical claim is already put in doubt by the paper's own tables. Table 2 shows D&R preferred over B-GST on the Captions style-match axis, and Table 5 shows B-GST below BT on Political automatic style accuracy, while no human style evaluation is reported for Political or Gender. These are not hidden assumptions: they are the reported evidence failing to support the headline. The reader's conditional verdict is still appropriate, but the condition needs to be more specific: the style-axis claims for Captions and Political must be backed by style-focused human evaluation, and the abstract must be narrowed if those evaluations do not favor B-GST. I therefore keep a conditional posture while disagreeing that the weakest point is the Delete Transformer's internal mechanism.","tokens_in":12056,"tokens_out":8933,"duration_ms":97596,"concrete_test":"Run MTurk or expert pairwise evaluation on a random sample of Political and Captions test sentences, asking specifically for target-style match, with at least 3 independent judgments per item and reported inter-annotator agreement and bootstrap 95% CIs. Pre-specify that B-GST is declared superior only if its style-match preference proportion over D&R/BT exceeds 0.5 and the CI excludes 0.5. Also re-analyze the released pairwise judgments for the Captions style axis; if D&R remains preferred, the abstract's 'outperform across 5 datasets' claim must be revised or removed.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The abstract and Section 4.3 claim that the models outperform state-of-the-art systems across all five datasets on sentiment, gender, and political slant transfer. The paper's own evaluation data do not establish the style-transfer half of this claim. In Table 2, on CAPTIONS target style match, D&R is preferred over B-GST 47.75% vs 34% (18.5% 'None'), so B-GST is not the better style-transfer system on that axis. Section 4.1 states that for POLITICAL and GENDER annotators rated only content and fluency, explicitly because target style strength is hard for MTurkers to judge. The only style signal for POLITICAL is Table 5's automatic target-style accuracy, where B-GST scores 71.2 versus BT's 88.0. Thus 'political slant transfer' superiority is not demonstrated and is contradicted by the paper's own automatic metric; the large BLEU/fluency advantages are compatible with B-GST copying source content rather than changing style. This is not merely an unverified design assumption but a mismatch between the central claim and the reported measurements.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a non-parallel text style transfer system built on a Delete-Retrieve-Generate (DRG) framework. The Delete component uses attention weights from a fine-tuned BERT-based classifier (Delete Transformer) to identify and remove style attributes; the Retrieve component selects a similar target-style sentence via TF-IDF; the Generate component is a decoder-only GPT-based Transformer (GST) fine-tuned with a reconstruction objective, with two variants (B-GST and G-GST). The authors report human and automatic evaluations on Yelp, Amazon, Captions, Political, and Gender datasets, claiming state-of-the-art performance, and they propose GLEU as an automatic evaluation metric.","tokens_in":12368,"tokens_out":5848,"duration_ms":53118,"significance":"The idea of using attention from a single head-layer of a Transformer as style-importance scores is novel and potentially useful, and the use of a pre-trained GPT model for generation is a sensible extension of the DRG framework. The authors make code and data available, and they include a human evaluation of the delete mechanism. However, the significance is currently limited by evaluation weaknesses: key claims of style-transfer superiority are not consistently supported by the reported data, and the statistical basis of the human evaluation is not documented.","major_comments":[{"comment":"The abstract and Section 4.3 claim that the models 'outperform state-of-art systems across 5 datasets on sentiment, gender and political slant transfer.' Table 2 shows that on CAPTIONS, D&R is preferred over B-GST on target style match (47.75% vs 34%), so B-GST is not superior on the style-transfer axis for that dataset. For POLITICAL and GENDER, Section 4.1 states that annotators only rated content and fluency because target style strength is hard to judge; the only style signal for POLITICAL is the automatic accuracy in Table 5, where B-GST scores 71.2% versus BT's 88.0%. Thus the paper's own evidence contradicts the style-transfer superiority part of the claim, and the claim should be revised or additional style evaluations supplied.","section":"Abstract and Section 4.3, Table 2"},{"comment":"The human evaluation results in Tables 2 and 3 are reported as pairwise preference percentages with no information about the number of annotators, confidence intervals, or significance tests. For example, the 34% vs 47.75% difference on CAPTIONS style match cannot be assessed for reliability without such information. Please report annotator counts and at least approximate confidence intervals or a significance test for the main comparisons.","section":"Section 4.1, Tables 2 and 3"},{"comment":"The abstract states that GLEU 'compare[s] better with human ratings than the predominantly used BLEU score,' but Section 4.3 explicitly says 'a detailed statistical correlation study is left for future work.' This is an unsupported claim; it should be backed by a correlation analysis (e.g., system-level or segment-level Spearman correlation with human judgments) or presented as a hypothesis rather than a finding.","section":"Abstract and Section 4.3"},{"comment":"The Delete Transformer is used both to select the head-layer pair for deletion (Eq. 5) and, during inference, to re-rank beam outputs by target-style match (Section 2.3.2). This means the automatic style accuracy in Tables 4 and 5 measures the model against the same classifier that selects its outputs, which can inflate style scores. Additionally, the deletion proportion gamma and smoothing parameter lambda are tuned per dataset, but no sensitivity analysis is provided, and the validity of the single head-layer assumption is only tested on 200 sentences. Please include a sensitivity analysis and clarify the consequences of the beam-selection procedure for the automatic style metric.","section":"Sections 2.1.1 and 2.3.2"},{"comment":"The FastText style classifiers used for automatic style accuracy have accuracies of 80% on CAPTIONS and 82% on GENDER (Section 4.2). For a binary task, 80% accuracy means the classifier is wrong on 20% of examples, so automatic style accuracy on these datasets is a noisy measure. Since the POLITICAL and GENDER style comparisons rely on this metric, its limitations should be discussed and, where possible, supplemented with human style judgments.","section":"Section 4.2"}],"minor_comments":[{"comment":"The paper repeatedly uses 'state-of-art' instead of 'state-of-the-art' (e.g., abstract, Section 1, Section 3.2); please fix this typo consistently.","section":"Throughout"},{"comment":"Table 6 refers to colored attributes, but the printed text does not show colors; use bold or underline to mark attributes.","section":"Table 6"},{"comment":"Equation (2) uses softmax over w to define α_{h,l}(w), which is a distribution over tokens; if this normalized value is used as an importance score, please state this explicitly and clarify that the score is not a raw attention weight.","section":"Equation (2)"},{"comment":"Section 2.2 states that TF-IDF retrieval gives 'best retrieval results' without presenting a comparison; please add a small table or a quantitative justification for this choice.","section":"Section 2.2"},{"comment":"Figure 2 is referenced in Section 4.3 but the actual plot is not visible in the manuscript text; please ensure the figure is included and legible.","section":"Section 4.3, Figure 2"}],"recommendation":"major_revision","confidential_remarks":"This is a borderline case. The approach has merit, but the abstract's claim of state-of-the-art style transfer is stronger than what the reported measurements support, particularly on CAPTIONS (style match) and POLITICAL (automatic style accuracy). The human evaluation also lacks basic statistical information. These issues are addressable through revision, so I would not reject the paper outright, but the authors need to align their claims with the evidence and strengthen the evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: the paper builds a real, usable style-transfer system by combining BERT-attention deletion, GPT-based generation, and beam re-ranking with the delete model, and it releases code and data. The main claim in the abstract, however, goes beyond what its own tables show, and the evaluation is thinner than the prose suggests.\n\nThe genuinely new piece is the Delete Transformer: using a specific BERT head-layer pair's attention weights to mask style tokens, chosen by a validation criterion, and then re-ranking generation beams with that same delete model. That's a sensible adaptation, and the authors' MTurk check that it deletes all style attributes 89% of the time versus D&R's 67% is a meaningful comparison. The two generator variants, B-GST and G-GST, are a clean way to separate blind generation from guided generation, even if G-GST underperforms due to retrieval.\n\nHuman evals show B-GST is strongly preferred over D&R and BT on content and fluency for most datasets. That alone is a practical contribution. But the abstract's 'outperform state-of-art systems across 5 datasets' is contradicted by their own numbers. On CAPTIONS, D&R beats B-GST on human style match (47.75% vs 34%). On POLITICAL, annotators didn't rate style at all, and the automatic style accuracy is lower than BT's (71.2 vs 88.0). So the style-transfer half of the headline claim is not demonstrated. The authors do acknowledge GLEU correlation is left for future work, which is honest, but the central claim still needs qualification.\n\nThe evaluation also lacks the basics: no annotator counts, no confidence intervals, no significance tests. And the deletion head-layer selection and threshold gamma are tuned per dataset with no sensitivity analysis, so we don't know how fragile the delete step is. These are not fatal to the system, but they are problems with the strength of the claims.\n\nWho is this for? Anyone working on non-parallel style transfer who wants a strong baseline and a worked example of using pretrained Transformers inside DRG. It deserves a serious referee: the artifacts are real, the ideas are specific, and the weaknesses are fixable in revision. I'd send it to review but ask for rewritten claims, statistical rigor, and at least a small sensitivity study.","headline":"A genuinely useful BERT-deletion + GPT-generation combination for non-parallel style transfer, but the headline claim overstates what the paper's own tables show.","tokens_in":12845,"tokens_out":2064,"would_cite":true,"duration_ms":20143,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A Transformer-based delete-and-generate pipeline outperforms prior style-transfer systems on five datasets, and the paper proposes GLEU as a metric that tracks human preferences better than BLEU.","keywords":["text style transfer","delete-retrieve-generate","attention-based attribute deletion","generative style transformer","GLEU metric","sentiment transfer","gender transfer","political slant transfer"],"falsifier":"Run B-GST on the Yelp sentiment task while replacing the chosen head-layer attention scores with random token scores that delete the same fraction of tokens; if human raters do not noticeably prefer the attention-based deletions for content preservation and style match, then the claim that attention identifies style attributes is not doing the work.","tokens_in":11906,"feed_emoji":"🎭","tokens_out":9281,"duration_ms":85016,"temperature":0.7,"pith_summary":"This paper argues that text style transfer without parallel corpora can be done reliably by separating style words from content with a Transformer's own attention weights, then regenerating the sentence with a pre-trained generative Transformer. The authors build a Delete Transformer, which scores each token by how much attention the classifier's [CLS] token pays to it, deletes the top-scoring fraction, and retrieves target-style material from a corpus. A Generative Style Transformer then rewrites the remaining content in the target style, either blind to specific target attributes (B-GST) or guided by retrieved ones (G-GST). In human evaluations on sentiment, gender, and political-slant datasets, B-GST is preferred over the previous best systems for content, fluency, style match, and overall quality, and the paper proposes GLEU as an automatic metric that aligns with human judgments better than BLEU.","feed_headline":"Transformer pipeline wins style transfer on five datasets","feed_subtitle":"Human raters prefer its rewrites over prior systems for content, fluency, and style, without parallel data.","key_machinery":"The load-bearing components are three. The Delete Transformer is a pre-trained Transformer text classifier whose per-head, per-layer attention weights onto its [CLS] token serve as importance scores for each input token; the paper selects the single head-layer pair that, after removing the top $\\gamma|x|$ tokens, most raises the classifier's confusion about the sentence's style (Eqs. 2 and 5). The Generative Style Transformer (GST) is a decoder-only Transformer language model, pre-trained on a large text corpus, that takes content, with or without retrieved target attributes, and the target style as input and generates the output sentence token by token. The two variants, B-GST (blind to target attributes) and G-GST (guided by retrieved attributes), share this machinery but differ in whether a Retrieve step supplies explicit target attribute tokens. A third component, the GLEU metric, scores outputs by n-gram overlap among source, reference, and generated sentence, letting it penalize wrongly changed content words while rewarding successful style edits.","core_discovery":"The central discovery is that a Transformer classifier's per-head attention onto its [CLS] token can act as a deletion policy for style transfer. The Delete Transformer identifies style-bearing words by choosing the head-layer pair whose attention, when the highest-attended tokens are removed, most confuses the classifier about the sentence's style; the remaining tokens are treated as content. A decoder-only Transformer language model, pre-trained on a large unlabeled corpus, is then fine-tuned to reconstruct sentences from their own content, or from content plus retrieved or noisy attributes, using the target style as a prompt. The paper reports that in human evaluations on five datasets covering sentiment, gender, and political slant, its blind variant B-GST is preferred over the previous best systems for content preservation, fluency, style match, and overall quality, and that GLEU, an n-gram metric that compares source, reference, and output, correlates better with those human judgments than BLEU does.","pith_inferences":["If the head-layer attention selection generalizes, the same delete mechanism could be applied to formality, toxicity, or dialect rewriting without retraining a classifier on each new style pair.","The gap between B-GST and G-GST points to the Retrieve component as the bottleneck; a learned attribute predictor or retrieval over latent attribute vectors could restore controllability without sacrificing quality.","GLEU's fit with human ratings on these five datasets suggests style-transfer evaluation can follow grammatical-error-correction practice; a systematic correlation study across more models and human-judgement setups could settle which metric should be standard.","The selected head-layer pair and the deletion-rate parameter are tuned per dataset; a sensitivity analysis of these choices would tell whether the method's gains are robust or concentrated in a narrow operating point."],"forward_implications":["On the Yelp, Amazon, and Captions datasets, human raters prefer B-GST over the previous Delete-and-Retrieve system for content, fluency, style match, and overall quality.","On the Political and Gender datasets, human raters prefer B-GST over back-translation for content and fluency.","Because B-GST keeps output length closely correlated with the source sentence length, the generated rewrites are less prone to the drift seen in earlier systems.","GLEU scores for B-GST are consistently the highest across datasets, which the paper reads as evidence that the model makes only necessary edits.","G-GST offers controllability: at inference the user can supply desired target attributes directly, bypassing the Retrieve step, which is useful when the target corpus lacks similar sentences."],"supporting_citations":[{"why":"Supplies the Delete-Retrieve-Generate framework and the D&R and DeleteOnly baselines and datasets that this paper extends and compares against.","marker":"Li et al. (2018)"},{"why":"Defines the Transformer attention mechanism with Q/K/V that the Delete Transformer and GST are built on.","marker":"Vaswani et al. (2017)"},{"why":"Provides the pre-trained Transformer classifier whose attention weights the Delete Transformer repurposes for attribute deletion.","marker":"Devlin et al. (2018)"},{"why":"Introduces input reduction, the deletion procedure the paper adapts with attention-based importance scores.","marker":"Feng et al. (2018)"},{"why":"Supplies the pre-trained decoder-only Transformer language model that GST fine-tunes for generation.","marker":"Radford et al."},{"why":"Provides the back-translation baseline and the Political and Gender datasets that B-GST is compared against.","marker":"Prabhumoye et al. (2018)"},{"why":"Defines GLEU, which the paper proposes as an automatic evaluation metric for style transfer.","marker":"Napoles et al. (2015)"}],"fun_headline_variants":["Attention-based deletion powers style transfer without parallel data","Transformer beats state-of-art in five style transfer tasks","Deleting style via attention improves text rewriting","New metric GLEU aligns better with human style ratings","Zero-parallel style transfer via attention-guided deletion"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the attention weights from one chosen head-layer pair of the style classifier mark style words, not content words; if that fails, the delete step either throws away content or leaves style in place, and the pipeline cannot do both.","fun_headline_variants_meta":{"raw":{"variants":["Attention-based deletion powers style transfer without parallel data","Transformer beats state-of-art in five style transfer tasks","Deleting style via attention improves text rewriting","New metric GLEU aligns better with human style ratings","Zero-parallel style transfer via attention-guided deletion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000168,"raw_usage":{"total_tokens":1228,"prompt_tokens":880,"completion_tokens":348,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":496,"completion_tokens_details":{"reasoning_tokens":274}},"tokens_in":496,"tokens_out":348,"duration_ms":3928,"temperature":1.0,"reasoning_tokens":274,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:13:28.287780+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run B-GST on the Yelp sentiment task while replacing the chosen head-layer attention scores with random token scores that delete the same fraction of tokens; if human raters do not noticeably prefer the attention-based deletions for content preservation and style match, then the claim that attention identifies style attributes is not doing the work.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the back-translation baseline and the Political and Gender datasets that B-GST is compared against."}],"review_version":1}