REVIEW 4 major objections 6 minor 1 cited by
StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding
T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read StructBERT shows that pre-training a BERT-style model to reconstruct shuffled trigrams and to classify whether a paired sentence is next, previous, or random improves downstream language understanding, reaching an average GLUE score of…
desk verdict StructBERT adds two simple auxiliary pre-training objectives with plausible gains, but the headline SOTA numbers mix ensembles and dev-only results, and the 'entirely attributed' claim outruns the evidence. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing device is the pair of auxiliary pre-training objectives added to BERT's masked-language objective. The word structural objective shuffles tokens within randomly chosen trigrams and trains a softmax classifier on the encoder's output vectors to predict the original token at each shuffled position, with a trigram length K equal to 3 to balance reconstructability and robustness. The sentence structural objective pools the [CLS] output and trains a three-way classifier to distinguish next, previous, and random sentences, making the model aware of inter-sentence order in both directions. These two objectives are jointly trained with the masked-language objective in a single model, and the ablation study shows that removing either one degrades downstream performance on most tasks.
What would settle it
Pre-train the same BERT architecture from scratch on exactly the same data and compute budget but without the two structural objectives, and compare downstream GLUE, SNLI, and SQuAD scores; if that baseline matches or exceeds StructBERT, the objectives are not the cause. A second check: in the released ablation, if disabling either objective leaves all downstream scores unchanged, that objective carries no weight.
Extended reading notes
Core claim
On its own terms, the discovery is that two simple order-based objectives are sufficient to improve BERT. The word structural objective shuffles the three tokens within a randomly selected trigram, after the usual 15% token masking, and trains the encoder to place each shuffled token back in its original position. The sentence structural objective replaces BERT's binary next-sentence prediction with a three-way choice: the paired sentence is the next sentence, the previous sentence, or a random sentence from another document, with each case occurring one third of the time. The paper reports that these objectives improve downstream results over BERT on GLUE, SNLI, and SQuAD v1.1, and that continuing pre-training from RoBERTa with the same objectives yields a new state-of-the-art GLUE average of 89.0. The authors attribute the improvement entirely to the new pre-training objectives because the network architecture is identical to BERT.
Load-bearing premise
The central claim rests on the assumption that the measured gains come from the two new pre-training objectives rather than from the extra pre-training data, longer training, ensembling, or fine-tuning choices, because the model architecture is identical to BERT and the headline GLUE result starts from RoBERTa and uses an ensemble.
Editorial extensions
If this is right
- On single-sentence tasks such as CoLA, the word structural objective appears to drive most of the gain, suggesting that grammatical acceptability benefits from explicit word-order reconstruction.
- On sentence-pair tasks such as MNLI, QQP, and SQuAD, the sentence structural objective contributes the larger share, so bidirectional sentence-order pre-training transfers to relationship reasoning.
- The objectives generalize beyond the original BERT: continuing pre-training from RoBERTa with the same two tasks produces the reported GLUE average of 89.0, beating the RoBERTa ensemble baseline.
- Because the architecture is unchanged, any improvement is, per the paper, attributable to the pre-training tasks, which makes the objectives portable to other Transformer encoders.
Reading between the lines
- A natural extension the paper does not test is applying the same two objectives at other granularities, such as phrases, clauses, or paragraphs, which could strengthen long-range discourse understanding.
- The three-way sentence-direction task makes the model aware of discourse direction, so it may also benefit generation or summarization tasks that arrange sentences into coherent order, though no such experiment appears in the paper.
- A stricter controlled test, pre-training BERT from scratch with and without the structural objectives under identical data, compute, and fine-tuning, would isolate the contribution of the objectives from the effects of continued pre-training and ensembling in the headline GLUE number.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes StructBERT, an extension of BERT that adds two auxiliary pre-training objectives: a word structural objective that shuffles trigrams and asks the model to recover the original order, and a sentence structural objective that classifies a second sentence as next, previous, or random. The authors claim that these objectives improve downstream NLU performance and report state-of-the-art results on GLUE (89.0 average), SQuAD v1.1 (93.0 F1), and SNLI (91.7 accuracy). They also present an ablation study in which removing either structural objective degrades performance on several downstream tasks. The central claim is that the improvements are entirely attributable to the two new pre-training objectives because the architecture is identical to BERT.
Significance. If the central claim is supported, the contribution is valuable: the two objectives are architecture-neutral, simple to implement on top of BERT-style pre-training, and the ablation in Table 4 shows consistent degradation when either objective is removed. The design is intuitive and is likely to be adopted in later pre-training work. However, the headline state-of-the-art numbers are not cleanly attributable to the proposed objectives, because they come from configurations that mix continued pre-training from RoBERTa, ensembling, dev-only evaluation, and leaderboard submissions under a different model name. The internal ablation is the strongest evidence, but it does not include a matched BERT baseline trained under identical data, compute, and fine-tuning conditions. The paper would be publishable after the attribution claims are either made precise and supported by controlled experiments, or softened to match the actual evidence.
major comments (4)
- [Section 3.1.2 (SNLI)] The sentence 'Since the network architecture of our model is identical to that of BERT, this improvement is entirely attributed to the new pre-training objectives' is not supported by the experiments. Identical architecture does not control for pre-training data, number of training steps, compute budget, or the fine-tuning protocol. Table 4 reports a BERTBase row, but nothing in the paper indicates that this row was produced by re-training BERT with StructBERT's 40-epoch, 64-V100 schedule and the same fine-tuning grid; it appears to be the published BERT numbers. Without a matched baseline, the observed gains could come from any of these factors rather than from the two structural objectives.
- [Section 3.1.1 and Table 1 (GLUE)] The abstract's claim of 'pushing the state-of-the-art on the GLUE benchmark to 89.0' is misleading because this number is the StructBERTRoBERTa ensemble submitted under the name ALICE, not the from-scratch StructBERTBase or StructBERTLarge single model. This configuration uses continued pre-training from released RoBERTa weights, an ensemble, and a different, much larger pre-training corpus. As a result, the 89.0 score cannot be used to establish the effectiveness of the two new objectives. Please report the exact configuration and provide a non-ensemble, from-scratch comparison if the claim is to be maintained.
- [Section 3.1.1 (MNLI attribution)] The statement that the MNLI improvement 'is entirely attributed to our new training objectives' because the authors 'fine-tuned MNLI only on its in-domain data' is not a valid attribution. The comparison is against published models with different pre-training data, model sizes, and fine-tuning procedures. A gain over a leaderboard baseline cannot be ascribed to one modeling change without a controlled experiment that varies only the pre-training objectives.
- [Section 3.2, Table 3, and footnote 3 (SQuAD)] The abstract's SQuAD v1.1 'F1 score ... to 93.0' is a dev-set ensemble score, and the paper's own footnote 3 states that the test result was not available at submission time due to a CodaLab server crash. Presenting this as a state-of-the-art result is not supported. In addition, Table 3 contains an internally inconsistent baseline: XLNet(single+DA) has test EM 89.9 and test F1 85.0, which is impossible given that EM cannot exceed F1 for the same predictions and is also inconsistent with the dev row (88.9/94.5) and with published XLNet results. This casts doubt on the reliability of the table and should be corrected.
minor comments (6)
- [Section 2.4] There is a typo: '64 Telsa V100 GPU cards' should be '64 Tesla V100 GPU cards.'
- [Section 3] The phrase 'Standford Natural Language inference' should be 'Stanford Natural Language Inference.'
- [Section 3.1.1] The description of the fine-tuning setup is inconsistent: RTE is listed both as a task fine-tuned after MNLI (STILTs) and as one of the 'other tasks' fine-tuned only on in-domain data. Please clarify the exact fine-tuning procedure for each task.
- [Equation (2)] The notation in Eq. (2) is confusing: the conditioning variables t1,...,tK appear to denote token identities, while the predicted events pos1=t1,...,posK=tK mix positions and token identities. Please rewrite the objective so that the relationship between tokens, positions, and the softmax predictions is unambiguous.
- [Table 4 and Section 3.3] The ablation reports averages over 8 runs but gives no variance or statistical significance. Adding standard deviations or a significance test would strengthen the claim that the degradation from removing each objective is meaningful.
- [Section 4.2] The sentence 'The task is general and yet challenging, and once is especially important for natural language generation' contains a typo: 'once' should likely be 'one' or 'once' should be replaced with a clearer phrase.
Circularity Check
No circularity found: the structural pre-training objectives are validated on external benchmarks and the ablation directly toggles the proposed objectives.
full rationale
The paper's central derivation is not circular under the defined standards. The proposed word structural objective and sentence structural objective are auxiliary self-supervised training signals defined on unlabeled text (Eq. 2 and the three-way sentence sampling), not functions of downstream benchmark labels. Claims of state-of-the-art performance are evaluated on external test sets (GLUE evaluation server, SNLI test set, SQuAD v1.1), and the ablation in Table 4 removes one objective at a time while holding the architecture fixed, so the comparison directly tests the contribution of the stated objectives. No parameter is fitted to a benchmark subset and then renamed as a prediction. The paper cites prior work for the BERT architecture and training data, but these are independent external resources, not a self-citation chain, and the authors do not invoke a uniqueness theorem or an ansatz imported from their own prior work. The skeptical concern that the phrase 'this improvement is entirely attributed to the new pre-training objectives' (Section 3.1.2) lacks a fully controlled BERT baseline re-trained under identical data and compute is a question of experimental attribution and internal validity, not circularity; the reported gains could be confounded by compute, data, or fine-tuning protocol, but the objectives are not defined in terms of the benchmark outcomes. The impossible XLNet test F1 value in Table 3 is a correctness/typographical risk, not a circular-reasoning defect. Accordingly, no circular step can be quoted or exhibited, and the circularity score is 0.
Assumptions & free parameters
free parameters (5)
- Trigram shuffling length K =
3
- Shuffled trigram percentage =
5%
- Sentence objective sampling probabilities =
1/3 each for next, previous, random
- Objective loss weighting =
Equal weights
- Downstream fine-tuning grid =
Batch sizes 16/24/32, learning rates 2e-5/3e-5/5e-5, epochs 2/3, dropout 0.05/0.1
assumptions (6)
- domain assumption BERT architecture, embeddings, and WordPiece tokenization are taken as given from Devlin et al. (2018).
- domain assumption Pre-training on English Wikipedia (2,500M words) and BookCorpus is sufficient for downstream transfer.
- ad hoc to paper The reported improvement is entirely attributable to the two new pre-training objectives.
- domain assumption The official GLUE leaderboard and downstream dev and test labels are reliable ground truth.
- domain assumption RoBERTa's released checkpoint is a valid initialization for continued pre-training with the new objectives.
- domain assumption Elman's word-order sensitivity result transfers to Transformer pre-training.
Cite this review
Pith. "Pith review of StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding." pith.science (2026). https://pith.science/paper/S5TKTKM7
@misc{pith2026190804577,
author = {Pith},
title = {Pith review of: StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding},
year = {2026},
howpublished = {\url{https://pith.science/paper/S5TKTKM7}},
note = {Machine review of arXiv:1908.04577}
}
read the original abstract
Recently, the pre-trained language model, BERT (and its robustly optimized version RoBERTa), has attracted a lot of attention in natural language understanding (NLU), and achieved state-of-the-art accuracy in various NLU tasks, such as sentiment classification, natural language inference, semantic textual similarity and question answering. Inspired by the linearization exploration work of Elman [8], we extend BERT to a new model, StructBERT, by incorporating language structures into pre-training. Specifically, we pre-train StructBERT with two auxiliary tasks to make the most of the sequential order of words and sentences, which leverage language structures at the word and sentence levels, respectively. As a result, the new model is adapted to different levels of language understanding required by downstream tasks. The StructBERT with structural pre-training gives surprisingly good empirical results on a variety of downstream tasks, including pushing the state-of-the-art on the GLUE benchmark to 89.0 (outperforming all published models), the F1 score on SQuAD v1.1 question answering to 93.0, the accuracy on SNLI to 91.7.
Figures
Forward citations
Cited by 1 Pith paper
-
Can bidirectional encoder become the ultimate winner for downstream applications of foundation models?
A review of bidirectional encoder models (BERT and variants) and their performance on GLUE and SQuAD relative to one-way generative models.
Reference graph
Works this paper leans on
-
[1]
The fifth pascal recognizing textual entailment challenge
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. In TAC, 2009
work page 2009
-
[2]
A bottom-up approach to sentence ordering for multi-document summarization
Danushka Bollegala, Naoaki Okazaki, and Mitsuru Ishizuka. A bottom-up approach to sentence ordering for multi-document summarization. Information processing & management, 46(1):89–109, 2010
work page 2010
-
[3]
A large annotated corpus for learning natural language inference
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326, 2015
arXiv 2015
-
[4]
Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055, 2017
arXiv 2017
-
[5]
Xinchi Chen, Xipeng Qiu, and Xuanjing Huang. Neural sentence ordering. arXiv preprint arXiv:1607.06952, 2016
arXiv 2016
-
[6]
Bert: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018
arXiv 2018
-
[7]
Automatically constructing a corpus of sentential paraphrases
William B Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. InProceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005
2005
-
[8]
Finding structure in time
Jeffrey L Elman. Finding structure in time. Cognitive science, 14(2):179–211, 1990
1990
Show all 35 references
-
[9]
A comparison of neural models for word ordering
Eva Hasler, Felix Stahlberg, Marcus Tomalin, Adri de Gispert, and Bill Byrne. A comparison of neural models for word ordering. arXiv preprint arXiv:1708.01809, 2017
2017 arXiv
-
[10]
Gaussian error linear units (gelus)
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016
2016 arXiv
-
[11]
Long short-term memory
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997
1997
-
[12]
Spanbert: Improving pre-training by representing and predicting spans
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. Spanbert: Improving pre-training by representing and predicting spans. arXiv preprint arXiv:1907.10529, 2019
1907 arXiv
-
[13]
The winograd schema challenge
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning , 2012
2012
-
[14]
Multi-task deep neural networks for natural language understanding
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. arXiv preprint arXiv:1901.11504, 2019
1901 arXiv
-
[15]
RoBERTa: A robustly optimized BERT pretraining approach.arXiv preprint arXiv:1907.11692, 2019
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach.arXiv preprint arXiv:1907.11692, 2019
1907 arXiv
-
[16]
Learned in translation: Contextualized word vectors
Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. Learned in translation: Contextualized word vectors. In Advances in Neural Information Processing Systems , pages 6294–6305, 2017
2017
-
[17]
Recurrent neural network based language model
Tomáš Mikolov, Martin Karafiát, Lukáš Burget, JanˇCernock`y, and Sanjeev Khudanpur. Recurrent neural network based language model. In Eleventh annual conference of the international speech communication association , 2010
2010
-
[18]
Deep contextualized word representations
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv:1802.05365, 2018
2018 arXiv
-
[19]
Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks
Jason Phang, Thibault Févry, and Samuel R Bowman. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088, 2018. 8
2018 arXiv
-
[20]
Improving language under- standing by generative pre-training
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. URL https://s3-us-west-2. amazonaws. com/openai-assets/research- covers/languageunsupervised/language understanding paper . pdf, 2018
2018
-
[21]
Squad: 100,000+ questions for machine comprehension of text
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016
2016 arXiv
-
[22]
Snorkel: Rapid training data creation with weak supervision
Alexander Ratner, Stephen H Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. Snorkel: Rapid training data creation with weak supervision. Proceedings of the VLDB Endowment , 11(3):269–282, 2017
2017
-
[23]
Building applied natural language generation systems
Ehud Reiter and Robert Dale. Building applied natural language generation systems. Natural Language Engineering, 3(1):57–87, 1997
1997
-
[24]
Word ordering without syntax
Allen Schmaltz, Alexander M Rush, and Stuart M Shieber. Word ordering without syntax. arXiv preprint arXiv:1604.08633, 2016
2016 arXiv
-
[25]
Recursive deep models for semantic compositionality over a sentiment treebank
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language process...
2013
-
[26]
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems , pages 5998–6008, 2017
2017
-
[27]
Glue: A multi-task benchmark and analysis platform for natural language understanding
Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018
2018 arXiv
-
[28]
Neural network acceptability judgments.arXiv preprint arXiv:1805.12471, 2018
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments.arXiv preprint arXiv:1805.12471, 2018
2018 arXiv
-
[29]
A broad-coverage challenge corpus for sentence understanding through inference
Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426, 2017
2017 arXiv
-
[30]
Google’s neural machine translation system: Bridging the gap between human and machine translation
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016
2016 arXiv
-
[31]
Enhancing pre-trained language representations with rich knowledge for machine reading comprehension
An Yang, Quan Wang, Jing Liu, Kai Liu, Yajuan Lyu, Hua Wu, Qiaoqiao She, and Sujian Li. Enhancing pre-trained language representations with rich knowledge for machine reading comprehension. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguist...
2019
-
[32]
Xlnet: Generalized autoregressive pretraining for language understanding
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237, 2019
1906 arXiv
-
[33]
Qanet: Combining local convolution with global self-attention for reading comprehension
Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V Le. Qanet: Combining local convolution with global self-attention for reading comprehension. arXiv preprint arXiv:1804.09541, 2018
2018 arXiv
-
[34]
Discriminative syntax-based word ordering for text generation
Yue Zhang and Stephen Clark. Discriminative syntax-based word ordering for text generation. Computational linguistics, 41(3):503–538, 2015
2015
-
[35]
Aligning books and movies: Towards story-like visual explanations by watching movies and reading books
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer ...
2015
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.