{"id":"5d16bcec-90f4-44a1-8484-fd65d7b0f1ac","arxiv_id":"1908.04577","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Adding word-order and sentence-order reconstruction tasks to BERT pre-training improves performance on GLUE, SNLI, and SQuAD v1.1 benchmarks.","lead":"The authors add two pre-training tasks to BERT: reconstructing the order of shuffled word trigrams and predicting whether the second sentence is next, previous, or random. They report state-of-the-art scores on GLUE, SNLI, and SQuAD v1.1, though the biggest GLUE gain comes from a RoBERTa-initialized ensemble.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Attribution of gains to the structural objectives is uncontrolled: no BERT baseline is re-trained under identical data, compute, and fine-tuning, and the GLUE/SQuAD SOTA numbers mix ensembles, continued pre-training, and dev-only results.","rationale":"The reader's weakest assumption identifies the same insecure link: the paper claims that gains are 'entirely attributed' to the new objectives based on architecture identity, but not on controlled training conditions. I could not find a stronger internal inconsistency than the impossible XLNet F1 in Table 3, but that is a baseline-entry error; it affects data reliability without directly refuting the main mechanism. The ablation is suggestive and the SNLI result is a genuine test-set SOTA, so the paper should not be rejected. However, because the headline GLUE and SQuAD claims mix configurations and the causal attribution lacks a controlled BERT baseline, a conditional verdict remains appropriate: require the authors to disclose the ALICE submission, provide a same-protocol BERT baseline, and release checkpoints or code. No change to the reader's verdict is needed.","tokens_in":10202,"tokens_out":8615,"duration_ms":79410,"concrete_test":"Pre-train BERTBase under exactly the same protocol used for StructBERTBase: same Wikipedia+BookCorpus data, 40 epochs, 64 V100s, Adam 1e-4, same batch size, and the same fine-tuning grid as Table 4, then compare Table 4's BERTBase row. If the re-trained BERTBase matches or exceeds StructBERTBase on CoLA/MNLI/SQuAD, the structural objectives' marginal contribution is not established; if StructBERTBase still leads by the reported margins, the attribution concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central causal claim appears in Section 3.1.2: 'Since the network architecture of our model is identical to that of BERT, this improvement is entirely attributed to the new pre-training objectives.' This attribution is load-bearing because the entire contribution is the two new objectives, but the evidence never compares against a BERT baseline pre-trained under the same conditions. Table 4 lists BERTBase numbers that appear to be the published BERT values, not a re-run with StructBERTBase's 40-epoch schedule, 64-V100 setup, and fine-tuning grid; without that control, gains may reflect data, compute, or fine-tuning protocol rather than the structural objectives. The GLUE 'state-of-the-art' 89.0 in Table 1 is a StructBERTRoBERTa ensemble submitted under the name ALICE, so it cannot isolate the objectives' effect. The SQuAD 93.0 is a dev-set number (footnote 3 states no test result), and Table 3 contains an impossible baseline F1 (XLNet test EM 89.9/F1 85.0), indicating the reported comparisons are not reliable. The cleanest evidence is the ablation in Table 4, but even it lacks the controlled BERT row needed to support the phrase 'entirely attributed.'","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes StructBERT, an extension of BERT that adds two auxiliary pre-training objectives: a word structural objective that shuffles trigrams and asks the model to recover the original order, and a sentence structural objective that classifies a second sentence as next, previous, or random. The authors claim that these objectives improve downstream NLU performance and report state-of-the-art results on GLUE (89.0 average), SQuAD v1.1 (93.0 F1), and SNLI (91.7 accuracy). They also present an ablation study in which removing either structural objective degrades performance on several downstream tasks. The central claim is that the improvements are entirely attributable to the two new pre-training objectives because the architecture is identical to BERT.","tokens_in":10482,"tokens_out":4381,"duration_ms":43938,"significance":"If the central claim is supported, the contribution is valuable: the two objectives are architecture-neutral, simple to implement on top of BERT-style pre-training, and the ablation in Table 4 shows consistent degradation when either objective is removed. The design is intuitive and is likely to be adopted in later pre-training work. However, the headline state-of-the-art numbers are not cleanly attributable to the proposed objectives, because they come from configurations that mix continued pre-training from RoBERTa, ensembling, dev-only evaluation, and leaderboard submissions under a different model name. The internal ablation is the strongest evidence, but it does not include a matched BERT baseline trained under identical data, compute, and fine-tuning conditions. The paper would be publishable after the attribution claims are either made precise and supported by controlled experiments, or softened to match the actual evidence.","major_comments":[{"comment":"The sentence 'Since the network architecture of our model is identical to that of BERT, this improvement is entirely attributed to the new pre-training objectives' is not supported by the experiments. Identical architecture does not control for pre-training data, number of training steps, compute budget, or the fine-tuning protocol. Table 4 reports a BERTBase row, but nothing in the paper indicates that this row was produced by re-training BERT with StructBERT's 40-epoch, 64-V100 schedule and the same fine-tuning grid; it appears to be the published BERT numbers. Without a matched baseline, the observed gains could come from any of these factors rather than from the two structural objectives.","section":"Section 3.1.2 (SNLI)"},{"comment":"The abstract's claim of 'pushing the state-of-the-art on the GLUE benchmark to 89.0' is misleading because this number is the StructBERTRoBERTa ensemble submitted under the name ALICE, not the from-scratch StructBERTBase or StructBERTLarge single model. This configuration uses continued pre-training from released RoBERTa weights, an ensemble, and a different, much larger pre-training corpus. As a result, the 89.0 score cannot be used to establish the effectiveness of the two new objectives. Please report the exact configuration and provide a non-ensemble, from-scratch comparison if the claim is to be maintained.","section":"Section 3.1.1 and Table 1 (GLUE)"},{"comment":"The statement that the MNLI improvement 'is entirely attributed to our new training objectives' because the authors 'fine-tuned MNLI only on its in-domain data' is not a valid attribution. The comparison is against published models with different pre-training data, model sizes, and fine-tuning procedures. A gain over a leaderboard baseline cannot be ascribed to one modeling change without a controlled experiment that varies only the pre-training objectives.","section":"Section 3.1.1 (MNLI attribution)"},{"comment":"The abstract's SQuAD v1.1 'F1 score ... to 93.0' is a dev-set ensemble score, and the paper's own footnote 3 states that the test result was not available at submission time due to a CodaLab server crash. Presenting this as a state-of-the-art result is not supported. In addition, Table 3 contains an internally inconsistent baseline: XLNet(single+DA) has test EM 89.9 and test F1 85.0, which is impossible given that EM cannot exceed F1 for the same predictions and is also inconsistent with the dev row (88.9/94.5) and with published XLNet results. This casts doubt on the reliability of the table and should be corrected.","section":"Section 3.2, Table 3, and footnote 3 (SQuAD)"}],"minor_comments":[{"comment":"There is a typo: '64 Telsa V100 GPU cards' should be '64 Tesla V100 GPU cards.'","section":"Section 2.4"},{"comment":"The phrase 'Standford Natural Language inference' should be 'Stanford Natural Language Inference.'","section":"Section 3"},{"comment":"The description of the fine-tuning setup is inconsistent: RTE is listed both as a task fine-tuned after MNLI (STILTs) and as one of the 'other tasks' fine-tuned only on in-domain data. Please clarify the exact fine-tuning procedure for each task.","section":"Section 3.1.1"},{"comment":"The notation in Eq. (2) is confusing: the conditioning variables t1,...,tK appear to denote token identities, while the predicted events pos1=t1,...,posK=tK mix positions and token identities. Please rewrite the objective so that the relationship between tokens, positions, and the softmax predictions is unambiguous.","section":"Equation (2)"},{"comment":"The ablation reports averages over 8 runs but gives no variance or statistical significance. Adding standard deviations or a significance test would strengthen the claim that the degradation from removing each objective is meaningful.","section":"Table 4 and Section 3.3"},{"comment":"The sentence 'The task is general and yet challenging, and once is especially important for natural language generation' contains a typo: 'once' should likely be 'one' or 'once' should be replaced with a clearer phrase.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"This manuscript's core idea is reasonable and the ablation is the strongest part of the paper, but the abstract and several in-text passages make attribution claims that go beyond what the experiments show. The GLUE 89.0 result and the SQuAD 93.0 F1 are not clean evidence for the proposed objectives because of ensembling, continued pre-training, and dev-only evaluation. I would like to see either a matched from-scratch BERT baseline or a substantial softening of the causal claims before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, what's actually new: StructBERT adds two auxiliary pre-training tasks to BERT — a word-order objective that shuffles trigrams and asks the model to reconstruct them, and a three-way sentence-order objective that predicts whether the second sentence is next, previous, or random. That combination is new, and the paper does the right thing by ablating both objectives in Table 4. That ablation is the cleanest evidence in the paper, and it shows real, consistent gains from the sentence objective on sentence-pair tasks and from the word objective on CoLA. Credit where due: the idea is simple and the empirical story is mostly coherent.\n\nThe soft spots are about attribution and presentation. The claim that \"this improvement is entirely attributed to the new pre-training objectives\" (Section 3.1.2) is too strong. The BERT baselines in Table 4 are published numbers, not a BERT re-trained under StructBERT's exact data, compute, and fine-tuning protocol. Without that control, you can't rule out that some of the gap is the 40-epoch schedule or the fine-tuning grid. The headline GLUE 89.0 is a StructBERT-RoBERTa ensemble submitted under the name ALICE, not the base model; the abstract and intro don't make that clear. Similarly, the SQuAD 93.0 is dev-only; the test result never came back. And Table 3 has an impossible number — XLNet test F1 of 85.0 against an EM of 89.9 — which makes you wonder how carefully the tables were assembled. No code or checkpoints are released, so independent verification isn't possible from the preprint.\n\nNone of this kills the core idea. The ablation supports the usefulness of both objectives, and the direction is consistent with later work. But the paper overstates certainty, and the messy SOTA claims need to be untangled. A reviewer should ask for a same-protocol BERT baseline, clearer model naming, and a fix to that table.\n\nWho is this for? Anyone working on pre-training objectives. It's worth reading and worth a serious referee. It should not be desk-rejected, but it needs revision before I'd trust the causal story. Fix the attribution, release code, and the paper becomes a solid contribution.","headline":"StructBERT adds two simple auxiliary pre-training objectives with plausible gains, but the headline SOTA numbers mix ensembles and dev-only results, and the 'entirely attributed' claim outruns the evidence.","tokens_in":11029,"tokens_out":2807,"would_cite":true,"duration_ms":26498,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"StructBERT shows that pre-training a BERT-style model to reconstruct shuffled trigrams and to classify whether a paired sentence is next, previous, or random improves downstream language understanding, reaching an average GLUE score of…","keywords":["StructBERT","structural pre-training","word structural objective","sentence structural objective","GLUE benchmark","SQuAD","SNLI","natural language understanding"],"falsifier":"Pre-train the same BERT architecture from scratch on exactly the same data and compute budget but without the two structural objectives, and compare downstream GLUE, SNLI, and SQuAD scores; if that baseline matches or exceeds StructBERT, the objectives are not the cause. A second check: in the released ablation, if disabling either objective leaves all downstream scores unchanged, that objective carries no weight.","tokens_in":9991,"feed_emoji":"🧩","tokens_out":6833,"duration_ms":62524,"temperature":0.7,"pith_summary":"The paper sets out to show that a masked-language model can become better at understanding by explicitly training it on order at two scales. It extends BERT with a word-level objective that shuffles tokens inside randomly chosen trigrams and forces the model to reconstruct the original order, and a sentence-level objective that asks whether a second sentence is the next sentence, the previous sentence, or a random sentence. The authors report that these structural pre-training tasks improve results over BERT across GLUE, SNLI, and SQuAD v1.1, and that when added on top of RoBERTa the model reaches an average GLUE score of 89.0. The SQuAD F1 of 93.0 is reported on the dev set, with test-set results not yet returned at submission. The payoff, if the results hold, is evidence that word order and sentence direction are learnable, transferable signals that do not require any explicit syntactic annotation.","feed_headline":"Two order-prediction tasks push GLUE score to 89.0","feed_subtitle":"Reordering trigrams and predicting sentence direction lifts GLUE to 89.0, SQuAD F1 to 93.0, SNLI to 91.7.","key_machinery":"The load-bearing device is the pair of auxiliary pre-training objectives added to BERT's masked-language objective. The word structural objective shuffles tokens within randomly chosen trigrams and trains a softmax classifier on the encoder's output vectors to predict the original token at each shuffled position, with a trigram length K equal to 3 to balance reconstructability and robustness. The sentence structural objective pools the [CLS] output and trains a three-way classifier to distinguish next, previous, and random sentences, making the model aware of inter-sentence order in both directions. These two objectives are jointly trained with the masked-language objective in a single model, and the ablation study shows that removing either one degrades downstream performance on most tasks.","core_discovery":"On its own terms, the discovery is that two simple order-based objectives are sufficient to improve BERT. The word structural objective shuffles the three tokens within a randomly selected trigram, after the usual 15% token masking, and trains the encoder to place each shuffled token back in its original position. The sentence structural objective replaces BERT's binary next-sentence prediction with a three-way choice: the paired sentence is the next sentence, the previous sentence, or a random sentence from another document, with each case occurring one third of the time. The paper reports that these objectives improve downstream results over BERT on GLUE, SNLI, and SQuAD v1.1, and that continuing pre-training from RoBERTa with the same objectives yields a new state-of-the-art GLUE average of 89.0. The authors attribute the improvement entirely to the new pre-training objectives because the network architecture is identical to BERT.","pith_inferences":["A natural extension the paper does not test is applying the same two objectives at other granularities, such as phrases, clauses, or paragraphs, which could strengthen long-range discourse understanding.","The three-way sentence-direction task makes the model aware of discourse direction, so it may also benefit generation or summarization tasks that arrange sentences into coherent order, though no such experiment appears in the paper.","A stricter controlled test, pre-training BERT from scratch with and without the structural objectives under identical data, compute, and fine-tuning, would isolate the contribution of the objectives from the effects of continued pre-training and ensembling in the headline GLUE number."],"forward_implications":["On single-sentence tasks such as CoLA, the word structural objective appears to drive most of the gain, suggesting that grammatical acceptability benefits from explicit word-order reconstruction.","On sentence-pair tasks such as MNLI, QQP, and SQuAD, the sentence structural objective contributes the larger share, so bidirectional sentence-order pre-training transfers to relationship reasoning.","The objectives generalize beyond the original BERT: continuing pre-training from RoBERTa with the same two tasks produces the reported GLUE average of 89.0, beating the RoBERTa ensemble baseline.","Because the architecture is unchanged, any improvement is, per the paper, attributable to the pre-training tasks, which makes the objectives portable to other Transformer encoders."],"supporting_citations":[{"why":"Provides the BERT architecture and the masked-LM and next-sentence prediction baselines that StructBERT extends and compares against.","marker":"[6]"},{"why":"Supplies the word-order sensitivity motivation that the word structural objective is built on.","marker":"[8]"},{"why":"RoBERTa is the pretrained checkpoint from which StructBERTRoBERTa continues pre-training and the main ensemble baseline for the 89.0 GLUE result.","marker":"[15]"},{"why":"Defines the STILTs two-stage fine-tuning procedure applied to MRPC, RTE, and STS-B in the GLUE experiments.","marker":"[19]"},{"why":"Defines the GLUE benchmark and its evaluation server used for the state-of-the-art claim of 89.0.","marker":"[27]"},{"why":"Supplies the SNLI dataset on which the 91.7 accuracy is reported.","marker":"[3]"},{"why":"Supplies the SQuAD v1.1 dataset and leaderboard for the F1 93.0 claim.","marker":"[21]"},{"why":"XLNet is a published ensemble baseline that the StructBERTRoBERTa ensemble surpasses on the GLUE leaderboard.","marker":"[32]"}],"fun_headline_variants":["Word and sentence order predictions improve BERT","Shuffled trigram and sentence order tasks boost BERT","StructBERT: two order tasks lift GLUE to 89.0","Order-based pre-training beats BERT on GLUE and SQuAD","Predicting word and sentence order sharpens BERT"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim rests on the assumption that the measured gains come from the two new pre-training objectives rather than from the extra pre-training data, longer training, ensembling, or fine-tuning choices, because the model architecture is identical to BERT and the headline GLUE result starts from RoBERTa and uses an ensemble.","fun_headline_variants_meta":{"raw":{"variants":["Word and sentence order predictions improve BERT","Shuffled trigram and sentence order tasks boost BERT","StructBERT: two order tasks lift GLUE to 89.0","Order-based pre-training beats BERT on GLUE and SQuAD","Predicting word and sentence order sharpens BERT"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000332,"raw_usage":{"total_tokens":1846,"prompt_tokens":947,"completion_tokens":899,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":815}},"tokens_in":563,"tokens_out":899,"duration_ms":10768,"temperature":1.0,"reasoning_tokens":815,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:38:25.311329+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Pre-train the same BERT architecture from scratch on exactly the same data and compute budget but without the two structural objectives, and compare downstream GLUE, SNLI, and SQuAD scores; if that baseline matches or exceeds StructBERT, the objectives are not the cause. A second check: in the released ablation, if disabling either objective leaves all downstream scores unchanged, that objective carries no weight.","supporting_citations":[],"review_version":1}