{"id":"1510a92a-60f3-43a6-9087-5cbae7e09f1e","arxiv_id":"2411.18885","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A soft syntactic regularizer that enforces orthogonality between sentence-span representations improves syntactic generalization and out-of-distribution robustness in transformer LMs without architectural changes.","lead":"TreeReg adds a differentiable penalty to transformer language models that encourages hidden states to treat syntactic phrases as context-independent units. Models trained with this penalty show better syntactic generalization and sometimes better out-of-distribution perplexity, without any change to the network architecture.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline SG/PTB gains may be inflated because the layer/head configuration for the main BLLIP-LG experiment was chosen by maximizing the same SG and PTB test metrics that are reported as results in Table 2.","rationale":"The reader's formal weakest_assumption concerns the adequacy of the last-token hidden state as a span representation, which is a real theoretical risk but is strongly mitigated by the empirical success and by the Allen-Zhu and Li citation. I do not think that is the most load-bearing issue. The reader's rationale also explicitly flags test-set-driven hyperparameter selection, and that is the concern I find most serious: Appendix G shows the main BLLIP-LG configuration was chosen by optimizing the same SG and PTB test metrics that later appear as headline results. This affects the magnitude of the central claims, not just their interpretation. I still do not think the paper should be rejected: the method is transparent, code is released, and several controls (randomized parses, cosine-similarity loss, ablations on parsed-data amount) indicate that the auxiliary parse signal, rather than generic regularization, drives part of the effect. But the reported numbers should be treated as conditional on a selection process that used the evaluation sets. A validation-based re-selection is a feasible, concrete check that would settle whether the headline margins survive. I therefore keep the reader's CONDITIONAL verdict rather than moving to ACCEPT or REJECT.","tokens_in":18986,"tokens_out":10357,"duration_ms":102748,"concrete_test":"Re-run the Appendix G layer/head sweep using only a validation split for selection: for example, choose the configuration that maximizes SG accuracy on a held-out subset of SyntaxGym items and minimizes perplexity on the BLLIP-LG development set, then freeze that configuration before computing the final Table 2 numbers on the SG test suite and the PTB test set. If the chosen configuration's SG gain over Base LM is materially smaller than the reported 8.1 points, or if a different configuration wins on validation, then the headline gains are partly selection artifacts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The main from-scratch result (Section 5.1, Table 2) uses TREE REG at layer 12 on 2 of 8 attention heads. Appendix G describes how this configuration was chosen: the authors varied the layer over [2,4,6,8,10,12,14] and the number of heads over [2,4,8], and selected the configuration that gives the highest overall SyntaxGym performance and the lowest PTB perplexity. These are the same test sets used to report the paper's headline improvements (an 8.1-point SG gain and a 9.2% PTB perplexity reduction). Selecting hyperparameters by optimizing the evaluation metrics on the evaluation sets makes the reported gains optimistic: with 21 configurations examined, the best one will look better than the method's typical performance even under pure noise. The concern is not that TREE REG has no effect; the randomized-parse and cosine-similarity ablations in the text provide evidence of a real syntactic signal. The issue is that the magnitude of the central quantitative claims, especially the SG gain and the PTB OOD improvement, is not established by the reported protocol. A validation-based selection could shrink the headline margins substantially, and the paper does not report how the chosen configuration would perform if selected on held-out data.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TreeReg, an auxiliary loss that converts constituency bracketing decisions from silver parses into differentiable orthogonality constraints on transformer hidden states. The loss is added to the standard LM objective without architectural changes. The authors report improved syntactic generalization on BLiMP and SyntaxGym, lower out-of-distribution perplexity on PTB, better sample efficiency, and gains in continued pretraining of Sheared Llama and fine-tuning on MultiNLI, including a randomized-parse control that supports the causal role of the parse supervision.","tokens_in":19257,"tokens_out":10460,"duration_ms":89141,"significance":"If the quantitative results hold, this is a useful and novel contribution: it provides a simple, computationally light way to inject hierarchical inductive bias into transformers without modifying the architecture or inference procedure, in contrast to prior approaches that alter attention or add parameters. The randomized-parse experiment is a strong control, and the code release supports reproducibility. The main reservation is that the headline magnitudes are selected on the same test sets used for reporting, so the size of the central gains is not yet established; nevertheless, the method's effectiveness is supported across several independent settings.","major_comments":[{"comment":"The central quantitative claims in Table 2 are based on a configuration (layer 12, 2 of 8 attention heads) that was selected by maximizing SyntaxGym performance and minimizing PTB perplexity on the same test sets used to report those numbers. Because 21 configurations were examined (layers 2,4,6,8,10,12,14 times heads 2,4,8), the reported 8.1-point SG gain and 9.2% PTB improvement are best-of-sweep values and are therefore optimistic. The authors should either choose the configuration on a held-out validation split (e.g., a validation portion of BLLIP-LG or a separate validation benchmark) and then report test metrics for that configuration, or report the distribution of SG and PTB results across all 21 configurations (e.g., mean±std) and base the headline claims on typical or pre-registered performance. This issue is load-bearing because the abstract and Section 5.1's primary quantitative claims rely on the selected numbers.","section":"Appendix G / §5.1, Table 2"},{"comment":"The abstract's claim that \"LMs pre-trained with TreeReg on natural language corpora such as WikiText-103 achieve up to 10% lower perplexities on out-of-distribution data\" is contradicted by the first block of Table 3, the exact WikiText-103 setting with no LM training on BLLIP-LG: TreeReg increases PTB perplexity from 331.5 to 411.1, a 24% relative degradation. The PTB improvement (2.4 points, about 4.5% relative) appears only when LM training is additionally performed on BLLIP-LG batches, which is a mixed-data setup rather than the advertised \"different data sources\" scenario. The abstract and Section 5.3 should be qualified to state the condition under which OOD perplexity improves, or the claim should be limited to the same-corpus settings. As written, the general claim overstates the evidence and the failure case is not acknowledged.","section":"§5.3 / Abstract, Table 3"}],"minor_comments":[{"comment":"The abstract reports \"41.2 points\" of mitigation on adversarial NLI benchmarks, but the corresponding numbers in Table 5 (MoNLI decrease without TreeReg: 50.3 to 1.9 = 48.4; with TreeReg: 50.3 to 43.5 = 6.8) give a mitigation of 48.4 - 6.8 = 41.6 points. Please correct the value.","section":"Abstract / §6.2, Table 5"},{"comment":"The text states that TreeReg \"maximizes SCIN for spans corresponding to input constituents while simultaneously minimizing it for other spans,\" but the loss in Eq. (4) only contrasts alternative splits of gold constituents; spans that are not split candidates of any gold constituent are never directly scored or penalized. Consider rephrasing to \"contrasts the gold split against alternative splits of each constituent\" or explaining how the contrastive signal also controls non-candidate spans.","section":"§3.2, Eq. (4)-(5), Alg. 1"},{"comment":"The row \"No LM on BLLIP-LG\" shows a 24% PTB perplexity degradation for TreeReg (411.1 vs 331.5). This failure case should be explicitly discussed in Section 5.3 and reconciled with the abstract's general claim about OOD perplexity improvements, since it directly limits the practical flexibility claim in §3.3.","section":"Table 3"},{"comment":"Figure 7's axis label reads \"PTB Perplexity on SG test suites,\" which is a copy-paste error; the figure plots PTB perplexity against layer and head count, not SyntaxGym performance.","section":"Appendix G, Figure 7"},{"comment":"The justification for using the last token's hidden state as the span representation relies on a citation (Allen-Zhu and Li, 2023) about PCFG-trained transformers. Since the SCIN loss directly supervises these states, an additional diagnostic or representation-level ablation would help substantiate the design choice.","section":"Appendix A.1"},{"comment":"Most results are reported from a single run without standard deviations. Given the hyperparameter sensitivity indicated in Appendix G, reporting variance over seeds for the main configurations would substantially strengthen the reliability of the claimed gains.","section":"Tables 2-5"},{"comment":"The comparison with PLM, TG, and Pushdown LM is based on previously published numbers; please state whether those baselines were trained with the same data size, steps, and hyperparameters, or treat the comparison as indicative rather than controlled.","section":"§5.1, Table 2"},{"comment":"The statement that \"TreeReg more than doubles the sample efficiency of syntactic generalization\" is based only on the SyntaxGym benchmark; the conclusion should qualify that the claim refers to that test suite.","section":"§5.2 / Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The test-set selection in Appendix G is the main obstacle to accepting the reported magnitudes. If the authors can provide a validation-selected configuration or a distribution of results across the hyperparameter sweep, the paper would be much stronger. The manuscript is otherwise within scope and the core idea is interesting, but the current reporting protocol makes the headline gains unverifiable as stated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one if you care about injecting syntax into transformers without touching the architecture. The idea is simple: take silver constituency parses, define a span contextual independence score (SCIN) from attention-head hidden states, and add a loss that pushes constituent spans to be orthogonal to their context while pushing non-constituents apart. No architectural changes, no inference overhead. That's genuinely new, as far as I know—prior work either modifies attention or adds parameters.\n\nThe empirical work is decent. They test it on grokking tasks, from-scratch LM pretraining, continued pretraining of Sheared Llama, and NLI fine-tuning. There's a randomized-parse control, ablations on the two SCIN terms, and they release code. The parse recovery results (F1 around 90) are a nice plus.\n\nSoft spots, in order of importance. First, the headline numbers in Table 2 come from picking the layer and number of heads that gave the best SyntaxGym and PTB results on the test sets themselves (Appendix G). That's 21 configurations, and they selected the best. So the 8.1-point SG gain and 9.2% PTB improvement are likely optimistic; the true effect is probably smaller. However, across most hyperparameter settings TreeReg does beat the Base LM on both metrics, so the qualitative finding is robust. Still, they should either use a validation split or report the distribution over configurations.\n\nSecond, the WikiText-103 claim. The abstract says \"up to 10% lower perplexities on out-of-distribution data,\" but the only OOD improvement on PTB comes when they also train on BLLIP-LG batches. Without that, PTB perplexity gets worse (411 vs 331). The authors acknowledge this in Section 5.3, but the abstract oversells it.\n\nThird, the NLI robustness result doesn't include a non-syntactic regularizer control, so we don't know if the effect is syntax-specific or just \"any auxiliary loss helps.\" The randomized-parse control from pretraining is the best evidence they have, but it's not the same setting.\n\nThe core assumption that the last token's hidden state represents the whole span is exactly that—an assumption. It's grounded in some prior work and the empirics are okay, but it's worth flagging.\n\nBottom line: this is a solid, publishable paper with a real contribution, but the headline magnitudes should be read with caution. It deserves serious refereeing; the main fixes are easy—report validation-based hyperparameter selection and phrase the WikiText results more carefully. I'd take it to reading group and would cite it.\n\nRecommendation: send it out for peer review, with a request to address the test-set selection issue before acceptance.","headline":"TreeReg is a genuinely new regularizer for injecting syntax into transformers, but the headline gains are inflated by test-set hyperparameter selection and the abstract oversells the WikiText results.","tokens_in":19765,"tokens_out":3331,"would_cite":true,"duration_ms":26487,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TreeReg injects tree structure into transformer LMs with an auxiliary orthogonality loss, improving syntactic generalization by up to 9.5 points and out-of-distribution perplexity by up to 9.2% with no architectural change.","keywords":["TreeReg","syntactic inductive bias","orthogonality constraints","constituency parsing","transformer language models","syntactic generalization","out-of-distribution perplexity","Span Contextual Independence Score"],"falsifier":"Take a test set of sentences where the most informative word of a phrase is not its last word, such as 'the decision to resign' in 'the decision to resign was announced', decode each sentence with the greedy SCIN parser, and compare against gold parse trees: if the recovered trees systematically miss exactly those phrases while matching elsewhere, the last-token span vector is not carrying the information the orthogonality constraint depends on.","tokens_in":18791,"feed_emoji":"🌳","tokens_out":10307,"duration_ms":81120,"temperature":0.7,"pith_summary":"TreeReg is an auxiliary loss that tells a transformer language model which spans of words should behave like independent constituents. It converts bracketings from silver constituency parses into orthogonality constraints on hidden states: a constituent's vector should have zero overlap with the context immediately before and after it. The paper claims this soft syntactic bias improves syntactic generalization by up to 9.5 points on SyntaxGym and lowers out-of-distribution perplexity by up to 9.2%, with the architecture and inference procedure left untouched. If true, this is a practical recipe for adding hierarchical structure to standard transformer LMs, roughly doubling their sample efficiency on syntactic generalization and making fine-tuned models more robust on adversarial NLI benchmarks.","feed_headline":"Tree regularizer boosts syntax generalization by 9.5 points","feed_subtitle":"An orthogonality term on hidden states also cuts out-of-distribution perplexity, with no architecture change.","key_machinery":"The Span Contextual Independence Score (SCIN) is the core object. For a span $S_{i;j}$, with $h_k$ the L2-normalized concatenation of hidden states from a chosen subset of attention heads at a chosen layer, it is defined as $\\mathrm{SCIN}(i,j) = \\|\\mathrm{orth}(h_j,h_{i-1})\\| + \\|\\mathrm{orth}(h_{j+1},h_j)\\|$, where $\\mathrm{orth}(x,y)=x-(x^{\\top}y)y$ is the component of $x$ orthogonal to $y$. The first term asks that the span be independent of what precedes it; the second asks that the following token be independent of the span. The TreeReg loss sums, over every constituent in the silver parse, a log-loss that contrasts the score of the silver split with the softmax over all possible splits, thereby raising SCIN on constituents and lowering it on non-constituents. The same greedy read of SCIN scores recovers an induced parse tree at inference without changing the model.","core_discovery":"The central claim is that the hidden states of an autoregressive transformer can be shaped into a tree-structured computation by a regularizer that is simply added to the language-model loss. For each span in a silver constituency parse, TreeReg drives up the Span Contextual Independence Score (SCIN), defined as the L2 norm of the component of the span's last-token hidden state orthogonal to the hidden state at the preceding boundary, plus the norm of the orthogonal component of the following hidden state relative to the span's; non-constituent spans are pushed in the opposite direction. In an optimally trained model, the tree that maximizes cumulative SCIN over its spans recovers the constituency parse, so the regularizer acts as a soft, differentiable re-encoding of syntax. The authors show that models trained this way from scratch on BLLIP-LG beat an otherwise identical base LM by 2.6 points on BLiMP and 8.1 points on SyntaxGym, lower PTB perplexity by 9.2%, and exceed the baseline's best syntactic generalization with less than half the training data; continued pretraining of Sheared Llama-1.3B with TreeReg improves syntactic generalization, and fine-tuning with TreeReg on MultiNLI reduces the drop on MoNLI from 48.4 points to 6.8 points relative to the unfinetuned model.","pith_inferences":["A testable extension beyond the paper is to swap the last-token span vector in Eq. (1) for a mean-pooled or attention-pooled representation; if the gains vanish, the paper's assumption that the final token summarizes the span is doing the work, and if they persist, the mechanism is more general.","The paper's greedy parse decoder only sees SCIN on spans that were possible splits of supervised constituents, so the induced tree need not be the global SCIN-maximizing tree; a CKY decoder over all spans would reveal whether the bias is encoding a genuine parse or only a local approximation.","Because randomized parses still help BLiMP but not SyntaxGym, some of TreeReg's effect may be a generic regularity push rather than the specific tree shape; comparing TreeReg against an equal-cost regularizer on random spans would isolate the syntactic content.","The layerwise F1 curve in Figure 4, rising to layer 12 and collapsing after, is a diagnostic that could be used on any new model to check whether the regularized circuit is actually the one doing syntactic work."],"forward_implications":["If TreeReg is correct, an unchanged transformer LM architecture can be given a syntactic inductive bias without additional parameters or inference-time cost.","Standard LMs will need less than half the training data to reach the same level of syntactic generalization when TreeReg is added.","Out-of-distribution text (PTB) will be modeled with up to 9.2% lower perplexity when TreeReg is used during pretraining.","Continued pretraining of an LLM with TreeReg should improve BLiMP, SyntaxGym, and PTB perplexity over continued pretraining without it.","Fine-tuning an LLM with TreeReg will preserve much more of its accuracy on adversarial NLI datasets compared with standard fine-tuning."],"supporting_citations":[{"why":"Supplies the tree-projection and contextual-independence idea that TreeReg operationalizes as orthogonality constraints.","marker":"Murty et al., 2023c"},{"why":"Provides the evidence that last-token hidden states summarize prefixes, which is the load-bearing assumption behind using $h_j$ as the span vector in SCIN.","marker":"Allen-Zhu and Li, 2023"},{"why":"Defines the tense inflection and question formation diagnostic tasks used for the grokking warm-up experiments.","marker":"McCoy et al., 2020"},{"why":"Provides BLiMP, the minimal-pair benchmark used to measure syntactic generalization.","marker":"Warstadt et al., 2020"},{"why":"Provides SyntaxGym, the surprisal-constraint test suite where TreeReg gains up to 9.5 points.","marker":"Gauthier et al., 2020"},{"why":"Releases BLLIP-LG, the parsed corpus used for from-scratch pretraining and continued pretraining.","marker":"Hu et al., 2020"},{"why":"Supplies Sheared Llama-1.3B, the pretrained LM used for continued pretraining and MultiNLI fine-tuning experiments.","marker":"Xia et al., 2024"},{"why":"Provides MultiNLI, whose parses and train split are used in the fine-tuning robustness experiments.","marker":"Williams et al., 2018"}],"fun_headline_variants":["TreeReg: syntax boost 9.5 pts without new architecture","Hidden states learn tree structure, cut data needs in half","Orthogonality term injects syntax, outperforms with less data","Tree-shaped hidden states: syntax gains, lower perplexity","A regularizer that sneaks tree syntax into any transformer"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the hidden state of the last token of a span faithfully represents that span, so that making that vector orthogonal to its surrounding context really captures the independence of the grammatical phrase.","fun_headline_variants_meta":{"raw":{"variants":["TreeReg: syntax boost 9.5 pts without new architecture","Hidden states learn tree structure, cut data needs in half","Orthogonality term injects syntax, outperforms with less data","Tree-shaped hidden states: syntax gains, lower perplexity","A regularizer that sneaks tree syntax into any transformer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00024,"raw_usage":{"total_tokens":1571,"prompt_tokens":1054,"completion_tokens":517,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":670,"completion_tokens_details":{"reasoning_tokens":432}},"tokens_in":670,"tokens_out":517,"duration_ms":5469,"temperature":1.0,"reasoning_tokens":432,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:47:08.509137+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a test set of sentences where the most informative word of a phrase is not its last word, such as 'the decision to resign' in 'the decision to resign was announced', decode each sentence with the greedy SCIN parser, and compare against gold parse trees: if the recovered trees systematically miss exactly those phrases while matching elsewhere, the last-token span vector is not carrying the information the orthogonality constraint depends on.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the tense inflection and question formation diagnostic tasks used for the grokking warm-up experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides MultiNLI, whose parses and train split are used in the fine-tuning robustness experiments."}],"review_version":1}