{"id":"d3f98343-291f-4b2e-8d69-0990edb0fba5","arxiv_id":"2501.05503","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Fine-tuning a single added layer on a synthetic polypersonal Russian marks causes BERT to separate the new grammar from standard Russian in its internal representations and to reduce the perplexity gap.","lead":"The authors added an artificial grammatical rule, polypersonal verb agreement, to Russian and trained a BERT model to recognize it. They report that adding a small fine-tuned layer lets the model keep ordinary Russian and modified Russian separate in its internal geometry while improving language-model scores.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Causal attribution is undermined by an internal contradiction: §3.1 freezes BERT to prevent it learning polypersonality, but §3.3 describes a 10-epoch unfrozen MLM pretraining on polypersonal data; if that stage was used, the §4.3 bottleneck-distance shift cannot be attributed to the added layer.","rationale":"The Reader's verdict is CONDITIONAL, and I agree that the evidence is under-powered and that the bottleneck-distance interpretation is threatened by tokenization and positional-encoding artifacts. However, I judge an even more load-bearing problem to be the unresolved procedural contradiction about whether the base BERT itself was trained on polypersonal data. Section 3.1 says BERT is frozen to prevent it from acquiring polypersonality; Section 3.3 says RuBERT was pretrained for 10 epochs with all layers unfrozen on both regular and polypersonal sentences. These cannot both describe the same model pipeline. If the unfrozen stage was used, then the comparison in Section 4.3 between 'pre-trained BERT layer outputs' and 'fine-tuned' outputs is not a clean before/after comparison of adding a layer; it is a comparison that may reflect direct modification of the base encoder. That would invalidate the paper's main causal statement, regardless of how the topological distances are interpreted. The Reader's rationale does mention 'an inconsistency about what the base BERT saw before freezing,' so there is partial agreement, but the chosen weakest assumption was tokenization rather than this procedural ambiguity. I do not propose to change the verdict from CONDITIONAL because the contradiction might be resolved in the authors' favor: perhaps the 10-epoch stage was a separate adaptation step and the actual experimental models had BERT frozen from the start. That is exactly what a code inspection or a controlled rerun would settle. The tokenization confound remains a valid additional concern, but the freezing inconsistency is more fundamental because it undercuts causal attribution before any geometric interpretation is needed.","tokens_in":6533,"tokens_out":5663,"duration_ms":55671,"concrete_test":"Inspect the training configuration or released code to determine whether the 10-epoch unfrozen MLM stage of §3.3 was applied before freezing in the models whose outputs are compared in §4.3. If it was, rerun the entire pipeline with BERT strictly frozen throughout, training only the MLM head or LSTM+MLM head on the polypersonal data, and recompute the bottleneck-distance comparison between base and polypersonal sentences. If the distance shift disappears or is markedly smaller, the observed geometric separation is attributable to BERT's own fine-tuning rather than to the added layer, and the central claim fails. Reporting the tokenization and sequence-length statistics for base versus polypersonal sentences in the same rerun would additionally control for the tokenization confound acknowledged in §4.3.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is causal: the abstract says 'adding even a single grammatical layer causes the model to separate the new and old grammatical systems,' and the conclusion says 'fine-tuning layers creates disentangled and linearly separable representations.' For this to hold, the base BERT must not itself have learned polypersonality, so the only trained component is the added layer. The manuscript directly contradicts this precondition. Section 3.1 states: 'The first block is RuBERT... This block is frozen during training as we do not want RuBERT to acquire information about polypersonality.' Section 3.3 states: 'At first, in order to adapt the model to our data, RuBERT was pretrained for 10 epochs with all the layers unfrozen on MLM task using regular Russian sentences and modified polypersonal Russian sentences mixed in equal amounts.' If this 10-epoch unfrozen stage was part of the pipeline that produced the models analyzed in §4.3, then the 'pretrained BERT layer outputs' used as a baseline are not from vanilla RuBERT; BERT's own weights were updated on polypersonal text. The observed bottleneck-distance shifts in Figure 3 could then be caused by BERT itself, not by the single additional grammar layer. The later claim in §4.3 that 'BERT never actually encountered polypersonal sentences during pretraining' is only true for the original RuBERT checkpoint, and it is contradicted by the §3.3 procedure if that procedure was actually applied. This ambiguity is the load-bearing weak point: the paper's headline causal statement is not supported by the experiments as described, because the counterfactual condition (frozen base) is not consistently maintained.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies how a BERT model adapted to an artificial grammatical feature (polypersonal agreement in Russian) changes its internal representations. The authors fine-tune RuBERT with an added MLM head and, in one variant, an additional LSTM layer, on data containing polypersonal suffix or prefix verb forms. They report pseudo-perplexity, layerwise MLM probing results, and bottleneck distances between Vietoris-Rips persistence diagrams built from token representations. The central claim is that adding even one grammatical layer causes the model to separate old and new grammatical systems in latent space and improves perplexity adaptation, with the conclusion asserting that fine-tuned layers create 'disentangled and linearly separable representations.'","tokens_in":6796,"tokens_out":4936,"duration_ms":46818,"significance":"If the causal claim were established, the paper would offer a striking and practically relevant result: a small trainable module attached to a frozen language model can acquire a novel grammatical category and localize it in a distinct region of the latent space, with geometric changes visible through topological data analysis. The study's strengths include the use of minimal pairs, the choice of pseudo-perplexity scoring appropriate for multi-token words, and the explicit acknowledgment of tokenization as a possible confound. The bottleneck-distance methodology is a nonstandard and potentially informative lens on fine-tuning. However, the central causal conclusion is currently not supported by the evidence as presented, because of a direct contradiction in the training pipeline and the absence of any variance or significance assessment in the geometric comparisons.","major_comments":[{"comment":"Section 3.1 states that RuBERT is frozen during training 'as we do not want RuBERT to acquire information about polypersonality,' but Section 3.3 states that 'RuBERT was pretrained for 10 epochs with all the layers unfrozen on MLM task using regular Russian sentences and modified polypersonal Russian sentences mixed in equal amounts.' These statements cannot both describe the pipeline that produced the models analyzed in Section 4.3. If the unfrozen pretraining stage was used, then the baseline 'BERT layer outputs' in Figure 3 are not from vanilla RuBERT, and the observed bottleneck-distance shift cannot be attributed to the added grammar layer; the statement in Section 4.3 that 'BERT never actually encountered polypersonal sentences during pretraining' is then false. The authors must clarify whether the frozen-pipeline models produced all reported results, or rerun the analysis with a strictly frozen base BERT, before the causal claim in the abstract can be evaluated.","section":"§3.1 and §3.3"},{"comment":"The geometric evidence is underpowered. Bottleneck distances are reported as averages over 1,000 sampled sentences without any variance, confidence interval, or significance test, so the visual differences in Figure 3 could plausibly be driven by outliers or noise. Furthermore, the Conclusion's assertion that fine-tuning 'creates disentangled and linearly separable representations' is not supported by any direct measurement of linear separability (e.g., linear probe accuracy, margin, or a separation index); no such analysis appears in Section 4.3. Without these quantitative supports, the paper overstates what the bottleneck-distance comparisons demonstrate.","section":"§4.3 and Conclusion"},{"comment":"The paper itself concedes that 'the clear identification of the sources is difficult. We suggest that it may be explained both by tokenization (hence positional encoding effects) and linguistic improbability.' This concession directly weakens the abstract's causal claim that adding a grammatical layer 'causes the model to separate the new and old grammatical systems.' The observed bottleneck-distance shifts could reflect surface tokenization of the nonce affixes rather than a newly acquired grammatical representation. A concrete control would be to compare persistence diagrams for sentences in which the same affix shape is used without the polypersonal agreement (or after removing affix tokens), and to show that the geometric separation is specific to the grammatical function rather than to the token string.","section":"§4.3"}],"minor_comments":[{"comment":"Reference [19] is cited for the Vietoris-Rips filtration, but [19] is the VisBERT paper; a standard source for Vietoris-Rips filtrations and persistence diagrams should be cited instead.","section":"References"},{"comment":"The sentence 'ChatGPT is sometimes capable of outperforming full-shot models in such tasks.' appears twice in consecutive paragraphs and should be removed.","section":"Introduction"},{"comment":"There are several typographical issues: 'modeles’' in §3.3, 'сonstructed' with a Cyrillic 'с' in §3.3, 'an noticable' and 'significantlymore' in the Conclusion, and 'Fig, 2' in §4.2.","section":"Throughout"},{"comment":"The text refers to the 'BERT-Small tokenizer' when selecting the affix paradigm, but the experimental model is RuBERT; please clarify which tokenizer was used and whether it matches the final model.","section":"§3.2"},{"comment":"Experimental details needed for reproducibility are missing, including the LSTM hidden dimension, optimizer, learning rate, batch size, number of training examples, and the exact split of the fiction-text corpus into train and test sets.","section":"§3.3"},{"comment":"The figures would benefit from axis labels, units, and error bars; in particular, Figure 3 does not define the scale or meaning of the bottleneck-distance axis in the caption.","section":"Figures 1–3"}],"recommendation":"major_revision","confidential_remarks":"The central idea is interesting and the paper is likely a good fit for a workshop or short-paper venue, but the contradiction between §3.1 and §3.3 is a load-bearing flaw that must be resolved by clarification or by rerunning the analysis with a strictly frozen base model. I would also ask the authors to supply error bars or significance tests for the bottleneck distances and to either measure linear separability directly or remove that claim from the Conclusion. These issues are fixable within the scope of the manuscript, so I do not recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Candid take: the paper's central claim—that adding a single grammar layer causes BERT to separate novel polypersonal morphology—is not supported by the experiments as described. Section 3.1 says BERT is frozen to prevent it learning polypersonality, but Section 3.3 says RuBERT was pretrained for 10 epochs unfrozen on a 50/50 mix of regular and polypersonal sentences. If that stage was run, then BERT itself saw polypersonal text, so the §4.3 comparison between 'pre-trained BERT outputs' and 'fine-tuned layer outputs' does not isolate the added layer. The stress-test note is on target: this is a load-bearing contradiction, not an editing slip.\n\nThe novelty is real. Russian polypersonal agreement is a clean minimal-pair manipulation: an unambiguous suffix or prefix on transitive verbs, no new tokens, and a clear contrast between base and modified text. The prefix/suffix split and the LSTM-vs-MLP head comparison are thoughtful. The perplexity result—fine-tuned models narrow the gap between base and polypersonal text—is a simple, honest observation that should survive scrutiny. The layer-wise probing (probability/rank of correct inflection) also gives a coherent picture, with the prefix case plateauing early and suffix peaking at the last layer.\n\nSoft spots beyond the contradiction: the bottleneck-distance evidence is underpowered. Averages over 1,000 sentences are reported with no variance, no confidence intervals, no significance tests. The conclusion calls the representations 'linearly separable' without any linear-separability measurement. And the authors themselves list tokenization/positional-encoding effects as a possible alternative explanation for the geometry shift, then don't control for it. No code or data is released, so reproducibility is limited.\n\nWho's this for? Researchers interested in morphosyntax in BERT's latent space, or in topological data analysis of embeddings. It deserves serious peer review because the setup is novel and the perplexity observation is worth validating, but only with major revision: fix the contradiction (either report results with the truly frozen model or use the adapted BERT as the proper baseline), add statistical support for the geometry, and drop or substantiate the linear-separability language. I would not cite it in its current form.","headline":"An original but methodologically tangled interpretability study; the headline causal claim is undercut by a contradiction in the training pipeline.","tokens_in":7402,"tokens_out":3648,"would_cite":false,"duration_ms":32845,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding a single fine-tuned layer to a frozen BERT separates a newly introduced grammatical system from the original language inside the model's latent space.","keywords":["BERT latent space","polypersonal agreement","topological data analysis","persistence diagrams","bottleneck distance","fine-tuning","grammar acquisition","interpretability"],"falsifier":"Replace the person-number affix with a matched nonsense suffix that triggers the same tokenization changes but encodes no agreement, and compare the bottleneck distances to the genuine polypersonal case; if the distances are comparable, the effect is an artifact of surface tokenization rather than grammatical separation.","tokens_in":36,"feed_emoji":"🧠","tokens_out":7850,"duration_ms":130225,"temperature":0.7,"pith_summary":"The paper asks what happens inside BERT when a language gains a grammatical feature it never had. It constructs a modified Russian where transitive verbs carry a suffix or prefix marking the person and number of their direct object, then trains a small head on a frozen BERT and watches perplexity and the shape of internal representations. The central claim is that one extra layer is enough to push the new grammar into a separate region of latent space, leaving the old Russian geometry mostly intact while improving overall perplexity. The evidence is topological: bottleneck distances between persistence diagrams of sentence embeddings grow between standard and polypersonal sentences, mainly in the added grammar module rather than in the frozen BERT body.","feed_headline":"One layer is enough for BERT to split new grammar from old","feed_subtitle":"A new verb-agreement system gets its own region in BERT's latent space, so grammar can be added without full retraining.","key_machinery":"The load-bearing object is the pair of persistence diagrams built from a sentence's token embeddings via Vietoris-Rips filtration, compared with the bottleneck distance. A persistence diagram records which clusters or holes in the point cloud of embeddings survive across scales, and the bottleneck distance is the largest shift needed to match one diagram to another. Comparing these distances between standard and polypersonal sentences, across BERT outputs and grammar-module outputs, is what grounds the claim that fine-tuning changes representation topology rather than merely shifting average vectors.","core_discovery":"The paper's central discovery is that fine-tuning a single layer before the prediction head creates a topology change in BERT's latent space: representations of sentences containing the new polypersonal affix are moved away from representations of standard Russian, while the old geometry is left largely intact. The authors measure this as a bottleneck distance between Vietoris-Rips persistence diagrams built from token embeddings, and they find that the separation appears in the grammar module rather than in the frozen BERT body. Prefix marking produces slightly more separation than suffix marking, and the LSTM variant shows the same separation with smaller distances, which they read as the recurrent layer keeping representations more clustered. On the strength of these observations they conclude that fine-tuning layers creates disentangled and linearly separable representations by changing the representation topology, and that this is why the fine-tuned model reduces perplexity on both text types compared with the unadapted checkpoint.","pith_inferences":["The conclusion's phrase \"linearly separable\" is stronger than the reported bottleneck-distance measurements, which compare topology but never train a linear classifier; a linear probe on BERT versus grammar-module outputs is the direct test of that wording.","Since the paper reports no variance across the 1,000 sampled sentences and says the distance source could lie in tokenization or positional encoding, part of the observed separation may be a surface-form artifact; replacing the genuine affix with a matched nonce suffix would isolate the grammatical contribution.","The same minimal-pair protocol should transfer to other absent inflectional categories, such as evidentiality or noun incorporation, to check whether topological disentangling is a general mechanism for adding grammar rather than a peculiarity of polypersonality."],"forward_implications":["A frozen pretrained encoder can gain a brand-new inflectional category by learning a small layer, while leaving the geometry of the original language mostly unchanged.","Because the new system appears in its own region of latent space, grammatical knowledge can be inspected and potentially extracted without full-model fine-tuning.","Fine-tuning a single layer narrows the perplexity gap between standard and polypersonal text, so adaptation is real on the model's own metric.","Layer-wise probing shows the new affix is predicted with rising confidence in later layers, matching the general pattern that higher layers carry syntax.","The same architecture works with either suffix or prefix marking, but prefix marking is harder for overall perplexity while easier for token-level prediction."],"supporting_citations":[{"why":"Supplies the PLL-word-l2r scorer used for pseudo-perplexity measurement on multi-token words.","marker":"[17]"},{"why":"Defines the bottleneck distance used to compare persistence diagrams between layers and text types.","marker":"[18]"},{"why":"Provides prior evidence that fine-tuning changes the topological structure of embeddings more in higher layers, which this work builds on.","marker":"[14]"},{"why":"The 'broken' Russian gender-agreement experiment this work contrasts its added-feature setup against.","marker":"[4]"},{"why":"Justifies tying output embeddings to input embeddings in the fine-tuned head, reducing parameters.","marker":"[16]"}],"fun_headline_variants":["Single layer splits BERT's grammar map","One tuned layer separates BERT grammars","BERT's grammar split: one layer suffices","Fine-tune one layer, see grammar topology shift","Polypersonal grammar gets its own BERT region"],"cache_read_input_tokens":9472,"weakest_assumption_plain":"The geometric conclusion assumes that the bottleneck distances between persistence diagrams of standard and polypersonal sentences reflect the model's grammatical differentiation, not surface tokenization or positional-encoding artifacts of the added affix.","fun_headline_variants_meta":{"raw":{"variants":["Single layer splits BERT's grammar map","One tuned layer separates BERT grammars","BERT's grammar split: one layer suffices","Fine-tune one layer, see grammar topology shift","Polypersonal grammar gets its own BERT region"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000155,"raw_usage":{"total_tokens":1164,"prompt_tokens":842,"completion_tokens":322,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":458,"completion_tokens_details":{"reasoning_tokens":251}},"tokens_in":458,"tokens_out":322,"duration_ms":3678,"temperature":1.0,"reasoning_tokens":251,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:13:50.389253+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the person-number affix with a matched nonsense suffix that triggers the same tokenization changes but encodes no agreement, and compare the bottleneck distances to the genuine polypersonal case; if the distances are comparable, the effect is an artifact of surface tokenization rather than grammatical separation.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the PLL-word-l2r scorer used for pseudo-perplexity measurement on multi-token words."},{"cited_title":"Stability of Persistence Diagrams","cited_arxiv_id":null,"evidence_quote":"Defines the bottleneck distance used to compare persistence diagrams between layers and text types."},{"cited_title":"TopoBERT: Exploring the topology of fine-tuned word representations","cited_arxiv_id":null,"evidence_quote":"Provides prior evidence that fine-tuning changes the topological structure of embeddings more in higher layers, which this work builds on."},{"cited_title":"E., & Kazakova, T","cited_arxiv_id":null,"evidence_quote":"The 'broken' Russian gender-agreement experiment this work contrasts its added-feature setup against."},{"cited_title":"Socher (2016)","cited_arxiv_id":null,"evidence_quote":"Justifies tying output embeddings to input embeddings in the fine-tuned head, reducing parameters."}],"review_version":1}