{"id":"6959369a-ed6a-40e3-863d-93000542d505","arxiv_id":"2501.06278","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"RoBERTa and DistilBERT align slightly better with fMRI brain responses than BERT, and removing punctuation yields a small improvement in BERT's later-layer alignment.","lead":"This study compared four modern language models (RoBERTa, DistilBERT, ALBERT, ELECTRA) against fMRI brain scans of people reading a story, using a published brain-alignment method. The authors report that RoBERTa and DistilBERT match brain activity slightly better than BERT, and that removing punctuation improves BERT's alignment a little.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The punctuation-removal result is confounded with token-identity and attention-mask changes; no control separates mechanical effects from semantic processing, so the brain-level conclusion is not yet supported.","rationale":"The paper makes two empirical claims: that RoBERTa (and DistiliBERT) align better with fMRI than the reproduced BERT baseline, and that BERT's alignment improves when punctuation is replaced by [PAD], supporting a conclusion that the brain makes limited semantic use of punctuation. The first claim is a fairly direct model comparison, though it depends on the authors' own BERT reproduction rather than the original Tonva-Wehbe numbers. The second claim is the more novel and the more load-bearing for the paper's title and conclusion. The weakest point is the punctuation counterfactual: replacing punctuation with [PAD] changes token embeddings, attention masks, and effective context, so the improvement could be a mechanical artifact of the model's tokenization rather than evidence about human semantic processing of punctuation. The reader's weakest_assumption identifies exactly this issue, and I agree. I do not see grounds for rejection: the encoding pipeline is a standard ridge-regression and searchlight framework, the data are public, and the paper is transparent about its baseline reproduction difficulty. However, the conclusion currently overstates what the [PAD] perturbation can establish. A matched control condition is cheap and would settle whether the concern lands. If the control shows the effect is mechanical, the brain-level claim should be dropped or substantially weakened; if the effect survives in both control arms, the conditional concerns would be addressed. Since the reader already conditioned the verdict on this and related statistical issues, my stress-test does not move the verdict.","tokens_in":12779,"tokens_out":4796,"duration_ms":51448,"concrete_test":"Run the same BERT encoding pipeline with two additional conditions on the chapter text: (A) replace each punctuation token with a rare non-punctuation token such as 'zqx' while keeping attention masks unchanged; (B) replace an equal number of randomly selected content words with [PAD] while keeping punctuation intact. If condition A reproduces the layer 7–12 improvement, the effect is about token identity or contextualization rather than punctuation absence; if condition B also improves alignment, the effect is not punctuation-specific. Report all 12 layers and all sequence lengths with per-subject variance, and compare the layer 7–12 improvement against a permutation null over which layers and sequence lengths are selected.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central inference about punctuation (§4, 'Results with Removing Punctuation' and §5) is that removing punctuation tokens makes BERT align better with fMRI, and that this shows 'the brain might have limited use of punctuation... semantically.' The experimental manipulation in §3.3, scenarios 3–4, replaces '–', '...', '—', '.', '?' with the [PAD] token. This is not a clean semantic deletion: [PAD] has its own learned embedding, it changes the attention mask (PAD positions are ignored by the model), and it changes the effective context available when computing contextualized representations. Any of these mechanical changes could shift ridge-regression encoding accuracy without reflecting whether the human brain extracts semantic content from punctuation. The reported improvement appears 'only on layers 7–12' with a maximum boost of almost 1.5%; this layer selection is post hoc and no error bars, significance tests, or multiple-comparison corrections are reported. Without a control condition that matches the number and position of replaced tokens while varying only their semantic status, the observed alignment gain cannot be uniquely attributed to the semantic role (or lack thereof) of punctuation in the brain.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper applies the Toneva and Wehbe fMRI-alignment pipeline to compare four transformer variants (RoBERTa, DistiliBERT, ALBERT, ELECTRA) against a BERT baseline, using publicly available fMRI data from subjects reading a chapter of Harry Potter. It then tests four punctuation-removal scenarios in which punctuation or fixation symbols are replaced with [UNK] or [PAD] tokens, re-extracts model features, and measures searchlight classification accuracy. The main claims are that RoBERTa aligns most closely with neural activity, surpassing BERT, and that BERT's alignment accuracy improves when punctuation is replaced with [PAD], especially in layers 7–12, which the authors interpret as evidence that the brain makes limited semantic use of punctuation.","tokens_in":12950,"tokens_out":2558,"duration_ms":28203,"significance":"If the central claims held, the paper would provide a useful extension of brain-alignment evaluations to newer transformer architectures and would demonstrate a model-driven way to generate hypotheses about the neural processing of punctuation. The work has real strengths: it uses publicly available data and code, follows a published pipeline, and is transparent about the failure to reproduce the original BERT results exactly, including a qualitative comparison in Figure 2. The alignment scores are computed against external fMRI data with no fitted constants, and the paper's layer-by-layer analyses follow an established methodology. However, the load-bearing conclusions about punctuation rest on a manipulation that is confounded with mechanical tokenization changes, and the reported effects lack statistical validation; these issues affect the central claims rather than just the presentation.","major_comments":[{"comment":"The punctuation-removal manipulation, scenarios 3 and 4, replaces punctuation tokens with [PAD]. This does not isolate the semantic role of punctuation: [PAD] has its own learned embedding, changes the attention mask so that those positions are ignored, and changes the effective token sequence seen by the model. Any of these mechanical effects could alter ridge-regression encoding accuracy without reflecting how the human brain processes punctuation. The manuscript provides no control condition that replaces the same tokens with a semantically neutral but attention-preserving alternative (e.g., [UNK]) or that otherwise varies only the semantic status of the removed tokens. The conclusion in §5 that 'the brain might have limited use for punctuation symbols to understand semantically a sentence' is therefore not uniquely supported by the reported experiments.","section":"§3.3, Removing punctuation"},{"comment":"The reported improvement from punctuation removal is described as occurring 'only on layers 7-12' with a maximum boost of almost 1.5%, but no significance tests, error bars, or multiple-comparison corrections are provided. The layer range appears to be selected post hoc, and the claim that layer 6 acts as a 'divisor' is asserted without statistical support. Because the underlying searchlight accuracies vary across subjects, folds, and layers, the authors should report across-subject standard errors or confidence intervals and perform a permutation or bootstrap test comparing the punctuation-removed condition against the baseline, with correction for the number of layers and sequence lengths tested.","section":"§4, Results with Removing Punctuation"},{"comment":"The authors state that they were not able to exactly reproduce the original BERT results from [53], and they use their own reproduced results as the baseline for all model comparisons. This is disclosed honestly, but it means the reported differences between RoBERTa or DistiliBERT and the baseline could be within the range of the reproduction discrepancy rather than reflecting genuine model differences. The manuscript should quantify the reproducibility of its own baseline (e.g., across random seeds or PCA initializations) and show that the model comparisons are robust to this variability, or compare all models against the original published numbers on a common metric.","section":"§4, BERT baseline reproduction"},{"comment":"The conclusion that the brain has limited semantic use of punctuation goes beyond what the encoding-alignment results can show. The experiments demonstrate that replacing punctuation with [PAD] can change the alignment of one transformer model (BERT) with fMRI data; they do not directly measure the brain's use of punctuation. Even if the mechanical confound were addressed, the inference would require additional evidence, such as behavioral reading measures or brain-region-specific analyses that link punctuation processing to semantic processing. The conclusion should be substantially tempered or supplemented with converging evidence.","section":"§5, Conclusion"}],"minor_comments":[{"comment":"The model names contain typographical artifacts such as 'RoBER T a' and 'DistiliBER T'; these should be corrected to RoBERTa and DistiliBERT throughout the text.","section":"§3.2"},{"comment":"There are several typographical errors in author and reference names, for example 'Toneva and Whebe' should be 'Toneva and Wehbe'; a careful proofreading pass is needed.","section":"Throughout"},{"comment":"The fixation-replacement scenarios (scenarios 1 and 2) are grouped together with punctuation-removal scenarios in the phrase 'in all four punctuation-modification scenarios,' although the fixation symbol is not punctuation. This conflation makes the summary of results harder to interpret; the two types of manipulations should be reported separately.","section":"§3.3 and §4"},{"comment":"The two panels of Figure 2 use different y-axis ranges, which makes the quantitative comparison between original and reproduced BERT results visually misleading; the authors note this in the caption, but it would be clearer to use a common scale or to overlay the two curves.","section":"§4, Figure 2"},{"comment":"The footnotes referring to the data and code appear as bare placeholders ('the data and the original code can be found at this link', '4') without URL or repository identifiers; these should be completed with working links or DOIs.","section":"Footnotes"}],"recommendation":"major_revision","confidential_remarks":"The punctuation-removal result is the most distinctive contribution, but as written it is not yet supported because the [PAD] replacement confounds semantic deletion with token-identity and attention-mask changes. The authors' own prior symposium work (ref. 31) appears to be the origin of this line of investigation, yet the manuscript does not discuss how the present results extend or differ from that earlier report; the editor may wish to ask for that clarification. With added control conditions, proper statistical testing, and a more cautious conclusion, the paper could become a solid empirical contribution to the brain-alignment literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline for me: the four-model sweep is a genuinely new, useful benchmark, but the punctuation-brain conclusion is not supported by the experiment as run.\n\nWhat's actually new: nobody had run Toneva & Wehbe's encoding pipeline on RoBERTa, DistilBERT, ALBERT, and ELECTRA against the same fMRI data, and that comparison is worth having. The authors are also honest that they could not exactly reproduce the original BERT baseline and give plausible reasons. That transparency earns credit.\n\nWhat the paper does well: it follows a published pipeline, uses public data, and makes its code and data available. The qualitative pattern — RoBERTa and DistilBERT slightly above BERT, ELECTRA below, accuracy falling with sequence length — is consistent with the prior literature. As a benchmark note, this is fine.\n\nThe soft spots are real and they sit right on the two headline claims. First, there are no error bars, significance tests, or multiple-comparison corrections anywhere. The reported differences between models are small, and without subject-level variance we have no idea if they exceed noise. Second, the punctuation manipulation is confounded. Replacing '.' and '?' with [PAD] changes the token identities, changes the attention mask (PAD positions are masked), and changes the effective context the model sees. The stress-test note is exactly right: there is no control condition that holds the position and number of replaced tokens constant while varying only semantic status. So the finding that BERT aligns better without punctuation cannot be uniquely attributed to the brain's semantic processing of punctuation. The authors even report the effect appears only on layers 7–12 and is at most ~1.5% — that is post hoc layer selection, and it undercuts the confidence of the conclusion. The not-exactly-reproduced BERT baseline is a third issue; comparing against a baseline that differs quantitatively from the original is workable, but it adds another layer of uncertainty.\n\nAlso minor: the punctuation result overlaps with their own 2022 symposium paper and the current paper doesn't state what is added beyond that.\n\nWho it's for: labs doing brain-encoding model comparisons or deciding whether to keep punctuation in their stimulus preprocessing. It is a useful data point, not an established result.\n\nRecommendation: send it to peer review, but with a serious request for revision. The model sweep deserves to be published in some form. The punctuation-brain claim needs either a proper control condition or a clear 'exploratory' framing with error bars. I would not accept the claims as stated.","headline":"Useful four-model brain-encoding benchmark, but the punctuation finding is confounded and the headline claims outrun the statistics.","tokens_in":13529,"tokens_out":2524,"would_cite":false,"duration_ms":24405,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that RoBERTa's internal representations align more closely with fMRI brain activity during narrative reading than BERT's, and that removing punctuation improves BERT's alignment in later layers, suggesting the brain…","keywords":["fMRI","transformer models","brain alignment","punctuation","semantic processing","RoBERTa","BERT","searchlight classification"],"falsifier":"Run a control condition that removes or replaces the same number of non-punctuation tokens, or that swaps punctuation for random punctuation tokens, and check whether the roughly 1.5% later-layer accuracy gain in BERT still appears; if it does, the punctuation-specific interpretation is falsified, and if it disappears, the interpretation is supported.","tokens_in":12535,"feed_emoji":"🧠","tokens_out":8637,"duration_ms":75024,"temperature":0.7,"pith_summary":"Using fMRI recordings of eight people reading a story chapter, this paper tests whether newer transformer models represent language in a way that tracks human brain activity, and whether punctuation contributes to that alignment. The authors train ridge-regression maps from each model layer's features to brain responses and score them with searchlight classification, following an established alignment procedure, and compare four BERT-derived models against BERT itself. They find that RoBERTa aligns most closely with brain activity, with DistilBERT also ahead of BERT, while ELECTRA and ALBERT do not beat the baseline. In four punctuation-removal scenarios, replacing punctuation with padding or unknown tokens raises BERT's alignment accuracy in layers 7 to 12, with the largest gain near 1.5%, and weakens the usual accuracy drop at longer sequence lengths. The authors take this as evidence that the brain makes limited semantic use of punctuation and needs it less as context grows.","feed_headline":"RoBERTa outperforms BERT at predicting brain activity","feed_subtitle":"Strip punctuation and BERT's brain-fit improves, hinting punctuation carries little semantic weight.","key_machinery":"The mechanism is a two-part alignment pipeline. A ridge regression learns a linear mapping from PCA-reduced transformer features, extracted per layer and per sequence length (4, 5, 10, 15, 20, 25, 30, 35, 40 words), to delayed fMRI BOLD responses, and then searchlight classification scores each voxel's neighbourhood by whether predicted brain images place a correct 20-TR chunk closer than a random incorrect chunk over 1000 trials. The punctuation manipulation is the second load-bearing part: four scenarios replace the fixation symbol or punctuation marks with [UNK] or [PAD] tokens before feature extraction, and the comparison of layer-wise accuracy across these scenarios is what carries the claim about punctuation's semantic role.","core_discovery":"The paper's central claim is that, under the brain-alignment procedure, RoBERTa produces layer-wise representations that are more predictive of fMRI responses during natural story reading than BERT's, and DistilBERT also outperforms the BERT baseline, whereas ELECTRA and ALBERT do not. The paper further claims that when punctuation symbols are replaced with [PAD] or [UNK] tokens before feature extraction, BERT's alignment improves specifically in the later layers, 7 through 12, with a peak gain of almost 1.5%, and the accuracy loss at longer sequence lengths is reduced. The authors interpret this pattern as evidence that the brain makes limited semantic use of punctuation and relies on it less as contextual length grows, and they note that layer 6 appears to divide earlier from later layers, consistent with earlier findings that the first six BERT layers are less brain-aligned than the last six.","pith_inferences":["Beyond the paper: a control condition that replaces punctuation with ordinary low-information tokens, rather than [PAD] or [UNK], could separate the semantic effect of punctuation from mechanical tokenization and attention-mask effects; the paper does not include such a control.","Beyond the paper: the layer-7-to-12 locus makes a testable prediction that ablating or freezing later BERT layers should remove the punctuation benefit, a manipulation the paper does not perform.","Beyond the paper: the alignment score could be turned into a model-selection screen that ranks pretrained models by neural fit before downstream fine-tuning, which would extend the comparison beyond the four models tested."],"forward_implications":["If RoBERTa's training choices yield more brain-aligned representations, then pretraining decisions such as masked-language-modelling objectives and longer training are candidate levers for making models neurally plausible.","If BERT's later layers, 7 to 12, become more brain-aligned when punctuation is masked, then the semantic contribution of punctuation is concentrated in those layers and is small in magnitude.","If the accuracy drop with longer context is reduced when punctuation is removed, then punctuation contributes less to comprehension as more context accumulates.","If DistilBERT keeps pace with BERT despite being smaller, then model compression does not destroy the brain-relevant semantic information captured in layer representations."],"supporting_citations":[{"why":"Supplies the alignment procedure: ridge regression from model features to fMRI responses and searchlight evaluation.","marker":"[53]"},{"why":"Provides the fMRI dataset of eight subjects reading a story chapter and the reading paradigm the analysis uses.","marker":"[54]"},{"why":"Defines RoBERTa, the model found to be most brain-aligned, and the training choices contrasted with BERT.","marker":"[58]"},{"why":"Defines DistilBERT, the compressed model whose layer alignment is compared against BERT.","marker":"[49]"},{"why":"Defines ELECTRA, whose discriminative pretraining yields alignment that does not surpass BERT.","marker":"[10]"},{"why":"Defines ALBERT, whose shared-layer architecture is tested for brain alignment.","marker":"[32]"},{"why":"Defines BERT, the baseline model whose reproduced alignment is the comparison point.","marker":"[13]"},{"why":"Prior evidence that punctuation is largely insignificant semantically for neural models, which the brain result extends.","marker":"[14]"},{"why":"Prior work showing punctuation affects reading behaviour, motivating the question of its semantic role.","marker":"[37]"}],"fun_headline_variants":["RoBERTa beats BERT in brain alignment, punctuation removal boosts BERT","Punctuation hurts BERT's brain alignment, RoBERTa leads","Removing punctuation improves BERT's brain-fit, RoBERTa still tops","RoBERTa best matches brain activity, BERT benefits from stripped punctuation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes that replacing punctuation with [PAD] or [UNK] tokens isolates the semantic role of punctuation, but those replacements also change token identities, attention masks, and effective sequence length, so the measured alignment gain could come from mechanical tokenization effects rather than from how the brain processes punctuation.","fun_headline_variants_meta":{"raw":{"variants":["RoBERTa beats BERT in brain alignment, punctuation removal boosts BERT","Punctuation hurts BERT's brain alignment, RoBERTa leads","Removing punctuation improves BERT's brain-fit, RoBERTa still tops","RoBERTa best matches brain activity, BERT benefits from stripped punctuation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000771,"raw_usage":{"total_tokens":3357,"prompt_tokens":833,"completion_tokens":2524,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":449,"completion_tokens_details":{"reasoning_tokens":2443}},"tokens_in":449,"tokens_out":2524,"duration_ms":15431,"temperature":1.0,"reasoning_tokens":2443,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:06:31.433094+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a control condition that removes or replaces the same number of non-punctuation tokens, or that swaps punctuation for random punctuation tokens, and check whether the roughly 1.5% later-layer accuracy gain in BERT still appears; if it does, the punctuation-specific interpretation is falsified, and if it disappears, the interpretation is supported.","supporting_citations":[{"cited_title":"In: Advances in Neural Information Processing Systems","cited_arxiv_id":null,"evidence_quote":"Supplies the alignment procedure: ridge regression from model features to fMRI responses and searchlight evaluation."},{"cited_title":"In: Proceedings of the 20th Chinese National Confer- ence on Computational Linguistics","cited_arxiv_id":null,"evidence_quote":"Defines RoBERTa, the model found to be most brain-aligned, and the training choices contrasted with BERT."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines DistilBERT, the compressed model whose layer alignment is compared against BERT."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines ELECTRA, whose discriminative pretraining yields alignment that does not surpass BERT."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines ALBERT, whose shared-layer architecture is tested for brain alignment."},{"cited_title":"In: Proceedings of the Probability and Meaning Conference (PaM 2020)","cited_arxiv_id":null,"evidence_quote":"Prior evidence that punctuation is largely insignificant semantically for neural models, which the brain result extends."},{"cited_title":"Functional Linguistics 3(1), 6 (May 2016)","cited_arxiv_id":null,"evidence_quote":"Prior work showing punctuation affects reading behaviour, motivating the question of its semantic role."}],"review_version":1}