{"id":"d4831372-e884-450e-803b-6bf9a28717e6","arxiv_id":"2501.03855","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Pretraining ELC-BERT and MLSM on 13 million isiXhosa words improves POS tagging and NER over a RoBERTa baseline, with ELC-BERT matching XLM-R on NER.","lead":"Two data-efficient language model architectures from the BabyLM challenge, ELC-BERT and MLSM, were pretrained from scratch on a 13 million word isiXhosa corpus and beat a standard RoBERTa baseline on part-of-speech tagging and named entity recognition. The results suggest sample-efficient training can help low-resource languages, but also show that missing high-quality pretraining data is the main barrier.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Hyperparameter confound between BabyLM models and RoBERTa baseline (Appendix A, Table 2) leaves the architecture-effect claim untested; a matched-configuration rerun is needed.","rationale":"I read the paper as an empirical case study claiming that BabyLM architectures yield data-efficiency gains for low-resource isiXhosa. For that claim to hold, the comparison against the vanilla RoBERTa baseline must isolate architecture. It does not, due to the hyperparameter differences in Appendix A, Table 2. This is the weakest assumption because it is internal to the experiment design and directly affects the headline result; the skyline comparisons with XLM-R are less central since they are explicitly cross-model. The paper is otherwise clear, reports standard deviations, and honestly discusses limitations. My concern does not overturn the paper, but it makes the stated conclusion conditional on a matched-configuration rerun. The reader's weakest_assumption identified the same confound, so I agree. No change to the reader's CONDITIONAL verdict is needed.","tokens_in":9019,"tokens_out":3715,"duration_ms":33331,"concrete_test":"Retrain the RoBERTa baseline on the same WURA isiXhosa corpus using ELC-BERT's pretraining hyperparameters (learning rate 5e-4, sequence length 128, batch size 128, 200 epochs), with identical fine-tuning and evaluation. Also, as a cross-check, train ELC-BERT with RoBERTa's hyperparameters (LR 5e-5, SL 512, batch 8) if compute allows. If the RoBERTa baseline reaches ELC-BERT's POS/NER scores under matched hyperparameters, the architecture-specific conclusion is unsupported; if the gap persists, the claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparison in Table 1 is not architecture-controlled. Appendix A (Table 2) shows RoBERTa pretrained with LR 5e-5, sequence length 512, batch size 8, while ELC-BERT uses LR 5e-4, sequence length 128, batch size 128, and MLSM uses LR 1e-4, sequence length 128, batch size 64. These settings imply a 4x difference in tokens per batch (RoBERTa: 4096; ELC-BERT: 16384; MLSM: 8192) and a 10x learning-rate difference, so the models receive very different optimization trajectories over the same 200 epochs. The claimed +3.2 NER F1 for ELC-BERT over RoBERTa could therefore reflect batch size, sequence length, or learning rate rather than ELC-BERT's weighted residual connections or MLSM's semantic prediction objective. The paper's mechanism analyses (Figures 2 and 3) are suggestive but do not control for these variables. Because the abstract and conclusion attribute the gains to data-efficient architectures, the load-bearing assumption is that architecture, not hyperparameters, drives the results; the paper never tests this. The finding remains plausible, but as reported it is a comparison of training configurations, not architectures.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper pretrains two BabyLM architectures (ELC-BERT and MLSM) on a 13-million-word isiXhosa corpus (WURA) and evaluates them on MasakhaPOS, MasakhaNER, and MasakhaNEWS, comparing against a RoBERTa baseline pretrained from scratch on the same corpus and against three large multilingual PLMs. The authors report that both BabyLMs outperform the RoBERTa baseline on POS tagging and NER (with ELC-BERT gaining +3.2 F1 on NER), that ELC-BERT in some tasks beats XLM-R, and that neither BabyLM outperforms RoBERTa on news topic classification. They attribute the gains to the architectures' sample-efficient design and present visual analyses of ELC-BERT layer contributions and MLSM semantic categories.","tokens_in":9287,"tokens_out":2301,"duration_ms":22649,"significance":"If the central claim—that BabyLM architectures provide data-efficiency gains for a genuinely low-resource language—is supported, the paper would be a useful contribution to low-resource NLP, showing that architectural innovations from the English BabyLM challenge transfer to a very different language and data regime. The paper has notable strengths: it uses public pretraining corpora and public benchmarks, reports means and standard deviations over five finetuning runs, and includes a limitations section that is candid about the single-language scope. However, the empirical comparison that supposedly isolates the architecture effect is confounded with major hyperparameter differences, so the headline conclusion is currently not established.","major_comments":[{"comment":"The central comparison in Table 1 is not architecture-controlled. Appendix A, Table 2 shows that RoBERTa is pretrained with learning rate 5e-5, sequence length 512, and batch size 8, while ELC-BERT uses 5e-4, 128, and 128 and MLSM uses 1e-4, 128, and 64. These settings produce a 4x difference in tokens per batch between RoBERTa (4096) and ELC-BERT (16384), a 2x difference for MLSM (8192), and a 10x learning-rate difference. The abstract and conclusion attribute the performance differences to the BabyLM architectures, but a comparison that varies architecture, learning rate, batch size, and sequence length simultaneously does not support that attribution. A matched-configuration rerun (RoBERTa trained with the same LR/batch/sequence-length settings as the BabyLMs, or BabyLMs trained with RoBERTa's settings) or a series of ablations is needed before the data-efficiency claim can be accepted.","section":"Appendix A, Table 2"},{"comment":"The claimed gains are not uniform across tasks. For POS tagging, ELC-BERT is 87.7±0.5 versus RoBERTa 87.0±0.1 and MLSM 87.4±0.1; the ELC-BERT gain of 0.7 points is within one standard deviation of its own estimate, and the MLSM gain is 0.4 points. For NTC, both BabyLMs are below RoBERTa by 2.2–2.6 F1. Only the NER gains (ELC-BERT +3.2, MLSM +1.6) are clearly outside the reported noise. The sentence in Section 4 that 'both BabyLMs outperform the baseline on POS and NER' therefore overstates the evidence, and the paper should either report significance tests/effect sizes or temper the claim accordingly.","section":"Section 4, Table 1"},{"comment":"The mechanism analyses are suggestive but do not control for the confounds above, and they are based on a small number of hand-picked examples. Figure 3 shows semantic overlap for four named entities (two persons, two locations), and Appendix B extends this to four POS-tagged words; no quantitative measure of overlap is given for the full evaluation set, and there is no comparison with a RoBERTa-based semantic distribution. The claim that 'this pattern indicates that MLSM effectively encodes the semantic properties of these named entities, to which we attribute its NER performance gains' is therefore a post-hoc interpretation rather than an established causal link. The wording should be softened, or the analysis expanded to a systematic evaluation.","section":"Section 5, Figures 2 and 3"}],"minor_comments":[{"comment":"There is a typo in 'computate requirements' in the paragraph beginning 'ELC-BERT demonstrates superior efficiency'; it should read 'compute requirements'.","section":"Section 4"},{"comment":"The caption says 'We boldface best per-category performance and underline best overall,' but the table as rendered does not show underlining, making 'best overall' ambiguous. Please clarify how 'best overall' is defined and ensure the formatting matches the description.","section":"Table 1 caption"},{"comment":"Figure 1 plots downstream performance against pretraining epochs but does not show the standard deviations reported in Table 1; adding error bars or bands would help readers assess whether the early-training differences are meaningful.","section":"Figure 1"},{"comment":"The paper says the training scripts from the BabyLM submissions were used, but no URL or repository link is given. Providing the exact scripts and configurations would improve reproducibility.","section":"Appendix A"},{"comment":"The 'barbecue' example is English-only; adding an isiXhosa example would make the description of MLSM more directly connected to the paper's language of study.","section":"Section 2.2.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest and well-scoped, and the limitations section is exemplary. The main issue is that the headline comparison is confounded with hyperparameters, which is fixable with additional experiments or a substantial reframing of the claims. I would not reject: the dataset and evaluation choices are sound, and the NER gains may well survive a controlled comparison, but the manuscript in its current form does not support the architecture-specific conclusion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper for what it is: a clean, well-written empirical study that pretrains ELC-BERT and MLSM from scratch on 13m words of isiXhosa and evaluates them on three Masakha tasks. The numbers are new, the data and benchmarks are public, and the authors report standard deviations over five finetuning runs. That is a solid contribution to low-resource NLP, especially for under-resourced African languages.\n\nThe main soft spot is exactly where the stress-test points. The comparison in Table 1 is not architecture-controlled. Appendix A shows RoBERTa with LR 5e-5, sequence length 512, batch size 8, while ELC-BERT runs at LR 5e-4, sequence length 128, batch size 128, and MLSM at LR 1e-4, sequence length 128, batch size 64. That is a 10x learning-rate difference and a 4x difference in tokens per batch. The claimed +3.2 NER F1 for ELC-BERT over RoBERTa could easily come from those settings rather than from weighted residual connections or semantic prediction. The paper never tests this, and the abstract overstates the result by attributing the gains to data-efficient architectures. The POS gain (0.7 points) is within one standard deviation, and the NTC result reverses direction. So the evidence for an architecture effect is thinner than the framing suggests.\n\nTo the paper's credit, the authors are transparent about the hyperparameters and include honest limitations. They note the single language, the limited task set, and the data-quality gap. The qualitative analyses of layer weights and semantic categories are suggestive, not conclusive. None of this is fatal; the result is plausible and worth checking. But as reported, the paper is more accurately a comparison of training configurations than of architectures.\n\nI would send this to peer review, but conditionally. The reviewer should ask for a matched-configuration rerun: same learning rate, batch size, and sequence length across models, or at least a systematic ablation. If that is not feasible, the conclusion should be reframed as \"BabyLM recipes transfer to isiXhosa\" rather than \"BabyLM architectures beat vanilla pretraining.\" With that change, the paper would be a useful empirical data point for people working on low-resource LMs and for the BabyLM community.","headline":"Useful new benchmark numbers for isiXhosa with BabyLM architectures, but the headline claim is weakened by a real hyperparameter confound between the BabyLMs and the RoBERTa baseline.","tokens_in":9861,"tokens_out":1586,"would_cite":false,"duration_ms":15963,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BabyLMs trained on 13 million isiXhosa words beat a standard model on POS tagging and NER.","keywords":["BabyLM","data-efficient language modelling","low-resource languages","isiXhosa","ELC-BERT","masked latent semantic modeling","part-of-speech tagging","named entity recognition"],"falsifier":"Retrain the RoBERTa baseline with the BabyLM hyperparameters (learning rate 5e-4, batch size 128, sequence length 128) on the same WURA corpus and compare POS and NER scores; if the gap narrows to near zero, the architecture-specific claim is unsupported.","tokens_in":8810,"feed_emoji":"🗣️","tokens_out":5375,"duration_ms":43122,"temperature":0.7,"pith_summary":"The paper tries to establish that sample-efficient 'BabyLM' architectures, developed for a fixed English corpus, transfer to a genuinely low-resource language. Pretraining ELC-BERT and MLSM on a 13-million-word isiXhosa corpus yields better part-of-speech tagging and named-entity recognition than a vanilla RoBERTa model trained on the same data, and in some cases beats the large multilingual XLM-R model. The authors care because low-resource languages typically have far less text than even the BabyLM constraint, so architectures that learn well from small data could provide real gains. They also argue the remaining gap to the best multilingual models is caused by the absence of curated, developmentally plausible pretraining data for isiXhosa.","feed_headline":"BabyLMs beat vanilla model on isiXhosa with 13m words","feed_subtitle":"Small-data architectures lift POS tagging and NER, sometimes beating XLM-R, for a language with scarce corpora.","key_machinery":"The argument rides on two architectural mechanisms. ELC-BERT replaces standard residual connections with learned, layer-specific weights, so each layer combines previous layers' outputs in a weighted rather than equal way, which the authors link to stronger word-level representations and faster, more sample-efficient pretraining. MLSM replaces token-identity prediction with masked latent semantic modeling: a teacher model produces sparse semantic categories for masked tokens, and a student model is trained to predict those categories, learning coarser but more robust semantic structure that helps named-entity recognition. Both models are pretrained on the 13m-word WURA isiXhosa corpus, and evaluated on MasakhaPOS, MasakhaNER, and MasakhaNEWS.","core_discovery":"On the paper's own terms, the central discovery is that two BabyLM architectures pretrained from scratch on the WURA isiXhosa corpus outperform a RoBERTa baseline on the word-level tasks of POS tagging and NER: ELC-BERT reaches 87.7 POS accuracy and 88.6 NER F1, against RoBERTa's 87.0 and 85.4, while MLSM reaches 87.4 and 87.0. ELC-BERT also beats the XLM-R skyline on both of these tasks, and the authors attribute the gains to each architecture's specific mechanism for learning with limited data. The paper is careful to note that these models do not beat the strongest multilingual skylines on all tasks, and that the limiting factor appears to be pretraining-data quality rather than the architectures themselves.","pith_inferences":["Our inference: since only isiXhosa is tested, transfer to other low-resource languages is plausible but unproven.","Our inference: because the baseline and BabyLMs differ in learning rate, batch size, and sequence length, the true size of the architectural advantage is uncertain; a matched-hyperparameter comparison would sharpen the claim.","Our inference: the layer-weighting analysis suggests ELC-BERT's emphasis on the embedding layer is tied to syntactic information, hinting that the architecture could combine well with richer morphological pretraining for Bantu languages."],"forward_implications":["Data-efficient architectures can narrow the gap for low-resource languages without large pretraining corpora.","ELC-BERT's learned layer weighting appears to be the more transferable of the two mechanisms, matching its English BabyLM success.","Named-entity recognition benefits more than topic classification from these architectures, suggesting word-level tasks are the sweet spot.","Pretraining-data quality, not just architecture, determines how far these gains go; curated corpora remain a bottleneck.","Cross-lingual transfer models like XLM-R can be beaten on some tasks by smaller models trained from scratch on one language."],"supporting_citations":[{"why":"Defines the BabyLM challenge and supplies the premise that sample-efficient pretraining works on constrained English data, which this paper extends to isiXhosa.","marker":"Warstadt et al., 2023"},{"why":"Introduces ELC-BERT, the top-performing BabyLM architecture whose learned layer weighting is the paper's main object of study.","marker":"Georges Gabriel Charpentier and Samuel, 2023"},{"why":"Introduces masked latent semantic modeling, the MLSM pretraining objective the paper evaluates.","marker":"Berend, 2023b"},{"why":"Provides the WURA isiXhosa corpus, the 13m-word pretraining dataset all models in the paper are trained on.","marker":"Oladipo et al., 2023"},{"why":"Supplies the MasakhaPOS dataset used to evaluate part-of-speech tagging.","marker":"Dione et al., 2023"},{"why":"Supplies the MasakhaNER dataset used to evaluate named entity recognition, where the largest gains appear.","marker":"Adelani et al., 2022"},{"why":"Supplies the MasakhaNEWS news topic classification dataset, where BabyLMs do not beat the baseline.","marker":"Adelani et al., 2023"},{"why":"Provides XLM-R, the massively multilingual skyline that ELC-BERT beats on POS and NER.","marker":"Conneau et al., 2020"},{"why":"Defines the RoBERTa architecture used as the vanilla baseline pretrained on the same isiXhosa corpus.","marker":"Liu et al., 2019"}],"fun_headline_variants":["BabyLMs rival XLM-R on isiXhosa with just 13M words","Small-data BabyLMs beat vanilla, sometimes XLM-R","ELC-BERT tops XLM-R on isiXhosa POS and NER","13M words: BabyLMs outperform vanilla and sometimes XLM-R"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes the performance difference between the BabyLMs and the RoBERTa baseline is caused by architecture, but the models also differ in learning rate, batch size, and sequence length, so a confound is untested.","fun_headline_variants_meta":{"raw":{"variants":["BabyLMs rival XLM-R on isiXhosa with just 13M words","Small-data BabyLMs beat vanilla, sometimes XLM-R","ELC-BERT tops XLM-R on isiXhosa POS and NER","13M words: BabyLMs outperform vanilla and sometimes XLM-R"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000883,"raw_usage":{"total_tokens":3809,"prompt_tokens":933,"completion_tokens":2876,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":2790}},"tokens_in":549,"tokens_out":2876,"duration_ms":25115,"temperature":1.0,"reasoning_tokens":2790,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:44:49.004806+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the RoBERTa baseline with the BabyLM hyperparameters (learning rate 5e-4, batch size 128, sequence length 128) on the same WURA corpus and compare POS and NER scores; if the gap narrows to near zero, the architecture-specific claim is unsupported.","supporting_citations":[],"review_version":1}