{"id":"dfd0b1c6-3863-4fe8-af21-439edd81da67","arxiv_id":"2412.03098","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A dynamically updated surprisal-based curriculum, run over a small ELC-BERT model, improves BabyLM 2024 EWOK and GLUE scores over official baselines but hurts BLiMP.","lead":"This paper tests a curriculum learning method called ACLM, where a language model chooses its own next training examples based on which sentences it is most surprised by, and applies it to a compact BERT model for the BabyLM 2024 challenge. The updated method improves scores on world-knowledge and common-sense benchmarks compared to official baselines, while grammar-focused scores drop.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The causal claim that ACLM drives EWOK/GLUE gains is undercut by an under-specified control: ACLM's effective batch (Table 2: 512×4) and training steps are not matched to the ELC-BERT B512 row, so the gap could be a schedule artifact, not the curriculum.","rationale":"The reader's conditional verdict is appropriately cautious: the paper's own Limitations section disclaims statistical significance testing, and the reported gains rest on single runs. However, the sharpest problem is not only missing confidence intervals but the absence of a controlled comparison: the ACLM runs and the non-ACLM ELC-BERT Baselines differ in unspecified training-schedule details. Table 2 provides hyperparameters for ACLM only, including batch size 512, gradient accumulation 4, and 31,250 training steps; the analogous values for the B32 and B512 controls are not stated. Because Algorithm 2 repeatedly trains on a growing active set, the number of optimizer steps and effective batch size may differ from the control, making it impossible to know whether the EWOK/GLUE advantage is a property of the surprisal-based curriculum or simply of a different effective training schedule. The reader's proposed fix (multiple seeds and significance tests) is necessary but not sufficient: a statistically significant gap between ACLM and an unmatched control would still not identify the causal ingredient. The concrete test above, a compute-matched random-order baseline and a random-selection ACLM variant, would settle whether the central causal claim survives. This does not change the conditional verdict; it sharpens the condition under which the paper's claim should be accepted.","tokens_in":6911,"tokens_out":8962,"duration_ms":88711,"concrete_test":"Run ELC-BERT with ACLM's exact effective configuration (batch size 512, gradient accumulation 4, total optimizer steps 31,250) but with ordinary random-order training over the full corpus, using the same optimizer and learning-rate schedule. If this matched-compute control reaches ACLM's EWOK/GLUE scores, the curriculum effect is not established. Additionally, run ACLM with random kNN selection under the identical schedule to isolate whether the surprisal criterion, rather than the active-set growth pattern, drives the improvement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is that the non-ACLM ELC-BERT B32/B512 rows are valid 'very similar settings' controls for ACLM. The paper never reports the gradient-accumulation values, total optimizer steps, or number of epochs for those control runs; Table 2, labeled 'for ACLM models,' lists batch size 512, gradient accumulation 4, and 31,250 training steps. If B512 uses accumulation 1 (effective batch 512) while ACLM uses effective batch 2048, or if ACLM runs more total steps because Algorithm 2 trains for n_epochs each iteration, the higher EWOK (70.0–72.1 vs. 64.9–65.2) and GLUE (63.5–65.7 vs. 61.0–63.4) scores could be caused by compute or batch-size differences rather than by the dynamic surprisal curriculum. The paper's Limitations section explicitly disclaims significance testing, and the B32/B512 baselines are single runs; both issues compound the configurational confound. Without a matched-compute random-order control, the abstract's 'outperform' claim is not attributable to ACLM.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an updated Active Curriculum Language Modeling (ACLM) system for the BabyLM 2024 strict-small track. The method iteratively selects training items using the model's own surprisal values, with a dynamically recomputed surprisal space, and is applied to ELC-BERT. The authors report results on BLiMP, EWOK, and GLUE (Table 1), showing that ACLM variants outperform both the official BabyLM 2024 baselines and their non-ACLM ELC-BERT runs on EWOK and GLUE, while underperforming on BLiMP. They also discuss batch-size sensitivity and mention several limitations, including the lack of significance testing.","tokens_in":7346,"tokens_out":2516,"duration_ms":27588,"significance":"If the causal claim holds, the paper provides evidence that a learner-directed, dynamically updated curriculum can improve world-knowledge and common-sense benchmarks under data-constrained conditions, which is directly relevant to the BabyLM line of work and to curriculum learning more broadly. The manuscript is transparent about its limitations and makes code publicly available, which supports reproducibility. However, the central comparative claim currently rests on single runs without significance testing and without fully matched training schedules, so the significance is conditional on additional experimental support.","major_comments":[{"comment":"The central claim that ACLM improves over the non-ACLM ELC-BERT runs and over the official baselines on EWOK and GLUE is not supported by significance testing or multiple random seeds. The Limitations section explicitly states that 'We do not have a full analysis of the statistical significances of the differences in the scores,' and Table 1 reports single-run accuracy values. Observed gaps such as EWOK 70.0–72.1 versus 64.9–65.2 and GLUE 63.5–65.7 versus 61.0–63.4 may be within run-to-run variance; the authors should report confidence intervals, bootstrap estimates, or results over several seeds before attributing these differences to the ACLM curriculum.","section":"§4, Table 1; Limitations"},{"comment":"The non-ACLM ELC-BERT B32 and B512 runs are not documented as controls in sufficient detail. Table 2 lists hyperparameters for the ACLM models only, including batch size 512, gradient accumulation 4, and 31,250 training steps; the gradient-accumulation values, effective batch sizes, total optimizer steps, and number of epochs for the B32 and B512 runs are not reported. Since Algorithm 2 trains for n_epochs per iteration and the value of n_epochs is not given, the ACLM runs may have a different effective batch size (512×4 = 2048) or a different total number of updates than the B512 control. The paper should provide a matched-compute or matched-effective-batch random-order control to support the claim that the improvements are due to the curriculum rather than to schedule or compute differences.","section":"§4, Table 2, Algorithm 2"},{"comment":"The text states that 'the overall consistency of outperformance of ACLM on the filtered BLiMP suggests that ACLM is having an effect,' but Table 1 shows differences of only about one or two points (e.g., filtered BLiMP ACLM-D7 51.3 versus B512 49.1 and B32 47.9) with no significance testing and no seed variation. This statement overstates the evidence and should be removed or supported with statistical analysis.","section":"§4, BLiMP paragraph"},{"comment":"The abstract's claim that 'they outperform the BabyLM 2024 official baselines on common-sense and world-knowledge tasks' is accurate only for the specific single runs reported in Table 1; as written, it can be read as a general claim. The abstract should qualify the result as applying to the runs in this study, pending the significance and control analyses described above.","section":"Abstract and §4"}],"minor_comments":[{"comment":"There are several typos and inconsistent spellings: 'surpisal' in the Figure 1 caption, 'suprisal' alongside 'surprisal' in Section 2, 'peturbations' in the Introduction, and 'acquistion' in the Introduction. These should be corrected.","section":"Abstract, Section 2, Figure 1 caption"},{"comment":"Table 2 is labeled 'Small (Submitted Model)' but it is unclear whether the same hyperparameters apply to all four ACLM variants (D7, D32, D64, D128) or whether the surprisal-space dimensionality affects any listed hyperparameters. The table caption and text should clarify this.","section":"Table 2 heading"},{"comment":"The parenthetical remark about a reviewer suggesting a switch in averaging procedure is informal for a journal article; consider moving such discussion to a more explicit comparison of evaluation pipelines or removing it.","section":"Section 2, footnote 2"}],"recommendation":"major_revision","confidential_remarks":"This manuscript is honest about its limitations, but the headline comparison is not yet statistically grounded. For a journal-level archival publication, the authors would need to add multiple seeds or significance testing and, crucially, a matched effective-batch and compute control. For a shared-task system report, the current level of evidence might be acceptable, but for this venue I recommend major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is small but real: the authors update their ACLM procedure so the surprisal space is recomputed from the model itself each iteration, and they run it over ELC-BERT for BabyLM 2024. The EWOK and GLUE numbers beat the official baselines and their own non-ACLM ELC-BERT runs, and the effect is consistent across all four surprisal-space dimensionalities. That is a reasonable empirical contribution to the BabyLM line, and the paper is written honestly—it flags the lack of significance testing, the English-only scope, and the batch-size sensitivity of BLiMP. Credit is also due for releasing code. The soft spot is not that they lack significance tests; for a workshop shared task that is a common and acknowledged limitation. The deeper problem is that the ACLM runs and the non-ACLM controls are not matched on compute. Table 2 reports ACLM with batch size 512 and gradient accumulation 4, so effective batch 2048, with 31,250 training steps. The ELC-BERT B512 row is presented as the 'very similar settings' control, but the paper never tells us its accumulation value or total steps. If B512 uses accumulation 1 (effective batch 512), or if Algorithm 2's multiple epochs per iteration push ACLM past 31,250 effective steps, then the EWOK and GLUE gaps could simply be a batch-size or compute effect. The authors themselves show that batch size matters a lot for BLiMP, so it is not safe to assume it does not matter for EWOK and GLUE. To claim the curriculum drives the improvement, they need a random-order control trained with the same effective batch, step count, and learning-rate schedule. That said, the paper does not hide these weaknesses. The Limitations section is candid, and the discussion of batch-size sensitivity is fair. The central claim—ACLM improves EWOK/GLUE—is plausible but unproven as stated. Given the shared-task context, I would not desk-reject it, but a serious referee should ask for matched-compute controls and at least a few seeds before the comparison is treated as evidence about curriculum learning. This paper will mainly interest people working on curriculum learning or on squeezing performance from small models. I would not cite it for the causal claim, but I might cite it as an example of a dynamic surprisal-based curriculum. Worth a reading-group slot for the methodology, not for the headline.","headline":"Modest, honest BabyLM system paper whose headline EWOK/GLUE gains are real but not yet attributable to ACLM because the controls are confounded on effective batch size.","tokens_in":714,"tokens_out":835,"would_cite":false,"duration_ms":24835,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a dynamically updated surprisal curriculum, driven by the model's own uncertainty, improves a strong small language model on world-knowledge and common-sense tasks over official baselines.","keywords":["active curriculum learning","surprisal oracle","BabyLM","ELC-BERT","sample-efficient language modeling","learner-directed curriculum","world knowledge evaluation","dynamic curriculum"],"falsifier":"Re-run the ACLM and non-ACLM ELC-BERT models under identical settings for, say, ten random seeds, compute confidence intervals on EWOK and GLUE accuracies, and check whether the ACLM advantage over its own control persists. If the gap shrinks to within the noise, the central claim that the dynamic surprisal curriculum drives the improvement collapses; likewise, if a control with randomly selected batches under the same update schedule matches ACLM, then the specific surprisal-based selection is not the cause.","tokens_in":6746,"feed_emoji":"🧠","tokens_out":7019,"duration_ms":56254,"temperature":0.7,"pith_summary":"The paper argues that letting a language model steer its own training order, by repeatedly re-measuring which sentences in its pool are most surprising to it at that moment, can push a strong small architecture to better world-knowledge and common-sense scores. This is the 'dynamic surprisal oracle': each iteration the model picks its least-certain training item, finds pool sentences with similar surprisal profiles, adds them to the active set, and rebuilds the surprisal space from its current state. The authors apply this procedure on top of ELC-BERT and report consistent gains over the official BabyLM 2024 baselines (LTG-BERT and BabyLlama) on EWOK and GLUE, alongside weaker grammatical-inference scores on BLiMP. A reader should care because the result suggests that, independent of architecture and data volume, the order in which a learner encounters data is a handle worth controlling. The authors are careful to note that without significance testing the score gaps are provisional.","feed_headline":"Surprisal-driven curriculum lifts small BERT on world-knowledge tests","feed_subtitle":"The model's own surprisal re-orders training each round; on EWOK and GLUE it beats the 2024 baselines and its own non-ACLM runs.","key_machinery":"The mechanism is the active curriculum itself: at each iteration, the model under training (ELC-BERT) computes per-sentence average surprisal over its active training set, selects the sentence with the highest surprisal as the 'least certain' item, and then uses $k$-nearest-neighbours on the surprisal vectors of the as-yet-unseen pool to pull in the most similar items, which join the training set for the next epoch. The update from the previous year's ACLM is that the surprisal space is re-computed every iteration from the current model rather than fixed in advance from a trigram model, making the curriculum dynamically learner-directed. The surprisal vectors are reduced to 7, 32, 64, or 128 dimensions, and the paper finds that this dimensionality choice makes little difference once the input length is uniformly 128 tokens.","core_discovery":"The central claim is that a learner-directed curriculum, updated by the model's own changing notion of surprisal, improves a strong data-constrained language model on world-knowledge and common-sense tasks relative to passive training. Concretely, ACLM over ELC-BERT scores 70.0–72.1 on EWOK against 48.9 (LTG-BERT) and 50.7 (BabyLlama), and 63.5–65.7 on GLUE against 60.3 and 63.3, while the non-ACLM ELC-BERT runs sit at 64.9–65.2 on EWOK and 61.0–63.4 on GLUE. The authors interpret this as evidence that controlling the order of learning can matter, and they connect it speculatively to developmental dependencies among tasks; they also report that the same models underperform on fine-grained grammatical inferences (BLiMP), which they attribute mostly to the much smaller batch size they used in all their runs. They do not claim to have a complete account of why the curriculum helps, and they explicitly flag the absence of significance tests, multiple-seed averages, and an exploration of hyperparameter space as limitations.","pith_inferences":["A testable extension the paper does not run: apply ACLM on top of the original full-size ELC-BERT configuration rather than the batch-size-limited runs, to see whether the curriculum still improves EWOK and GLUE when the base model is not handicapped.","The paper's cognitive reading suggests a direct probe: inspect the actual order of sentences selected by ACLM and test whether it correlates with human developmental orderings (e.g., concreteness or syntactic complexity), which the paper only speculates about.","If the EWOK and GLUE gains survive significance testing, the surprisal criterion may be capturing a general 'zone of proximal development' signal that could be combined with other informativeness measures, since the authors note that a single measure is unlikely to be optimal."],"forward_implications":["Learning order is a real lever: the same architecture, data, and batch size produce better EWOK and GLUE scores when the training set is curated by the model's own surprisal rather than fixed in advance.","The curriculum can be a portable add-on: ACLM is model-agnostic in principle, and here it transfers to a different base architecture (ELC-BERT) from last year's RoBERTa-based system.","Hyperparameter sensitivity of strong baselines can confound curriculum comparisons: the non-ACLM ELC-BERT runs underperform on BLiMP, which the authors attribute to batch size, so future comparisons need matched hyperparameters to isolate the curriculum effect.","The dimensionality of the surprisal signature may not matter when input lengths are uniform, so future work should test variable-length inputs where the reduction scheme could bite."],"supporting_citations":[{"why":"Defines the original ACLM procedure whose update this paper implements.","marker":"Hong et al., 2023"},{"why":"Supplies the ELC-BERT architecture and training recipe that this paper builds on.","marker":"Charpentier and Samuel, 2023"},{"why":"Provides the LTG-BERT baseline that the paper compares against.","marker":"Samuel et al., 2023"},{"why":"Defines the BabyLM 2024 strict-small task and its evaluation setup.","marker":"Choshen et al., 2024"},{"why":"The BLiMP benchmark used for grammatical-inference evaluation.","marker":"Warstadt et al., 2020"},{"why":"The EWOK benchmark for world-knowledge evaluation.","marker":"Ivanova et al., 2024"},{"why":"The GLUE benchmark used for common-sense and entailment evaluation.","marker":"Wang et al., 2018"},{"why":"Provides the active curriculum learning framing that ACLM adapts by removing the human labeler.","marker":"Jafarpour et al., 2021"}],"fun_headline_variants":["Surprisal-based curriculum beats baselines on world-knowledge tasks","Active curriculum lifts BERT on EWOK and GLUE but not grammar","Model's own surprisal reorders training, boosting common-sense scores","Learner-directed training order improves small BERT on world knowledge","Surprisal-driven ACLM outperforms passive training on EWOK"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reported score gaps between ACLM and its non-ACLM control reflect the curriculum rather than run-to-run noise or evaluation details, since the paper reports no significance tests, confidence intervals, or multiple-seed averages.","fun_headline_variants_meta":{"raw":{"variants":["Surprisal-based curriculum beats baselines on world-knowledge tasks","Active curriculum lifts BERT on EWOK and GLUE but not grammar","Model's own surprisal reorders training, boosting common-sense scores","Learner-directed training order improves small BERT on world knowledge","Surprisal-driven ACLM outperforms passive training on EWOK"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000233,"raw_usage":{"total_tokens":1515,"prompt_tokens":986,"completion_tokens":529,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":602,"completion_tokens_details":{"reasoning_tokens":434}},"tokens_in":602,"tokens_out":529,"duration_ms":5253,"temperature":1.0,"reasoning_tokens":434,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:45:58.414760+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the ACLM and non-ACLM ELC-BERT models under identical settings for, say, ten random seeds, compute confidence intervals on EWOK and GLUE accuracies, and check whether the ACLM advantage over its own control persists. If the gap shrinks to within the noise, the central claim that the dynamic surprisal curriculum drives the improvement collapses; likewise, if a control with randomly selected batches under the same update schedule matches ACLM, then the specific surprisal-based selection is not the cause.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the original ACLM procedure whose update this paper implements."},{"cited_title":"Hu, Tal Linzen, Aaron Mueller, Candance Ross, Alex Warstadt, Ethan Wilcox, Adina Williams, and Chengxu Zhuang","cited_arxiv_id":null,"evidence_quote":"Defines the BabyLM 2024 strict-small task and its evaluation setup."}],"review_version":1}