{"id":"3fe45201-b3cd-4223-963d-ae373d66e994","arxiv_id":"2501.12660","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Simple knowledge distillation from mBERT produces smaller, faster Tagalog-only transformers that match the teacher on some tasks and lag on NER.","lead":"The authors distill a small Tagalog-only transformer from the multilingual BERT model using standard knowledge distillation, then fine-tune it on three Tagalog benchmarks. The distilled Base model runs about twice as fast as mBERT and matches it on hate speech and NLI, but is clearly worse on named entity recognition.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Efficiency claim omits distillation cost: reported speedups cover only downstream finetuning, so 'much more efficient' is not established for the full pipeline.","rationale":"The reader's strongest claim includes 'much more efficient,' and the reader's rationale states that the efficiency claim is well supported. I found that the efficiency claim rests on an omitted and potentially large cost: the distillation step itself. The paper reports only downstream finetuning runtimes, so the claimed speedup does not compare full pipelines. This is a load-bearing gap because the abstract promises efficiency for compute-scarce settings, and a reader deciding whether to adopt the method needs the total cost. The performance side also has a visible NER gap, but the efficiency omission is cleaner and directly testable. I therefore disagree with the reader's assessment that the efficiency claim is well supported, while agreeing that the overall verdict should remain conditional pending additional reporting. The proposed concrete test would resolve whether the concern lands or whether the efficiency claim survives after accounting for distillation.","tokens_in":6469,"tokens_out":7118,"duration_ms":77228,"concrete_test":"Report the wall-clock time and GPU-hours for the full distillation step (three epochs on the OSCAR Tagalog split with the frozen mBERT teacher), and combine it with the finetuning runtimes in Table 2 to form an end-to-end cost per task. Then compare against finetuning mBERT directly on the same hardware. If the total dBERT pipeline time is not below mBERT finetuning time for at least the Base model, the paper should revise 'much more efficient' to 'more parameter- and finetuning-efficient after a one-time distillation cost.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim has two parts: on-par performance and much greater efficiency. The efficiency part is currently unsupported because Table 2's 'Avg. Speed' measures only finetuning runtime of the downstream task. The method itself requires an additional front-loaded step: three epochs of distillation over the OSCAR Tagalog split, with a frozen mBERT teacher producing soft targets for masked tokens. No runtime, GPU-hours, FLOPs, or corpus size is reported for this step. For a researcher in a low-resource setting who needs one Tagalog model, the relevant comparison is mBERT finetuning versus distillation plus student finetuning. If the distillation step is expensive, the 1.97x and 5.23x finetuning speedups in Table 2 may not translate into end-to-end savings. The paper's own framing ('very simple and computationally cheap') makes this omission material rather than cosmetic, because the abstract explicitly promises efficiency. The missing number is easy to supply and would settle whether the headline efficiency claim survives.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a simple knowledge distillation method to extract language-specific Transformers from a multilingual BERT teacher. Using Tagalog as a case study, the authors distill two student models (dBERT Base and dBERT Tiny) on the OSCAR Tagalog corpus with a combination of KL divergence and MLM loss, then evaluate them on NER, hate speech detection, and NLI. The main claim is that these smaller single-language models perform on-par with strong baselines while being much more efficient during downstream finetuning. The paper also presents ablations on distillation data size, teacher conditioning, and student initialization, and discusses hypotheses about negative interference and embedding transfer.","tokens_in":6657,"tokens_out":4873,"duration_ms":44833,"significance":"If the efficiency claim were measured over the full pipeline and the performance comparisons included uncertainty estimates, this would be a practically valuable result for low-resource NLP: a simple, architecture-agnostic way to obtain a lightweight Tagalog model from mBERT, with speedups during finetuning and competitive accuracy on two of three tasks. The paper's strengths include the use of standard benchmarks, a clear baseline set, three relevant ablations, and a surprisingly non-obvious negative result that copying teacher embeddings hurts the distilled student. However, the significance as stated is reduced by the missing distillation cost, which is essential for a fair efficiency comparison, and by the absence of variance reporting that would support the claimed teacher-beating result. The limitations section candidly acknowledges the narrow scope (one MMT, one language), but this does not address the measurement gaps in the central claims.","major_comments":[{"comment":"The efficiency claim is not supported end-to-end. The abstract and Section 3 describe the method as 'much more efficient' and 'very simple and computationally cheap,' but Table 2's 'Avg. Speed' column reports only downstream finetuning runtime. The method requires a front-loaded distillation step: three epochs on OSCAR's Tagalog split with a frozen mBERT teacher producing soft targets (Section 2.1). The paper reports no corpus size, GPU-hours, FLOPs, or wall-clock time for this step. For a practitioner who needs one Tagalog model, the relevant comparison is mBERT finetuning versus distillation plus student finetuning. Please report the distillation cost and provide an end-to-end efficiency comparison, or explicitly amend the efficiency claim to refer only to finetuning.","section":"Section 2.1 and Table 2"},{"comment":"The performance claim that dBERT Base outperforms its teacher on hate speech classification by 1.86% (Section 3.1, Table 3) is based on a single run, with no seeds, variance, or significance test reported. Given typical finetuning variance, a 1.9-point advantage is not established as reliable. Additionally, the NER result in Table 2 (dBERT Base 0.8074 vs mBERT 0.8925) is an 8.5 F1 gap, which contradicts the 'on-par with strong baselines' claim in the abstract; the text in Section 3 describes this as 'lags slightly behind,' but the magnitude is substantial. Please provide multiple seeds with mean and standard deviation or significance tests, and qualify the on-par claim per task.","section":"Section 3.1 and Table 2"},{"comment":"The sentence 'Once we go down to half the training data, the original only outperforms the student model by 4.35% – a sub 1% degradation in performance!' is misleading. The degradation relative to mBERT is 4.35%, not sub 1%; the 'sub 1%' figure appears to refer to the incremental degradation between the 80% and 50% conditions (3.43% to 4.35%, a 0.92 percentage point increase), but this is not what the sentence says. Please rephrase to state the comparison explicitly, for example: 'the additional degradation from reducing the distillation data from 80% to 50% is under 1 percentage point.'","section":"Section 3.1"}],"minor_comments":[{"comment":"Equation (1) is malformed: the line breaks and the extra equals sign make the formula unreadable. It should read L_distil = α_KL * KL(out_student || out_teacher) + α_MLM * MLM(out_student, out_teacher).","section":"Section 2.1, Eq. (1)"},{"comment":"The values of α_KL, α_MLM, and the temperature parameter are not reported anywhere, so the distillation setup is not reproducible from the text alone.","section":"Section 2.1"},{"comment":"The paper does not report the size of the OSCAR Tagalog split used for distillation, which is important for interpreting both the efficiency claim and the data-amount ablation in Table 3.","section":"Section 2.1 and Table 3"},{"comment":"The runtime numbers are rounded down, and no hardware details are given; please state the GPU/CPU used and provide exact values or a note on source.","section":"Table 2"},{"comment":"In the Limitations section, 'high-resource langauge' is a typo for 'language.'","section":"Limitations"},{"comment":"The 'Avg. Speed' column averages speedups over tasks with very different runtimes; please define the averaging scheme and consider reporting per-task speedups as well.","section":"Table 2"},{"comment":"The hypotheses that 'negative interference' in the teacher explains the conditioning and initialization results are plausible but are not directly tested; a sentence clarifying that these are interpretations, not established mechanisms, would strengthen the exposition.","section":"Sections 3.2 and 3.3"}],"recommendation":"major_revision","confidential_remarks":"This is a well-written empirical study with a simple, potentially useful method. However, the central efficiency claim lacks the distillation cost, and the performance claims rest on single runs. These are fixable within the scope of the paper: report the distillation cost and multi-seed results. The paper's scope (one language, one MMT) is narrow but the authors acknowledge it in the Limitations section; if the requested measurements are added, the contribution would be suitable for a specialized NLP venue. I recommend major revision rather than rejection because the method and ablations are valuable and the missing evidence is straightforward to supply."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: this is a modest but honest empirical paper. The central claim — that you can distill mBERT into a small Tagalog-only student that finetunes about 2x (Base) and 5x (Tiny) faster than mBERT with only modest accuracy drops on two of three tasks — is supported by the tables. The ablations are the real value: teacher conditioning makes things worse, and copying or freezing teacher embeddings also makes things worse. Those negative results are worth knowing.\n\nWhat is genuinely new: not the KD recipe itself; Ansell et al. 2023 and Wibowo et al. 2024 already distill language-specific students. The new part is the controlled comparison of conditioning and initialization choices, and the counterintuitive findings that both hurt.\n\nThe soft spots are real but not fatal. The stress-test note is correct: Table 2's runtime only covers downstream finetuning. The distillation step costs three epochs over OSCAR's Tagalog split, but no GPU-hours or corpus size is given. That front-loaded cost is directly relevant to the 'much more efficient' framing, and it's an easy number to supply. Second, there are no seeds or error bars anywhere. The 1.86% accuracy edge over mBERT on hate speech is well within noise; I wouldn't put any weight on it. The NER gap (0.807 vs 0.892) is the more honest summary: the student roughly matches on classification and NLI, but clearly lags on token-level NER. 'On-par' is too strong. Third, Section 3.1's 'sub 1% degradation' is misleading — the accuracy drop from 80% to 50% data is 0.92%, but that is the change in the gap to mBERT, not the total gap. Also, no code or hyperparameters are released.\n\nThe limitations section is unusually honest about the single language and single teacher, which I credit. The writing is clear.\n\nWho this is for: anyone doing low-resource NLP, especially Tagalog/Filipino, and people working on distillation. It deserves serious peer review; the missing runtime and error bars are fixable in revision. I'd recommend sending it out, with a request to add distillation cost, report variance, and soften the 'on-par' claims. Not a reading group pick for me, but a reasonable one for a low-resource or systems group.","headline":"A useful empirical study with real negative results, but the efficiency claim needs the distillation cost and error bars before the headline fully lands.","tokens_in":7200,"tokens_out":2771,"would_cite":false,"duration_ms":27427,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Simple distillation turns a multilingual BERT into a faster Tagalog-only model that matches or beats the teacher on some tasks.","keywords":["knowledge distillation","low-resource languages","Tagalog","multilingual transformers","mBERT","efficient language models","single-language models"],"falsifier":"Run the identical recipe on another low-resource language in mBERT, using a benchmark of comparable difficulty; if the distilled student does not match the teacher within a few points or does not yield the training-speed advantage, the general claim that this simple distillation extracts robust single-language models fails. The paper itself already shows one such limit in NER, so a broader benchmark where the student trails consistently would falsify the 'on-par with strong baselines' claim.","tokens_in":6257,"feed_emoji":"⚡","tokens_out":6364,"duration_ms":52581,"temperature":0.7,"pith_summary":"The paper proposes taking a frozen multilingual BERT (mBERT), distilling it into a smaller, Tagalog-only transformer using masked-language-model-style distillation on Tagalog text, and then finetuning that student on downstream tasks. The authors' central claim is that this simple recipe produces single-language models that perform on par with strong baselines on Tagalog benchmarks while training much faster. On hate-speech classification the distilled base model (77.29% accuracy) even beats its teacher mBERT (75.43%), and it trails the full Tagalog-RoBERTa only slightly on NLI; the tiny variant trades more accuracy for a 5.23x average training speedup. The result matters because researchers working on low-resource languages often lack the compute to finetune large multilingual models for just one language.","feed_headline":"Distilled Tagalog BERT matches its teacher at double speed","feed_subtitle":"A small language-specific student distilled from mBERT trains twice as fast and still matches strong baselines on Tagalog tasks.","key_machinery":"The core object is the distillation setup: a frozen mBERT teacher and a blank student (dBERT Base or dBERT Tiny) trained on OSCAR's Tagalog split with $L_{\\mathrm{distil}} = \\alpha_{\\mathrm{KL}} \\mathrm{KL}(\\mathrm{out}_{\\mathrm{student}} \\| \\mathrm{out}_{\\mathrm{teacher}}) + \\alpha_{\\mathrm{MLM}} \\mathrm{MLM}(\\mathrm{out}_{\\mathrm{student}}, \\mathrm{out}_{\\mathrm{teacher}})$, plus a temperature to soften the logits. This transfers the teacher's Tagalog behavior into a model with fewer layers or a smaller hidden dimension, and the downstream comparison isolates whether the extracted model keeps the teacher's language knowledge without the interference and cost of the full multilingual model.","core_discovery":"On three Tagalog tasks, a blank student transformer distilled from frozen mBERT with a mix of KL-divergence and masked-language-model loss—dBERT Base—matches or beats the multilingual teacher on hate-speech classification and natural-language inference while training about twice as fast; it lags clearly on named-entity recognition. A smaller dBERT Tiny gives a 5.23x average speedup but only stays competitive on hate-speech classification. The paper also finds that reducing distillation data to 50% costs less than a point of accuracy beyond the 80% reduction, that conditioning the teacher with extra MLM before distillation hurts the student, and that copying the teacher's embeddings into the student hurts NER performance.","pith_inferences":["Editorial: If the distillation is effectively compressing mBERT's Tagalog subspace while discarding other-language parameters, the same recipe should transfer to other low-resource languages in mBERT, provided the teacher has nontrivial knowledge of that language; testing on a language with weaker teacher performance would separate data sufficiency from teacher quality.","Editorial: The negative results for embedding copying suggest that the student benefits from learning a fresh Tagalog-focused embedding space; a testable extension is comparing distillation with randomly initialized versus teacher-initialized embeddings across several target languages to see whether the finding is Tagalog-specific.","Editorial: The authors do not report the size of the OSCAR Tagalog split or the loss weights, so the strongest version of their claim—that three epochs is always enough—has not yet been separated from corpus-size effects; a controlled sweep over distillation length and corpus size would define the actual boundary.","Editorial: The same distillation target could be evaluated on generative or instruction-following multilingual models, which the authors list as future work; if instruction-following behavior survives the transfer, the method would give low-resource communities a faster path to usable language-specific assistants."],"forward_implications":["If the recipe holds, low-resource research groups can get a language-specific transformer from an existing MMT in three epochs of distillation and then finetune it at roughly half the training time.","Because 50% of the distillation corpus retains most of the hate-speech accuracy, the data requirement for distillation appears smaller than the full OSCAR split; the authors connect this to how much Tagalog the teacher already knows.","Teacher conditioning and embedding initialization, two common enhancements in distillation work, are shown to hurt rather than help when the goal is a single-language student, so the simple path is also the better path for this setting.","The small variant's limits on NER and NLI suggest capacity, not just distillation time, constrains how much of a multilingual teacher's knowledge can be extracted.","The method is presented as a general-use base for further task-specific work, not just a one-off finetuned model."],"supporting_citations":[{"why":"Provides the mBERT teacher model that is frozen and distilled.","marker":"Devlin et al., 2019"},{"why":"Supplies the teacher-student distillation objective used to train the student.","marker":"Hinton et al., 2015"},{"why":"Supplies the OSCAR Tagalog corpus used as distillation training data.","marker":"Ortiz Suárez et al., 2019"},{"why":"Provides Tagalog-RoBERTa, the full monolingual baseline, and the TLUnified resource family.","marker":"Cruz and Cheng, 2022"},{"why":"Provides DistilmBERT, the distilled multilingual baseline, and the embedding-initialization practice the authors ablate.","marker":"Sanh et al., 2020"},{"why":"Provides the Hatespeech Filipino dataset used for the main accuracy comparison.","marker":"Cabasag et al., 2019"},{"why":"Provides the NewsPH NLI dataset used to evaluate entailment.","marker":"Cruz et al., 2021"},{"why":"Provides the TLUnified NER dataset where the distilled models show their largest gap.","marker":"Miranda, 2023"},{"why":"Represents the layer-copying initialization tricks the authors deliberately avoid and compare against.","marker":"Jiao et al., 2020"}],"fun_headline_variants":["Distilled Tagalog BERT trains 2x faster, matches on 2 tasks","Distilled Tagalog BERT beats teacher on hate speech and NLI","Small Tagalog BERT from distillation rivals mBERT, 2x faster","Distilled Tagalog BERT: 2x speed, but NER lags"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result rests on the assumption that three epochs of distillation over the OSCAR Tagalog split are enough to transfer mBERT's Tagalog knowledge, even though the paper does not report the corpus size or the loss weights that define those epochs.","fun_headline_variants_meta":{"raw":{"variants":["Distilled Tagalog BERT trains 2x faster, matches on 2 tasks","Distilled Tagalog BERT beats teacher on hate speech and NLI","Small Tagalog BERT from distillation rivals mBERT, 2x faster","Distilled Tagalog BERT: 2x speed, but NER lags"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001332,"raw_usage":{"total_tokens":5337,"prompt_tokens":784,"completion_tokens":4553,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":400,"completion_tokens_details":{"reasoning_tokens":4467}},"tokens_in":400,"tokens_out":4553,"duration_ms":34598,"temperature":1.0,"reasoning_tokens":4467,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T16:55:24.211123+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical recipe on another low-resource language in mBERT, using a benchmark of comparable difficulty; if the distilled student does not match the teacher within a few points or does not yield the training-speed advantage, the general claim that this simple distillation extracts robust single-language models fails. The paper itself already shows one such limit in NER, so a broader benchmark where the student trails consistently would falsify the 'on-par with strong baselines' claim.","supporting_citations":[],"review_version":1}