{"id":"77b425dc-dbc2-4d56-b919-12ee9ff230f5","arxiv_id":"2412.13922","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Adapting Llama-3.1-8B to Basque through continual pre-training, translated instruction tuning, and DPO yields state-of-the-art Basque NLU and instruction-following for sub-10B models.","lead":"This paper tests how to adapt the Llama-3.1-8B English model to Basque by adding Basque text, translated instructions, and preference data. It reports big gains and produces the best Basque sub-10B models, but the headline results rely on a small manual evaluation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 24-point instruction-following claim rests entirely on a 100-item test scored by one native speaker with no reliability data; the 7-point DPO gain is within sampling noise, so the alignment-stage conclusion is not yet supported.","rationale":"The reader's conditional verdict is well calibrated. The NLU claim is supported by 12 objective benchmarks with consistent gains, so the continual-pretraining result is credible. The instruction-following half rests on a 100-item, single-annotator manual evaluation, which the paper's own Limitations section explicitly hedges. I identify the same weak spot but sharpen it: even before considering annotator subjectivity, several key stage-wise differences in Tables 3 and 4 are within sampling noise. The DPO 7-point improvement and the 9-point benefit of Basque pre-training for instruction tuning are not statistically significant at n=100. This does not invalidate the headline 24-point gap, but it does mean the alignment-stage conclusion is not currently supported. Since the reader already set CONDITIONAL for essentially this reason, my read does not change the verdict. If the proposed re-scoring shows poor agreement or reverses the ordering, the instruction-following and DPO conclusions should be substantially weakened, while the NLU claims may still stand.","tokens_in":18942,"tokens_out":11061,"duration_ms":107840,"concrete_test":"Have two additional native Basque speakers, blinded to model identity, independently score the same 100 model outputs using the paper's rubric; compute per-model correct rates and Cohen's kappa. If kappa is below 0.6 or the re-scored 6% versus 30% ordering does not survive, the instruction-following headline and the DPO-stage conclusion should be downgraded. As a secondary analytical check, report Fisher exact p-values for the 23% versus 30% and 14% versus 23% comparisons.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is the manual instruction-following evaluation in Section 4.2 (Tables 3 and 4). One native Basque speaker classified 100 sampled No_Robots instructions as correct/partially correct/wrong; no inter-annotator agreement, no confidence intervals, and no blinded re-scoring are reported. The 'correct' category requires a subjective judgment that a response 'fully addressed' the task and introduced no hallucinations, so absolute rates depend on one annotator's leniency. The Limitations section itself says the conclusions 'should be interpreted within the context of the nature and size of this test set.' A separate, annotator-independent problem is statistical: with n=100, the DPO 23% to 30% gain and the Basque-pre-training 14% to 23% gain are both within sampling error (Fisher exact p>0.1), so the stage-wise claims for alignment and for the benefit of pre-training are underpowered. The headline 6% to 30% difference is significant, but the alignment-stage contribution is not. Without releasing the 100 outputs or providing a reliability check, the 24-point claim and especially the 7-point DPO claim are not auditable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes and evaluates a three-stage pipeline for adapting Llama-3.1-8B to Basque: (1) continual pre-training on about 521 million words of Basque (ZelaiHandi) mixed 80:20 with English (FineWeb); (2) LoRA instruction tuning on machine-translated No_Robots and SlimOrca datasets; (3) DPO alignment on machine-translated UltraFeedback. Using 12 Basque NLU benchmarks, the authors report a +12.47 average improvement over Llama-3.1-8B (from 48.75 to 61.22). For instruction-following, they rely on a 100-instruction manual test scored by a single native Basque speaker into correct/partially correct/wrong categories and report a 24-point improvement in the correct rate (6% for Llama-3.1-8B-instruct to 30% for Llama-eus-8B-instruct-DPO). The paper introduces four manually translated benchmark subsets and new Basque training corpora (No_Robots_eu, SlimOrca_eu, UltraFeedback_eu).","tokens_in":19193,"tokens_out":11889,"duration_ms":96061,"significance":"If the results hold, the NLU improvement is consistent across all 12 benchmarks and provides a practically valuable sub-10B Basque model, with the released datasets and evaluation subsets being useful resources for the Basque NLP community. The paper also gives a detailed and reproducible account of training configurations, mixing ratios, and carbon footprint. However, the instruction-following claims are built on a small, single-annotator manual evaluation with no reliability or significance testing; several stage-wise incremental gains are within sampling noise. The 'state-of-the-art' and 'first instructed Basque LLM' assertions require a broader comparison and are not yet fully supported. The central NLU contribution is credible, but the alignment-stage conclusions need substantial strengthening or rephrasing.","major_comments":[{"comment":"The manual instruction-following evaluation uses 100 instructions from the No_Robots test set scored by one native Basque speaker into correct/partially correct/wrong. The paper reports no inter-annotator agreement, no confidence intervals, and no evidence of blinded scoring; the rubric requires subjective judgments about whether a response fully addressed the task and introduced hallucinations. This makes the 24-point improvement and the 7-point DPO gain non-auditable. The authors should publish the 100 inputs and all model outputs, obtain at least a second annotation on a random subset, and report a reliability measure such as Cohen's kappa. Without this, the headline instruction-following claims cannot be independently verified.","section":"Section 4.2, Tables 3-4"},{"comment":"With n=100, the incremental improvements between consecutive pipeline stages are not statistically supported. The DPO gain (23% to 30% correct) and the benefit of Basque pre-training for instruction tuning (14% to 23%) are not significant (Fisher exact two-tailed p>0.1 and p≈0.14, respectively); the 15% to 23% difference between No_Robots_eu and SlimOrca_eu is also non-significant. The paper labels these effects as 'significant benefits' and states that translated preference data are 'viable' based on these differences. Please report exact p-values or confidence intervals and temper the stage-wise conclusions; only the full-pipeline comparison (6% vs 30%) appears statistically robust.","section":"Section 4.4 and 5.2, Tables 3-4"},{"comment":"The paper claims that Llama-eus-8B-instruct is 'the first instructed LLM for Basque' and achieves state-of-the-art performance among sub-10B models. The related work discussion only treats Latxa (Etxaniz et al., 2024) as a family of foundational models, but the cited Latxa work also describes instruction-tuned variants. If such models exist, they must be included in the instruction-following comparison and the 'first' claim must be revised. Please clarify whether any instruction-tuned Basque model predates this work and, if so, compare against it.","section":"Introduction and Section 2"},{"comment":"The abstract attributes the '24-point improvement in instruction-following performance' to 'instruction tuning and human preference alignment,' but the 6% baseline is Llama-3.1-8B-instruct and the 30% result also includes the effect of continual pre-training, which alone accounts for a 9-point gain over Llama-3.1-8B+SlimOrca_eu. Thus the 24-point gain is the cumulative effect of the full pipeline, not solely the instruction-tuning and alignment stages. The wording should be corrected to avoid this misattribution.","section":"Abstract and Section 6"}],"minor_comments":[{"comment":"There are typos in this section, including 'intruction tunining' and 'intructions datasets'; these should be corrected.","section":"Section 4.4"},{"comment":"For the four newly translated subsets (ARC_HT_eu_sample, Winogrande_HT_eu_sample, MMLU_HT_eu_sample, HellaSwag_HT_eu_sample), please report the sample sizes in the table and, if possible, 95% confidence intervals, so that the reader can assess the precision of the 12.47 average gain.","section":"Section 3.3, Table 1"},{"comment":"The reported machine translation quality (19.3 BLEU and 52.2 chrF++) is not discussed; a short analysis of how translation errors might affect the downstream instruction-tuning results would strengthen the paper.","section":"Section 4.1"},{"comment":"The unqualified phrase 'state-of-the-art among sub-10B parameter models' should be scoped to 'among the compared baselines' or supported by a broader comparison (e.g., Qwen2, Gemma, or other open sub-10B models).","section":"Introduction and Conclusions"},{"comment":"The Basque example for the Summarization category contains a likely typo ('nitxoko' instead of 'nitzat' or a similar form); please proofread all Basque examples in the appendix.","section":"Appendix E"},{"comment":"The OpenOrca reference has a malformed URL ('https://https://huggingface.co/...') and should be fixed.","section":"References"},{"comment":"The caption should explicitly explain the lighter-color shading for 'partially correct' answers, and the figure should be checked for legibility in print.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The NLU portion of the paper is solid and the released benchmark subsets and training corpora would be valuable to the Basque NLP community. The main weakness is the instruction-following evaluation: a 100-item, single-annotator test with no reliability or significance analysis cannot support the current stage-wise claims or the 'state-of-the-art' and 'first instructed Basque LLM' assertions. The authors' own Limitations section acknowledges the small test set, but the abstract and conclusions do not carry that caveat. I recommend a revision that adds a second annotation, confidence intervals, and exact statistical tests, or substantially softens the claims. Also, please verify the existence of instruction-tuned Latxa models before finalizing the 'first' claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a competent and useful case study, and the strongest claim in it—the +12.47 NLU gain from continual pretraining—looks solid. The instruction-following numbers are directionally plausible but rest on a thin evaluation that is not robust enough to support the stage-by-stage conclusions the paper draws.\n\nWhat is new: the first Basque instruction-tuned model under 10B, four manually translated benchmark subsets for Basque, and a clean three-stage comparison of continual pretraining, LoRA SFT, and DPO. The NLU evaluation uses twelve established tasks and the LM Evaluation Harness, so the measured improvement from 48.75 to 61.22 is credible and consistent with prior work in Japanese, Estonian, and Chinese. The paper also reports the small English performance loss after pretraining, which is the right control. The carbon reporting is a nice touch.\n\nWhere it is soft: the manual instruction-following evaluation is 100 examples, scored by one native speaker, with no inter-annotator agreement, no confidence intervals, and no released outputs. The authors themselves write that the conclusions \"should be interpreted within the context of the nature and size of this test set.\" That is an honest admission, but it undercuts the stage-wise attribution. Given n=100, the DPO gain from 23% to 30% and the pretraining benefit from 14% to 23% are both within sampling error; only the aggregate 6% to 30% difference is clearly significant. So the central claim that the pipeline works is supported, but the specific claims about which stage adds value are not. There is also a mild circularity concern: the evaluation instructions come from the No_Robots family and the training sets are translated from No_Robots/SlimOrca, so the test set may not be fully independent of training data. The translated benchmarks are done by one translator without a validation round; that is minor but worth noting.\n\nWho this is for: anyone working on low-resource LLM adaptation will get value from the recipe and the benchmarks. It deserves a serious referee. I would send it out, but ask the authors for a second annotator, confidence intervals, and the 100 model outputs before the numbers are taken at face value.","headline":"Solid, useful case study: the NLU gain is credible, but the instruction-following claims need a larger, reliability-checked evaluation before they support stage-by-stage conclusions.","tokens_in":19717,"tokens_out":2554,"would_cite":true,"duration_ms":22396,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A complete instruction-following model for Basque can be built almost entirely from translated English data, via continual pre-training, LoRA instruction tuning, and DPO, gaining more than 12 NLU points and 24 instruction-following points.","keywords":["low-resource languages","Basque","continual pre-training","instruction tuning","direct preference optimization","machine-translated data","LLM adaptation","natural language understanding"],"falsifier":"Re-run the same 100-instruction Basque test with three or more independent native-speaker raters and report per-model correct rates with confidence intervals; if the 30% versus 6% advantage of Llama-eus-8B-instruct-DPO over Llama-3.1-8B-instruct collapses or the 30% versus 23% DPO-over-SFT gap reverses, the paper's central instruction-following claim is not stable. A larger automatically scored or human-rated sample would settle the same question with more power.","tokens_in":18764,"feed_emoji":"🗣️","tokens_out":9536,"duration_ms":69306,"temperature":0.7,"pith_summary":"This paper argues that a complete instruction-following model for a low-resource language can be built almost entirely from translated English data, provided the base model is first adapted to the target language. The authors test this on Basque by continually pre-training Llama-3.1-8B on about 521 million words of clean Basque text mixed 80:20 with English, then instruction-tuning with LoRA on machine-translated versions of the No_Robots and SlimOrca datasets, then applying DPO on a machine-translated version of UltraFeedback. Across twelve Basque NLU benchmarks, the resulting foundational model averages 61.22, up from 48.75 for its English-centric base, and on a 100-item manual Basque instruction test the final model is fully correct on 30% of items versus 6% for the stock instruct model. The authors conclude that each stage contributes, that translated data is a viable substitute for native Basque instruction data, and that the resulting models are the strongest Basque models under 10 billion parameters. If this reasoning transfers to other low-resource languages, it would give a cheap, reproducible recipe for building instruct models without expensively curated native corpora.","feed_headline":"Basque model: +12 language understanding, +24 instruction-following","feed_subtitle":"Translated English data alone may be enough to build instruction-following models for low-resource languages.","key_machinery":"The load-bearing mechanism is a three-stage transfer pipeline built on an 80:20 Basque-English mixture during continual pre-training, with language-matched sequence packing; LoRA (low-rank adaptation) for instruction tuning; and DPO (direct preference optimization) on translated preference pairs. The 80:20 mix is the object that carries the argument: it transfers functional English competences to Basque while preventing catastrophic forgetting. The instruction and preference stages are carried by two machine-translated datasets, SlimOrca_eu for supervised fine-tuning and UltraFeedback_eu for preference alignment, generated with a single machine translation system.","core_discovery":"The paper's central claim is that a complete instruction-following model can be developed for a low-resource language by adapting an 8-billion-parameter English-centric model at three successive stages. First, full continual pre-training on a curated 521-million-word Basque corpus mixed 80:20 with English raises average performance on 12 Basque NLU benchmarks from 48.75 to 61.22, beating existing Basque-specific models of comparable size and nearly matching a 70B Basque model. Second, LoRA instruction tuning on machine-translated No_Robots and SlimOrca data makes the model follow Basque instructions: 23% fully correct and 41% partially correct on the manual test, versus 6% and 26% for Llama-3.1-8B-instruct. Third, DPO on a machine-translated UltraFeedback pushes the fully correct rate to 30%. The resulting Llama-eus-8B and Llama-eus-8B-instruct are, the paper claims, the best Basque models under 10B parameters; in English the pre-trained model loses only 1.96 points on average, leaving a 13.28-point Basque-English gap.","pith_inferences":["If the translated-data results hold for Basque, a similar 80:20 continual pre-training plus translated SFT/DPO recipe is a cheap candidate for other low-resource languages whose base LLM already has some multilingual coverage; the recipe's success likely depends on how much of the target language the tokenizer and base model already encode.","The 24-point instruction-following gain is measured on a single native speaker's rating of 100 items; a re-rating with multiple annotators and per-item confidence intervals would either firm up or revise the ranking of DPO versus SFT, and could be done without retraining.","Because the English version of the same 100-item test scores 91% correct, the remaining Basque gap is partly a translation-quality and task-complexity problem; improving machine translation quality for the SFT data may yield further gains without more Basque data.","The paper's own dataset-comparison appendix suggests adding more, noisier Basque data yields only marginal NLU gains (61.22 to 61.84), so the next bottleneck for Basque is probably instruction-following and generation quality rather than raw understanding."],"forward_implications":["Languages without native instruction corpora can nevertheless obtain instruction-following models by translating existing English SFT and preference data, as long as a solid target-language foundation model is available.","Continual pre-training on a relatively small, clean corpus of about half a billion words is enough to surpass much larger Basque-specific models in NLU, so dataset quality can substitute for dataset scale.","Starting preference training from a target-language-adapted instruct model is necessary: DPO on the English-centric Llama-3.1-8B-instruct degraded its Basque correctness from 6% to 2%.","Basque still trails English by 13.28 average points in NLU and the instructed model lags far behind Llama-3.1-8B-instruct in English, so the pipeline reduces but does not close the language gap.","Larger synthetic instruction data (SlimOrca_eu) beat a smaller curated set (No_Robots_eu) for instruction-following, implying scale matters more than source quality for translated SFT data."],"supporting_citations":[{"why":"Supplies the Llama-3.1-8B base model and instruct baseline whose Basque performance the pipeline is built to beat.","marker":"Dubey et al., 2024"},{"why":"Provides ZelaiHandi, the roughly 521-million-word freely licensed Basque corpus used for continual pre-training.","marker":"San Vicente et al., 2024"},{"why":"Motivates the 80:20 Basque-English data mix during continual pre-training to prevent catastrophic forgetting.","marker":"Kuulmets et al., 2024"},{"why":"Supplies the Latxa Basque model family and Basque benchmarks (EusProficiency, EusReading, EusExams, EusTrivia) used as baselines and evaluation tasks.","marker":"Etxaniz et al., 2024"},{"why":"Supplies No_Robots, source of the machine-translated Basque instruction dataset and the 100-item manual test set.","marker":"Rajani et al., 2023"},{"why":"Supplies SlimOrca, the larger synthetic instruction dataset whose translation yields the best instruction-tuned model.","marker":"Lian et al., 2023"},{"why":"Supplies UltraFeedback, the preference dataset whose Basque translation is used for DPO alignment.","marker":"Cui et al., 2023a"},{"why":"Provides the DPO objective used to align the instructed model to translated preference data.","marker":"Rafailov et al., 2023"},{"why":"Provides the evaluation harness used to score all models on the NLU benchmarks.","marker":"Gao et al., 2024"}],"fun_headline_variants":["Basque LLM: +12 NLU, +24 instruction-following via translated data","From English to Basque: 8B model gains 24 points in instruction following","Low-resource win: Basque LLM beats larger models with just 521M words","Basque instruct model: +24 points using only translated English data","Llama-eus-8B: Basque SOTA under 10B params, +12 NLU, +24 instruction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline instruction-following improvements rest on a single native Basque speaker rating only 100 translated instructions into correct, partially correct, or wrong, with no second annotator to check the ratings.","fun_headline_variants_meta":{"raw":{"variants":["Basque LLM: +12 NLU, +24 instruction-following via translated data","From English to Basque: 8B model gains 24 points in instruction following","Low-resource win: Basque LLM beats larger models with just 521M words","Basque instruct model: +24 points using only translated English data","Llama-eus-8B: Basque SOTA under 10B params, +12 NLU, +24 instruction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00064,"raw_usage":{"total_tokens":2947,"prompt_tokens":945,"completion_tokens":2002,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":561,"completion_tokens_details":{"reasoning_tokens":1886}},"tokens_in":561,"tokens_out":2002,"duration_ms":13223,"temperature":1.0,"reasoning_tokens":1886,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:38:54.342530+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same 100-instruction Basque test with three or more independent native-speaker raters and report per-model correct rates with confidence intervals; if the 30% versus 6% advantage of Llama-eus-8B-instruct-DPO over Llama-3.1-8B-instruct collapses or the 30% versus 23% DPO-over-SFT gap reverses, the paper's central instruction-following claim is not stable. A larger automatically scored or human-rated sample would settle the same question with more power.","supporting_citations":[],"review_version":1}