{"id":"9ecf4272-bf20-4863-9ae3-00a9efbdfa53","arxiv_id":"2412.13375","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Bilingual embedding alignment plus instruction tuning improves Persian classification in Llama-2, while English-to-Persian transfer is marginal and task-dependent.","lead":"This paper tests recipes for teaching Persian to Llama-2 using parameter-efficient fine-tuning, from embedding alignment to LoRA pre-training and instruction tuning. It finds bilingual alignment helps Persian classification with little English loss, but knowledge transfer from English to Persian is limited to simple tasks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 5 undercuts the central attribution: Persian classification is 0.56 for Llama-2, Em-aligned, and Fa-pretrained alike, so instruction tuning, not bilingual alignment, drives the reported gain.","rationale":"The reader's weakest assumption concerns train/test overlap via SuperNatural-Instructions, which affects external validity. I agree that is a real problem, but the more load-bearing issue is internal: even taking the reported numbers at face value, the design lacks a clean no-alignment control that differs only in alignment. The Llama-2 row in Table 5 is such a control, and it matches the aligned models on Persian classification exactly (0.56/0.56/0.56). So the central mechanism named in the abstract is not identified by the experiments. The overlap issue bears on whether the absolute gains are real; the attribution issue bears on whether the paper's explanation of those gains is correct. A reframed paper that says instruction tuning with mixed-language data improves Persian classification, and that further alignment mainly helps under transfer-only conditions, would be consistent with the tables. That is a substantial revision, but the underlying experiments are still informative, so CONDITIONAL rather than REJECT is appropriate.","tokens_in":15702,"tokens_out":9275,"duration_ms":88310,"concrete_test":"Evaluate Llama-2, Em-aligned, and Fa-pretrained on the same Persian classification tasks before instruction tuning (zero-shot, and after each alignment stage without any instruction data), with 3–5 seeds and reported standard deviations. If Em-aligned does not outperform Llama-2 before instruction tuning while both improve equally after it, the improvement is driven by instruction tuning and the 'through bilingual data alignment' wording must be withdrawn; if Em-aligned already beats the no-alignment baseline before instruction tuning, the attribution would be supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Table 5 is the comparison the abstract's first claim relies on, and it does not support the causal attribution to bilingual alignment. After identical instruction tuning with English+Persian instructions, Persian average classification is 0.56 for Llama-2 (no vocabulary expansion, no embedding alignment, no Persian pre-training), 0.56 for Em-aligned, and 0.56 for Fa-pretrained; the gain over random (0.36) is shared by the no-alignment baseline. The English-side claim is also internally strained: English summarization falls from 0.35 (Llama-2) to 0.32 (Em-aligned) and 0.23 (Fa-pretrained), so 'no adverse impact on English tasks' holds only if restricted to classification and is contradicted by Section 7's own admission that alignment negatively impacts English text generation. To support the headline, the paper needs either an explicit control that isolates the alignment stages or a reframed claim attributing the classification gain to instruction tuning with Persian data.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies how to extend Llama-2-7B to Persian under parameter-efficient constraints. It proposes a multi-stage pipeline: extending the tokenizer with 20,000 Persian subword tokens, aligning newly added embeddings via frozen-transformer next-token prediction on monolingual and parallel English-Persian data, continued pretraining with LoRA on Persian text, and bilingual instruction tuning. The authors evaluate four model variants (Llama-2, Llama-2-noLoRA, Em-aligned, Fa-pretrained) on Persian and English classification tasks (multiple-choice QA, sentiment, entailment) and generation tasks (summarization, translation), and additionally study knowledge transfer with English-only plus translation instructions, input translation, limited instruction data, and comparisons with multilingual and state-of-the-art models. The central claims are that bilingual embedding alignment improves Persian classification without adverse impact on English tasks, that cross-lingual alignment provides minimal benefit in low-resource settings, and that knowledge transfer from English to Persian is marginal and mostly helps simple classification tasks.","tokens_in":15890,"tokens_out":3171,"duration_ms":28236,"significance":"If the central claims were firmly established, the paper would provide practical guidance for adapting English-centric LLMs to lower-resource languages under PEFT constraints, and its systematic stage-by-stage evaluation would be a useful reference. The paper has genuine strengths: it releases code, covers a non-Latin low-resource language, evaluates both classification and generation, includes unseen-task generalization checks, and compares against several recent open and closed models. However, the headline attribution of classification gains to bilingual alignment is directly undermined by the paper's own Table 5, and the scoping of the 'no adverse impact' claim is internally inconsistent with the reported English summarization drops and with the conclusion section. These issues are load-bearing rather than cosmetic, so the central claims need correction or additional experimental support before the paper can be accepted.","major_comments":[{"comment":"The abstract's first claim, that bilingual data alignment enhances Persian classification accuracy, is not supported by Table 5. For Persian tasks, average classification accuracy is 0.56 for Llama-2 (no vocabulary expansion, no alignment), 0.56 for Em-aligned, and 0.56 for Fa-pretrained; the improvement over random (0.36) is shared by the non-aligned instruction-tuned baseline. This indicates that instruction tuning with Persian instructions, rather than the alignment stages, drives the reported gain. The paper needs either an explicit control that isolates the alignment stages while holding instruction tuning fixed, or a reframed claim that attributes the classification improvement to instruction tuning with Persian data.","section":"§5.2, Table 5"},{"comment":"The claim of 'no adverse impact and sometimes even improvements on English tasks' is contradicted by the paper's own numbers: English summarization drops from 0.35 (Llama-2) to 0.32 (Em-aligned) and 0.23 (Fa-pretrained). Section 7 explicitly concedes that 'this alignment negatively impacts the model's English text generation.' The abstract and Section 1 should therefore scope the 'no adverse impact' claim to classification tasks, or the conflict with Section 7 should be resolved.","section":"§5.2, §7, Table 5"},{"comment":"The evaluation setup has a train/test overlap confound that affects both the classification-gain claim and the limited-transfer claim. English evaluation tasks are selected from SuperNatural-Instructions, the same source used for instruction tuning, and Persian multiple-choice, entailment, and translation evaluations also come from SuperNatural-Instructions. Section 6 acknowledges that observed improvements 'may be influenced by similarities between the training and test sets, such as instruction formats.' The authors should evaluate on held-out sources or explicitly verify that the specific test instances and templates were not present in the instruction-tuning data.","section":"§3.2, §5.1, §6"},{"comment":"All comparative claims rest on single-run point estimates without error bars, confidence intervals, or significance testing. Many differences are within one or two points of accuracy (e.g., Persian averaged classification 0.56 vs. 0.56, English averaged classification 0.52 vs. 0.56), and generation scores are reported without variance. Without repeated runs or statistical testing, the claims of 'sometimes even improvements' and 'minimal benefits' are not robust; the paper should report variance or at least multiple seeds for the central comparisons.","section":"Tables 5–9"}],"minor_comments":[{"comment":"The header 'Perian sourceType' appears to be a typo and should read 'Persian sourceType'; also consider aligning the table columns for readability.","section":"Table 1"},{"comment":"The phrase 'only the heads and embeddings are trained' is ambiguous; please specify whether 'heads' means the LM head, attention heads, or both, and clarify what is frozen in each stage.","section":"§4.2.1"},{"comment":"Table 4 reports the same parameter counts for pretraining and instruction-tuning (6.20%), but Section 4.2.2 says LoRA weights are updated during pretraining and later during instruction tuning; a sentence clarifying which components are trainable in each stage would prevent confusion.","section":"§4.3, Table 4"},{"comment":"The description of the Llama-2-noLoRA model says it has 'limited Persian tokens' but the model description for Llama-2 says the same; since noLoRA has the original 32k vocabulary, please clarify whether Llama-2-noLoRA uses the expanded vocabulary or not.","section":"§5.1"},{"comment":"The text refers to 'the orange box' in Figure 2, but the figure is not in color in this format; consider describing the location concretely (e.g., 'the highlighted box').","section":"Figure 2"},{"comment":"The conclusion says alignment 'negatively impacts the model's English text generation,' which contradicts the abstract's 'no adverse impact' phrasing; align the wording across abstract, introduction, and conclusion after the major revisions are made.","section":"§7"}],"recommendation":"major_revision","confidential_remarks":"The paper contains a useful empirical study and a reproducible pipeline, but the headline attribution to bilingual alignment is not supported by the paper's own control comparison in Table 5. A major revision that either adds an isolation control or substantially reframes the claims, plus explicit handling of the train/test overlap and reporting of variance, would be needed before publication. The mismatch between the abstract and Section 7 should be resolved in the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the Persian adaptation paper. The headline claim is not supported by the paper's own Table 5: Persian average classification is 0.56 for plain Llama-2, Em-aligned, and Fa-pretrained alike after instruction tuning. The gain over random (0.36) is already present in the unaligned baseline, so the abstract's attribution of the classification boost to bilingual alignment is unsupported. What the experiments actually show is that instruction tuning with Persian data is what matters for classification, while alignment shows up in generation and translation. The 'no adverse impact on English' claim is also too broad: English summarization drops from 0.35 to 0.32 and 0.23 across the alignment stages, and Section 7 itself admits alignment negatively impacts English text generation. This is a reframing problem, not a fatal flaw, but it is load-bearing for the abstract.\n\nWhat the paper does well is the staged comparison itself. Vocabulary expansion, embedding alignment, LoRA pre-training, and instruction tuning are evaluated one stage at a time, including a translation-only condition that asks how far English plus translation instructions can carry Persian tasks. That is genuinely useful for practitioners. The limited-instruction experiment, where the Fa-pretrained model gains the most from instruction tuning, is a concrete observation. The code is public, and the limitations section is honest about compute constraints.\n\nThe soft spots are addressable. No error bars, single runs, so the equal 0.56s may hide variance. Entailment and translation are evaluated on SuperNatural-Instructions, which was also the instruction-tuning source, so those numbers are not clean evidence of language ability; the paper acknowledges this only in the Section 6 comparison with Qwen/Gemma, not in the main analysis. The Gemini 1.5 comparison is overstated: Table 9 shows Gemini at 0.74 average classification against this model's 0.56, so 'on par' is wrong.\n\nWho is this for? Someone building a PEFT pipeline to add a low-resource language to Llama and wanting a stage-by-stage recipe. It is not a methods paper, and the conclusions need tightening. The experimental matrix is worth a referee's time; I would send it to review and ask for revised claims, variance estimates, and an overlap check on the evaluation tasks.","headline":"The staged PEFT recipe is useful, but Table 5 undermines the paper's central claim that bilingual alignment drives the Persian classification gains, and the 'no adverse impact' claim is contradicted by the paper's own generation results.","tokens_in":16436,"tokens_out":2857,"would_cite":true,"duration_ms":27551,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Bilingual embedding alignment, not heavy pretraining, is what lets Llama classify new-language tasks, and it leaves English mostly intact.","keywords":["Persian language adaptation","parameter-efficient fine-tuning","cross-lingual transfer","embedding alignment","instruction tuning","low-resource languages","classification","text generation"],"falsifier":"Take the same training recipe and evaluate on Persian tasks whose instruction templates and instances come from a source completely disjoint from those used in instruction tuning; if the classification gains from bilingual alignment vanish, the gains are format familiarity rather than Persian understanding.","tokens_in":15525,"feed_emoji":"🗣️","tokens_out":4953,"duration_ms":42972,"temperature":0.7,"pith_summary":"This paper asks what it takes to teach a largely English-only large language model a new, low-resource language using parameter-efficient fine-tuning, with Persian as the test case. It argues that embedding-level bilingual alignment is the decisive ingredient for classification: aligning Persian embeddings with English ones through parallel data is enough to lift Persian classification accuracy, without hurting English and sometimes improving it. For generation, additional monolingual Persian pretraining helps, but it costs some English summarization quality. The paper also argues that cross-lingual transfer from English to Persian without Persian instruction data is shallow, helping mostly simple tasks like sentiment analysis.","feed_headline":"Bilingual alignment lifts Persian classification, not generation","feed_subtitle":"Adding Persian via low-rank tuning helps Llama classify without hurting English, but deeper transfer stays shallow.","key_machinery":"The central mechanism is a staged parameter-efficient pipeline: first, vocabulary expansion by merging 20,000 SentencePiece Persian tokens into Llama-2's 32,000-token vocabulary; second, embedding alignment, where all transformer layers are frozen and only embeddings and the LM head are trained on monolingual Persian and parallel English-Persian next-token prediction; third, text-generation pretraining with low-rank adaptation (LoRA, rank 8 and alpha 32) on monolingual Persian; and fourth, instruction tuning on Persian and English instructions, including English-Persian translation instructions. The load-bearing step is the second one, bilingual embedding alignment, which the paper shows is sufficient for classification gains and preserves English ability.","core_discovery":"The paper's central claim is that when extending a largely English-only LLM to Persian with parameter-efficient fine-tuning, the stage that matters most for classification is embedding-level bilingual alignment: training only the embedding and head layers on monolingual Persian plus English-Persian parallel text yields Persian classification accuracy equal to or better than further full-weight LoRA pretraining, with no English degradation and sometimes English gains. For generation tasks, deeper monolingual pretraining helps, but at a cost of English summarization performance. Cross-lingual transfer from English to Persian without Persian instructions is weak, succeeding mainly on simple sentiment classification and failing on textual entailment, and with very limited Persian instruction data the model's initial Persian knowledge outweighs alignment.","pith_inferences":["Because all English evaluation tasks come from the same instruction-tuning source, the measured classification gains may partly reflect instruction-format familiarity rather than general Persian ability; a truly held-out Persian task format would test this.","The alignment benefit may scale with base-model quality: a stronger base model with more Persian tokens could show deeper transfer than Llama-2 demonstrates, so the shallow-transfer result may be specific to weak initial Persian support.","This bilingual-alignment recipe, applied with only embedding and head updates, may generalize to other low-resource languages with non-Latin scripts, where embedding alignment is the primary bottleneck.","In practical deployment, translating inputs to English is a cheaper alternative to target-language adaptation for classification, but not for generation, where the adapted model retains an advantage in summarization."],"forward_implications":["Classification adaptation to a new language can be achieved by aligning embeddings with bilingual parallel data; additional monolingual pretraining is not necessary for classification.","Adding a language through embedding alignment does not degrade English classification and sometimes improves it, though further Persian pretraining can lower English summarization quality.","When only limited instruction data is available, the model's initial target-language comprehension matters more than cross-lingual alignment, so base-model choice dominates.","Knowledge transfer from English to Persian without Persian instruction data is limited: it helps simple classification tasks like sentiment analysis but not complex tasks like textual entailment.","Translating Persian inputs to English improves classification accuracy for the adapted models but does not help summarization, suggesting generation is more sensitive to input language."],"supporting_citations":[{"why":"Supplies the base Llama-2-7B model whose limited Persian ability motivates the adaptation pipeline.","marker":"Touvron et al. (2023)"},{"why":"Introduces LoRA, the parameter-efficient method used for pretraining and instruction tuning throughout the paper.","marker":"Hu et al. (2022)"},{"why":"Provides the SentencePiece subword model used to build the 20,000-token Persian vocabulary extension.","marker":"Kudo and Richardson (2018)"},{"why":"Contributes the MIZAN English-Persian parallel corpus used in the bilingual embedding-alignment stage.","marker":"Kashefi (2018)"},{"why":"Contributes the TEP Persian-English parallel corpus used in the bilingual embedding-alignment stage.","marker":"Pilehvar et al. (2011)"},{"why":"Contributes the PEPC parallel corpus used in the bilingual embedding-alignment stage.","marker":"Karimi et al. (2018)"},{"why":"Supplies the SuperNatural-Instructions dataset, which is both the main instruction-tuning source and the source of all English evaluation tasks, hence the overlap caveat.","marker":"Wang et al. (2022)"},{"why":"Supports the claim that more diverse instruction data improves the generalizability of instruction-tuned models.","marker":"Wang et al. (2023)"}],"fun_headline_variants":["Alignment aids Persian classification, not generation","Persian LLM: classification gains, generation stalls","Low-rank tuning: embedding alignment for Persian tasks","Bilingual alignment boosts Persian classification only","Persian adaptation: alignment powers classification, not generation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that the Persian classification scores reflect language ability, not the model's familiarity with instruction templates and examples that overlap between training and test sets.","fun_headline_variants_meta":{"raw":{"variants":["Alignment aids Persian classification, not generation","Persian LLM: classification gains, generation stalls","Low-rank tuning: embedding alignment for Persian tasks","Bilingual alignment boosts Persian classification only","Persian adaptation: alignment powers classification, not generation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00022,"raw_usage":{"total_tokens":1408,"prompt_tokens":866,"completion_tokens":542,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":482,"completion_tokens_details":{"reasoning_tokens":473}},"tokens_in":482,"tokens_out":542,"duration_ms":5290,"temperature":1.0,"reasoning_tokens":473,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:10:40.793269+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same training recipe and evaluate on Persian tasks whose instruction templates and instances come from a source completely disjoint from those used in instruction tuning; if the classification gains from bilingual alignment vanish, the gains are format familiarity rather than Persian understanding.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the MIZAN English-Persian parallel corpus used in the bilingual embedding-alignment stage."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the TEP Persian-English parallel corpus used in the bilingual embedding-alignment stage."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the PEPC parallel corpus used in the bilingual embedding-alignment stage."}],"review_version":1}