{"id":"8c75d663-8ff9-4099-a1c1-549eceff696b","arxiv_id":"2508.12662","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Fine-tuning Llama-3-8B on synthetic Hindi-English code-switched CommonSenseQA data raises Hindi accuracy from 54% to 85.6% while English accuracy rises from 78% to 90.4%.","lead":"The paper tests whether fine-tuning an AI language model on artificially code-switched Hindi-English questions improves how well it answers commonsense questions in Hindi without hurting its English performance. If the effect holds up, it points toward a cheap way to make large language models more equitable for speakers of low-resource languages.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing monolingual fine-tuning controls leave the core attribution to code-switching untested; Section 8 admits this gap.","rationale":"The reader's weakest assumption is the load-bearing issue, and I agree with it. The paper is a clearly labeled work-in-progress, and it deserves credit for releasing the synthetic dataset, using five-fold cross-validation, and majority-voting over non-deterministic inference. But the experimental design cannot distinguish the effect of code-switching from the effect of task-specific fine-tuning on CommonSenseQA, because no monolingual control condition was run. This matters directly for both headline claims: 'substantial improvements in LRL performance' and 'preserving or enhancing performance in HRLs' are causal claims about the training distribution. The fact that English accuracy also rises substantially after fine-tuning on code-switched questions is especially hard to attribute to code-switching, since no original English questions appear in the CMI2 training set; learning the benchmark format and the English answer-choice schema is the more parsimonious explanation. The paper's own Section 8 lists monolingual benchmarking as future work, which is an explicit admission that the discriminating experiment is missing. A secondary statistical concern is that with five folds and no error bars on the baseline, the CMI2-vs-CMI1/3 differences may not be reliable, but this is separate from the attribution problem. Since the empirical numbers are internally consistent and the authors are transparent about the missing control, the conditional verdict is appropriate; I would not accept the causal claim as established nor reject the work outright.","tokens_in":9404,"tokens_out":5226,"duration_ms":52611,"concrete_test":"Run the same five-fold QLoRA protocol with two added control conditions: (1) fine-tune on original English CommonSenseQA training questions (English answers); (2) fine-tune on fully Hindi-translated CommonSenseQA training questions. Evaluate both on the same English and Hindi test sets as Table 1. If control (1) reproduces CMI2's English (90.4%) and Hindi (85.6%) accuracies within fold-level standard deviation, the improvement is task fine-tuning, not code-switching. If control (2) drops English accuracy below the 78% base while CMI2 keeps it near 90.4%, the catastrophic-forgetting claim gains support. Report per-fold means with standard deviations and multiple random seeds for a significance estimate.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Every fine-tuned model was trained on synthetic rewrites of the same CommonSenseQA questions used at evaluation, with the English answer choices preserved, and the design includes no monolingual English-only or Hindi-only fine-tuning control. The headline gains (English 78→90.4%, Hindi 54→85.6% for CMI2 in Table 1) are therefore fully compatible with the simpler explanation that the model learned the benchmark's task format and English answer-choice schema, independent of any benefit from language mixing. The paper's Section 8 acknowledges the gap: \"We also plan to benchmark our approach against models fine-tuned on fully translated monolingual datasets to contrast the specific effects of code-switching from direct target-language exposure.\" The related claim that code-switching avoids catastrophic forgetting is also asserted without the relevant comparison: the paper never fine-tunes on Hindi-only data to demonstrate that such fine-tuning degrades English; the only HRL reference is the base model. So the narrow claim (these datasets improve over base) is supported, but the causal claim central to the paper's contribution is not.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript proposes fine-tuning an open-weight LLM (LLaMA-3-8B-Instruct) on synthetic Hindi-English code-switched text as a way to close the performance gap between high-resource and low-resource languages on commonsense reasoning. The authors generate code-switched versions of CommonSenseQA using GPT-3.5 and CoCoa, with three Code-Mixing Index ranges (low, medium, high), while keeping answer choices in English. They fine-tune the base model separately on each generated dataset and evaluate English and Hindi accuracy under five-fold cross-validation. Table 1 reports that the medium-mixing (CMI 2) model reaches 90.4% English and 85.6% Hindi accuracy, versus 78.0% and 54.0% for the base model, and the authors interpret this as evidence that code-switched fine-tuning improves low-resource performance without harming high-resource performance.","tokens_in":9574,"tokens_out":4989,"duration_ms":47357,"significance":"The question addressed is timely: equitable multilingual LLM performance is an important fairness problem, and controlled synthetic code-switching is a plausible intervention. The paper's positive aspects include making the synthetic code-switched dataset and code public, evaluating three controlled mixing levels, and being transparent about limitations, including the absence of monolingual controls. If the causal claim were established, the result would be practically valuable for adapting LLMs to low-resource languages. However, the current experimental design cannot yet separate the effect of code-switching from the effect of task-specific fine-tuning on the evaluation benchmark, so the significance is conditional on additional controls.","major_comments":[{"comment":"All fine-tuned models are trained on rewrites of the same CommonSenseQA questions used for evaluation, and no monolingual fine-tuning control is included. The large gains over the base model (e.g., English 78.0% to 90.4% and Hindi 54.0% to 85.6% for CMI 2 in Table 1) are therefore fully compatible with the model learning the task format and answer-choice schema from English-only or Hindi-only fine-tuning, independent of code-switching. Section 8 lists the comparison with fully translated monolingual fine-tuning as future work, but that comparison is needed to support the paper's central contribution as currently stated. Please add English-only and Hindi-only fine-tuned controls, and ideally a control fine-tuned on English questions with the same answer choices, and show that code-switched training outperforms them.","section":"§4.1–4.2, Table 1"},{"comment":"The claim that code-switched fine-tuning avoids catastrophic forgetting is not tested. The paper never fine-tunes on Hindi-only data and then evaluates on English, so it cannot show that such monolingual fine-tuning degrades English performance; the only English reference point is the base model. The 'preserving or enhancing HRL performance' result relative to the base model does not establish that code-switching is what preserves it. This claim should either be removed or supported with the missing Hindi-only control.","section":"§1 and §8"},{"comment":"Per-fold accuracy varies widely (e.g., CMI 1 English ranges 66–98 percentage points across folds; CMI 1 Hindi has a standard deviation of 16.16 pp; GPTgen Hindi has 12.76 pp), and the CMI 2 versus CMI 3 differences are small relative to this variance. With five folds and no significance testing or confidence intervals, the ranking of CMI configurations and the claim that CMI 2 is the best are not robust. Report statistical comparisons or at least per-condition confidence intervals, and add baseline per-fold results, which are currently absent from both Table 1 and Table B2.","section":"§4.2 and Appendix B, Table B2"},{"comment":"The Hindi evaluation set construction is underspecified: the text states that the same testing dataset was 'translated into Hindi' but does not describe the translation method, whether a professional translator or machine translation was used, or whether the answer choices were translated. Since all Hindi accuracy scores depend on this translation, please document the procedure and any quality checks.","section":"§4.2"}],"minor_comments":[{"comment":"Baseline standard deviations are not reported, and baseline per-fold scores are not given in Table B2, making it impossible to assess whether the base-model gains exceed within-condition variability.","section":"Table 1 and Table B2"},{"comment":"The GPT-3.5 prompt and few-shot examples used for code-switched generation are not included; please provide them in an appendix for reproducibility.","section":"§3.1"},{"comment":"The QLoRA configuration (rank, alpha, dropout) is not reported; these are standard details needed to reproduce the fine-tuning.","section":"§3.3"},{"comment":"The capitalization of 'commonSenseQA' is inconsistent; use 'CommonSenseQA' consistently.","section":"Throughout"},{"comment":"The phrase 'we rely on the evaluation metrics released with the CoCoa paper' is vague; specify which metrics were used and how they were applied to the generated data.","section":"§3.2"},{"comment":"The analogy between moderate code-switching improving human bilingual performance and the CMI 2 result is suggestive but is not evidence for the model behavior; please phrase it as an analogy rather than a supporting result.","section":"§5"}],"recommendation":"major_revision","confidential_remarks":"This is a non-archival SRW paper describing work in progress. For a full archival venue, the central attribution claim requires the monolingual controls and significance testing described above. The authors are transparent about the missing comparison in Section 8, but the abstract and contributions currently overstate what the data show. I would encourage the editor to treat this as an exploratory paper whose value will depend on the revision adding the missing controls."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a work-in-progress paper that does something genuinely useful — it builds a synthetic Hindi-English code-switched version of CommonSenseQA with three controlled CMI ranges and reports that the medium-mix version (CMI2) produces the best balance, with Hindi accuracy going from 54% to 85.6% and English from 78% to 90.4% after fine-tuning LLaMA-3-8B. The dataset is released, the per-fold results are in the appendix, and the limitations section is refreshingly honest.\n\nWhat's new: the controlled CMI comparison and the finding that medium mixing beats low and high mixing. That is a concrete, testable result, and the released data lets others reproduce or build on it.\n\nThe soft spot is exactly what the authors admit in Section 8: they never fine-tune on monolingual English-only or Hindi-only data. With the evaluation on the same CommonSenseQA benchmark used for training, the gains over the base model could simply come from learning the task format and the English answer-choice schema. That would explain the English improvement as easily as code-switching does. The catastrophic-forgetting claim is also unsupported — they never fine-tune on Hindi-only data to show that English degrades, so they can't demonstrate that code-switching avoids it. The Table 1 standard deviations are large enough that some comparisons across folds are noisy; the CMI2 vs. base gap is large, but the CMI1 vs. CMI3 ordering is not robust.\n\nNone of this is hidden. The paper calls itself work in progress and lists the monolingual control as planned future work. The problem is that the abstract and introduction state the causal claim as demonstrated, not as a hypothesis. So the reviewable claim should be the narrow one: fine-tuning on these datasets improves both languages over base. That claim is supported.\n\nWho this is for: anyone working on multilingual fine-tuning or code-switched data will find the dataset and the CMI-comparison useful. It is not yet a general result — one language pair, one 8B model, one commonsense QA benchmark.\n\nMy recommendation: send it to a serious referee. The central attribution needs to be fixed before it can be cited as evidence for code-switching, but the dataset and the empirical setup deserve scrutiny and the authors have been transparent about the gap.\n\nRegards.","headline":"A useful controlled dataset and a clear empirical result, but the missing monolingual baseline leaves the paper's central claim unproven.","tokens_in":10120,"tokens_out":2203,"would_cite":true,"duration_ms":21361,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning on synthetic code-switched text substantially improves low-resource language common-sense accuracy without degrading high-resource language performance.","keywords":["code-switching","multilingual language models","low-resource languages","common sense reasoning","fine-tuning","cross-lingual transfer","synthetic data","Code-Mixing Index"],"falsifier":"Fine-tune the same base model on the same CommonSenseQA questions in English only (or Hindi only) with the same budget; if the English-only model matches or exceeds the Hindi accuracy gains of the code-switched model, the effect is not caused by code-switching.","tokens_in":9205,"feed_emoji":"🌐","tokens_out":5761,"duration_ms":50611,"temperature":0.7,"pith_summary":"The paper proposes that fine-tuning a multilingual large language model on synthetically code-switched text—questions that mix Hindi and English in controlled proportions—can improve its accuracy on low-resource languages like Hindi while preserving or even improving its accuracy on English. On CommonSenseQA, the best configuration raises Hindi accuracy from 54% to 85.6% and English from 78% to 90.4%. The authors interpret this as evidence that code-switching helps the model build shared cross-linguistic representations, offering an alternative to monolingual fine-tuning, which often causes catastrophic forgetting of high-resource languages. They release a synthetic Hinglish version of CommonSenseQA with three code-mixing intensities.","feed_headline":"Hindi accuracy jumps from 54% to 85.6% via code-switched fine-tuning","feed_subtitle":"The medium code-mixing setting also lifts English accuracy, from 78% to 90.4%, on CommonSenseQA.","key_machinery":"The controlling object is the Code-Mixing Index (CMI), a value from 0% to 50% that measures how evenly two languages are mixed in a text based on per-language word counts and language-independent tokens. The paper generates synthetic code-switched questions with three CMI bands—low, medium, high—plus a less controlled GPT-generated set, then fine-tunes the model on each. The medium CMI band is claimed to provide the optimal balance for cross-lingual transfer.","core_discovery":"The central empirical claim is that controlled code-mixing during fine-tuning transfers reasoning skill across languages. Using a controllable code-switched text generator to convert English CommonSenseQA questions into Hinglish at three levels of Code-Mixing Index, the authors fine-tune a LLaMA-3-8B-Instruct model with QLoRA. The medium-mixing configuration yields the largest gains: Hindi accuracy rises to 85.6% (from a 54% baseline) and English accuracy to 90.4% (from 78%). The paper argues this balances cross-linguistic transfer without sacrificing either language, echoing findings from bilingualism research that moderate bilingual experience aids native-language performance.","pith_inferences":["Because the authors fine-tune on exactly the benchmark they evaluate, the gains may partly reflect task-specific adaptation rather than code-switching itself; a monolingual English fine-tune control would isolate the effect.","The observed optimum at medium CMI suggests a testable hypothesis: there is an ideal code-mixing ratio for cross-lingual transfer, and future work could sweep CMI continuously to map that curve.","The approach could extend to other language pairs and base models, but only if the generation method can produce coherent code-switched text for those pairs and tokenizers support both scripts."],"forward_implications":["Code-switched fine-tuning can serve as a more equitable alternative to monolingual fine-tuning for low-resource languages, avoiding the typical trade-off of degrading high-resource performance.","The approach requires only synthetic data, sidestepping the scarcity of naturally occurring code-switched corpora for low-resource languages.","The released dataset with three CMI levels enables systematic studies of how mixing intensity affects multilingual model behavior.","If the effect generalizes, the method could narrow the performance gap for other low-resource languages without retraining from scratch."],"supporting_citations":[{"why":"Supplies the controllable code-switched generation model (CoCoa) used to create the three CMI-ratio datasets.","marker":"Mondal et al. (2022)"},{"why":"Provides the CommonSenseQA benchmark, the source of questions for the synthetic datasets and the evaluation task.","marker":"Talmor et al. (2018)"},{"why":"Defines the LLaMA-3-8B-Instruct base model, whose tokenizer and multilingual capability make the fine-tuning experiment possible.","marker":"Dubey et al. (2024)"},{"why":"Provides QLoRA, the efficient fine-tuning method used to adapt the model on the code-switched data.","marker":"Dettmers et al. (2023)"},{"why":"Defines the Code-Mixing Index formula used to measure and control the language ratios in the generated text.","marker":"Das and Gambäck (2014)"},{"why":"Documents performance degradation in high-resource languages after monolingual low-resource fine-tuning, which motivates the proposed code-switching alternative.","marker":"Guo et al. (2023)"}],"fun_headline_variants":["Code-switching boosts Hindi accuracy from 54% to 85.6%","Moderate code-mixing lifts Hindi and English both","Fine-tuning on Hinglish closes language gap in LLMs","Code-switched data: Hindi +31.6pts, English +12.4pts","Multilingual parity via controlled code-switching"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The gains attributed to code-switching could instead come from fine-tuning on the CommonSenseQA task itself, because the paper did not compare against English-only or Hindi-only fine-tuned models.","fun_headline_variants_meta":{"raw":{"variants":["Code-switching boosts Hindi accuracy from 54% to 85.6%","Moderate code-mixing lifts Hindi and English both","Fine-tuning on Hinglish closes language gap in LLMs","Code-switched data: Hindi +31.6pts, English +12.4pts","Multilingual parity via controlled code-switching"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00025,"raw_usage":{"total_tokens":1505,"prompt_tokens":851,"completion_tokens":654,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":467,"completion_tokens_details":{"reasoning_tokens":564}},"tokens_in":467,"tokens_out":654,"duration_ms":6235,"temperature":1.0,"reasoning_tokens":564,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:19:20.509148+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fine-tune the same base model on the same CommonSenseQA questions in English only (or Hindi only) with the same budget; if the English-only model matches or exceeds the Hindi accuracy gains of the code-switched model, the effect is not caused by code-switching.","supporting_citations":[{"cited_title":"CoCoa: An Encoder- Decoder Model for Controllable Code-switched Gen- eration","cited_arxiv_id":null,"evidence_quote":"Supplies the controllable code-switched generation model (CoCoa) used to create the three CMI-ratio datasets."}],"review_version":2}