{"id":"e3b0b9bf-e604-4bb8-ae5a-ee2dfa384aa9","arxiv_id":"2504.19070","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"Fine-tuning Qwen2.5-3B and 7B with LoRA on synthetic Hinglish dialogues raises human-rated fluency and coherence, though the benchmark is small and synthetic.","lead":"This paper builds a Hinglish chatbot by fine-tuning small open-weights language models on dialogues generated by a larger AI model. It reports that the fine-tuned small models score close to a 70B model on a small, AI-judged test, but the data and code are not released.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"All training and test dialogues come from Gemini-2.0-Flash; without a natural Hinglish held-out set, the reported gains and 'competitive with 70B' claim are unestablished.","rationale":"The reader identified the same load-bearing weakness: the evaluation is synthetic, generated by the same model that produced the training data, and the only human evaluation is very small. I agree that this is the critical point. The paper's premise is plausible, but the evidence as presented cannot establish the headline claim about real Hinglish conversation because there is no independent naturalistic benchmark. The internal inconsistencies in the reported numbers and the absence of released code/data further weaken confidence, but the fundamental issue is external validity. Since the reader already recommended REJECT and my concern reinforces that verdict rather than shifting it, the verdict remains unchanged.","tokens_in":5295,"tokens_out":2628,"duration_ms":24757,"concrete_test":"Collect a held-out set of at least 200 natural Hinglish conversation prompts from real chat logs, social media, or a pre-existing human-written Hinglish conversational corpus, with no Gemini-generated content. Run the final Qwen2.5-3B and Qwen2.5-7B LoRA fine-tuned models, their base versions, and LLaMA-70B on these prompts. Have bilingual annotators blindly rate fluency, coherence, persona adherence, and appropriateness, and also perform forced-choice preference between models. If the fine-tuned models no longer beat their base versions, or if the 7B model no longer approaches LLaMA-70B on this naturalistic set, the central claim is unsupported. Repeating the CMI and BERTScore computation on this set would provide a quantitative check of whether the measured gains transfer.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that small fine-tuned models can be competitive with a 70B model on Hinglish conversation—depends on evaluation data that is not independent of the training data. Section 4.2 states that the full dataset was \"generated entirely through synthetically generated dialogues\" using Gemini-2.0-Flash, and Section 6.1 splits that same synthetic corpus 80:10:10 for train/validation/test. The test prompts therefore come from the same generator, the same prompt style, and the same topic template as the training data. A model fine-tuned on Gemini-style Hinglish is measured against Gemini-style Hinglish; it can score well by reproducing that style without demonstrating the ability to handle real romanized Hinglish with its spelling variation and code-switching patterns. The benchmark against LLaMA 70B in Section 6.2 uses 100 prompts from this same synthetic distribution with GPT-4o as judge, which further favors outputs that resemble the judge's own generation conventions rather than human Hinglish. The human A/B test uses only 10 prompts, which is too small to establish a reliable preference distribution and is described only as \"surveyors\" with no demographic or annotation detail. Without an external, naturalistic Hinglish evaluation, the improvements in Tables 1-2 and the claim of approaching 70B performance remain benchmark artifacts rather than established conversational capability.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes fine-tuning small multilingual models (Qwen2.5-3B, Qwen2.5-7B, Gemma3-4B) with LoRA/QLoRA on a synthetically generated Hinglish dialogue corpus produced by Gemini-2.0-Flash. The authors report improvements in fluency, coherence, gender correctness, and persona adherence after fine-tuning, claim that their 7B model approaches LLaMA-70B performance on a 100-prompt benchmark judged by GPT-4o, and conclude that small fine-tuned models can be competitive for Hinglish conversation while remaining computationally efficient. The entire training, validation, and test data come from the same synthetic generation process, and the human A/B test uses only ten prompts.","tokens_in":5665,"tokens_out":5719,"duration_ms":47162,"significance":"If the central claim were supported, the result would be practically useful for building lightweight conversational agents for low-resource, code-mixed languages. The paper does provide a clear problem statement, a sensible choice to avoid BLEU/ROUGE, and a reproducible-looking fine-tuning pipeline with LoRA. However, the experimental design is self-referential: training and test data share the same Gemini-2.0-Flash generator, and the human evaluation is too small and underdescribed to validate the claim. The numerical results also contain internal contradictions (e.g., Table 1 vs. text). As submitted, the evidence does not establish that the small models generalize to natural Hinglish conversation or that they are genuinely competitive with a 70B model.","major_comments":[{"comment":"The entire dataset, including the test split, was generated by Gemini-2.0-Flash under the same prompting framework (Section 4.2; Section 6.1 splits this corpus 80:10:10). A model fine-tuned on this corpus is therefore evaluated against inputs drawn from the same distribution and style as its training data; high scores may reflect imitation of the generator's Hinglish style rather than mastery of real romanized Hinglish. Without an external, naturally occurring Hinglish evaluation set, the reported gains and the 'competitive with 70B' claim are unestablished.","section":"Sections 4.2 and 6.1"},{"comment":"The text and Table 1 disagree on the identity and magnitude of the reported improvements. Table 1 is titled 'Qwen2.5 3B Model Comparison' and shows a single LoRA column with fluency +41.4% and gender correctness +8.9%, while the text attributes a 41.4% fluency gain to '3B_Hinglish_LoRA_LD', a 34.6% gain to '3B_Hinglish_LoRA', and a 25.6% gender improvement to '3B_Hinglish_LoRA'. The table contains no LD/non-LD distinction, and the 25.6% figure does not appear anywhere in the table. These inconsistencies make the quantitative results irreproducible.","section":"Table 1 and Section 6.2"},{"comment":"The human A/B test uses only ten prompts, and the paper does not report the number of surveyors, their language background, or any significance test. The reported preference rates are also internally inconsistent: a fine-tuned preference rate of 87.8% would leave at most 12.2% for the base model, not the stated '12-39%'. This cannot support the claim of an 'overwhelming preference' for the fine-tuned model.","section":"Section 6.2, A/B test"},{"comment":"The comparison against LLaMA 70B uses GPT-4o as judge on 100 prompts drawn from the same synthetic distribution as the training data, and no human evaluation of the LLaMA outputs is reported. The score differences (4.6 vs. 4.1 vs. 3.86) are presented without confidence intervals or significance tests. Given the generator-aligned evaluation, the claim that the 7B model 'approaches' 70B performance is not supported.","section":"Section 6.2, LLaMA 70B comparison"}],"minor_comments":[{"comment":"Figure 1 is referenced but not included in the manuscript text; please add the figure or remove the reference.","section":"Section 6.2"},{"comment":"The description of human evaluation ('Fluent bilingual evaluators rated model outputs...') lacks details on the number of evaluators, the number and source of prompts, and inter-annotator agreement; please specify.","section":"Section 6.1"},{"comment":"The 'BERT-F1 score of 0.828' is said to indicate similarity to 'human-generated references', but the source of these references is not described; please clarify.","section":"Section 6.2"},{"comment":"Gemma3-4B is listed as a final model but no experimental results for it appear in Tables 1 or 2; please either report its results or explain its omission.","section":"Section 5"},{"comment":"Several entries are incomplete, e.g., 'Google. 2024. Google gemini app.' and 'TheDevastator. 2021' lack URLs or retrieval details; please complete the bibliography.","section":"References"},{"comment":"The sentence 'Across 10 diverse prompts, the fine-tuned model was consistently preferred, with surveyor preference rates as high as 87.8%, compared to just 12-39% for the base model' is ambiguous and numerically inconsistent; please rephrase with per-prompt or aggregate results.","section":"Section 6.2"}],"recommendation":"reject","confidential_remarks":"The manuscript appears to be an early-stage report rather than a complete research paper. The core evaluation is circular, and the internal numerical inconsistencies (Table 1 vs. text) suggest the reported results have not been carefully checked. Even with a major revision, the authors would need to collect external naturalistic Hinglish evaluation data and rerun the human and LLaMA comparisons; without that, the central claim cannot be accepted. I recommend rejection, but I would be open to reconsidering a substantially revised version with independent evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper is a reasonable engineering exercise in fine-tuning small models on synthetic Hinglish dialogues, and the new corpus is a real artifact. But the central claim that small models can approach 70B performance on Hinglish conversation is not supported, because the evaluation is largely circular: the test set, the LLaMA-70B comparison prompts, and the training data all come from Gemini-2.0-Flash. There are also number inconsistencies that make me trust the tables less.\n\nWhat is genuinely new: the authors built a ~3,000-dialogue (~5M token) synthetic Hinglish conversational corpus and report LoRA fine-tuning results for Qwen2.5-3B and 7B on that corpus. I could not find that exact combination in the cited literature. They also avoided BLEU/ROUGE and used CMI, BERTScore, and human preference, which is the right instinct for code-mixed text.\n\nThe soft spots are serious. Section 4.2 says all data was generated via Gemini-2.0-Flash; Section 6.1 splits that same synthetic corpus 80:10:10. The test prompts therefore come from the same generator, the same style, and the same topic templates as training. A model can score well on this benchmark by imitating Gemini's Hinglish style without generalizing to real romanized Hinglish. The LLaMA-70B benchmark (100 prompts, GPT-4o judge) reinforces that: it favors outputs that resemble the judge's own conventions. The human A/B test has only 10 prompts with no annotator demographics or agreement numbers. That is too small to prove the preference results.\n\nThere are also internal inconsistencies. The text claims a 25.6% gain in gender correctness for the 3B LoRA model, but Table 1 shows +8.9%. The text distinguishes LoRA and LoRA_LD variants but the tables only show one LoRA column, making the label unclear. These are not stylistic nits; they make it hard to know what exactly was evaluated.\n\nNone of this means the idea is wrong. Small models fine-tuned on good code-mixed data is a sensible direction, and the paper is transparent that the data is synthetic. But the current evidence cannot support 'competitive with LLaMA 70B.' The paper would need an external, naturalistic Hinglish test set and a larger, better-documented human evaluation before the claim stands.\n\nWho is this for? People working on code-mixed conversational AI who want a baseline on synthetic data + LoRA for Hinglish. It is a useful data point but not a proven result. I would send it to peer review because the artifact and the questions are real, but I would expect major revision.","headline":"A useful new synthetic Hinglish corpus and a plausible fine-tuning approach, but the circular evaluation and internal number inconsistencies leave the headline claims unestablished.","tokens_in":6122,"tokens_out":3925,"would_cite":false,"duration_ms":36652,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning small models on synthetic Hinglish data can reach near-70B conversational quality.","keywords":["Hinglish","code-mixed dialogue","synthetic training data","LoRA fine-tuning","sample efficiency","low-resource NLP","conversational AI","human preference evaluation"],"falsifier":"Collect a preference test on at least 50 naturally occurring Hinglish conversational exchanges from real users, asking bilingual raters to choose between the fine-tuned 3B or 7B model and a 70B model; if the fine-tuned models are not preferred on real exchanges despite winning on synthetic prompts, the paper's central claim is falsified.","tokens_in":5075,"feed_emoji":"💬","tokens_out":8160,"duration_ms":71897,"temperature":0.7,"pith_summary":"The paper tries to establish that parameter-efficient fine-tuning on carefully generated synthetic code-mixed dialogue lets small language models handle conversational Hinglish nearly as well as models an order of magnitude larger. The authors build a fully synthetic multi-turn Hinglish corpus, normalize spelling variants, and fine-tune 3B and 7B models with LoRA and QLoRA. Their reported gains are substantial: a 3B model improves 41.4% in Hinglish fluency and 42.4% in coherence, and a 7B fine-tuned model scores 4.1 against 4.6 for a 70B model on a 100-prompt benchmark. If correct, practical Hinglish chatbots do not require 70B-scale compute or large curated corpora; quality synthetic data plus targeted fine-tuning can substitute for raw scale.","feed_headline":"Fine-tuned 3B model nearly matches a 70B on Hinglish chat","feed_subtitle":"Synthetic Hinglish data plus LoRA lifts a small model's fluency 41% and coherence 42%.","key_machinery":"The load-bearing mechanism is the synthetic data pipeline: an API-driven instruction-tuned model generates over 3,000 structured multi-turn Hinglish dialogues, standardized by normalizing high-frequency romanized spelling variants, then used to fine-tune small open-weight models with LoRA and QLoRA. This pipeline supplies the high-quality code-mixed training signal that existing Hinglish datasets lack, and the parameter-efficient fine-tuning keeps the compute budget low enough that a 3B or 7B model becomes a practical deployment target.","core_discovery":"The central claim, stated on the paper's own terms, is that models with fewer parameters, when appropriately fine-tuned on high-quality code-mixed data, can achieve competitive performance for Hinglish conversation generation while maintaining computational efficiency. Operationally, the paper shows that LoRA fine-tuning on a synthetic Hinglish corpus raises human-rated fluency and coherence by double digits for a 3B model, and that a 7B model approaches the judged quality of a 70B model on a 100-prompt comparison. The authors interpret this as evidence that data quality outweighs raw parameter count for low-resource code-mixed dialogue, and that established metrics such as BLEU and ROUGE are poorly suited to valid Hinglish variation.","pith_inferences":["The ten-prompt human preference test and the synthetic evaluation prompts mean the current evidence is style-matched, since both training and evaluation are generated by the same API-driven model; a preference study on real, naturally occurring Hinglish exchanges would show whether the gains reflect competence in everyday code-mixing or alignment with the generator's style.","The competitive 7B-versus-70B comparison relies on a single judge model, so swapping that judge for human raters on the same 100 prompts would reveal whether the near-parity is an artifact of judge preference.","The sample-efficiency claim could be tested directly by varying the synthetic corpus size, such as 100, 500, and 3,000 dialogues, and measuring fluency gains, which would expose the marginal value of additional synthetic data for code-mixed dialogue.","The normalization of spelling variants assumes a small set of high-frequency canonical forms; a systematic study of which romanization variants matter most for downstream fluency would help extend the approach to other romanized languages."],"forward_implications":["If the claim holds, Hinglish chatbots can run on much smaller compute budgets, making on-device or low-resource deployment practical.","Benchmarks for code-mixed dialogue should follow this paper's example and avoid BLEU and ROUGE, using human preference and code-mixing-aware metrics instead.","A 3B model fine-tuned this way can serve as a lightweight default, with 7B reserved for higher-quality responses, making a 70B baseline unnecessary for most Hinglish interactions.","The same synthetic-data-plus-LoRA recipe is a candidate template for other code-mixed Indian languages such as Tanglish, Benglish, and Manglish.","Models below 3B parameters still struggle even with quality data, so the result does not remove all scale requirements."],"supporting_citations":[{"why":"Establishes the precedent that small-parameter models can perform well on a low-resource language with limited data, motivating the sample-efficiency approach.","marker":"Matzopoulos et al., 2025"},{"why":"Shows continued pre-training and synthetic augmentation on Hindi-English data for a 4B model, supporting the feasibility of targeted small-model adaptation.","marker":"Raviraj et al., 2024"},{"why":"Supplies the API-driven instruction-tuned model used to generate the entire synthetic Hinglish training corpus.","marker":"Google, 2024"},{"why":"Provides the Qwen2.5 base models that the paper fine-tunes, including the 3B and 7B variants used in the main experiments.","marker":"An Yang et. al., 2024"},{"why":"Documents the Gemma 3 family, the other evaluated architecture, and the trade-off between model capacity and efficiency.","marker":"DeepMind, 2025b"},{"why":"Supplies a Hinglish-English dataset that was considered but rejected for lacking conversational depth, supporting the paper's data-scarcity premise.","marker":"Srivastava and Singh, 2021"}],"fun_headline_variants":["Small fine-tuned model rivals 70B on Hinglish chat","LoRA + synthetic data shrink Hinglish model while keeping quality","Data quality beats size for Hinglish chatbots","Efficient Hinglish AI: 3B model nearly matches 70B","Sample-efficient Hinglish: small model, big performance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the synthetic Hinglish dialogues produced by the API-driven generator, plus the ten-prompt human preference test, represent real everyday Hinglish closely enough that measured gains transfer to actual users.","fun_headline_variants_meta":{"raw":{"variants":["Small fine-tuned model rivals 70B on Hinglish chat","LoRA + synthetic data shrink Hinglish model while keeping quality","Data quality beats size for Hinglish chatbots","Efficient Hinglish AI: 3B model nearly matches 70B","Sample-efficient Hinglish: small model, big performance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0002,"raw_usage":{"total_tokens":1318,"prompt_tokens":828,"completion_tokens":490,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":444,"completion_tokens_details":{"reasoning_tokens":411}},"tokens_in":444,"tokens_out":490,"duration_ms":4848,"temperature":1.0,"reasoning_tokens":411,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T06:01:26.690894+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a preference test on at least 50 naturally occurring Hinglish conversational exchanges from real users, asking bilingual raters to choose between the fine-tuned 3B or 7B model and a 70B model; if the fine-tuned models are not preferred on real exchanges despite winning on synthetic prompts, the paper's central claim is falsified.","supporting_citations":[{"cited_title":"BabyLMs for isiXhosa: Data-Efficient Language Modelling in a Low-Resource Context","cited_arxiv_id":"2501.03855","evidence_quote":"Establishes the precedent that small-parameter models can perform well on a low-resource language with limited data, motivating the sample-efficiency approach."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies a Hinglish-English dataset that was considered but rejected for lacking conversational depth, supporting the paper's data-scarcity premise."}],"review_version":1}