{"id":"09e859d9-37e3-48a7-bc79-0ba7c93e5204","arxiv_id":"2505.00114","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Fine-tuning on 3,000 culturally authentic Lebanese sentences appears to beat 140,000 translated sentences, but the evidence is weakened by a potentially non-independent evaluation set and conflicting FLoRes results.","lead":"A team at the American University of Beirut fine-tuned an open language model to translate Lebanese Arabic, testing small culturally authentic data against much larger translated data. Their headline result is undermined by a possibly overlapping test set and contradictory scores on an existing benchmark.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LebEval is not shown to be disjoint from the LW training data; the only consistent LW advantage appears on LebEval, so the central 'data quality over quantity' claim rests on a possible train/test leakage.","rationale":"The reader identified the weakest assumption correctly: the independence of LebEval from the LW training data. The paper makes a strong causal claim about authenticity, but its evidence reduces to a 70-sentence benchmark whose disjointness from the 3K training sentences is never established. The FLoRes results in Table 1 actually point the other way, with NN-trained models better on FLoRes in nearly every configuration. This pattern is exactly what would be observed if LebEval were not independent: LW models match or memorize the conversational podcast domain and win only on the author-created test set. No alternative reading rescues the headline claim, because 'consistently' is contradicted outside LebEval. I would not change the reader's REJECT verdict. I also credit the paper's useful contributions—LebEval as a resource, the contrastive/grammar instruction comparisons, and the released code/data—but the central empirical claim is not supported by the current experimental design. The small test size and absence of error bars further weaken the 1–2 point differences, but the independence issue is primary.","tokens_in":13327,"tokens_out":8245,"duration_ms":83238,"concrete_test":"Run the head-to-head comparison (Instruct-Cont-LW vs Instruct-Cont-NN, 0-shot/3-shot/C3-shot, 3 random seeds) on a held-out native Lebanese test set built from a source verified to share no episodes or sentences with LanguageWave, e.g., a different Lebanese podcast channel or transcribed interviews. If LW does not consistently win on that held-out set, the LebEval result is a domain-match artifact and the central claim fails; if LW still wins, the independence concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline claim that LW-trained models 'consistently outperform' NN-trained models is contradicted by the paper's own FLoRes results: in Table 1, NN matches or beats LW in 11 of 12 non-vanilla FLoRes comparisons (e.g., Instruct-Cont-NN C3-shot 89.1 vs Instruct-Cont-LW 87.4). The LW advantage appears only on LebEval, the authors' new 70-sentence benchmark (Section 3.4). The load-bearing assumption is that LebEval is independent evidence: LW training data (Section 3.1) is described only as ~3K sentences from 'a Lebanese podcast' (95 episodes), while LebEval is built from 15 episodes of the 'Levantine Arabic Made Easier' podcast. The paper provides no episode IDs, no sentence-level deduplication, and no overlap analysis, so it never rules out that LebEval sentences—or the same conversational domain—occur in LW training. Because both datasets are native conversational Lebanese, the systematic LW advantage on LebEval can be fully explained by train/test overlap or domain match rather than by 'data quality over quantity.' Without LebEval's independence, the central claim has no supporting evidence; the FLoRes numbers point the other way.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies fine-tuning Aya23-8B for Lebanese-to-English translation, comparing adapters trained on a small native podcast-derived dataset (LW, about 3K sentences) with adapters trained on a larger translated corpus (NN, 140K sentences) under basic, contrastive, and grammar-hint instruction formats, with and without curriculum training, and under zero-shot, few-shot, and contrastive prompting. The paper also introduces LebEval, a 70-sentence evaluation set constructed by the authors from a Lebanese podcast, and evaluates all models with reference-free xCOMET on LebEval and a 500-sentence FLoRes subset. The headline claims are that LW-trained models consistently outperform NN-trained models, that contrastive fine-tuning paired with contrastive prompting is best, and that authentic evaluation benchmarks are necessary because FLoRes scores overestimate dialect translation quality.","tokens_in":13628,"tokens_out":11447,"duration_ms":103847,"significance":"Should these claims hold, the paper would contribute to low-resource dialect MT and to the 'data quality over quantity' debate. The experimental design is broad (three instruction types, two data sources, three prompting strategies, two curricula, and a preference-alignment comparison), the work targets an under-resourced dialect, and the authors state that code and data are released, which supports reproducibility. The introduction of LebEval is also a potentially useful resource. However, the central quality-over-quantity claim is contradicted by the paper's own FLoRes results, and the only benchmark that shows an aggregate LW advantage is LebEval, whose independence from the LW training distribution is not established. Section 7 candidly lists limitations for the synthetic grammar data and preference alignment, but it does not address either of these load-bearing issues. As presented, the empirical basis for the headline conclusion is not sufficient.","major_comments":[{"comment":"The abstract and Section 4.2 claim that LW-trained models 'consistently' outperform NN-trained models, but the FLoRes columns in Table 1 contradict this. Across the 12 paired NN/LW comparisons (Instruct-MT, Instruct-Cont, CONT+MT, and Gram+CONT+MT in each prompting setting), NN-trained models score higher than LW-trained models in 11 cases (e.g., 0-shot Instruct-MT: 87.6 vs 86.9; C3-shot Instruct-Cont: 89.1 vs 87.4); LW leads only in the C3-shot Instruct-MT setting, by 0.3 points (87.0 vs 86.7). Even on LebEval the pattern is not universal: LW wins eight of the twelve paired comparisons, NN wins two (0-shot Instruct-Cont 71.8 vs 71.7; C3-shot Instruct-MT 71.1 vs 71.0), and two are ties. Because the paper's central claim is stated as a consistent advantage, the authors' own Table 1 is evidence against it, and Section 7 does not acknowledge the contradiction.","section":"Section 4.2, Table 1"},{"comment":"LebEval is not shown to be independent of the LW training data. Section 3.1 describes LW as approximately 3K sentences from 95 episodes of a Lebanese podcast; Section 3.4 describes LebEval as 70 sentences from 15 episodes of the 'Levantine Arabic Made Easier' podcast. The paper does not identify the LW podcast, does not provide episode IDs, does not perform or report sentence-level deduplication against LW, and contains no statement that the two corpora are disjoint. Because both corpora are native conversational Lebanese and the LW advantage is largest on LebEval, the observed result may be a train/test leakage or domain-match artifact rather than evidence for data quality over quantity. This missing overlap analysis is load-bearing and must be supplied before any conclusion can be drawn from LebEval.","section":"Sections 3.1 and 3.4"},{"comment":"The paper interprets the raw xCOMET gap between FLoRes (85.5 for the base model) and LebEval (68.7) as evidence that culturally generic benchmarks misrepresent dialect translation quality. However, xCOMET scores are not calibrated across test sets; the two benchmarks differ in source genre, sentence length, transcription quality, and difficulty. A matched calibration or pairwise human judgments would be needed to support the claim that FLoRes is too easy or culturally generic. As written, the cross-benchmark conclusion in Section 4.2 is not supported.","section":"Section 4.2, Table 1 (Vanilla row)"},{"comment":"No confidence intervals, bootstrap errors, or significance tests accompany any of the headline comparisons. LebEval contains 70 sentences, and many of the reported gaps are 0.1-0.7 xCOMET points (e.g., 0-shot Instruct-Cont on LebEval: 71.8 vs 71.7; C3-shot Instruct-MT on FLoRes: 86.7 vs 87.0). Without seed variance or interval estimates, the ranking of adapters and prompting strategies in Tables 1 and 2 cannot be distinguished from metric noise, especially because many configurations are compared at once.","section":"Sections 3.4 and 4.2"}],"minor_comments":[{"comment":"The text contains an unresolved placeholder: 'With the rise of powerful open-source LLMs such as BLOOM (cite appropriately) and LLaMA' should be replaced with a proper citation.","section":"Appendix A"},{"comment":"The row label 'Instuct-MT-NN' should read 'Instruct-MT-NN', and the model name is written both as Aya23-8B and Aya23-8b; please standardize.","section":"Table 1"},{"comment":"There is a typo ('Altough'), and the benchmark name is spelled inconsistently as FLoRes and FLORES; please unify.","section":"Section 3.4"},{"comment":"The xCOMET-10.7B metric is used without a version identifier or citation; because xCOMET versions change the score scale, this should be specified for reproducibility.","section":"Section 3.4"},{"comment":"The frequency-based few-shot selection method relies on an unspecified 'large Lebanese corpus' and an unreported frequency threshold; these details should be given for reproducibility.","section":"Section 4.1"},{"comment":"The abstract states that datasets and code are available on GitHub, but no repository URL is provided; the GitHub release of LebEval and LW should be clearly linked.","section":"Abstract and Section 3.1"},{"comment":"No manual validation or quality checks are reported for the grammar instructions synthesized by Claude 3.5 Sonnet; since Section 7 acknowledges this limitation, a brief description of any validation performed would be helpful.","section":"Section 2.3 and Section 7"}],"recommendation":"reject","confidential_remarks":"To the editor: The paper has a useful experimental design and the release of datasets and code is commendable, but the central claim is contradicted by the paper's own Table 1 on FLoRes, and the only benchmark that shows an aggregate LW advantage (LebEval) is not shown to be independent of the LW training distribution. These are load-bearing issues that require substantial additional evidence and a reframing of the claims. Should the authors resubmit with an overlap analysis, significance testing, and a more restricted claim, the work could be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth a look for the LebEval benchmark and the careful comparison of fine-tuning regimes on Lebanese, but the central claim as stated doesn't survive its own results. On FLoRes, the NN-trained models beat the LW-trained models in nearly every setting (e.g., Instruct-Cont-NN 3-shot 89.1 vs Instruct-Cont-LW 87.4). The only place LW consistently wins is LebEval, a 70-sentence set the authors built from a Lebanese podcast. They never show that LebEval is disjoint from the LanguageWave training data, which also comes from a Lebanese podcast, and they give no episode IDs or dedup analysis. Even without literal overlap, both are conversational Lebanese, so the LW advantage on LebEval could simply be domain match, not 'data quality over quantity.' That's a load-bearing flaw in the abstract and Section 4.2.\n\nWhat's actually new: LebEval is a small but genuinely native evaluation set, and the paper reports a clean comparison of basic, contrastive, and grammar-hint SFT plus two curriculum schedules on Aya23-8B. The contrastive-tuning result, best when paired with contrastive prompting, is consistent with prior work, and the negative result for curriculum learning is a useful data point. They also release code and data, which helps reproducibility.\n\nSoft spots beyond the main one: no error bars or significance tests on the 70-sentence set, so differences under 2 points are not clearly real; the few-shot K and selection ablation appears to tune on the test sets themselves; and the grammar-hint experiments rely on synthesized data from Claude, which they acknowledge. None of these are fatal on their own, but they compound the core problem.\n\nOverall: the paper is not publishable in its current form. The right fix is to either reframe the claim as 'LW helps on native conversational data, not on FLoRes' or to provide real evidence of independence and significance. As is, I'd recommend major revision with a careful reviewer, not rejection without review — the benchmark and the empirical map could be useful to the community once the claims are made honest.\n\nRecommendation: send to peer review, but flag the FLoRes contradiction and the LebEval independence question to the reviewers.","headline":"A useful new benchmark and a clean empirical setup are undermined by a headline claim that the paper's own Table 1 contradicts; the LW advantage only shows up on a small author-built test set of unclear independence.","tokens_in":14109,"tokens_out":6294,"would_cite":false,"duration_ms":56475,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning Aya23-8B on 3,000 authentic Lebanese sentences beats training on 140,000 translated ones.","keywords":["Lebanese dialect","machine translation","low-resource languages","contrastive fine-tuning","contrastive prompting","cultural authenticity","data quality over quantity","LebEval benchmark"],"falsifier":"Build a held-out Lebanese-English test set from a different source genre, such as social-media comments or transcribed news interviews, that does not share the podcast domain of Language Wave, and compare the LW-trained adapters against the NN-trained adapters on it; if the 140K-data models match or beat the 3K-data model there, the authenticity-over-quantity claim collapses into a domain-match effect. As a second check, have native speakers rate the best model's translations for idiom preservation and cultural appropriateness, and see whether the xCOMET gains track human preferences.","tokens_in":13139,"feed_emoji":"🗣️","tokens_out":8402,"duration_ms":77416,"temperature":0.7,"pith_summary":"The paper sets out to show that for translating the low-resource Lebanese dialect, a few thousand culturally authentic sentence pairs beat a much larger corpus of translated text, and that teaching a model to prefer good translations over bad ones improves quality further. It reports systematic fine-tuning experiments on the open-source Aya23-8B model, comparing basic translation instructions, contrastive instructions with chosen and rejected outputs, and grammar-hint instructions, trained on either the 3K-sentence Language Wave dataset or a 140K-sentence non-native collection. It also introduces LebEval, a 70-sentence evaluation set drawn from native Lebanese podcast content, and finds that models score much lower there than on the translated FLoRes benchmark. The best result comes from contrastive fine-tuning paired with contrastive prompting, which the paper reads as evidence against the assumption that more training data always means better translation.","feed_headline":"3,000 native sentences beat 140,000 translated ones","feed_subtitle":"Fine-tuning Aya23-8B on authentic Lebanese data plus contrastive training gives the best dialect translations.","key_machinery":"The load-bearing mechanism is contrastive instruction tuning. Each training example concatenates a chosen translation and a rejected translation, separated by '<rather than>' and accompanied by a hint field stating the preference, so the model learns to distinguish good from bad output; at inference the same idea is applied by including good and bad examples in a contrastive prompt. The other components are the Aya23-8B base model fine-tuned with efficient low-rank adaptation, the small culturally aware Language Wave dataset versus the larger non-native collection, and the LebEval benchmark used for evaluation.","core_discovery":"On the paper's own terms, the central discovery is that data provenance matters more than data volume for dialectal translation: fine-tuning on the culturally aware Language Wave dataset, roughly 3,000 sentences from a Lebanese podcast, consistently outperforms fine-tuning on roughly 140,000 sentences of translated subtitles and travel feedback, across every prompting strategy tested. Among the training styles, contrastive instruction tuning, where the model sees a preferred translation and a rejected one in the same training example, outperforms plain translation tuning, and the gap widens when contrastive prompting is used at inference, with the Instruct-Cont-LW adapter reaching the highest xCOMET score of 74.4 on LebEval. The paper also claims that native-content evaluation is essential: the base model and all adapters score far higher on FLoRes than on LebEval, while curriculum learning without separate model stages yields no significant gains, and preference optimization with model-generated rejected translations underperforms contrastive fine-tuning.","pith_inferences":["Because LebEval and Language Wave are both drawn from Lebanese podcast content, the small-data advantage may partly reflect domain match rather than cultural authenticity; a test set from a different genre would separate the two explanations.","The contrastive-tuning result suggests a broader principle for dialectal and cultural translation: explicit negative examples teach the model which fluent-sounding renderings are wrong, which plain parallel data cannot do.","The same design could be tested on other dialects or on culturally rich language pairs outside Arabic; if the pattern holds, it would generalize the authenticity-over-volume claim beyond Lebanese.","A human evaluation focused on idiom preservation and cultural appropriateness would tell whether the xCOMET gains correspond to the cultural nuance that matters for this translation task."],"forward_implications":["A practical recipe for low-resource dialect translation emerges: collect a few thousand authentic parallel sentences from native speakers and prefer that over scraping large translated corpora.","Contrastive training data can be built cheaply by pairing golden translations with model-generated suboptimal outputs, and the same good/bad contrast can be moved into the prompt at test time.","Scores on translated benchmarks such as FLoRes overstate how well a model handles real dialectal text, so native-content benchmarks are a more demanding and more honest yardstick.","Curriculum learning and grammar-hint tuning did not pay off under the tested settings, so gains from those strategies should not be assumed without dedicated study.","Preference-alignment methods that rely on model-generated rejected examples can underperform simple contrastive fine-tuning in this setting."],"supporting_citations":[{"why":"Provides the open-source Aya23-8B base model that all fine-tuning experiments start from.","marker":"(Aryabumi et al., 2024)"},{"why":"Supplies the Language Wave culturally aware parallel dataset and the finding that xCOMET correlates best with human judgment for Lebanese translation.","marker":"(Yakhni and Chehab, 2025)"},{"why":"Provides MADAR, part of the larger non-native training collection used as the comparison condition.","marker":"(Bouamor et al., 2018)"},{"why":"Provides Open Subtitles, the other non-native dataset in the 140K training collection.","marker":"(Krubínski et al., 2023)"},{"why":"Establishes the contrastive alignment instruction method that the paper adapts for dialectal translation.","marker":"(Mao and Yu, 2024)"},{"why":"Provides the error-guided contrastive prompting approach used to build contrastive prompts.","marker":"(Jiao et al., 2023a)"},{"why":"Supplies the FLoRes benchmark used as the translated-data comparison test set.","marker":"(team et al., 2022)"},{"why":"Defines contrastive preference optimization (CPO), the preference alignment method that underperforms in the comparison.","marker":"(Xu et al., 2024b)"}],"fun_headline_variants":["3k native sentences beat 140k translated for dialect translation","Contrastive tuning yields best low-resource dialect model","Data source matters more than data size in LLM translation","Authentic data beats bigger sets for Lebanese translation","Smaller, native dataset wins over massive translated corpus"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that smaller culturally authentic data beats larger non-native data rests on LebEval being an independent test set, but the paper never shows that the LebEval sentences are disjoint from the Language Wave training sentences or that the two podcast-derived sources do not share a domain.","fun_headline_variants_meta":{"raw":{"variants":["3k native sentences beat 140k translated for dialect translation","Contrastive tuning yields best low-resource dialect model","Data source matters more than data size in LLM translation","Authentic data beats bigger sets for Lebanese translation","Smaller, native dataset wins over massive translated corpus"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000218,"raw_usage":{"total_tokens":1418,"prompt_tokens":903,"completion_tokens":515,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":436}},"tokens_in":519,"tokens_out":515,"duration_ms":5313,"temperature":1.0,"reasoning_tokens":436,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:50:33.766263+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a held-out Lebanese-English test set from a different source genre, such as social-media comments or transcribed news interviews, that does not share the podcast domain of Language Wave, and compare the LW-trained adapters against the NN-trained adapters on it; if the 140K-data models match or beat the 3K-data model there, the authenticity-over-quantity claim collapses into a domain-match effect. As a second check, have native speakers rate the best model's translations for idiom preservation and cultural appropriateness, and see whether the xCOMET gains track human preferences.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Language Wave culturally aware parallel dataset and the finding that xCOMET correlates best with human judgment for Lebanese translation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides MADAR, part of the larger non-native training collection used as the comparison condition."},{"cited_title":"Tuning LLMs with Contrastive Alignment Instructions for Machine Translation in Unseen, Low-resource Languages","cited_arxiv_id":"2401.05811","evidence_quote":"Establishes the contrastive alignment instruction method that the paper adapts for dialectal translation."}],"review_version":1}