{"id":"786b6a0d-6edd-470e-9dea-daaeaf779f5f","arxiv_id":"2411.09073","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"CHAI trains a reward model on GPT-4o preference labels and uses PPO to align Llama-3.1-8B for English-to-Hinglish translation, claiming a 25.66% human win-rate improvement over baselines.","lead":"This paper introduces CHAI, a framework that uses AI-generated preference labels and reinforcement learning to improve code-mixed Hinglish translation in a large language model. The authors report a 25.66% win-rate gain over open-source baselines in human evaluation, but the evidence is uneven across baselines and lacks statistical rigor.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 25.66% win-rate claim is not statistically grounded: against the strongest SFT baseline the human win rate is only 52.15% (near chance), and no confidence intervals, significance tests, or inter-annotator agreement are reported.","rationale":"The most load-bearing link in the argument is the human win-rate result, because the automatic metrics do not uniformly favor CHAI: Table 3 shows CHAI below πsft-1 on chrF and chrF++ on both MixMT and HinGE, so the headline has to be carried by the preference evaluations. Both the reader and I place the same weakest assumption here: the preference evaluations are point estimates from a very small convenience sample with no inferential statistics. This is not a matter of disagreeing with the field's consensus; it is an internal evidential gap. The proposed binomial/bootstrap check is cheap and decisive: if the confidence interval for the strongest-baseline comparison straddles 50%, the averaged 25.66% claim cannot be accepted. I do not escalate to REJECT because the method is plausible, the code is released, and the COMET gains and sentiment results provide some independent positive signals; but the central quantitative claim should remain conditional on the release of raw judgments and the computed intervals. I mark verdict_should_be as UNCHANGED because the reader's conditional verdict already captures this concern.","tokens_in":17832,"tokens_out":10279,"duration_ms":132498,"concrete_test":"Recompute the CHAI-vs-πsft-1 human win rate from raw per-annotator judgments on MixMT-2022 and HinGE (the released repository should contain the vote data) and report: (a) an exact binomial 95% CI for the majority-vote win rate on each test set and on the pooled set; (b) a clustered bootstrap CI that resamples source sentences to account for sentence-level correlation across annotators; and (c) Fleiss' kappa among the three annotators. If any of the CIs includes 50% — as the point estimate 52.15% suggests — the pairwise claim against the strongest baseline should be downgraded, and the 25.66% headline should be replaced by per-baseline margins with intervals.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that CHAI-powered LLMs outperform state-of-the-art open-source LLMs by 25.66% in human-adjudicated win rate rests entirely on the human evaluation described in §4 and Appendix A.14. The 25.66% is an average of pairwise win-rate margins over three baselines, and it is dominated by the 76.91% margin against the custom πsft-2 baseline; against the stronger πsft-1 baseline the human win rate is only 52.15%, essentially chance. With 376 test sentences in MixMT and three convenience-sampled annotators, a 52.15% majority-vote win rate is not statistically distinguishable from 50% (exact binomial 95% CI roughly 47–57%, p≈0.4), and no confidence intervals, significance tests, or inter-annotator agreement are reported anywhere. Since automatic metrics in Table 3 are mixed (COMET improves but chrF/chrF++ are lower than the SFT baselines), the human preference result is the sole support for the headline; as reported, it does not establish a robust 25.66% advantage.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CHAI, a reinforcement-learning-from-AI-feedback (RLAIF) framework for improving code-mixed machine translation, specifically English-to-Hinglish. The pipeline uses GPT-4o to annotate pairwise preferences among candidate translations, trains a Bradley-Terry reward model on those AI labels, and fine-tunes Llama-3.1-8B-Instruct with PPO. The central claim is that CHAI-powered LLMs outperform state-of-the-art open-source LLMs by 25.66% in human-adjudicated win rate on code-mixed translation, with additional evidence from automatic metrics, cross-lingual transfer experiments, and code-mixed sentiment analysis.","tokens_in":17997,"tokens_out":4699,"duration_ms":45740,"significance":"If the headline result is correct, the paper would be a valuable first demonstration of RLAIF for code-mixed NLP, with a practical recipe (LLM-as-annotator, reward modeling, PPO) that could extend to other low-resource code-mixed tasks. The authors contribute useful ablations: the comparison with DPO (Table A5), the SFT-on/off study (Table 2), and the temperature sweep (Figure 2). The code is promised to be public. However, the contribution's weight rests almost entirely on the human-preference evaluation, and that evaluation is not yet statistically grounded. The cross-lingual results also contain an internal contradiction with the paper's own automatic metrics.","major_comments":[{"comment":"The headline claim of a 25.66% win-rate improvement is not supported by statistically meaningful evidence. Against the stronger SFT baseline πsft-1, the human-adjudicated win rate is only 52.15% (Figure 3). With the reported test sizes (376 MixMT + 395 HinGE sentences, possibly averaged across these), a 52.15% win rate is not distinguishable from 50% (approximate 95% CI roughly 48.6–55.7%, p > 0.2). No confidence intervals, significance tests, or inter-annotator agreement are reported anywhere in Section 4 or Appendix A.14. Because the automatic metrics in Table 3 are mixed — CHAI is below the SFT baselines on chrF/chrF++ on both datasets — the human preference result is the sole support for the abstract's claim, and as reported it does not establish a robust advantage.","section":"Section 4, 'Evaluation Metrics' and Figure 3; Appendix A.14"},{"comment":"The paper never defines how the 25.66% figure is computed. Figure 3 reports per-baseline win rates (60.65%, 52.15%, 76.91% against πbase, πsft-1, πsft-2, respectively), but the abstract and introduction quote 25.66% without a formula or a description of the aggregation. The manuscript should state exactly how this number is derived (e.g., average of win-rate margins above 50%) and should report the per-comparison win rates with uncertainty bounds for each dataset separately.","section":"Abstract and Section 4, Figure 3"},{"comment":"The text claims that 'all classic metrics consistently favor CHAI-LLM' when describing cross-lingual transfer, but Table 4 contradicts this for the English→French direction: chrF drops from 34.88 (πbase) to 22.07 (CHAI-LLM), and chrF++ drops from 31.52 to 19.85. The later statement that the RLAIF procedure improves transfer on 'two out of three language pairs' is accurate for chrF, but the 'consistently favor' sentence is incorrect and must be corrected. The near-chance LLM win rate for English→Bengali (50.11%) also undercuts the strength of the cross-lingual claim.","section":"Section 4, 'Cross-lingual Transferability' and Table 4"},{"comment":"The automatic metrics do not corroborate the human-preference claim. On both MixMT 2022 and HinGE, CHAI-LLM has lower chrF and chrF++ than both SFT baselines (e.g., MixMT chrF: 42.68 vs. 48.57 for πsft-1), while COMET increases by only about 0.02–0.04. The paper dismisses this with a general reference to Winata et al. (2024), but no evidence is provided that chrF is an invalid indicator for Hinglish code-mixed translation specifically. The authors should either report metric–human correlation on this task or discuss more directly why the SFT baselines are preferred by chrF while the human evaluators (weakly) prefer CHAI.","section":"Table 3 and Section 4, 'Impact of CHAI on Translation Quality'"},{"comment":"A core contribution is the claim that LLM annotators can accurately label code-mixed translation preferences. The maximum alignment with human preferences reported in Table 1 is 61.8%, which is modest and is measured on a single 1000-sample evaluation set (Appendix A.3) with no confidence intervals or inter-annotator agreement for the human labels. The reward model is trained on these noisy AI labels, so the entire RLAIF pipeline is only as trustworthy as the 61.8% alignment. The paper should provide a more complete assessment of this agreement (per-category, with variance) and discuss the implications of the noise for reward-model training.","section":"Table 1 and Section 3, 'Collecting Preference Data Using LLMs'"}],"minor_comments":[{"comment":"The description of the human evaluation protocol mentions only the MixMT 2022 test set, while Figure 3 and the text refer to win rates on both MixMT 2022 and HinGE. Please clarify which test sets were used for human adjudication and whether the reported win rates are per-dataset or averaged.","section":"Section 4, 'Evaluation Metrics'"},{"comment":"The phrase 'outperform ... by 25.66% (in terms of win rate)' is ambiguous because win rate is bounded by 0–100%, and the reader cannot tell whether 25.66% is an absolute margin over 50%, a relative improvement, or an average of different margins. Please define the quantity precisely.","section":"Abstract and Conclusion"},{"comment":"There are several typographical errors and inconsistent spellings: 'developmennt' (Section 7), 'renumeration' (Section 7), 'alternated' in the human-evaluation rules (Appendix A.10), 'RAILF' instead of 'RLAIF' (Appendix A.5), and inconsistent 'Llama'/'LlaMA' spellings. A thorough proofread is needed.","section":"Throughout"},{"comment":"The recruitment details state only that three annotators were recruited from a university via convenience sampling and paid $25/hour; no information is given about proficiency screening, annotator agreement, or the decision to use only three annotators. This is material to the reliability of the headline result.","section":"Appendix A.14"}],"recommendation":"major_revision","confidential_remarks":"The paper makes a substantial claim, but as it stands the evidence is not yet convincing. The authors should be encouraged to add a rigorous statistical analysis of the human evaluation (confidence intervals, significance tests, and inter-annotator agreement), to define the 25.66% figure explicitly, and to correct the cross-lingual text that is contradicted by Table 4. If these points are addressed, the contribution could be acceptable; without them, the headline result is not established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is the first RLAIF pipeline aimed at code-mixed translation, and it ships a clean three-stage recipe: LLM-as-annotator for preferences, Bradley-Terry reward model, and PPO. The prompt-design study for the annotator is genuinely useful: they systematically compare basic, rule-augmented, CoT, and few-shot prompts, and find that rule-augmented 0-shot works best (61.8% agreement with human labels). That is honest negative evidence about CoT and few-shot for code-mixed annotation. They also report that adding SFT before RL hurts, and they remove it; that's a useful ablative finding. The central claim is where things get soft. The 25.66% win-rate improvement in the abstract is a real overstatement. It is the average of pairwise win margins over three baselines, and it leans on a 76.91% win rate against a much weaker second SFT baseline. Against the stronger SFT baseline, the human win rate is 52.15%, which with three majority-vote annotators on roughly 376 sentences is basically chance. No confidence intervals, no significance tests, no inter-annotator agreement. The automatic metrics are mixed: COMET improves, chrF/chrF++ actually drop against both SFT baselines, so the human preference result is doing all the work, and it isn't statistically supported. Some other soft spots, smaller ones. The cross-lingual section says CHAI improves transfer on two of three language pairs, but the En+Be result is 50.11% to 49.89%, a toss-up. The DPO comparison likely under-tunes DPO (beta=0.1, lr=5e-6) relative to the RLAIF setup. And the code link looks like a placeholder repo name, which makes the reproducibility promise hard to verify. That said, the framework is plausible and the paper is honest about its limitations: it acknowledges the SFT failure, the single language pair, the single model size, and the noise in code-mixed data. The work is not a shabby parlor trick; it's a serious empirical attempt with an inflated headline. Who is this for: people working on alignment for low-resource or code-mixed scenarios. I would not take the headline at face value, but I would send it to a referee because the core question, whether RLAIF can improve code-mixed translation, is worth a careful look. My recommendation: engage, but require the authors to add significance testing, report IAA, use a held-out validation set for hyperparameters, and reword the abstract to match what the data actually show.","headline":"Useful first RLAIF-for-code-mixed-translation recipe, but the 25.66% headline is statistically unsupported; the paper deserves review but only after the authors add significance testing and temper claims.","tokens_in":764,"tokens_out":1796,"would_cite":true,"duration_ms":44460,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that CHAI, an RLAIF pipeline using GPT-4o preference labels, lifts human-preferred Hinglish translation quality of a Llama-3.1 model by 25.66% in win rate.","keywords":["code-mixed translation","Hinglish","RLAIF","LLM as annotator","machine translation","preference data","PPO","Llama-3.1-8B-Instruct"],"falsifier":"Run pairwise human evaluation of CHAI-LLM versus the strongest SFT baseline on the MixMT and HinGE test sets with at least 20 native Hinglish speakers, record individual preferences, and compute a confidence interval for the win rate; if the interval contains 50% or excludes the reported margin, the central human-preference claim is not supported. An inter-annotator agreement no better than chance would similarly undermine the averaging.","tokens_in":17553,"feed_emoji":"🌐","tokens_out":8128,"duration_ms":81455,"temperature":0.7,"pith_summary":"The paper claims that a general-purpose alignment pipeline called CHAI — reinforcement learning from AI feedback (RLAIF) built on GPT-4o preference labels — can substantially improve how large language models translate monolingual English text into code-mixed Hinglish. The authors report that the CHAI-tuned Llama-3.1-8B-Instruct model beats state-of-the-art open baselines by 25.66% in human-adjudicated win rate on code-mixed translation, and also outscores them on COMET and on downstream Hinglish sentiment classification. The argument matters because human preference annotation for code-mixed languages is scarce and expensive; if LLM annotators can supply reliable preference data, alignment-based improvement becomes scalable for other low-resource code-mixed languages. The paper also reports a cautionary internal result: supervised fine-tuning on the parallel corpus hurt final quality, and chrF-style lexical metrics favor the SFT baseline even where human and LLM judges prefer CHAI outputs.","feed_headline":"AI feedback lifts code-mixed translation win rate by 25.66%","feed_subtitle":"GPT-4o preference labels replace costly human annotation to make Hinglish output read more naturally","key_machinery":"The load-bearing machinery is the AI preference-labeling loop. CHAI pairs each English source sentence with two Hinglish translations, has GPT-4o (with randomly switched candidate order, three temperature settings, and majority voting) pick the better one, trains a reward model on those binary labels via the Bradley-Terry model, and optimizes the policy with PPO using a KL penalty to stay near the base distribution. The same preference pipeline is also used to compute win rates, with Gemini-1.5-Flash and three human annotators acting as judges. The paper's key finding about this machinery is that simple rule-augmented prompts outperform chain-of-thought and few-shot prompting for LLM annotators on code-mixed data, and that the SFT stage of a conventional RLAIF recipe is counterproductive in this setting.","core_discovery":"On the paper's own terms, the central discovery is that preference-based alignment works for code-mixed translation when the preference labels come from an AI annotator. CHAI first asks GPT-4o, at three temperatures, to choose the better of two Hinglish translations of the same English sentence; majority vote gives a binary label. Those labels train a Bradley-Terry reward model, and a PPO policy update with a KL penalty aligns Llama-3.1-8B-Instruct to the reward. The authors find that basic rule-augmented prompts align LLM annotators with human judges 61.8% of the time, and that the resulting CHAI-LLM wins human pairwise comparisons against the base model, two SFT baselines, and a DPO baseline, while improving Hinglish sentiment accuracy and showing mostly positive cross-lingual transfer to English-Bengali and English-Spanish. The lexical metrics chrF and chrF++ do not track the human preference result, which the authors attribute to known weaknesses of n-gram metrics for code-mixed output.","pith_inferences":["Beyond the paper: because the LLM annotator agrees with humans only about 62% of the time, preference data carries an annotator-model-specific bias; a different labeler than GPT-4o could produce a differently flavored 'best Hinglish'.","Beyond the paper: the human evaluation's three-person convenience sample means the claimed 25.66% margin is a point estimate with unknown spread; rerunning with a larger, more diverse annotator pool would likely shrink the margin even if the direction survives.","Beyond the paper: the counterproductive SFT result suggests code-mixed corpus noise, not model capacity, may be the binding constraint; a testable extension is to filter noisy parallel data before SFT and see whether quality improves.","Beyond the paper: the same RLAIF recipe could be applied to code-mixed summarization or language identification once preference data are generated, since the sentiment gains suggest task transfer within code-mixed understanding."],"forward_implications":["If the win-rate results hold, LLM-generated preference labels can replace expensive human annotation for code-mixed translation alignment, making RLAIF practical for other low-resource language pairs.","The CHAI-trained model should generalize beyond translation: the same alignment improved Hinglish sentiment accuracy and F1 by double digits on two datasets.","The cross-lingual gains, especially to English-Spanish code-mixing, suggest some preference knowledge learned on Hinglish transfers to other code-mixed pairs, while English-French did not clearly benefit.","Evaluation of code-mixed systems should weight human or LLM preference over chrF and chrF++, since the lexical metrics contradict the judged preference ordering.","Because SFT lowered final quality, instruction-tuned base models may need a modified RLAIF recipe that skips or rethinks the supervised fine-tuning stage."],"supporting_citations":[{"why":"Supplies the MixMT parallel corpus and test set used for English-to-Hinglish translation.","marker":"(Srivastava and Singh, 2022)"},{"why":"Supplies the ALL-CS Hinglish parallel corpus used to build the preference dataset.","marker":"(Tarunesh et al., 2021)"},{"why":"Supplies the HinGE test set used for evaluating translation quality.","marker":"(Srivastava and Singh, 2021)"},{"why":"GPT-4o is the LLM annotator that generates the preference labels at three temperatures.","marker":"(OpenAI et al., 2024)"},{"why":"Provides the Bradley-Terry model used to train the reward model from binary preference pairs.","marker":"(Bradley and Terry, 1952)"},{"why":"Provides the PPO algorithm used to optimize the policy against the reward model.","marker":"(Schulman et al., 2017)"},{"why":"Introduced the RLAIF/Constitutional AI paradigm that CHAI adapts to code-mixed translation.","marker":"(Bai et al., 2022)"},{"why":"Establishes the RLAIF-vs-RLHF procedure and the win-rate evaluation used to compare CHAI with baselines.","marker":"(Lee et al., 2024)"},{"why":"Llama-3.1-8B-Instruct is the base model that CHAI aligns.","marker":"(Grattafiori et al., 2024)"}],"fun_headline_variants":["AI feedback improves code-mixed translation by 25.66%","Using AI preferences to align LLMs for code-mixed translation","CHAI framework: AI feedback trains LLM for Hinglish translation","Preference learning from GPT-4o lifts code-mixed translation quality","Reinforcement learning with AI feedback enhances code-mixed translation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire 25.66% human-preference claim rests on majority votes from only three convenience-recruited annotators, with no confidence intervals, significance tests, or inter-annotator agreement reported, and the human win rate over the strongest SFT baseline is only 52.15%.","fun_headline_variants_meta":{"raw":{"variants":["AI feedback improves code-mixed translation by 25.66%","Using AI preferences to align LLMs for code-mixed translation","CHAI framework: AI feedback trains LLM for Hinglish translation","Preference learning from GPT-4o lifts code-mixed translation quality","Reinforcement learning with AI feedback enhances code-mixed translation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000331,"raw_usage":{"total_tokens":1887,"prompt_tokens":1035,"completion_tokens":852,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":651,"completion_tokens_details":{"reasoning_tokens":761}},"tokens_in":651,"tokens_out":852,"duration_ms":7528,"temperature":1.0,"reasoning_tokens":761,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:04:46.688002+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run pairwise human evaluation of CHAI-LLM versus the strongest SFT baseline on the MixMT and HinGE test sets with at least 20 native Hinglish speakers, record individual preferences, and compute a confidence interval for the win rate; if the interval contains 50% or excludes the reported margin, the central human-preference claim is not supported. An inter-annotator agreement no better than chance would similarly undermine the averaging.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MixMT parallel corpus and test set used for English-to-Hinglish translation."},{"cited_title":"HinGE: A Dataset for Generation and Evaluation of Code-Mixed Hinglish Text","cited_arxiv_id":"2107.03760","evidence_quote":"Supplies the HinGE test set used for evaluating translation quality."}],"review_version":1}