Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

When Life Gives You Samples: The Benefits of Scaling up Inference Compute for Multilingual LLMs

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper shows that generating five candidate outputs and selecting among them with multilingual-aware sampling and judging outperforms single-sample decoding across languages, adding up to 17.3 win-rate points for an 8B model.

desk verdict A genuinely multi-task, seven-language study of sampling and selection for inference-time scaling, with practical methods and honest limitations, but the headline win-rate results would be stronger with per-language judge agreement checks and error bars. read the letter →

arxiv 2506.20544 v1 pith:SZWQ72VG submitted 2025-06-25 cs.CL cs.AI

Ammar Khairi , Daniel D'souza , Ye Shen , Julia Kreutzer , Sara Hooker This is my paper
classification cs.CLcs.AI
keywords inference-timecomputemultilingualLLMshedgedsamplingCHOPScross-lingualMBRBest-of-Ntest-timescalingopen-endedgeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that spending a small amount of extra inference-time compute—generating five candidate outputs instead of one—can reliably improve multilingual LLM quality, provided both how samples are drawn and how the best one is chosen are adapted to language diversity. English-centric recipes fail in non-English languages because high-temperature sampling raises the risk of poor outputs more sharply outside English; the authors' 'hedged sampling' counters this by always including a greedy decoding in the pool. For selection, they propose two methods: CHOPS, which asks an LLM judge to write a task-specific checklist and pick the best answer in one pass, and X-MBR, which extends minimum Bayes risk selection with evidence samples drawn in a stronger language. Together these methods add up to +17.3 win-rate points for Aya Expanse 8B and +9.0 for Command-A (111B) on multilingual open-ended benchmarks, against single-sample decoding, and +6.8 average points against Gemini 2.0 Flash.

What carries the argument

The machinery is a two-stage pipeline for parallel inference-time scaling. Sampling creates a pool of $N=5$ candidates. The paper's hedged sampling mixes stochastic samples at temperature $\tau \in [0.7, 0.9]$ with one deterministic greedy ($\tau=0$) output, because non-English languages show higher 'risk'—the worst sample's quality drops faster at high temperature—so the greedy sample acts as a safety net. Selection then picks one candidate. The paper compares Best-of-N with a reward model (scoring each sample independently), Minimum Bayes Risk (MBR) with pairwise LLM-judge comparisons, and two new methods: CHOPS, which asks an LLM judge to first write a prompt-specific evaluation checklist and then select the best candidate in a single pass ($O(1)$ judge calls); and X-MBR, which extends the MBR evidence set with $M=3$ extra samples written in a dominant language (Chinese for English prompts, English otherwise), leveraging cross-lingual generation to sharpen selection.

What would settle it

Run the same hedged sampling with CHOPS and X-MBR on m-ArenaHard in the seven test languages, then have human annotators rate the selected outputs and compare win rates per language against GPT-4o's ratings. If the methods' advantage shrinks or reverses for some languages when ground truth is human preference rather than the automated judge, the central claim would be shown to be judge-dependent.

Watch

Extended reading notes

Core claim

The paper's central claim is that parallel inference-time scaling—sample multiple outputs, then select one—generalizes to multilingual, multi-task settings only if both stages are language-aware. Existing selection methods that work in English (likelihood, similarity-based MBR) fail or underperform cross-lingually, while Best-of-N with a specialized reward model is a strong but not optimal baseline. The paper introduces hedged sampling (mixing high-temperature stochastic samples with the deterministic greedy output), CHOPS (one-pass checklist-based judge selection), and X-MBR (cross-lingual evidence for MBR). On test sets spanning open-ended generation (m-ArenaHard-v2.0), math reasoning (MGSM) and machine translation (WMT24++), across seven languages and two 8B models, the best combinations beat single-sample greedy decoding on every task and language, and also improve the outputs of the much larger Command-A model when it judges its own samples.

Load-bearing premise

The reported gains depend on the judges used for selection (Command A) and for evaluation (GPT-4o) agreeing with human preferences in every language; the paper acknowledges this reliance but does not measure whether judge preferences are biased toward certain languages, styles, or output lengths.

Editorial extensions

If this is right

  • Five well-chosen samples can replace a much larger and more expensive model for many multilingual generation tasks.
  • Existing English-tuned reward models are not the ceiling: generalist multilingual LLM judges can outperform a specialized RM for Best-of-N in open-ended multilingual generation.
  • The same recipe transfers to a 111B model selecting its own outputs, so the gains do not depend on a size gap between generator and judge.
  • The recipe is cheap: CHOPS costs $O(1)$ judge calls, and X-MBR requires only $N+M=8$ total samples and $O(N(N+M))$ pairwise comparisons.
  • Scaling curves are steep at $N=5$; beyond $N=10$ gains become less predictable, so small budgets are a sensible operating point.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The cross-lingual evidence idea suggests a testable extension: for languages with no strong auxiliary language, evidence drawn from several languages or from machine-translated prompts may give a similar boost.
  • The checklist mechanism in CHOPS may be doing the real calibration work: asking the judge to operationalize quality criteria before reading candidates could reduce the length or style biases that plague LLM judges; this could be tested by comparing CHOPS against judges primed with fixed rubrics.
  • Because the gains appear in open-ended generation and translation, synthetic-data pipelines that produce multilingual training data could adopt the same selection step to filter higher-quality samples at negligible extra cost.
  • The paper only tests high-resource languages; if the pattern holds, the risk-hedging motivation is even stronger for lower-resource languages, where sample variance is larger.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper studies parallel inference-time scaling (sampling N=5 outputs and selecting one) for multilingual, multi-task generation. The authors evaluate Aya-Expanse-8B, Qwen3-8B, and Command-A (111B) across seven languages on m-ArenaHard open-ended tasks, MGSM math reasoning, and WMT24++ translation. They introduce hedged sampling (mixing greedy with stochastic samples), CHOPS (one-pass checklist-based judge selection), and X-MBR (MBR with cross-lingual evidence in English/Chinese), reporting consistent win-rate, accuracy, and XComet gains over greedy decoding, with headline improvements such as +17.3 win-rate points for Aya on m-ArenaHard and +9.0 for Command-A. The paper argues that English-centric inference-time scaling strategies do not transfer across languages and tasks.

Significance. The contribution is practically relevant and empirically broad: the authors compare against strong reward-model BoN baselines, use separate dev, devtest, and test splits, and report results on open-ended generation, math, and MT across seven languages. If the judge-based gains are real, the proposed methods are low-cost and useful for production multilingual systems. However, the central quantitative claims rest entirely on LLM-judge win rates, and the paper explicitly acknowledges in its Limitations that judge alignment is not addressed; the absence of any cross-lingual validation of the judges is a significant gap that must be closed before the headline deltas can be taken at face value.

major comments (3)
  1. [Limitations; §3.3.2; Eq. (8)] The manuscript's central win-rate claims are evaluated with GPT-4o (m-ArenaHard) while CHOPS and X-MBR select using Command A. The Limitations section ('Reliance on judge alignment') concedes that the methods 'do not directly address' the issue, and no per-language agreement analysis between either judge and human preferences is provided. This matters because X-MBR (Eq. 8) scores target-language candidates against English/Chinese pseudo-references, so selection can favor outputs that match the judge's stylistic or length preferences in those evidence languages rather than genuine quality. M-RewardBench (Table 5) measures preference accuracy on a different task, not agreement on open-ended win rates. Please add a cross-lingual judge-bias check: either per-language human agreement for GPT-4o and Command A on a sample of m-ArenaHard, or a judge-substitution experiment (e.g., re-running the final evaluation with a different judge model) to demonstrate that the reported deltas are not artifacts.
  2. [§3.3.2; Tables 3 and 4] All quantitative results, including the headline '+17.3', '+9.0', and '+6.8', are point estimates without confidence intervals or significance tests. In Table 3, many deltas are small (e.g., +0.20 XComet for Aya WMT), and Table 8 shows negative per-language X-MBR deltas on WMT; a paired bootstrap over the 250/498 prompts could establish which gains are stable. Without such intervals, the claim of 'consistent gains across languages' is not supported at the reported level of precision. Please add uncertainty estimates for the main tables.
  3. [§3.3.2 vs Appendix F, Table 8] The text in §3.3.2 states 'We find improvements over the greedy baseline in all tasks, languages, and methods,' but Appendix F Table 8 reports negative X-MBR deltas for most WMT language pairs (e.g., Aya: Chinese -1.05, Japanese -3.62, Russian -2.73; Qwen: Chinese -0.23, Japanese -0.60, Russian -1.26). Even the averaged test result in Table 3 is only +0.20 for Aya WMT, and X-MBR underperforms RM-BoN and CHOPS on translation in both tables. The blanket claim and the recipe's recommendation of X-MBR for 'language disparities' therefore need to be qualified, for example restricted to open-ended generation.
minor comments (4)
  1. [Abstract; §3.3.2, Table 4] Please clarify in the abstract that the +6.8 average against Gemini is the X-MBR result specifically; CHOPS gives +5.0 and BoN +6.4 in Table 4, so the current wording ('our combined sampling and selection methods') overstates the consistency across methods.
  2. [§1; §3.1] Section 1 contains a typo ('and and how well'), and §3.1 defines X-MBR without stating explicitly that the M=3 evidence samples are an additional inference budget beyond the N=5 candidates; please state this in the method description.
  3. [Figure 4a] The x-axis labels ('Single Temperature Hedged Single Hedged Random Random Temperature') are hard to parse; consider a cleaner legend that distinguishes single vs multi-temperature and hedged vs non-hedged.
  4. [Table 5] M-RewardBench is a preference-accuracy benchmark; the text should clarify that it does not directly measure agreement with human open-ended quality judgments, which is precisely the gap noted in the Limitations.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: headline gains are held-out empirical evaluations against external metrics; selection judges and hyperparameters are inputs, not derived predictions.

full rationale

The paper's derivation chain is empirical rather than formal, and I found no load-bearing step that reduces to its own inputs. The central claims (+17.3, +9.0, +6.8 win-rate points) are measured on held-out test splits (m-ArenaHard-v2.0, MGSM, WMT24++) with external evaluation metrics (GPT-4o win rate, exact match, XComet-XL). Sampling and selection hyperparameters (temperature 0.7, min-p 0.2, hedged greedy inclusion, and the X-MBR evidence-language rule) were chosen on dev splits and applied to the test set only once, which is standard practice and not a fitted-input-called-prediction loop. CHOPS and X-MBR use Command A as a selection judge, but the evaluation judge is GPT-4o; a selection model is an input to the pipeline, not a quantity derived from the evaluation. The self-improvement experiment uses Command A for both generation and selection, but the resulting win rates are judged by GPT-4o against greedy single-sample outputs, so the claim is not forced by construction. Some references are to the authors' own models and benchmarks (Aya Expanse, m-ArenaHard, M-RewardBench, Command A), but these support the experimental setup rather than supply an unverified theorem, uniqueness claim, or ansatz that the results depend on. The Limitations section explicitly acknowledges that judge-alignment is not addressed; that is a validity risk for the multilingual win-rate claims, not a circularity in the derivation. Because final results are evaluated against external or held-out metrics and no parameter is fitted to the test distribution, the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim depends on a small set of hand-chosen hyperparameters (temperature, min-p, evidence language, evidence size) and on the domain assumption that LLM judges faithfully evaluate cross-lingual quality. No new entities are introduced; the methods are algorithms rather than postulated objects.

free parameters (4)
  • sampling temperature tau = 0.7
    Chosen from dev-set exploration for highest best-case performance across tasks and models; used for hedged single-temperature sampling.
  • min-p threshold p = 0.2
    Adopted for token-level pruning in hedged sampling; value chosen from dev experiments (Figure 10).
  • cross-lingual evidence size M = 3
    Set to 3 additional samples for X-MBR; no tuning curve shown, chosen by hand.
  • X-MBR evidence language mapping = Chinese for English prompts, English otherwise
    Selected based on dev-set comparison (Table 2), which showed En/Zh evidence outperforming same-language or random evidence.
assumptions (4)
  • domain assumption GPT-4o is a reliable and unbiased judge of answer quality across the 7 test languages
    Win-rate metrics are computed by GPT-4o; if the judge is biased by language or style, reported gains may be artifacts.
  • domain assumption The LLM judges (Command A) select outputs in a way that transfers to the GPT-4o-judged win-rate metric
    Selection uses Command A pairwise/checklist comparisons; final evaluation uses GPT-4o; any misalignment between the two judges could weaken results.
  • domain assumption The 7 high-resource languages are representative of multilingual settings
    The paper acknowledges in Limitations that results may not transfer to underrepresented languages.
  • domain assumption Exact-match accuracy on MGSM and XComet-XL on WMT are appropriate metrics for the respective tasks
    Standard metrics, but translation quality metrics may have biases.

how reviews work

0 comments
Cite this review

Pith. "Pith review of When Life Gives You Samples: The Benefits of Scaling up Inference Compute for Multilingual LLMs." pith.science (2026). https://pith.science/paper/SZWQ72VG

@misc{pith2026250620544,
  author       = {Pith},
  title        = {Pith review of: When Life Gives You Samples: The Benefits of Scaling up Inference Compute for Multilingual LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SZWQ72VG}},
  note         = {Machine review of arXiv:2506.20544}
}
read the original abstract

Recent advancements in large language models (LLMs) have shifted focus toward scaling inference-time compute, improving performance without retraining the model. A common approach is to sample multiple outputs in parallel, and select one of these as the final output. However, work to date has focused on English and a handful of domains such as math and code. In contrast, we are most interested in techniques that generalize across open-ended tasks, formally verifiable tasks, and across languages. In this work, we study how to robustly scale inference-time compute for open-ended generative tasks in a multilingual, multi-task setting. Our findings show that both sampling strategy based on temperature variation and selection strategy must be adapted to account for diverse domains and varied language settings. We evaluate existing selection methods, revealing that strategies effective in English often fail to generalize across languages. We propose novel sampling and selection strategies specifically adapted for multilingual and multi-task inference scenarios, and show they yield notable gains across languages and tasks. In particular, our combined sampling and selection methods lead to an average +6.8 jump in win-rates for our 8B models on m-ArenaHard-v2.0 prompts, against proprietary models such as Gemini. At larger scale, Command-A (111B model) equipped with our methods, shows +9.0 improvement in win-rates on the same benchmark with just five samples against single-sample decoding, a substantial increase at minimal cost. Our results underscore the need for language- and task-aware approaches to inference-time compute, aiming to democratize performance improvements in underrepresented languages.

Figures

Figures reproduced from arXiv: 2506.20544 by the authors.

Figure 1
Figure 1. Overview of the multilingual multi-task experimental scope. New methods are marked with . Name Data Splits (# Prompts per Language) Metric dev devtest test Arena m-ArenaHard (250/250) m-ArenaHard-v2.0 (498) Win rate MGSM GSM8K-instruct-parallel (250/250) MGSM (258) Accuracy WMT WMT24/15 dev (997/1.5k) NTREX (1997) WMT24++ (960) XComet-XL [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Multilingual win-rates gains vs greedy output on the dev set of m-ArenaHard as we increase the sample size from 1 to 20. Performance improvements are steepest at low sample sizes (3-5) with more modest changes beyond that for both selection methods. Results shown for Aya-8B across French, Japanese, and Russian. to (1) greedy decoding for intrinsic comparison and (2) Gemini (gemini-2.0-flash) outputs for extrinsic co… view at source ↗
Figure 3
Figure 3. Quality under single temperature sampling For each temperature, we evaluate best, worst and mean quality for N = 5 samples from Aya for English, Japanese, French and Russian on each of the dev sets of the tasks (rows: Arena, MGSM, WMT). While best-case scores improve over greedy outputs (τ = 0), the effect varies across languages and tasks, with a notable drop in worst-case quality for Japanese and Russian Arena and… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Performance analysis of temperature sampling methods. Left: Sampling from a one vs multiple temperatures, with and without hedging: Win rates over greedy outputs on mArenaHard with Judge MBR and N = 5 samples, averaged across models. We choose τ = 0.7 as single tempera…
Figure 5
Figure 5. Figure 5: Comparison of baselines vs RM and LLM Judge on N = 5 generations in terms of the win-rate comparing to greedy outputs on mArenaHard. Averaged across models, and non-English languages. samples and performing N × (N + M) pairwise comparisons, approximating O(N2 ) complex…
Figure 6
Figure 6. Figure 6: Self-improvement with parallel scaling: Command A win rates against the greedy single sample baseline for each of the selection methods. Our methods, which also use Command A for selection, outperform the RM BoN in the majority of languages. report the delta in win-rat…
Figure 7
Figure 7. Figure 7: Arena: Evaluation score under different temperatures with N = 5 samples. (5.3%), whereas CHOPS gives a more balanced outcome of 6.8% and 7.1% win-rates over greedy in English and non-English languages respectively. We hypothesize that this shift comes form the fact tha…
Figure 8
Figure 8. Figure 8: MGSM: Evaluation score under different temperatures with N = 5 samples. 50 60 70 80 90 French Best Mean Worst Japanese Russian 0.0 0.3 0.70.80.91.0 50 60 70 80 90 0.0 0.3 0.70.80.91.0 0.0 0.3 0.70.80.91.0 Temperature Aya Expanse Qwen3 Comet [PITH_FULL_IMAGE:figures/fu…
Figure 9
Figure 9. Figure 9: WMT Evaluation score under different temperatures with N = 5 samples. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_9.png]
Figure 10
Figure 10. Figure 10: Effect of adding min-p sampling across different selection methods. Min-p provides consistent improvements [PITH_FULL_IMAGE:figures/full_fig_p031_10.png]
Figure 11
Figure 11. Figure 11: Comparison of CHOPS (with checklists) versus OPS (without checklists): (Left) The self-generated [PITH_FULL_IMAGE:figures/full_fig_p031_11.png]
Figure 12
Figure 12. Figure 12: BoN Reward score as we increase the sample size from from 1 sample to 40 samples [PITH_FULL_IMAGE:figures/full_fig_p031_12.png]
Figure 13
Figure 13. Figure 13: Scaling pool sample size using τ = 0.7 hedged sampling for selected languages with different selection methods on mArenaHard dev set. 32 [PITH_FULL_IMAGE:figures/full_fig_p032_13.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CodeChemist: Test-Time Scaling for Low-Resource Code Generation via Functional Knowledge Transfer

    cs.SE 2025-10 conditional novelty 6.0 of 10

    Using Python-generated test oracles and multi-temperature sampling, CodeChemist selects low-resource-language code by execution pass rate and improves Pass@1 up to 69.5% over vanilla sampling.

Reference graph

Works this paper leans on

45 extracted references · 12 canonical work pages · cited by 1 Pith paper

  1. [1]

    The multilingual alignment prism: Aligning global and local preferences to reduce harm

    Arash Ahmadian Aakanksha, Beyza Ermis, Seraphina Goldfarb-Tarrant, Julia Kreutzer, Marzieh Fadaee, Sara Hooker, et al. The multilingual alignment prism: Aligning global and local preferences to reduce harm. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 12027–12049,

    work page 2024

  2. [6]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    arXiv

  3. [8]

    Ticking all the boxes: Generated checklists improve llm evaluation and generation.arXiv preprint arXiv:2410.03608,

    Jonathan Cook, Tim Rocktäschel, Jakob Foerster, Dennis Aumiller, and Alex Wang. Ticking all the boxes: Generated checklists improve llm evaluation and generation.arXiv preprint arXiv:2410.03608,

    arXiv

  4. [9]

    RLHF can speak many languages: Unlocking multilingual preference optimization for LLMs

    John Dang, Arash Ahmadian, Kelly Marchisio, Julia Kreutzer, Ahmet Üstün, and Sara Hooker. RLHF can speak many languages: Unlocking multilingual preference optimization for LLMs. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.),Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 13134–13156, Miami, Florida...

    arXiv 2024

  5. [10]

    Daniel Deutsch, Eleftheria Briakou, Isaac Caswell, Mara Finkelstein, Rebecca Galor, Juraj Juraska, Geza Kovacs, Alison Lui, Ricardo Rei, Jason Riesa, et al

    URL https://arxiv.org/abs/2505.08751. Daniel Deutsch, Eleftheria Briakou, Isaac Caswell, Mara Finkelstein, Rebecca Galor, Juraj Juraska, Geza Kovacs, Alison Lui, Ricardo Rei, Jason Riesa, et al. Wmt24++: Expanding the language coverage of wmt24 to 55 languages & dialects.arXiv preprint arXiv:2502.12404,

    arXiv

  6. [11]

    Optimizing temperature for language models with multi-sample inference.arXiv preprint arXiv:2502.05234,

    18 Weihua Du, Yiming Yang, and Sean Welleck. Optimizing temperature for language models with multi-sample inference.arXiv preprint arXiv:2502.05234,

    arXiv

  7. [14]

    doi:10.18653/v1/2022.sumeval-1.4

    Association for Computational Linguistics. doi:10.18653/v1/2022.sumeval-1.4. URLhttps://aclanthology.org/2022.sumeval-1.4/. Patrick Fernandes, António Farinhas, Ricardo Rei, José G. C. de Souza, Perez Ogayo, Graham Neubig, and Andre Martins. Quality-aware decoding for neural machine translation. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vla...

    doi:10.18653/v1/2022.sumeval-1.4 2022

  8. [15]

    doi:10.18653/v1/2022.naacl-main.100

    Association for Computational Linguistics. doi:10.18653/v1/2022.naacl-main.100. URL https://aclanthology .org/2022.naacl-main.100/. Markus Freitag, Behrooz Ghorbani, and Patrick Fernandes. Epsilon sampling rocks: Inves- tigating sampling strategies for minimum Bayes risk decoding for machine translation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.)...

    work page doi:10.18653/v1/2022.naacl-main.100 2022

Show all 45 references
  1. [16]

    doi:10.18653/v1/2023.findings-emnlp.617

    Associa- tion for Computational Linguistics. doi:10.18653/v1/2023.findings-emnlp.617. URL https://aclanthology.org/2023.findings-emnlp.617/. Ashim Gupta and Vivek Srikumar. Test-time scaling with repeated sampling improves multilingual text generation,

    2023 doi

  2. [17]

    Srishti Gureja, Lester James V Miranda, Shayekh Bin Islam, Rishabh Maheshwary, Drishti Sharma, Gusti Winata, Nathan Lambert, Sebastian Ruder, Sara Hooker, and Marzieh Fadaee

    URLhttps://arxiv.org/abs/2505.21941. Srishti Gureja, Lester James V Miranda, Shayekh Bin Islam, Rishabh Maheshwary, Drishti Sharma, Gusti Winata, Nathan Lambert, Sebastian Ruder, Sara Hooker, and Marzieh Fadaee. M- rewardbench: Evaluating reward models in multilingual settings...

    arXiv

  3. [18]

    Truncation sampling as language model desmoothing

    John Hewitt, Christopher Manning, and Percy Liang. Truncation sampling as language model desmoothing. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.),Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 3414–3427, Abu Dhabi, United Arab Emirates...

    2022

  4. [19]

    doi:10.18653/v1/2022 .findings-emnlp.249

    Association for Computational Linguistics. doi:10.18653/v1/2022 .findings-emnlp.249. URLhttps://aclanthology.org/2022.findings-emnlp.249/. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. InInternational Conference on...

    2022 doi

  5. [20]

    Audrey Huang, Adam Block, Qinghua Liu, Nan Jiang, Dylan J Foster, and Akshay Krishnamurthy

    URL https://arxiv.org/abs/2407.05694. Audrey Huang, Adam Block, Qinghua Liu, Nan Jiang, Dylan J Foster, and Akshay Krishnamurthy. Is best-of-n the best of them? coverage, scaling, and optimality in inference-time alignment.arXiv preprint arXiv:2503.21878,

    arXiv

  6. [22]

    doi:10.18653/v1/2024.emnlp-main.248

    Association for Computational Linguistics. doi:10.18653/v1/2024.emnlp-main.248. URL https://aclanthology .org/2024.emnlp-main.248/. Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, ...

    2024 doi

  7. [23]

    doi:10.18653/v1/2024.wmt-1.109

    Association for Computational Linguistics. doi:10.18653/v1/2024.wmt-1.109. URL https://aclanthology.org/2024.wmt-1.109/. Julia Kreutzer, Eleftheria Briakou, Sweta Agrawal, Marzieh Fadaee, and Kocmi Tom. D\’ej\a vu: Multilingual llm evaluation through the lens of machine transl...

    2024 arXiv

  8. [25]

    From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline.arXiv preprint arXiv:2406.11939,

    Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline.arXiv preprint arXiv:2406.11939,

    arXiv

  9. [26]

    Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou

    URLhttps://openreview.net/f orum?id=v8L0pN6EOi. Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-reward: Bag of tricks for reward modeling in llms.arXiv preprint arXiv:2410.18451,

    arXiv

  10. [27]

    Ayomide Odumakinde, Daniel D’souza, Pat Verga, Beyza Ermis, and Sara Hooker

    URLhttps: //openreview.net/forum?id=FBkpCyujtS. Ayomide Odumakinde, Daniel D’souza, Pat Verga, Beyza Ermis, and Sara Hooker. Multilingual arbi- trage: Optimizing data pools to accelerate multilingual progress.arXiv preprint arXiv:2408.14960,

    arXiv

  11. [28]

    José Pombal, Dongkeun Yoon, Patrick Fernandes, Ian Wu, Seungone Kim, Ricardo Rei, Graham Neubig, and André FT Martins

    URLhttps://kar.kent.ac.uk/105743/. José Pombal, Dongkeun Yoon, Patrick Fernandes, Ian Wu, Seungone Kim, Ricardo Rei, Graham Neubig, and André FT Martins. M-prometheus: A suite of open multilingual llm judges.arXiv preprint arXiv:2504.04953,

  12. [29]

    The effect of sampling temperature on problem solving in large language models

    Matthew Renze. The effect of sampling temperature on problem solving in large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.),Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 7346–7356, Miami, Florida, USA, November

    2024

  13. [30]

    doi:10.18653/v1/2024.findings-emnlp.432

    Association for Computational Linguistics. doi:10.18653/v1/2024.findings-emnlp.432. URL https://aclanthology.org/2024.findings-emnlp.432/. 21 Chufan Shi, Haoran Yang, Deng Cai, Zhisong Zhang, Yifan Wang, Yujiu Yang, and Wai Lam. A thorough examination of decoding methods in th...

    2024 doi

  14. [31]

    doi:10.18653/v1/2024.emnlp-main.489

    Association for Computational Linguistics. doi:10.18653/v1/2024.emnlp-main.489. URL https://aclantholo gy.org/2024.emnlp-main.489/. Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et a...

    2024 arXiv

  15. [32]

    Guijin Son, Jiwoo Hong, Hyunwoo Ko, and James Thorne

    URLhttps://openreview.net/forum?i d=4FWAwZtd2n. Guijin Son, Jiwoo Hong, Hyunwoo Ko, and James Thorne. Linguistic generalizability of test-time scaling in mathematical reasoning.arXiv preprint arXiv:2502.17407,

    arXiv

  16. [33]

    The good, the bad, and the greedy: Evaluation of LLMs should not ignore non-determinism

    Yifan Song, Guoyin Wang, Sujian Li, and Bill Yuchen Lin. The good, the bad, and the greedy: Evaluation of LLMs should not ignore non-determinism. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.),Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the ...

    2025

  17. [34]

    ISBN 979-8-89176-189-6

    Association for Computational Linguistics. ISBN 979-8-89176-189-6. doi:10.18653/v1/2025.naacl-long.211. URL https: //aclanthology.org/2025.naacl-long.211/. Miloš Stanojević, Amir Kamran, Philipp Koehn, and Ondřej Bojar. Results of the wmt15 metrics shared task. InProceedings o...

    2025 doi

  18. [36]

    Nandan Thakur, Jianmo Ni, Gustavo Hernandez Abrego, John Wieting, Jimmy Lin, and Daniel Cer

    URL https://openreview.net/forum?id=348hfcprUs. Nandan Thakur, Jianmo Ni, Gustavo Hernandez Abrego, John Wieting, Jimmy Lin, and Daniel Cer. Leveraging LLMs for synthesizing training data across many languages in multilingual dense retrieval. In Kevin Duh, Helena Gomez, and St...

    2024

  19. [37]

    doi:10.18653/v1/2024.naacl-long.426

    Association for Computational Linguistics. doi:10.18653/v1/2024.naacl-long.426. URL https://aclanthology.org/2024.naacl-long.426/. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, ...

    2024 arXiv

  20. [38]

    Aya model: An instruction finetuned open-access multilingual language model.arXiv preprint arXiv:2402.07827,

    Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, et al. Aya model: An instruction finetuned open-access multilingual language model.arXiv preprint arXiv:2402.07827,

    arXiv

  21. [39]

    Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev

    URL https://openreview.net/forum?id=1PL1NIMMrw. Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training top-performing reward models.arXiv prepr...

    arXiv

  22. [40]

    Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang

    URLhttps://openreview.net/forum?id=7xCS K9BLPy. Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Scaling inference computa- tion: Compute-optimal inference for problem-solving with language models. InThe 4th Workshop on Mathematical Reasoning and AI at NeurI...

    2024 arXiv

  23. [41]

    doi:10.18653/v1/2024.acl-long.405

    Association for Computational Linguistics. doi:10.18653/v1/2024.acl-long.405. URL https://aclanthology.org/2024.acl-long.405/. Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token pre...

    2024 arXiv

  24. [42]

    non-thinking

    Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl.498. URLhttps://aclanthology.org/2024.findings-acl.498/. 24 A Detailed Experimental Setup Models and Language CoverageFor the multilingual generative model, we consider two 8B models from different f...

    2024 doi

  25. [43]

    For the larger models (Command A and Gemini 2.0 Flash), we use their dedicated hosted APIs

    Model Serving:We use vLLM [Kwon et al., 2023] to generate outputs from our 8B models (Aya andQwen), loading them with FP8 quantization and a maximum sequence length of 8,192 tokens. For the larger models (Command A and Gemini 2.0 Flash), we use their dedicated hosted APIs. For...

    2023

  26. [44]

    where its agreement with pairwise human preferences from Chatbot Arena battles in multiple languages is close to GPT-4o’s, with particular strengths in Chinese, Vietnamese, French, Turkish and Dutch. On the English RewardBench benchmark [Lambert et al., 2024], classifier RMs a...

    2024

  27. [45]

    "" Listing 1: Prompt Used for Checklisted One Pass Selection (CHOPS) 30

    and Stroebl et al. [2024]. The results in Figure 13 illustrates this issue: Win rate improvements over greedy are not developing smoothly across languages, sometimes even dropping below zero, so that it is not always the case that sampling more will result in a larger improvem...

    2024

  28. [1985]

    doi:https://doi.org/10.1016/S0 364-0213(85)80012-4

    ISSN 0364-0213. doi:https://doi.org/10.1016/S0 364-0213(85)80012-4. URL https://www.sciencedirect.com/science/article/pii/S0364021 385800124. Arash Ahmadian, Seraphina Goldfarb-Tarrant, Beyza Ermis, Marzieh Fadaee, Sara Hooker, et al. Mix data or merge models? optimizing for d...

    arXiv

  29. [2002]

    doi: 10.3115/1118693.1118712

    Association for Computational Linguistics. doi: 10.3115/1118693.1118712. URLhttps://doi.org/10.3115/1118693.1118712. 20 Shankar Kumar and William Byrne. Minimum Bayes-risk decoding for statistical machine translation. InProceedings of the Human Language Technology Conference o...

  30. [2015]

    Inference scaling{F} laws: The limits of llm resampling with imperfect verifiers.arXiv preprint arXiv:2411.17501,

    Benedikt Stroebl, Sayash Kapoor, and Arvind Narayanan. Inference scaling{F} laws: The limits of llm resampling with imperfect verifiers.arXiv preprint arXiv:2411.17501,

  31. [2019]

    doi:10.18653/v1/P19-1365

    Association for Computational Linguistics. doi:10.18653/v1/P19-1365. URLhttps://aclanthology.org/P19-1365/. Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An open so...

    doi

  32. [2020]

    doi: 10.18653/v1/2020.coling-main.398

    International Committee on Computational Linguistics. doi: 10.18653/v1/2020.coling-main.398. URLhttps://aclanthology.org/2020.coling-main.398/. Bryan Eikema and Wilker Aziz. Sampling-based approximations to minimum Bayes risk decoding for neural machine translation. In Yoav Go...

    2020 doi

  33. [2021]

    Command a: An enterprise-ready large language model.arXiv preprint arXiv:2504.00698,

    Team Cohere, Arash Ahmadian, Marwan Ahmed, Jay Alammar, Yazeed Alnumay, Sophia Althammer, Arkady Arkhangorodsky, Viraat Aryabumi, Dennis Aumiller, Raphaël Avalos, et al. Command a: An enterprise-ready large language model.arXiv preprint arXiv:2504.00698,

    arXiv

  34. [2022]

    doi:10.18653/v1/2022.emnlp-main.754

    Association for Computational Linguistics. doi:10.18653/v1/2022.emnlp-main.754. URL https://aclanthology.org/2022.em nlp-main.754/. Christian Federmann, Tom Kocmi, and Ying Xin. NTREX-128 – news test references for MT evaluation of 128 languages. In Kabir Ahuja, Antonios Anast...

    2022 doi

  35. [2023]

    doi:10.18653/v1/2023.bigpicture-1.9

    Association for Computational Linguistics. doi:10.18653/v1/2023.bigpicture-1.9. URL https://aclanthology.org/2023.bigpicture-1.9/. Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling infe...

    2023 arXiv

  36. [2024]

    Do we truly need so many samples? multi-llm repeated sampling efficiently scale test-time compute.arXiv preprint arXiv:2504.00762,

    Jianhao Chen, Zishuo Xun, Bocheng Zhou, Han Qi, Qiaosheng Zhang, Yang Chen, Wei Hu, Yuzhong Qu, Wanli Ouyang, and Shuyue Hu. Do we truly need so many samples? multi-llm repeated sampling efficiently scale test-time compute.arXiv preprint arXiv:2504.00762,

  37. [2025]

    Breaking language barriers in multilingual mathematical reasoning: Insights and observations.arXiv preprint arXiv:2310.20246,

    Nuo Chen, Zinan Zheng, Ning Wu, Ming Gong, Dongmei Zhang, and Jia Li. Breaking language barriers in multilingual mathematical reasoning: Insights and observations.arXiv preprint arXiv:2310.20246,

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.