{"id":"e37638ae-9222-4380-a974-1b614330e911","arxiv_id":"2412.18299","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Averaging next-token probabilities across paraphrased prompts in a single batched pass gives small but consistent gains on translation, code, and simplification benchmarks.","lead":"Large language models usually answer a question based on one prompt. This paper tests a simple trick: ask the model several slightly reworded versions of the same prompt in one batch, average the per-token probabilities, and let the averaged distribution choose each next word. The authors report small gains on translation, code generation, and text simplification, suggesting a cheap way to stabilize outputs when the model's probabilities are accessible.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Per-seed data in Appendix B show MT and code gains are within seed noise; without significance tests the claimed consistent improvement over any single prompt is not established.","rationale":"The reader's CONDITIONAL verdict is appropriate, and I would not change it. I partially agree with the reader: the malformed code-generation p2 prompt in Appendix A.2 is a real problem and invalidates the code experiments as evidence for the prompt-equivalence mechanism. However, the most load-bearing concern for the broad central claim is statistical. The reported tables are averages over 10 seeds with no intervals or tests, and the seed-level data that are included let us see that many of the headline gains are within seed noise. Concretely, the two MT directions highlighted in the abstract (en→zh and en→ja) have paired t-statistics of about 1.3 and 0.5, respectively, when comparing Ours against the better single prompt. Code pass@10 is marginal at best, with several negative per-seed differences. Text simplification is the one task where the improvement is consistently positive. A second, smaller inconsistency is that Algorithm 1 appears to average logits rather than the probability distributions required by Equation 3; this needs clarification, but it is secondary to the statistical gap because the published seed data already allow a decisive test of the headline claim. The proposed paired significance test on the Appendix B tables would settle whether the central claim is supported.","tokens_in":15885,"tokens_out":17184,"duration_ms":145615,"concrete_test":"Using the per-seed values in Appendix Tables 15, 16, 18, and 19, compute for each seed and condition d = score(Ours) − max(score(p1), score(p2)) and run a one-sided paired t-test and Wilcoxon signed-rank test per condition, with Benjamini–Hochberg correction across the 8 translation directions, 3 pass@k values, and 2 LENS conditions. If the MT directions and code pass@k comparisons do not survive correction at p<0.05, the reported claim of consistent improvement over any single prompt is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is empirical: averaged token distributions from n prompt variants beat every single prompt. The evidence is 10-seed averages with no significance tests, and the seed-level tables in Appendix B let us check this directly. For en→zh d-BLEU, the per-seed differences Ours−p1 are -3.67, -0.20, +6.48, +3.22, +0.89, +1.41, +7.04, -3.36, +3.34, -0.01, giving a paired t≈1.3 (p≈0.22). For en→ja, differences versus the better of p1/p2 are -5.02, +2.65, +4.87, +0.63, +0.52, +1.00, +0.44, +1.09, -4.13, +3.10 (t≈0.5). Code pass@10 differences versus the better single prompt are negative on 3 of 10 seeds and t≈2.1; pass@1 has large negative outliers. Only text simplification is consistently positive (t≈4.8). With 8 translation directions and three k values, some positive averages are expected by chance. Because n=2 was selected on the same test sets (Tables 7–8), the reported configuration is also optimistically chosen. These issues, not the prompt-equivalence assumption alone, are what block the central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes M-Ped, a multi-prompt ensemble decoding method for LLMs. For a single input X, the method constructs n semantically equivalent prompt variants, runs them as one batched inference pass through the same model, and at each decoding step averages the n token-level probability distributions before sampling or selecting the next token. To make the batch feasible, the paper recommends left padding so that all prompts in the batch share the same input length. The method is evaluated on document-level machine translation with Llama-3.1-8B-Instruct (eight IWSLT 2017 directions), code generation with CodeLlama-7B-Python (HumanEval pass@k), and text simplification with Llama-3.1-8B-Instruct (SimpEval_2022 LENS). The paper reports consistent improvements over either single prompt in all three tasks, plus additional experiments on decoding strategies, model sizes, prompt count n, multilingual prompts, and MBR combination.","tokens_in":16128,"tokens_out":5680,"duration_ms":53424,"significance":"If the claimed gains are real, this is an attractively simple and low-cost way to improve generation quality: it requires no extra models or training, only a set of paraphrase prompts and a batched forward pass. The paper is commendable for reporting per-seed results in Appendix B and for addressing the practical padding issue. However, the central claim is empirical, and the current evidence does not establish it: the per-seed variability is large, no significance testing or multiple-comparison correction is reported, the prompt count n=2 is selected on the same test sets used for the headline numbers, and the code-generation prompt p2 shown in Appendix A.2 is syntactically malformed. The core contribution is plausible and potentially useful, but the manuscript needs substantial revision before the claimed improvements can be accepted.","major_comments":[{"comment":"The seed-level results do not support the claim that M-Ped improves over every single prompt. For example, en→ja p1 values range from 4.69 to 14.02 d-BLEU, and the per-seed differences between Ours and the better of p1/p2 are negative on several seeds; a paired test on the reported 10 seeds gives an approximately non-significant result. Similar patterns appear in the code-generation data in Table 16. Since the paper makes point comparisons ('approximately increase by 1.5 points' in Section 3.1) without confidence intervals or significance tests, the headline improvements are not established.","section":"Appendix B, Table 15; Section 3.1"},{"comment":"Algorithm 1 does not implement the ensemble average described by Eq. (3). As written, the algorithm splits pre_logits into n parts, repeats each part n times along the batch dimension, and divides by n; for n=2 this returns the original rows unchanged rather than averaging the n prompt rows. The correct operation would be to average over the prompt dimension, e.g., pre_logits.view(n, part_size, vocab_size).mean(dim=0). This discrepancy makes the reported implementation non-reproducible and must be fixed.","section":"2.1, Algorithm 1"},{"comment":"The choice n=2 is tuned on the same test sets that produce the headline results in Tables 2 and 3. Section 4.3 justifies the standard configuration by comparing n=2, 3, 4 on HumanEval and SimpEval_2022, then uses this configuration for the main claims. Because no held-out validation or selection correction is reported, the reported gains are optimistically biased and the '2-3 prompts are sufficient' conclusion is not a validated finding.","section":"4.3, Tables 7 and 8"},{"comment":"The paper's second code-generation prompt is syntactically malformed. Inserting the triple-quoted string '\"\"\" This is a good code.' before a function that already contains triple-quoted docstrings creates nested triple quotes and a syntax error, and it does not preserve the semantic content of p1 as claimed in Section 3.2. The code-generation results in Tables 2 and 6 therefore rest on a defective prompt construction, and the claim that p2 'does not alter the meaning of the original prompt' is contradicted by the appendix.","section":"Appendix A.2, Tables 12 and 13"},{"comment":"There is a numerical inconsistency in the main machine-translation result: Table 1 reports Ours en→zh d-BLEU as 18.88, but the corresponding average in Table 15 is 18.38. Since Tables 1 and 15 are supposed to report the same experiment, this discrepancy undermines confidence in the reported values and must be reconciled.","section":"Table 1 vs. Table 15"}],"minor_comments":[{"comment":"The sums are written as i=0 to n, which gives n+1 terms; they should be i=1 to n (or the indexing should be otherwise corrected).","section":"Eqs. (2) and (3)"},{"comment":"The model description says 'CodeLlama-7B-Python-hf' but then immediately refers to 'CodeLlama-13B-Python-hf is a member of the Code Llama series'; this appears to be a typo and should be corrected.","section":"Section 3.2"},{"comment":"The asterisk on Ours under Top-k (67.85%) has no corresponding footnote explaining what it marks.","section":"Table 5"},{"comment":"The rows labeled 'A VG' should be 'AVG' or 'Average' for clarity.","section":"Appendix B tables"}],"recommendation":"major_revision","confidential_remarks":"The manuscript presents a simple and potentially useful recipe, but the evidence base is currently too weak for acceptance. The lack of significance tests, the n=2 selection on the test data, and the Algorithm 1/prompt inconsistencies are all fixable in revision; if the authors can supply corrected results and proper statistical support, the paper could be publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: M-Ped is a simple ensemble trick that averages token-level probabilities from several paraphrase prompts. The math is standard ensemble decoding with the model index swapped for a prompt index—Eq. 3 is Eq. 2 with names changed. The novelty is thin. But the paper is not useless; the seed-level tables in Appendix B are the most honest thing in it, and they tell a different story from the abstract.\n\nWhat the paper does well: it reports 10-seed results for all three tasks, which is more than most LLM papers do, and it runs ablations across decoding strategies, model sizes, number of prompts, and MBR combination. The left-padding approach for batched decoding is a reasonable engineering detail. Those are real contributions.\n\nThe soft spots are the load-bearing ones. The stress-test note is right. For en→zh and en→ja, the per-seed differences between M-Ped and the better single prompt are mostly within noise; my own paired t-test on the appendix values gives t≈1.3 for en→zh and t≈0.5 for en→ja. Code pass@10 is borderline (t≈2.1, p≈0.06) and pass@1 has two large negative outliers. Only text simplification is solidly positive (t≈4.8). So the central claim—that it improves over any single prompt—is not established for two of the three tasks. The n=2 configuration is chosen from Tables 7–8 on the same test sets used for the headline results, which adds an optimistic bias. The code p2 prompt (Appendix A.2) is syntactically broken: it opens a triple-quoted string before the function definition and never closes it, contradicting their claim that p2 doesn't change the meaning. A less obvious but important issue: the method multiplies compute by n even if latency is amortized; the paper glosses over this cost-benefit.\n\nThe paper is an empirical recipe with a simple idea, not a theoretical contribution. It deserves a serious referee because the idea is reasonable and the per-seed data are there to evaluate it. But the referee should demand paired significance tests, a corrected code prompt, a comparison against prior multi-prompt/sampling baselines, and an honest statement about the compute multiplier. As written, the abstract and conclusions overstate what the data show.","headline":"M-Ped renames standard ensemble decoding and the headline gains don't survive the paper's own per-seed data; the only robust effect is on text simplification.","tokens_in":16760,"tokens_out":8819,"would_cite":false,"duration_ms":60728,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that averaging the token-level probability distributions of several paraphrased prompts in a single batched decode improves generation quality over any one prompt, with gains of roughly 0.5–2 points on translation, code…","keywords":["multi-prompt ensemble decoding","inner-batch ensemble","left-padding","large language models","document-level machine translation","code generation","text simplification","pass@k"],"falsifier":"Re-analyze the per-seed results in Tables 15–17 with a paired test comparing the ensemble score against the better-performing single prompt for each language pair and task; if the ensemble does not win on a significant majority of the ten seeds, or the paired difference is not significant, the claimed advantage over the best single prompt is not established.","tokens_in":15633,"feed_emoji":"🧩","tokens_out":9610,"duration_ms":74620,"temperature":0.7,"pith_summary":"Multi-prompt ensemble decoding (M-Ped) proposes that instead of running one prompt through an LLM, you can batch several paraphrases of the same request, average the per-token probability distributions across the batch, and decode from that averaged distribution. The paper reports that this Inner-Batch Ensemble consistently beats the best individual prompt on document-level machine translation (d-BLEU), code generation (pass@k), and text simplification (LENS), with improvements of roughly 0.5 to 2 points depending on task and direction. Because all n prompts share a single batched forward pass, the method costs essentially the same inference time as a single prompt, with only extra memory for the additional batch slots and padding tokens. If the pattern holds, it offers a nearly free way to raise LLM output quality without retraining or deploying multiple models.","feed_headline":"One batched pass, n prompts: ensemble decoding boosts LLM output","feed_subtitle":"Averaging token probabilities across prompt paraphrases lifts d-BLEU, pass@k, and LENS scores with no extra inference cost.","key_machinery":"The load-bearing object is the Inner-Batch Ensemble formula (Equation 3), together with the batched decoding procedure in Algorithm 1 that implements it. At each decoding step, the n prompts' logit vectors are partitioned by batch position, each partition is repeated n times, and the repeated copies are averaged, producing a single combined distribution from which the next token is drawn. A Left-Padding strategy makes the batch feasible by aligning all prompt lengths to the longest input on the left side, so that the source tokens occupy the same relative positions and right-padding does not sever the input-output connection. The uniformity of the averaging means the method introduces no learned weights; the only design choice is the list of paraphrases.","core_discovery":"On its own terms, the paper's central claim is that the diversity needed for effective ensemble decoding can come from prompts rather than from model parameters. Classical ensemble decoding averages the predictive distributions of n different models on one prompt; M-Ped keeps a single model and averages the distributions produced by n semantically equivalent prompts, defining the ensemble probability of token $y_j$ as $P(y_j | y_{0:j-1}, X) = (1/n) \\sum_{i=1}^{n} P(y_j | y_{0:j-1}, X, P_i; \\theta)$. The paper presents this as a drop-in replacement: the same batched logits are sliced, repeated, and averaged inside the decoder. Its evidence is empirical and cross-task: d-BLEU gains on IWSLT 2017 in eight language directions, pass@k gains on HumanEval with two CodeLlama sizes, and LENS gains on SimpEval_2022, with ablation studies suggesting that two to three prompts capture most of the benefit.","pith_inferences":["Since the method's gain presumably comes from reducing the variance of the next-token distribution, the same batched-averaging trick should transfer to other cheap sources of input diversity, such as mixing system-prompt languages, varying temperature, or shuffling few-shot example order; the paper only varies instruction phrasing.","The reported per-seed tables show wide overlapping ranges between single-prompt and ensemble scores, so part of the headline improvement may reflect seed noise rather than a systematic ensemble effect; a paired significance test across the ten seeds would settle this.","The code-generation p2 prompt appends a triple-quoted string 'This is a good code.' inside the function body, which is syntactically part of the function definition and may change the problem's meaning; those results are therefore not a clean test of averaging equivalent paraphrases.","Left-padding as a mechanism is independent of the ensemble idea and could be reused in other batched decoding setups; the paper leaves open how it interacts with position-dependent attention biases such as RoPE or ALiBi."],"forward_implications":["On IWSLT 2017 document translation, M-Ped raises d-BLEU by about 1.5 points for en→zh and en→ja and by roughly 0.5 points in the other six tested directions relative to the best single prompt.","On HumanEval, M-Ped improves pass@1, pass@5, and pass@10 by about 1 percentage point with CodeLlama-7B-Python, and pass@10 by nearly 2 points with CodeLlama-13B-Python.","On SimpEval_2022, M-Ped improves LENS by about 1.5 points with reference and 0.5 points without reference over the best single prompt.","The gains appear with Top-p and Top-k decoding, shrink under beam search, and plateau once the prompt count reaches three, so two to three paraphrases are the recommended operating point.","Because all prompts are decoded in one batched forward pass, the wall-clock inference cost is comparable to a single-prompt decode, making the ensemble nearly compute-neutral."],"supporting_citations":[{"why":"Defines deep ensembles that average predictive distributions of independently trained models; this is the standard-ensemble template whose model index M-Ped replaces with a prompt index.","marker":"Lakshminarayanan et al., 2017"},{"why":"Supplies the general ensemble principle that combining several predictors can beat the best single predictor, which motivates seeking prompt-level diversity.","marker":"Zhou et al., 2002"},{"why":"Review of ensemble deep learning that the paper draws on to justify ensemble decoding as a quality-boosting technique.","marker":"Ganaie et al., 2022"},{"why":"Closest prior work: multi-prompt decoding that builds a diverse candidate set and selects via Minimum Bayes Risk; M-Ped contrasts its own inference-time averaging with this result-space selection.","marker":"Heineman et al., 2024"},{"why":"Boosted prompt ensembles that select few-shot prompts to maximize ensemble accuracy; the paper contrasts this prompt-selection approach with its fixed-paraphrase averaging.","marker":"Pitis et al., 2023"},{"why":"Defines BLEU, the metric family behind the d-BLEU scores used to measure machine-translation gains.","marker":"Papineni et al., 2002"},{"why":"Introduces pass@k and the HumanEval benchmark used for code-generation evaluation.","marker":"Chen et al., 2021"},{"why":"Introduces LENS, the learnable text-simplification metric used for the simplification experiments.","marker":"Maddela et al., 2023"}],"fun_headline_variants":["Prompt ensemble decoding: one model, multiple prompts, better outputs","Average token odds across prompt variants to boost LLM generation","Multi-prompt ensemble: cheap performance lift for LLMs","Prompt diversity in decoding: better results without extra models","Ensemble decoding by prompt averaging: simpler, stronger LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The n prompts must be semantically equivalent paraphrases of the same instruction; if any prompt diverges in meaning, the averaged token distribution is an average over different tasks and the ensemble has no coherent interpretation.","fun_headline_variants_meta":{"raw":{"variants":["Prompt ensemble decoding: one model, multiple prompts, better outputs","Average token odds across prompt variants to boost LLM generation","Multi-prompt ensemble: cheap performance lift for LLMs","Prompt diversity in decoding: better results without extra models","Ensemble decoding by prompt averaging: simpler, stronger LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000368,"raw_usage":{"total_tokens":1978,"prompt_tokens":954,"completion_tokens":1024,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":570,"completion_tokens_details":{"reasoning_tokens":942}},"tokens_in":570,"tokens_out":1024,"duration_ms":9829,"temperature":1.0,"reasoning_tokens":942,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:50:34.430380+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-analyze the per-seed results in Tables 15–17 with a paired test comparing the ensemble score against the better-performing single prompt for each language pair and task; if the ensemble does not win on a significant majority of the ten seeds, or the paired difference is not significant, the claimed advantage over the best single prompt is not established.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the general ensemble principle that combining several predictors can beat the best single predictor, which motivates seeking prompt-level diversity."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Closest prior work: multi-prompt decoding that builds a diverse candidate set and selects via Minimum Bayes Risk; M-Ped contrasts its own inference-time averaging with this result-space selection."}],"review_version":1}