Pith. sign in

REVIEW 4 major objections 5 minor 54 references

Reflection-Window Decoding: Text Generation with Selective Refinement

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proves that stepwise greedy decoding can deviate from the same-length globally optimal response even for an oracle LLM, and proposes reflection-window decoding to pause and rewrite a short window when next-token entropy is high.

desk verdict The theory survives without its strongest assumption; the experiments are the weak link. read the letter →

arxiv 2502.03678 v3 pith:YY7DRFOK submitted 2025-02-05 cs.CL cs.AIcs.LG

Zeyu Tang , Zhenhao Chen , Xiangchen Song , Loka Li , Yunlong Deng , Yifan Shen , Guangyi Chen , Peter Spirtes
show 1 more author
Kun Zhang
This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords autoregressivedecodingsuboptimalitytextgenerationreflectionwindowselectiverefinementconditionalentropybeamsearchlargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that greedy, one-token-at-a-time decoding is structurally unable to guarantee the best whole response of the same length even when the model's probabilities are perfect, and that this gap can be detected while decoding is still in progress. Its theorem gives a necessary condition: at the first prefix where the true best response overtakes the greedy one, the greedy model's next-token probability is forced below an explicit positive number $\epsilon_L < 1$, so a low-confidence step is a signal that earlier tokens may already be wrong. The paper then proposes a decoding algorithm that treats uncertainty as a pause signal: a sliding window of recent tokens is inspected, and when conditional entropy stays high across the window, generation stops and only that window is rewritten by a short beam search. The empirical results report that this selective refinement improves judged answer quality and reasoning-benchmark accuracy over greedy, beam search, and top-k/top-p baselines while regenerating only a small fraction of tokens. If the claim holds, decoding gains a cheap route toward better global optimality without abandoning autoregressive speed.

What carries the argument

The load-bearing object is the threshold $\epsilon_L$ of Theorem 3.6, defined as the ratio of the global optimum's joint prefix probability at length L to the greedy prefix's joint probability at length L-1. The theorem shows that if the global optimum overtakes the greedy prefix at L, and stays ahead, then the greedy model's conditional probability for any next token at L is strictly smaller than $\epsilon_L$, and $\epsilon_L < 1$. That inequality does the argument's work: it converts the abstract fact that the greedy path is suboptimal into a concrete, locally observable signature, namely suppressed next-token probabilities, which the paper operationalizes as high conditional entropy over a sliding window of size $d$. The complementary machinery is Algorithm 1's two-pointer reflection window: a fast pointer advances ordinary decoding, a slow pointer marks the last refined position, and a pause criterion triggers regeneration of only the $d$ tokens between them using a length-$d$ beam search.

What would settle it

Take a fixed model and prompt, compute the greedy sequence and the true global optimum of the same length, and evaluate both sequences' joint probabilities at every prefix length; if there exist lengths $L < M$ where the global optimum's prefix is strictly more probable at $L$ but strictly less probable at $M$, Assumption 3.5 fails and the proof of Theorem 3.6 no longer applies. A tiny synthetic vocabulary with three tokens can exhibit such a distribution directly.

Watch

Extended reading notes

Core claim

The paper's central claim is that purely autoregressive (greedy) decoding can miss the globally most probable same-length response even when the model is an oracle that perfectly reproduces every autoregressive conditional probability. Let the greedy sequence and the global optimum first become distinguishable at prefix length L, meaning that at L the global optimum's prefix becomes strictly more probable than the greedy prefix and, by Assumption 3.5, stays more probable at all longer prefixes. Then the greedy token at position L was chosen under a conditional probability strictly below an explicitly defined threshold $\epsilon_L$, which is itself strictly below 1; in fact the deviation must have started at some earlier step $K < L$. The paper reads this as a warning signal: noticeable uncertainty in the next-token distribution can indicate that the generated history has already gone suboptimal. To act on it, the paper introduces reflection-window decoding, in which generation proceeds with slow and fast pointers, and when the conditional entropy of the last $d$ tokens exceeds a threshold, the content between the pointers is regenerated with a short fixed-length beam search before generation continues.

Load-bearing premise

The load-bearing premise is Assumption 3.5: once the globally optimal prefix overtakes the greedy prefix at some length, that advantage never reverses at any longer length; this monotonicity is assumed rather than derived from any property of natural-language distributions, and the paper does not empirically test it.

Editorial extensions

If this is right

  • If Theorem 3.6 is right, any purely autoregressive decoder, even an oracle model, carries a hard limit on how often it can reach the same-length global optimum, so correction during decoding becomes a necessary component for that optimum rather than an optional extra.
  • The $\epsilon_L$ bound provides a theoretically grounded proxy for detecting a suboptimal generation history while it is being written: rising next-token uncertainty should trigger inspection or refinement.
  • Reflection-window decoding with window size $d=4$ and entropy threshold $\sigma=0.5$ regenerates only about 3.5% to 5.5% of tokens on the paper's reasoning benchmark, so most of the output still comes from cheap greedy-style decoding.
  • On judged multi-turn chat responses, the method beats greedy decoding in 66.67% of cases while beam search beats greedy in only 46.3%, and it stays comparable to or better than beam search across several model families.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the theorem's condition is necessary but not sufficient, so the entropy pause criterion will also fire at harmless branching points where the greedy token is in fact optimal; the paper's regeneration ratio does not separate useful rewrites from wasted ones.
  • Editorial inference: the two components are pluggable, so the same sliding window could host a learned trigger that predicts a global-optimum overtake directly, or a non-beam refinement model; the paper does not test these swaps.
  • Editorial inference: if Assumption 3.5 fails on natural-language distributions, the exact bound may be too pessimistic even though the windowing heuristic still helps; a robustness study under relaxed monotonicity would show whether the empirical gains depend on the theorem or on the heuristic alone.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper studies the suboptimality of purely autoregressive decoding for LLM text generation. The authors define the globally optimal length-T response as the sequence maximizing the ground-truth joint conditional probability f(X1:T | X≤0), and the stepwise-optimal response as the sequence obtained by taking token-by-token argmaxes of the model's conditionals g. Under an oracle-LLM assumption (Assumption 3.3, exact autoregressive factorization), a strict-preference/injectivity assumption (Assumption 3.4), and an irreversibility assumption (Assumption 3.5), they prove Theorem 3.6: if the stepwise-optimal response is not the globally optimal one, and L is the first prefix length at which the global optimum overtakes the stepwise prefix, then the deviation must have begun at some step K < L, and the stepwise model's conditional probability at step L is bounded above by a positive number epsilon_L strictly below 1. The paper interprets this as a signal that noticeable token-level uncertainty can flag deviation from global optimality. On the empirical side, the authors propose reflection-window decoding (Algorithm 1): greedy generation with a sliding window of size d; when the conditional entropies of the last d generated tokens all exceed a threshold sigma, a fixed-length beam search regenerates the window and generation resumes.

Significance. The theoretical result is clean and, as far as I can tell, correctly proved; a notable point is that the proof of Theorem 3.6 in Appendix A actually uses only the M = L inequality in Eq. (7), so the stated Assumption 3.5 is stronger than necessary. The contribution is nonetheless modest in scope: it is an elementary but useful formalization of the well-known fact that coordinate-wise (greedy) optimization is not globally optimal, giving a falsifiable necessary condition (a bounded-away-from-1 conditional probability at the first overtaking prefix) that links uncertainty to potential deviation. The method itself is simple, principled, and readily implementable, and the paper is commendably honest in stating that its empirical evaluations do not rely on the oracle assumption. The semi-synthetic sanity check in Section 4 is a reasonable way to demonstrate that the phenomenon occurs with actual LLMs, and the detailed per-subject tables in Appendix C are valuable for scrutiny.

major comments (4)
  1. [5.3 / Table 1] The headline MMLU result does not support the claimed improvement over greedy decoding: on Phi3-Medium the macro average is 78.15% for reflection-window decoding versus 78.14% for greedy, a difference of 0.01 percentage points, and the per-category row shows the method losing ground on two of the four categories (Social Sciences 83.65% vs 83.91%; Others 81.10% vs 81.59%) while gaining only on STEM. No error bars, number of seeds, or significance tests are reported anywhere in the paper, and the same single-run issue applies to Table 2, where the gains over top-k/top-p sampling (0.88 to 2.76 percentage points) are also unreplicated. As written, the claim in Section 5.3 that the method 'achieves the highest average accuracy' and the abstract's 'significant improvement' are not supported; the authors should report variance over seeds or bootstrap intervals, test the relevant differences, and adjust the claims accordingly.
  2. [5.3 / Figure 5 and Tables 6-7] The MT-Bench evidence for 'clearly outperforming' greedy and beam search is not established. The win rates (66.67% for reflection-window vs greedy, 46.3% for beam vs greedy) are computed as wins divided by wins plus losses with ties excluded, which can inflate the apparent margin, and no confidence intervals are given; with the 80 MT-Bench questions, a 66.67% rate carries a wide binomial interval. The comparison also lacks any control for response length, which is a known confound for LLM judges, and reflection-window decoding could plausibly change response length by inserting regenerated segments. The score-based results in the appendix tell a more modest story: on Llama3.1-8B at d=4 the method scores 7.89 versus greedy 7.88 and beam 7.63, and on Mistral-Nemo the best configuration (d=3) gives 7.93 versus 7.91 for beam search. The paper should report confidence intervals for the win rates, address length confounding explicitly, and reconcile the win-rate and score-based narratives.
  3. [5.4 / Table 3] The claimed efficiency advantage over beam search is not supported by the reported metrics. Table 3 reports a regeneration ratio of 3.5 to 5.5 percent of tokens and the number of regeneration calls, but no wall-clock time, FLOPs, or latency-equivalent measurement appears anywhere in the paper; since each regeneration call is itself a beam search of width 4 over a window of length d, the token fraction does not quantify the actual compute overhead. The claim in the introduction that the method is 'more efficient' than beam search requires a direct timing comparison under matched hardware, such as tokens per second or total forward passes.
  4. [5.1, Eq. (4); Tables 4-5] The contribution of the pausing criterion itself is not isolated. The ablations in Tables 4 and 5 vary the threshold sigma and the window size d, but they never compare the entropy-gated pause schedule against a non-adaptive control, such as regenerating the window on a fixed cadence (every d tokens) or at random positions while holding the number of regeneration calls fixed. Without such a control, the observed gains could be due solely to occasional windowed beam-search refinement regardless of where the pauses occur, so the theory-guided selectivity of the mechanism is unverified; the paper would also be strengthened by reporting whether the regenerated windows actually coincide with positions of high conditional entropy, which would test the link to Theorem 3.6.
minor comments (5)
  1. [Appendix A / Assumption 3.5] Assumption 3.5 is stronger than necessary for Theorem 3.6: the proof in Appendix A uses only the M = L inequality in Eq. (7), and the strict preferences in Eq. (10) follow from the minimality of L together with Assumption 3.4. Removing the assumption (or stating it as a remark) would strengthen the theorem.
  2. [Appendix B.3] There is a typo in Appendix B.3: 'which is calculate by the product' should read 'which is calculated as the product'.
  3. [5.3] In Section 5.3, 'generates tokens autogressively' should read 'generates tokens autoregressively'.
  4. [3 / Theorem 3.6 and Theorem A.1] The bound in Theorem 3.6 and the appendix version (Theorem A.1) are written in slightly different orders (epsilon_L < 1 versus 1 > epsilon_L > max g); the two statements should be brought into exact agreement.
  5. [4 / Figure 4] Figure 4 does not state the number of prompts or generation histories used at each starting position, nor is the beam width shown in the legend; please add the sample size and the proxy settings to the caption.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the theoretical bound is derived from stated assumptions, the pausing criterion is a heuristic rather than a fitted prediction, and the beam-search overlap is a baseline-fairness concern, not an equivalence.

full rationale

The derivation chain is self-contained. Definitions 3.1 and 3.2 fix the two optimization targets; Assumptions 3.3, 3.4, and 3.5 state the oracle factorization, strict preference, and monotonicity; Appendix A proves Theorem 3.6 by chaining inequalities among f and g values. No equation in the proof re-uses its conclusion as an input, and no fitted parameter is renamed as a prediction: the entropy threshold sigma and window size d are explicit hyperparameters with robustness ablations in Tables 4, 5, 6, 7, 8, 9, and 10. The only overlap that could look suspicious is that beam search is used both as the refinement subroutine (ReGenerate) and as a comparison baseline; this is a legitimate baseline-selection and attribution concern, but the superiority claim is not forced by construction, since the method replaces only short windows and still generates greedily outside pauses. The Assumption 3.5 discussion raised by the reader concerns robustness of a sufficient monotonicity condition, not circularity; the final epsilon_L bound uses the M = L inequality together with the strict L-1 preference obtained from minimality of L, so the theorem does not reduce to Assumption 3.5. Empirical weaknesses, such as the 0.01 percentage point MMLU gap, absent error bars, and length-uncontrolled MT-Bench judge comparisons, affect evidentiary strength rather than circularity of the derivation. The footnote citation to Spirtes et al. (1993) is a modeling-consistency remark, not load-bearing support for the central claim.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim rests on two tuned hyperparameters (sigma=0.5, d=4), two beam widths, and three explicit assumptions. Assumption 3.5 is the most fragile: it asserts an irreversibility that is not true for arbitrary distributions and is never checked. Assumptions 3.3 and 3.4 are idealizations. No new physical or probabilistic entities are introduced; the reflection window, slow/fast pointers, and pause criterion are algorithmic constructs, not invented entities.

free parameters (4)
  • sigma (entropy threshold) = 0.5 (default)
    Pause trigger threshold in Eq. 4; robustness tested over 0.1 to 1.0 in Table 4, with 0.25 best on one MMLU subset.
  • d (reflection window size) = 4 (default)
    Number of tokens in the sliding reflection window; tested over 2 to 6, with d=6 best on one subset and d=3 or d=5 best on MT-Bench.
  • beam width for ReGenerate = 4
    Fixed-length beam search used for refinement, chosen to match the vanilla beam search baseline width for fairness.
  • beam width for sanity-check proxy = 10
    Beam width used in Section 4 as an approximate stand-in for the globally optimal response, acknowledged as an upper-bound proxy.
assumptions (3)
  • domain assumption Assumption 3.3 (Oracle LLM): the chain-rule product of next-token probabilities exactly equals the ground-truth joint probability for any response length.
    Idealizes a perfectly trained LLM; the paper notes the empirical parts do not rely on this assumption.
  • domain assumption Assumption 3.4 (Strict Preference Among Same-Length Sequences): ground-truth probabilities of distinct length-T responses are never equal.
    A generic-position, no-ties condition required for the strict inequalities in the proof of Theorem 3.6.
  • ad hoc to paper Assumption 3.5 (Irreversible Advantage Once Manifested): if the globally optimal prefix beats the stepwise-optimal prefix at length L, it stays ahead for all longer prefixes.
    This monotonicity is not a property of general joint distributions, is not validated empirically, and is load-bearing for the chain of inequalities in Appendix A (Eq. 7).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reflection-Window Decoding: Text Generation with Selective Refinement." pith.science (2026). https://pith.science/paper/YY7DRFOK

@misc{pith2026250203678,
  author       = {Pith},
  title        = {Pith review of: Reflection-Window Decoding: Text Generation with Selective Refinement},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YY7DRFOK}},
  note         = {Machine review of arXiv:2502.03678}
}
read the original abstract

The autoregressive decoding for text generation in large language models (LLMs), while widely used, is inherently suboptimal due to the lack of a built-in mechanism to perform refinement and/or correction of the generated content. In this paper, we consider optimality in terms of the joint probability over the generated response, when jointly considering all tokens at the same time. We theoretically characterize the potential deviation of the autoregressively generated response from its globally optimal counterpart that is of the same length. Our analysis suggests that we need to be cautious when noticeable uncertainty arises during text generation, which may signal the sub-optimality of the generation history. To address the pitfall of autoregressive decoding for text generation, we propose an approach that incorporates a sliding reflection window and a pausing criterion, such that refinement and generation can be carried out interchangeably as the decoding proceeds. Our selective refinement framework strikes a balance between efficiency and optimality, and our extensive experimental results demonstrate the effectiveness of our approach.

Figures

Figures reproduced from arXiv: 2502.03678 by the authors.

Figure 1
Figure 1. Illustrative diagrams of different dependence [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of our approach to address the inherent shortcoming of autoregressive decoding for text generation, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. A concrete example demonstrating our reflection-window decoding. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: The probability that greedy decoding can attain globally optimal response, with respect to the number of newly [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Comparison of win rates between beam search [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Regeneration ratio on MMLU college-level subjects with Llama3.1-8B [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Test Sample #28 18 [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Test Sample #29 19 [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Test Sample #7 20 [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 22 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    J., Javaheripi, M., Kauffmann, P., et al

    Abdin, M., Aneja, J., Behl, H., Bubeck, S., Eldan, R., Gunasekar, S., Harrison, M., Hewett, R. J., Javaheripi, M., Kauffmann, P., et al. Phi-4 technical report. arXiv preprint arXiv:2412.08905, 2024 a

    arXiv 2024

  3. [3]

    A., Awan, A

    Abdin, M., Jacobs, S. A., Awan, A. A., Aneja, J., Awadallah, A., Awadalla, H., Bach, N., Bahree, A., Bakhtiari, A., Behl, H., et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024 b

    arXiv 2024

  4. [4]

    Constitutional ai: Harmlessness from ai feedback

    Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022

    arXiv 2022

  5. [5]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33: 0 1877--1901, 2020

    1901

  6. [6]

    Accelerating large language model decoding with speculative sampling

    Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., and Jumper, J. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023 a

    arXiv 2023

  7. [7]

    Teaching large language models to self-debug

    Chen, X., Lin, M., Sch \"a rli, N., and Zhou, D. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128, 2023 b

    arXiv 2023

  8. [8]

    DeepSeek-R1 : Incentivizing reasoning capability in LLMs via reinforcement learning

    DeepSeek-AI. DeepSeek-R1 : Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

Show all 54 references
  1. [9]

    Hierarchical neural story generation

    Fan, A., Lewis, M., and Dauphin, Y. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 889--898, 2018

    2018

  2. [10]

    T., and Gibson, E

    Fedorenko, E., Piantadosi, S. T., and Gibson, E. A. Language is primarily a tool for communication rather than thought. Nature, 630 0 (8017): 0 575--586, 2024

    2024

  3. [11]

    The capacity for moral self-correction in large language models

    Ganguli, D., Askell, A., Schiefer, N., Liao, T., Luko s i \=u t \.e , K., Chen, A., Goldie, A., Mirhoseini, A., Olsson, C., Hernandez, D., et al. The capacity for moral self-correction in large language models. arXiv preprint arXiv:2302.07459, 2023

    2023 arXiv

  4. [12]

    Gemini: A family of highly capable multimodal models

    Gemini Team . Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  5. [13]

    Gemma: Open models based on gemini research and technology

    Gemma Team . Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024

    2024 arXiv

  6. [14]

    Measuring massive multitask language understanding

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  7. [15]

    Learning to write with cooperative discriminators

    Holtzman, A., Buys, J., Forbes, M., Bosselut, A., Golub, D., and Choi, Y. Learning to write with cooperative discriminators. arXiv preprint arXiv:1805.06087, 2018

    2018 arXiv

  8. [16]

    The curious case of neural text degeneration

    Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020

    2020

  9. [17]

    Thinking, Fast and Slow

    Kahneman, D. Thinking, Fast and Slow. Farrar, Straus and Giroux, 2011

    2011

  10. [18]

    Language models can solve computer tasks

    Kim, G., Baldi, P., and McAleer, S. Language models can solve computer tasks. arXiv preprint arXiv:2303.17491, 2023

    2023 arXiv

  11. [19]

    W., Gholami, A., and Keutzer, K

    Kim, S., Mangalam, K., Moon, S., Malik, J., Mahoney, M. W., Gholami, A., and Keutzer, K. Speculative decoding with big little decoder. Advances in Neural Information Processing Systems, 36, 2024

    2024

  12. [20]

    On writing: A memoir of the craft

    King, S. On writing: A memoir of the craft. Pocket Books, 2000

    2000

  13. [21]

    Fast inference from transformers via speculative decoding

    Leviathan, Y., Kalman, M., and Matias, Y. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp.\ 19274--19286. PMLR, 2023

    2023

  14. [22]

    K., Matheos, G., Zhi-Xuan, T., Ghavamizadeh, M., Gothoskar, N., Russell, S., and Mansinghka, V

    Lew, A. K., Matheos, G., Zhi-Xuan, T., Ghavamizadeh, M., Gothoskar, N., Russell, S., and Mansinghka, V. K. SMCP 3: Sequential monte carlo with probabilistic program proposals. In International conference on artificial intelligence and statistics, pp.\ 7061--7088. PMLR, 2023

    2023

  15. [23]

    X., Nie, J.-Y., and Wen, J.-R

    Li, J., Tang, T., Zhao, W. X., Nie, J.-Y., and Wen, J.-R. Elmer: A non-autoregressive pre-trained language model for efficient and effective text generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022

    2022

  16. [24]

    L., Holtzman, A., Fried, D., Liang, P., Eisner, J., Hashimoto, T., Zettlemoyer, L., and Lewis, M

    Li, X. L., Holtzman, A., Fried, D., Liang, P., Eisner, J., Hashimoto, T., Zettlemoyer, L., and Lewis, M. Contrastive decoding: Open-ended text generation as optimization. In The 61st Annual Meeting of the Association for Computational Linguistics, 2023

    2023

  17. [25]

    R., and Eisner, J

    Lin, C.-C., Jaech, A., Li, X., Gormley, M. R., and Eisner, J. Limitations of autoregressive models and their alternatives. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-H...

    2021

  18. [26]

    The llama 3 herd of models

    Llama Team . The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  19. [27]

    Self-refine: Iterative refinement with self-feedback

    Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. Self-refine: Iterative refinement with self-feedback. In Advances in Neural Information Processing Systems, volume 36, 2024

    2024

  20. [28]

    Mistral nemo, 2024

    MistralAI. Mistral nemo, 2024. URL https://mistral.ai/news/mistral-nemo/

    2024

  21. [29]

    and Wright, S

    Nocedal, J. and Wright, S. J. Numerical Optimization. Springer, 1999

    1999

  22. [30]

    GPT -4 technical report

    OpenAI. GPT -4 technical report. arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  23. [31]

    GPT -4o system card

    OpenAI. GPT -4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  24. [32]

    Pan, L., Saxon, M., Xu, W., Nathani, D., Wang, X., and Wang, W. Y. Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies. arXiv preprint arXiv:2308.03188, 2023

    2023 arXiv

  25. [33]

    Causality

    Pearl, J. Causality. Cambridge University Press, 2009

    2009

  26. [34]

    Is temperature the creativity parameter of large language models? In Proceedings of the 15th International Conference on Computational Creativity, 2024

    Peeperkorn, M., Kouwenhoven, T., Brown, D., and Jordanous, A. Is temperature the creativity parameter of large language models? In Proceedings of the 15th International Conference on Computational Creativity, 2024

    2024

  27. [35]

    Language models are unsupervised multitask learners

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. OpenAI Blog, 1 0 (8): 0 9, 2019

    2019

  28. [36]

    The writing strategies book: Your everything guide to developing skilled writers

    Serravallo, J. The writing strategies book: Your everything guide to developing skilled writers. Heinemann Portsmouth, NH, 2017

    2017

  29. [37]

    Long horizon temperature scaling

    Shih, A., Sadigh, D., and Ermon, S. Long horizon temperature scaling. In International Conference on Machine Learning, pp.\ 31422--31434. PMLR, 2023

    2023

  30. [38]

    R., and Yao, S

    Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K. R., and Yao, S. Reflexion: Language agents with verbal reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  31. [39]

    Causation, Prediction, and Search

    Spirtes, P., Glymour, C., and Scheines, R. Causation, Prediction, and Search. Springer New York, 1993

    1993

  32. [40]

    T., Ro, J

    Sun, Z., Suresh, A. T., Ro, J. H., Beirami, A., Jain, H., and Yu, F. Spectr: Fast speculative decoding via optimal transport. Advances in Neural Information Processing Systems, 36, 2024

    2024

  33. [41]

    and Z ilinskas, A

    T \"o rn, A. and Z ilinskas, A. Global Optimization, volume 350. Springer, 1989

    1989

  34. [42]

    Llama 2: Open foundation and fine-tuned chat models

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  35. [43]

    Llms cannot find reasoning errors, but can correct them! arXiv preprint arXiv:2311.08516, 2023

    Tyen, G., Mansoor, H., Chen, P., Mak, T., and C a rbune, V. Llms cannot find reasoning errors, but can correct them! arXiv preprint arXiv:2311.08516, 2023

    2023 arXiv

  36. [44]

    N., Kaiser, L., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  37. [45]

    K., Tavanaei, A., and Bouyarmane, K

    Wei, C., Koo, K. K., Tavanaei, A., and Bouyarmane, K. Confidence-aware sub-structure beam search (cabs): Mitigating hallucination in structured data generation with large language models. arXiv preprint arXiv:2406.00069, 2024

    2024 arXiv

  38. [46]

    Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting

    Wu, H., Xu, J., Wang, J., and Long, M. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in neural information processing systems, 34: 0 22419--22430, 2021

    2021

  39. [47]

    Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation

    Xia, H., Ge, T., Wang, P., Chen, S.-Q., Wei, F., and Sui, Z. Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp.\ 3909--3925, 2023

    2023

  40. [48]

    Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding

    Xia, H., Yang, Z., Dong, Q., Wang, P., Li, Y., Ge, T., Liu, T., Li, W., and Sui, Z. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. arXiv preprint arXiv:2401.07851, 2024

    2024 arXiv

  41. [49]

    X., Kan, M.-Y., He, J., and Xie, M

    Xie, Y., Kawaguchi, K., Zhao, Y., Zhao, J. X., Kan, M.-Y., He, J., and Xie, M. Self-evaluation guided beam search for reasoning. Advances in Neural Information Processing Systems, 36, 2024

    2024

  42. [50]

    Qwen2 technical report

    Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024 a

    2024 arXiv

  43. [51]

    Language-informed beam search decoding for multilingual machine translation

    Yang, Y., Lee, S., and Tadepalli, P. Language-informed beam search decoding for multilingual machine translation. arXiv preprint arXiv:2408.05738, 2024 b

    2024 arXiv

  44. [52]

    React: Synergizing reasoning and acting in language models

    Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022

    2022 arXiv

  45. [53]

    Judging LLM -as-a-judge with MT -bench and chatbot arena

    Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging LLM -as-a-judge with MT -bench and chatbot arena. In Advances in Neural Information Processing Systems, volume 36, pp.\ 46595--46623, 2023

    2023

  46. [54]

    Deductive beam search: Decoding deducible rationale for chain-of-thought reasoning

    Zhu, T., Zhang, K., Xie, J., and Su, Y. Deductive beam search: Decoding deducible rationale for chain-of-thought reasoning. arXiv preprint arXiv:2401.17686, 2024

    2024 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.