Pith. sign in

REVIEW 3 major objections 5 minor 9 cited by

Self-Improvement in Language Models: The Sharpening Mechanism

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Models can improve themselves without external feedback, this paper argues, by sharpening: using their likelihood as a self-reward to train toward high-quality responses — provably up to a coverage limit that exploration removes.

desk verdict A genuinely new statistical framework for feedback-free self-improvement, with matching lower and upper bounds; the theory is real, the realizability caveat is acknowledged and real, and the empirics are the weakest part. read the letter →

arxiv 2412.01951 v2 pith:CKWX33GB submitted 2024-12-02 cs.AI cs.CLcs.LGstat.ML

Audrey Huang , Adam Block , Dylan J. Foster , Dhruv Rohatgi , Cyril Zhang , Max Simchowitz , Jordan T. Ash , Akshay Krishnamurthy This is my paper · ORCID
classification cs.AIcs.CLcs.LGstat.ML MSC 68T0768Q3262C20
keywords self-improvementsharpeningbest-of-NsamplingcoveragecoefficientsamplecomplexityminimaxoptimalityRLHFexploration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Language models appear to get better by critiquing and refining their own generations, even though no new information enters the system; this paper argues the paradox dissolves once self-improvement is read as sharpening. Models are better verifiers than generators, so the model's own sequence-level likelihood is a usable self-reward, and post-training toward high-likelihood responses turns expensive inference-time search into an amortized fine-tuned model. The paper formalizes sharpening as a statistical problem and proves fundamental limits: any algorithm must pay a coverage cost set by how much probability the base model puts on its own best responses, and SFT-style training on best-of-N selections is minimax optimal against that bound. The decisive result is that this coverage cost is not a law of nature: an RLHF-style variant with online exploration provably replaces coverage with a structural complexity measure, yielding an exponential sample-complexity gain for simple model classes. Experiments back the mechanism — picking the most likely of several sampled responses beats greedy decoding on most model-task pairs, and part of the gain survives amortization — so the paper's practical message is that self-improvement is not about creating knowledge but about measuring and buying down coverage.

What carries the argument

The machinery is a sample-and-evaluate oracle model: the learner sees prompts $x \sim \mu$, draws $N$ responses per prompt from the base policy $\pi_{\mathrm{base}}$, and observes their likelihoods, with total queries $m = n \cdot N$ as the cost measure. Three named objects carry the argument. The coverage coefficient $C_{\mathrm{cov}} = \mathbb{E}_{x\sim\mu}[1/\pi_{\mathrm{base}}(y^\star(x)|x)]$ is the expected inverse probability of the best response; the lower bound makes it the universal price of sharpening. The margin condition (Assumption 4.2), a multiplicative gap between the best and second-best responses, converts reward suboptimality into guarantees on the sharpened model's mass. The sequential extrapolation coefficient $\mathrm{SEC}(\Pi)$ measures how well the model class supports online extrapolation and replaces coverage in the exploration-based algorithm. The optimization core is the KL-regularized objective $\mathbb{E}_\pi[\log \pi_{\mathrm{base}}(y|x)] - \beta D_{\mathrm{KL}}(\pi \| \pi_{\mathrm{base}})$, whose exact optimum $\pi^\star_\beta(y|x) \propto \pi_{\mathrm{base}}(y|x)^{1+1/\beta}$ converges to the sharpening target as $\beta \to 0$, together with the MLE generalization bound that drives the SFT analysis.

What would settle it

On any model-task pair, sample a large batch of temperature-1 responses and compare the sequence-level log-likelihood distributions of correct versus incorrect completions; the sharpening mechanism predicts the correct ones stochastically dominate (the pattern in the paper's Figure 6), so a task where conditioning on correctness leaves the likelihood unchanged or shifted lower — where best-of-N by log-likelihood selects no better than chance — would falsify maximum-likelihood sharpening as a general mechanism. A second, theory-level check: in a linear softmax class that satisfies the margin condition but has tiny coverage, Theorem 4.4 predicts XPO reaches $(\epsilon,\delta)$-sharpening in $\mathrm{poly}(d)$ samples regardless of coverage; measuring exponential growth in $d$ would refute the exploration result.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that maximum-likelihood sharpening — training toward the base model's sequence-level arg-max responses under $r_{\mathrm{self}}(y|x) = \log \pi_{\mathrm{base}}(y|x)$ — is the correct formal target for self-improvement, and that two algorithm families provably reach it. The arg-max response is a genuinely hidden object: computationally NP-hard in general (Appendix E), yet fully determined by knowledge already in the model. In the sample-and-evaluate framework, the paper proves a lower bound that any algorithm must pay the coverage coefficient $C_{\mathrm{cov}} = \mathbb{E}_{x\sim\mu}[1/\pi_{\mathrm{base}}(y^\star(x)|x)]$, the expected inverse probability of the best response (Theorem 3.1). SFT-Sharpening — sample $N$ responses, keep the most likely, fine-tune on it — matches that bound and is minimax optimal when the base model covers its good responses (Theorem 4.1). The paper's sharpest claim is that the coverage price is not fundamental: RLHF-Sharpening implemented as a DPO objective with an exploration bonus (XPO) achieves sample complexity governed by the sequential extrapolation coefficient $\mathrm{SEC}(\Pi)$ instead of $C_{\mathrm{cov}}$ (Theorem 4.3), and for linear softmax models this is an exponential improvement, $\mathrm{poly}(d)$ rather than $\exp(\Omega(d))$ samples (Theorem 4.4). Empirically the sharpening mechanism holds up: best-of-N selection by raw log-likelihood beats greedy decoding on most model-task pairs, and SFT-Sharpening recovers a substantial fraction of the inference-time gain.

Load-bearing premise

The guarantees rest on realizability — the model family $\Pi$ must contain the sharpened distribution the algorithm is chasing (the best-of-N induced model for SFT, the KL-regularized optimum for RLHF) — and the paper itself shows in Appendix E.2 that for a two-layer linear softmax class the sharpened model is unrepresentable in any same-form class, so the bounds would be vacuous even with unlimited samples.

Editorial extensions

If this is right

  • SFT-Sharpening — sample $N$ responses, keep the most likely under the base model, fine-tune on them — provably learns an $(\epsilon,\delta)$-sharpened model with $O(C_{\mathrm{cov}} \log|\Pi|/(\delta \epsilon^2))$ samples and is minimax optimal for constant $\delta$, so the common recipe of sampling many responses and fine-tuning on the best has a firm statistical justification whenever the base m
  • An adaptive-sampling variant that stops drawing responses once the current best is confident improves the accuracy dependence from $1/\epsilon^2$ to $1/\epsilon$ (Theorem D.1).
  • RLHF-Sharpening via DPO reaches the same guarantee using only two responses per prompt, at the price of a margin condition measuring how clearly the best response outscores its rivals (Theorem 4.2).
  • With online exploration (XPO), sample complexity scales with $\mathrm{SEC}(\Pi)$ rather than $C_{\mathrm{cov}}$ (Theorem 4.3); for linear softmax classes this is $\mathrm{poly}(d)$ instead of $\exp(\Omega(d))$, an exponential improvement on instances where SFT-Sharpening provably fails (Theorems 4.3 and 4.4).
  • Empirically, best-of-N selection by raw sequence-level log-likelihood beats greedy decoding on most of the model-task pairs tested, and training with SFT-Sharpening recovers a substantial fraction of that inference-time gain — the amortization the theory predicts (Section 5).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A diagnostic the paper leaves implicit: the gap between inference-time best-of-N accuracy and the SFT-Sharpened model's accuracy is a direct, measurable estimate of the non-representability cost flagged in Appendix E.2, and monitoring it tells practitioners whether coverage or realizability is the binding constraint.
  • Theorem 4.3 suggests a concrete allocation rule beyond the paper's text: when coverage is poor, spend online interaction on prompts whose sampled responses have low likelihood — where the verifier signal is most informative — rather than raising $N$ uniformly; exploratory allocation should beat uniform best-of-N exactly on instances where SFT-Sharpening collapses.
  • The lower bound likely transfers to any self-reward that is monotone in response quality — length-normalized likelihood, majority vote, model-as-judge — so the coverage bottleneck proved for log-likelihood probably constrains those richer rewards too unless exploration is applied; the paper's own experiments with those rewards are consistent with a shared mechanism.
  • Read broadly, sharpening recasts self-improvement as compute transfer rather than knowledge creation: buy generations at inference time, bank them as weights in fine-tuning, and the exchange rate is set by coverage — a framing that makes test-time compute and post-training compute directly comparable currencies.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces a formal statistical framework for "sharpening" in language models: using the base model's own sequence-level log-likelihood as a self-reward and training or fine-tuning a model to place large mass on high-likelihood responses. It defines a sample-and-evaluate oracle with sample complexity m = n·N, introduces a coverage coefficient Ccov, proves an information-theoretic lower bound on sharpening, and analyzes two families of algorithms: SFT-Sharpening (maximum likelihood on best-of-N responses) and RLHF-Sharpening (implemented via DPO and via an exploration-based XPO variant). The main theorems give upper bounds that match the lower bound up to logarithmic factors for SFT-Sharpening under a coverage assumption, and show that the XPO variant can replace the coverage coefficient with the sequential extrapolation coefficient SEC(Π). The paper also reports experiments validating that best-of-N sampling with the log-likelihood reward improves accuracy over greedy decoding on several model/dataset pairs, and that a small-scale SFT-Sharpening implementation partially amortizes this gain.

Significance. The paper is one of the first to give a rigorous statistical treatment of self-improvement without external feedback, and it identifies a clean quantity—the coverage coefficient—as a fundamental limit for a natural class of algorithms. The proofs are detailed and appear internally consistent; the lower bound is constructed against explicit model classes with no fitted constants, and the empirical section is honest in reporting cases where the self-reward fails. The coverage-free result via SEC is a plausible and interesting extension of prior work on exploration in RLHF. However, the central optimality claims are conditional on realizability assumptions that the paper itself shows can fail in natural two-layer softmax classes, which substantially tempers the significance of the headline results.

major comments (3)
  1. [Abstract and §4.1 (Theorem 4.1, Remark 4.1, Proposition E.2)] The abstract and the informal discussion state that SFT-Sharpening is minimax optimal 'whenever the initial model has sufficient coverage,' but this omits the load-bearing realizability condition. Theorem 4.1 requires Assumption 4.1 (πBoN_N ∈ Π) in addition to bounded coverage, yet the lower bound in Theorem 3.1 is constructed over a model class Π for which this realizability condition does not hold: the best-of-N distributions induced by the class members are not themselves in Π. The upper and lower bounds therefore match on different problem classes, so the phrase 'minimax optimal' is not fully justified for the class of instances satisfying the theorem's assumptions. Moreover, Proposition E.2 gives a natural two-layer linear softmax class (H=2, V=[n]) in which no same-form model with any norm budget places more than 1/2 mass on the unique sequence-level argmax, so Assumption 4.1 fails for the sharpened target and Theorem 4.1 provides no guarantee even with unlimited samples. Remark 4.1 acknowledges the issue, but the abstract and Section 4.1 should not present the result as an unconditional coverage-based optimality statement; the authors should either prove a lower bound for the restricted class satisfying realizability or qualify the claim explicitly.
  2. [§4.2.2 (Theorem 4.3, Assumption 4.3) and Appendix E.2 (Proposition E.2)] The claim that RLHF-Sharpening 'bypasses the need for coverage' inherits the same representational fragility. Theorem 4.3 relies on Assumption 4.3 (π*_β ∈ Π), and the formal statement in Theorem J.2 requires β to be at most O(γ_margin / log(2|Y|/δ)), which forces π*_β to be very close to the sharpened distribution. Proposition E.2 applies equally to π*_β: in the same two-layer linear softmax class, every model in any class of the same form fails to place more than 1/2 mass on y*, so Assumption 4.3 is violated for any β below the threshold required by Theorem J.2. Consequently, the exploration result does not overcome the representational obstruction, and the coverage-free statement should be explicitly conditioned on the sharpened target being representable in the chosen model class.
  3. [§4.1, Eq. (7) and Theorem 3.1] The claim that the sample complexity bound in Eq. (7) makes SFT-Sharpening 'minimax optimal in the sample-and-evaluate framework when δ is constant' is not stated with enough precision. The lower bound in Theorem 3.1 is proved only for the δ = 1/2 regime, while the upper bound in Eq. (7) contains an explicit 1/δ factor; the phrase 'matches ... up to polynomial dependence on δ' does not establish minimax optimality for the (ϵ,δ)-sharpening objective for arbitrary δ. The authors should either extend the lower bound to cover general δ or state the exact sense in which the dependence on δ is immaterial. This is a local but load-bearing issue for the 'minimax optimal' terminology used in the contributions.
minor comments (5)
  1. [§4.1 display after Theorem 4.1] The displayed definition of bπBoN writes 'arg max_{π∈Π} Σ_{i=1}^n log πbase(yBoN_i | xi)', but the MLE objective should be 'log π(yBoN_i | xi)', using the likelihood of the model being fit rather than the base model. This typo is likely to confuse readers and should be corrected.
  2. [Abstract] The first sentence contains a grammatical error: 'a language models evaluates' should read 'a language model evaluates'.
  3. [Figure 3(d)] The subplot labeled '(d)' is captioned as 'Greedy: Accuracy (%)' in the text, but its axis label reads 'Log Probability over Greedy'; the axis label is inconsistent with the caption and should be fixed.
  4. [§5.2] The phrase 'subsamplng the 50 generated samples' contains a typo ('subsamplng') and should read 'subsampling'.
  5. [Theorem 4.3 (informal) and Theorem J.2] The informal sample complexity bound in Theorem 4.3, m = eO(SEC(Π)·log(|Π|/ρ)/(γ_margin^2 δ^2 ϵ^2)), omits the factor R_max^2 that appears in the formal Theorem J.2. Since R_max can be as large as log|Y|, the informal statement is misleading and should either include R_max or state the dependence explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the sample-complexity theorems are proof-based, and the self-citations to XPO and chi-squared DPO transfer general RLHF results whose assumptions do not include the sharpening conclusion.

full rationale

The paper's central claims are derived rather than assumed. The sharpening target y*(x) is defined as the argmax of log pi_base(y|x), and SFT-Sharpening and RLHF-Sharpening are analyzed as statistical estimators of that target under explicit realizability and coverage assumptions (Assumptions 4.1, 4.3, 4.4). Theorems 4.1-4.4 and the lower bound Theorem 3.1 are proved from sample-complexity arguments (MLE/Hellinger bounds, concentration inequalities, elliptic potential arguments) with no fitted constant or empirical measurement entering the sample-complexity statements. The coverage coefficient Ccov appears both in the lower bound and in the upper bounds, but this matching is the content of minimax optimality rather than a tautology: the lower bound is constructed on an explicit hard class with Ccov bounded by C, and the upper bound is proven for arbitrary pi_base satisfying the stated coverage condition. The appeals to Xie et al. (2024) for XPO and Huang et al. (2024) for the DPO equivalence are citations to general RLHF results with stated assumptions (realizability, bounded density ratios, bounded SEC) that do not include the sharpening target; Appendix J.2 also gives a proof sketch of the XPO guarantee, and the cited results are parameter-free rather than fitted to the present experiments. The acknowledged representational failure in Appendix E.2 (Proposition E.2) shows that Assumptions 4.1 and 4.3 can fail for a two-layer linear softmax class, so the guarantees are conditional rather than unconditional; however, a conditional guarantee under a stated realizability assumption is not a circular argument. I find no step in which a prediction or theorem reduces by construction to its own input.

Assumptions & free parameters 4 free parameters · 8 assumptions · 1 invented entities

The central theoretical claims rest on realizability, margin, and concentrability assumptions about the model class and base policy, plus the unproven sharpening mechanism hypothesis that self-log-likelihood tracks quality. The empirical free parameters are the LoRA/checkpoint-selection choices, which are tuned without a clean held-out validation split.

free parameters (4)
  • LoRA rank = 16 (Phi3.5-Mini), 8 (Mistral-7B)
    Selected by sweeping over LoRA rank for the SFT-Sharpening experiments (Section 5.4, Table 2).
  • Weight decay = 0.1, 0.5, 0.0, 1.0 across model-dataset pairs
    Selected by hyperparameter sweep for SFT-Sharpening; chosen values are dataset-dependent (Table 2).
  • Best checkpoint index = not reported
    SFT-Sharpening results report 'best model checkpoint during training' without a held-out validation set, effectively selecting on the test metric and inflating reported lift (Section 5.4).
  • Number of epochs = 30 (Phi3.5-Mini), 10 (Mistral-7B)
    Chosen due to a known numerical issue with LoRA batch size >1 on Mistral (Section 5.4).
assumptions (8)
  • domain assumption Assumption 4.1: The best-of-N distribution πBoN_N lies in the model class Π for SFT-Sharpening.
    Realizability of the distillation target; required for Theorem 4.1 and shown in Appendix E.2 to be violable for natural classes.
  • domain assumption Assumption 4.3: The KL-regularized optimum π*_β lies in Π for RLHF-Sharpening.
    Realizability for the DPO/XPO analyses (Theorems 4.2 and 4.3).
  • domain assumption Assumption 4.2 (margin): the argmax response is at least a (1+γ_margin) factor more probable than any other response.
    Needed for RLHF-Sharpening to translate reward suboptimality into sharpening error; not needed for SFT-Sharpening.
  • domain assumption Assumption 4.4 (concentrability): Cπ ≤ Cconc and C_{πbase/π;β} ≤ Closs for all π in Π.
    Bounded distribution shift between the model class and base model; used in Theorem 4.2; authors note this is stronger than SFT coverage.
  • domain assumption Assumption 4.5 (bounded log probabilities): |log(1/πbase(y|x))| ≤ Rmax.
    Controls reward range for the XPO guarantee (Theorem 4.3).
  • ad hoc to paper The self-reward log πbase(y|x) is a meaningful proxy for response quality.
    This is the sharpening mechanism hypothesis; the paper provides empirical evidence (Section 5) that is positive but mixed, and it is not proven by the theory.
  • standard math Standard maximum likelihood estimation guarantees (Lemma F.1, Wong and Shen 1995) and martingale concentration (Freedman) hold.
    Used in proofs of Theorems 4.1, D.1, and J.1.
  • standard math NP-hardness of max-cut is assumed for Proposition E.1.
    Used to show sequence-level argmax is NP-hard.
invented entities (1)
  • The sharpening mechanism (formal hypothesis that high self-reward responses are high quality)
    purpose: Explains why self-improvement works without external feedback and defines the objective (Eq. 1).
    No evidence outside the paper's own experiments; those experiments show positive but inconsistent correlation between log-likelihood and correctness (Figures 1 and 6).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Self-Improvement in Language Models: The Sharpening Mechanism." pith.science (2026). https://pith.science/paper/CKWX33GB

@misc{pith2026241201951,
  author       = {Pith},
  title        = {Pith review of: Self-Improvement in Language Models: The Sharpening Mechanism},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CKWX33GB}},
  note         = {Machine review of arXiv:2412.01951}
}
read the original abstract

Recent work in language modeling has raised the possibility of self-improvement, where a language models evaluates and refines its own generations to achieve higher performance without external feedback. It is impossible for this self-improvement to create information that is not already in the model, so why should we expect that this will lead to improved capabilities? We offer a new perspective on the capabilities of self-improvement through a lens we refer to as sharpening. Motivated by the observation that language models are often better at verifying response quality than they are at generating correct responses, we formalize self-improvement as using the model itself as a verifier during post-training in order to ``sharpen'' the model to one placing large mass on high-quality sequences, thereby amortizing the expensive inference-time computation of generating good sequences. We begin by introducing a new statistical framework for sharpening in which the learner aims to sharpen a pre-trained base policy via sample access, and establish fundamental limits. Then we analyze two natural families of self-improvement algorithms based on SFT and RLHF. We find that (i) the SFT-based approach is minimax optimal whenever the initial model has sufficient coverage, but (ii) the RLHF-based approach can improve over SFT-based self-improvement by leveraging online exploration, bypassing the need for coverage. Finally, we empirically validate the sharpening mechanism via inference-time and amortization experiments. We view these findings as a starting point toward a foundational understanding that can guide the design and evaluation of self-improvement algorithms.

Figures

Figures reproduced from arXiv: 2412.01951 by the authors.

Figure 1
Figure 1. Validation of maximum-likelihood sharpening, via Best-of- [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Percent lift in accuracy of inference-time BoN-sharpening over greedy decoding in each task as [PITH_FULL_IMAGE:figures/full_fig_p015_2.png] view at source ↗
Figure 3
Figure 3. Performance of alternative self-reward functions for inference-time BoN sharpening: Percent [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Effect of inference-time BoN-sharpening on [PITH_FULL_IMAGE:figures/full_fig_p027_4.png]
Figure 5
Figure 5. Figure 5: Effect of N on average sequence level log-probabilities for inference-time BoN-sharpening on various model-task pairs, compared to greedy decoding baseline. As predicted by theory, the likelihood of sequences sampled with BoN-sharpening increases with N. 28 [PITH_FULL…
Figure 6
Figure 6. Figure 6: Distribution of sequence-level log-probabilities for responses sampled with temperature 1, conditioned [PITH_FULL_IMAGE:figures/full_fig_p029_6.png]
Figure 7
Figure 7. Figure 7: Evolution of Phi3.5-Mini under SFT-Sharpening (N = 50) on different datasets, as measured by (i) % lift over Greedy in accuracy; and (ii) difference in average sequence-level log-probability of generated responses under the reference model. The fine-tuned model learns …
Figure 8
Figure 8. Figure 8: Evolution of Mistral-7B-Instruct-v0.3 under SFT-Sharpening (N = 50) on MATH, as measured by (i) % lift over Greedy in accuracy; and (ii) difference in average sequence-level log-probability of generated responses under the reference model. 0K 5K 10K 15K 20K 25K 30K Che…
Figure 9
Figure 9. Figure 9: Effect of N on SFT-Sharpening for Phi3.5-Mini on MATH. We report (a) % lift in accuracy over greedy; and (b) lift in sequence-level log-likelihood (averaged over the dataset). In both cases, we see that increasing N leads to greater lift, in accordance with theory. 31 …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. From Prior to Pro: Efficient Skill Mastery via Distribution Contractive RL Finetuning

    cs.RO 2026-03 accept novelty 6.0 of 10

    Residual off-policy RL with selective BC regularization and value-guided sampling contracts a pretrained generative robot policy around successful actions, reaching high success on hard long-horizon tasks from pixels ...

  2. A Task-Centric Theory for Iterative Self-Improvement with Easy-to-Hard Curricula

    cs.LG 2026-02 conditional novelty 6.0 of 10

    Iterative self-improvement provably keeps improving only when initial performance lies in a moderate difficulty interval, and easy-to-hard curricula beat fixed mixtures under moderate difficulty separation and suffici...

  3. e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMs

    cs.LG 2025-06 conditional novelty 6.0 of 10

    e3 couples asymmetric skill chaining, negative-gradient RL, and a difficulty/budget curriculum so a 1.7B model extrapolates test-time compute to 2x its training budget and sets reported <2B state-of-the-art on AIME/HMMT 2025.

  4. Sample Complexity and Representation Ability of Test-time Scaling Paradigms

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Best-of-n sampling provably needs about 1/Δ samples versus 1/Δ² for self-consistency, and a constructed Transformer can route among experts using verifier feedback to reach near-optimal final responses.

  5. Reinforcing General Reasoning without Verifiers

    cs.LG 2025-05 conditional novelty 6.0 of 10

    VeriFree trains LLMs with RL by maximizing the likelihood of the reference answer after generated reasoning, matching verifier-based RL without any verifier.

  6. Temporal Self-Rewarding Language Models: Decoupling Chosen-Rejected via Past-Future

    cs.CL 2025-08 conditional novelty 5.0 of 10

    Anchoring rejected responses to the initial model and choosing responses from a future model raises AlpacaEval 2.0 win rate from 19.69 to 29.44 for Llama3.1-8B.

  7. Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challenges

    cs.LG 2025-02 conditional novelty 5.0 of 10

    Iterative self-training on a model's own correct outputs, with simple length and voting filters, lets transformers generalize to far longer arithmetic and path-finding problems than they saw in training.

  8. Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation

    cs.LG 2026-02 conditional novelty 4.0 of 10

    A taxonomy-driven survey arguing that reward design is the central mechanism shaping reliable LLM reasoning, with maps of reward paradigms, reward-hacking failure modes, and benchmark pitfalls.

  9. LIAR: Leveraging Inference Time Alignment (Best-of-N) to Jailbreak LLMs in Seconds

    cs.CL 2024-12 reject novelty 4.0 of 10

    LIAR shows that best-of-N sampling of suffixes from a GPT-2 model jailbreaks several aligned LLMs with low-perplexity prompts and far faster time-to-attack than training-based attacks.

Reference graph

Works this paper leans on

68 extracted references · 14 canonical work pages · cited by 9 Pith papers

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone.arXiv:2404.14219,

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone.arXiv:2404.14219,

    arXiv

  2. [3]

    the obstacle in natural, concrete model classes is not just computational but in factrepresentational (Proposition E.2). Regarding the latter point, we note that while amortized Bayesian inference has received extensive investigation empirically (Beal, 2003; Gershman and Goodman, 2014; Swersky et al., 2020; Bengio et al., 2021; Hu et al., 2023), we are un...

    work page 2003

  3. [5]

    Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv:2204.05862, 2022a

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, ...

    arXiv

  4. [6]

    J.1.2 Proof of Theorem 4.2′ Proof of Theorem 4.2′

    = O(k4 + 1), where the third-to-last line uses Lemma J.1. J.1.2 Proof of Theorem 4.2′ Proof of Theorem 4.2′. For any modelπ ∈ Π, define J(π) := Eπ[log πbase(y | x)]. Let bπ ∈ Π denote the model returned by the DPO algorithm in Eq.(8). LetEπ,π ′[·] denote shorthand forEx∼µ,y∼π(x),y′∼π′(x)[·], and for anyr : X × Y →R define ∆r(x, y, y′) := r(x, y) − r(x, y′...

    work page 2024

  5. [8]

    Large language monkeys: Scaling inference compute with repeated sampling.arXiv:2407.21787,

    19 Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling.arXiv:2407.21787,

    arXiv

  6. [12]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805,

    arXiv

  7. [15]

    Foundations of reinforcement learning and interactive decision making

    Dylan J Foster and Alexander Rakhlin. Foundations of reinforcement learning and interactive decision making. arXiv:2312.16730,

    arXiv

  8. [16]

    The statistical complexity of interactive decision making

    Dylan J Foster, Sham M Kakade, Jian Qian, and Alexander Rakhlin. The statistical complexity of interactive decision making. arXiv:2112.13487,

    arXiv

Show all 68 references
  1. [17]

    REBEL: Reinforcement learning via regressing relative rewards.arXiv:2404.16767,

    Zhaolin Gao, Jonathan D Chang, Wenhao Zhan, Owen Oertell, Gokul Swamy, Kianté Brantley, Thorsten Joachims, J Andrew Bagnell, Jason D Lee, and Wen Sun. REBEL: Reinforcement learning via regressing relative rewards.arXiv:2404.16767,

    arXiv

  2. [20]

    Measuring massive multitask language understanding.arXiv:2009.03300,

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv:2009.03300,

    2009 arXiv

  3. [21]

    Measuring mathematical problem solving with the math dataset.arXiv:2103.03874,

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset.arXiv:2103.03874,

    arXiv

  4. [24]

    Amortizing intractable inference in large language models.arXiv:2310.04363,

    Edward J Hu, Moksh Jain, Eric Elmoznino, Younesse Kaddar, Guillaume Lajoie, Yoshua Bengio, and Nikolay Malkin. Amortizing intractable inference in large language models.arXiv:2310.04363,

    arXiv

  5. [25]

    Correcting the mythos of KL-regularization: Direct alignment without overparameterization via Chi-squared Preference Optimization.arXiv:2407.13399,

    Audrey Huang, Wenhao Zhan, Tengyang Xie, Jason D Lee, Wen Sun, Akshay Krishnamurthy, and Dylan J Foster. Correcting the mythos of KL-regularization: Direct alignment without overparameterization via Chi-squared Preference Optimization.arXiv:2407.13399,

    arXiv

  6. [26]

    Large language models can self-improve.arXiv:2210.11610,

    Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve.arXiv:2210.11610,

    arXiv

  7. [27]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv:2310.06825,

    arXiv

  8. [29]

    Provably mitigating overoptimization in RLHF: Your SFT loss is implicitly an adversarial regularizer

    Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang, Jose Blanchet, and Zhaoran Wang. Provably mitigating overoptimization in RLHF: Your SFT loss is implicitly an adversarial regularizer. arXiv:2405.16436,

    arXiv

  9. [30]

    Auto-regressive next-token predictors are universal learners.arXiv:2309.06979,

    Eran Malach. Auto-regressive next-token predictors are universal learners.arXiv:2309.06979,

    arXiv

  10. [31]

    If beam search is the answer, what was the question? arXiv:2010.02650,

    Clara Meister, Tim Vieira, and Ryan Cotterell. If beam search is the answer, what was the question? arXiv:2010.02650,

    2010 arXiv

  11. [32]

    Controlled decoding from language models

    Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, et al. Controlled decoding from language models. arXiv:2310.17022,

    arXiv

  12. [34]

    West-of-n: Synthetic preference generation for improved reward modeling.arXiv:2401.12086,

    Alizée Pace, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. West-of-n: Synthetic preference generation for improved reward modeling.arXiv:2401.12086,

    arXiv

  13. [35]

    Language model self-improvement by reinforcement learning contemplation.arXiv:2305.14483,

    Jing-Cheng Pang, Pengyuan Wang, Kaiyuan Li, Xiong-Hui Chen, Jiacheng Xu, Zongzhang Zhang, and Yang Yu. Language model self-improvement by reinforcement learning contemplation.arXiv:2305.14483,

    arXiv

  14. [36]

    Understanding the gains from repeated self-distillation

    Divyansh Pareek, Simon S Du, and Sewoong Oh. Understanding the gains from repeated self-distillation. arXiv:2407.04600,

    arXiv

  15. [37]

    The entropy enigma: Success and failure of entropy minimization.arXiv:2405.05012,

    Ori Press, Ravid Shwartz-Ziv, Yann LeCun, and Matthias Bethge. The entropy enigma: Success and failure of entropy minimization.arXiv:2405.05012,

    arXiv

  16. [38]

    Recursive introspection: Teaching language model agents how to self-improve.arXiv:2407.18219,

    Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar. Recursive introspection: Teaching language model agents how to self-improve.arXiv:2407.18219,

    arXiv

  17. [41]

    Bond: Aligning LLMs with Best-of-N distillation

    Pier Giuseppe Sessa, Robert Dadashi, Léonard Hussenot, Johan Ferret, Nino Vieillard, Alexandre Ramé, Bobak Shariari, Sarah Perrin, Abe Friesen, Geoffrey Cideron, et al. Bond: Aligning LLMs with Best-of-N distillation. arXiv:2407.14622,

    arXiv

  18. [42]

    Scaling LLM test-time compute optimally can be more effective than scaling model parameters.arXiv:2408.03314,

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling model parameters.arXiv:2408.03314,

    arXiv

  19. [43]

    Understanding preference fine-tuning through the lens of coverage.arXiv:2406.01462,

    Yuda Song, Gokul Swamy, Aarti Singh, J Andrew Bagnell, and Wen Sun. Understanding preference fine-tuning through the lens of coverage.arXiv:2406.01462,

    arXiv

  20. [44]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    arXiv

  21. [45]

    Tent: Fully test-time adaptation by entropy minimization.arXiv:2006.10726,

    Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization.arXiv:2006.10726,

    2006 arXiv

  22. [46]

    Self-taught evaluators.arXiv:2408.02666,

    Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi-Yu, Richard Yuanzhe Pang, Maryam Fazel-Zarandi, Jason Weston, and Xian Li. Self-taught evaluators.arXiv:2408.02666,

    arXiv

  23. [47]

    Chain-of-thought reasoning without prompting.arXiv:2402.10200,

    Xuezhi Wang and Denny Zhou. Chain-of-thought reasoning without prompting.arXiv:2402.10200,

    arXiv

  24. [48]

    Self-instruct: Aligning language models with self-generated instructions.arXiv:2212.10560,

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions.arXiv:2212.10560,

    arXiv

  25. [49]

    Meta-rewarding language models: Self-improving alignment with llm-as-a-meta- judge

    Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. Meta-rewarding language models: Self-improving alignment with llm-as-a-meta- judge. arXiv:2407.19594, 2024a. Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck...

    arXiv

  26. [50]

    Exploratory preference optimization: Harnessing implicit Q*-approximation for sample-efficient RLHF

    Tengyang Xie, Dylan J Foster, Akshay Krishnamurthy, Corby Rosset, Ahmed Awadallah, and Alexander Rakhlin. Exploratory preference optimization: Harnessing implicit Q*-approximation for sample-efficient RLHF. arXiv:2405.21046,

    arXiv

  27. [51]

    Gibbs sampling from human feedback: A provable KL-constrained framework for RLHF.arXiv:2312.11456,

    Wei Xiong, Hanze Dong, Chenlu Ye, Han Zhong, Nan Jiang, and Tong Zhang. Gibbs sampling from human feedback: A provable KL-constrained framework for RLHF.arXiv:2312.11456,

    arXiv

  28. [52]

    Asymptotics of language model alignment.arXiv:2404.01730,

    Joy Qiping Yang, Salman Salamatian, Ziteng Sun, Ananda Theertha Suresh, and Ahmad Beirami. Asymptotics of language model alignment.arXiv:2404.01730,

    arXiv

  29. [53]

    A theoretical analysis of Nash learning from human feedback under general KL-regularized preference.arXiv:2402.07314,

    Chenlu Ye, Wei Xiong, Yuheng Zhang, Nan Jiang, and Tong Zhang. A theoretical analysis of Nash learning from human feedback under general KL-regularized preference.arXiv:2402.07314,

    arXiv

  30. [54]

    Self-rewarding language models.arXiv:2401.10020,

    Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models.arXiv:2401.10020,

    arXiv

  31. [55]

    Model Dataset Weight Decay LoRA Rank Phi3.5-Mini MATH 0.1 16 Phi3.5-Mini GSM8k 0.5 16 Phi3.5-Mini ProntoQA 0.0 16 Mistral-7B-Instruct-v0.3 MATH 1.0 8 Table 2: Hyperparameters for training-time sharpening experiments withSFT-Sharpening. 0 10 20 30 40 50 N 70 80 90 100 110 120 1...

    2024

  32. [56]

    LLM-as-a-Judge

    on MATH, as measured by (i) % lift over Greedy in accuracy; and (ii) difference in average sequence-level log-probability of generated responses under the reference model. 0K 5K 10K 15K 20K 25K 30K Checkpoint 0 5 10 15 20 25% Lift over Greedy % Lift in Accuracy over Greedy N=1...

    2022

  33. [57]

    and was popularized as a test-time adaptation method in computer vision (c.f., Wang et al., 2020; Press et al., 2024). Maximum-likelihood sharpening, especially via RL, is closely related in that Eq.(3) with β → 0 and rself = log πbase maximizes Eπ[log πbase(y | x)] rather tha...

    2020

  34. [58]

    More sophisticated inference-time search strategies such tree search and MCTS (Yao et al., 2024; Wan et al., 2024; Mudgal et al., 2023; Zhao et al.,

    can be viewed as instances of inference-time sharpening for specific choices of the self-reward functionrself. More sophisticated inference-time search strategies such tree search and MCTS (Yao et al., 2024; Wan et al., 2024; Mudgal et al., 2023; Zhao et al.,

    2024

  35. [59]

    Perhaps most closely related to our work is Frei et al

    for binary classification and regression, aims to provide convergence guarantees for self-training in stylized setups such 32 as linear models (Mobahi et al., 2020; Das and Sanghavi, 2023; Das et al., 2024; Pareek et al., 2024), with Allen-Zhu and Li (2020) giving guarantees f...

    2020

  36. [61]

    Proof of Proposition C.1

    and γ ∈ [0, 1), as long as N ≥ log(ρ−1) πbase(y⋆γ(x) | x) , 33 inference-time best-of-N sampling produces a responseby ∈ y⋆ γ(x) with probability at least1 − ρ. Proof of Proposition C.1. Fix a promptx ∈ X, failure probabilityρ ∈ (0, 1), and parameterγ ∈ (0, 1). By definition o...

    1995

  37. [62]

    We quantify the quality of a sharpened model as follows

    we define y⋆ γ(x) := y | πbase(y | x) ≥ (1 − γ) · max y∈Y πbase(y | x) as the set of(1 − γ)-approximate maximizers forlog πbase(y | x). We quantify the quality of a sharpened model as follows. Definition F.1(Sharpened model). We say that a modelbπ is (ϵ, δ, γ)-sharpened relati...

    1995

  38. [63]

    Now a⊤Γ−1 t−1a ≤ 1/λ with probability 1, where λ = λmin(Γ0)

    Now observe that log det(Γt) = log det(Γt−1 + At) = log det(Γt−1) + log det(Id + Γ−1/2 t−1 AtΓ−1/2 t−1 ) = log det(Γt−1) + log det Ea∼pt h Id + Γ−1/2 t−1 aa⊤Γ−1/2 t−1 i ≥ log det(Γt−1) + Ea∼pt log det(Id + Γ−1/2 t−1 aa⊤Γ−1/2 t−1 ) = log det(Γt−1) + Ea∼pt log(1 + a⊤Γ−1 t−1a). N...

    2014

  39. [64]

    Note that unlike the non-adaptive framework, the distribution overmi depends on the underlying instance I with which the algorithm interacts

    42 Lower bound under adaptive sample-and-evaluate oracle.In the adaptive framework, we letmi denote the (potentially random) number of tuples(x, y,log πbase(y | x)) observed by the algorithm in which x = xi. Note that unlike the non-adaptive framework, the distribution overmi ...

    2017

  40. [66]

    From here, Bernstein’s inequality and a union bound implies that with probability at least1 − ρ, Eπbase,πbase ∆bπ(x, y, y′) − ∆π⋆ β (x, y, y′) 2 I n ∆bπ ≤ Bn,ρ, ∆π⋆ β ≤ Bn,ρ o ≲ B2 n,ρ log(|Π|ρ−1) n =: ε2 stat. In particular, if we combine this with Eq.(22) and setη = Bn,ρ, th...

    2024

  41. [67]

    Initialize: π(1) ← πbase, D(0) ← ∅

    input: Base model πbase : X →∆(Y), reward functionr : X × Y →R, number of iterationsT ∈ N, KL regularization coefficient β >0, optimism coefficientα >0. Initialize: π(1) ← πbase, D(0) ← ∅. for iteration t = 1, . . . , Tdo Generate sample: (x(t), y(t), ey(t)) via x(t) ∼ µ, y(t)...

    2024

  42. [68]

    Also suppose thatπ⋆ β ∈ Π where π⋆ β(y | x) ∝ π1+β−1 base (y | x)

    so that T ≥ CJ.2 R2 maxSEC(Π) log(2N (Π, ϵdisc)T /ρ) ϵ2δ2β2 and ϵdisc ≤ cJ.2 ϵδp SEC(Π)T where SEC(Π) := SEC(Π, r, T, β, R2 max; πbase). Also suppose thatπ⋆ β ∈ Π where π⋆ β(y | x) ∝ π1+β−1 base (y | x). Then applying Algorithm 1 with base modelπbase, reward functionr, iterati...

    2016

  43. [1973]

    Chain of thought empowers transformers to solve inherently serial problems.arXiv:2402.12875,

    Zhiyuan Li, Hong Liu, Denny Zhou, and Tengyu Ma. Chain of thought empowers transformers to solve inherently serial problems.arXiv:2402.12875,

    arXiv

  44. [1983]

    GPT-4 technical report.arXiv:2303.08774,

    OpenAI. GPT-4 technical report.arXiv:2303.08774,

    arXiv

  45. [1994]

    Towards a theory of model distillation.arXiv preprint arXiv:2403.09053,

    Enric Boix-Adsera. Towards a theory of model distillation.arXiv preprint arXiv:2403.09053,

    arXiv

  46. [1995]

    Butterfly effects of SGD noise: Error amplification in behavior cloning and autoregression.arXiv:2310.11428,

    Adam Block, Dylan J Foster, Akshay Krishnamurthy, Max Simchowitz, and Cyril Zhang. Butterfly effects of SGD noise: Error amplification in behavior cloning and autoregression.arXiv:2310.11428,

    arXiv

  47. [2004]

    BoNBoN alignment for large language models and the sweetness of best-of-n sampling.arXiv:2406.00832,

    Lin Gui, Cristina Gârbacea, and Victor Veitch. BoNBoN alignment for large language models and the sweetness of best-of-n sampling.arXiv:2406.00832,

    arXiv

  48. [2006]

    Self-play fine-tuning converts weak language models to strong language models.arXiv:2401.01335,

    Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models.arXiv:2401.01335,

    arXiv

  49. [2011]

    In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning.arXiv:2101.06329,

    Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning.arXiv:2101.06329,

    arXiv

  50. [2014]

    Palm 2 technical report.arXiv:2305.10403,

    Google. Palm 2 technical report.arXiv:2305.10403,

    arXiv

  51. [2015]

    Lora: Low-rank adaptation of large language models.arXiv:2106.09685,

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models.arXiv:2106.09685,

    arXiv

  52. [2016]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv:1707.06347,

    arXiv

  53. [2017]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv:2110.14168,

    arXiv

  54. [2018]

    Distillation≈ early stopping? Harvesting dark knowledge utilizing anisotropic information retrieval for overparameterized neural network.arXiv:1910.01255,

    Bin Dong, Jikai Hou, Yiping Lu, and Zhihua Zhang. Distillation≈ early stopping? Harvesting dark knowledge utilizing anisotropic information retrieval for overparameterized neural network.arXiv:1910.01255,

    1910 arXiv

  55. [2019]

    The llama 3 herd of models.arXiv:2407.21783,

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.arXiv:2407.21783,

    arXiv

  56. [2020]

    Variational best-of-n alignment.arXiv:2407.06057,

    Afra Amini, Tim Vieira, and Ryan Cotterell. Variational best-of-n alignment.arXiv:2407.06057,

    arXiv

  57. [2021]

    Distilling the knowledge in a neural network.arXiv:1503.02531,

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.arXiv:1503.02531,

    arXiv

  58. [2022]

    Towards understanding ensemble, knowledge distillation and self-distillation in deep learning.arXiv:2012.09816,

    Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning.arXiv:2012.09816,

    2012 arXiv

  59. [2023]

    Retraining with predicted hard labels provably increases model accuracy

    Rudrajit Das, Inderjit S Dhillon, Alessandro Epasto, Adel Javanmard, Jieming Mao, Vahab Mirrokni, Sujay Sanghavi, and Peilin Zhong. Retraining with predicted hard labels provably increases model accuracy. arXiv:2406.11206,

    arXiv

  60. [2024]

    Transferring inductive biases through knowledge distillation

    Samira Abnar, Mostafa Dehghani, and Willem Zuidema. Transferring inductive biases through knowledge distillation. arXiv:2006.00555,

    2006 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.