Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper proves a quantitative upper bound on the alignment gap left by prompt optimization of a frozen language model, and supplies a closed-form optimal prompter.

desk verdict The framework is sensible and the decomposition is new, but Theorem 6.1 is false as stated without a nonnegativity assumption on the reward; the experiments don't test the theorem anyway. read the letter →

arxiv 2501.03486 v1 pith:ETQXKHTF submitted 2025-01-07 cs.LG cs.AI

Prashant Trivedi , Souradip Chakraborty , Avinash Reddy , Vaneet Aggarwal , Amrit Singh Bedi , George K. Atia This is my paper · ORCID
classification cs.LGcs.AI MSC 68T0768T50
keywords promptoptimizationLLMalignmentRLHFfrozenlanguagemodelsuboptimalityboundtotalvariationdistanceKL-regularizedclosed-formoptimalprompter
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that prompt optimization is a theoretically grounded alternative to RLHF for aligning language models whose weights cannot be changed. Treating the prompter as a trainable distribution $\rho(x'|x)$ in front of a frozen policy $\pi_F$, it derives the closed-form optimal prompter and then bounds the suboptimality gap $J(\pi^*) - J(\tilde{\pi}_{\rho^*})$ relative to the RLHF-optimal policy. The bound splits the gap into three readable pieces: how far the frozen model sits from $\pi^*$, how much the reference prompter moves the frozen model's output distribution, and the KL divergence budget the optimal prompter spends. If this is right, alignment can be pursued, measured, and partly closed on frozen or black-box models instead of by parameter fine-tuning.

What carries the argument

The load-bearing object is the closed-form optimal prompter of Lemma 5.1, a Gibbs-style distribution that reweights the reference prompter by the exponentiated expected reward of the frozen model on the rewritten prompt. It is what makes the suboptimality bound in Theorem 6.1 possible: the optimality inequality for $\rho^*$ converts the prompter's contribution into a negative KL term, so the bound shrinks exactly as the prompter spends KL budget to move away from $\rho_{\mathrm{sft}}$. The remaining two terms are total-variation distances that capture, respectively, the baseline quality of $\pi_F$ relative to $\pi^*$ and the sensitivity of $\pi_F$ to prompt changes under $\rho_{\mathrm{sft}}$.

What would settle it

Train the prompter on a single dataset as in Section 7.1, record the realized $D_{\mathrm{KL}}(\rho_{\mathrm{ppo}}\|\rho_{\mathrm{sft}})$, estimate the two total-variation terms from samples, and compare the measured gap $J(\pi^*) - J(\tilde{\pi}_{\rho_{\mathrm{ppo}}})$ with the right-hand side of Theorem 6.1. A violation of the inequality, or a large mismatch between the PPO-trained prompter and the closed-form $\rho^*$ computed from Eq. (10), would show the experiments do not test the theorem.

Watch

Extended reading notes

Core claim

Align-Pro's central assertion is that the best prompter for a frozen model is the solution to a KL-regularized reward maximization, and that this prompter makes the RLHF gap quantitative rather than empirical. Concretely, Lemma 5.1 gives $\rho^*(x'|x) \propto \rho_{\mathrm{sft}}(x'|x)\exp(R(x,x')/\lambda)$, where $R(x,x')$ is the expected reward of the frozen model on the rewritten prompt, and Theorem 6.1 bounds the gap by $r_{\max}\mathbb{E}[d_{\mathrm{TV}}(\pi^*,\pi_F)] + r_{\max}\mathbb{E}[d_{\mathrm{TV}}(\pi_F(\cdot|x),\pi_F(\cdot|x'))] - \lambda \mathbb{E}[D_{\mathrm{KL}}(\rho^*\|\rho_{\mathrm{sft}})]$. The paper claims this decomposition shows exactly where prompt optimization loses to fine-tuning: in the frozen model's intrinsic misalignment and in the reference prompter's destabilizing rewrites, offset by the KL budget that the optimized prompter is allowed to spend. The experiments with PPO-trained prompters on three datasets are presented as proof of concept that a real prompter can realize the effect and beat the no-fine-tuning baseline.

Load-bearing premise

The bound and the experiments both rest on the assumption that the PPO-trained prompter described in Section 7.1 converges to the closed-form $\rho^*$ of Eq. (10), and that the reward used in training is the same $r^*$ defining $\pi^*$; the paper does not report $\lambda$ or verify this convergence.

Editorial extensions

If this is right

  • If the frozen model already equals the RLHF-optimal policy, the first total-variation term vanishes and the bound reduces to prompt sensitivity minus KL budget.
  • If the reference prompter is a delta distribution, the prompt-sensitivity term vanishes, isolating the frozen-model quality and the KL budget.
  • Because the third term is negative, spending KL divergence budget on the prompter can only shrink the worst-case gap, giving a quantitative rationale for prompt engineering on frozen models.
  • The win-rate and mean-reward experiments on three datasets are offered as evidence that a PPO-trained prompter realizes this effect, beating no fine-tuning and approaching the RLHF oracle.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The bound turns $\lambda$ into a measurable alignment budget: one could estimate the two total-variation terms from data and choose $\lambda$ so that the negative KL term offsets them, which the paper leaves implicit.
  • The theorem assumes the same reward $r^*$ defines both $\pi^*$ and the prompter objective; a natural stress test is to train the prompter with one reward and evaluate with another, probing whether the inequality survives reward misspecification.
  • The prompt-sensitivity term predicts that a low-variance reference prompter, or an ensemble of prompters whose rewrites cancel, should shrink the second term; comparing Align-Pro's gap under different reference prompters would test that prediction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Align-Pro, a prompt optimization framework for aligning frozen LLMs without parameter fine-tuning. It formulates prompter learning as maximizing the expected reward of the frozen model under a KL-regularization penalty relative to a reference prompter, derives a closed-form optimal prompter (Lemma 5.1), and proves a suboptimality bound comparing the resulting policy with the RLHF optimal policy (Theorem 6.1). Experiments on UltraFeedback, HelpSteer, and Orca with PPO-trained prompters report improved mean reward and win rate over a no-fine-tuning baseline.

Significance. If Theorem 6.1 were correct as stated, the decomposition would be a useful conceptual contribution: it separates the prompt-optimization gap into frozen-model quality, reference-prompt sensitivity, and a KL-budget term. The closed-form prompter and the accompanying proof effort are valuable. However, the theorem currently omits a necessary reward-range assumption, and the proof's total-variation bounds fail for signed rewards. In addition, the experiments train the prompter with PPO against a proxy reward model and never report the KL coefficient lambda, so they do not test the object analyzed in the theorem. These issues are local and repairable, but the central theoretical claim needs correction before publication.

major comments (3)
  1. [Theorem 6.1 / Eq. (11) and Appendix B, Eqs. (26), (31)] The central bound is not valid for the general bounded-reward setting stated. The proof uses the inequality |E_{pi1}[r] - E_{pi2}[r]| <= rmax * d_TV(pi1, pi2), which holds for nonnegative rewards with supremum rmax, but not for signed rewards. For r in [-rmax, rmax], the sharp constant is (sup r - inf r) = 2 rmax, and the paper never states a nonnegativity assumption or defines rmax. A concrete counterexample: take a degenerate prompt distribution at x, two prompts a and b with rho_sft(a|x)=rho_sft(b|x)=0.5, responses y1 and y2, rewards r*(x,y1)=1 and r*(x,y2)=-1, pi_F(y1|x)=pi_F(y1|a)=1, pi_F(y2|b)=1, and lambda=1. Then pi* = delta_{y1}, the first TV term in Eq. (11) is 0, the second is 0.5, D_KL(rho*||rho_sft) is approximately 0.328, and the RHS is 0.172, while J(pi*) - J(e_rho*) = 1 - (0.881 - 0.119) = 0.238, violating the inequality. The theorem can be repaired by replacing rmax with the reward range (sup r - inf r) or by explicitly assuming r* in [0, rmax] and defining rmax, but as written the statement is false.
  2. [Section 7.1] The experiments do not instantiate the object analyzed in Theorem 6.1. The theorem concerns the exact maximizer rho* of Eq. (9) with a known lambda and reference rho_sft. The experiments instead train the prompter with PPO using a separate reward model (RM-Gemma-2B); the value of lambda in Eq. (9) is never reported, and no convergence or approximation argument shows that the PPO-trained prompter approaches rho*. Consequently, the win rates in Table 1 and the mean/variance results in Figures 2 and 3 do not test the theoretical claim. The authors should either train with the Eq. (9) objective and report lambda, or provide a quantitative bound connecting the PPO solution to rho*.
  3. [Equation (11) and discussion following Theorem 6.1] The negative KL term in Eq. (11) is imported directly from the optimality condition used to define rho* (Appendix B, Eq. (32)). It is therefore a restatement of the prompter's own objective rather than an independently measured quantity. The text should say this explicitly; as written, the interpretation that prompt optimization 'reduces the suboptimality bound' through the third term may mislead readers into thinking the bound relies on an external improvement rather than on the deliberate choice of the KL-regularized objective in Eq. (9).
minor comments (5)
  1. [Appendix A, Eqs. (16)-(19)] The proof of Lemma 5.1 differentiates the objective without enforcing the normalization constraint sum_{x'} rho(x'|x)=1, then normalizes the resulting expression; the final formula is correct, but the derivation should use a Lagrangian multiplier or cite the standard KL-regularized optimization argument.
  2. [Section 7.1, first paragraph] The text lists both F1 and F2 as Llama-3.1-8B-Instruct, but Table 1 and Figures 2-3 indicate that the second frozen model is Qwen-2.5-7B-Instruct; please correct this typo.
  3. [Equation (4) versus Eq. (2)] Equation (4) uses pi_F as the reference policy in the closed-form RLHF solution, while Eq. (2) defines the KL penalty relative to pi_sft; please clarify whether pi_F = pi_sft or redefine the reference appropriately.
  4. [Table 1] Win rates are computed on 100 samples without confidence intervals or significance tests; given the small sample, the claim of consistent improvement should be accompanied by error bars or a statistical test.
  5. [Lemma 5.1, statement] The term 'log partition function' for Z(x) is a misnomer; Z(x) is the partition function, while log Z(x) is the log-partition function.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's bound follows from its stated optimization objective, and the empirical claims are benchmarked externally.

full rationale

Theorem 6.1's negative KL term is the only place one might suspect self-referentiality, but it is derived legitimately: inequality (32) uses the definition of rho* as the maximizer of objective (9), and inequality (34) rearranges that optimality condition. This is not an independent prediction smuggled in; it is a mathematical consequence of the prompter's defining optimization problem. The first two terms in Eq. (11) are structural TV distances that do not depend on rho*, so the bound is not simply a restatement of the objective. Lemma 5.1's closed form is a standard KL-regularized softmax derived from an externally cited source [48], not from the present authors' prior work. The only same-author citation ([43]) appears as related-work context for red-teaming and is not load-bearing. Experiments are evaluated against a no-fine-tuning baseline with an external reward model and a GPT-4 judge, so the empirical win rates do not reduce to the training objective. A reviewer concern that Theorem 6.1 requires nonnegative rewards (or a reward-range correction) is a correctness issue, not a circularity issue; the paper's own limitations section does not assert any circular dependency. Therefore no circularity is found.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The ledger shows one free parameter (lambda), five structural assumptions, and no invented entities. The core math is standard KL-regularized optimization; the main burden is the assumption that a PPO-trained prompter realizes the exact theoretical solution.

free parameters (1)
  • lambda (prompter KL coefficient) = not reported
    Controls the KL regularizer in Eq (9) and appears with a negative sign in Theorem 6.1; no experimental value is specified, so the bound's third term cannot be evaluated.
assumptions (5)
  • domain assumption The effective policy is e_pi_rho(y|x) = sum_x' pi_F(y|x') rho(x'|x).
    Equation (3); this mixture model is the central modeling choice for prompt optimization.
  • domain assumption The same ground-truth reward r* defines the RLHF optimal policy and the prompter objective.
    Equations (4) and (8)-(9); in practice only a proxy reward model is used, so this is an idealized assumption.
  • ad hoc to paper The optimal prompter is the exact maximizer of Eq (9) with reference rho_sft.
    Theorem 6.1 relies on rho* from Eq (10), but Section 7.1 trains the prompter via PPO; the link between PPO and rho* is assumed, never proven.
  • standard math TV-distance bound: |E_P f - E_Q f| <= rmax d_TV(P,Q) for |f| <= rmax.
    Used in the proof of Theorem 6.1 to bound Delta1 and Delta3.
  • standard math The exponential-tilt form of the optimal policy solves the KL-regularized RLHF objective.
    Equation (4) is taken from [46,47]; needed to define pi* and to derive the closed-form prompter.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment." pith.science (2026). https://pith.science/paper/ETQXKHTF

@misc{pith2026250103486,
  author       = {Pith},
  title        = {Pith review of: Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ETQXKHTF}},
  note         = {Machine review of arXiv:2501.03486}
}
read the original abstract

The alignment of large language models (LLMs) with human values is critical as these models become increasingly integrated into various societal and decision-making processes. Traditional methods, such as reinforcement learning from human feedback (RLHF), achieve alignment by fine-tuning model parameters, but these approaches are often computationally expensive and impractical when models are frozen or inaccessible for parameter modification. In contrast, prompt optimization is a viable alternative to RLHF for LLM alignment. While the existing literature has shown empirical promise of prompt optimization, its theoretical underpinning remains under-explored. We address this gap by formulating prompt optimization as an optimization problem and try to provide theoretical insights into the optimality of such a framework. To analyze the performance of the prompt optimization, we study theoretical suboptimality bounds and provide insights in terms of how prompt optimization depends upon the given prompter and target model. We also provide empirical validation through experiments on various datasets, demonstrating that prompt optimization can effectively align LLMs, even when parameter fine-tuning is not feasible.

Figures

Figures reproduced from arXiv: 2501.03486 by the authors.

Figure 1
Figure 1. A basic overview of the prompt optimization framework. A prompter modifies the prompt [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Reward mean comparisons. Figure shows the reward mean across the chosen datasets. Align-Pro shows an improvement over the no fine-tuning approach. We employ two prompters P1 (Phi-3.5-Instruct) and P2 (Qwen-2.5-1.5B-Instruct), along with two frozen LLMs, denoted as F1 (Llama-3.1-8B-Instruct) and F2 (Llama-3.1-8B-Instruct). The oracle is fine-tuned LLM via RLHF. (a) (b) (c) [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Reward variance comparisons. Align-Pro has the least variance compared to Oracle and no fine-tuning approach. Due to the prompter’s precise guidance, the frozen LLM generates almost similar responses in terms of helpfulness and coherence, which results in less diverse responses. We use the following terminologies for the prompters and the frozen models: P1 (Phi-3.5-Instruct), P2 (Qwen-2.5-1.5B-Instruct), F1 (Llama-3… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SafeAgent: Safeguarding LLM Agents via an Automated Risk Simulator

    cs.AI 2025-05 conditional novelty 6.0 of 10

    This paper introduces AutoSafe, an automated pipeline that generates agent risk scenarios, samples safe actions via self-reflection, and fine-tunes LLM agents to improve safety on synthetic and real-world benchmarks.

Reference graph

Works this paper leans on

57 extracted references · 54 canonical work pages · cited by 1 Pith paper

  1. [1]

    Secrets of rlhf in large language models part ii: Reward modeling

    Wang B, Zheng R, Chen L, Liu Y, Dou S, Huang C, et al. Secrets of rlhf in large language models part ii: Reward modeling. arXiv preprint arXiv:240106080. 2024. 12

    work page 2024

  2. [2]

    A survey of reinforcement learning from human feedback

    Kaufmann T, Weng P, Bengs V, Hüllermeier E. A survey of reinforcement learning from human feedback. arXiv preprint arXiv:231214925. 2023

    work page 2023

  3. [3]

    More RLHF, More Trust? On The Impact of Human Preference Alignment On Language Model Trustworthiness

    Li AJ, Krishna S, Lakkaraju H. More RLHF, More Trust? On The Impact of Human Preference Alignment On Language Model Trustworthiness. arXiv preprint arXiv:240418870. 2024

    work page 2024

  4. [4]

    Safe rlhf: Safe reinforcement learning from human feedback

    Dai J, Pan X, Sun R, Ji J, Xu X, Liu M, et al. Safe rlhf: Safe reinforcement learning from human feedback. arXiv preprint arXiv:231012773. 2023

    work page 2023

  5. [5]

    Principled Reinforcement Learning with Human Feedback from Pairwise orK-wise Comparisons

    Zhu B, Jiao J, Jordan MI. Principled Reinforcement Learning with Human Feedback from Pairwise orK-wise Comparisons. arXiv preprint arXiv:230111270. 2023

    work page 2023

  6. [6]

    A general theoretical paradigm to understand learning from human preferences

    Azar MG, Rowland M, Piot B, Guo D, Calandriello D, Valko M, et al. A general theoretical paradigm to understand learning from human preferences. arXiv preprint arXiv:231012036. 2023

    work page 2023

  7. [7]

    Fine-Tuning Language Models from Human Preferences

    Ziegler DM, Stiennon N, Wu J, Brown TB, Radford A, Amodei D, et al. Fine-Tuning Language Models from Human Preferences. arXiv preprint arXiv:190908593. 2019

    work page 2019

  8. [8]

    Open problems and fundamental limitations of reinforcement learning from human feedback

    Casper S, Davies X, Shi C, Gilbert TK, Scheurer J, Rando J, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:230715217. 2023

    work page 2023

Show all 57 references
  1. [9]

    Training language models to follow instructions with human feedback

    Ouyang L, Wu J, Jiang X, Almeida D, Wainwright C, Mishkin P, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems. 2022;35:27730-44

    2022

  2. [10]

    Black-Box Prompt Learning for Pre-trained Language Models

    Diao S, Huang Z, Xu R, Li X, Yong L, Zhou X, et al. Black-Box Prompt Learning for Pre-trained Language Models. Transactions on Machine Learning Research. 2023

    2023

  3. [11]

    AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts

    Shin T, Razeghi Y, Logan IV RL, Wallace E, Singh S. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In: Proc. EMNLP

  4. [12]

    Prompt Optimization with Human Feedback

    Lin X, Dai Z, Verma A, Ng SK, Jaillet P, Low BKH. Prompt Optimization with Human Feedback. arXiv preprint arXiv:240517346. 2024

    2024

  5. [13]

    Learning overparameterized neural networks via stochastic gradient descent on structured data

    Li Y, Liang Y. Learning overparameterized neural networks via stochastic gradient descent on structured data. Advances in neural information processing systems. 2018;31

    2018

  6. [14]

    The Power of Scale for Parameter-Efficient Prompt Tuning

    Lester B, Al-Rfou R, Constant N. The Power of Scale for Parameter-Efficient Prompt Tuning. In: Proc. EMNLP; 2021. p. 3045-59

    2021

  7. [15]

    PRewrite: Prompt Rewriting with Reinforcement Learning

    Kong W, Hombaiah SA, Zhang M, Mei Q, Bendersky M. PRewrite: Prompt Rewriting with Reinforcement Learning. arXiv preprint arXiv:240108189. 2024

    2024

  8. [16]

    PromptAgent: Strategic planning with language models enables expert-level prompt optimization

    Wang X, Li C, Wang Z, Bai F, Luo H, Zhang J, et al. PromptAgent: Strategic planning with language models enables expert-level prompt optimization. arXiv preprint arXiv:231016427. 2023. 13

    2023

  9. [17]

    Alpacafarm: A simulation framework for methods that learn from human feedback

    Dubois Y, Li CX, Taori R, Zhang T, Gulrajani I, Ba J, et al. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems. 2024;36

    2024

  10. [18]

    Fine-tuning language models from human preferences

    Ziegler DM, Stiennon N, Wu J, Brown TB, Radford A, Amodei D, et al. Fine-tuning language models from human preferences. arXiv preprint arXiv:190908593. 2019

    2019

  11. [19]

    RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs

    Chaudhari S, Aggarwal P, Murahari V, Rajpurohit T, Kalyan A, Narasimhan K, et al. RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs. arXiv preprint arXiv:240408555. 2024

    2024

  12. [20]

    Direct preference optimization: Your language model is secretly a reward model

    Rafailov R, Sharma A, Mitchell E, Manning CD, Ermon S, Finn C. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems. 2024;36

    2024

  13. [21]

    Slic-hf: Sequence likelihood calibration with human feedback

    Zhao Y, Joshi R, Liu T, Khalman M, Saleh M, Liu PJ. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:230510425. 2023

    2023

  14. [22]

    Direct Preference Optimization with an Offset

    Amini A, Vieira T, Cotterell R. Direct Preference Optimization with an Offset. arXiv preprint arXiv:240210571. 2024

    2024

  15. [23]

    A general theoretical paradigm to understand learning from human preferences

    Azar MG, Guo ZD, Piot B, Munos R, Rowland M, Valko M, et al. A general theoretical paradigm to understand learning from human preferences. In: International Conference on Artificial Intelligence and Statistics. PMLR; 2024. p. 4447-55

    2024

  16. [24]

    Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model

    Gou Q, Nguyen CT. Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model. arXiv preprint arXiv:240319443. 2024

    2024

  17. [25]

    LiPO: Listwise Preference Optimization through Learning-to-Rank

    Liu T, Qin Z, Wu J, Shen J, Khalman M, Joshi R, et al. LiPO: Listwise Preference Optimization through Learning-to-Rank. arXiv preprint arXiv:240201878. 2024

    2024

  18. [26]

    Filtered Direct Preference Optimization

    Morimura T, Sakamoto M, Jinnai Y, Abe K, Air K. Filtered Direct Preference Optimization. arXiv preprint arXiv:240413846. 2024

    2024

  19. [27]

    Generalized Preference Optimization: A Unified Approach to Offline Alignment

    Tang Y, Guo ZD, Zheng Z, Calandriello D, Munos R, Rowland M, et al. Generalized Preference Optimization: A Unified Approach to Offline Alignment. arXiv preprint arXiv:240205749. 2024

    2024

  20. [28]

    Beyond reverse kl: Generalizing direct preference optimization with diverse divergence constraints

    Wang C, Jiang Y, Yang C, Liu H, Chen Y. Beyond reverse kl: Generalizing direct preference optimization with diverse divergence constraints. arXiv preprint arXiv:230916240. 2023

    2023

  21. [29]

    Efficient Exploration for LLMs

    Dwaracherla V, Asghari SM, Hao B, Van Roy B. Efficient Exploration for LLMs. arXiv:240200396. 2024

    2024

  22. [30]

    ORPO: Monolithic Preference Optimization without Reference Model; 2024

    Hong J, Lee N, Thorne J. ORPO: Monolithic Preference Optimization without Reference Model; 2024. Available from:https://arxiv.org/abs/2403.07691. 14

    2024 arXiv

  23. [31]

    Intuitive Fine-Tuning: Towards Simplifying Alignment into a Single Process; 2024

    Hua E, Qi B, Zhang K, Yu Y, Ding N, Lv X, et al.. Intuitive Fine-Tuning: Towards Simplifying Alignment into a Single Process; 2024. Available from:https://arxiv. org/abs/2405.11870

    2024 arXiv

  24. [32]

    Toward Human Readable Prompt Tuning: Kubrick’s The Shining is a good movie, and a good prompt too? In: Proc

    Shi W, Han X, Gonen H, Holtzman A, Tsvetkov Y, Zettlemoyer L. Toward Human Readable Prompt Tuning: Kubrick’s The Shining is a good movie, and a good prompt too? In: Proc. EMNLP; 2023. p. 10994-1005

    2023

  25. [33]

    Prefix-Tuning: Optimizing Continuous Prompts for Generation

    Li XL, Liang P. Prefix-Tuning: Optimizing Continuous Prompts for Generation. In: Proc. ACL; 2021. p. 4582-97

    2021

  26. [34]

    Factual Probing Is [MASK]: Learning vs

    Zhong Z, Friedman D, Chen D. Factual Probing Is [MASK]: Learning vs. Learning to Recall. In: Proc. NAACL; 2021. p. 5017-33

    2021

  27. [35]

    Clip-Tuning: Towards Derivative-free Prompt Learning with a Mixture of Rewards

    Chai Y, Wang S, Sun Y, Tian H, Wu H, Wang H. Clip-Tuning: Towards Derivative-free Prompt Learning with a Mixture of Rewards. In: Proc. EMNLP (Findings); 2022. p. 108-17

    2022

  28. [36]

    Black-box tuning for language-model-as-a-service

    Sun T, Shao Y, Qian H, Huang X, Qiu X. Black-box tuning for language-model-as-a-service. In: Proc. ICML; 2022. p. 20841-55

    2022

  29. [37]

    BBTv2: Pure Black-Box Optimization Can Be Comparable to Gradient Descent for Few-Shot Learning

    Sun T, He Z, Qian H, Huang X, Qiu X. BBTv2: Pure Black-Box Optimization Can Be Comparable to Gradient Descent for Few-Shot Learning. In: Proc. EMNLP; 2022. p. 3916-30

    2022

  30. [38]

    MultiPrompter: Cooperative Prompt Optimization with Multi-Agent Reinforcement Learning

    Kim DK, Sohn S, Logeswaran L, Shim D, Lee H. MultiPrompter: Cooperative Prompt Optimization with Multi-Agent Reinforcement Learning. arXiv preprint arXiv:231016730. 2023

    2023

  31. [39]

    Curiosity-driven red-teaming for large language models

    Hong ZW, Shenfeld I, Wang TH, Chuang YS, Pareja A, Glass J, et al. Curiosity-driven red-teaming for large language models. arXiv preprint arXiv:240219464. 2024

    2024

  32. [40]

    Discovering Language Model Behaviors with Model-Written Evaluations

    Perez E, Ringer S, Lukoši¯ ut˙ e K, Nguyen K, Chen E, Heiner S, et al.. Discovering Language Model Behaviors with Model-Written Evaluations. arXiv; 2022. Available from: https://arxiv.org/abs/2212.09251

    2022 arXiv

  33. [41]

    Gradient-based language model red teaming

    Wichers N, Denison C, Beirami A. Gradient-based language model red teaming. arXiv preprint arXiv:240116656. 2024

    2024

  34. [42]

    Learning diverse attacks on large language models for robust red-teaming and safety tuning

    Lee S, Kim M, Cherif L, Dobre D, Lee J, Hwang SJ, et al. Learning diverse attacks on large language models for robust red-teaming and safety tuning. arXiv preprint arXiv:240518540. 2024

    2024

  35. [43]

    LIAR: Leveraging Alignment (Best-of-N) to Jailbreak LLMs in Seconds

    Beetham J, Chakraborty S, Wang M, Huang F, Bedi AS, Shah M. LIAR: Leveraging Alignment (Best-of-N) to Jailbreak LLMs in Seconds. arXiv preprint arXiv:241205232. 2024

    2024

  36. [44]

    Rank analysis of incomplete block designs: I

    Bradley RA, Terry ME. Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika. 1952;39(3/4):324-45. 15

    1952

  37. [45]

    Large Language Models Are Human-Level Prompt Engineers

    Zhou Y, Muresanu AI, Han Z, Paster K, Pitis S, Chan H, et al. Large Language Models Are Human-Level Prompt Engineers. In: Proc. ICLR; 2023

    2023

  38. [46]

    Advantage-weighted regression: Simple and scalable off-policy reinforcement learning

    Peng XB, Kumar A, Zhang G, Levine S. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:191000177. 2019

    2019

  39. [47]

    Reinforcement learning by reward-weighted regression for operational space control

    Peters J, Schaal S. Reinforcement learning by reward-weighted regression for operational space control. In: Proceedings of the 24th international conference on Machine learning

  40. [48]

    Direct preference optimization: Your language model is secretly a reward model

    Rafailov R, Sharma A, Mitchell E, Ermon S, Manning CD, Finn C. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:230518290. 2023

    2023

  41. [49]

    ULTRAFEEDBACK: Boosting Language Models with Scaled AI Feedback

    Cui G, Yuan L, Ding N, Yao G, He B, Zhu W, et al. ULTRAFEEDBACK: Boosting Language Models with Scaled AI Feedback. In: Forty-first International Conference on Machine Learning; 2024

    2024

  42. [50]

    Helpsteer: Multi-attribute helpfulness dataset for steerlm

    Wang Z, Dong Y, Zeng J, Adams V, Sreedhar MN, Egert D, et al. Helpsteer: Multi-attribute helpfulness dataset for steerlm. arXiv preprint arXiv:231109528. 2023

    2023

  43. [51]

    Orca: Progressive learning from complex explanation traces of gpt-4

    Mukherjee S, Mitra A, Jawahar G, Agarwal S, Palangi H, Awadallah A. Orca: Progressive learning from complex explanation traces of gpt-4. arXiv preprint arXiv:230602707. 2023. 16 Appendix A Proof of Lemma 5.1 Lemma 5.1.Let R(x, x′) := Ey∼πF (·|x′)[r∗(x, y)], andλ >0 be the prom...

    2023

  44. [52]

    Consider the term∆1 as ∆1 = Ey∼π∗(·|x)[r∗(x, y)] − Ey∼πF (·|x)[r∗(x, y)] ≤ rmax[dT V(π∗(·|x), πF (·|x))], (26) where the upper bound follows from the definition of TV norm. Next, to bound the term∆∗ 2, we first observe that Ey∼πF (·|x)[r∗(x, y)] = Ex′∼ρsft(·|x),y∼πF (·|x)[r∗(x...

    2008

  45. [53]

    While this aspect certainly played a crucial role, it oversimplifies the broader economic and structural issues

    The Subprime Mortgage Crisis as the Sole Cause Misconception: The financial crisis of 2008 is often reduced to a narrative focused solely on the subprime mortgage market, which implies that it was primarily driven by reckless lending practices by American lenders to low-income...

    2008

  46. [54]

    These tools allowed financial institutions to shift risk off their balance sheets and increase leverage, ultimately contributing to instability

    Financial Innovation as Inherently Evil Misconception: Financial innovations such as securitization and derivatives are frequently portrayed as the root cause of the crisis. These tools allowed financial institutions to shift risk off their balance sheets and increase leverage...

    2008

  47. [55]

    Birdhouses and Animal Habitats:Smaller bottles can serve as habitats for birds or insects

    Craft Projects Planters and Terrariums:Plastic bottles can be transformed into plant pots for herbs, flowers, or small plants. Birdhouses and Animal Habitats:Smaller bottles can serve as habitats for birds or insects. Decorative Art:Create decorative objects like figurines, wi...

  48. [56]

    Garden Tools: Convert old bottles into garden markers, plant markers, or simple tools like a mini watering sprayer

    Upcycling in Gardening Watering Can Repurposing:Clean and cut the top off a plastic bottle, invert it to fit inside the body of the bottle, making a unique watering can with a built-in reservoir. Garden Tools: Convert old bottles into garden markers, plant markers, or simple t...

  49. [57]

    Covers and Protectors:Use them as covers for plants during winters or protect delicate surfaces in transit

    Homes and Storage Solutions Toolboxes and Organizers:Modify plastic bottles to create compartments for storing tools, hardware, or small items. Covers and Protectors:Use them as covers for plants during winters or protect delicate surfaces in transit. DIY Desk Accessories:Make...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.