Pith. sign in

REVIEW 3 major objections 7 minor 1 cited by

ACING: Actor-Critic for Instruction Learning in Black-Box LLMs

T0 review · 3 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ACING turns prompt discovery for black-box LLMs into a 165-query actor-critic search that beats human-written prompts on most instruction-induction tasks.

desk verdict A solid prompt-optimization paper with a useful off-policy actor-critic recipe, but the abstract oversells the headline numbers and the critic's added value over a no-critic ablation looks thin. read the letter →

arxiv 2411.12736 v2 pith:NZWTWYAS submitted 2024-11-19 cs.CL cs.AIcs.LGcs.SYeess.SYmath.OC

Salma Kharrat , Fares Fourati , Marco Canini This is my paper · ORCID
classification cs.CLcs.AIcs.LGcs.SYeess.SYmath.OC
keywords promptoptimizationblack-boxLLMactor-criticinstructioninductioncontinuousactionspacesoftmaximumentropyreinforcementlearningchain-of-thoughtprompting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ACING treats instruction optimization for black-box LLMs as a stateless, continuous-action reinforcement-learning problem: an action is a low-dimensional vector, projected into a soft-prompt embedding and decoded by a frozen white-box LLM into a discrete instruction, whose task score from the black-box LLM is the reward. The paper claims that an off-policy actor-critic with twin critics and entropy-regularized exploration discovers instructions that beat human-written prompts on 76% of instruction-induction tasks and outperform four automatic baselines on 33 tasks spanning instruction induction, summarization, and zero-shot chain-of-thought reasoning, all within a fixed 165-query budget. If this holds, prompt engineering for API-only models can be automated with lightweight networks and no gradient access.

What carries the argument

The load-bearing object is the projected action vector: an action $a\in[0,1]^{d'}$ with $d'=10$ is multiplied by a fixed random matrix $P$ with entries from $\mathrm{Uniform}(-1,1)$ to form a soft prompt $z=Pa$ in the embedding space of a frozen Vicuna-13B decoder, which turns $z$ and five exemplars into a discrete instruction $\tau$. A stateless off-policy actor-critic, consisting of a Gaussian policy network plus two critics whose value estimate is the minimum of the two, updates on the reward $r=\frac{1}{m}\sum_i q(\hat y_i,y_i)$ from the black-box LLM, with an adaptive entropy temperature to keep exploration alive under the 165-query budget. This machinery converts discrete prompt search into a continuum bandit and lets the method explore an infinite instruction space without model gradients.

What would settle it

Run ACING on the same 33 tasks with the 165-query budget but replace the critic with random search over the same action space; if random search matches ACING's median, the critic's learned generalization is not what drives the reported gains. A second check: sample pairs of actions within a small radius, decode them, and measure text overlap of the resulting instructions; low correlation would show the projection-decoder map is too discontinuous for the critic to learn from.

Watch

Extended reading notes

Core claim

The central claim is that a continuous latent prompt space, searched with soft actor-critic style updates, is a better substrate for black-box instruction optimization than discrete token search or finite candidate pools. ACING reports a median test score of 0.71 across 30 instruction-induction tasks, roughly 10 points above the best baseline median, winning 14 of 30 tasks overall and 13 of the 23 hardest tasks; it also leads on SAMSum summarization and on GSM8K and AQUA-RAT zero-shot reasoning. The discovered instructions remain readable, with a median Flesch Reading Ease of 70.8, and are selected by validation score alone before being evaluated on held-out test sets.

Load-bearing premise

The paper assumes that a random 10-dimensional projection into embedding space, decoded by a frozen Vicuna-13B, maps nearby actions to usefully related instructions; if that map is noisy or flat, the critic cannot generalize and the search degrades toward random sampling.

Editorial extensions

If this is right

  • Prompt optimization for API-only LLMs can be driven entirely by task scores, with no token-level access or gradient information.
  • The same untuned hyperparameters transfer across 33 tasks, suggesting ACING is a general-purpose optimizer rather than a per-task tuner.
  • Discovered prompts are naturally interpretable and readable, so the optimization output can be deployed or inspected directly.
  • Because rewards often peak within 60-80 of the 165 calls, the fixed budget can be shortened or split into exploration plus re-ranking to gain about 5 more points in median score.
  • Swapping the frozen decoder from Vicuna-13B to WizardLM-13B lifts median accuracy by 8 points, so the approach can ride on stronger open decoders as they appear.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same actor-critic loop could optimize not just the instruction text but also the exemplars or the decoding template, since both enter the same decoder pipeline.
  • If the critic's generalization across actions is the real driver, comparing ACING against a non-parametric baseline such as random search over the same action space would isolate how much of the reported 10-point median gain comes from the learned value function.
  • The random projection $P$ is a single draw; a testable extension is ensembling multiple projections or learning the projection to reduce the chance that a particular draw makes the action-to-prompt map uninformative.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper introduces ACING, an actor-critic framework for optimizing natural-language instructions for black-box LLMs. It encodes candidate instructions as continuous latent vectors, obtained by linearly projecting a low-dimensional action vector through a fixed random matrix and decoding to text with a frozen white-box model (Vicuna-13B). ACING trains a stateless off-policy actor with twin critics and entropy regularization, using the validation score of the black-box LLM as reward under a fixed 165-query budget. The paper reports results on 30 instruction-induction tasks, two zero-shot chain-of-thought datasets, and one summarization dataset, claiming that ACING outperforms human prompts in 76% of instruction-induction tasks and improves on automatic baselines by up to 33 points with a 10-point median gain. The appendices contain ablations on the critic, action dimensionality, exemplar count, budget splitting, and decoder choice, plus human and readability evaluations of the generated prompts.

Significance. If its headline claims were supported, ACING would be a useful contribution: it demonstrates a lightweight, API-compatible alternative to LLM-based prompt proposers, provides a broad evaluation across 33 datasets and several black-box models, and ships open source with detailed ablations. The framework itself is clearly specified and the experimental effort is substantial. However, the main quantitative claims as stated are not consistent with the paper's own tables, and the absence of a random-search control leaves the core attribution — that the actor-critic mechanism, rather than the continuous latent space and decoder, drives the improvements — unproven.

major comments (3)
  1. [Abstract; Section 4.1; Table 2; Table 19] The claim that ACING 'outperforms human-written prompts in 76% of instruction-induction tasks' is not supported by the reported comparisons. Table 2, the only direct human-vs-ACING table in the main text, shows ACING winning on 7 of 12 tasks where scores differ (58%). Table 19, the full comparison, shows ACING as best on 16 of 30 tasks (53%) if 'best-performing' is exclusive; no table in the paper yields 76%. The abstract also states 'gains of up to 33 points,' but Table 2 contains a larger 39-point gain (Rhymes, 0.61 to 1.00). Please specify the exact task set, the tie-breaking rule, and the reference baseline for these headline numbers.
  2. [Abstract; Section 4.2; Table 3; Table 9] The '10-point median improvement over the best automatic baseline' is computed on the 23-task subset of Table 3 (median 0.69 vs 0.59 for InstructZero). On the full 30-task instruction-induction set (Table 9), the median gain over the best baseline is 5 points (0.71 vs 0.66), and the CoT (Table 5) and summarization (Table 4) gains are at most 4 points. Thus the 10-point median claim across 33 tasks is not supported by the reported results; the authors should either report a combined 33-task comparison with a single aggregate median or revise the claim to refer only to the 23-task subset.
  3. [Section 3.3, Eq. (3); Section 4.2; Appendix F.1; Section 7 (Limitations)] The paper does not include a random-search baseline over the same continuous action space defined by the fixed projection matrix P and the frozen Vicuna decoder. Such a control is necessary to attribute the observed improvements to the off-policy actor-critic mechanism rather than to the latent-space representation and validation-based prompt selection. The paper's own Limitations section concedes that 'strong prompts can occasionally arise even from random search,' and Appendix F.1 (Table 12) shows that a pure policy-gradient without a critic reaches a median of 0.70 vs ACING's 0.71, with ACING winning on 21 of 33 tasks. Adding a random-search or random-action baseline with identical decoder, exemplars, budget, and selection rule would clarify whether the actor-critic learning itself contributes beyond the continuous latent space.
minor comments (7)
  1. [Section 4.1] The statement that 'ACING improves the median task score from 0.81 to 0.86' refers only to the 12-task subset of Table 2; please clarify that this is not the median over the full 30-task set.
  2. [Table 3; Table 9] The '# best-performing tasks' counts treat ties as wins (for example, Letters_list is tied between InstructZero and ACING); please state the tie-breaking convention explicitly.
  3. [Section 4.2; Statistical significance test] The Wilcoxon signed-rank test should state whether it uses per-seed scores or per-task means, whether the test is one-sided or two-sided, and the number of paired observations; currently these details are missing.
  4. [Appendix F.5; Table 16] The action-dimensionality results are reported for a fixed seed only; the text should note that these are single-seed observations and that variability across seeds may be substantial, especially given the large standard deviations seen elsewhere.
  5. [Table 19] The bottom rows report '# best-performing tasks 14 16' (summing to 30) while the table displays only 21 rows; please clarify whether the counts refer to the full 30-task set or to the displayed subset, and if the latter, explain the discrepancy.
  6. [Table 21 vs Table 9] For several tasks the best-instruction test score in Table 21 differs from the mean in Table 9 (e.g., auto_debugging 0.375 vs 0.25, sentence_similarity 0.21 vs 0.13); please explain how the 'best' instruction is selected and why these values differ.
  7. [Section 4; typos] There is a typo 'Moroever' at the start of Section 4; please correct it.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: ACING optimizes an external validation reward and tests on held-out data; missing random-search control is an evaluation gap, not a definitional reduction.

full rationale

The derivation is not circular. ACING's objective in Eq. (3) maximizes the black-box validation score q(y, f(τ(Pa)⊕x)), and the critic loss in Eq. (4) and actor objective in Eq. (6) are trained on that observed reward signal, not on the test metric. The final instruction is selected using the validation set V and then evaluated on a separate test set T (Section 3.3, after Step 4; Section 2.1). No fitted parameter is renamed as a prediction, and no equation reduces to its own input by construction. The random-projection plus frozen-decoder construction is an explicit design ansatz, and the paper ablates it (action dimensionality in Appendix F.5, decoder choice in Appendix F.7), which is the appropriate way to expose its assumptions. The paper also candidly concedes in Section 7 that strong prompts can occasionally arise even from random search, and the absence of a random-search control over the same latent space is a legitimate experimental-control concern, but that is a correctness/evidence issue, not circularity. The policy-gradient-without-critic ablation in Appendix F.1 (median 0.70 vs. 0.71) suggests the critic's contribution is incremental, but again this speaks to effect size and attribution, not to a definitional identity. The only self-citation, Fourati et al. (2024) in Section 7, is a future-research pointer and is not load-bearing in any argument for ACING's effectiveness. The central claim is therefore self-contained against external validation and test benchmarks, with no circular step identified.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a small number of hand-picked hyperparameters and on two unproven modeling assumptions: that the random projection plus frozen decoder creates a smooth action-to-instruction mapping, and that validation rewards generalize to test performance. The paper provides no new physical or conceptual entities.

free parameters (4)
  • Action dimension d' = 10
    Chosen to match prior work (InstructZero, INSTINCT) for fairness; ablations in Appendix F.5 show d'=40 yields more best-performing tasks, so the main result depends on this hand-picked value.
  • Number of soft tokens Nz = 5
    Set to 5 following prior work; ablations show sensitivity to this choice, so the performance is conditional on it.
  • Query budget T = 165
    Adopted for fair comparison; Appendix F.3 shows many tasks converge earlier, but the fixed budget is a hand-chosen experimental constraint, not an optimized quantity.
  • Entropy target Htarget = not reported
    The adaptive temperature in Eq. (7) depends on a target entropy whose value is not specified in the paper; this is a free choice that affects exploration.
assumptions (4)
  • ad hoc to paper Random projection P with Uniform(-1,1) entries maps low-dimensional actions to soft-prompt space while preserving enough signal for optimization.
    Invoked in Eq. (3) and Section 2.2; there is no theoretical guarantee, and ablations in Appendix F.5 show performance varies with the action dimension.
  • domain assumption The frozen white-box model h decodes any projected soft prompt plus exemplars into a coherent, task-relevant discrete instruction.
    Used throughout Section 3.3 and Figure 2; Appendix F.7 shows changing the decoder from Vicuna to WizardLM changes median scores, so the assumption is load-bearing.
  • domain assumption The validation set V is representative of the task distribution D, so maximizing average validation reward generalizes to the test set T.
    Standard supervised-learning assumption stated in Section 2.1, Eq. (1) and (2); if V is too small or unrepresentative, the optimized prompt may overfit.
  • domain assumption The black-box LLM f behaves as a fixed conditional distribution, and the empirical reward is an unbiased estimate of the expected score.
    Assumed implicitly in the critic loss (Eq. 4) and in the reward computation in Section 3.3; API stochasticity is acknowledged but not modeled.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ACING: Actor-Critic for Instruction Learning in Black-Box LLMs." pith.science (2026). https://pith.science/paper/NZWTWYAS

@misc{pith2026241112736,
  author       = {Pith},
  title        = {Pith review of: ACING: Actor-Critic for Instruction Learning in Black-Box LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NZWTWYAS}},
  note         = {Machine review of arXiv:2411.12736}
}
read the original abstract

The effectiveness of Large Language Models (LLMs) in solving tasks depends significantly on the quality of their instructions, which often require substantial human effort to craft. This underscores the need for automated instruction optimization. However, optimizing instructions is particularly challenging when working with black-box LLMs, where model parameters and gradients are inaccessible. We introduce ACING, an actor-critic reinforcement learning framework that formulates instruction optimization as a stateless, continuous-action problem, enabling exploration of infinite instruction spaces using only black-box feedback. ACING automatically discovers prompts that outperform human-written prompts in 76% of instruction-induction tasks, with gains of up to 33 points and a 10-point median improvement over the best automatic baseline in 33 tasks spanning instruction-induction, summarization, and chain-of-thought reasoning. Extensive ablations highlight its robustness and efficiency. An implementation of ACING is available at https://github.com/salmakh1/ACING.

Figures

Figures reproduced from arXiv: 2411.12736 by the authors.

Figure 1
Figure 1. Pipeline of ACING. At each iteration, a soft prompt and task exemplars are fed to the white-box model to generate an instruction. This instruction queries the black-box LLM, whose outputs are scored. The resulting score is returned to the agent as a reward, which is used to update its networks and adjust its policy. Both LLMs remain frozen throughout. an output yˆj = f(τ (z) ⊕ xj ). The output yˆj is then evaluated … view at source ↗
Figure 2
Figure 2. Illustration of the prompt generation and testing inside the environment using the larger_animal dataset as an example. architecture proves effective: the critic generalizes reward signals in prompt space, stabilizes updates by reducing reward variance, and enhances robust￾ness via a twin-critic setup to mitigate overesti￾mation. Empirically, this actor-critic formulation consistently outperforms no-critic baselines… view at source ↗
Figure 3
Figure 3. Ranking distributions across cognitive tasks for all algorithms, ordered increasing by median rank. Budget efficiency. While ACING uses a fixed 165-query budget for fair comparison with prior work, it often converges well before the budget is exhausted. As illustrated in Appendix F.3, many tasks reach optimal rewards within 60–80 queries, and some within 10–20, showing strong sample efficiency under constrained sett… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Reward plots for running ACING on various selected tasks, showing the highest achieved reward on the [PITH_FULL_IMAGE:figures/full_fig_p017_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Every Call is Precious: Global Optimization of Black-Box Functions with Unknown Lipschitz Constants

    cs.LG 2025-02 conditional novelty 7.0 of 10

    ECP is a no-regret global optimization algorithm for Lipschitz black-box functions with unknown constant, achieving minimax-optimal regret via an expanding acceptance region.

Reference graph

Works this paper leans on

61 extracted references · 37 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Eshaan Agarwal, Joykirat Singh, Vivek Dani, Raghav Magazine, Tanuja Ganu, and Akshay Nambi. 2024. Promptwizard: Task-aware prompt optimization framework. arXiv preprint arXiv:2405.18369

    arXiv 2024

  4. [4]

    Yekun Chai, Shuohuan Wang, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2022. https://aclanthology.org/2022.findings-emnlp.8 Clip-Tuning: Towards Derivative-free Prompt Learning with a Mixture of Rewards . In EMNLP

    work page 2022

  5. [5]

    Jiuhai Chen, Lichang Chen, Heng Huang, and Tianyi Zhou. 2023 a . https://arxiv.org/abs/2304.03262 When do you need Chain-of-Thought Prompting for ChatGPT? Preprint, arXiv:2304.03262

    arXiv 2023

  6. [6]

    Lichang Chen, Jiuhai Chen, Tom Goldstein, Heng Huang, and Tianyi Zhou. 2023 b . Instructzero: Efficient instruction optimization for black-box large language models. arXiv preprint arXiv:2306.03082

    arXiv 2023

  7. [7]

    Lichang Chen, Jiuhai Chen, Tom Goldstein, Heng Huang, and Tianyi Zhou. 2024. https://proceedings.mlr.press/v235/chen24e.html InstructZero: Efficient Instruction Optimization for Black-Box Large Language Models . In ICML

    work page 2024

  8. [8]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. https://lmsys.org/blog/2023-03-30-vicuna/ Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90\

    work page 2023

Show all 61 references
  1. [9]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    2021 arXiv

  2. [10]

    Meri Coleman and Ta Lin Liau. 1975. A computer readability formula designed for machine scoring. Journal of Applied Psychology, 60(2):283

    1975

  3. [11]

    Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric Xing, and Zhiting Hu. 2022. https://aclanthology.org/2022.emnlp-main.222 RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning . In EMNLP

    2022

  4. [12]

    Chrisantha Fernando, Dylan Sunil Banarse, Henryk Michalewski, Simon Osindero, and Tim Rockt \"a schel. 2024. Promptbreeder: Self-referential self-improvement via prompt evolution. In International Conference on Machine Learning, pages 13481--13544. PMLR

    2024

  5. [13]

    Rudolph Flesch. 1948. A new readability yardstick. Journal of applied psychology, 32(3):221

    1948

  6. [14]

    Fares Fourati, Vaneet Aggarwal, and Mohamed-Slim Alouini. 2024. https://proceedings.mlr.press/v235/fourati24a.html Stochastic Q-learning for Large Discrete Action Spaces . In ICML

    2024

  7. [15]

    Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. https://aclanthology.org/D19-5409 SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization . In Workshop on New Frontiers in Summarization

    2019

  8. [16]

    Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. 2024. https://openreview.net/forum?id=ZG3RaNIsO8 Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers . In ICLR

    2024

  9. [17]

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018. https://proceedings.mlr.press/v80/haarnoja18b.html Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor . In ICML

    2018

  10. [18]

    Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. 2019. https://arxiv.org/abs/1812.05905 Soft Actor-Critic Algorithms and Applications . Preprint, arXiv:1812.05905

    2019 arXiv

  11. [19]

    Bowman, and Omer Levy

    Or Honovich, Uri Shaham, Samuel R. Bowman, and Omer Levy. 2023. https://aclanthology.org/2023.acl-long.108 Instruction Induction: From Few Examples to Natural Language Task Descriptions . In ACL

    2023

  12. [20]

    Wenyang Hu, Yao Shu, Zongmin Yu, Zhaoxuan Wu, Xiangqiang Lin, Zhongxiang Dai, See-Kiong Ng, and Bryan Kian Hsiang Low. 2024. Localized zeroth-order prompt optimization. arXiv preprint arXiv:2403.02993

    2024 arXiv

  13. [21]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    2024 arXiv

  14. [22]

    J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel

    1975

  15. [23]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. 2017. https://arxiv.org/abs/1412.6980 Adam: A Method for Stochastic Optimization . Preprint, arXiv:1412.6980

    2017 arXiv

  16. [24]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Proc. NeurIPS, pages 22199--22213

    2022

  17. [25]

    Vijay Konda and John Tsitsiklis. 1999. Actor-critic algorithms. Advances in neural information processing systems, 12

    1999

  18. [26]

    Minchan Kwon, Gaeun Kim, Jongsuk Kim, Haeil Lee, and Junmo Kim. 2024. Stableprompt: Automatic prompt tuning using reinforcement learning for large language models. arXiv preprint arXiv:2410.07652

    2024 arXiv

  19. [27]

    Tor Lattimore and Csaba Szepesvári. 2020. https://doi.org/10.1017/9781108571401 Bandit Algorithms . Cambridge University Press

    2020 doi

  20. [28]

    Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. https://aclanthology.org/2021.emnlp-main.243 The Power of Scale for Parameter-Efficient Prompt Tuning . In EMNLP

    2021

  21. [29]

    Xiang Lisa Li and Percy Liang. 2021. https://aclanthology.org/2021.acl-long.353 Prefix-Tuning: Optimizing Continuous Prompts for Generation . In ACL-IJCNLP

    2021

  22. [30]

    Chin-Yew Lin. 2004. ROUGE : A package for automatic evaluation of summaries. In Text summarization branches out, pages 74--81

    2004

  23. [31]

    Xiaoqiang Lin, Zhongxiang Dai, Arun Verma, See-Kiong Ng, Patrick Jaillet, and Bryan Kian Hsiang Low. 2024 a . Prompt optimization with human feedback. arXiv preprint arXiv:2405.17346

    2024 arXiv

  24. [32]

    Xiaoqiang Lin, Zhaoxuan Wu, Zhongxiang Dai, Wenyang Hu, Yao Shu, See-Kiong Ng, Patrick Jaillet, and Bryan Kian Hsiang Low. 2024 b . https://proceedings.mlr.press/v235/lin24r.html Use Your INSTINCT: INSTruction optimization for LLMs usIng Neural bandits Coupled with Transformer...

    2024

  25. [33]

    Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proc. Annual Meeting of the ACL, pages 158--167

    2017

  26. [34]

    Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. https://doi.org/10.1145/3560815 Pre-Train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing . ACM Computing Surveys, 55(9)

    2023 doi

  27. [35]

    Swaroop Mishra, Daniel Khashabi, Chitta Baral, Yejin Choi, and Hannaneh Hajishirzi. 2021. https://aclanthology.org/2022.findings-acl.50 Reframing Instructional Prompts to GPTk's Language . In ACL

    2021

  28. [36]

    Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. https://proceedings.mlr.press/v48/mniha16.html Asynchronous Methods for Deep Reinforcement Learning . In ICML

    2016

  29. [37]

    OpenAI. 2023 a . ChatGPT . https://chat.openai.com

    2023

  30. [38]

    OpenAI. 2023 b . https://openai.com/chatgpt ChatGPT: A Conversational AI Model

    2023

  31. [39]

    OpenAI. 2023 c . https://arxiv.org/abs/2303.08774 GPT-4 Technical Report . Preprint, arXiv:2303.08774

    2023 arXiv

  32. [40]

    Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. 2023. https://aclanthology.org/2023.emnlp-main.494 Automatic Prompt Optimization with ``Gradient Descent'' and Beam Search . In EMNLP

    2023

  33. [41]

    Yunzhe Qi, Jinjin Tian, Ruirui Li, Tianci Liu, Tianxin Wei, Hui Liu, Xianfeng Tang, Monica Xiao Cheng, and Jingrui He. 2025. https://openreview.net/forum?id=m0ddLnNvXS Automatic task-aware instruction optimizer for black-box llms

    2025

  34. [42]

    Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. 2018. https://openreview.net/forum?id=ryQu7f-RZ On the Convergence of Adam and Beyond . In ICLR

    2018

  35. [43]

    Laria Reynolds and Kyle McDonell. 2021. https://doi.org/10.1145/3411763.3451760 Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm . In CHI EA

    2021

  36. [44]

    Chengshuai Shi, Kun Yang, Jing Yang, and Cong Shen. 2024. https://openreview.net/forum?id=q4mtFqw8PR Best Arm Identification for Prompt Learning under a Limited Budget . In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models

    2024

  37. [45]

    Weijia Shi, Xiaochuang Han, Hila Gonen, Ari Holtzman, Yulia Tsvetkov, and Luke Zettlemoyer. 2023. https://aclanthology.org/2023.findings-emnlp.733 Toward Human Readable Prompt Tuning: Kubrick's The Shining is a good movie, and a good prompt too? In EMNLP

    2023

  38. [46]

    Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. 2020. https://aclanthology.org/2020.emnlp-main.346 AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts . In EMNLP

    2020

  39. [47]

    Aleksandrs Slivkins. 2019. http://dx.doi.org/10.1561/2200000068 Introduction to Multi-Armed Bandits . Foundations and Trends in Machine Learning , 12(1-2)

    2019 doi

  40. [48]

    Tianxiang Sun, Zhengfu He, Hong Qian, Xuanjing Huang, and Xipeng Qiu. 2022 a . https://aclanthology.org/2022.emnlp-main.259 BBTv2: Towards a Gradient-Free Future with Large Language Models . In EMNLP

    2022

  41. [49]

    Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. 2022 b . https://proceedings.mlr.press/v162/sun22e.html Black-Box Tuning for Language-Model-as-a-Service . In ICML

    2022

  42. [50]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. https://arxiv.org/abs/2302.13971 LLaMA:...

    2023 arXiv

  43. [51]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models . In NeurIPS

    2022

  44. [52]

    Frank Wilcoxon. 1992. Individual comparisons by ranking methods. In Breakthroughs in statistics: Methodology and distribution, pages 196--202. Springer

    1992

  45. [53]

    Zhaoxuan Wu, Xiaoqiang Lin, Zhongxiang Dai, Wenyang Hu, Yao Shu, See-Kiong Ng, Patrick Jaillet, and Bryan Kian Hsiang Low. 2024. Prompt optimization with ease? efficient ordering-aware automated selection of exemplars. arXiv preprint arXiv:2405.16122

    2024 arXiv

  46. [54]

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2024. https://openreview.net/forum?id=CfXh93NDgH WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex Instructions . In ICLR

    2024

  47. [55]

    Weihao Zeng, Can Xu, Yingxiu Zhao, Jian-Guang Lou, and Weizhu Chen. 2024. Automatic instruction evolving for large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6998--7018

    2024

  48. [56]

    Tianjun Zhang, Xuezhi Wang, Denny Zhou, Dale Schuurmans, and Joseph E Gonzalez. 2023. https://openreview.net/forum?id=gSHyqBijPFO TEMPERA: Test-Time Prompt Editing via Reinforcement Learning . In ICLR

    2023

  49. [57]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, and 3 others. 2024. https://arxiv.org/...

    2024 arXiv

  50. [58]

    Zexuan Zhong, Dan Friedman, and Danqi Chen. 2021. https://aclanthology.org/2021.naacl-main.398 Factual Probing Is [MASK]: Learning vs. Learning to Recall . In NAACL

    2021

  51. [59]

    Dongruo Zhou, Lihong Li, and Quanquan Gu. 2020. https://proceedings.mlr.press/v119/zhou20a.html Neural Contextual Bandits with UCB-based Exploration . In ICML

    2020

  52. [60]

    Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023. https://openreview.net/forum?id=92gvk82DE- Large Language Models are Human-Level Prompt Engineers . In ICLR

    2023

  53. [61]

    Brian D Ziebart. 2010. https://doi.org/10.1184/R1/6720692.v1 Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy . Ph.D. thesis, Carnegie Mellon University

    2010 doi

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.