Pith. sign in

REVIEW 2 major objections 7 minor 52 references

Grammar-Guided Evolutionary Search for Discrete Prompt Optimisation

T0 review · 2 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Grammar-guided evolutionary search reliably optimises long, detailed prompts for small language models, outperforming three leading prompt optimisers that often degrade the same task-model combinations.

desk verdict A promising but under-verified empirical recipe for prompt optimization on small LLMs: the method is genuinely novel in combination, but the superiority claim rests on unquantified budget matching and single test-set numbers. read the letter →

arxiv 2507.10326 v1 pith:VNFKFKML submitted 2025-07-14 cs.CL

Muzhaffar Hazman , Minh-Khoi Pham , Shweta Soundararajan , Goncalo Mordido , Leonardo Custode , David Lynch , Giorgio Cruciata , Yucheng Shi
show 5 more authors
Hongmeng Song Wang Chao Pan Yue Aleksandar Milenovic Alexandros Agapitos
This is my paper
classification cs.CL
keywords discretepromptoptimisationgrammar-guidedgeneticprogrammingevolutionarysearchlocalsurrogatemodelsmalllanguagemodelssensitivity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models become more sensitive to prompt wording as they get smaller, and the automated prompt optimisers that work on big models often collapse on small models and on long, detail-heavy prompts. This paper argues that the fix is to stop asking the model to edit its own prompt and instead run a grammar-guided evolutionary search over small, discrete edits to a manually written prompt, evolving the edit program rather than the prompt text. Across four domain-specific tasks and three small models (Llama3.2 3B, Llama3 8B, Gemma2 9B), the method improves on the baseline prompt in 10 of 12 task-model pairs, and a follow-up local search phase makes it the best or second-best method in all 12. If this holds, practitioners can deploy small, cheap models on specialised tasks without fine-tuning and without the performance cliff that current prompt-automation tools impose on them.

What carries the argument

The central machinery is a Backus–Naur-form grammar whose productions compose prompt-editing functions into executable programs, a setup the paper calls G3P DPO. Each individual in the population is an integer genotype that maps through the grammar to a phenotype: a set of edit sequences, one per prompt section, each a tree of list-manipulation and semantically aligned operations (LLM-based paraphrase and summarise, dictionary-based stop-word removal and synonym substitution) at word, sentence, or phrase chunking levels. Evolution runs 20 generations of 50 individuals with tournament selection, subtree crossover, and subtree mutation, evaluating fitness on 20 training rows per generation; a post-hoc local search then perturbs each index parameter of the best program, screens the resulting neighbours with a 10-model neural ensemble surrogate trained on Sentence-BERT embeddings of prompts, and real-evaluates the 25 highest-mean and 25 highest-variance neighbours.

What would settle it

Rerun PromptWizard, OPRO, and RL-Prompt on the same 12 task-model combinations with the total number of LLM evaluation calls equalised to the G3P + Local Search budget (20 generations x 50 individuals evaluated on 20 training rows, plus 50 local-search evaluations), reporting step counts and wall-clock time; if any baseline reaches or exceeds the +56% mean relative gain under matched calls, the claimed superiority would be shown to reflect budget allocation rather than search design.

Watch

Extended reading notes

Core claim

The paper claims that discrete prompt optimisation for small LLMs should be treated as a program-synthesis problem, not a text-editing problem. Its G3P approach evolves sequences of edit operations — swap, remove, re-add, duplicate, paraphrase, summarise, stop-word removal, and synonym substitution — applied at word, sentence, or phrase granularity to six modular sections of a base prompt (persona, task, output format, in-context demonstrations, context, chain-of-thought). A formal grammar restricts the space so every candidate is a syntactically valid prompt-producing program, and fitness is the target model's accuracy on a sample of training rows. The authors report a mean relative gain of +56% over the base prompts (against +5% for PromptWizard, −12% for OPRO, and −42% for RL-Prompt), best-or-second-best results in all 12 task-model combinations, and a narrowing of the performance gap between the weakest and strongest models from 35.6 points to 10.5, because the weakest model gains the most. They also document specific failure modes of the baselines — PromptWizard injecting a single training question into the instruction, OPRO rewriting class labels into invalid values, RL-Prompt producing near-zero performance — and argue that evolutionary selection and elitism naturally filter out such destructive edits.

Load-bearing premise

The headline comparison assumes the baselines ran on a search budget comparable to G3P's 20 generations of 50 individuals, but the paper caps OPRO and RL-Prompt at 10 training rows per step without reporting step counts, total LLM calls, or runtime, and it concedes in Section 5.2 that RL-Prompt's reduced steps were possibly too few.

Editorial extensions

If this is right

  • Practitioners can optimise long, detail-heavy prompts for small open-source models without fine-tuning, with the largest relative gains landing on the weakest model, such as Llama3 8B rising from 19.6% to 33.5% on ConvFinQA.
  • The inter-model performance spread across the four tasks drops from 35.6 points on the base prompts to 14.4 with G3P and 10.5 after local search, so the choice among small models matters less once prompts are optimised this way.
  • Because the grammar makes persona, output-format, and chain-of-thought sections optional, the search itself decides which sections a small model actually needs; 7 of the 12 optimised prompts left at least one section untouched.
  • The surrogate-assisted local search adds a further improvement in 11 of 12 task-model combinations beyond the evolved individual, meaning the local refinement phase is complementary rather than redundant to the evolutionary search.
  • The baselines' documented failures — unparseable outputs, invalid labels, and training-set artefacts copied into instructions — show that LLM-self-editing optimisers do not transfer safely to the small-model, long-prompt regime.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The grammar's modular section structure could double as an automatic ablation tool: which sections the evolved program keeps, edits, or drops gives a per-task diagnosis of what a small model actually needs in its prompt.
  • A natural extension is to seed the population with the baselines' own best outputs, or to add the information-injecting operations the paper lists as missing (instruction induction, self-reflection, phrase injection), combining content discovery with G3P's selection pressure.
  • Because the surrogate is trained only on prompts from one G3P run, its cost-saving benefit should be tested by scaling generations and checking whether the ensemble's screening accuracy keeps pace with the growing neighbourhood.
  • The claim that selection filters destructive edits could be probed directly by counting how many early-generation low-fitness candidates exhibit the same failure modes documented for the baselines and confirming they are eliminated by selection rather than never generated.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 7 minor

Summary. The paper proposes G3P DPO, a two-phase discrete prompt optimisation method aimed at smaller general-purpose LLMs on long, domain-specific prompts. Phase 1 uses grammar-guided genetic programming to evolve sequences of list-manipulation, dictionary-based, and LLM-based edit operations over six prompt sections; phase 2 applies surrogate-screened local search to the best evolved prompt. The method is evaluated on PubMedQA, ETHOS, TAT-QA, and ConvFinQA with Llama3.2 3B, Llama3 8B, and Gemma2 9B, and compared against PromptWizard, OPRO, and RL-Prompt. The authors report that G3P plus local search is the best or second-best optimisation method in all 12 task-model combinations, with a mean relative gain over the baseline prompt of +56%, versus +5%, -12%, and -42% for the three baselines.

Significance. If the empirical claims are supported, the paper makes a useful contribution. It addresses a genuine gap: most discrete prompt optimisation work assumes very large LLMs and short prompt templates, while this work targets smaller models on long, structured, domain-specific prompts. The use of grammar-constrained search, external task metrics as fitness signals, and a surrogate model to prune local-search candidates are sensible design choices, and the appendices provide useful detail on the grammar, hyperparameters, and baseline prompts. The qualitative failure-mode analysis of the baselines in Section 5.2 is also informative. However, the central contribution is a comparative claim, and the current evidence is weakened by two load-bearing issues: the baseline compute budgets are asserted but never quantified, and all results are single test-set numbers from a stochastic procedure with no repeated seeds or confidence intervals. These issues should be addressed before the comparative claims can be accepted.

major comments (2)
  1. [Sec. 4.2 and Table 4] The claim that OPRO and RL-Prompt were run at matched compute is not substantiated. Section 4.2 says these baselines were limited to 10 training rows per step 'to match that of our approach', and that RL-Prompt used 10 candidate prompts per step, but no step counts, total numbers of LLM forward passes, total numbers of LLM edit calls, or runtimes are reported for any method. G3P's own budget is substantial: 20 generations with population and offspring sizes of 50, 20 training rows per evaluated individual, one validation evaluation per generation, surrogate-model training, and a local-search phase that really evaluates 50 neighbours on both validation and training data. In addition, every phenotype containing paraphrase or summarise operations consumes LLM calls before fitness is measured. Section 5.2 explicitly concedes that RL-Prompt was run with fewer steps than its authors recommend. Because the headline result in Table 4 is a comparison of search methods, the evaluation budgets must be quantified and actually matched; as written, the reported superiority could be an artefact of compute disparity rather than search design. For PromptWizard, no budget information is reported at all beyond 'default settings'.
  2. [Table 4, Sec. 3.3, Sec. 3.4] All test results are single numbers from a single run, although the search procedure is stochastic in several places: training rows are resampled every generation, the initial population is generated with PTC2, local-search neighbours are constructed from randomly selected index perturbations, and the surrogate ensemble uses bootstrap sampling. Figures 3 and Appendix G plot the mean and standard deviation across individuals within one run, not across repeated runs. Several headline differences are small: for ETHOS with Gemma2 9B, G3P plus local search gives 83.7 versus 83.5 for RL-Prompt and 83.0 for the baseline prompt; for TAT-QA with Gemma2 9B, G3P plus local search gives 40.6 versus 40.5 for the baseline prompt. Without repeated seeds, confidence intervals, or a paired significance test, the statements that G3P plus local search is 'best or second-best for all combinations' and that the approach 'outperforms' three baselines are not quantitatively supported. This is a load-bearing issue because the paper's central claim is comparative and the observed margins include cases within plausible run-to-run noise.
minor comments (7)
  1. [Abstract and Sec. 5.1] The abstract's 'outperforms three state-of-the-art prompt optimisation approaches' is stronger than Table 4, where PromptWizard achieves the highest score in 3 of 12 task-model combinations (PubMedQA with Llama3 8B, ETHOS with Llama3.2 3B, and TAT-QA with Llama3 8B); please qualify the claim as an average or overall comparison.
  2. [Table 4 and Sec. 5.1] For ConvFinQA with Gemma2 9B, G3P plus local search (54.5) remains 12% below the baseline prompt (61.9); the abstract's 'only incurring minimal degradation' and the 'best or second-best' framing should state explicitly that the ranking is among optimisation methods and should discuss this case.
  3. [Sec. 3.4] The phrase 'sorrugated ensemble' appears to be a typo for 'surrogate ensemble'.
  4. [Appendix C] The production for ⟨cot_expr⟩ has mismatched angle brackets (given as ⟨cot_expr>⟩), and several terminal strings in the grammar are sentiment-classification specific despite the grammar being presented as general; please clarify whether these are illustrative placeholders.
  5. [Appendix G and Fig. 3] The x-axis in the training-fitness plots ends at generation 15, while Appendix D specifies 20 generations; if the plots are truncated or generations are indexed from 0 to 19 with a display quirk, this should be explained.
  6. [Sec. 4.1] The phrase 'syntactic comparison to ground-truth labels' is potentially misleading for TAT-QA and ConvFinQA, where evaluation requires executing or parsing formulas; please clarify how the metrics are computed for these tasks.
  7. [Reproducibility] The paper does not state code or data availability and does not list the random seeds used; given the many hyperparameters in Appendices D and E, releasing code and seed configurations would materially improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: external task fitness, held-out test evaluation, and independent baselines; budget-parity concerns are experimental fairness, not derivation circularity.

full rationale

The paper's derivation chain is self-contained. The fitness signal is an external task metric defined by each dataset's authors: 'The performance of a prompt on a given task is measured using syntactic comparison to ground-truth labels provided in the dataset,' which eliminates LLM self-scoring. G3P selects individuals on training fitness and generation-level validation fitness, and the final prompt is evaluated once on a held-out test set, so no parameter is fitted to test results. In local search, the surrogate model only ranks neighbours; the retained neighbour is selected by real LLM evaluation on the validation set plus a training sample, so the surrogate is not the source of the final comparison. The baselines (PromptWizard, OPRO, RL-Prompt) are independent published algorithms run with their own settings, not derived from G3P's outputs. The paper's admission that RL-Prompt 'was possibly too few to effectively apply the approach' and its unquantified claim that LLM-based evaluations were 'limit[ed] ... to match that of our approach' are legitimate experimental-fairness concerns that could affect the strength of the empirical comparison, but they do not make any claimed result equivalent to its inputs by construction. The only self-citation, reference [29] on grammar design, is background and not load-bearing. No self-definitional, fitted-input-as-prediction, or imported-uniqueness pattern is present.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The method is empirical; no constants are derived. The listed parameters are search hyperparameters chosen by hand, not fitted to test data. The axioms are standard supervised-evaluation assumptions plus design choices specific to the grammar and surrogate.

free parameters (6)
  • Number of generations = 20
    Fixed search budget chosen in App. D; affects final prompt quality and not fitted to test data.
  • Population size / offspring population size = 50 / 50
    Chosen in App. D; larger populations would likely improve results but increase compute.
  • Training rows sampled per generation = 20
    Resampled each generation; fitness estimates are noisy compared to full training sets.
  • Number of neighbours per index in local search = 10
    Enumeration creates 10 unique neighbours per index; selection keeps top 25 mean and top 25 variance.
  • Surrogate ensemble hyperparameters = layer sizes, dropout, batch size, lr selected via 5-fold CV
    Selected by hyperparameter testing on G3P-collected data; not fitted to test results.
  • LLM decoding parameters = temperature 0.0, max new tokens 2048
    Deterministic sampling, no quantization; fixed across all methods.
assumptions (4)
  • domain assumption Fitness measured on a random sample of the training set is a reliable proxy for performance on the validation/test distribution.
    Used throughout Sec. 3.3; if the 20-row samples are unrepresentative, elite selection is misled.
  • domain assumption The lexical/syntactic edit operations and the six-section prompt decomposition span a useful region of prompt space.
    The grammar in Sec. 3.2 and Appendix C restricts search to these operations; useful edits outside this space cannot be found.
  • domain assumption The surrogate model's predicted fitness correlates with true fitness for ranking neighbours.
    Local search (Sec. 3.4) selects 50 neighbours via surrogate mean/variance before real evaluation; poor ranking would degrade local search.
  • domain assumption Ground-truth labels and syntactic comparison are correct and sufficient for evaluating prompt quality.
    Sec. 4.1 states performance is measured by syntactic comparison to ground-truth labels; any formatting mismatch counts as error.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Grammar-Guided Evolutionary Search for Discrete Prompt Optimisation." pith.science (2026). https://pith.science/paper/VNFKFKML

@misc{pith2026250710326,
  author       = {Pith},
  title        = {Pith review of: Grammar-Guided Evolutionary Search for Discrete Prompt Optimisation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VNFKFKML}},
  note         = {Machine review of arXiv:2507.10326}
}
read the original abstract

Prompt engineering has proven to be a crucial step in leveraging pretrained large language models (LLMs) in solving various real-world tasks. Numerous solutions have been proposed that seek to automate prompt engineering by using the model itself to edit prompts. However, the majority of state-of-the-art approaches are evaluated on tasks that require minimal prompt templates and on very large and highly capable LLMs. In contrast, solving complex tasks that require detailed information to be included in the prompt increases the amount of text that needs to be optimised. Furthermore, smaller models have been shown to be more sensitive to prompt design. To address these challenges, we propose an evolutionary search approach to automated discrete prompt optimisation consisting of two phases. In the first phase, grammar-guided genetic programming is invoked to synthesise prompt-creating programmes by searching the space of programmes populated by function compositions of syntactic, dictionary-based and LLM-based prompt-editing functions. In the second phase, local search is applied to explore the neighbourhoods of best-performing programmes in an attempt to further fine-tune their performance. Our approach outperforms three state-of-the-art prompt optimisation approaches, PromptWizard, OPRO, and RL-Prompt, on three relatively small general-purpose LLMs in four domain-specific challenging tasks. We also illustrate several examples where these benchmark methods suffer relatively severe performance degradation, while our approach improves performance in almost all task-model combinations, only incurring minimal degradation when it does not.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

52 extracted references · 33 canonical work pages

  1. [1]

    Agarwal, J

    E. Agarwal, J. Singh, V . Dani, R. Magazine, T. Ganu, and A. Nambi. Promptwizard: Task-aware prompt optimization framework. arXiv preprint arXiv:2405.18369, 2024

    arXiv 2024

  2. [2]

    T. Bäck, D. B. Fogel, and Z. Michalewicz. Handbook of evolutionary computation. Release, 97(1):B1, 1997

    work page 1997

  3. [3]

    K. Bikov. Financial data analysis with fine-tuned large language models,

    work page

  4. [4]

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Am...

    work page 2020

  5. [5]

    K. Chen, X. Li, T. Yang, H. Wang, W. Dong, and Y . Gao. Mdteamgpt: A self-evolving llm-based multi-agent framework for multi-disciplinary team medical consultation, 2025

    work page 2025

  6. [6]

    Z. Chen, S. Li, C. Smiley, Z. Ma, S. Shah, and W. Y . Wang. Convfinqa: Exploring the chain of numerical reasoning in conversational finance question answering. arXiv preprint arXiv:2210.03849, 2022

    arXiv 2022

  7. [7]

    W. Cui, J. Zhang, Z. Li, H. Sun, D. Lopez, K. Das, B. A. Malin, and S. Kumar. Phaseevo: Towards unified long-context prompt optimiza- tion for large language models. In First Workshop on Long-Context F oundation Models@ ICML 2024, 2024

    work page 2024

  8. [8]

    M. Deng, J. Wang, C.-P. Hsieh, Y . Wang, H. Guo, T. Shu, M. Song, E. Xing, and Z. Hu. RLPrompt: Optimizing discrete text prompts with reinforcement learning. In Y . Goldberg, Z. Kozareva, and Y . Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3369–3391, Abu Dhabi, United Arab Emirates, Dec. 20...

    doi:10.18653/v1/2022.emnlp-main.222 2022

Show all 52 references
  1. [9]

    Y . Dong, K. Luo, X. Jiang, Z. Jin, and G. Li. Pace: Improving prompt with actor-critic editing for large language model. arXiv preprint arXiv:2308.10088, 2023

    2023 arXiv

  2. [10]

    Y . Fei, Y . Hou, Z. Chen, and A. Bosselut. Mitigating label biases for in-context learning, 2023

    2023

  3. [11]

    Fernando, D

    C. Fernando, D. Banarse, H. Michalewski, S. Osindero, and T. Rock- täschel. Promptbreeder: Self-referential self-improvement via prompt evolution, 2023

    2023

  4. [12]

    L. Giray. Prompt engineering with chatgpt: a guide for academic writers. Annals of biomedical engineering , 51(12):2629–2633, 2023

    2023

  5. [13]

    Q. Guo, R. Wang, J. Guo, B. Li, K. Song, X. Tan, G. Liu, J. Bian, and Y . Yang. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers, 2024

    2024

  6. [14]

    Z. Guo, P. Wang, Y . Wang, and S. Yu. Improving small language models on pubmedqa via generative data augmentation, 2023

    2023

  7. [15]

    Hsieh, S

    C.-J. Hsieh, S. Si, F. X. Yu, and I. S. Dhillon. Automatic engineering of long prompts. arXiv preprint arXiv:2311.10117, 2023

    2023 arXiv

  8. [16]

    Q. Jin, B. Dhingra, Z. Liu, W. W. Cohen, and X. Lu. Pubmedqa: A dataset for biomedical research question answering. arXiv preprint arXiv:1909.06146, 2019

    1909 arXiv

  9. [17]

    M. R. Kabir, R. M. Sultan, I. H. Asif, J. Ibn Ahad, F. Rahman, M. R. Amin, N. Mohammed, and S. Rahman. Beyond labels: Aligning large language models with human-like reasoning. arXiv preprint arXiv:2408.11879, 2024

    2024 arXiv

  10. [18]

    Kojima, S

    T. Kojima, S. S. Gu, M. Reid, Y . Matsuo, and Y . Iwasawa. Large language models are zero-shot reasoners, 2023

    2023

  11. [19]

    W. Kong, S. Hombaiah, M. Zhang, Q. Mei, and M. Bendersky. PRewrite: Prompt rewriting with reinforcement learning. In L.-W. Ku, A. Mar- tins, and V . Srikumar, editors,Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V olume 2: Short Pa- ...

    2024

  12. [20]

    J. R. Koza. Genetic programming - on the programming of computers by means of natural selection . Complex adaptive systems. MIT Press, 1993

    1993

  13. [22]

    Y . Li, X. Yue, Z. Xu, F. Jiang, L. Niu, B. Y . Lin, B. Ramasubramanian, and R. Poovendran. Small models struggle to learn from strong reasoners, 2025

    2025

  14. [23]

    Y . B. Li and K. Wu. Spell: Semantic prompt evolution based on a llm, 2023

    2023

  15. [24]

    Z. Li, B. Peng, P. He, M. Galley, J. Gao, and X. Yan. Guiding large language models via directional stimulus prompting. Advances in Neural Information Processing Systems, 36:62630–62656, 2023

    2023

  16. [25]

    Y . Lu, M. Bartolo, A. Moore, S. Riedel, and P. Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In S. Muresan, P. Nakov, and A. Villavicencio, ed- itors, Proceedings of the 60th Annual Meeting of the Association fo...

    2022 doi

  17. [26]

    S. Luke. Two fast tree-creation algorithms for genetic programming. IEEE Transactions on Evolutionary Computation , 4(3):274–283, 2000

    2000

  18. [27]

    S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Y . Goldberg, Z. Kozareva, and Y . Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Na...

    2022 doi

  19. [28]

    Mollas, Z

    I. Mollas, Z. Chrysopoulou, S. Karlos, and G. Tsoumakas. Ethos: a multi- label hate speech detection dataset. Complex & Intelligent Systems , 8(6): 4663–4678, 2022

    2022

  20. [29]

    Nicolau and A

    M. Nicolau and A. Agapitos. Understanding grammatical evolution: Grammar design. In C. Ryan, M. O’Neill, and J. J. Collins, editors, Handbook of Grammatical Evolution , pages 23–53. Springer, 2018

    2018

  21. [30]

    R. Pan, S. Xing, S. Diao, W. Sun, X. Liu, K. Shum, J. Zhang, R. Pi, and T. Zhang. Plum: Prompt learning using metaheuristics. In L.-W. Ku, A. Martins, and V . Srikumar, editors,Findings of the Association for Computational Linguistics ACL 2024 , pages 2177–2197, Bangkok, Thail...

    2024

  22. [31]

    D. Paul, M. Ismayilzada, M. Peyrard, B. Borges, A. Bosselut, R. West, and B. Faltings. Refiner: Reasoning feedback on intermediate represen- tations. arXiv preprint arXiv:2304.01904, 2023

    2023 arXiv

  23. [32]

    Prasad, P

    A. Prasad, P. Hase, X. Zhou, and M. Bansal. Grips: Gradient-free, edit- based instruction search for prompting large language models. arXiv preprint arXiv:2203.07281, 2022

    2022 arXiv

  24. [33]

    gradient descent

    R. Pryzant, D. Iter, J. Li, Y . Lee, C. Zhu, and M. Zeng. Automatic prompt optimization with “gradient descent” and beam search. In H. Bouamor, J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 7957–79...

    2023 doi

  25. [34]

    Ramnath, K

    K. Ramnath, K. Zhou, S. Guan, S. S. Mishra, X. Qi, Z. Shen, S. Wang, S. Woo, S. Jeoung, Y . Wang, H. Wang, H. Ding, Y . Lu, Z. Xu, Y . Zhou, B. Srinivasan, Q. Yan, Y . Chen, H. Ding, P. Xu, and L. L. Cheong. A systematic survey of automatic prompt optimization techniques, 2025

    2025

  26. [35]

    Sahoo, A

    P. Sahoo, A. K. Singh, S. Saha, V . Jain, S. Mondal, and A. Chadha. A systematic survey of prompt engineering in large language models: Techniques and applications, 2024

    2024

  27. [36]

    Schnabel and J

    T. Schnabel and J. Neville. Symbolic prompt program search: A structure- aware approach to efficient compile-time prompt optimization, 2024

    2024

  28. [37]

    Sclar, Y

    M. Sclar, Y . Choi, Y . Tsvetkov, and A. Suhr. Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting, 2024

    2024

  29. [38]

    Singhal, T

    K. Singhal, T. Tu, J. Gottweis, R. Sayres, E. Wulczyn, M. Amin, L. Hou, K. Clark, S. R. Pfohl, H. Cole-Lewis, D. Neal, Q. M. Rashid, M. Schaek- ermann, A. Wang, D. Dash, J. H. Chen, N. H. Shah, S. Lachgar, P. A. Mansfield, S. Prakash, B. Green, E. Dominowska, B. Agüera y Arcas...

  30. [39]

    X. Wang, C. Li, Z. Wang, F. Bai, H. Luo, J. Zhang, N. Jojic, E. P. Xing, and Z. Hu. PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization. arXiv preprint arXiv:2310.16427, 2023

    2023 arXiv

  31. [40]

    doi: 10.1038/s41591-024-03423-7

    doi

  32. [41]

    P. A. Whigham et al. Grammatically-based genetic programming. In Proceedings of the workshop on genetic programming: from theory to real-world applications, volume 16, pages 33–41. Citeseer, 1995

    1995

  33. [42]

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023

    2023

  34. [43]

    C. Yang, X. Wang, Y . Lu, H. Liu, Q. V . Le, D. Zhou, and X. Chen. Large language models as optimizers. In The Twelfth International Conference on Learning Representations, 2024

    2024

  35. [44]

    H. Xu, Y . Chen, Y . Du, N. Shao, W. Yanggang, H. Li, and Z. Yang. GPS: Genetic prompt search for efficient few-shot learning. In Y . Goldberg, Z. Kozareva, and Y . Zhang, editors,Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 81...

    2022 doi

  36. [45]

    Zhang, X

    T. Zhang, X. Wang, D. Zhou, D. Schuurmans, and J. E. Gonzalez. Tem- pera: Test-time prompting via reinforcement learning. arXiv preprint arXiv:2211.11890, 2022

    2022 arXiv

  37. [46]

    Zhang, T

    L. Zhang, T. Ergen, L. Logeswaran, M. Lee, and D. Jurgens. Sprig: Improving large language model performance by system prompt opti- mization. arXiv preprint arXiv:2410.14826, 2024

    2024 arXiv

  38. [47]

    F. Zhu, W. Lei, Y . Huang, C. Wang, S. Zhang, J. Lv, F. Feng, and T.-S. Chua. Tat-qa: A question answering benchmark on a hybrid of tabular and textual content in finance. arXiv preprint arXiv:2105.07624, 2021

    2021 arXiv

  39. [48]

    Zheng, H

    C. Zheng, H. Zhou, F. Meng, J. Zhou, and M. Huang. Large language models are not robust multiple choice selectors, 2024

    2024

  40. [49]

    J. Zhuo, S. Zhang, X. Fang, H. Duan, D. Lin, and K. Chen. Prosa: Assessing and understanding the prompt sensitivity of llms. In Findings of the Association for Computational Linguistics: EMNLP 2024 , pages 1950–1976, 2024

    2024

  41. [50]

    F. Zhu, Z. Liu, F. Feng, C. Wang, M. Li, and T. S. Chua. Tat-llm: A specialized language model for discrete reasoning over financial tabular and textual data. In Proceedings of the 5th ACM International Confer- ence on AI in Finance , ICAIF ’24, page 310–318, New York, NY , USA,

  42. [53]

    ⟨X_task_prompt⟩ |=

    B. Zoph, C. Raffel, D. Schuurmans, D. Yogatama, D. Zhou, D. Metzler, E. H. Chi, J. Wei, J. Dean, L. B. Fedus, M. P. Bosma, O. Vinyals, P. Liang, S. Borgeaud, T. B. Hashimoto, and Y . Tay. Emergent abilities of large language models. TMLR, 2022. Appendices A Dataset Details Tab...

    2022

  43. [2024]

    ISBN 9798400710810

    Association for Computing Machinery. ISBN 9798400710810. doi: 10.1145/3677052.3698685

  44. [2025]

    Accessed April 24, 2025

    Preprint. Accessed April 24, 2025

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.