Diverse Prompts: Illuminating the Prompt Space of Large Language Models with MAP-Elites

Gabriel Machado Santos; Marcelo Zanchetta do Nascimento; Rita Maria da Silva Julia

REVIEW 4 major objections 5 minor 1 cited by

Diverse Prompts: Illuminating the Prompt Space of Large Language Models with MAP-Elites

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A grammar-guided MAP-Elites search produces more diverse, high-performing prompts than random search, and the resulting maps expose task-specific prompt rules.

desk verdict Novel CFG-plus-MAP-Elites prompt mapping, but the headline coverage claim is not yet supported because noisy 50-shot fitness plus archive selection can explain the gap. read the letter →

arxiv 2504.14367 v1 pith:VNWJHB6G submitted 2025-04-19 cs.CL cs.AI

Gabriel Machado Santos , Rita Maria da Silva Julia , Marcelo Zanchetta do Nascimento This is my paper · ORCID

classification cs.CLcs.AI

keywords promptengineeringMAP-Elitesquality-diversitysearchcontext-freegrammarlargelanguagemodelsdiversityBigBenchLite

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that prompt diversity for large language models can be systematically generated and studied by combining a context-free grammar with the MAP-Elites search algorithm. The authors argue that MAP-Elites covers more of the prompt-feature space with high-quality prompts than random search, and that the resulting archive reveals task-dependent relationships between prompt structure and model accuracy. The value is practical: instead of hand-tuning prompts, a practitioner could use this approach to map which prompt designs work for which task types, and the maps suggest that no single prompt style wins everywhere.

What carries the argument

The mechanism is a context-free grammar whose expansion path acts as the genotype: production choices decide whether a prompt includes a context-role, how many examples it uses (zero-shot, few-shot, or many-shot), and its reasoning depth (no chain-of-thought, one step, or multiple steps), along with task-specific entries, requests, instructions, and examples. MAP-Elites then evolves a population of these genotypes, scoring each prompt by accuracy on 50 task instances, binning individuals by number of examples, prompt length, and reasoning depth, and keeping only the best occupant per bin. The structured archive is what carries the argument: coverage of bins by high-accuracy individuals is the measure of diversity with quality, and the bin distribution is what supports the task-specific correlations.

What would settle it

Run the full 56-run protocol several times with different random seeds, then take every archived prompt and score it on the complete task set; if coverage counts or which features look significant change across seeds or against full-set scores, the reported diversity advantage and feature correlations are not stable.

Watch

Extended reading notes

Core claim

The paper's central claim is that structuring prompt generation with a context-free grammar and evolving prompts with MAP-Elites produces a broader archive of high-performing, phenotypically varied prompts than random sampling. Across seven BigBench Lite tasks and four instruction-tuned models, MAP-Elites covers over 60% of the feature space with individuals above 55% accuracy in 21 of 28 runs, versus 6 for random search, with statistically significant differences on two tasks. The archive analysis shows task-specific patterns: zero-shot prompts dominate for logical-deduction and knowledge-heavy tasks like LD3 and KU, while few-shot and many-shot prompts help contextual tasks like PDSD and Winowhy; context-role and chain-of-thought depth show little measurable effect. The authors conclude that exploring the prompt space in this way is a method for both optimization and characterization of how prompt structure affects LLM behavior.

Load-bearing premise

The method assumes that a prompt's fitness measured on 50 randomly drawn task instances is stable enough to decide which prompts fill the archive and which features appear correlated with performance.

Editorial extensions

If this is right

Prompt search can be cast as a quality-diversity problem: the archive itself is a deliverable, not just the best prompt.
Zero-shot prompts are not a compromise for reasoning-heavy tasks; on LD3, SQA, and KU they outperform few-shot prompts, so simple designs should be part of any search.
Few-shot and many-shot prompts carry the top performers on contextual pattern-recognition tasks like PDSD, SSB, and Winowhy, consistent with in-context learning effects.
Static context-role templates and chain-of-thought depth had negligible measured impact, meaning those features should not be the primary lever in prompt optimization.
Reported correlations between prompt features and accuracy are generally weak, so the method is more useful for mapping task-specific regions than for deriving global prompt-design rules.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Inference: because each prompt is scored on only 50 task instances without repeated runs, evaluation noise could be inflating or deflating the coverage gap; rerunning with fixed seeds would test this.
Inference: the same grammar-plus-archive recipe could be applied to larger models, and the zero-shot dominance on reasoning tasks may shrink or reverse if larger models exploit demonstrations differently.
Inference: the null context-role result may be an artifact of static templates; task-conditioned or dynamically generated roles could reveal a real effect.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a genuine but modest step. Combining a CFG genotype with MAP-Elites to map prompt structure space is new, and the write-up is honest about weak correlations and resource limits. The problem is the headline claim — “significantly outperforming random generation” — rests on a comparison that selection on noisy fitness could explain.

What’s new: prior work used evolutionary sampling or grammar prompting separately; here the CFG gives controlled variation over shots, reasoning depth, and context, and MAP-Elites bins by those phenotypes. That is a reasonable way to ask which prompt structures matter per task. The paper ships code and data (Zenodo), reports 56 runs, and openly lists limitations in Section VI.A. The per-task findings — zero-shot best for LD3 and SQA, examples helpful for Winowhy and SSB — are plausible and consistent with the broader in-context learning literature.

Where it is soft: The 21/28 vs 6/28 coverage count is the empirical core, but fitness is measured on 50 random instances, and the archive keeps the max of noisy estimates. Random search has no optimizer pushing candidates across the threshold, so the gap could be selection bias rather than richer exploration. The high-performer z-tests in Section V.C compare against the archive population itself, which is not independent, so the significance is inflated. Only two of seven tasks reach statistical significance on coverage, effect sizes are mostly small to medium, and the correlations are near zero — the authors themselves admit large-sample significance is likely. No repeated runs, no variance, no confidence intervals. These are addressable, but they mean the central quantitative claim is not yet established.

Verdict: this deserves a serious referee, not a desk reject. The method is worth engaging, and the questions are the right ones. I would send it back for major revision: add repeated runs or variance estimates, add a matched best-of-k random baseline to control for selection-on-noise, and redo the z-tests against an independent sample or otherwise account for archive dependency. If those checks hold, this becomes a solid contribution. As is, I treat the qualitative trends as suggestive rather than demonstrated.

Referee Report

4 major / 5 minor

Summary. The paper proposes combining a context-free grammar (CFG) with the MAP-Elites quality-diversity algorithm to explore the prompt space of LLMs. Genotypes are CFG expansion paths that control phenotypic traits such as number of examples, reasoning depth, and role context; fitness is the proportion of correct answers on 50 task instances per prompt. The authors evaluate the method on seven BigBench Lite tasks with four sub-10B instruction-tuned LLMs, comparing MAP-Elites against random search in terms of phenotypic coverage, correlation between prompt features and accuracy, and the distribution of features among high-performing prompts. The central claim is that MAP-Elites 'consistently outperformed Random Search' in coverage, with statistical significance in two tasks and high-performer coverage above 60% in 21 of 28 runs versus 6 of 28 for Random Search (Section V.A, Table I). The conclusion states that the method 'effectively enhances prompt diversity, significantly outperforming random generation.' Public code and data are provided via Zenodo.

Significance. If the central claim were fully established, the paper would make a useful contribution to automated prompt engineering by showing that a quality-diversity search over a grammar-defined prompt space yields both higher coverage of the phenotypic space and actionable insights about which prompt structures help which tasks. The paper is honest in listing resource-driven limitations, and the public reproduction artifacts are a real strength. However, the headline claim currently rests on a small number of runs, noisy fitness estimates, and statistical comparisons that are partly circular. The observed effect is plausible, but the evidence as reported does not yet support the strong conclusion in the abstract and Section VI.

major comments (4)

[Section IV.C, Algorithm 1, Eq. (1)] The archive's keep-best-in-bin rule, combined with fitness estimated from only 50 task instances, makes the headline coverage advantage vulnerable to selection on noise. Each prompt's fitness is the proportion correct on 50 random instances, and the archive stores the individual with the highest observed fitness per bin. If true accuracy is near 0.5, a single 50-trial evaluation has probability about 0.24 of exceeding the 55% high-performer threshold, and with several candidates per bin the chance that at least one crosses the threshold is large. Random Search, as described, has no equivalent selection pressure, so the 21-of-28 versus 6-of-28 coverage counts in Table I may reflect the number of noisy draws per bin rather than a genuine quality-diversity advantage. The paper reports no repeated runs and no confidence intervals for these counts. Please add repeated runs or a matched best-of-k random baseline, and report variance or confidence bounds for the coverage statistics.
[Section IV.C] The random-search baseline is underspecified. The text says only that 'we used random sampling as a baseline for diversity' and gives no details on the number of individuals sampled, whether the same evaluation budget (500 evaluations) is used, or whether any best-of-k selection is applied. Without this information, the coverage comparison in Table I may conflate search effectiveness with evaluation budget. The authors should specify the random-search protocol precisely, including the number of samples, the allocation of fitness evaluations, and whether the same 55% threshold is applied to the same set of evaluated prompts.
[Section V.C, Table III] The z-tests for feature over-representation compare the high-performer subset to the overall population of 500 individuals from which those high performers were drawn. This is not an independent comparison: any subset defined by a fitness threshold will tend to differ from the full population on features correlated with fitness, and since the archive keeps only the best individual per bin, the 'high performers' are also the survivors of within-bin selection. The dagger significances in Table III are therefore partly definitional and do not establish that MAP-Elites preferentially explores those structures. A valid test would compare the archive's high performers against an independently sampled random population, or against the complete set of evaluated genotypes with an appropriate multiple-testing correction.
[Section V.A, Table I] The statement that 'MAP-Elites consistently outperformed Random Search' is not supported by the reported statistics. Only SSB and Winowhy reach p < 0.05, multiple comparisons across the 28 runs are not accounted for, and in several rows Random Search reports higher coverage (e.g., PDSD/LLaMA-3-1 high-performer coverage of 80% vs. 72%). The conclusion in Section VI that the method 'significantly outperforming random generation' is an overstatement relative to the evidence. The authors should either temper the claim to the tasks where significance holds or provide a proper multi-run analysis with variance estimates.

minor comments (5)

[Section III.B] There is a typo in the sentence 'improve the fitness os such solutions' which should read 'improve the fitness of such solutions.'
[Section III.B, Algorithm 1] The pseudocode would benefit from an explicit description of the mutation operators and the archive replacement rule, rather than only a textual description; this would make the 'Add or replace' step unambiguous.
[Section V.B, Figure 2] The heatmap in Figure 2 is referenced but not visible in the manuscript text; please ensure it is legible and that all correlation values and significance markers are readable in the published version.
[Section IV.C] The paper states that LLM temperature was set to 0, but does not discuss whether API-side nondeterminism was otherwise controlled; a brief note on this would strengthen reproducibility.
[Section V.A] The description of the statistical tests (Chi-Square, p-value, Cramer's V) is brief; please state the contingency table structure and the sample size underlying each test, since the table reports only percentages.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central comparison is empirical and the fitness signal comes from external task labels.

full rationale

The paper's derivation chain is self-contained on the key points examined. Fitness is defined externally as the proportion of correctly solved task instances (Eq. 1, f = num correct / num evaluations), evaluated against task datasets from BigBench Lite. The MAP-Elites archive is populated by comparing these measured fitness values, not by any quantity derived from the claimed conclusions. The central claim that MAP-Elites outperforms Random Search in coverage is an empirical comparison: Table I reports measured coverage percentages, chi-square statistics, p-values, and effect sizes for each of the 28 runs, and the high-performer threshold (>55%) is an independent criterion applied to both methods. Although MAP-Elites is by design a coverage-seeking algorithm, the paper does not assert that coverage advantage is true by construction; it reports observed outcomes, including tasks where the difference is not significant, which is consistent with an empirical finding rather than a tautology. The task-specific insights in Sections V.B and V.C are descriptive analyses of the measured data, and the paper explicitly acknowledges the weakness of the correlations, so these are not fitted parameters renamed as predictions. There are no load-bearing self-citations: the cited MAP-Elites paper [6] and BigBench sources are standard external references, and none of the authors' own prior work is used to justify the method's validity. Concerns about 50-sample fitness noise affecting archive selection are threats to experimental robustness, not circularity, because the fitness estimate is still an independent measurement of task performance. No equation is shown to reduce to its own input, no fitted value is relabeled as a prediction, and no uniqueness claim is imported from the authors' other work. The honest finding is therefore no significant circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claims rest on the grammar's coverage of prompt space, the stability of 50-instance fitness estimates, and the choice of phenotypic bins and hand-written templates. These are reasonable but untested modeling choices, and the statistical analyses assume independence that the archive structure violates. No invented entities are introduced.

free parameters (5)

Number of context-role templates (C) = 10
Hand-crafted generic context-role phrases (c1..c10), constant across tasks; limits alignment with task-specific nuances. Section III.A.
Number of reasoning-depth templates (T) = 10
Hand-crafted thought templates with step counts 1..10. Section III.A.
High-performer threshold = 55% accuracy
Used to define high performers in coverage and z-test analyses (Section V.A). Chosen by hand without justification.
MAP-Elites hyperparameters = population 50, iterations 10, mutation rate 40%, mutation chance 40%, num evaluations 50, bin sizes (2,25,2)
Hand-set for computational feasibility on a single MacBook; no sensitivity analysis. Section IV.C.
Evaluation subset size = 50 task instances per fitness
Fitness is computed on 50 random instances with no reported seed, adding noise to archive selection. Algorithm 1 and Section IV.C.

assumptions (5)

domain assumption The CFG production rules adequately capture the prompt structures that matter for LLM performance.
The grammar limits the search space to combinations of shots, context-role templates, and reasoning-depth templates; other prompt dimensions such as wording, ordering, and formatting are not explored. Section III.A.
domain assumption MAP-Elites archive bins over number of examples, prompt length, and reasoning depth capture meaningful phenotypic traits.
Bin sizes (2,25,2) discretize these features; other traits like context inclusion are not binned, so the archive structure may miss relevant variation. Sections III.B and IV.C.
domain assumption Fitness from 50 task instances is a stable proxy for prompt quality on the full task.
No repeat evaluations or confidence intervals are reported; high variance is possible for small datasets, affecting archive replacement and all downstream statistics. Algorithm 1.
ad hoc to paper The 10 context and 10 thought templates are representative of generic role and reasoning prompts.
Empirically instantiated by the authors with no external benchmark or selection criterion given. Section III.A.
standard math Statistical tests (chi-square, z-test, Spearman) assumptions hold on the archive data.
Independence assumptions may be violated because individuals in the archive and population are related through mutations and selection. Sections V.A, V.B, and V.C.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Diverse Prompts: Illuminating the Prompt Space of Large Language Models with MAP-Elites." pith.science (2026). https://pith.science/paper/VNWJHB6G

@misc{pith2026250414367,
  author       = {Pith},
  title        = {Pith review of: Diverse Prompts: Illuminating the Prompt Space of Large Language Models with MAP-Elites},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VNWJHB6G}},
  note         = {Machine review of arXiv:2504.14367}
}

read the original abstract

Prompt engineering is essential for optimizing large language models (LLMs), yet the link between prompt structures and task performance remains underexplored. This work introduces an evolutionary approach that combines context-free grammar (CFG) with the MAP-Elites algorithm to systematically explore the prompt space. Our method prioritizes quality and diversity, generating high-performing and structurally varied prompts while analyzing their alignment with diverse tasks by varying traits such as the number of examples (shots) and reasoning depth. By systematically mapping the phenotypic space, we reveal how structural variations influence LLM performance, offering actionable insights for task-specific and adaptable prompt design. Evaluated on seven BigBench Lite tasks across multiple LLMs, our results underscore the critical interplay of quality and diversity, advancing the effectiveness and versatility of LLMs.

Figures

Figures reproduced from arXiv: 2504.14367 by the authors.

**Figure 1.** Feature space coverage for the Logical Deduction [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Self-Evolving Agent Harnesses via Gated Semantic Quality-Diversity
cs.CL 2026-07 conditional novelty 6.0 of 10

A gated, statistically-checked self-evolution loop improves frozen agents' harnesses by +9 to +15.5 points on sealed tests across six benchmarks, retaining 86-147% of the training gain.

Reference graph

Works this paper leans on

39 extracted references · 13 canonical work pages · cited by 1 Pith paper

[1]

A review on large language models: Architectures, applications, taxonomies, open issues and challenges,

M. A. K. Raiaan, M. S. H. Mukta, K. Fatema, N. M. Fahad, S. Sakib, M. M. J. Mim, J. Ahmad, M. E. Ali, and S. Azam, “A review on large language models: Architectures, applications, taxonomies, open issues and challenges,” IEEE Access , 2024

work page 2024
[2]

The prompt report: A systematic survey of prompting techniques,

S. Schulhoff, M. Ilie, N. Balepur, K. Kahadze, A. Liu, C. Si, Y . Li, A. Gupta, H. Han, S. Schulhoff et al., “The prompt report: A systematic survey of prompting techniques,” arXiv preprint arXiv:2406.06608 , 2024

arXiv 2024
[3]

Prompt tuning in code intelligence: An experimental evaluation,

C. Wang, Y . Yang, C. Gao, Y . Peng, H. Zhang, and M. R. Lyu, “Prompt tuning in code intelligence: An experimental evaluation,” IEEE Transactions on Software Engineering , 2023

work page 2023
[4]

Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery,

Y . Wen, N. Jain, J. Kirchenbauer, M. Goldblum, J. Geiping, and T. Gold- stein, “Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery,” Advances in Neural Information Processing Systems, vol. 36, 2024

2024
[5]

Prompt engineering for chatgpt: a quick guide to techniques, tips, and best practices,

S. Ekin, “Prompt engineering for chatgpt: a quick guide to techniques, tips, and best practices,” Authorea Preprints, 2023

2023
[6]

Illuminating search spaces by mapping elites,

J.-B. Mouret and J. Clune, “Illuminating search spaces by mapping elites,” arXiv preprint arXiv:1504.04909 , 2015

arXiv 2015
[7]

Exploring the prompt space of large language models through evolutionary sampling,

M. Saletta and C. Ferretti, “Exploring the prompt space of large language models through evolutionary sampling,” in Proceedings of the Genetic and Evolutionary Computation Conference , 2024, pp. 1345–1353

work page 2024
[8]

Big-bench lite,

G. Research, “Big-bench lite,” https://github.com/google/BIG-bench/ blob/main/bigbench/benchmark tasks/keywords to tasks.md# big-bench-lite, 2023, accessed: December 31, 2024

work page 2023

Show all 39 references

[9]

Distributed representations of words and phrases and their composi- tionality,

T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, “Distributed representations of words and phrases and their composi- tionality,” Advances in neural information processing systems , vol. 26, 2013

2013
[10]

Glove: Global vectors for word representation,

J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors for word representation,” in Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) , 2014, pp. 1532–1543

2014
[11]

Bert: Pre-training of deep bidirectional transformers for language understanding,

J. Devlin, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805 , 2018

2018 arXiv
[12]

Improving language understanding by generative pre- training,

A. Radford, “Improving language understanding by generative pre- training,” 2018

2018
[13]

Attention is all you need,

A. Vaswani, “Attention is all you need,” Advances in Neural Information Processing Systems, 2017

2017
[14]

Language mod- els are few-shot learners,

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language mod- els are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020

1901
[15]

Llama: Open and efficient foundation language models,

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar et al. , “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023

2023 arXiv
[16]

Megatron-lm: Training multi-billion parameter language models using model parallelism,

M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catan- zaro, “Megatron-lm: Training multi-billion parameter language models using model parallelism,” arXiv preprint arXiv:1909.08053 , 2019

1909 arXiv
[17]

Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing,

P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig, “Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing,” ACM Computing Surveys, vol. 55, no. 9, pp. 1–35, 2023

2023
[18]

Rethinking the role of demonstrations: What makes in-context learning work?

S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer, “Rethinking the role of demonstrations: What makes in-context learning work?” arXiv preprint arXiv:2202.12837 , 2022

2022 arXiv
[19]

Chain-of-thought prompting elicits reasoning in large language models,

J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou et al. , “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems , vol. 35, pp. 24 824–24 837, 2022

2022
[20]

Training millions of personalized dialogue agents,

P.-E. Mazar ´e, S. Humeau, M. Raison, and A. Bordes, “Training millions of personalized dialogue agents,” arXiv preprint arXiv:1809.01984 , 2018

2018 arXiv
[21]

Exploiting cloze questions for few shot text classification and natural language inference,

T. Schick and H. Sch ¨utze, “Exploiting cloze questions for few shot text classification and natural language inference,” arXiv preprint arXiv:2001.07676, 2020

2001 arXiv
[22]

A survey on evaluation of large language models,

Y . Chang, X. Wang, J. Wang, Y . Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y . Wang et al. , “A survey on evaluation of large language models,” ACM Transactions on Intelligent Systems and Technology , vol. 15, no. 3, pp. 1–45, 2024

2024
[23]

Quality diversity: A new frontier for evolutionary computation,

J. K. Pugh, L. B. Soros, and K. O. Stanley, “Quality diversity: A new frontier for evolutionary computation,” Frontiers in Robotics and AI , vol. 3, p. 202845, 2016

2016
[24]

Robots that can adapt like animals,

A. Cully, J. Clune, D. Tarapore, and J.-B. Mouret, “Robots that can adapt like animals,” Nature, vol. 521, no. 7553, pp. 503–507, 2015

2015
[25]

Deep curiosity search: Intra-life exploration can improve performance on challenging deep reinforcement learning problems,

C. Stanton and J. Clune, “Deep curiosity search: Intra-life exploration can improve performance on challenging deep reinforcement learning problems,” arXiv preprint arXiv:1806.00553 , 2018

2018 arXiv
[26]

Understanding innovation en- gines: Automated creativity and improved stochastic optimization via deep learning,

A. Nguyen, J. Yosinski, and J. Clune, “Understanding innovation en- gines: Automated creativity and improved stochastic optimization via deep learning,” Evolutionary computation , vol. 24, no. 3, pp. 545–572, 2016

2016
[27]

Prefix-tuning: Optimizing continuous prompts for generation,

X. L. Li and P. Liang, “Prefix-tuning: Optimizing continuous prompts for generation,” arXiv preprint arXiv:2101.00190 , 2021

2021 arXiv
[28]

Auto- prompt: Eliciting knowledge from language models with automatically generated prompts,

T. Shin, Y . Razeghi, R. L. Logan IV , E. Wallace, and S. Singh, “Auto- prompt: Eliciting knowledge from language models with automatically generated prompts,” arXiv preprint arXiv:2010.15980 , 2020

2010 arXiv
[29]

Exploring the limits of transfer learning with a unified text-to-text transformer,

C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of machine learning research, vol. 21, no. 140, pp. 1–67, 2020

2020
[30]

Grammar prompting for domain-specific language generation with large language models,

B. Wang, Z. Wang, X. Wang, Y . Cao, R. A Saurous, and Y . Kim, “Grammar prompting for domain-specific language generation with large language models,” Advances in Neural Information Processing Systems , vol. 36, 2024

2024
[31]

Chomsky, Aspects of the Theory of Syntax

N. Chomsky, Aspects of the Theory of Syntax . MIT press, 2014, no. 11

2014
[32]

Beyond the imitation game: Quantifying and extrapolating the capabilities of language models,

A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso et al. , “Beyond the imitation game: Quantifying and extrapolating the capabilities of language models,” arXiv preprint arXiv:2206.04615 , 2022

2022 arXiv
[33]

Hugging face inference endpoints,

H. Face, “Hugging face inference endpoints,” https://huggingface.co/ inference-endpoints/dedicated, 2024, accessed: December 18, 2024

2024
[34]

Starling-lm-7b-alpha,

H. Face and B. N. Lab, “Starling-lm-7b-alpha,” https://huggingface. co/berkeley-nest/Starling-LM-7B-alpha, 2024, accessed: December 18, 2024

2024
[35]

Llama-3.1-8b-instruct,

H. Face and M. AI, “Llama-3.1-8b-instruct,” https://huggingface.co/ meta-llama/Llama-3.1-8B-Instruct, 2024, accessed: December 18, 2024

2024
[36]

Phi-3.5-mini-instruct,

H. Face and Microsoft, “Phi-3.5-mini-instruct,” https://huggingface.co/ microsoft/Phi-3.5-mini-instruct, 2024, accessed: December 18, 2024

2024
[37]

Qwen2.5-7b-instruct,

H. Face and Q. Team, “Qwen2.5-7b-instruct,” https://huggingface.co/ Qwen/Qwen2.5-7B-Instruct, 2024, accessed: December 18, 2024

2024
[38]

Schema-learning and rebinding as mechanisms of in-context learning and emergence,

S. Swaminathan, A. Dedieu, R. Vasudeva Raju, M. Shanahan, M. Lazaro-Gredilla, and D. George, “Schema-learning and rebinding as mechanisms of in-context learning and emergence,” Advances in Neural Information Processing Systems , vol. 36, 2024

2024
[39]

Large lan- guage models are zero-shot reasoners,

T. Kojima, S. S. Gu, M. Reid, Y . Matsuo, and Y . Iwasawa, “Large lan- guage models are zero-shot reasoners,” Advances in neural information processing systems, vol. 35, pp. 22 199–22 213, 2022

2022

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

A review on large language models: Architectures, applications, taxonomies, open issues and challenges,

M. A. K. Raiaan, M. S. H. Mukta, K. Fatema, N. M. Fahad, S. Sakib, M. M. J. Mim, J. Ahmad, M. E. Ali, and S. Azam, “A review on large language models: Architectures, applications, taxonomies, open issues and challenges,” IEEE Access , 2024

work page 2024

[2] [2]

The prompt report: A systematic survey of prompting techniques,

S. Schulhoff, M. Ilie, N. Balepur, K. Kahadze, A. Liu, C. Si, Y . Li, A. Gupta, H. Han, S. Schulhoff et al., “The prompt report: A systematic survey of prompting techniques,” arXiv preprint arXiv:2406.06608 , 2024

arXiv 2024

[3] [3]

Prompt tuning in code intelligence: An experimental evaluation,

C. Wang, Y . Yang, C. Gao, Y . Peng, H. Zhang, and M. R. Lyu, “Prompt tuning in code intelligence: An experimental evaluation,” IEEE Transactions on Software Engineering , 2023

work page 2023

[4] [4]

Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery,

Y . Wen, N. Jain, J. Kirchenbauer, M. Goldblum, J. Geiping, and T. Gold- stein, “Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery,” Advances in Neural Information Processing Systems, vol. 36, 2024

2024

[5] [5]

Prompt engineering for chatgpt: a quick guide to techniques, tips, and best practices,

S. Ekin, “Prompt engineering for chatgpt: a quick guide to techniques, tips, and best practices,” Authorea Preprints, 2023

2023

[6] [6]

Illuminating search spaces by mapping elites,

J.-B. Mouret and J. Clune, “Illuminating search spaces by mapping elites,” arXiv preprint arXiv:1504.04909 , 2015

arXiv 2015

[7] [7]

Exploring the prompt space of large language models through evolutionary sampling,

M. Saletta and C. Ferretti, “Exploring the prompt space of large language models through evolutionary sampling,” in Proceedings of the Genetic and Evolutionary Computation Conference , 2024, pp. 1345–1353

work page 2024

[8] [8]

Big-bench lite,

G. Research, “Big-bench lite,” https://github.com/google/BIG-bench/ blob/main/bigbench/benchmark tasks/keywords to tasks.md# big-bench-lite, 2023, accessed: December 31, 2024

work page 2023