Adaptive Prompting: Ad-hoc Prompt Composition for Social Bias Detection

Barbara Hammer; Eyke H\"ullermeier; Fabian Fumagalli; Henning Wachsmuth; Maximilian Muschalik; Maximilian Splieth\"over; Tim Knebler

REVIEW 2 major objections 6 minor 77 references

Adaptive Prompting: Ad-hoc Prompt Composition for Social Bias Detection

T0 review · 2 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Per-input prompt selection can beat any single fixed prompt composition in social bias detection.

desk verdict Solid empirical paper on per-instance prompt composition selection, but the CobraFrames failure is likely an objective mismatch (BCE vs macro F1) rather than the unlearnability the authors claim. read the letter →

arxiv 2502.06487 v1 pith:VUAQUS6R submitted 2025-02-10 cs.CL

Maximilian Spliethöver , Tim Knebler , Fabian Fumagalli , Maximilian Muschalik , Barbara Hammer , Eyke Hüllermeier , Henning Wachsmuth This is my paper · ORCID

classification cs.CL

keywords adaptivepromptingpromptcompositionsocialbiasdetectionlargelanguagemodelsin-contextlearningShapleyinteractionsencoder-basedselectionengineering

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that the most effective prompt for a large language model is not fixed: it depends on the text being classified, and a learned selector can choose the right combination of prompting techniques for each input. The authors test this on social bias detection with three LLMs and three datasets, building 64 prompt compositions from five techniques and training an encoder to predict which composition will produce a correct label. On StereoSet the per-instance selector beats the best fixed composition for all three LLMs (macro F1 0.809 vs 0.800 for Mistral, 0.781 vs 0.706 for Command-R, 0.853 vs 0.817 for Llama 3); on SBIC it improves over the best fixed composition for Llama 3 and is competitive for Mistral. The approach fails on CobraFrames, where the encoder cannot learn meaningful input-to-composition links, so the paper's central claim is conditional on that learnability.

What carries the argument

The load-bearing object is the prompt composition: a fixed-order combination of the base task description and input with any subset of five techniques, where the in-context demonstrations technique has three mutually exclusive variants, yielding $2^4 \cdot (3+1) = 64$ compositions. The mechanism that carries the argument is an encoder-based selection model: a fine-tuned DeBERTa-v3-large with a sigmoid regression head trained by binary cross-entropy to estimate, for each input, the likelihood that each of the 64 compositions will generate a correct label; at inference it applies the highest-likelihood composition. The Shapley value and Shapley interaction analysis over the composition game supplies the second part: it attributes performance to individual techniques and their pairwise synergies, supporting the claim that technique interactions matter and that no single composition is universally best.

What would settle it

On a held-out test set, compute the per-instance oracle composition (the one that yields the correct label), then train the encoder on the training split. If the encoder's chosen composition is correct no more often than choosing the single most frequent composition from the training set, the central claim is refuted; CobraFrames with Mistral (0.580 adaptive vs 0.604 best fixed) is a concrete case where this already appears to happen.

Watch

Extended reading notes

Core claim

The paper's central claim is that prompt compositions—ordered sets of discrete prompting techniques—can and should be chosen ad hoc per input instance rather than optimized once per dataset or model. The authors define 64 compositions from five techniques (persona, definition, in-context demonstrations in one of three selection modes, directional stimulus, and reasoning steps), collect LLM labels for every composition on training texts, then fine-tune a DeBERTa-v3-large encoder to output a likelihood for each composition and pick the argmax. They report that this adaptive prompting outperforms the best single test-set composition on StereoSet for all three LLMs, and on SBIC for Llama 3. They also find that the performance of compositions is volatile across models and texts, that adding more techniques does not monotonically improve results, and that Shapley interaction analysis reveals both positive and negative pairwise interactions between techniques.

Load-bearing premise

The whole approach rests on the assumption that a text-encoder can learn, from the input text alone, which of many prompt compositions will make the LLM produce a correct label; the paper itself finds this fails on CobraFrames.

Editorial extensions

If this is right

Prompt selection can be moved from human trial-and-error to a learned per-instance decision, reducing the cost of finding a good prompt for a new LLM or dataset.
A fixed best-on-validation composition is not an upper bound; input-dependent selection can beat it, as shown on StereoSet and SBIC.
Compositions that combine more techniques are not automatically better; Shapley interactions show that individually helpful techniques can hurt when combined, so selection must consider interactions.
The approach transfers to other tasks with smaller gains: on sentiment analysis, natural language inference, and question answering it beats the base composition and the best-validation composition in the reported experiments.
On datasets where text content does not predict which prompt will succeed, such as CobraFrames, adaptive prompting falls back toward the level of a fixed or random selector.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension is to restrict the selector to the small set of compositions that disagree most often on a validation set; the gains should concentrate there, and the computational cost of collecting labels would drop sharply.
The CobraFrames failure suggests a diagnostic for when to trust adaptive prompting: measure the mutual information between the input and the oracle-optimal composition. If it is near zero, a fixed composition is the safer choice.
If the encoder is choosing the most frequent composition on hard datasets, the approach degenerates to best-fixed selection; comparing the selector's chosen frequencies to per-composition correctness on the training split, as the paper does, is a cheap sanity check.
The gains over best fixed compositions are modest in absolute terms; a sympathetic reading is that adaptive prompting is a safety property—it avoids the worst compositions—rather than a large average improvement, and that is what the cross-dataset results suggest.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's the short version: this is a solid, hard-working empirical paper on per-instance prompt composition selection for social bias detection. The idea is genuinely new in the framing—predicting which of 64 hand-designed discrete prompt compositions will work for each input text, rather than searching for one global prompt. The authors compare against the best composition on the test set (a generous baseline) and their adaptive method wins on StereoSet across all three LLMs and on SBIC for Llama 3. They also ship a code link, report many ablations, and are candid that the method fails on CobraFrames.

What the paper does well: the evaluation is extensive—three LLMs (7B to 70B), three datasets, 64 compositions, five seeds, plus a Shapley interaction analysis. The finding that prompt composition performance is highly volatile and that instance-specific selection can beat the best fixed composition on the test set is credible and useful for practitioners. The Shapley interaction analysis also gives real insight into which techniques combine well and which interact negatively.

But there are real issues. The CobraFrames failure is the main one. The authors attribute it to the encoder not learning meaningful text-composition connections, but the stress-test analysis surfaces a more likely culprit: the encoder is trained with binary cross-entropy on per-instance correctness, while the evaluation metric is macro F1. On CobraFrames, the test set is 1862 positive vs 77 negative. A selector that gets 84% of instances right can still produce a macro F1 near 0.58 because it misses most of the 77 minority-class instances. That is an objective mismatch, not necessarily a learnability failure. The paper's own numbers in Table 5 support this—the encoder's per-instance accuracy is actually comparable to the other datasets. So the 'robustly' claim in the abstract is too strong; at best, the method is robust on reasonably balanced data.

I also wish the significance testing were sturdier: a one-sided t-test over five seeds without multiple-comparison correction, and no confidence intervals for the headline macro F1 numbers. This is a moderate concern, not fatal—the StereoSet gains are large enough that they would likely survive a more conservative test, but the SBIC results (0.790 vs 0.792 with Mistral) would not survive.

Who should read it: anyone working on automatic prompting, few-shot selection, or bias detection with LLMs. The idea is subfield-level, not paradigm-shifting, but it is a real step beyond global prompt search. It deserves a serious referee: I would accept this for review and ask for the objective-mismatch analysis and corrected significance testing as major revisions. The cross-dataset generalization is weak, but they acknowledge that too. Overall, a paper worth engaging with despite the flagged concerns.

Referee Report

2 major / 6 minor

Summary. The paper proposes an adaptive prompting approach that trains a DeBERTa-based encoder to predict, for each input text, which of 64 hand-defined prompt compositions will make a target instruction-tuned LLM produce a correct binary social-bias label. The approach is evaluated on three datasets (StereoSet, SBIC, CobraFrames) with three LLMs (Mistral-7B, Command-R-35B, Llama-3-70B) and compared against fixed compositions, an ensemble, a fine-tuned DeBERTa, and Shapley-value-based composition selection. The central empirical claim is that per-instance composition selection can outperform the best fixed composition on the test set: on StereoSet this holds for all three LLMs, on SBIC it holds only for Llama 3, and on CobraFrames adaptive prompting is below the best fixed test composition for all three LLMs. The paper also includes a Shapley interaction analysis of the prompting techniques and additional experiments on three other NLP tasks.

Significance. The idea of selecting prompt compositions per input instance is a useful and relatively underexplored direction, and the experimental setup is extensive, covering three LLMs of different sizes, three datasets, and a large number of compositions. The comparison against Best on Test is appropriately conservative, as it uses oracle knowledge of test-set performance, and the Shapley interaction analysis provides interpretable insight into how prompting techniques interact. The StereoSet results, where adaptive prompting beats the best fixed test composition for all three LLMs, are the strongest evidence for the approach. However, the failure on CobraFrames, together with the statistical reporting limitations, substantially weakens the robustness and generality claims made in the abstract and conclusion. If the objective-mismatch issue is resolved and the significance analysis is made more rigorous, the contribution would be solid for a specialized NLP methods venue.

major comments (2)

[§3.2, §5.5, Tables 4, 5, 8] The encoder is trained with binary cross-entropy on whether each composition produces the correct binary label per instance, which optimizes per-instance accuracy, but it is evaluated with macro F1. On CobraFrames, the test split is highly imbalanced (1862 positive vs 77 negative instances, Table 4), so an accuracy-trained selector can be correct on the vast majority of instances while making errors on the 77 negatives, which dominate macro F1. Table 5 shows that the encoder's selected compositions yield correct labels for 84.6%, 83.3%, and 82.0% of CobraFrames instances for Mistral, Command-R, and Llama 3 respectively, yet Table 8 shows adaptive macro F1 is below Best on Test for all three LLMs. The paper's diagnosis in §5.5 that the encoder "can likely not learn meaningful connections between the inputs and compositions" is therefore not the most consistent explanation; the accuracy-vs-macro-F1 objective mismatch is a load-bearing alternative that directly affects the claimed robustness. The authors should either re-analyze the failure with per-class breakdowns or adjust the training objective (e.g., cost-sensitive or metric-aware learning), and revise the explanation accordingly.
[§B.3, Tables 1, 7, 8] The significance testing is selective and lacks multiple-comparison correction. As stated in §B.3, a one-sided independent t-test over five seeds is used only when adaptive prompting shows the best overall results, and no correction is applied across the many comparisons (three LLMs, multiple datasets, multiple baselines). Table 1, for instance, reports p<.05 and p<.01 for three LLM comparisons without accounting for the multiple tests, and no confidence intervals are given for the macro F1 scores. This weakens the headline claim that adaptive prompting "robustly ensures high detection performance" and "is best in several settings." I recommend reporting all tested comparisons, applying a multiple-comparison correction (e.g., Bonferroni or Holm), and providing per-seed variances or confidence intervals.

minor comments (6)

[Abstract and Conclusion] The statement "Our approach robustly ensures high detection performance" is too strong given that adaptive prompting underperforms Best on Test on CobraFrames for all three LLMs (Table 8). Suggest softening to something like "can achieve high detection performance in several settings."
[Tables 9–11 captions] There are typos in the captions of Tables 9, 10, and 11: "defintion" should be "definition."
[Acknowledgments] The Acknowledgments contain a typo: "ChaptGPT" should be "ChatGPT."
[§3.2] The rationale for avoiding a multi-class setup, namely "to avoid a few dominant compositions from possibly being preferred over others consistently," would benefit from a more detailed explanation of how this preference would arise and why a sigmoid output avoids it.
[Tables 1, 7, 8] The main results tables report adaptive prompting as a single macro F1 value without standard deviations or confidence intervals, even though it is averaged over five seeds; including the variance would aid interpretation of the significance claims.
[Figure 9 and Table 4] The majority baseline value (0.364) shown for CobraFrames in Figure 9 is not clearly defined; given the class imbalance in Table 4, it would be helpful to state how the majority baseline macro F1 is computed (e.g., the macro F1 of always predicting the majority class).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the adaptive selector is trained on LLM pseudo-labels but evaluated against held-out ground truth.

full rationale

The central derivation is not circular. The approach collects LLM labels for all 64 compositions on training texts (Step 1), trains a DeBERTa encoder to predict which composition yields a correct label (Section 3.2, BCE loss), and then, for unknown test texts, selects the arg-max composition and feeds it to the same LLM (Step 3). The reported macro-F1 results (Tables 1, 7, 8) are computed against ground-truth labels on held-out test splits, so the prediction is not the training target by construction; it is a genuine generalization claim. The Best-on-Val and Best-on-Test baselines are oracle/selection baselines, not fitted parameters of the adaptive model. The Shapley analysis (Section 3.3, Appendix C) is a post-hoc explanation of technique contributions and is not used to fit the adaptive selector's test predictions. Self-citations to shapiq (Muschalik et al. 2025, Fumagalli et al. 2023) are citations to a public library implementing standard Shapley interactions; they are not invoked to forbid alternatives or to validate the main result. The only internal diagnostic that resembles the training objective (Table 5) is evaluated on test data, so it measures generalization rather than reducing the result to a fit. No equation in the paper defines the reported F1 as an optimized constant. The accuracy-versus-macro-F1 mismatch on CobraFrames is a performance and robustness concern, not a circularity issue, because it does not make the evaluation equivalent to the training signal. Therefore no significant circularity is present.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities or formal axioms. It relies on hand-designed prompt templates, demonstration counts, a learnability assumption, and dataset-specific preprocessing choices. The main risk is not circularity but the fragility of the learnability assumption, which the authors themselves observe on CobraFrames.

free parameters (2)

In-context demonstration counts per dataset = 7 (SBIC), 4 (StereoSet), 11 (CobraFrames)
Hand-chosen in Appendix A to cover bias categories; no ablation shows how sensitive results are to these counts.
CobraFrames binarization keyword list = presence of 'offensive', 'microaggression', 'xenophobic', etc.
Ad hoc conversion of offensiveness labels to binary bias (Appendix A.3); this defines the evaluation labels and therefore the central F1 numbers.

assumptions (4)

ad hoc to paper The pre-defined ordering and compatibility constraints in Section 3.1 define all meaningful prompt compositions.
The search space size 64 follows from this hand-designed structure; any better prompt outside this space is unreachable, so the claim of optimality is relative to this space.
domain assumption Constrained decoding to 'Yes'/'No' provides a reliable binary bias classifier.
Described in Appendix B.1; if the LLM cannot express bias judgments under this constraint, all collected labels and the encoder training signal are invalid.
domain assumption Macro F1 is the appropriate performance metric for the evaluation.
Used throughout Sections 4 and 5; with CobraFrames' extreme class imbalance (Table 4), macro F1 is sensitive to rare-class behavior, and the 'optimal' composition is defined relative to it.
domain assumption The DeBERTa-v3-large encoder can learn a mapping from input text to optimal composition.
This is the core learnability premise; Section 5.5 reports it fails on CobraFrames, which is why the central claim is not fully general.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Adaptive Prompting: Ad-hoc Prompt Composition for Social Bias Detection." pith.science (2026). https://pith.science/paper/VUAQUS6R

@misc{pith2026250206487,
  author       = {Pith},
  title        = {Pith review of: Adaptive Prompting: Ad-hoc Prompt Composition for Social Bias Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VUAQUS6R}},
  note         = {Machine review of arXiv:2502.06487}
}

read the original abstract

Recent advances on instruction fine-tuning have led to the development of various prompting techniques for large language models, such as explicit reasoning steps. However, the success of techniques depends on various parameters, such as the task, language model, and context provided. Finding an effective prompt is, therefore, often a trial-and-error process. Most existing approaches to automatic prompting aim to optimize individual techniques instead of compositions of techniques and their dependence on the input. To fill this gap, we propose an adaptive prompting approach that predicts the optimal prompt composition ad-hoc for a given input. We apply our approach to social bias detection, a highly context-dependent task that requires semantic understanding. We evaluate it with three large language models on three datasets, comparing compositions to individual techniques and other baselines. The results underline the importance of finding an effective prompt composition. Our approach robustly ensures high detection performance, and is best in several settings. Moreover, first experiments on other tasks support its generalizability.

Figures

Figures reproduced from arXiv: 2502.06487 by the authors.

**Figure 2.** The three steps of our adaptive prompting approach: (1) Bias labels are collected for all considered prompt [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Social bias detection results on StereoSet (oth [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (7 more)

**Figure 4.** Figure 4: Network plots of the shapley interactions for the three evaluated LLMs on StereoSet (others in Appendix [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

**Figure 5.** Figure 5: Force plots of Shapley values for three variants (top: category in-context demonstrations, middle: [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]

**Figure 6.** Figure 6: Network plots of the shapley interactions for the three evaluated LLMs on SBIC. [PITH_FULL_IMAGE:figures/full_fig_p021_6.png]

**Figure 7.** Figure 7: Network plots of the shapley interactions for the three evaluated LLMs on CobraFrames. [PITH_FULL_IMAGE:figures/full_fig_p021_7.png]

**Figure 8.** Figure 8: Social bias detection results on SBIC: Macro [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]

**Figure 9.** Figure 9: Social bias detection results on CobraFrames: [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]

**Figure 10.** Figure 10: An example prompt composition for StereoSet bias inference using all prompting techniques evaluated [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

77 extracted references · 31 canonical work pages

[1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Alberto Mario Ceballos Arroyo, Monica Munnangi, Jiuding Sun, Karen Y. C. Zhang, Denis Jered McInerney, Byron C. Wallace, and Silvio Amir. 2024. http://arxiv.org/abs/2407.09429 Open ( Clinical ) LLMs are Sensitive to Instruction Phrasings . arXiv preprint

work page Pith review arXiv 2024
[4]

Fatemeh Torabi Asr, Mohammad Mazraeh, Alexandre Lopes, Vasundhara Gautam, Junette Gonzales, Prashanth Rao, and Maite Taboada. 2021. https://doi.org/10.1371/journal.pone.0245533 The Gender Gap Tracker : Using Natural Language Processing to measure gender bias in media . PLOS ONE, 16(1)

work page doi:10.1371/journal.pone.0245533 2021
[5]

Tilman Beck, Hendrik Schuff, Anne Lauscher, and Iryna Gurevych. 2024. https://aclanthology.org/2024.eacl-long.159 Sensitivity, Performance , Robustness : Deconstructing the Effect of Sociodemographic Prompting . In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics ( Volume 1: Long Papers ) , pages ...

work page 2024
[6]

Gormley, and Graham Neubig

Amanda Bertsch, Maor Ivgi, Uri Alon, Jonathan Berant, Matthew R. Gormley, and Graham Neubig. 2024. https://doi.org/10.48550/arXiv.2405.00200 In- Context Learning with Long - Context Models : An In - Depth Exploration . arXiv preprint

arXiv doi:10.48550/arxiv.2405.00200 2024
[7]

Sebastian Bordt and Ulrike von Luxburg. 2023. From Shapley Values to Generalized Additive Models and back . In International Conference on Artificial Intelligence and Statistics (AISTATS 2023) , volume 206 of Proceedings of Machine Learning Research, pages 709--745. PMLR

work page 2023
[8]

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

arXiv 2020

Show all 77 references

[9]

Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. https://papers.nips.cc/paper_files/paper/2018/hash/4c7a167bb329bd92580a99ce422d6fa6-Abstract.html e- SNLI : Natural Language Inference with Natural Language Explanations . In Advances in Neural In...

2018
[10]

Minje Choi, Jiaxin Pei, Sagar Kumar, Chang Shu, and David Jurgens. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.699 Do LLMs Understand Social Knowledge ? Evaluating the Sociability of Large Language Models with SocKET Benchmark . In Proceedings of the 2023 Conference on E...

2023 doi
[11]

CohereForAI. 2024. https://huggingface.co/CohereForAI/c4ai-command-r-v01 Command- R model card

2024
[12]

Erik Derner, Sara Sansalvador de la Fuente, Yoan Gutiérrez, Paloma Moreda, and Nuria Oliver. 2024. https://doi.org/10.48550/arXiv.2406.13677 Leveraging Large Language Models to Measure Gender Bias in Gendered Languages . arXiv preprint

2024 doi arXiv
[13]

Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.88 Toxicity in chatgpt: Analyzing persona-assigned language models . In Findings of the Association for Computational Linguistics ...

2023 doi
[14]

Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. 2024. https://doi.org/10.48550/arXiv.2301.00234 A Survey on In -context Learning . arXiv preprint

2024 doi arXiv
[15]

Schelten

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, and Alan et al. Schelten. 2024. https://doi.org/10.48550/arXiv.2407.21783 The Llama 3 Herd of Models . arXiv preprint

2024 doi arXiv
[16]

Micha Elsner and Jordan Needle. 2023. https://doi.org/10.18653/v1/2023.sigmorphon-1.2 Translating a low-resource language using GPT -3 and a human-readable dictionary . In Proceedings of the 20th SIGMORPHON workshop on Computational Research in Phonetics , Phonology , and Morp...

2023 doi
[17]

Federico Errica, Giuseppe Siracusano, Davide Sanvito, and Roberto Bifulco. 2024. https://doi.org/10.48550/arXiv.2406.12334 What Did I Do Wrong ? Quantifying LLMs ' Sensitivity and Consistency to Prompt Engineering . arXiv preprint

2024 doi arXiv
[18]

Fabian Fumagalli, Maximilian Muschalik, Eyke Hüllermeier, Barbara Hammer, and Julia Herbinger. 2024 a . https://doi.org/10.48550/arXiv.2412.17152 Unifying Feature-Based Explanations with Functional ANOVA and Cooperative Game Theory . arXiv preprint

2024 doi arXiv
[19]

Fabian Fumagalli, Maximilian Muschalik, Patrick Kolpaczki, Eyke H \"u llermeier, and Barbara Hammer. 2024 b . https://openreview.net/forum?id=d5jXW2H4gg Kernel SHAP - IQ : Weighted least square optimization for shapley interactions . In Forty-first International Conference on ...

2024
[20]

Fabian Fumagalli, Maximilian Muschalik, Patrick Kolpaczki, Eyke Hüllermeier, and Barbara Hammer. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/hash/264f2e10479c9370972847e96107db7f-Abstract-Conference.html SHAP - IQ : Unified Approximation of any-order Shapley In...

2023
[21]

Ungar, and Brenda Curtis

Salvatore Giorgi, Tingting Liu, Ankit Aich, Kelsey Isman, Garrick Sherman, Zachary Fried, João Sedoc, Lyle H. Ungar, and Brenda Curtis. 2024. https://doi.org/10.48550/arXiv.2406.14462 Explicit and Implicit Large Language Model Personas Generate Opinions but Fail to Replicate D...

2024 doi arXiv
[22]

Andreas Grivas, Antonio Vergari, and Adam Lopez. 2024. https://doi.org/10.1609/aaai.v38i11.29110 Taming the Sigmoid Bottleneck : Provably Argmaxable Sparse Multi - Label Classification . Proceedings of the AAAI Conference on Artificial Intelligence, 38(11):12208--12216. Number: 11

2024 doi
[23]

Hyeonmin Ha, Jihye Lee, Wookje Han, and Byung-Gon Chun. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.159 Meta- Learning of Prompt Generation for Lightweight Prompt Engineering on Language - Model -as-a- Service . In Findings of the Association for Computational Lingui...

2023 doi
[24]

Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. https://openreview.net/forum?id=sE7-XhLxHA DeBERTaV3 : Improving DeBERTa using ELECTRA - Style Pre - Training with Gradient - Disentangled Embedding Sharing . Kigali, Rwanda

2023
[25]

Sui He. 2024. https://doi.org/10.48550/arXiv.2403.00127 Prompting ChatGPT for Translation : A Comparative Analysis of Translation Brief and Persona Prompts . arXiv preprint

2024 doi arXiv
[26]

Rem Hida, Masahiro Kaneko, and Naoaki Okazaki. 2024. https://doi.org/10.48550/arXiv.2407.03129 Social Bias Evaluation for Large Language Models Requires Prompt Variations . arXiv preprint

2024 doi arXiv
[27]

Bowman, and Omer Levy

Or Honovich, Uri Shaham, Samuel R. Bowman, and Omer Levy. 2023. https://doi.org/10.18653/v1/2023.acl-long.108 Instruction Induction : From Few Examples to Natural Language Task Descriptions . In Proceedings of the 61st Annual Meeting of the Association for Computational Lingui...

2023 doi
[28]

Dirk Hovy and Diyi Yang. 2021. https://doi.org/10.18653/v1/2021.naacl-main.49 The Importance of Modeling Social Factors of Language : Theory and Practice . In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics : Hu...

2021 doi
[29]

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

2023 doi arXiv
[30]

Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts

Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. 2023. https://doi.org/10.48550/arXiv.2310.03714 DSPy : Compiling...

2023 doi arXiv
[31]

Itay Levy, Ben Bogin, and Jonathan Berant. 2023. https://doi.org/10.18653/v1/2023.acl-long.78 Diverse Demonstrations Improve In -context Compositional Generalization . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ( Volume 1: Long P...

2023 doi
[32]

Xiang Lisa Li and Percy Liang. 2021. https://doi.org/10.18653/v1/2021.acl-long.353 Prefix- Tuning : Optimizing Continuous Prompts for Generation . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer...

2021 doi
[33]

Zekun Li, Baolin Peng, Pengcheng He, Michel Galley, Jianfeng Gao, and Xifeng Yan. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/hash/c5601d99ed028448f29d1dae2e4a926d-Abstract-Conference.html Guiding Large Language Models via Directional Stimulus Prompting . Advan...

2023
[34]

Andy Liu, Mona Diab, and Daniel Fried. 2024 a . https://doi.org/10.48550/arXiv.2405.20253 Evaluating Large Language Model Biases in Persona - Steered Generation . arXiv preprint

2024 doi arXiv
[35]

Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022 a . https://doi.org/10.18653/v1/2022.deelio-1.10 What Makes Good In - Context Examples for GPT -3? In Proceedings of Deep Learning Inside Out ( DeeLIO 2022): The 3rd Workshop on Knowledg...

2022 doi
[36]

Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024 b . https://doi.org/10.1162/tacl_a_00638 Lost in the Middle : How Language Models Use Long Contexts . Transactions of the Association for Computational Linguistics...

2024 doi
[37]

Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. https://doi.org/10.1145/3560815 Pre-train, Prompt , and Predict : A Systematic Survey of Prompting Methods in Natural Language Processing . ACM Computing Surveys, 55(9):195:1--195:35

2023 doi
[38]

Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2022 b . https://doi.org/10.18653/v1/2022.acl-short.8 P- Tuning : Prompt Tuning Can Be Comparable to Fine -tuning Across Scales and Tasks . In Proceedings of the 60th Annual Meeting of the Ass...

2022 doi
[39]

Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. https://doi.org/10.18653/v1/2022.acl-long.556 Fantastically Ordered Prompts and Where to Find Them : Overcoming Few - Shot Prompt Order Sensitivity . In Proceedings of the 60th Annual Meeting of...

2022 doi
[40]

Lundberg, Gabriel G

Scott M. Lundberg, Gabriel G. Erion, Hugh Chen, Alex J. DeGrave, Jordan M. Prutkin, Bala Nair, Ronit Katz, Jonathan Himmelfarb, Nisha Bansal, and Su - In Lee. 2020. https://doi.org/10.1038/s42256-019-0138-9 From local explanations to global understanding with explainable AI fo...

2020 doi
[41]

Lundberg and Su - In Lee

Scott M. Lundberg and Su - In Lee. 2017. https://proceedings.neurips.cc/paper/2017/hash/8a20a8621978632d76c43dfd28b67767-Abstract.html A unified approach to interpreting model predictions . In Advances in Neural Information Processing Systems 30: Annual Conference on Neural In...

2017
[42]

Zeeshan Memon, Muhammad Arham, Adnan Ul-Hasan, and Faisal Shafait. 2024. https://openreview.net/forum?id=d0jQuZe6k0 LLM - Informed Discrete Prompt Optimization

2024
[43]

Marius Mosbach, Tiago Pimentel, Shauli Ravfogel, Dietrich Klakow, and Yanai Elazar. 2023. https://doi.org/10.18653/v1/2023.findings-acl.779 Few-shot Fine -tuning vs. In -context Learning : A Fair Comparison and Evaluation . In Findings of the Association for Computational Ling...

2023 doi
[44]

Maximilian Muschalik, Hubert Baniecki, Fabian Fumagalli, Patrick Kolpaczki, Barbara Hammer, and Eyke Hüllermeier. 2025. https://proceedings.neurips.cc/paper_files/paper/2024/hash/eb3a9313405e2d4175a5a3cfcd49999b-Abstract-Datasets_and_Benchmarks_Track.html shapiq: Shapley Inter...

2025
[45]

Maximilian Muschalik, Fabian Fumagalli, Barbara Hammer, and Eyke H \" u llermeier. 2024. https://doi.org/10.1609/AAAI.V38I13.29352 Beyond treeshap: Efficient computation of any-order shapley interactions for tree ensembles . In Thirty-Eighth AAAI Conference on Artificial Intel...

2024 doi
[46]

Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. https://doi.org/10.18653/v1/2021.acl-long.416 StereoSet : Measuring stereotypical bias in pretrained language models . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Intern...

2021 doi
[47]

Joan Plepi, Charles Welch, and Lucie Flek. 2024. https://aclanthology.org/2024.findings-acl.387 Perspective Taking through Generating Responses to Conflict Situations . In Findings of the Association for Computational Linguistics ACL 2024 , pages 6482--6497, Bangkok, Thailand ...

2024
[48]

Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. https://doi.org/10.3115/v1/S14-2004 SemEval -2014 Task 4: Aspect Based Sentiment Analysis . In Proceedings of the 8th International Workshop on Semantic Eva...

2014 doi
[49]

Maximus Powers, Umang Mavani, Harshitha Reddy Jonala, Ansh Tiwari, and Hua Wei. 2024. https://doi.org/10.48550/arXiv.2410.08388 GUS - Net : Social Bias Classification in Text with Generalizations , Unfairness , and Stereotypes . arXiv preprint

2024 doi
[50]

Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.378 Measuring and Narrowing the Compositionality Gap in Language Models . In Findings of the Association for Computational Linguistics : EMNLP 2...

2023 doi
[51]

Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. https://openai.com/blog/language-unsupervised/ Improving Language Understanding by Generative Pre - Training

2018
[52]

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf Language Models are Unsupervised Multitask Learners . OpenAI blog, 1(8)

2019
[53]

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. http://jmlr.org/papers/v21/20-074.html Exploring the Limits of Transfer Learning with a Unified Text -to- Text Transformer . Journal of Machine L...

2020
[54]

Nils Reimers and Iryna Gurevych. 2019. https://doi.org/10.18653/v1/D19-1410 Sentence- BERT : Sentence Embeddings using Siamese BERT - Networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference...

2019 doi
[55]

Tal Ridnik, Emanuel Ben-Baruch, Nadav Zamir, Asaf Noy, Itamar Friedman, Matan Protter, and Lihi Zelnik-Manor. 2021. Asymmetric loss for multi-label classification. In Proceedings of the IEEE / CVF international conference on computer vision ( ICCV ) , pages 82--91

2021
[56]

Benedek Rozemberczki, Lauren Watson, P \' e ter Bayer, Hao - Tsung Yang, Oliver Kiss, Sebastian Nilsson, and Rik Sarkar. 2022. https://doi.org/10.24963/IJCAI.2022/778 The shapley value in machine learning . In Proceedings of the Thirty-First International Joint Conference on A...

2022 doi
[57]

Smith, and Yejin Choi

Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020. https://www.aclweb.org/anthology/2020.acl-main.486 Social Bias Frames : Reasoning about Social and Power Implications of Language . In Proceedings of the 58th Annual Meeting of the Ass...

2020
[58]

Timo Schick, Sahana Udupa, and Hinrich Schütze. 2021. https://doi.org/10.1162/tacl_a_00434 Self- Diagnosis and Self - Debiasing : A Proposal for Reducing Corpus - Based Bias in NLP . Transactions of the Association for Computational Linguistics, 9:1408--1424

2021 doi
[59]

L. S. Shapley. 1953. A Value for n-Person Games . In Contributions to the Theory of Games (AM-28), Volume II, pages 307--318. Princeton University Press

1953
[60]

Emily Sheng, Josh Arnold, Zhou Yu, Kai-Wei Chang, and Nanyun Peng. 2021. https://doi.org/10.48550/arXiv.2104.08728 Revealing Persona Biases in Dialogue Systems . arXiv preprint

2021 doi arXiv
[61]

Kashun Shum, Shizhe Diao, and Tong Zhang. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.811 Automatic Prompt Augmentation and Selection with Chain -of- Thought from Labeled Data . In Findings of the Association for Computational Linguistics : EMNLP 2023 , pages 12113--...

2023 doi
[62]

Maximilian Spliethöver, Sai Nikhil Menon, and Henning Wachsmuth. 2024. https://aclanthology.org/2024.findings-acl.553 Disentangling Dialect from Social Bias via Multitask Learning to Improve Fairness . In Findings of the Association for Computational Linguistics ACL 2024 , pag...

2024
[63]

Maximilian Spliethöver and Henning Wachsmuth. 2020. https://www.aclweb.org/anthology/2020.argmining-1.9 Argument from Old Man 's View : Assessing Social Bias in Argumentation . In Proceedings of the 7th Workshop on Argument Mining , pages 76--87, Online. Association for Comput...

2020
[64]

Maja Stahl, Leon Biermann, Andreas Nehring, and Henning Wachsmuth. 2024. https://aclanthology.org/2024.bea-1.23/ Exploring LLM prompting strategies for joint essay scoring and feedback generation . In Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educa...

2024
[65]

Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. https://doi.org/10.18653/v1/N19-1421 CommonsenseQA : A Question Answering Challenge Targeting Commonsense Knowledge . In Proceedings of the 2019 Conference of the North American Chapter of the Associatio...

2019 doi
[66]

Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Q...

2022 doi arXiv
[67]

Jacob-Junqi Tian, David Emerson, Sevil Zanjani Miyandoab, Deval Pandya, Laleh Seyyed-Kalantari, and Faiza Khan Khattak. 2024. https://doi.org/10.48550/arXiv.2306.04735 Soft-prompt Tuning for Large Language Models to Evaluate Bias . arXiv preprint

2024 doi arXiv
[68]

Paulina Toro Isaza, Guangxuan Xu, Toye Oloko, Yufang Hou, Nanyun Peng, and Dakuo Wang. 2023. https://doi.org/10.18653/v1/2023.acl-long.359 Are Fairy Tales Fair ? Analyzing Gender Bias in Temporal Narrative Event Chains of Children 's Fairy Tales . In Proceedings of the 61st An...

2023 doi
[69]

Che - Ping Tsai, Chih - Kuan Yeh, and Pradeep Ravikumar. 2023. Faith-Shap: The Faithful Shapley Interaction Index . Journal of Machine Learning Research, 24(94):1--42

2023
[70]

Le, and Denny Zhou

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html Chain-of- Thought Prompting Elicits Rea...

2022
[71]

Xinchao Xu, Zeyang Lei, Wenquan Wu, Zheng-Yu Niu, Hua Wu, and Haifeng Wang. 2023. https://doi.org/10.18653/v1/2023.findings-acl.90 Towards Zero - Shot Persona Dialogue Generation with In - Context Learning . In Findings of the Association for Computational Linguistics : ACL 20...

2023 doi
[72]

Sohee Yang, Jonghyeon Kim, Joel Jang, Seonghyeon Ye, Hyunji Lee, and Minjoon Seo. 2024. https://doi.org/10.1162/tacl_a_00666 Improving Probability -based Prompt Selection Through Unified Evaluation and Analysis . Transactions of the Association for Computational Linguistics, 1...

2024 doi
[73]

Zamfirescu-Pereira, Richmond Y

J.D. Zamfirescu-Pereira, Richmond Y. Wong, Bjoern Hartmann, and Qian Yang. 2023. https://doi.org/10.1145/3544548.3581388 Why Johnny Can ’t Prompt : How Non - AI Experts Try (and Fail ) to Design LLM Prompts . In Proceedings of the 2023 CHI Conference on Human Factors in Comput...

2023
[74]

Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022. https://doi.org/10.48550/arXiv.2210.03493 Automatic Chain of Thought Prompting in Large Language Models . arXiv preprint

2022 doi arXiv
[75]

Le, and Ed H

Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V. Le, and Ed H. Chi. 2022. Least-to- Most Prompting Enables Complex Reasoning in Large Language Models

2022
[76]

Hwang, Swabha Swayamdipta, and Maarten Sap

Xuhui Zhou, Hao Zhu, Akhila Yerukola, Thomas Davidson, Jena D. Hwang, Swabha Swayamdipta, and Maarten Sap. 2023 a . https://doi.org/10.18653/v1/2023.findings-acl.392 COBRA Frames : Contextual Reasoning about Effects and Harms of Offensive Statements . In Findings of the Associ...

2023 doi
[77]

Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023 b . https://doi.org/10.48550/arXiv.2211.01910 Large Language Models Are Human - Level Prompt Engineers . arXiv preprint

2023 doi arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Alberto Mario Ceballos Arroyo, Monica Munnangi, Jiuding Sun, Karen Y. C. Zhang, Denis Jered McInerney, Byron C. Wallace, and Silvio Amir. 2024. http://arxiv.org/abs/2407.09429 Open ( Clinical ) LLMs are Sensitive to Instruction Phrasings . arXiv preprint

work page Pith review arXiv 2024

[4] [4]

Fatemeh Torabi Asr, Mohammad Mazraeh, Alexandre Lopes, Vasundhara Gautam, Junette Gonzales, Prashanth Rao, and Maite Taboada. 2021. https://doi.org/10.1371/journal.pone.0245533 The Gender Gap Tracker : Using Natural Language Processing to measure gender bias in media . PLOS ONE, 16(1)

work page doi:10.1371/journal.pone.0245533 2021

[5] [5]

Tilman Beck, Hendrik Schuff, Anne Lauscher, and Iryna Gurevych. 2024. https://aclanthology.org/2024.eacl-long.159 Sensitivity, Performance , Robustness : Deconstructing the Effect of Sociodemographic Prompting . In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics ( Volume 1: Long Papers ) , pages ...

work page 2024

[6] [6]

Gormley, and Graham Neubig

Amanda Bertsch, Maor Ivgi, Uri Alon, Jonathan Berant, Matthew R. Gormley, and Graham Neubig. 2024. https://doi.org/10.48550/arXiv.2405.00200 In- Context Learning with Long - Context Models : An In - Depth Exploration . arXiv preprint

arXiv doi:10.48550/arxiv.2405.00200 2024

[7] [7]

Sebastian Bordt and Ulrike von Luxburg. 2023. From Shapley Values to Generalized Additive Models and back . In International Conference on Artificial Intelligence and Statistics (AISTATS 2023) , volume 206 of Proceedings of Machine Learning Research, pages 709--745. PMLR

work page 2023

[8] [8]

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

arXiv 2020