S2LPP: Small-to-Large Prompt Prediction across LLMs

Liang Cheng; Mark Steedman; Tianyi Li; Zhaowei Wang

REVIEW 3 major objections 4 minor 18 references

S2LPP: Small-to-Large Prompt Prediction across LLMs

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A small language model can pick the prompt template that performs near-optimally on a much larger model, because prompt preferences are consistent across model sizes.

desk verdict Useful practical pipeline built on a consistency claim that is real but statistically overstated; the RRoP results carry the paper. read the letter →

arxiv 2505.20097 v1 pith:LKWS5HYH submitted 2025-05-26 cs.CL

Liang Cheng , Tianyi LI , Zhaowei Wang , Mark Steedman This is my paper · ORCID

classification cs.CL

keywords promptengineeringselectionsmall-to-largetransferpreferenceconsistencyquestionansweringnaturallanguageinferenceretrieval-augmentedgenerationchain-of-thoughtprompting

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that large language models of different sizes—within the same family and usually across families—prefer the same prompt templates, so the best prompt found with a small, cheap model is near-optimal for a big, expensive one. This is tested on open-domain question answering (Google-RE, T-REx) and natural-language inference (Levy/Holt) across fourteen models. Building on this consistency, the paper proposes S2LPP, a pipeline that uses a small selection model to score candidate prompts on a few examples and then applies the winning template to the target model. The reported result is that S2LPP matches the performance of oracle prompts on GPT-3.5 while cutting the computational cost of prompt search, and the same idea is extended to context selection in retrieval-augmented generation and chain-of-thought selection in arithmetic reasoning. If the consistency holds, prompt engineering for new or fast-evolving large models can largely be delegated to small models.

What carries the argument

The mechanism that carries the argument is prompt-preference consistency across model sizes, formalized as the Proportion of Optimal-Prompt Matches (POPM). For each relation or task, the optimal prompt is the one with the highest accuracy on the evaluation set; POPM is the fraction of relations where two models share that same argmax prompt. S2LPP turns this regularity into a three-step pipeline: generate candidate natural-language prompts with a strong generator, score each candidate with a small selection model on a few hundred examples, and run the winning prompt on the large target model. The Recovery Rate of Performance (RRoP) then reports how much of the target model's oracle-prompt accuracy the selected prompt recovers.

What would settle it

Take a fixed task and a set of candidate prompts, and estimate each prompt's accuracy with enough samples and repeated seeds to obtain confidence intervals. If, for a substantial share of relations, the best and second-best prompts are statistically indistinguishable while different model sizes pick different prompts, the consistency claim would lose its footing. A simpler falsifier: find one relation where a small and a large model from the same family reliably prefer different prompts under tight intervals; that single counterexample breaks the within-family claim.

Watch

Extended reading notes

Core claim

The central discovery is that the prompt template achieving the highest accuracy on a development set is stable across model sizes within a family, and often across families: the same template is optimal for small, medium, and large versions of LLaMA, DeepSeek, Vicuna, and related models. The paper quantifies this with the Proportion of Optimal-Prompt Matches (POPM), counting the fraction of relations where two models pick the same best prompt, and reports high POPM values even when comparing smaller models to GPT-3.5. On top of this, S2LPP selects the prompt with the highest accuracy on a small development set using a small model, then uses that prompt for the target model. The paper measures success with the Recovery Rate of Performance (RRoP), and reports that the selected prompts outperform first-generated, average, and manual prompt baselines, approaching the oracle upper bound. The consistency is also shown to extend to choosing which retrieved paragraph to use as context in RAG and which chain-of-thought examples to include in arithmetic reasoning.

Load-bearing premise

The paper assumes that the single prompt with the highest measured accuracy on the development set is a stable, non-noisy property of each model, so that overlapping best-prompt choices reflect genuine preference consistency rather than ties broken by sampling noise.

Editorial extensions

If this is right

Prompt engineering for a new large model can be done on a small model from the same or a different family, drastically reducing the compute needed to find a good template.
The best prompt found by S2LPP approaches the oracle prompt found by searching directly on the target model, so cost savings come with only a small accuracy gap.
The consistency transfers beyond prompt templates to other choices, such as which retrieved paragraph to use as context and which chain-of-thought examples to present.
If preferences trace back to pretraining distributions, the same selected prompts may stay performant for newly released large models without re-running the search.
The approach cuts the cost of chain-of-thought prompt selection by sixty percent relative to the baseline in the arithmetic reasoning experiment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the consistency is driven by shared pretraining data, then a measurable prediction follows: POPM should be higher between models trained on overlapping corpora than between models trained on deliberately disjoint ones, and a test of that comparison would sharpen the claim.
The argmax-based definition of the optimal prompt could hide near-ties: when the top prompts differ in accuracy only within sampling noise, the reported consistency may overstate how strongly models agree. A bootstrap or confidence-interval analysis of the best-prompt choice would clarify this.
The paper relies on ChatGPT to generate the candidate prompts; if that generator biases templates toward OpenAI-style phrasing, the consistency might partly reflect a shared generator rather than a property of the task. Using open-source generators for the same pipeline would separate these explanations.
The same small-to-large selection scheme could serve as a cheap diagnostic for prompt sensitivity across model families, letting practitioners spot brittle templates without running a full grid search on the large model.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my read. The genuinely useful part of this paper is the S2LPP pipeline: using a small model to pick a prompt for a large model, with evaluation across fourteen models and extensions to RAG context selection and CoT. That's a practical advance, and the numbers back it up—LLaMA-2-7B as selector recovers about 94-95% of oracle accuracy on QA and 92% on NLI. The code is public, and the authors cite the prior consistency work (Si et al., Wang et al., Li et al.) openly. The consistency finding itself is not new, but the breadth of the cross-family study is.

The weak spot is the central consistency claim. POPM is exact argmax agreement over a 100-sample dev set, with no confidence intervals. Near-ties among prompts are likely, so the argmax can flip on noise. Table 2 reports POPM relative to GPT-3.5, not within-family pairs, which is what the §3.4 language claims. The direct within-family evidence is in a few figures, and Appendix D admits LLaMA-2-70B breaks the pattern on PlaceOfBirth. So 'consistent preference' is overstated.

That said, the applied method survives this critique. RRoP measures performance recovery rather than exact prompt identity, and near-tied prompts can still give near-oracle accuracy. The paper would be more honest if it led with RRoP and the pipeline, and treated exact argmax consistency as a secondary observation. As written, the consistency section needs statistical strengthening, but the core method is solid.

I'd send this to peer review. It's a reproducible empirical study with a deployable technique. A serious referee would ask for confidence intervals, within-family POPM, and more target models than GPT-3.5. Those are fixable. The paper isn't a breakthrough, but it's a useful contribution to prompt engineering cost reduction.

Referee Report

3 major / 4 minor

Summary. The paper studies whether LLMs of different sizes prefer the same prompt templates. In §3, the authors evaluate 10 automatically generated prompts per relation on open-domain QA (Google-RE, T-REX) and 5 manual prompts on NLI (Levy/Holt) across LLaMA-2, LLaMA-3, Vicuna, and DeepSeek-R1 families. They define the Proportion of Optimal-Prompt Matches (POPM) as the fraction of relations where two models have the same single highest-accuracy prompt, and report high POPM values mostly against GPT-3.5 as the reference. In §4, they propose S2LPP: use a small 'selection' model to choose the best prompt from ChatGPT-generated candidates on a 100-example development set, then run the chosen prompt on a larger target model (GPT-3.5). They report accuracy gains over baselines and Recovery Rate of Performance (RRoP) close to the oracle, with extensions to RAG context selection and CoT prompt selection.

Significance. If the consistency claim holds, S2LPP would be a practical, low-cost way to avoid running large models on many candidate prompts. The proposed pipeline is simple, reproducible (the authors release a code link), and the empirical results in Table 3 and Figure 4 suggest the method works in the tested setting. The extensions to RAG and CoT broaden the scope. However, the headline consistency result is currently supported by an exact-argmax metric without uncertainty quantification, and the within-family claim is not the quantity tabulated in Table 2. These issues are fixable and do not undermine the applied S2LPP results, which measure performance recovery rather than exact prompt identity.

major comments (3)

[§3.3–3.4] The POPM metric is defined as exact agreement of the single highest-accuracy prompt, but the paper reports no confidence intervals, significance tests, or tie handling. On the full datasets, prompt accuracies for the top candidates may be within sampling noise, so the argmax can flip without any genuine change in preference. Please report bootstrap confidence intervals for POPM and/or a tie-aware metric (e.g., the fraction of relations where the small model's optimal prompt is in the target model's top-k). Without this, the statement in §3.4 that 'These findings demonstrate a consistent preference...' is stronger than the evidence supports.
[§3.4, Table 2, Appendix E] The POPM scores in Table 2 are computed against GPT-3.5, not between small and large versions of the same family, so the within-family claim in §3.4 is not the quantity tabulated. The direct within-family evidence is limited to Google-RE (3 relations; Figures 1 and 6) and the T-REX table in Appendix E (Table 7). Table 7 itself shows several mismatches between LLaMA-2-7B and LLaMA-2-13B (e.g., 'named after', 'field of work', 'occupation', 'has part', 'capital of', 'play', 'location', 'headquarters location'), and Appendix D explicitly concedes an exception for LLaMA-2-70B on PlaceOfBirth. Please present within-family POPM or per-relation match counts and revisit the strength of the conclusion.
[§4.2, §4.4, Limitations] All S2LPP experiments use GPT-3.5 as the target model, as the Limitations section acknowledges. The sentence in §4.4 that 'other language models can also be effectively utilized as target models' is not supported by any experiment, and the abstract's 'across fourteen LLMs' refers to selection models, not target models. Since the title and motivation promise small-to-large prediction, the applicability to other large targets should either be demonstrated or the claims should be tempered to the GPT-3.5 setting.

minor comments (4)

[§3.2] There is a typo: 'Natutral Language Inference' should be 'Natural Language Inference'.
[Appendix B] The list of manual prompts contains 'prompt 3' twice, and the following sentence refers to 'LLaMA-7B' and 'LLaMA-13B' while the paper otherwise uses 'LLaMA-2'; please correct the numbering and the model names.
[Table 7] The table caption mentions blue/red highlighting for matches/mismatches, but this is not visible in the text version; please use an accessible format such as bold or symbols.
[§4.2.3] The first-generated baseline is justified by the statement that 'the first prompt also tends to be the most favored prompt'; this claim is not supported by data or a citation and should be either substantiated or removed.

Circularity Check

0 steps flagged · score 2.0 of 10

No derivation reduces to its inputs; the consistency finding and S2LPP transfer are empirical, with only minor non-load-bearing self-citations.

full rationale

The paper is an empirical study rather than a formal derivation: the central claim is that argmax-optimal prompt templates agree across model sizes, measured by POPM, and the S2LPP method transfers a small model's development-set prompt choice to a larger target model. No fitted parameter is renamed as a prediction, and no equation defines the reported accuracy or RRoP in terms of the consistency claim. The closest structural concerns are (1) self-citations to overlap-author prior work for the NLI prompt templates and the directional subset, and (2) the reuse of the same datasets for the consistency analysis in §3 and the S2LPP evaluation in §4. These are not load-bearing circularities: the cited prior work supplies experimental artifacts rather than the asserted consistency premise, and S2LPP is evaluated by held-out accuracy on GPT-3.5 after selection is done on a separate 100-sample development set, so the result is not forced by construction. The methodological limitation that POPM uses exact argmax agreement without confidence intervals, and that Table 2 reports matches against GPT-3.5 rather than within-family pairs, is a robustness concern, not a circularity.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new theoretical entities; it relies on standard datasets and models. The free parameters are experimental choices (dev size, candidate count, temperature) that are not fitted or varied, and the axioms are practical assumptions about measurement reliability and representativeness.

free parameters (3)

development set size = 100 samples
Chosen by hand for all tasks; no sensitivity analysis is reported, and the stability of the optimal prompt ranking at this sample size is not established.
number of candidate prompts = 10
The paper fixes 10 ChatGPT-generated candidates per relation/task without testing whether the selection result is robust to the number or diversity of candidates.
ChatGPT generation temperature = 0
Fixed to 0 for reproducibility, but this is a hand-chosen setting and the exact generated prompt set for T-REX is not fully released.

assumptions (3)

domain assumption String-containment accuracy is a valid measure of QA correctness.
Used to rank prompts in §3.3; prior work (Petroni et al. 2019) uses Precision@1, and the paper argues equivalence, but exact-match on entities may miss semantically correct answers.
domain assumption The 100-sample development set yields a reliable estimate of each prompt's true performance.
The entire S2LPP selection step in §4.1 depends on this; with noisy small samples, the argmax prompt could be arbitrary.
domain assumption The generated prompt candidates are representative of the space of natural language prompts.
Conclusion (a) generalizes to prompt preference, but only for the specific ChatGPT-generated candidate sets; no sensitivity to prompt generation method is tested.

how reviews work

0 comments

Cite this review

Pith. "Pith review of S2LPP: Small-to-Large Prompt Prediction across LLMs." pith.science (2026). https://pith.science/paper/LKWS5HYH

@misc{pith2026250520097,
  author       = {Pith},
  title        = {Pith review of: S2LPP: Small-to-Large Prompt Prediction across LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LKWS5HYH}},
  note         = {Machine review of arXiv:2505.20097}
}

read the original abstract

The performance of pre-trained Large Language Models (LLMs) is often sensitive to nuances in prompt templates, requiring careful prompt engineering, adding costs in terms of computing and human effort. In this study, we present experiments encompassing multiple LLMs variants of varying sizes aimed at probing their preference with different prompts. Through experiments on Question Answering, we show prompt preference consistency across LLMs of different sizes. We also show that this consistency extends to other tasks, such as Natural Language Inference. Utilizing this consistency, we propose a method to use a smaller model to select effective prompt templates for a larger model. We show that our method substantially reduces the cost of prompt engineering while consistently matching performance with optimal prompts among candidates. More importantly, our experiment shows the efficacy of our strategy across fourteen LLMs and its applicability to a broad range of NLP tasks, highlighting its robustness

Figures

Figures reproduced from arXiv: 2505.20097 by the authors.

**Figure 2.** The figure illustrates the accuracy of differ [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** The workflow of S2LPP on open-domain QA: [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: The Recovery Rate of Performance (RRoP) across various LLMs on QA tasks. RRoP scores exceeding [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Accuracy of different models in the prompt [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

**Figure 6.** Figure 6: The figure illustrates the accuracy of different prompts across Vicuna, StableBeluga and LLaMA-2-chat [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 16 canonical work pages

[1]

If [premise], then [hypothesis]

prompt0: "If [premise], then [hypothesis]."

work page
[2]

[P], so [H]

prompt 1: "[P], so [H]."

work page
[3]

[P] entails [H]

prompt 2: "[P] entails [H]"

work page
[4]

We present the man- ually crafted prompt templates below and highlight the best-performed prompt template on the target model, GPT-3.5, in bold

for textual entailments. We present the man- ually crafted prompt templates below and highlight the best-performed prompt template on the target model, GPT-3.5, in bold

work page
[5]

[H], because [P]

prompt 3: "[H], because [P]." The prompt0 outperforms another prompt template in GPT-3.5 and LLaMA-7B, LLaMA-13B, and Vicuna-13B models. The prompt 0 achieves the second highest accuracy among other templates on Vicuna-7B, where the optimal prompt is prompt3. C Automatically Generated Prompt Templates from ChatGPT As discussed in §4, we introduce the S2LP...

work page
[6]

If [P] is true, must [H] also be true?

prompt5: "If [P] is true, must [H] also be true?"

work page
[7]

Does the truth of [P] guarantee the truth of [H]?

prompt6: "Does the truth of [P] guarantee the truth of [H]?"

work page
[8]

[P], which means that [H]

prompt 3: "[P], which means that [H]."

work page

Show all 18 references

[9]

Can we derive [H] from [P]?

prompt 8: "Can we derive [H] from [P]?"
[10]

Can [H] be inferred from [P]?

prompt 0: "Can [H] be inferred from [P]?" 2.prompt 1: "Does [P] entail [H]?"
[11]

Is it true that [P] leads to [H]?

prompt 2: "Is it true that [P] leads to [H]?"
[12]

Is [H] a necessary consequence of [P]?

prompt3: "Is [H] a necessary consequence of [P]?"
[13]

Do we conclude [H] from [P]?

prompt 4: "Do we conclude [H] from [P]?"
[16]

Is [H] a logical consequence of [P]?

prompt7: "Is [H] a logical consequence of [P]?"
[18]

Is [H] implied by [P]?

prompt 9: "Is [H] implied by [P]?" We also present the generated prompt templates for open-domain QA in Table 6. In this table, the optimal prompt templates for the target model, GPT-3.5, are highlighted in bold. D Consistency across Different Models Besides the LLaMA-3 and De...

2019
[2021]

DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z

Training verifiers to solve math word prob- lems.arXiv preprint arXiv:2110.14168. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihon...

arXiv
[2022]

Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller

Training language models to follow instruc- tions with human feedback.Advances in neural in- formation processing systems, 35:27730–27744. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language Models as...

2019 arXiv
[2023]

Simran Arora, Avanika Narayan, Mayee F Chen, Lau- rel Orr, Neel Guha, Kush Bhatia, Ines Chami, and Christopher Re

Falcon-40B: an open large language model with state-of-the-art performance. Simran Arora, Avanika Narayan, Mayee F Chen, Lau- rel Orr, Neel Guha, Kush Bhatia, Ines Chami, and Christopher Re. 2022. Ask me anything: A sim- ple strategy for prompting language models. InThe Eleven...

2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

If [premise], then [hypothesis]

prompt0: "If [premise], then [hypothesis]."

work page

[2] [2]

[P], so [H]

prompt 1: "[P], so [H]."

work page

[3] [3]

[P] entails [H]

prompt 2: "[P] entails [H]"

work page

[4] [4]

We present the man- ually crafted prompt templates below and highlight the best-performed prompt template on the target model, GPT-3.5, in bold

for textual entailments. We present the man- ually crafted prompt templates below and highlight the best-performed prompt template on the target model, GPT-3.5, in bold

work page

[5] [5]

[H], because [P]

prompt 3: "[H], because [P]." The prompt0 outperforms another prompt template in GPT-3.5 and LLaMA-7B, LLaMA-13B, and Vicuna-13B models. The prompt 0 achieves the second highest accuracy among other templates on Vicuna-7B, where the optimal prompt is prompt3. C Automatically Generated Prompt Templates from ChatGPT As discussed in §4, we introduce the S2LP...

work page

[6] [6]

If [P] is true, must [H] also be true?

prompt5: "If [P] is true, must [H] also be true?"

work page

[7] [7]

Does the truth of [P] guarantee the truth of [H]?

prompt6: "Does the truth of [P] guarantee the truth of [H]?"

work page

[8] [8]

[P], which means that [H]

prompt 3: "[P], which means that [H]."

work page