Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

The Order Effect: Investigating Prompt Sensitivity to Input Order in LLMs

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read API LLMs still flip answers when prompt order changes.

desk verdict Transparent but statistically thin measurement study extending a known phenomenon to newer models; the qualitative claim is plausible, the quantitative deltas are not. read the letter →

arxiv 2502.04134 v2 pith:6A7UGZZ6 submitted 2025-02-06 cs.CL

Bryan Guan , Tanya Roosta , Peyman Passban , Mehdi Rezagholizadeh This is my paper · ORCID
classification cs.CL
keywords inputorderpromptsensitivitydependencelargelanguagemodelsfew-shotpromptingmultiple-choicequestionsrelevancejudgmentLLMreliability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that current LLMs reached through APIs still change their answers when the order of the input pieces is shuffled, even on standard benchmarks. Across paraphrasing, relevance judgment, and multiple-choice tasks, the authors find that randomizing sentence, passage, or option order usually lowers F1 scores, with drops up to roughly 13 percent depending on model and task. Few-shot examples sometimes reduce the gap but do not remove it, and in one comparison-preference task shuffling actually improved scores, so the effect is not a simple uniform penalty. A sympathetic reader would care because order-dependent answers mean that evaluation pipelines, automated judges, and high-stakes advice built on these models can flip with cosmetic input changes.

What carries the argument

The experimental machinery is a four-condition comparison: each task is run zero-shot and few-shot, each in original and randomly shuffled order, and the order effect is quantified as the percentage change in F1 between shuffled and original conditions. The shuffle is the intervention; it randomly reassigns positions of sentences, passages, or options while keeping content identical. The delta formula, $\Delta = (F1_{\text{shuffled}} - F1_{\text{original}}) / F1_{\text{shuffled}} \times 100$, makes the comparison explicit, and the paper uses the consistency of negative deltas across tasks as evidence for an autoregressive explanation.

What would settle it

Rerun all five tasks with at least ten independent repetitions per condition using identical decoding settings and report the distribution of deltas; if 95% confidence intervals around the 1-3% F1 gaps include zero for multiple tasks, those particular order effects would not survive, though the larger MSMARCO and DeepSeek drops might. A second test would compare shuffled-condition variance against repeated same-order runs to separate genuine order effects from decoding stochasticity.

Watch

Extended reading notes

Core claim

The central claim is that order sensitivity persists in API-accessible LLMs despite recent advances. Using five datasets and three models, the authors compare zero-shot original-order prompts against shuffled-order prompts, and few-shot versions of both, measuring relative F1 change. In MRPC, MSMARCO, MMLU, and MedMCQA, shuffling generally degrades performance; the largest consistent drops appear on the longer-input MSMARCO task, and DeepSeek tends to be more affected than the GPT models. WebGPT is the exception, where shuffling often improves scores. The paper argues that no single task-level pattern explains the direction of the effect, and that the persistent directional decline supports an autoregressive, order-dependent processing account.

Load-bearing premise

The load-bearing premise is that the F1 differences between original and shuffled runs are real effects of order, not ordinary run-to-run noise, since each condition was evaluated once with no random seeds, repeated runs, or confidence intervals.

Editorial extensions

If this is right

  • Benchmark scores that report a single number without order sensitivity can overstate model reliability.
  • LLM-based evaluation and judging pipelines may return different verdicts when candidate or option order changes.
  • Few-shot prompting cannot be assumed to fix order dependence; in some settings it made gaps larger.
  • Input length amplifies the vulnerability, so long-context applications deserve extra robustness checks.
  • The WebGPT counterexample implies that some tasks can improve under shuffling, so mitigation should be task-aware.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because each configuration was measured once without seeds or confidence intervals, some of the smaller 1-3% deltas could reflect sampling noise; the paper's strongest evidence is the consistent direction across many task-model combinations, not any single number.
  • If the autoregressive explanation is right, then training or fine-tuning with order-randomized prompts should shrink the effect; that is a testable extension the paper does not run.
  • The category-level MMLU pattern, where text-comprehension subjects worsened and reasoning-heavy subjects held up, hints that shallow surface-form reliance drives part of the effect; a direct test would compare matched questions with high versus low lexical overlap.
  • A practical consequence not developed in the paper: order sensitivity could be measured cheaply as a standard robustness metric by reporting the spread of accuracy across a small set of shuffles.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies whether three API-accessible LLMs (GPT-4o, GPT-4o mini, DeepSeek R1-Distill-Llama-70B) are sensitive to the order of input elements across five tasks: paraphrase detection (MRPC), passage relevance (MSMARCO), multiple-choice QA (MMLU, MedMCQA), and pairwise preference comparison (WebGPT). For each task, the authors compare performance in zero-shot and few-shot settings under original versus shuffled input order. The reported results show F1 declines after shuffling on MRPC, MSMARCO, MMLU, and MedMCQA for most models and settings, while WebGPT mostly shows improvements after shuffling. The paper concludes that input order significantly affects performance and that shuffled inputs lead to measurable declines, with few-shot prompting providing only partial and inconsistent mitigation.

Significance. If the central claim were established, the paper would provide a useful, up-to-date confirmation that recent closed-source LLMs remain order-sensitive on standard benchmarks, complementing earlier work by Sclar et al. (2024), Zheng et al. (2024), and Pezeshkpour & Hruschka (2023). The experimental design is transparent: the authors use public datasets, report their prompts, and cover a reasonable range of tasks and models. The paper also explicitly situates itself in the practical scenario of non-technical users interacting through APIs, which is a relevant perspective. However, the empirical evidence as presented is not statistically supported: every reported delta is a single point estimate with no repeated runs, confidence intervals, or significance testing, and the WebGPT results contradict the paper's own 'consistent decline' narrative. The paper's value would be substantially increased by adding repeated-shuffle experiments with error bars and significance tests, and by reconciling or reinterpreting the WebGPT exception.

major comments (3)
  1. [Section 4, Tables 1-5] The central claim that shuffling 'significantly affects' performance and leads to 'measurable declines' rests entirely on single-run F1 deltas. Each configuration in Tables 1-5 is measured once, with no repeated shuffles, no random seeds, no confidence intervals, and no reported decoding temperature or sampling settings. For stochastic API models, a difference of 0.0 to 3 F1 points (e.g., MRPC few-shot GPT-4o mini and DeepSeek, MMLU zero-shot GPT-4o, MedMCQA few-shot GPT-4o) is within ordinary run-to-run variability, as the authors themselves acknowledge in citing Sclar et al. (2024), who report sizable variance for similar APIs. The paper must report repeated experiments (e.g., multiple shuffles and seeds), error bars, and a significance test before the qualitative claim of a consistent degradation can be accepted.
  2. [Section 4, Table 5 and Section 4.1] The WebGPT results are not an edge case but a direct counterexample to the paper's stated finding that 'changing the input order generally leads to performance degradation' and that 'shuffling the input sequence almost always led to decreased accuracy.' Table 5 shows large positive deltas after shuffling in most settings, including +21.05 for DeepSeek zero-shot and +10.86 for GPT-4o zero-shot. The paper acknowledges this and offers speculation about training exposure to WebGPT, but does not test this explanation or consider the simpler alternative that the metric is noisy. This inconsistency undermines the central claim; the authors should either restrict the claim to specific tasks, provide a mechanism that explains both directions, or add controls that distinguish order effects from noise.
  3. [Section 4, MMLU category analysis] The category-level analysis in the MMLU section is internally inconsistent and statistically unsupported. The text first states that categories such as abstract algebra, conceptual physics, high school mathematics, and machine learning 'showed performance declines, whereas others such as philosophy, prehistory, and world religions showed improvements.' It then concludes that 'for text-based categories (e.g. philosophy) involving reading comprehension, LLMs are more sensitive to input order' and that for 'complex, reasoning-intensive tasks (e.g. algebra), LLMs may be more resilient.' These two statements directly contradict each other (philosophy improved, algebra declined). Moreover, the category-level deltas are computed on single-run subsets of a few hundred examples with no error bars, making them unreliable. This analysis should be rewritten or removed.
minor comments (5)
  1. [Table 1] The delta formula in the caption is unusual: the relative change is computed with the shuffled F1 in the denominator rather than the original F1. This makes the delta depend on the order of subtraction in a nonstandard way; the paper should use the original-order F1 as the baseline and define the sign consistently.
  2. [Abstract and Section 1] The phrase 'measurable declines' is too strong given the lack of statistical evidence; 'numerically lower F1 scores in four of five tasks' would be more accurate. Also, in the Introduction, the example with the Lego prompt reports two responses from the ChatGPT web interface, but the paper does not state that this experiment was repeated or that the observed flip is representative.
  3. [Section 3, general] The paper never reports key decoding parameters for the API calls, such as temperature, top-p, or the number of samples per prompt. Without these, the single-run results cannot be reproduced or compared with other studies.
  4. [Appendix B] The WebGPT few-shot example is labeled 'Few-shot, shuffled order Experiment 2 - 3 shot,' but the experiment is Experiment 5 and the shuffle status is unclear. Also, the prompt contains the typo 'adament' for 'adamant.'
  5. [Conclusion] The sentence 'our experiments revealed that consistently placing the correct answer in a specific position creates a pattern that the LLM learns, impacting its subsequent predictions' appears to describe an observation not supported by any reported analysis in Section 4; either add data for this claim or remove it.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is a direct empirical measurement of order sensitivity on external benchmarks, with no fitted parameters, no derived predictors, and no load-bearing self-citations.

full rationale

The paper's central claim, that shuffling input order changes F1 scores, is an empirical finding computed from standard external datasets (MRPC, MSMARCO, MMLU, MedMCQA, WebGPT) and publicly accessible API models. There is no derivation chain in which a conclusion is defined into an input: order sensitivity is operationalized as the difference in F1 between original and shuffled prompts, and the measured deltas are presented as observations rather than as predictions from a fitted model. No parameter is fitted to a subset of the data and then relabeled as a prediction, no uniqueness theorem is imported from the authors' own prior work, and no ansatz is smuggled in via self-citation. The references to prior work (Sclar et al., Zheng et al., Pezeshkpour & Hruschka, McIlroy-Young et al.) are external and are used to motivate the study, not to justify the reported measurements. The paper itself explicitly acknowledges that no consistent pattern emerged and that the problem remains unresolved, which further confirms that the findings are not manufactured by construction. The main weaknesses, including single-run measurements without confidence intervals or decoding details, are threats to statistical validity, not circularity. Because the empirical measurements are self-contained against external benchmarks, the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no mathematical parameters, fitted constants, or invented entities. Its quantitative conclusions rest on experimental assumptions about model stability, single-run evaluation, and benchmark validity, which are listed above. These are reasonable for an exploratory study but are not explicitly verified.

assumptions (3)
  • domain assumption The named model versions accessed via API or web interface are stable and representative during the experiment period.
    The paper does not report exact model version strings or access dates except a single screenshot note; if models were updated or routed differently between conditions, the deltas could be affected. This is assumed throughout Section 4.
  • domain assumption A single inference pass per item, with unspecified decoding settings, is a valid basis for comparing original and shuffled performance.
    All tables report one run per example with no temperature, top-p, or seed information. The delta computations treat these point estimates as exact, which is a load-bearing assumption for the paper's quantitative conclusions.
  • domain assumption The datasets used as ground truth (MRPC, MSMARCO, MMLU, MedMCQA, WebGPT) are appropriate and their labels are not contaminated in a way that specifically biases order sensitivity.
    The authors speculate about WebGPT contamination in Section 4 but do not test it; overall they assume benchmark labels are reliable for measuring order effects.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Order Effect: Investigating Prompt Sensitivity to Input Order in LLMs." pith.science (2026). https://pith.science/paper/6A7UGZZ6

@misc{pith2026250204134,
  author       = {Pith},
  title        = {Pith review of: The Order Effect: Investigating Prompt Sensitivity to Input Order in LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6A7UGZZ6}},
  note         = {Machine review of arXiv:2502.04134}
}
read the original abstract

As large language models (LLMs) become integral to diverse applications, ensuring their reliability under varying input conditions is crucial. One key issue affecting this reliability is order sensitivity, wherein slight variations in the input arrangement can lead to inconsistent or biased outputs. Although recent advances have reduced this sensitivity, the problem remains unresolved. This paper investigates the extent of order sensitivity in LLMs whose internal components are hidden from users (such as closed-source models or those accessed via API calls). We conduct experiments across multiple tasks, including paraphrasing, relevance judgment, and multiple-choice questions. Our results show that input order significantly affects performance across tasks, with shuffled inputs leading to measurable declines in output accuracy. Few-shot prompting demonstrates mixed effectiveness and offers partial mitigation; however, fails to fully resolve the problem. These findings highlight persistent risks, particularly in high-stakes applications, and point to the need for more robust LLMs or improved input-handling techniques in future development.

Figures

Figures reproduced from arXiv: 2502.04134 by the authors.

Figure 1
Figure 1. How GPT-4o responds to the same question when the order of choices is reversed. The calls were made on Tuesday, May 6th, at 16:29 EST. B PROMPT EXAMPLES An example of a zero-shot, original order prompt from the MMLU dataset: MMLU, Zero-shot, Original order Question: Determine whether the polynomial in Z[x] satisfies an Eisenstein criterion for irreducibility over Q. 8xˆ3 + 6xˆ2 - 9x + 24 Options: 1) Yes, with p=2. 2… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Innate Economic Preferences of Language Models

    econ.EM 2026-07 conditional novelty 6.0 of 10

    Language models' softmax token choice is exactly a random utility model, letting logits identify preferences: twelve models show risk aversion, IIA violations, and fine-tuning can set a target risk attitude.

  2. Answer-Centric or Reasoning-Driven? Uncovering the Latent Memory Anchor in LLMs

    cs.CL 2025-06 conditional novelty 5.0 of 10

    Across ten LLMs, masking the final answer inside a complete reasoning chain causes a 26.9-point accuracy drop, evidence that models anchor to answers, not reasoning templates.

Reference graph

Works this paper leans on

13 extracted references · 13 canonical work pages · cited by 2 Pith papers

  1. [1]

    Albany is a city in Stearns County, Minnesota, United States

    For the unincorporated community in southeast Minnesota named West Albany, see West Albany, Minnesota. Albany is a city in Stearns County, Minnesota, United States. The population was 2,561 at the 2010 census. It is part of the St. Cloud Metropolitan Statistical Area. 2)

    work page 2010

  2. [3]

    Out of people who lived in different houses, 71% lived in this county

    Place of birth for U.S.-born residents: 70% of the 56307 zip code residents lived in the same house 5 years ago. Out of people who lived in different houses, 71% lived in this county. Out of people who lived in different counties, 50% lived in Minnesota. 92% of the 56307 zip code residents lived in the same house 1 year ago

    work page

  3. [4]

    Option 1

    No. Based on the given question and four options, which one is the right answer? Please respond with only “Option 1”, “Option 2”, “Option 3”, or “Option 4” as your final answer, without any additional explanation. An example of a zero-shot, shuffled order prompt from the MSMARCO dataset: 10 Work in progress MSMARCO, Zero-shot, Shuffled order Query: albany...

    work page

  4. [5]

    City of Albany, MN Demographic Information

    City of Albany, MN Zip Codes. City of Albany, MN Demographic Information. * Demographic data is based on information taken from the 2000 Census. City of Albany, MN covers 1 Area Code. City of Albany, MN covers 1 Zip Code. 15 Cities within 15 Miles of the City of Albany, MN

    work page 2000

  5. [6]

    2 Bachelor’s degree or higher: 15.4%

    For population 25 years and over in 56307: 1 High school or higher: 87.4%. 2 Bachelor’s degree or higher: 15.4%. 3 Graduate or professional degree: 3.3 4 %. Unemployed: 3. 5 2%. Mean travel time to work (commute): 23.6 minutes

    work page

  6. [7]

    Albany is a city in Stearns County, Minnesota, United States

    Sponsored Topics. Albany is a city in Stearns County, Minnesota, United States. The population was 2,561 at the 2010 census. It is part of the St. Cloud Metropolitan Statistical Area. 7)

    work page 2010

  7. [8]

    Albany is mentioned 87 times on our forum: Latest news from Albany, MN collected exclusively by city-data.com from local newspapers, TV , and radio stations

    Recent posts about Albany, Minnesota on our local forum with over 2,000,000 registered users. Albany is mentioned 87 times on our forum: Latest news from Albany, MN collected exclusively by city-data.com from local newspapers, TV , and radio stations. Ancestries: German (55.6%), Irish (10.0%), Polish (5.9%), Norwegian (5.4%), Swedish (2.8%), United States (2.6%)

    work page

  8. [9]

    2 Bachelor’s degree or higher: 15.4%

    For population 25 years and over in Albany: 1 High school or higher: 86.7%. 2 Bachelor’s degree or higher: 15.4%. 3 Graduate or professional degree: 4.4 4 %. Unemployed: 4. 5 3%. Mean travel time to work (commute): 23.0 minutes

    work page

Show all 13 references
  1. [10]

    Option 1

    Albany, Minnesota, as per 2017 US Census estimate, has a community population of 2,662 people. Albany is located in Stearns County, 20 miles west of St. Cloud and 80 miles northwest of Minneapolis/St. Paul on Interstate 94 (I-94). Albany has direct access to State Highway 238,...

    2017

  2. [11]

    metastatic breast carcinoma

  3. [12]

    Option 1

    Fibrous dysplasia Based on the given question and four options, which one is the right answer? Please respond with only “Option 1”, “Option 2”, “Option 3”, or “Option 4” as your final answer, without any additional explanation. To ensure you clearly understand my intention, I ...

  4. [13]

    Meningitis Answer: Option 1 12 Work in progress Example 2: Question: Which is NOT a third generation Cephalosporin Options:

  5. [14]

    1) osteosar- coma

    Cefuroxime Answer: Option 4 Example 3: ... Answer: Option 2 Example 4: ... Answer: Option 1 Example 5: ... Answer: Option 3 The prompt above includes a question with shuffled choices, meaning the first choice, “1) osteosar- coma”, is not necessarily the first choice in its ori...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.