Pith. sign in

REVIEW 3 major objections 8 minor 30 references

Revisiting LLM Value Probing Strategies: Are They Robust and Expressive?

T0 review · 3 major / 8 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Probing an LLM's values is unreliable: all three common methods shift under superficial formatting changes and barely track the model's choices in value-laden scenarios.

desk verdict Robustness findings are solid and worth taking seriously, but the value-behavior correlation analysis has a pseudo-replication flaw that undermines the paper's central expressiveness claim. read the letter →

arxiv 2507.13490 v1 pith:6YL2O6EE submitted 2025-07-17 cs.CL

Siqi Shen , Mehar Singh , Lajanugen Logeswaran , Moontae Lee , Honglak Lee , Rada Mihalcea This is my paper
classification cs.CL
keywords LLMvalueprobingtokenlogitsequenceperplexitytextgenerationselectionbiasdemographicpromptingvalue-actionagreementWorldValuesSurvey
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the value orientations researchers extract from large language models depend heavily on arbitrary design choices, not just on the model itself. It compares three widely used probing strategies — token logits, sequence perplexity, and free-text generation — and finds that all three produce large variance when the prompt wording or option labels are rearranged. It then tests expressiveness by asking whether probed values respond to demographic context and whether they predict the model's ratings of value-relevant actions. The central finding is that values only weakly correlate with action preferences, so current value-probing methods provide limited insight into actual LLM behavior. A sympathetic reader would take this as a caution that value measurements need to be validated against behavior, not taken at face value.

What carries the argument

The central objects are the three value-probing scoring methods: token logit, the softmax probability of the option token; sequence perplexity, the normalized inverse perplexity of the full answer sequence; and text generation, the empirical frequency of each option over repeated samples. Robustness is measured by mismatch rate (does the top answer flip?) and Jensen-Shannon distance (does the whole distribution shift?). Expressiveness is measured by Earth Mover's Distance alignment with World Values Survey country distributions under demographic prompting, and by Pearson and Spearman correlation between probed probability weights and action ratings on the synthesized scenario dataset.

What would settle it

Run the value-action agreement test with scenarios and actions written by humans and independently rated for value-pole validity; if the correlation between probed values and action ratings rises from roughly 0.1–0.3 to above 0.5, the paper's conclusion about poor expressiveness would be falsified as a dataset artifact. A second check would compare the top-choice match rate between probed values and the model's own action preference on a per-question basis, rather than only an aggregate correlation.

Watch

Extended reading notes

Core claim

The paper's central claim is that LLM value representations are neither robust nor expressive enough to support the conclusions typically drawn from them. Robustness is evaluated by perturbing prompt style (default, prefixed, one-shot) and option order or labels, measuring both answer mismatch and Jensen-Shannon divergence; sequence perplexity is the most stable, but all methods show high variance, especially in mid-size models, and robustness under prompt changes correlates with robustness under selection bias. Expressiveness is tested in two ways: demographic prompting with country personas improves alignment with human survey distributions for token and sequence methods but is largely missed by free-text generation, and a newly synthesized dataset of scenarios with paired value-based actions shows only weak (0.1–0.3) correlation between probed value weights and action ratings. The conclusion is that probed values capture limited information about how the model actually behaves in value-related scenarios.

Load-bearing premise

The weakest link is the assumption that the GPT-4-generated scenarios and action pairs really instantiate the intended value dimensions; if they are biased or mislabeled, the observed weak correlation between probed values and action ratings would be an artifact of dataset construction rather than evidence about value probing.

Editorial extensions

If this is right

  • Any single-prompt or single-format value probe should not be treated as a stable measurement; more reliable results require averaging across multiple prompt styles and option orders.
  • Sequence perplexity is the most robust of the three methods against input variation, so it is the safer choice when only a probability distribution over options is needed.
  • Free-text generation is a poor channel for detecting demographic steering: even when the underlying model shifts with country context, the sampled text often fails to reflect that shift.
  • Because probed values correlate only weakly with value-based action ratings, value-probing results alone cannot be used to predict or control LLM behavior.
  • Larger models are more stable under perturbation, but stability alone does not imply expressiveness, since a constant distribution would be perfectly robust and meaningless.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If probed values do not track action preferences, a plausible explanation is that LLMs do not maintain a coherent value latent; instead they may answer using surface cues. A direct test would apply a value-steering intervention and check whether action ratings shift in the predicted direction.
  • The weak value-action correlation also suggests a practical caution for alignment evaluation: value questionnaires may measure prompt-compliance rather than genuine belief, and should be supplemented with behavioral benchmarks.
  • The robustness results imply that past findings about cultural or demographic value alignment in LLMs may be partly an artifact of prompt formatting; re-analysis with multiple templates could change country-level rankings.
  • A concrete extension would be to build the scenario dataset with human-written actions instead of GPT-4-generated ones; if correlations rise substantially, the weak agreement is partly a data artifact rather than a model property.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 8 minor

Summary. The paper systematically compares three value-probing strategies (token logits, sequence perplexity, and free-text generation) for extracting value distributions from LLMs. Using 206 World Values Survey questions and ten open-weight models, it evaluates robustness under prompt-style and answer-option perturbations, then proposes two expressiveness tasks: one measuring how demographic prompting shifts probed values toward country-level human survey responses, and another correlating probed value weights with model ratings of actions in GPT-4-generated value-related scenarios. The paper reports that all probing methods are unstable under input perturbations, that sequence perplexity is the most robust, that demographic prompting generally improves alignment (least so for text generation), and that probed values only weakly correlate with action preferences. It also releases the synthesized value-action dataset.

Significance. If the central claims hold, this is a valuable cautionary study for the LLM evaluation community. Its strengths are the breadth of models (ten open models across three families), the grounding of the demographic-alignment analysis in external WVS human data, the head-to-head comparison of three widely used probing methods, and the public release of a new value-action scenario dataset. The robustness findings are consistent across models and are likely to be influential. However, the expressiveness claims, especially the headline conclusion that value probing provides limited insight into behavior, currently rest on a statistical analysis that pools non-independent observations, and on a dataset whose construction may bias the result. These issues are fixable but currently make the central behavioral claim unproven.

major comments (3)
  1. [§5.2, Figure 4] The pooled Pearson and Spearman correlations treat each scenario-action pair as an independent observation, but the value probability weight is a single per-question quantity that is repeated for every scenario generated from that question. For each of the 206 WVS questions, the two pole weights are computed once (after averaging over prompt and selection-bias variants) and then paired with ten scenario-level action scores, yielding 20 data points with only two distinct value weights. This pseudo-replication means the effective sample size is at most the number of questions, not the number of scenario-action pairs, so the significance stars in Figure 4 are anti-conservative. More importantly, the pooled correlation is dominated by between-question covariation between pole weights and average action scores, and it cannot support the claim that values only weakly correlate with behavior. Please re-analyze with question as the clustering unit (e.g., per-question correlations, mixed-effects models, or block bootstrap) and report confidence intervals.
  2. [§5.2, Table 5, §9] The value-action dataset is generated by GPT-4-turbo and filtered by GPT-4 self-critique, with no human validation; the Limitations section explicitly states that not all examples were examined by the authors. If the generated actions do not reliably correspond to the intended value poles, the observed weak correlation would be an artifact of dataset construction rather than evidence about value probing. The manuscript should either provide a human-validated subset with evidence that the filtering criteria are satisfied independently of the generating model, or demonstrate that the correlation conclusions are stable under stricter validity filters. Without such evidence, the strongest conclusion of the paper—that probed values provide limited insight into actual model behavior—is not load-bearing.
  3. [§5.1, Figure 5] The demographic-prompting alignment improvements are reported as point estimates without confidence intervals or significance tests. Claims such as 'token logits method sees the largest improvement' and 'text generation method is less steerable' are descriptive only, and the differences between methods could easily fall within sampling variability. Please add error bars across countries or questions, or run paired statistical tests, before drawing conclusions about which probing method is more or less steerable.
minor comments (8)
  1. [§3] The definition of text generation probability, p_text = n/N, leaves n and N undefined in the main text; please state that n is the number of times an option is selected and N is the total number of samples.
  2. [§4.1] The sentence 'For each perturbation on options, we take the average over all prompt styles to isolate its effect' should specify whether the averaged distribution is computed before or after computing mismatch/JS distances, since this affects the numerical results.
  3. [Table 2] The column headers 'Metric', 'Method', 'Correlation', 'p_value' are clear, but the method names 'option_probs', 'seq_probs', and 'text_probs' should be mapped explicitly to the 'Token', 'Seq', and 'Text' labels used in the main text and figures.
  4. [§5.2] The illustrative sentence 'ActionA can have a 0.7 total probability weight while receiving an action score of 8' should state the action-score scale (e.g., 0–10) and how scores are elicited from the model.
  5. [Figures 2–4] The significance-star markers ('*') in Figure 4 are not defined in the caption; please specify the p-value threshold and test used for each star.
  6. [§7.1] The statement 'In all our previous experiments, we considered tokens with and without space' needs a precise description of how the two token types were combined or averaged in the token-logit method, since this is a non-obvious implementation detail.
  7. [References] The reference for Alzahrani et al. is formatted as '2023. 2024.' in the bibliography; please correct the year/entry.
  8. [§10] The Ethics Statement contains a typo, 'rescriction' for 'restriction'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the robustness, demographic-alignment, and value-action analyses are empirical comparisons with no fitted input renamed as a prediction and no load-bearing self-citation.

full rationale

The paper's derivation chain is self-contained. The robustness study (Section 4) perturbs prompts and option formats and measures mismatch rate and Jensen-Shannon distance between the same model's outputs; these metrics are not derived from any fitted parameter, and the input-output relationship is not equivalent by construction. The demographic-prompting analysis (Section 5.1) is anchored to the externally collected World Values Survey human response distributions via Earth Mover's Distance, so the alignment improvement is measured against a ground truth independent of the probing methods. The value-action agreement task (Section 5.2) correlates probed value weights with action ratings obtained from the same model family; although this is self-referential in the sense that both variables are model outputs, the correlation is not analytically forced. The observed weak correlation is an empirical outcome that could, in principle, have been strong, and no equation in the paper reduces the correlation to the value weights by construction. The action-preference dataset is generated and filtered by GPT-4, which is a legitimate construct-validity concern for interpreting the weak correlation, but it is not a fitted input called a prediction, and the paper explicitly discloses this risk in its Limitations section. No uniqueness theorem, ansatz, or load-bearing result is imported from prior work by the same authors, and no known empirical pattern is merely renamed. The only substantive issues, such as potential pseudo-replication from pooling scenario-level action scores with a small number of question-level value weights, are statistical-correctness concerns rather than circularity. Accordingly, the paper has no significant circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The empirical claims depend on four domain assumptions: perturbation invariance is the right robustness criterion, WVS country distributions are valid ground truth, country prompts are an effective demographic lever, and the GPT-4-generated scenario dataset correctly instantiates value dimensions. No fitted free parameters or invented theoretical entities are introduced.

assumptions (4)
  • domain assumption A reliable value representation should be largely invariant to non-semantic changes in prompt and option format.
    This normative assumption is introduced in Section 4; if formatting changes are semantically relevant or if answer variance is legitimate, the high mismatch rates do not by themselves show that value probing is unreliable.
  • domain assumption Country-level World Values Survey response distributions are valid references for LLM value alignment.
    Invoked in Section 5.1 and the alignment metric in A.1; assumes that pooling human answers by country yields a meaningful cultural value distribution and that EMD over answer positions is a valid distance.
  • domain assumption Adding a country name to the prompt is an effective way to condition the model's value distribution toward that culture.
    Used in Section 5.1; the paper acknowledges debate around in-context conditioning but treats it as reasonable because it resembles end-user interaction.
  • domain assumption The GPT-4-generated and GPT-4-filtered scenarios and actions correctly instantiate the value dimensions being probed.
    Used in Section 5.2; if the scenarios do not faithfully realize the intended value contrast, the weak correlation between probed values and action ratings is an artifact of dataset construction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Revisiting LLM Value Probing Strategies: Are They Robust and Expressive?." pith.science (2026). https://pith.science/paper/6YL2O6EE

@misc{pith2026250713490,
  author       = {Pith},
  title        = {Pith review of: Revisiting LLM Value Probing Strategies: Are They Robust and Expressive?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6YL2O6EE}},
  note         = {Machine review of arXiv:2507.13490}
}
read the original abstract

There has been extensive research on assessing the value orientation of Large Language Models (LLMs) as it can shape user experiences across demographic groups. However, several challenges remain. First, while the Multiple Choice Question (MCQ) setting has been shown to be vulnerable to perturbations, there is no systematic comparison of probing methods for value probing. Second, it is unclear to what extent the probed values capture in-context information and reflect models' preferences for real-world actions. In this paper, we evaluate the robustness and expressiveness of value representations across three widely used probing strategies. We use variations in prompts and options, showing that all methods exhibit large variances under input perturbations. We also introduce two tasks studying whether the values are responsive to demographic context, and how well they align with the models' behaviors in value-related scenarios. We show that the demographic context has little effect on the free-text generation, and the models' values only weakly correlate with their preference for value-based actions. Our work highlights the need for a more careful examination of LLM value probing and awareness of its limitations.

Figures

Figures reproduced from arXiv: 2507.13490 by the authors.

Figure 1
Figure 1. Probing and evaluating the robustness and ex [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Effect of prompt styles to the value representation obtained with different methods. Measured by Mismatch(↓) on majority answer and JS distance (↓) on answer distribution. the answer is yes to all the questions. This process ensures that the scenarios and actions generated are plausible, relevant, and accurately reflect the influence of value orientations on behavior. Measuring Agreement. For each scenario, we ask t… view at source ↗
Figure 3
Figure 3. Effect of selection bias variations to the values representation obtained with different methods. Measured by Mismatch(↓) on majority answer and JS distance (↓) on answer distribution. Prompt style can drastically change the LLM values. We prompt all models with a set of tem￾plates that format the questions differently. We compare the output distribution from different tem￾plates pairwise and then take the average o… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Correlation between the value representation and action scoring [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Alignment Improvement with demographic [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

30 extracted references · 10 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Asif Agha. 2006. Language and social relations, volume 24. Cambridge University Press

    work page 2006

  4. [4]

    ElNokrashy, Mai AlKhamissi, and Mona Diab

    Badr AlKhamissi, Muhammad N. ElNokrashy, Mai AlKhamissi, and Mona Diab. 2024. https://api.semanticscholar.org/CorpusID:267759574 Investigating cultural alignment of large language models . In Annual Meeting of the Association for Computational Linguistics

    work page 2024

  5. [5]

    Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. https://arxiv.org/abs/2311.16867 The falcon series of open language models . Preprint, arXiv:2...

    arXiv 2023

  6. [6]

    Alzahrani, Hisham Abdullah Alyahya, Sultan Yazeed Alnumay, Shaykhah Alsubaie, Yusef Almushaykeh, Faisal A

    Norah A. Alzahrani, Hisham Abdullah Alyahya, Sultan Yazeed Alnumay, Shaykhah Alsubaie, Yusef Almushaykeh, Faisal A. Mirza, Nouf M. Alotaibi, Nora Altwairesh, Areeb Alowisheq, Saiful Bari, and Haidar Khan. 2023. 2024. https://api.semanticscholar.org/CorpusID:267412932 When benchmarks are targets: Revealing the sensitivity of large language model leaderboar...

    work page 2023

  7. [7]

    Anat Bardi and Shalom H Schwartz. 2003. Values and behavior: Strength and structure of relations. Personality and social psychology bulletin, 29(10):1207--1220

    work page 2003

  8. [8]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, and 12 others. 2020. https://arxiv.org/abs/2005.14165 Lan...

    arXiv 2020

Show all 30 references
  1. [9]

    Yong Cao, Li Zhou, Seolhwa Lee, Laura Cabello, Min Chen, and Daniel Hershcovich. 2023. https://arxiv.org/abs/2303.17466 Assessing cross-cultural alignment between chatgpt and human societies: An empirical study . Preprint, arXiv:2303.17466

    2023 arXiv

  2. [10]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  3. [11]

    Esin Durmus, Karina Nyugen, Thomas I. Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCandlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, and Deep Ganguli. 20...

    2023 arXiv

  4. [12]

    Christian Haerpfer, Ronald Inglehart, Alejandro Moreno, Christian Welzel, Kseniya Kizilova, Juan Diez-Medrano, Marta Lagos, Pippa Norris, Eduard Ponarin, and Bi Puranen. 2022. https://doi.org/10.14281/18241.24 World values survey: Round seven – country-pooled datafile version ...

    2022 doi

  5. [13]

    Katharina H \"a mmerl, Bjoern Deiseroth, Patrick Schramowski, Jind r ich Libovick \'y , Constantin Rothkopf, Alexander Fraser, and Kristian Kersting. 2023. https://doi.org/10.18653/v1/2023.findings-acl.134 Speaking multiple languages affects the moral bias of language models ....

    2023 doi

  6. [14]

    Geert Hofstede, Gert Jan Hofstede, and Michael Minkov. 2010. Cultures and Organizations: Software of the Mind, 3rd edition. McGraw-Hill Education, New York

    2010

  7. [15]

    Ronald Inglehart. 2005. Christian Welzel Modernization, Cultural Change, and Democracy The Human Development Sequence. Cambridge: Cambridge university press

    2005

  8. [16]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  9. [17]

    Rebecca L Johnson, Giada Pistilli, Natalia Menédez-González, Leslye Denisse Dias Duran, Enrico Panai, Julija Kalpokiene, and Donald Jay Bertulfo. 2022. https://arxiv.org/abs/2203.07785 The ghost in the machine has an american accent: value conflict in gpt-3 . Preprint, arXiv:2...

    2022 arXiv

  10. [18]

    Chenyang Lyu, Minghao Wu, and Alham Fikri Aji. 2024. https://api.semanticscholar.org/CorpusID:267770168 Beyond probabilities: Unveiling the misalignment in evaluating large language models . ArXiv, abs/2402.13887

    2024 arXiv

  11. [19]

    Marilù Miotto, Nicola Rossberg, and Bennett Kleinberg. 2022. https://arxiv.org/abs/2209.14338 Who is gpt-3? an exploration of personality, values and demographics . Preprint, arXiv:2209.14338

    2022 arXiv

  12. [20]

    Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward ...

    2023 arXiv

  13. [21]

    Sagnik Mukherjee, Muhammad Farid Adilazuarda, Sunayana Sitaram, Kalika Bali, Alham Fikri Aji, and Monojit Choudhury. 2024. Cultural conditioning or placebo? on the effectiveness of socio-demographic prompting. arXiv preprint arXiv:2406.11661

    2024 arXiv

  14. [22]

    Richard Nisbett. 2010. The Geography of Thought: How Asians and Westerners Think Differently... and. Simon and Schuster

    2010

  15. [23]

    Fabio Petroni, Tim Rockt \"a schel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. https://doi.org/10.18653/v1/D19-1250 Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language P...

    2019 doi

  16. [24]

    Held, and Diyi Yang

    Michael Joseph Ryan, William B. Held, and Diyi Yang. 2024. https://api.semanticscholar.org/CorpusID:267897555 Unintended impacts of llm alignment on global representation . In Annual Meeting of the Association for Computational Linguistics

    2024

  17. [25]

    Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. https://arxiv.org/abs/2303.17548 Whose opinions do language models reflect? Preprint, arXiv:2303.17548

    2023 arXiv

  18. [26]

    Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, and 1 others. 2024. A roadmap to pluralistic alignment. arXiv preprint arXiv:2402.05070

    2024 arXiv

  19. [27]

    Lindia Tjuatja, Valerie Chen, Tongshuang Wu, Ameet Talwalkwar, and Graham Neubig. 2024. https://doi.org/10.1162/tacl_a_00685 Do LLM s exhibit human-like response biases? a case study in survey design . Transactions of the Association for Computational Linguistics, 12:1011--1026

    2024 doi

  20. [28]

    Xinpeng Wang, Chengzhi Hu, Bolei Ma, Paul R \"o ttger, and Barbara Plank. 2024 a . Look at the text: Instruction-tuned language models are more robust multiple choice selectors than you think. arXiv preprint arXiv:2404.08382

    2024 arXiv

  21. [29]

    o ttger, Frauke Kreuter, Dirk Hovy, and Barbara Plank. 2024 b . https://api.semanticscholar.org/CorpusID:267782369

    Xinpeng Wang, Bolei Ma, Chengzhi Hu, Leon Weber-Genzel, Paul R \"o ttger, Frauke Kreuter, Dirk Hovy, and Barbara Plank. 2024 b . https://api.semanticscholar.org/CorpusID:267782369 "my answer is c": First-token probabilities do not match text answers in instruction-tuned langua...

    2024

  22. [30]

    Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2023. https://api.semanticscholar.org/CorpusID:261582594 Large language models are not robust multiple choice selectors . ArXiv, abs/2309.03882

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.