REVIEW 4 major objections 4 minor 2 cited by
PReSS: An Automated Black-Box Framework for Evaluating Political Stance Stability in LLMs
T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read The paper proposes PReSS, a black-box stress test that labels each LLM's topic-level political stance stable or unstable, and shows the labels predict which stances survive fine-tuning.
desk verdict A useful framework for measuring LLM stance stability, but a load-bearing contradiction in the UR/UL definitions and an overclaimed factor analysis undermine the current version. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying object is the four-class stance typology built from two binary signals: the persistence indicator $\delta(o,a)$, which is 1 when the model's stance survives both supportive and counter-argumentative prompting, and the bias-alignment indicator $I_B(o,b)$, which is 1 when the original stance matches the annotated rightward (or leftward) polarity of the statement. Their combination yields stable-left, unstable-left, stable-right, and unstable-right for every model–topic pair. This typology does the work: direction alone (left vs right) does not predict fine-tuning outcomes, but the stable/unstable split does, which is why the paper can treat stability as a moderating factor for debiasing and alignment.
What would settle it
Re-run PReSS on the two factor-homogeneous subsets of the 19 statements (items loading primarily on the first versus the second rotated factor). If stability scores and the UR→SR / UL→SR transition rates shift sharply between subsets, the single-axis assumption is load-bearing and the reported controllability numbers are partly an artifact of item selection. A cleaner test: run the mirror-image experiment with left-alignment DPO and check whether SL topics convert to stable-left at comparable rates — if not, the claimed stability–controllability link is specific to one intervention direction.
Extended reading notes
Core claim
On its own terms, the paper establishes that stance stability is a measurable, topic-specific property of LLMs and that PReSS labels carry predictive force for alignment interventions. Each response under the three argumentative conditions is coded by whether the stance persists ($\delta = 1$) and whether it aligns with the statement's right-bias direction ($I_B = 1$), producing the four classes. A model that is left-leaning overall can hold a stable-right stance on one topic and an unstable-left stance on another; the topic-wise stability score $S_{t,m}$ across three argument sets ranges from near zero to one within a single model. The decisive empirical claim concerns fine-tuning: transitions UR→SR and UL→SR occur with probabilities around 61.1% and 57.8% under right-alignment DPO, whereas SL→SR is significantly less frequent, so the framework's stability labels identify where fine-tuning has leverage and where a model preserves prior commitments. The paper further validates the black-box labels against semantic entropy, a white-box uncertainty measure, reaching AUROC 0.78, and reports that prompting with opposite-persona instructions rarely overturns a stable stance.
Load-bearing premise
The 19 Political Compass statements are treated as measuring a single left-right economic axis, yet the paper's own factor analysis keeps a second factor above the retention threshold (eigenvalue 1.62; 25.66% of variance after rotation) — if that second dimension is real, the left/right polarity labels that drive the whole typology could be confounded.
Editorial extensions
If this is right
- Debiasing or ideology-reversal pipelines should be targeted at topics labeled unstable, where the reported transition rates (roughly 58–61%) show fine-tuning has leverage, rather than applied uniformly across topics.
- Model-level left/right classifications should be replaced or supplemented by topic-level stance maps; a single model can be stable-right on some topics and unstable-left on others.
- Three black-box probes per topic (neutral, supporting, counter) can substitute for roughly 20 generations of white-box uncertainty estimation when the goal is to locate malleable political stances, since PReSS labels reach AUROC 0.78 against semantic entropy.
- Any application that requires the model to hold a consistent political position — tutoring, multi-agent debate, AI-powered persuasion — should audit the relevant topics with a stability probe before deployment.
- Because stability and instability are properties of model–topic pairs rather than of whole models, safety evaluations that sample only a few topics could miss either the most entrenched or the most flippable behaviors of the same system.
Reading between the lines
- If stability is a genuine model property, fine-tuning on a stable topic should require proportionally more preference data than on an unstable topic; the paper does not vary dataset size per topic, so this is a direct testable extension.
- The factor analysis retains a second factor (25.66% of rotated variance) by the Kaiser criterion, so part of what the paper labels left-right polarity may be a second ideological axis; re-running the typology within factor-homogeneous statement subsets would show whether the stability and transition results survive.
- The reported asymmetry — both unstable-right and unstable-left topics convert to stable-right under right-alignment — could reflect the intervention direction rather than an intrinsic property; a mirror experiment with left-alignment DPO should show whether SL conversions become the frequent ones.
- Because semantic entropy correlates with instability at AUROC 0.78, unstable stances may be cases of genuine model uncertainty; one could check whether unstable topics are also those with highest lexical diversity across unpressured paraphrases, connecting stance stability to the broader distinction between epistemic uncertainty and sycophantic compliance.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes PReSS, a black-box framework for measuring whether LLM political stances are stable or unstable under argumentative pressure, classifying each model-topic pair as stable-left, unstable-left, stable-right, or unstable-right. Applying PReSS to 12 instruction-tuned LLMs across 19 Political Compass economic statements, the authors report substantial topic-level variation in stability, a correlation between instability and semantic-entropy uncertainty (AUROC up to 0.78), and evidence that stable stances resist ideology reversal by prompting and DPO fine-tuning while unstable stances are more readily converted to stable-right. The paper argues that stability should be treated as a distinct dimension of political bias in LLM evaluation.
Significance. If the framework and its empirical claims withstand scrutiny, PReSS would be a useful and practical contribution: it offers a low-cost black-box protocol, uses open-ended rather than forced-choice prompts, and connects an interpretable stability taxonomy to both white-box uncertainty metrics and downstream controllability. The proposed distinction between stable and unstable stances is potentially valuable for alignment and safety evaluations. However, the paper's central claims currently rest on an internal inconsistency in the stance typology, an unvalidated unidimensionality assumption, and outcome definitions that are partly circular with respect to the predictor labels. Each of these issues is load-bearing for the headline conclusion that stability labels predict controllability, so the contribution cannot be accepted in its present form.
major comments (4)
- [§3.2, Eq. (3) and Table 1] The formal definition and the interpretive table define the unstable categories in opposite directions. Eq. (3) sets IB=1 exactly when the original stance o is right-aligned (for b=+1, o=+1; for b=-1, o=+1). For UR, δ=0 means a≠o, so a post-pressure stance is necessarily left. Thus UR denotes an original right stance that shifts left under pressure, and UL denotes an original left stance that shifts right. Table 1 states the reverse for both UR and UL. Since the labels UR and UL feed every downstream analysis, including Table 2, Fig. 8, and the §6.2 transition probabilities UR→SR (61.1%) and UL→SR (57.8%), the paper must specify which rule was implemented and correct either the equations or the interpretation. Without a code release or a worked example, the central controllability claims are not reproducible from the text.
- [§3.1 and Appendix B (Tables 5–7)] The factor analysis does not support the claim of a single dominant economic dimension. Both the unrotated solution (eigenvalues 5.72 and 1.63) and the rotated solution retain two factors under the Kaiser criterion, with the rotated second factor explaining 25.66% of the variance. Several items load substantially on Factor 2 (e.g., Q8 0.72, Q14 0.54, Q7 0.53, Q15 0.44), so the items are not unidimensional in the sample used. Since the left/right polarity labels assigned to the 19 statements depend on the single-axis assumption, this gap could confound the stance-direction classifications and, in turn, the stability labels. A parallel-analysis or eigenvalue-based justification is needed, or the analysis should be restricted to items that are demonstrably unidimensional.
- [§6 and §6.2] The SF/SU/ID outcomes used to demonstrate controllability are defined from the same PReSS stability labels that are the predictor of interest. SF is defined as stable stances of only one direction appearing across conditions, SU as stable stances of both orientations, and ID as no stable stance appearing; these are essentially restatements of the topic-level SL/SR/UL/UR classification under different prompts. Consequently, the high PSF rates in Figs. 5–6 may reflect the definition rather than independent evidence that stability predicts controllability. The paper should either define an outcome measure that does not presuppose the four-class taxonomy, or explicitly justify why the taxonomy-based outcome is not circular. In addition, the headline transition probabilities UR→SR≈61.1% and UL→SR≈57.8% and the claim that SL→SR is significantly less frequent are asserted in §6.2 without a table, confidence intervals, or per-model breakdowns, with the full table promised only for a later version. These numbers are the central evidence for the controllability claim and must be presented and accompanied by uncertainty estimates.
- [Abstract and Section 5.1] The abstract states that PReSS was applied to 9 LLMs, while the full text, Fig. 2, and Section 4 consistently report 12 models. This inconsistency should be resolved. Additionally, Section 5.1 reports Table 2 as using 'a single set of supporting- and counter-arguments,' while Section 5.2 and Section 6 use three independent argument sets; the paper should clarify which argument set was used for the baseline distribution and whether the reported percentages and confidence intervals account for variation across the three sets.
minor comments (4)
- [Throughout] There are several typographical errors, including 'tyree' in Appendix A and 'Y ue Dong' in the author block; these should be corrected.
- [Fig. 4 and §5.3] The AUROC values are reported only as printed on the figure, with no confidence intervals or significance tests across the 12 models; a small table with bootstrap intervals would strengthen the validation claim.
- [Appendix B, Table 7] The rotated loadings are not discussed item by item; the paper should identify which items are not clearly aligned with the intended polarity and explain their impact on the stance labels.
- [Section 8 (Limitations)] The limitations section is honest and useful, but it should be expanded to mention the internal typology inconsistency and the factor-analysis issue, rather than only topic coverage and axis dimensionality.
Circularity Check
No significant circularity: the controllability claim rests on empirical transition rates, not on defining the outcome into the input.
full rationale
The PReSS labels are operationalized in Eqs. (1)-(3) from observed baseline and post-argument stances plus the statement's annotated bias, and the Section 6 controllability evidence is an empirical transition analysis: UR->SR and UL->SR are reported at roughly 61.1% and 57.8% while SL->SR is less frequent. These rates are not forced by the definitions because an initially unstable topic can remain unstable (ID) or become stable in either direction, and an initially stable-left topic can remain stable-left, degrade to unstable, or flip; the coding merely labels the outcomes, it does not determine their probabilities. The Semantic Entropy validation (AUROC up to 0.78) is an independent white-box metric, and the fine-tuning and prompting interventions are separate manipulations from the argumentative pressure used to define stability. There are no load-bearing self-citations: the external references (Feng et al., PoliTune, Farquhar et al.) supply standard protocols or independent tools rather than the paper's own conclusions. The paper's stated limitations (single left-right axis, 19 topics, polarized agree-disagree semantics) are scope restrictions, not circular inputs. Two non-circular concerns are worth noting separately: Table 1's UR/UL interpretations reverse the direction implied by Eq. 3, since delta=0 with IB=1 means an original right stance that moves left, not right, and the SF/SU/ID outcome taxonomy is defined from the same stable/unstable labels it describes; but neither makes the central transition claim true by construction.
Assumptions & free parameters
assumptions (4)
- domain assumption The 19 selected Political Compass statements measure a single economic left-right dimension.
- domain assumption The BART-based NLI model correctly classifies open-ended model responses as agreement or disagreement.
- domain assumption Three DeepSeek-R1-32B-generated arguments per side constitute valid argumentative pressure.
- domain assumption Deterministic generation at temperature=0 isolates ideological variance from sampling noise.
Cite this review
Pith. "Pith review of PReSS: An Automated Black-Box Framework for Evaluating Political Stance Stability in LLMs." pith.science (2026). https://pith.science/paper/RLMSMLAI
@misc{pith2026250417052,
author = {Pith},
title = {Pith review of: PReSS: An Automated Black-Box Framework for Evaluating Political Stance Stability in LLMs},
year = {2026},
howpublished = {\url{https://pith.science/paper/RLMSMLAI}},
note = {Machine review of arXiv:2504.17052}
}
read the original abstract
Existing evaluations of political bias in large language models (LLMs) typically classify outputs as left- or right-leaning. We extend this perspective by examining how ideological tendencies vary across topics and how consistently models maintain their positions, a property we refer to as stability. To capture this dimension, we propose PReSS (Political Response Stability under Stress), an automated black-box framework that evaluates LLMs by jointly considering model and topic context, categorizing responses into four stance types: stable-left, unstable-left, stable-right, and unstable-right. Applying PReSS to 9 widely used LLMs across 19 political topics reveals substantial variation in stance stability; for instance, a model that is left-leaning overall can exhibit stable-right behavior on certain topics. This highlights the importance of topic-aware and fine-grained evaluation of political ideologies of LLMs. Moreover, stability has practical implications for controlled generation and model alignment: interventions such as debiasing or ideology reversal should explicitly account for stance stability. Our empirical analyses reveal that when models are prompted or fine-tuned to adopt the opposite ideology, unstable topic stances are more likely to change, whereas stable ones resist modification. Thus, treating stability as a moderating factor provides a principled foundation for understanding, evaluating, and guiding interventions in politically sensitive model behavior.
Figures
Figures from the paper (5 more)
Forward citations
Cited by 2 Pith papers
-
Progressive in Principle, Centrist in Practice: LLM Political Bias Is Instrument-Dependent
LLMs that look left-of-center on abstract political questionnaires align with centrist parties and often vote no when asked to decide real Swiss referenda, with large language-dependent variation.
-
When Models Refuse: Political Steerability and Feature Richness as Measures of Ideological Depth
Comparing two open LLMs, the paper claims that greater internal political feature richness predicts steerability and that refusals on benign prompts reflect capability deficits, but the causal evidence is missing.
Reference graph
Works this paper leans on
-
[1]
Introduction Large Language Models (LLMs) increasingly shape political discourse, raising important ques- tions about how they internalize and express ide- ological biases inherited from training data (Feng et al., 2023; Santurkar et al., 2023). Prior studies have primarily focused on assessing the overall ideological leanings of LLMs, reporting consisten...
arXiv 2023
-
[2]
We proposePReSS(Political Response Sta- bility under Stress), a black-box framework for evaluating large language models (LLMs) by measuringideological stabilityunder argu- mentative pressure. PReSS introduces a four- class stance taxonomy (stable-left,unstable- left,stable-right,unstable-right) to reveal nu- anced ideological behaviors beyond binary left...
-
[3]
Even within the same overall leaning, models exhibit diverse stable and unstable topic stances
Through an evaluation of 12 LLMs across 19 political topics, we show thatideological be- havior is topic-dependent. Even within the same overall leaning, models exhibit diverse stable and unstable topic stances. Moreover, stability predicts model controllability: unsta- ble topics shift more readily when models are prompted or fine-tuned toward the opposi...
-
[4]
We validate the reliability of the black-box PReSS framework using the white-boxSe- mantic Entropymetric (AUROC = 0.78), demonstrating that stance instability correlates strongly with model uncertainty. Together, these findings establish ideological stability as a measurable and interpretable dimension of LLM bias, with significant implications for ro- bu...
-
[5]
Background and Related Work Ideological Stance Detection in Political Sci- enceIdeological stance detection has long been a foundational problem in political science and com- putational social science (Burnham, 2024). It pro- vides an empirical lens on how language reflects and shapes ideology, and it supports analyses of polarization, framing, and repres...
work page 2024
-
[6]
Methodology Existing approaches to evaluating political bias in LLMs typically classify their overall ideological orientation (e.g., left or right) using closed-ended prompts. Such methods ignore how consistently models preserve their stance on individual topics when their initial opinion is challenged. In practice, models often adapt their position mid-r...
-
[7]
Curate and annotate a controlled set of topic- specific statements derived from the Political Compass economic axis, ensuring consistent unidimensional ideological polarity
-
[8]
Present these selected statements to candi- date LLMs and record their original stances (left/right) from their open-ended responses
Show all 21 references
-
[9]
Re-present the same statements under sup- porting and counter-argument conditions to simulate persuasive pressure and classify the stance of each outcome
-
[10]
We use these stages to quantify not only the ide- ological direction but also the stability of stance ex- pression under varying argumentative conditions
Compare post-argumentation stances to the baseline and categorize each outcome beyond left/right as one of four types: Stable-Left,Stable-Right,Unstable-Left, or Unstable-Right. We use these stages to quantify not only the ide- ological direction but also the stability of stan...
2023
-
[11]
Left and right-leaning candidates are shown in different colors
Experimental Setup RightLeft Llama-2-70b-chat Llama-3.2-3B-Instruct gemma-2-9b-it Llama-2-13b-chat Mistral-7B-Instruct-v0.3 Qwen2.5-3B-Instruct OLMo-7B-0724-Instruct Qwen2.5-3B DeepSeek-R1-Distill-Qwen-1.5B Right-FT-Llama-2-7b-chat Right-FT-Llama-2-13b-chat Right-FT-Llama-3.1-...
2024
-
[12]
Results In this section, we analyze the stance stability of the candidate LLMs across topics, treating the topic as input and stability as the resulting outcome. 5.1. Relationship Between Ideology and Stability We first quantify the stance stability of the models over each top...
-
[13]
Ideology Reversal Analysis Prompt engineering (Anagnostidis and Bulian,
-
[15]
Conclusion In this study, we examined how Large Language Models (LLMs) maintain or shift their political stances when subjected to argumentative pres- sure. Using the proposedPReSSframework, we classified each model’s topic-specific responses as stable or unstable beyond their...
-
[16]
Firstly, the analy- sis covers only 19 topics curated from the Political Compass test because it offered a well-structured set of statements
Limitations Our study has several limitations. Firstly, the analy- sis covers only 19 topics curated from the Political Compass test because it offered a well-structured set of statements. While these statements capture a diverse set of themes, a broader coverage of topics dra...
-
[17]
Bibliographical References Ahmed Agiza, Mohamed Mostagir, and Sherief Reda. 2024. Politune: Analyzing the impact of data selection and fine-tuning on economic and political biases in large language models. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, v...
2024
-
[2003]
Michael Laver and John Garry
Extracting policy positions from political texts using words as data.American political science review, 97(2):311–331. Michael Laver and John Garry. 2000. Estimating policy positions from political texts.American Journal of Political Science, pages 619–634. Gaël Le Mens and Ai...
2000
-
[2017]
from each according to his ability, to each according to his need
Estimating candidates’ political orienta- tion in a polarized congress.Political Analysis, 25(2):167–187. Kaiqi Y ang, Hang Li, Yucheng Chu, Yuping Lin, Tai-Quan Peng, and Hui Liu. 2024. Unpack- ing political bias in large language models: In- sights across topic polarization....
2024
-
[2019]
Mitchell Linegar, Rafal Kocielnik, and R Michael Alvarez
Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.arXiv preprint arXiv:1910.13461. Mitchell Linegar, Rafal Kocielnik, and R Michael Alvarez. 2023. Large language models and po- litical science.Frontiers in Politic...
1910 arXiv
-
[2024]
However, we find that they are significantly more resilient on topics with stable stances
and finetuning approaches like DPO are widely used to reverse these models’ ideologies efficiently (Rafailov et al., 2023; Agiza et al., 2024). However, we find that they are significantly more resilient on topics with stable stances. The core intuition is that a model’sstable...
2023
-
[2025]
arXiv preprint arXiv:2505.04171
Large language models are often politi- cally extreme, usually ideologically inconsistent, and persuasive even in informational contexts. arXiv preprint arXiv:2505.04171. Sotiris Anagnostidis and Jannis Bulian. 2024. How susceptible are llms to influence in prompts? arXiv prep...
2024 arXiv
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.