REVIEW 3 major objections 4 minor 2 cited by
Political Actor Agent: Simulating Legislative System for Roll Call Votes Prediction with Large Language Models
T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read The Political Actor Agent (PAA) claims that simulating legislators as role-playing LLM agents—with scalable profiles, multi-view planning, and a leadership influence mechanism—predicts roll-call votes more accurately and more…
desk verdict Useful agent-design paper, but the headline prediction numbers are contaminated by the models' pretraining, and the paper's own control doesn't rule it out. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the three-module agent pipeline: the Profile Construction Module (a scalable prompt-level profile holding personal information, constituency details, sponsorship activity, and a sample of 20 recent voting records per legislator), the Multi-view Planning Module (which decomposes the vote into trustee, delegate, and follower perspectives and synthesizes them), and the Simulated Legislative Action Module (an influence mechanism where leader agents L = {Speaker, Republican Leader, Democratic Leader, committee chair, caucus members} vote first via multi-view planning and the remaining agents vote conditioned on the leaders' outcomes: V_l = p(L), V_o = p(O | V_l)). The claim is that this pipeline—rather than any learned weights—carries the predictive power, and the profile module carries the largest share, as the ablation dropping it costs roughly 13 accuracy points.
What would settle it
Build a test set of roll-call votes on bills introduced after the LLM's knowledge cutoff (for example, bills in the current Congress with votes not present in the model's training data), run the same PAA prompts with the same 20-vote profiles, and compare accuracy against the embedding baselines trained only on the designated training split; if PAA's edge shrinks to noise on post-cutoff bills, that would show the reported advantage came from memorized historical votes rather than from the simulation mechanism.
Extended reading notes
Core claim
The central discovery claimed is that an agent-based paradigm—Political Actor Agent (PAA)—can match or exceed embedding-based political-actor models at roll-call vote prediction while also producing interpretable, multi-view decision reasoning. PAA constructs a scalable textual profile for each legislator (personal background, constituency demographics, sponsorship activity, and sampled past votes), decomposes the voting decision into trustee, delegate, and follower views synthesized into a final stance, and then simulates legislative dynamics by having leader agents (Speaker, party leaders, committee chair, relevant caucus members) vote first, with remaining agents' prompts conditioned on the leaders' votes. On the 117th–118th U.S. House data, PAAG (GPT-4o-mini) reaches 91.8/92.2 accuracy/F1 on split244 and 92.1/93.0 on split622, consistently above the ideal-point, graph-neural-network, and pre-trained baselines, while PAAL (Llama-3-70B) leads on macro-F1 among non-PAAG methods. The authors conclude that PAA offers a scalable and interpretable paradigm that degrades less than trained baselines when training data is scarce, and that its reasoning traces give political science insights into how legislators weigh constituency, expertise, and party leadership.
Load-bearing premise
The load-bearing premise is that the base LLM has not memorized the test roll-call labels during pretraining; if the model already knows that the 117th House passed H.R. 1096, then feeding it the bill title and asking for a vote can yield high accuracy by recall rather than by the simulated reasoning the paper describes.
Editorial extensions
If this is right
- With only 20 sampled past votes per legislator, PAA stays accurate as the training split shrinks, suggesting it can predict votes of newly elected legislators where embedding methods lack data.
- Because PAA returns trustee, delegate, and follower reasoning for each vote, it can generate per-legislator explanations that link a vote to constituency, expertise, career focus, and party leadership.
- PAA's influence mechanism conditions every non-leader vote on the leaders' predicted votes, so the framework explicitly models party and committee leadership effects without training a network to learn them.
- Removing any single profile component (personal info, constituency, sponsorship, voting records) costs accuracy, but the full-profile version with 20 sampled records outperforms versions fed the entire training set, indicating bounded context is better than exhaustive history.
Reading between the lines
- If the memorization concern is set aside, a direct testable extension is to run PAA on a legislature not in the LLM's pretraining corpus (for example, a non-English parliament with recent votes) to isolate the simulation's transfer value from any U.S.-specific knowledge.
- The result that longer voting histories hurt accuracy suggests a context-window bottleneck: a retrieval policy that selects the most bill-relevant past votes per legislator would be a natural improvement that the paper does not test.
- The leader-follower conditioning conflates two effects—the information content of leader votes and the prompting effect of 'here is what leaders did'—and the ablation removing the acting module cannot separate them; a targeted ablation that feeds leaders' votes without a leadership label would isolate the mechanism.
- The interpretability claim, if it holds, gives computational political science a cheap instrument for counterfactual analysis: swapping a legislator's district demographics or party label in the profile and observing how the predicted vote and its stated reasons change.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes the Political Actor Agent (PAA), an LLM-based framework that combines a scalable legislator profile, a multi-view planning module (trustee, delegate, follower), and a leader-follower influence mechanism to predict roll-call votes. It evaluates PAA on 117th-118th U.S. House voting data under three chronological splits, reporting that PAA with GPT-4o-mini (PAAG) outperforms five embedding-based baselines in accuracy and macro-F1 (e.g., 91.8 acc / 92.2 F1 on split244), while also providing human-readable decision rationales. Ablation studies in Section 4.3-4.4 decompose the contributions of the profile, planning, and action modules and probe the effect of profile components and length.
Significance. If the evaluation were clean, the paper would be a useful contribution: it offers an interpretable, data-efficient alternative to trained embedding models, and the modular design (profile/planning/action) is a natural way to inject political-science knowledge into LLM predictions. The ablations are thoughtful, and the consistency analysis in Section 4.5 directly addresses the hallucination concern, which is a strength. However, the significance of the headline accuracy claim hinges entirely on excluding pretraining memorization: because the test votes largely predate or overlap the base models' training data, the reported gains over baselines cannot currently be interpreted as predictive skill. This is a load-bearing, unresolved issue.
major comments (3)
- [§4.4 Analysis of Profile Module (RQ1, PAA-ano)] The RQ1 control PAA-ano anonymizes legislator names and bill numbers but leaves the full bill title and text in the prompt. For test votes from the 117th and 118th House, which predate or overlap the training windows of GPT-4o-mini and Llama-3-70B, the model can recall the eventual outcome of a bill (e.g., H.R. 1096 in Figure 5) from pretraining corpora such as news articles, Wikipedia, or legislative trackers. The reported 90.8 accuracy for PAA-ano therefore does not rule out label memorization; it only shows that names and bill numbers are not the retrieval cue. The conclusion in the same section that 'PAA likely relies on the information in our profile module for predictions' is not supported by this control. A necessary condition is to remove bill content from the prompt, or to restrict the test set to votes that postdate the model's knowledge cutoff.
- [§4.1-§4.2 Datasets, Baselines, and Chronological Split] The chronological splits (split244/433/622) do not prevent label leakage from pretraining because the base LLMs were trained on web text covering the same time period as many of the test votes. The comparison in Table 1 is therefore not a fair predictive comparison: the embedding baselines are fit only on the given training split, while the LLM has potential access to the test outcomes through its pretraining. The paper should report the knowledge cutoff dates of both base models, isolate a test subset of votes occurring after those cutoffs, and/or include a baseline with the same look-ahead exposure. Without this, the 91.8-92.1 PAAG accuracy cannot be taken as evidence of predictive skill over the baselines.
- [§4.4 PAA-Dec and interpretation of profile effects] The PAA-Dec experiment swaps legislator names but keeps bill text, so it shares the same memorization flaw as PAA-ano. The small accuracy drop from PAA-ano (90.8) to PAA-Dec (90.1) is interpreted as evidence that legislator information affects predictions, but it is equally consistent with the model using bill-content cues while being slightly perturbed by inconsistent name information. This weakens the paper's claims about the relative contributions of profile components (e.g., constituency information being least important), because all such comparisons operate under the unresolved contamination risk.
minor comments (4)
- [General reproducibility] The paper does not provide code, data, or the actual prompt templates; the appendix containing the prompts is referenced but not included in this arXiv version. The description of the profile module (Section 3.1) and the view prompts (Section 3.2) is too high-level to reproduce the method without guessing.
- [§4.2 Implementation] The paper does not report decoding parameters (temperature, top-p, number of samples) for the LLM experiments, despite emphasizing consistency and reporting standard deviations over five runs. These settings are important for interpreting the variance and the consistency results.
- [§4.5 Consistency Analysis] The consistency analysis is a positive feature, but the heatmap in Figure 4 would be more informative with a numerical summary, such as the proportion of agent-bill pairs that are correct in all 20 runs, and the test-set size should be stated.
- [References] The reference list contains duplicate entries: Majumdar et al. 2024a and 2024b are identical, and Zhou et al. 2024a and 2024b are identical; these should be merged or disambiguated.
Circularity Check
No significant circularity: the PAA pipeline keeps training and test data separate, and the reported predictions are not defined in terms of the test labels.
full rationale
The paper's derivation chain is not circular. PAA constructs legislator profiles by sampling 20 voting records from the training set only and then evaluates on a chronologically separated test set, so the predicted labels are not injected through the profile or any fitted parameter. The influence mechanism conditions other agents on leader-agent predictions, but those leader predictions are themselves generated from the same training-derived profiles, not from the test outcomes. The ablation studies modify the profile modules and report performance differences, which is a legitimate sensitivity analysis rather than a circular reduction. The paper contains no equations that define the output in terms of the input labels, and no parameter is fitted to the test set and then renamed as a prediction. The references include no load-bearing self-citations by the present authors; the cited prior work is used for baselines and background concepts, not to justify the validity of the method. The concern raised in the paper's own RQ1, that the base LLM may have encountered legislative information during pretraining, is a data-contamination and external-validity issue, not a circularity of argument. The PAA-ano control keeps bill text intact, so it does not fully rule out memorization, but an incomplete control is a correctness risk, not a circular step. Under the stated criteria, the central claim of accurate roll-call prediction via LLM agents has independent content and does not reduce by construction to its inputs.
Assumptions & free parameters
free parameters (3)
- k (number of sampled voting records in profile) =
20
- Leader-agent set L = {S, R, D, CC, CM} =
fixed composition in Eq. (1)
- View decomposition (trustee/delegate/follower) =
three fixed prompt templates
assumptions (4)
- domain assumption A legislator's vote can be fully encoded by trustee, delegate, and follower views.
- domain assumption Leaders vote first and followers are causally influenced by leaders' votes.
- domain assumption An LLM prompted with a profile and bill text simulates the legislator's decision faithfully.
- domain assumption Test roll-call outcomes are not available to the LLM through pretraining.
Cite this review
Pith. "Pith review of Political Actor Agent: Simulating Legislative System for Roll Call Votes Prediction with Large Language Models." pith.science (2026). https://pith.science/paper/PBUVIEYE
@misc{pith2026241207144,
author = {Pith},
title = {Pith review of: Political Actor Agent: Simulating Legislative System for Roll Call Votes Prediction with Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/PBUVIEYE}},
note = {Machine review of arXiv:2412.07144}
}
read the original abstract
Predicting roll call votes through modeling political actors has emerged as a focus in quantitative political science and computer science. Widely used embedding-based methods generate vectors for legislators from diverse data sets to predict legislative behaviors. However, these methods often contend with challenges such as the need for manually predefined features, reliance on extensive training data, and a lack of interpretability. Achieving more interpretable predictions under flexible conditions remains an unresolved issue. This paper introduces the Political Actor Agent (PAA), a novel agent-based framework that utilizes Large Language Models to overcome these limitations. By employing role-playing architectures and simulating legislative system, PAA provides a scalable and interpretable paradigm for predicting roll-call votes. Our approach not only enhances the accuracy of predictions but also offers multi-view, human-understandable decision reasoning, providing new insights into political actor behaviors. We conducted comprehensive experiments using voting records from the 117-118th U.S. House of Representatives, validating the superior performance and interpretability of PAA. This study not only demonstrates PAA's effectiveness but also its potential in political science research.
Figures
Figures from the paper (2 more)
Forward citations
Cited by 2 Pith papers
-
Digital Pantheon: Simulating and Auditing Coalition Formation with LLM Agents
An LLM multi-agent framework with party-specific fine-tuning and manifesto retrieval produces a stable N-VA-led coalition simulation, and a new lineage-tracing taxonomy claims to predict which simulated provisions mat...
-
ValuesRAG: Enhancing Cultural Alignment Through Retrieval-Augmented Contextual Learning
ValuesRAG retrieves and reranks value summaries of demographically similar WVS respondents and uses them as in-context evidence, beating four baselines on six regional survey datasets.
Reference graph
Works this paper leans on
-
[1]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Alexander, R. M. 2019. Representation and the Electoral College. Oxford University Press
work page 2019
-
[4]
P.; Busby, E
Argyle, L. P.; Busby, E. C.; Fulda, N.; Gubler, J. R.; Rytting, C.; and Wingate, D. 2023. Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3): 337--351
2023
-
[5]
A.; MacKnight, R.; and Gomes, G
Boiko, D. A.; MacKnight, R.; and Gomes, G. 2023. Emergent autonomous scientific research capabilities of large language models. arXiv preprint arXiv:2304.05332
arXiv 2023
-
[6]
V.; Yang, S.; Shah, D.; Hu, J.; and Rogers, T
Chuang, Y.-S.; Studdiford, Z.; Nirunwiroj, K.; Goyal, A.; Frigo, V. V.; Yang, S.; Shah, D.; Hu, J.; and Rogers, T. T. 2024. Beyond Demographics: Aligning Role-playing LLM-based Agents Using Human Belief Networks. arXiv preprint arXiv:2406.17232
arXiv 2024
-
[7]
Clinton, J.; Jackman, S.; and Rivers, D. 2004. The statistical analysis of roll call data. American Political Science Review, 98(2): 355--370
work page 2004
-
[8]
Dai, G.; Zhang, W.; Li, J.; Yang, S.; Rao, S.; Caetano, A.; Sra, M.; et al. 2024. Artificial Leviathan: Exploring Social Evolution of LLM Agents Through the Lens of Hobbesian Social Contract Theory. arXiv preprint arXiv:2406.14373
arXiv 2024
Show all 32 references
-
[9]
Davoodi, M.; Waltenburg, E.; and Goldwasser, D. 2020. Understanding the language of political agreement and disagreement in legislative texts. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 5358--5368
2020
-
[10]
Davoodi, M.; Waltenburg, E.; and Goldwasser, D. 2022. Modeling US state-level policies by extracting winners and losers from legislative texts. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 270--284
2022
-
[11]
Feng, S.; Chen, Z.; Zhang, W.; Li, Q.; Zheng, Q.; Chang, X.; and Luo, M. 2021. Kgap: Knowledge graph augmented political perspective detection in news media. arXiv preprint arXiv:2108.03861
2021 arXiv
-
[12]
Feng, S.; Tan, Z.; Chen, Z.; Wang, N.; Yu, P.; Zheng, Q.; Chang, X.; and Luo, M. 2022. PAR: Political actor representation learning with social context and expert knowledge. arXiv preprint arXiv:2210.08362
2022 arXiv
-
[13]
M.; and Blei, D
Gerrish, S. M.; and Blei, D. M. 2011. Predicting legislative roll calls from text. In Proceedings of the 28th International Conference on Machine Learning, ICML 2011
2011
-
[14]
Gu, Y.; Sun, Y.; Jiang, N.; Wang, B.; and Chen, T. 2014. Topic-factorized ideal point estimation model for legislative voting network. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, 183--192
2014
-
[15]
Horton, J. J. 2023. Large language models as simulated economic agents: What can we learn from homo silicus? Technical report, National Bureau of Economic Research
2023
-
[16]
Kim, J.; Kovach, M.; Lee, K.-M.; Shin, E.; and Tzavellas, H. 2024. Learning to be Homo Economicus: Can an LLM Learn Preferences from Choice. arXiv preprint arXiv:2401.07345
2024 arXiv
-
[17]
Kraft, P.; Jain, H.; and Rush, A. M. 2016. An embedding model for predicting roll-call votes. In Proceedings of the 2016 conference on empirical methods in natural language processing, 2066--2070
2016
-
[18]
Li, C.; and Goldwasser, D. 2019. Encoding social information with graph convolutional networks forpolitical perspective detection in news media. In Proceedings of the 57th annual meeting of the association for computational linguistics, 2594--2604
2019
-
[19]
Li, G.; Hammoud, H.; Itani, H.; Khizbullin, D.; and Ghanem, B. 2023. Camel: Communicative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, 36: 51991--52008
2023
-
[21]
Majumdar, S.; Elkind, E.; and Pournaras, E. 2024 b . Generative AI Voting: Fair Collective Choice is Resilient to LLM Biases and Inconsistencies. arXiv preprint arXiv:2406.11871
2024
-
[22]
Mou, X.; Li, Z.; Lyu, H.; Luo, J.; and Wei, Z. 2024. Unifying Local and Global Knowledge: Empowering Large Language Models as Political Experts with Knowledge Graphs. In Proceedings of the ACM on Web Conference 2024, 2603--2614
2024
-
[23]
Mou, X.; Wei, Z.; Chen, L.; Ning, S.; He, Y.; Jiang, C.; and Huang, X.-J. 2021. Align voting behavior with public statements for legislator representation learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internat...
2021
-
[24]
Mou, X.; Wei, Z.; and Huang, X. 2024. Unveiling the truth and facilitating change: Towards agent-based large-scale social movement simulation. arXiv preprint arXiv:2402.16333
2024 arXiv
-
[25]
Mou, X.; Wei, Z.; Zhang, Q.; and Huang, X.-J. 2023. Uppam: A unified pre-training architecture for political actor modeling based on language. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 11996--12012
2023
-
[26]
T.; and Rosenthal, H
Poole, K. T.; and Rosenthal, H. 1985. A spatial model for legislative roll call analysis. American journal of political science, 357--384
1985
-
[27]
Qian, C.; Cong, X.; Yang, C.; Chen, W.; Su, Y.; Xu, J.; Liu, Z.; and Sun, M. 2023. Communicative agents for software development. arXiv preprint arXiv:2307.07924
2023 arXiv
-
[28]
Vafa, K.; Naidu, S.; and Blei, D. 2020. Text-Based Ideal Points. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 5345--5357
2020
-
[29]
Wang, L.; Ma, C.; Feng, X.; Zhang, Z.; Yang, H.; Zhang, J.; Chen, Z.; Tang, J.; Chen, X.; Lin, Y.; et al. 2024. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6): 186345
2024
-
[30]
Wu, Z.; Wang, Z.; Xu, X.; Lu, J.; and Yan, H. 2023. Embodied task planning with large language models. arXiv preprint arXiv:2307.01848
2023 arXiv
-
[31]
C.; Korecki, M.; Dailisan, D.; Hausladen, C
Yang, J. C.; Korecki, M.; Dailisan, D.; Hausladen, C. I.; and Helbing, D. 2024. Llm voting: Human choices and ai collective decision making. arXiv preprint arXiv:2402.01766
2024 arXiv
-
[32]
Yang, Y.; Lin, X.; Lin, G.; Huang, Z.; Jiang, C.; and Wei, Z. 2021. Joint representation learning of legislator and legislation for roll call prediction. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, ...
2021
-
[34]
Zhou, Y.; Ni, Y.; Liu, X.; Zhang, J.; Liu, S.; Ye, G.; and Chai, H. 2024 b . Are Large Language Models Rational Investors? arXiv preprint arXiv:2402.12713
2024 arXiv
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.