{"id":"295d07bf-bdd5-40f4-a1ae-855c4efe53d7","arxiv_id":"2412.07144","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"PAA, a role-playing LLM agent with multi-view planning and leader-follower influence, reports 91.8-92.1% accuracy on U.S. House roll-call prediction.","lead":"This paper introduces PAA, an LLM agent system that role-plays U.S. House members, reasons through trustee, delegate, and follower viewpoints, and lets leader agents influence followers, to predict roll-call votes. On 117th-118th House data, PAA with GPT-4o-mini reports 91.8-92.1% accuracy, beating embedding-based baselines, with written vote explanations.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline accuracy may be a pretraining-memorization artifact: the test roll calls predate or overlap LLM training data, and the PAA-ano control leaves bill text intact, so the model could recall outcomes instead of simulating legislators.","rationale":"I read the paper in good faith. The framework is clearly described, the ablations are thoughtfully designed, and the numbers are internally consistent. But the central claim that PAA provides a scalable and interpretable paradigm for roll-call prediction rests on an untested assumption: that the base LLM has not memorized the test roll-call labels. The authors explicitly pose this as RQ1, yet their PAA-ano experiment does not answer it because bill content remains in the prompt and is sufficient for a pretrained model to retrieve the known outcome. The reader's weakest assumption identifies exactly this gap, and I agree with that assessment. The concrete post-cutoff evaluation would settle whether the concern lands: if the advantage disappears on votes after the models' training cutoffs, the headline comparison against baselines is not evidence of predictive skill. Until that check is run, the REJECT verdict is appropriate, and my stress-test does not change it.","tokens_in":10740,"tokens_out":4746,"duration_ms":56220,"concrete_test":"Evaluate PAAG and PAAL on a temporal holdout of roll-call votes whose outcomes were not publicly available before the base models' stated knowledge cutoffs, e.g., 118th House votes after 2023-10-01 for GPT-4o-mini and after 2023-03-01 for Llama-3-70B, using the same prompts, profile construction, and influence mechanism, and compare against the five baselines trained only on pre-cutoff votes. If PAAG's post-cutoff accuracy/F1 is not significantly better than the best baseline, the reported advantage is explained by pretraining memorization. A complementary check is to rerun PAA with bill titles and bill text replaced by opaque content-free placeholders; if accuracy collapses, the bill-content channel is what the model is exploiting.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that PAA predicts roll-call votes and outperforms trained baselines. For this to hold, the LLM's outputs must reflect the proposed simulation and reasoning, not memorized test labels. The evaluation uses real 117th-118th House votes, and a large fraction of the test votes occurred before or during the pretraining data windows of GPT-4o-mini and Llama-3-70B. The only direct contamination control is the PAA-ano ablation in Table 3, which randomizes legislator names and bill numbers but keeps the full bill title and text in the prompt. A model that has memorized the eventual fate of a bill from news articles, Wikipedia, or legislative-tracker text can answer correctly without using the constructed profile, the multi-view planning, or the influence mechanism. PAA-ano still achieving 90.8 accuracy therefore does not rule out contamination; it only shows that names and bill numbers are not the retrieval cue. The trained baselines, in contrast, are fit only on the given training split and cannot exploit the same look-ahead, so the comparison is not a valid test of predictive skill. This concern is load-bearing for both the accuracy and interpretability claims: if the model is recalling labels, the human-readable reasons in Figure 5 are post-hoc rationalizations rather than evidence about legislative behavior. The paper itself flags the risk as RQ1, but the designed experiment does not answer it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes the Political Actor Agent (PAA), an LLM-based framework that combines a scalable legislator profile, a multi-view planning module (trustee, delegate, follower), and a leader-follower influence mechanism to predict roll-call votes. It evaluates PAA on 117th-118th U.S. House voting data under three chronological splits, reporting that PAA with GPT-4o-mini (PAAG) outperforms five embedding-based baselines in accuracy and macro-F1 (e.g., 91.8 acc / 92.2 F1 on split244), while also providing human-readable decision rationales. Ablation studies in Section 4.3-4.4 decompose the contributions of the profile, planning, and action modules and probe the effect of profile components and length.","tokens_in":10923,"tokens_out":4291,"duration_ms":47776,"significance":"If the evaluation were clean, the paper would be a useful contribution: it offers an interpretable, data-efficient alternative to trained embedding models, and the modular design (profile/planning/action) is a natural way to inject political-science knowledge into LLM predictions. The ablations are thoughtful, and the consistency analysis in Section 4.5 directly addresses the hallucination concern, which is a strength. However, the significance of the headline accuracy claim hinges entirely on excluding pretraining memorization: because the test votes largely predate or overlap the base models' training data, the reported gains over baselines cannot currently be interpreted as predictive skill. This is a load-bearing, unresolved issue.","major_comments":[{"comment":"The RQ1 control PAA-ano anonymizes legislator names and bill numbers but leaves the full bill title and text in the prompt. For test votes from the 117th and 118th House, which predate or overlap the training windows of GPT-4o-mini and Llama-3-70B, the model can recall the eventual outcome of a bill (e.g., H.R. 1096 in Figure 5) from pretraining corpora such as news articles, Wikipedia, or legislative trackers. The reported 90.8 accuracy for PAA-ano therefore does not rule out label memorization; it only shows that names and bill numbers are not the retrieval cue. The conclusion in the same section that 'PAA likely relies on the information in our profile module for predictions' is not supported by this control. A necessary condition is to remove bill content from the prompt, or to restrict the test set to votes that postdate the model's knowledge cutoff.","section":"§4.4 Analysis of Profile Module (RQ1, PAA-ano)"},{"comment":"The chronological splits (split244/433/622) do not prevent label leakage from pretraining because the base LLMs were trained on web text covering the same time period as many of the test votes. The comparison in Table 1 is therefore not a fair predictive comparison: the embedding baselines are fit only on the given training split, while the LLM has potential access to the test outcomes through its pretraining. The paper should report the knowledge cutoff dates of both base models, isolate a test subset of votes occurring after those cutoffs, and/or include a baseline with the same look-ahead exposure. Without this, the 91.8-92.1 PAAG accuracy cannot be taken as evidence of predictive skill over the baselines.","section":"§4.1-§4.2 Datasets, Baselines, and Chronological Split"},{"comment":"The PAA-Dec experiment swaps legislator names but keeps bill text, so it shares the same memorization flaw as PAA-ano. The small accuracy drop from PAA-ano (90.8) to PAA-Dec (90.1) is interpreted as evidence that legislator information affects predictions, but it is equally consistent with the model using bill-content cues while being slightly perturbed by inconsistent name information. This weakens the paper's claims about the relative contributions of profile components (e.g., constituency information being least important), because all such comparisons operate under the unresolved contamination risk.","section":"§4.4 PAA-Dec and interpretation of profile effects"}],"minor_comments":[{"comment":"The paper does not provide code, data, or the actual prompt templates; the appendix containing the prompts is referenced but not included in this arXiv version. The description of the profile module (Section 3.1) and the view prompts (Section 3.2) is too high-level to reproduce the method without guessing.","section":"General reproducibility"},{"comment":"The paper does not report decoding parameters (temperature, top-p, number of samples) for the LLM experiments, despite emphasizing consistency and reporting standard deviations over five runs. These settings are important for interpreting the variance and the consistency results.","section":"§4.2 Implementation"},{"comment":"The consistency analysis is a positive feature, but the heatmap in Figure 4 would be more informative with a numerical summary, such as the proportion of agent-bill pairs that are correct in all 20 runs, and the test-set size should be stated.","section":"§4.5 Consistency Analysis"},{"comment":"The reference list contains duplicate entries: Majumdar et al. 2024a and 2024b are identical, and Zhou et al. 2024a and 2024b are identical; these should be merged or disambiguated.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The core issue is that the experimental design conflates memorization with prediction. I do not think this is a rejection in principle, because the paper could be repaired by adding a post-cutoff evaluation and a control that removes bill text from the prompt. The authors already show awareness of the risk via RQ1, which suggests the fix is within scope. In revision, I would also ask for the prompt templates and exact model/date information, as these are currently missing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a clean, readable agent paper with a real architectural contribution, but the central claim—that PAA predicts roll-call votes better than trained baselines—is not supported as submitted. The likely reason for the high accuracy is that the base LLMs already know the outcomes of the 117th/118th House votes, and the PAA-ano control leaves bill text intact.\n\nWhat's actually new: combining a scalable profile with multi-view planning (trustee/delegate/follower) and the leader-agent influence mechanism for roll-call prediction. That combination isn't in the cited ideal-point, graph, or plain-LLM-simulation work. The paper also does a decent job of ablating the modules, reporting consistency, and showing one interpretable example. The writing is clear and the ablations are thoughtful.\n\nWhere it falls short: the contamination problem is load-bearing. RQ1 is exactly the right question, but the designed experiment doesn't answer it. PAA-ano replaces names and bill numbers with random IDs but keeps the bill title and text, so a model that remembers 'the Commemorative Coin Act passed Yea' can answer without doing any simulation. PAA-Dec is a clever sanity check, but it only shows that names matter, not that the model is reasoning from the profile. There is no post-cutoff evaluation, and no no-bill-text baseline. Since the trained baselines only see the training split, the comparison is not a fair test of predictive skill.\n\nThe smaller issues: the equations for the influence mechanism are descriptive, not derivational, which is fine but shouldn't be oversold. The interpretability example is a single hand-picked case. No code, prompts, or data are released, so reproducibility is at sketch level. The lack of release matters more here because the whole contribution depends on prompt construction.\n\nThe paper is honestly written—the limitations section is straightforward, and RQ1 shows the authors know the risk. The citation pattern looks fine. The math is simple but not wrong.\n\nBottom line: I'd bring this to a reading group as a case study in LLM evaluation contamination, but I wouldn't cite the headline numbers. It deserves peer review, not a desk reject, because the framework is potentially useful and the flaw is fixable. Ask the authors for a post-cutoff split (e.g., votes after the model's knowledge cutoff), a control with bill text removed, and released artifacts. With those, the paper could be a solid contribution to computational political science.","headline":"Useful agent-design paper, but the headline prediction numbers are contaminated by the models' pretraining, and the paper's own control doesn't rule it out.","tokens_in":11568,"tokens_out":2377,"would_cite":false,"duration_ms":23371,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The Political Actor Agent (PAA) claims that simulating legislators as role-playing LLM agents—with scalable profiles, multi-view planning, and a leadership influence mechanism—predicts roll-call votes more accurately and more…","keywords":["roll-call vote prediction","political actor modeling","large language models","LLM agents","role-playing","legislative simulation","influence mechanism","interpretability"],"falsifier":"Build a test set of roll-call votes on bills introduced after the LLM's knowledge cutoff (for example, bills in the current Congress with votes not present in the model's training data), run the same PAA prompts with the same 20-vote profiles, and compare accuracy against the embedding baselines trained only on the designated training split; if PAA's edge shrinks to noise on post-cutoff bills, that would show the reported advantage came from memorized historical votes rather than from the simulation mechanism.","tokens_in":10408,"feed_emoji":"🗳️","tokens_out":5512,"duration_ms":49194,"temperature":0.7,"pith_summary":"The paper tries to establish that roll-call vote prediction can be reframed as an agent-simulation problem: instead of learning vector embeddings of legislators from large labeled datasets, you populate profiles of each legislator into an LLM, ask it to reason from trustee, delegate, and follower perspectives, and let an influence mechanism propagate leader votes to followers. If true, it would matter because vote prediction would no longer require hand-defined features, large training corpora, or opaque embedding vectors—it would work from a handful of recent votes, degrade gracefully with scarce data, and return human-readable justifications alongside each prediction. The reported numbers on 117th and 118th House votes put PAA with GPT-4o-mini at 91.8 accuracy / 92.2 macro-F1 on the smallest training split, ahead of five baselines, with the open-weight Llama-3-70B version best on macro-F1 among non-GPT runs. The paper's own ablation shows anonymizing legislator names and bill numbers only slightly reduces accuracy, which the authors read as evidence that the agent reasons from profile content rather than memorized identities.","feed_headline":"LLM agents simulate Congress to predict roll-call votes","feed_subtitle":"Role-playing legislators with trustee, delegate, and leader-influence views beat trained models and explain each vote.","key_machinery":"The load-bearing object is the three-module agent pipeline: the Profile Construction Module (a scalable prompt-level profile holding personal information, constituency details, sponsorship activity, and a sample of 20 recent voting records per legislator), the Multi-view Planning Module (which decomposes the vote into trustee, delegate, and follower perspectives and synthesizes them), and the Simulated Legislative Action Module (an influence mechanism where leader agents L = {Speaker, Republican Leader, Democratic Leader, committee chair, caucus members} vote first via multi-view planning and the remaining agents vote conditioned on the leaders' outcomes: V_l = p(L), V_o = p(O | V_l)). The claim is that this pipeline—rather than any learned weights—carries the predictive power, and the profile module carries the largest share, as the ablation dropping it costs roughly 13 accuracy points.","core_discovery":"The central discovery claimed is that an agent-based paradigm—Political Actor Agent (PAA)—can match or exceed embedding-based political-actor models at roll-call vote prediction while also producing interpretable, multi-view decision reasoning. PAA constructs a scalable textual profile for each legislator (personal background, constituency demographics, sponsorship activity, and sampled past votes), decomposes the voting decision into trustee, delegate, and follower views synthesized into a final stance, and then simulates legislative dynamics by having leader agents (Speaker, party leaders, committee chair, relevant caucus members) vote first, with remaining agents' prompts conditioned on the leaders' votes. On the 117th–118th U.S. House data, PAAG (GPT-4o-mini) reaches 91.8/92.2 accuracy/F1 on split244 and 92.1/93.0 on split622, consistently above the ideal-point, graph-neural-network, and pre-trained baselines, while PAAL (Llama-3-70B) leads on macro-F1 among non-PAAG methods. The authors conclude that PAA offers a scalable and interpretable paradigm that degrades less than trained baselines when training data is scarce, and that its reasoning traces give political science insights into how legislators weigh constituency, expertise, and party leadership.","pith_inferences":["If the memorization concern is set aside, a direct testable extension is to run PAA on a legislature not in the LLM's pretraining corpus (for example, a non-English parliament with recent votes) to isolate the simulation's transfer value from any U.S.-specific knowledge.","The result that longer voting histories hurt accuracy suggests a context-window bottleneck: a retrieval policy that selects the most bill-relevant past votes per legislator would be a natural improvement that the paper does not test.","The leader-follower conditioning conflates two effects—the information content of leader votes and the prompting effect of 'here is what leaders did'—and the ablation removing the acting module cannot separate them; a targeted ablation that feeds leaders' votes without a leadership label would isolate the mechanism.","The interpretability claim, if it holds, gives computational political science a cheap instrument for counterfactual analysis: swapping a legislator's district demographics or party label in the profile and observing how the predicted vote and its stated reasons change."],"forward_implications":["With only 20 sampled past votes per legislator, PAA stays accurate as the training split shrinks, suggesting it can predict votes of newly elected legislators where embedding methods lack data.","Because PAA returns trustee, delegate, and follower reasoning for each vote, it can generate per-legislator explanations that link a vote to constituency, expertise, career focus, and party leadership.","PAA's influence mechanism conditions every non-leader vote on the leaders' predicted votes, so the framework explicitly models party and committee leadership effects without training a network to learn them.","Removing any single profile component (personal info, constituency, sponsorship, voting records) costs accuracy, but the full-profile version with 20 sampled records outperforms versions fed the entire training set, indicating bounded context is better than exhaustive history."],"supporting_citations":[{"why":"Supplies the ideal-vector baseline that PAA must beat and the multidimensional embedding approach to roll-call prediction.","marker":"Kraft, Jain, and Rush 2016"},{"why":"Supplies the LSTM+GCN baseline and the cosponsorship-informed graph modeling that PAA compares against.","marker":"Yang et al. 2021"},{"why":"Supplies the Vote+MTL baseline that incorporates Twitter statements via relational graph convolution.","marker":"Mou et al. 2021"},{"why":"Supplies the PAR baseline combining social context and expert knowledge, the second-best non-agent method on the largest split.","marker":"Feng et al. 2022"},{"why":"Supplies the UPPAM pre-training baseline that maps language to politician representations, the strongest embedding-based competitor.","marker":"Mou et al. 2023"},{"why":"Provides the trustee/delegate/follower representation views that organize the multi-view planning module.","marker":"Alexander 2019"},{"why":"Supplies the role-playing architecture that PAA adapts for constructing legislator agents.","marker":"Li et al. 2023"},{"why":"Establishes the precedent of using LLM agents to simulate human samples, which PAA extends from demographic profiles to detailed political actor profiles.","marker":"Argyle et al. 2023"}],"fun_headline_variants":["LLM agents role-play Congress to predict votes with explanations","Simulated legislators: LLM agents outperform trained vote models","Political Actor Agent: interpretable roll-call forecasting via LLM role-play","LLM agents simulate legislative dynamics to predict roll calls"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the base LLM has not memorized the test roll-call labels during pretraining; if the model already knows that the 117th House passed H.R. 1096, then feeding it the bill title and asking for a vote can yield high accuracy by recall rather than by the simulated reasoning the paper describes.","fun_headline_variants_meta":{"raw":{"variants":["LLM agents role-play Congress to predict votes with explanations","Simulated legislators: LLM agents outperform trained vote models","Political Actor Agent: interpretable roll-call forecasting via LLM role-play","LLM agents simulate legislative dynamics to predict roll calls"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001034,"raw_usage":{"total_tokens":4382,"prompt_tokens":999,"completion_tokens":3383,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":3313}},"tokens_in":615,"tokens_out":3383,"duration_ms":24814,"temperature":1.0,"reasoning_tokens":3313,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:04:46.760797+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a test set of roll-call votes on bills introduced after the LLM's knowledge cutoff (for example, bills in the current Congress with votes not present in the model's training data), run the same PAA prompts with the same 20-vote profiles, and compare accuracy against the embedding baselines trained only on the designated training split; if PAA's edge shrinks to noise on post-cutoff bills, that would show the reported advantage came from memorized historical votes rather than from the simulation mechanism.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ideal-vector baseline that PAA must beat and the multidimensional embedding approach to roll-call prediction."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the LSTM+GCN baseline and the cosponsorship-informed graph modeling that PAA compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Vote+MTL baseline that incorporates Twitter statements via relational graph convolution."},{"cited_title":"PAR: Political Actor Representation Learning with Social Context and Expert Knowledge","cited_arxiv_id":"2210.08362","evidence_quote":"Supplies the PAR baseline combining social context and expert knowledge, the second-best non-agent method on the largest split."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the UPPAM pre-training baseline that maps language to politician representations, the strongest embedding-based competitor."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the trustee/delegate/follower representation views that organize the multi-view planning module."}],"review_version":1}