pith. sign in

arxiv: 2605.18395 · v1 · pith:CHLSQ2OOnew · submitted 2026-05-18 · 💻 cs.CY · cs.AI

Diagnosing Korean-Language LLM Political Bias via Census-Grounded Agent Simulation

Sungwoo Kang This is my paper

Pith reviewed 2026-05-19 23:59 UTC · model grok-4.3

classification 💻 cs.CY cs.AI
keywords LLM political biasKorean electionsagent-based simulationcensus databias mitigationelection predictionvoter modelingthird-party salience
0
0 comments X

The pith

Census-grounded agent simulations diagnose political biases in Korean LLMs and accurately predict real election outcomes.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces Dynamo-K, a framework that builds voter agents from Korean census data and prompts them with election scenarios to test how language models behave politically. It documents three recurring problems across models: moderate agents leaning progressive, third-party options disappearing from consideration, and failure to capture regional party strongholds. Reframing the scenarios and adding a learned reweighting step improve simulation accuracy, and the same pipeline correctly forecasts the winner in every recent presidential race plus the dominant party in a separate local contest. A sympathetic reader would care because the method offers a repeatable, low-cost way to check and adjust AI systems that might shape political discussion or analysis.

Core claim

Dynamo-K is a census-grounded simulation framework that evaluates four Korean-language LLMs on six elections from 2017 to 2025. It identifies three systematic failure modes: progressive bias in moderate agents that explicit mitigation reduces by a factor of 5.2 in mean absolute error, model-dependent third-party salience collapse, and bidirectional under-prediction of historical regional polarization. Scenario reframing recovers 62 percent of the 2017 mean absolute error, and a learned reweighting adapter calibrates opposing-valence models without using candidate names at training or test time. The framework predicts the winner in all three presidential races, including 2.1 percentage point

What carries the argument

Dynamo-K, the census-grounded agent simulation framework that constructs representative voter profiles from demographic data and elicits model responses through election scenario prompts.

If this is right

  • Explicit mitigation steps can cut mean absolute error by more than five times when addressing progressive bias in moderate voter profiles.
  • Reframing election scenarios restores visibility of third-party candidates and recovers most of the accuracy lost to salience collapse.
  • A learned reweighting adapter can align models with opposing political leanings without ever referencing specific candidate names.
  • The same pipeline supplies a scalable diagnostic that works across multiple election cycles and held-out local races.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same census-agent approach could be adapted to other countries or languages by swapping in local demographic data and election contexts.
  • Developers might embed similar reweighting adapters into production LLMs to reduce systematic political skew in general queries.
  • Testing the framework on non-election political topics could reveal whether the identified failure modes appear outside voting scenarios.
  • Combining the simulations with periodic real polling data might allow ongoing calibration as voter preferences shift.

Load-bearing premise

Census-derived agent profiles and election scenario prompts generate response distributions close enough to real Korean voter behavior to expose genuine model biases.

What would settle it

Running Dynamo-K on a future Korean national or local election and finding that the simulated vote shares or winner deviate substantially from the official results.

Figures

Figures reproduced from arXiv: 2605.18395 by Sungwoo Kang.

Figure 1
Figure 1. Figure 1: Dynamo-K pipeline architecture. Layers 1–2 are data infrastructure; layers 3–4 synthesize [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: System prompt for a moderate voter agent. The prompt encodes demographics, political [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Predicted vs. actual vote shares for the three presidential elections. The 2022 election [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
read the original abstract

Large language models (LLMs) exhibit systematic political biases in voter simulations, but their underlying mechanisms and cross-lingual generalizations remain poorly understood. We introduce Dynamo-K, a census-grounded simulation framework evaluating Korean-language LLM political behavior across four models on six Korean elections (2017-2025). Using this framework, we identify three systematic failure modes: (1) progressive bias in moderate agents, where explicit mitigation reduces Mean Absolute Error (MAE) by 5.2 times; (2) model-dependent third-party salience collapse, distinguishing between salience failure and decision bias; and (3) regional polarization collapse, where models bidirectionally under-predict historical party strongholds. To address these failures, we demonstrate that scenario reframing recovers 62% of 2017 MAE by restoring third-party visibility. Furthermore, we introduce a learned reweighting adapter that successfully calibrates opposing-valence models without relying on candidate names at train or test time. Validating our diagnostic framework, Dynamo-K accurately predicts 3/3 presidential winners - including a 2.1%p MAE on the highly contested 0.73%p-margin 2022 race - and correctly identifies the dominant party in a held-out local election. The pipeline is open-source and provides a scalable, cost-effective method for diagnosing LLM political behavior.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces Dynamo-K, a census-grounded agent simulation framework for diagnosing political biases in Korean-language LLMs. It evaluates four models across six elections (2017-2025), identifies three failure modes (progressive bias in moderate agents, third-party salience collapse, and regional polarization collapse), proposes mitigations via scenario reframing (recovering 62% of 2017 MAE) and a learned reweighting adapter, and validates the framework through accurate prediction of 3/3 presidential winners including 2.1%p MAE on the 2022 race (0.73%p margin) plus correct dominant-party identification in a held-out local election. The pipeline is open-source.

Significance. If the simulations prove representative, the work supplies a scalable, cost-effective, and reproducible method for cross-lingual LLM bias diagnosis, with explicit credit for the open-source pipeline and falsifiable election-outcome predictions. The identification of distinct failure modes (salience vs. decision bias) adds diagnostic granularity to the literature on LLM political simulation.

major comments (2)
  1. [Validation results] Validation paragraph (abstract and results): the claim that accurate winner prediction and 2.1%p MAE on the 2022 race validates the diagnostic framework is load-bearing, yet the manuscript supplies no comparison of simulated vs. observed vote shares by age band, region, or education, nor calibration against historical exit-poll marginals. Absent this check, the low MAE could result from prompt engineering that encodes known outcomes rather than emergent representative behavior.
  2. [Methods] Methods (agent construction): the central assumption that census-grounded profiles plus scenario prompts yield voter distributions sufficiently close to the Korean electorate is not supported by sensitivity analysis showing MAE stability under perturbations within census sampling error. This directly affects whether the reported bias failure modes can be attributed to the LLMs rather than simulation artifacts.
minor comments (2)
  1. [Abstract] Abstract: the phrase 'six Korean elections (2017-2025)' is used without listing the exact contests or the four models evaluated; adding this enumeration would aid immediate comprehension.
  2. [Abstract] Notation: MAE is introduced in the abstract without prior expansion, although later clarified; consistent first-use definition would improve readability.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive and detailed comments, which help clarify the validation and methodological foundations of Dynamo-K. We address each major comment below and outline revisions to strengthen the manuscript.

read point-by-point responses

  1. Referee: [Validation results] Validation paragraph (abstract and results): the claim that accurate winner prediction and 2.1%p MAE on the 2022 race validates the diagnostic framework is load-bearing, yet the manuscript supplies no comparison of simulated vs. observed vote shares by age band, region, or education, nor calibration against historical exit-poll marginals. Absent this check, the low MAE could result from prompt engineering that encodes known outcomes rather than emergent representative behavior.

    Authors: We agree that aggregate winner prediction and overall MAE, while informative, do not by themselves rule out the possibility of prompt-level encoding of known results. The 2022 race was selected precisely because its narrow 0.73%p margin makes such encoding unlikely without explicit vote-share cues (which are absent from the scenario prompts), and the held-out local election provides an additional falsifiable test. Nevertheless, the referee's point is well-taken: explicit demographic and exit-poll calibration would materially strengthen the claim that the observed behavior is emergent. In the revised manuscript we will add a new supplementary table and figure that compare simulated versus observed vote shares by age band and region for the 2022 election, using publicly available exit-poll marginals. We will also report calibration error against these marginals to address the concern directly. revision: yes

  2. Referee: [Methods] Methods (agent construction): the central assumption that census-grounded profiles plus scenario prompts yield voter distributions sufficiently close to the Korean electorate is not supported by sensitivity analysis showing MAE stability under perturbations within census sampling error. This directly affects whether the reported bias failure modes can be attributed to the LLMs rather than simulation artifacts.

    Authors: The agent population is constructed by stratified sampling directly from the most recent Korean census micro-data, so the marginal distributions on age, region, education, and income are matched by design. The three failure modes appear consistently across four independent models and six elections, which would be improbable if they were driven by sampling fluctuations within census error margins. That said, we accept that an explicit sensitivity check would make the attribution to LLM behavior more robust. We will therefore add a sensitivity analysis subsection in the revised Methods: we will re-sample the agent pool 50 times with demographic weights perturbed by the reported census sampling errors and demonstrate that both the aggregate MAE and the qualitative failure-mode signatures remain stable. These results will be reported in a new table. revision: yes

Circularity Check

0 steps flagged

No significant circularity; derivation relies on external election benchmarks

full rationale

The paper constructs agent profiles from census data and uses LLM responses under scenario prompts to generate simulated vote distributions, then compares those outputs directly against observed historical election results (presidential winners and margins from 2017-2022 plus a held-out local election). No equation or procedure is shown that defines the target vote shares in terms of the simulation outputs themselves, nor does any reported 'prediction' reduce to a parameter fitted on the same election data being evaluated. The learned reweighting adapter is described as operating without candidate names at train or test time, and the primary validation claims rest on out-of-sample matches to real-world outcomes rather than internal consistency. This structure keeps the central diagnostic claims independent of the inputs they are tested against.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The central claims rest on the representativeness of census-derived agent profiles and the validity of MAE as a bias measure; no explicit free parameters or invented entities are described in the abstract.

pith-pipeline@v0.9.0 · 5759 in / 1013 out tokens · 23805 ms · 2026-05-19T23:59:58.839206+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

  • IndisputableMonolith/Foundation/RealityFromDistinction.lean reality_from_one_distinction unclear
    ?
    unclear

    Relation between the paper passage and the cited Recognition theorem.

    Dynamo-K processes electoral prediction through a six-layer pipeline: data collection from government APIs and academic surveys, preprocessing into joint distributions, agent synthesis with census-grounded demographics, belief seeding and calibration, LLM-based vote simulation, and result aggregation with evaluation metrics.

  • IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear
    ?
    unclear

    Relation between the paper passage and the cited Recognition theorem.

    We identify three systematic failure modes: (i) progressive bias in moderate agents... (ii) model-dependent third-party salience collapse... (iii) regional polarization collapse...

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

19 extracted references · 19 canonical work pages

  1. [1]

    Dynamo: Large-scale political simulation with LLM agents

    Aaru. Dynamo: Large-scale political simulation with LLM agents. Company product description, accessed 2026.https://aaru.com

    work page 2026

  2. [2]

    L. P. Argyle, E. C. Busby, N. Fulda, J. R. Gubler, C. Rytting, and D. Wingate. Out of one, many: Using language models to simulate human samples.Political Analysis, 31(3):337–351, 2023

    work page 2023

  3. [3]

    M. D. Jenkins and H. J. Kim. The role of misogyny in the 2022 Korean presidential election: Under- standing the backlash against feminism in industrialized democracies.Journal of East Asian Studies, 24(2):169–196, 2024.https://doi.org/10.1017/jea.2024.11

    work page doi:10.1017/jea.2024.11 2022

  4. [4]

    S. Feng, C. Y. Park, Y. Liu, and Y. Tsvetkov. From pretraining data to language models to downstream tasks: Tracking the trails of political biases leading to unfair NLP models. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL 2023), pages11737–11762, 2023. Best Paper Award

    work page 2023

  5. [5]

    Deffuant, D

    G. Deffuant, D. Neau, F. Amblard, and G. Weisbuch. Mixing beliefs among interacting agents.Advances in Complex Systems, 3(01n04):87–98, 2000

    work page 2000

  6. [6]

    FlockVote: LLM-Empowered Agent-Based Modeling for Simulating U.S

    FlockVote. FlockVote: LLM-Empowered Agent-Based Modeling for Simulating U.S. Presidential Elec- tions. 2025

    work page 2025

  7. [7]

    N. E. Friedkin and E. C. Johnsen. Social influence and opinions.Journal of Mathematical Sociology, 15(3–4):193–206, 1990

    work page 1990

  8. [8]

    Weekly political opinion survey cumulative report

    Gallup Korea. Weekly political opinion survey cumulative report. Technical report, Gallup Korea Research Institute, 2024

    work page 2024

  9. [9]

    R. A. Holley and T. M. Liggett. Ergodic theorems for weakly interacting infinite systems and the voter model.The Annals of Probability, 3(4):643–663, 1975

    work page 1975

  10. [10]

    KoreanGeneralSocialSurveycumulativecodebook 2003–2023

    SungKyunKwanUniversitySurveyResearchCenter. KoreanGeneralSocialSurveycumulativecodebook 2003–2023. Technical report, KOSSDA, 2023

    work page 2003

  11. [11]

    W. C. Kang. Local economic voting and residence-based regionalism in South Korea: Evidence from the 2007 presidential election.Journal of East Asian Studies, 16(3):349–369, 2016

    work page 2007

  12. [12]

    Microdata Integrated Service: 2020 Population and Housing Census 2% sample

    Statistics Korea. Microdata Integrated Service: 2020 Population and Housing Census 2% sample. https://mdis.kostat.go.kr, 2020

    work page 2020

  13. [13]

    Election statistics system (선ᄀ ᅥᄐ ᅩ ᆼᄀ ᅨᄉ ᅵᄉ ᅳ테 ᆷ).https://info

    National Election Commission of Korea. Election statistics system (선ᄀ ᅥᄐ ᅩ ᆼᄀ ᅨᄉ ᅵᄉ ᅳ테 ᆷ).https://info. nec.go.kr, 2025

    work page 2025

  14. [14]

    J. S. Park, J. C. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein. Generative agents: Interactive simulacra of human behavior. InProceedings of UIST 2023, pages 1–22, 2023

    work page 2023

  15. [15]

    PolyPersona: Persona-Grounded LLM for Synthetic Survey Responses

    PolyPersona authors. PolyPersona: Persona-Grounded LLM for Synthetic Survey Responses. 2025

    work page 2025

  16. [16]

    Qwen3 technical report

    Qwen Team. Qwen3 technical report. Technical report, Alibaba Cloud, 2025.https://qwenlm.github. io/blog/qwen3/

    work page 2025

  17. [17]

    Santurkar, E

    S. Santurkar, E. Durmus, F. Ladhak, C. Lee, P. Liang, and T. Hashimoto. Whose opinions do language models reflect? InProceedings of ICML 2023, pages 29971–30004, 2023. 29

    work page 2023

  18. [18]

    W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica. Efficient memory management for large language model serving with PagedAttention. InProceedings of SOSP 2023, pages 611–626, 2023

    work page 2023

  19. [19]

    EXAONE 4.0: Unified large language models integrating non-reasoning and reason- ing modes

    LG AI Research. EXAONE 4.0: Unified large language models integrating non-reasoning and reason- ing modes. Technical report, LG AI Research, 2025. arXiv:2507.11407.https://huggingface.co/ LGAI-EXAONE/EXAONE-4.0-32B. 30

    work page arXiv 2025