REVIEW 8 cited by
Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We introduce a new type of test, called a Turing Experiment (TE), for evaluating to what extent a given language model, such as GPT models, can simulate different aspects of human behavior. A TE can also reveal consistent distortions in a language model's simulation of a specific human behavior. Unlike the Turing Test, which involves simulating a single arbitrary individual, a TE requires simulating a representative sample of participants in human subject research. We carry out TEs that attempt to replicate well-established findings from prior studies. We design a methodology for simulating TEs and illustrate its use to compare how well different language models are able to reproduce classic economic, psycholinguistic, and social psychology experiments: Ultimatum Game, Garden Path Sentences, Milgram Shock Experiment, and Wisdom of Crowds. In the first three TEs, the existing findings were replicated using recent models, while the last TE reveals a "hyper-accuracy distortion" present in some language models (including ChatGPT and GPT-4), which could affect downstream applications in education and the arts.
Forward citations
Cited by 8 Pith papers
-
Can AI Reason Like an Urban Planner? Benchmarking Large Language Models Against Professional Judgment
On UPBench, 25 LLMs show a non-monotonic planning curve—strong Remember/Analyze, weak Understand/Evaluate—with four failure modes that support differential, not blanket, AI delegation.
-
Large language models replicate and predict human cooperation across experiments in game theory
Llama-3.1-8B with a multi-step reasoning-and-filter prompt reproduces human cooperation rates across 121 dyadic games (MSD=0.031, r=0.89), outperforming Nash-equilibrium predictions (MSD=0.096, r=0.78).
-
Super-additive Cooperation in Language Model Agents
Language model agents cooperate more in a prisoner's dilemma when repeated interactions and inter-team competition are combined, but only for some models.
-
The Effect of State Representation on LLM Agent Behavior in Dynamic Routing Games
In a repeated Braess routing game, LLM agents given summarized, regret-based, and own-action-only state representations converge closer to Nash equilibrium and behave more stably than agents given full chat transcript...
-
Modeling Earth-Scale Human-Like Societies with One Billion Agents
Light Society scales LLM-agent social simulations to one billion agents by substituting most LLM interactions with a distilled surrogate model.
-
Using AI to replicate human experimental results: a motion study
Across four motion-verb psycholinguistic tasks, ChatGPT o1 responses correlated strongly with human judgments (Spearman rho around .69 to .96), but methodological gaps limit the strength of the replication claim.
-
Recalibrating the Compass: Integrating Large Language Models into Classical Research Methods
LLMs extend, rather than replace, classical social science methods, with a proposed three-tier bias framework for LLM-augmented surveys.
- Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs
Discussion (0). Continue with ORCID to comment.