REVIEW 2 major objections 5 minor 40 references
A hybrid LLM-ABM framework that updates contact networks from group mobility decisions better matches COVID-19 peak timing and size than static ABM alone.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-10 21:56 UTC pith:6CFGIGJI
load-bearing objection Solid engineering of a million-agent hybrid ABM-LLM loop, but the headline comparison rests on an unmatched baseline and an unvalidated behavioral model. the 2 major comments →
LLM-powered reasoning in agent-based modeling
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
When LLM-generated group-level decisions to stay home are used as a feedback loop that prunes outdoor edges in an activity-based temporal contact network, the resulting epidemic trajectories for COVID-19 in Salt Lake County align more closely with observed weekly cases in both peak timing and cumulative size than otherwise identical ABM-only simulations.
What carries the argument
The HALE framework: a hybrid loop in which an individual-based SIR ABM advances on daily weighted activity networks while, every week, 1,552 LLM agents (grouped by municipality, sex, race and age) return structured yes/no mobility decisions that deactivate the corresponding outdoor contacts for that week.
Load-bearing premise
The zero-shot answers of a single 8-billion-parameter language model, at temperature 0.2, correctly capture how real demographic groups change their outdoor activity when local infection rates rise.
What would settle it
Compare the LLM-derived weekly stay-home probabilities for each age-sex-municipality group against independent mobility traces (cell-phone, survey, or transit data) for Salt Lake County over the same months; systematic mismatch would falsify the behavioral premise.
If this is right
- Population-scale digital twins can incorporate near-real-time behavioral adaptation without requiring continuous high-resolution mobility surveys.
- Public-health scenario analysis can test how different demographic groups respond to incidence signals rather than assuming uniform compliance.
- The same hybrid pattern can be reused for other contagion or diffusion processes that depend on deliberate outdoor activity.
- Network construction methods that fill missing activity locations with spatial kernels become more useful once those edges can be dynamically switched off by LLM feedback.
Where Pith is reading between the lines
- If the LLM stay-home probabilities prove reliable, the same grouping scheme could be applied to other counties or countries with only modest re-prompting.
- The gradual rise in decline probability across age groups suggests that LLMs may encode a form of social-memory dynamics that pure ABMs usually must hard-code.
- The near-zero response for tiny unincorporated places implies that geographic specificity in the prompt is itself a first-order control on model fidelity.
- Coupling weather or policy announcements into the same weekly LLM query would be a low-cost next experiment that the architecture already supports.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces HALE, a hybrid framework that couples a large-scale activity-based temporal contact network ABM (SIR on ~1.1M synthetic agents for Salt Lake County) with weekly zero-shot Llama-3.1-8B inferences over 1,552 demographic-spatial groups. LLM yes/no decisions on outdoor activity selectively deactivate deliberate edges in G(t), allowing mobility to respond to simulated incidence. As a proof-of-concept for COVID-19 (Sept 2021–Feb 2022, Omicron parameters), the mean of 30 HALE runs better matches CDC peak timing and produces a total size ~2.21 imes observed incidence (argued to be consistent with asymptomatic under-ascertainment), while an ABM-only baseline with the same networks and parameters overestimates even after a post-hoc 35% random contact deactivation. Supporting analyses show age- and municipality-differentiated stay-home probabilities that rise gradually with incidence.
Significance. If the comparative claim holds, HALE offers a practical, scalable route to inject adaptive human behavior into population-scale digital twins without replacing every agent by a generative model—an important engineering contribution for policy-oriented epidemic ABMs that currently rely on static survey-derived activity schedules. Strengths that should be credited include the carefully specified network-construction pipeline (anchored vs. non-anchored activities, class-level school contacts, exponential kernels with literature λ values), the HPC server-client integration that keeps LLM inference tractable for millions of agents, the public Zenodo release of code and data, and the explicit ablation-style comparison of HALE vs. ABM-only. These elements make the work a concrete, reproducible step beyond purely conceptual LLM-ABM proposals.
major comments (2)
- Results §3.1 and Fig. 3: the central claim that HALE better captures peak timing and size rests on an unmatched ABM-only control. The paper states that 35% of contacts are randomly deactivated “for a fair comparison” so that average degree roughly matches HALE after LLM updates. Random thinning preserves the original activity mix and spatial structure; HALE instead removes whole outdoor-activity classes for entire demographic-spatial groups in an incidence-dependent, time-varying manner (Architecture §2.3.1–2.3.3). Consequently the improvement cannot be attributed to LLM reasoning rather than to any structured reduction of high-transmission activities or to the particular 35% level chosen after inspecting HALE output. A matched ablation that applies the same activity-type or group-level deactivation schedule without LLM input (or a sensitivity sweep over the deactivation fraction) is req
- §2.3.3 and Results §3.2: the load-bearing premise that zero-shot Llama-3.1-8B (temperature 0.2, structured yes/no) for 1,552 municipality×sex×race×age bins accurately reflects real group-level mobility responses is never validated against observed mobility, survey, or cell-phone data. The temperature choice itself is acknowledged to be decisive (temperature=0 yields near-certain stay-home; 0.7 yields ~0.5), yet no external calibration or sensitivity analysis is reported. Because every network update after each weekly LLM step depends on these outputs, the epidemic-curve improvement remains under-supported until the behavioral module is checked against independent mobility evidence or at least subjected to a systematic temperature/prompt ablation.
minor comments (5)
- Abstract and Introduction state the simulation window as “September 2020 to February 2022,” while Results §3 and Fig. 3 use September 2021–February 2022; the dates should be reconciled throughout.
- §2.2.1: the office degree distribution N(µ=21.154/8, σ²=10.58/8) and the out-of-class school average degree of 4 are taken from literature but never sensitivity-tested; a brief note on robustness would strengthen the network-construction claims.
- Fig. 5 caption and surrounding text: “Harriman” should be “Herriman”; several municipality names are inconsistently capitalized.
- The prompt template in §3.2 is given for a single illustrative agent; a short appendix listing the exact structured-output schema and any system prompt would improve reproducibility.
- References [23] (Zenodo) and the funding acknowledgment are present, but the main text never states the exact Llama-3.1-8B checkpoint or vLLM version used; adding these details would complete the methods.
Circularity Check
No significant circularity: HALE epidemic curves are generated from pretrained zero-shot LLM decisions plus ABM dynamics and compared to external CDC incidence; the 35% baseline adjustment and temperature choice do not force the main claim by construction.
full rationale
The paper's central empirical claim (HALE better matches observed COVID-19 peak timing and size in Salt Lake County than ABM-only under shared SIR parameters and networks) is an external comparison against CDC weekly case data, not a quantity derived from its own inputs. The LLM component uses a fixed pretrained Llama-3.1-8B model in zero-shot mode with structured yes/no outputs; its weights were not trained or fine-tuned on the Salt Lake County incidence series, so the mobility decisions that update G(t) after each weekly step are independent of the target curves. Network construction from NHTS/UrbanPop data and the SIR update rules are standard and do not define the observed incidence. The post-hoc random 35% edge deactivation in the ABM-only control (chosen to approximate HALE's average degree reduction) and the temperature=0.2 setting (selected for non-degenerate age-group patterns) are modeling choices that affect the strength of the comparative claim, but they do not make HALE's match to CDC data true by construction or reduce any reported prediction to a fitted parameter. No self-definitional equations, load-bearing self-citations of uniqueness results, or renamed known patterns appear. The derivation chain is therefore self-contained against the external benchmark.
Axiom & Free-Parameter Ledger
free parameters (6)
- LLM temperature =
0.2
- ABM-only contact deactivation fraction =
0.35
- Omicron R0 and recovery rate =
R0=9.5, γ=0.2
- Office contact degree distribution =
μ≈2.64, σ²≈1.32
- Out-of-class school average degree =
4
- Activity-specific travel distances λ =
activity-dependent
axioms (5)
- ad hoc to paper Zero-shot Llama-3.1-8B can produce demographically and spatially differentiated stay-home decisions that approximate real human responses to local incidence.
- domain assumption SIR compartmental dynamics with constant β and γ adequately describe Omicron transmission on the constructed activity networks.
- domain assumption Exponential distance kernel correctly generates contacts for non-anchored activities when exact locations are unknown.
- ad hoc to paper Grouping agents into 1,552 municipality × sex × race × age bins is a sufficient resolution for LLM behavioral inference.
- domain assumption Weekly LLM time steps (Δt_LLM = 7 days) match the relevant behavioral adaptation timescale.
invented entities (2)
-
HALE framework (hybrid ABM-LLM loop with deliberate-edge deactivation)
no independent evidence
-
1,552 LLM group agents (one per demographic-spatial bin)
no independent evidence
read the original abstract
Agent-based modeling (ABM) has the capability to model millions of individuals and their interactions, which is useful for policy making. However, ABMs have traditionally relied on static prior, which prevents the models from adapting to real-time changes. Our research provides a novel approach to addressing this information gap. Large language models (LLMs) offer new opportunities to predict human decision-making. Here, we introduce a scalable Hybrid Agent-based and Language-driven Epidemic (HALE) modeling framework that leverages LLMs to predict human decision-making in an ABM simulation. As a proof-of-concept, we use HALE to simulate COVID-19 and its effects in Salt Lake County, UT.
Figures
Reference graph
Works this paper leans on
-
[1]
Federal Highway Administration (FHWA).National Household Travel Survey.https : / / nhts.ornl.gov/. Accessed: 2025-08-21. 2025
work page 2025
-
[2]
Are all underimmunized measles clusters equally critical?
Sifat Afroj Moon, Achla Marathe, and Anil Vullikanti. “Are all underimmunized measles clusters equally critical?” In:Royal Society Open Science10.8 (2023)
work page 2023
-
[3]
Marvellous Akinlotan et al. “Travel for medical or dental care by race/ethnicity and rurality in the US: findings from the 2001, 2009 and 2017 National Household Travel Surveys”. In: Preventive Medicine Reports35 (2023), p. 102297. 14
work page 2001
-
[4]
Rachel Bachman.How Close Do You Need to Be to Your Gym?https://www.wsj.com/ articles/how-close-do-you-need-to-be-to-your-gym-1490111186. Accessed: 2024-01-
work page 2024
-
[5]
Decisions & distance: The rela- tionship between child care access and child care travel
Evelyn Blumenberg, Madeline Wander, and Zhiyuan Yao. “Decisions & distance: The rela- tionship between child care access and child care travel”. In:Journal of transport geography 114 (2024), p. 103756
work page 2024
-
[6]
Isaac Brodsky.H3: Uber’s Hexagonal Hierarchical Spatial Index.https://www.uber.com/ blog/h3/. Accessed: 2025-08-21. 2025
work page 2025
-
[7]
AgentTorch: Agent-based Modeling with Automatic Differentiation
Ayush Chopra et al. “AgentTorch: Agent-based Modeling with Automatic Differentiation”. In:Second Agent Learning in Open-Endedness Workshop
-
[8]
Large language models are zero-shot recognizers for activities of daily living
Gabriele Civitarese et al. “Large language models are zero-shot recognizers for activities of daily living”. In:ACM Transactions on Intelligent Systems and Technology16.4 (2025), pp. 1– 32
work page 2025
-
[9]
The United States COVID-19 Forecast Hub dataset
Estee Y Cramer et al. “The United States COVID-19 Forecast Hub dataset”. In:medRxiv (2021).doi:10.1101/2021.11.04.21265886.url:https://www.medrxiv.org/content/ 10.1101/2021.11.04.21265886v1
work page doi:10.1101/2021.11.04.21265886.url:https://www.medrxiv.org/content/ 2021
-
[10]
Mixing patterns between age groups in social networks
Sara Y Del Valle et al. “Mixing patterns between age groups in social networks”. In:Social Networks29.4 (2007), pp. 539–554
work page 2007
-
[11]
Abhimanyu Dubey et al. “The llama 3 herd of models”. In:arXiv e-prints(2024), arXiv– 2407
work page 2024
-
[12]
gov/surveys/ntps/estable/table/ntps/ntps2021_sflt07_t1s
National Center for Education Statistics.Average public school class size: Average class size in public K–12 schools, by school level, class type, and state: 2020–21.https://nces.ed. gov/surveys/ntps/estable/table/ntps/ntps2021_sflt07_t1s. Accessed: 2025-06-01. 2025
work page 2020
-
[13]
Agent-based modelling meets generative ai in social network simu- lations
Antonino Ferraro et al. “Agent-based modelling meets generative ai in social network simu- lations”. In:International Conference on Advances in Social Networks Analysis and Mining. Springer. 2024, pp. 155–170
work page 2024
-
[14]
JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models
Saibo Geng et al. “Jsonschemabench: A rigorous benchmark of structured outputs for lan- guage models”. In:arXiv preprint arXiv:2501.10868(2025)
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[15]
Navid Ghaffarzadegan et al. “Generative agent-based modeling: Unveiling social system dy- namics through coupling mechanistic models with generative artificial intelligence”. In:arXiv preprint arXiv:2309.11456(2023)
work page internal anchor Pith review Pith/arXiv arXiv 2023
-
[16]
Kin On Kwok et al. “Utilizing large language models in infectious disease transmission mod- elling for public health preparedness”. In:Computational and Structural Biotechnology Jour- nal23 (2024), pp. 3254–3257
work page 2024
-
[17]
Efficient memory management for large language model serving with pagedattention
Woosuk Kwon et al. “Efficient memory management for large language model serving with pagedattention”. In:Proceedings of the 29th symposium on operating systems principles. 2023, pp. 611–626
work page 2023
-
[18]
Jia’an Liu et al. “Agentic AI for sustainable development: Leveraging large language model- enhanced agent-based modeling for complex policy strategies”. In:Emerging Media3.3 (2025), pp. 401–413
work page 2025
-
[19]
Jodi L Liu, Bing Han, and Deborah A Cohen. “Beyond neighborhood food environments: distance traveled to food establishments in 5 US cities, 2009–2011”. In:Preventing chronic disease12 (2015), E126. 15
work page 2009
-
[20]
Ying Liu and Joacim Rockl¨ ov. “The effective reproductive number of the Omicron variant of SARS-CoV-2 is several times relative to Delta”. In:Journal of travel medicine29.3 (2022), taac037
work page 2022
-
[21]
Sifat A Moon and Caterina M Scoglio. “Contact tracing evaluation for COVID-19 transmis- sion in the different movement levels of a rural college town in the USA”. In:Scientific reports 11.1 (2021), p. 4891
work page 2021
-
[22]
Sifat A Moon et al. “A spatio-temporal individual-based network framework for West Nile virus in the USA: Spreading pattern of West Nile virus”. In:PLoS computational biology15.3 (2019), e1006875
work page 2019
-
[23]
Sifat Afroj Moon.HALE data and simulation. Apr. 2026.doi:10.5281/zenodo.19829913. url:https://doi.org/10.5281/zenodo.19829913
-
[24]
Long-distance dispersal and accelerating waves of disease: em- pirical relationships
Christopher C Mundt et al. “Long-distance dispersal and accelerating waves of disease: em- pirical relationships”. In:The American Naturalist173.4 (2009), pp. 456–466
work page 2009
-
[25]
A population data-driven workflow for COVID-19 modeling and learn- ing
Jonathan Ozik et al. “A population data-driven workflow for COVID-19 modeling and learn- ing”. In:The International Journal of High Performance Computing Applications35.5 (2021), pp. 483–499
work page 2021
-
[26]
Large language models can infer psychological disposi- tions of social media users
Heinrich Peters and Sandra C Matz. “Large language models can infer psychological disposi- tions of social media users”. In:PNAS nexus3.6 (2024), pgae231
work page 2024
-
[27]
Performance and biases of large language models in public opinion simulation
Yao Qu and Jue Wang. “Performance and biases of large language models in public opinion simulation”. In:Humanities and Social Sciences Communications11.1 (2024), pp. 1–13
work page 2024
-
[28]
Data Assimilation for Robust UQ Within Agent-Based Simulation on HPC Systems
Adam Spannaus et al. “Data Assimilation for Robust UQ Within Agent-Based Simulation on HPC Systems”. In:Proceedings of the Platform for Advanced Scientific Computing Con- ference. 2025, pp. 1–11
work page 2025
-
[29]
Rahul Subramanian, Qixin He, and Mercedes Pascual. “Quantifying asymptomatic infection and transmission of COVID-19 in New York City using observed cases, serology, and testing capacity”. In:Proceedings of the National Academy of Sciences118.9 (2021), e2019716118
work page 2021
-
[30]
Joe Tuccillo.Downscaling Synthetic Populations to Realistic Residential Locations. Tech. rep. Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States), 2024
work page 2024
-
[31]
Joseph Tuccillo et al. “UrbanPop: A spatial microsimulation framework for exploring demo- graphic influences on human dynamics”. In:Applied Geography151 (2023), p. 102844
work page 2023
-
[32]
UKHSA. “COVID-19 Omicron variant infectious period and transmission from people with asymptomatic compared with symptomatic infection: A rapid review”. In:GOV-14430.{UK Health Security Agency}(2023)
work page 2023
-
[33]
Alexander Sasha Vezhnevets et al. “Generative agent-based modeling with actions grounded in physical, social, or digital space using Concordia”. In:arXiv preprint arXiv:2312.03664 (2023)
work page internal anchor Pith review Pith/arXiv arXiv 2023
-
[34]
Asymptomatic SARS-CoV-2 infection by age: a global systematic review and meta-analysis
Bing Wang et al. “Asymptomatic SARS-CoV-2 infection by age: a global systematic review and meta-analysis”. In:The Pediatric Infectious Disease Journal42.3 (2023), pp. 232–239
work page 2023
-
[35]
Efficient Guided Generation for Large Language Models
Brandon T Willard and R´ emi Louf. “Efficient Guided Generation for Large Language Mod- els”. In:arXiv preprint arXiv:2307.09702(2023)
work page internal anchor Pith review Pith/arXiv arXiv 2023
-
[36]
Epidemic Modeling with Generative Agents
Ross Williams et al. “Epidemic modeling with generative agents”. In:arXiv preprint arXiv:2307.04986 (2023). 16
work page internal anchor Pith review Pith/arXiv arXiv 2023
-
[37]
Evaluating the ability of large Language models to predict human social decisions
Feng Xiao and XT XiaoTian Wang. “Evaluating the ability of large Language models to predict human social decisions”. In:Scientific Reports15.1 (2025), p. 32290
work page 2025
-
[38]
A baseline structure inventory with critical attribution for the US and its territories
Hsiuhan Lexie Yang et al. “A baseline structure inventory with critical attribution for the US and its territories”. In:Scientific Data11.1 (2024), p. 502
work page 2024
-
[39]
LLM-AIDSim: LLM-Enhanced Agent-Based Influence Diffusion Simulation in Social Networks
Lan Zhang et al. “LLM-AIDSim: LLM-Enhanced Agent-Based Influence Diffusion Simulation in Social Networks”. In:Systems13.1 (2025), p. 29
work page 2025
-
[40]
Exploring the role of large language models in the scientific method: from hypothesis to discovery
Yanbo Zhang et al. “Exploring the role of large language models in the scientific method: from hypothesis to discovery”. In:npj Artificial Intelligence1.1 (2025), p. 14. 17
work page 2025
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.