Pith. sign in

REVIEW 3 major objections 4 minor 40 references

This paper argues that collecting human preferences over the intermediate decision steps of generative agents—rather than only over final trajectories—provides an effective training signal, improving both local decision quality and long-hor

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 01:56 UTC pith:6IMNXLOH

load-bearing objection A genuinely useful dataset and interface, but the paper's headline claim that human step-level preferences drive the gains is not yet isolated from GPT-4o distillation. the 3 major comments →

arxiv 2607.14485 v1 pith:6IMNXLOH submitted 2026-07-16 cs.AI

Step-Level Preference Learning for Generative Agents in Social Simulations

Wenchang Gao , Pingyue Sheng , Lanlan Qiu , Yunfei Ma , Jian Zhao , Baicheng Chen , Kangda Wang , Yuyang Tian
show 2 more authors
Shunqiang Mao Tianxing He
This is my paper
classification cs.AI
keywords social simulationgenerative agentsstep-level preference learninghuman-in-the-loop annotationdirect preference optimizationlong-horizon behaviormemory importanceLLM-as-a-judge evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces an interactive simulation interface through which human annotators supervise the moment-by-moment decisions of generative agents: planning, memory retrieval, reflection, action, and dialogue. Using this interface, the authors collect 57,239 step-level preference pairs across 30 social events, then fine-tune open-weight language models with supervised learning and direct preference optimization on those pairs. On 10 held-out events with different scripts and goals, the trained models consistently improve all five whole-trajectory metrics over their base versions. The trained agents also spend time across behavioral categories much closer to a human reference, with the divergence dropping from 0.610 to 0.084. The paper's central claim is that fine-grained human preferences over intermediate decisions are an effective training signal for producing more socially competent long-horizon agents.

Core claim

Step-level human supervision is an effective training signal for generative agents in social simulations. For each triggered decision module, the model is shown the same partial observations, goals, and retrieved memories the agent sees, and generates three candidate outputs; a human annotator chooses the most competent one or writes a custom alternative. Training on these accepted outputs (and contrasting them against rejected ones) markedly improves open-weight models across location adherence, temporal adherence, role fulfillment, requirement consistency, and interaction quality, with most of the gain coming from supervised fine-tuning and smaller, dimension-dependent gains from direct pr

What carries the argument

The central mechanism is the step-level preference tuple (x, k, y+, y−), where x is the agent's decision context—partial observations, explicit goal, retrieved memories, and local state—k is the triggered module, y+ is the human-preferred output, and y− is a rejected alternative. The interaction interface exposes these decision contexts to annotators and records their choices across six modules. A module-conditioned backbone language model is then trained first by supervised fine-tuning on accepted outputs and second by direct preference optimization on the preference pairs, aligning local decisions with human judgments of decision competence.

Load-bearing premise

The central claim collapses if the LLM judges used to score trajectories reward fluent or stylistically polished outputs rather than genuine behavioral fidelity, since every reported improvement is measured by those judges.

What would settle it

Run a blind human evaluation of full trajectories from base versus trained agents on held-out events and check whether human raters consistently prefer the trained agents on the same five dimensions; if they do not, or if objective event-completion markers such as guests actually attending the event and information propagating correctly show no improvement, the claim that step-level supervision improves long-horizon behavior fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If step-level supervision is effective, future agent-alignment pipelines can shift annotation effort from whole trajectories to the internal decisions that generate behavior, yielding more actionable signal per annotation.
  • Because most gains come from supervised fine-tuning on accepted outputs, fixing poor local decisions may matter more than adding general reasoning capability when building socially competent agents.
  • Memory-importance recalibration alone is insufficient; improving long-horizon social behavior requires coordinated improvement across multiple decision modules.
  • Step-level preference learning moves agents' long-horizon time allocation toward human behavioral distributions, suggesting that local supervision does not induce myopic or collapsing behavior over multi-day simulations.
  • Combining step-level preferences with trajectory-level supervision is a promising direction for jointly optimizing internal decisions and global social outcomes.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the dataset's low human–LLM agreement on action and importance-score modules (around 30% top-1 match, near random) suggests that current LLMs systematically misjudge feasibility and information grounding; this points to a concrete failure mode that step-level preference data could expose in other agent architectures.
  • Editorial extension: an implicit but testable consequence is that step-level preference learning will transfer across event structures that share the same module decomposition; the paper only tests scenario-level transfer within one architecture, so cross-architecture transfer is an open question.
  • Editorial extension: because annotation cost is concentrated in a small number of annotators and single-label per step, the method's value depends on whether decision-competence judgments are more reproducible across annotators than subjective taste, which the paper does not measure.
  • Editorial extension: one could design a cheaper data-collection pipeline using model-generated critiques to mimic the step-level preference oracle, and measure how much of the observed gain survives without human annotators.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces SimPref, an interactive simulation interface for collecting step-level human preference supervision over the intermediate decisions of GA-style generative agents. The authors construct a dataset of 57,239 preference pairs spanning six agent modules across 30 social events, then train open-weight LLMs (Qwen2.5-7B/14B, Llama-3.1-8B) with SFT on human-accepted outputs and DPO on accepted--rejected pairs. They report consistent improvements over five whole-trajectory metrics on 10 held-out events, a shift in behavioral time allocation toward a human reference (KL 0.610 to 0.084), and qualitative case studies. The central claim is that step-level human supervision is an effective training signal for improving both local decision quality and long-horizon agent behavior.

Significance. If the central claim is established, this would be a useful contribution: it is the first step-level human preference dataset for GA-style social simulation, and the interface and training pipeline are valuable resources. The +MI ablation and the KL-to-human behavioral analysis are thoughtful attempts to isolate mechanisms and long-horizon effects. However, the attribution of the gains to human step-level supervision is currently not supported by the experimental design, and the evaluation relies entirely on LLM-as-a-judge scoring. The paper's own limitations section acknowledges the judge-bias and missing inter-annotator agreement, but those limitations are load-bearing for the headline results.

major comments (3)
  1. [§4.2, Table 1, §6.2] The central attribution is confounded by model distillation. Only 1,217 of 57,239 pairs (2.1%) are human-customized; the SFT stage trains almost entirely on GPT-4o-generated outputs that a human selected. SFT accounts for most of the Table 2 gains (e.g., Q7B-SFT +0.76 to +1.14 across metrics), yet no control is trained on the LLM's own top-1 candidate or on random GPT-4o candidates. The +MI ablation in §6.2 does not address this confound because it changes only retrieval scoring while holding training labels fixed. A control isolating human selection (e.g., SFT on GPT-4o top-1 vs. SFT on human-selected candidates) is needed to support the claim that step-level human supervision, rather than distillation of a stronger model, drives the improvements.
  2. [Table 2, §6.2] Only mean Likert scores are reported, without standard deviations, confidence intervals, or significance tests. With 10 held-out events and 3 episodes per method, the claimed consistent improvements—especially the modest, uneven DPO gains (e.g., L8B temporal adherence drops from 2.75 to 2.56; Q7B requirement consistency drops from 2.93 to 2.89)—may be within noise. Per-event score distributions, paired tests, or effect sizes are needed to substantiate the 'consistent improvement' and the DPO contribution claims.
  3. [§6.1, §7] All quantitative claims rest on GPT-5.2 and DeepSeek-v3.2 Likert judgments, and the human reference trajectories are scored by the same judges. The paper acknowledges evaluator bias in §7, but because the evaluation metrics (requirement consistency, role fulfillment, temporal adherence) closely mirror the annotation criteria, there is a nontrivial risk that the LLM judges reward stylistic artifacts of SFT/DPO rather than true behavioral fidelity. A human evaluation of a subset of trajectories, or at least a judge-agreement and robustness analysis, is necessary to ground the headline improvements.
minor comments (4)
  1. [§6.2] There are several typos/OCR artifacts: 'difficult' in the Introduction, 'sufficient' in §6.2, and 'A veraged' in §6.2. Please proofread.
  2. [Table 1, §5.2] The Top-1 Match numbers are compared against a naive 33% baseline, but for the Importance Score module the output space is smaller and chance agreement is higher. The paper notes this but does not quantify the adjusted baseline; please report a chance-corrected agreement or otherwise handle the output-space size.
  3. [§5.1, §7] The absence of inter-annotator agreement is acknowledged in §7, but given that the entire dataset rests on single-annotator judgments, a small overlap sample with agreement statistics would substantially strengthen the dataset's credibility.
  4. [§6.3] The KL divergence is computed over normalized frequencies of six behavioral categories for Q14B only. Please report the exact category definitions, the coding procedure, and the raw counts, since KL estimates over six bins can be unstable when counts are small.

Circularity Check

1 steps flagged

Annotation rubric and evaluation metrics overlap; central result still has held-out empirical content.

specific steps
  1. self definitional [§4.1 (annotation instructions) and §6.1 (evaluation metrics)]
    "Rather than subjective social taste, annotators judge decision competence, i.e., the feasibility of the output, its consistency with the current information state, its temporal and spatial appropriateness, and whether it advances the agent’s explicit goal ... The five metrics are ... Requirement Consistency (whether behavior remains consistent with event constraints, available information, and agent profile), ... and Role Fulfillment (whether behavior is consistent with the assigned persona and advances the explicit goal)."

    The y+ labels are selected by the same constructs that the evaluation metrics score: consistency with the information state is scored as Requirement Consistency; temporal/spatial appropriateness as Temporal/Location Adherence; advancing the explicit goal as Role Fulfillment. Thus Table 2 partly measures whether the model has internalized the annotation rubric rather than an independently defined fidelity property. This is only partial: evaluation is on held-out events with whole trajectories, and DPO contrasts rejected outputs, so the result is not forced by definition.

full rationale

No equation-level circularity, no fitted parameter renamed as a prediction, and no load-bearing uniqueness claim imported from self-citations. The paper makes an empirical supervised-learning claim: 57,239 step-level preferences from 30 events are used to train open-weight LLMs, which are then evaluated on 10 held-out events with disjoint scripts; base-model and +MI comparisons provide real (if imperfect) controls. The SFT-on-GPT-4o-candidates concern is a missing-control/confound issue, not a circularity. The paper itself acknowledges LLM-as-a-judge bias and the absence of inter-annotator agreement (§7), and it explicitly disclaims statistical independence for its case study (§6.4). The one circularity-adjacent point is the construct overlap between the annotation rubric and the five evaluation metrics, which are adapted from the authors’ own prior work [30]; because of that overlap, the headline improvements are partly self-referential. This raises the score slightly but does not amount to a derivation equivalent to its inputs.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central claim rests on empirical assumptions about the simulation architecture, the validity of LLM judges, and the reliability of single-annotator human labels. The free parameters are standard hyperparameters and inherited retrieval weights, none of which are fitted to the evaluation data. No new physical or conceptual entities are postulated beyond the SimPref interface itself, which is a concrete software artifact.

free parameters (4)
  • Retrieval score weights α, β, γ = not specified in paper (inherited from [23])
    The memory retrieval score score(m; q_t)=α·rel + β·recency + γ·imp (§3) is part of the GA backbone; values are not given. The +MI variant replaces the importance estimator, so the central result is not highly sensitive to these weights, but they are hand-specified in the inherited architecture.
  • DPO β = 0.05
    Chosen in Appendix B; standard DPO hyperparameter, not fitted to data.
  • LoRA rank/α/dropout = 16/16/0.05
    Chosen in Appendix B; applied to q,k,v,o projections (and q,v for Qwen-14B). Standard LoRA hyperparameters, not fitted.
  • Number of candidate outputs per trigger = 3
    Interface design choice; the naive random agreement baseline for Top-1 Match is 33%. No ablation of candidate count is reported.
axioms (4)
  • domain assumption LLM-as-a-judge Likert scores are a valid proxy for human-judged simulation quality
    All headline results in Table 2 are LLM-judged; no human evaluation of trained agents is reported (§6.1, acknowledged as a limitation in §7).
  • domain assumption GA-style modular architecture (memory, plan, reflect, act, talk) is an appropriate model of human social decision-making
    The entire training signal is defined over the module outputs of this architecture from [23] (§3).
  • domain assumption Single-annotator preference labels without overlap are reliable
    No inter-annotator agreement is measured; each decision is labeled by one of eight annotators (§5.1). The authors list this as a limitation (§7).
  • domain assumption GPT-4o-generated candidates span the plausible decision space
    Humans choose among three GPT-4o candidates or write their own; the diversity and coverage of the candidates are not characterized (§4.2).

pith-pipeline@v1.3.0-alltime-deepseek · 11782 in / 11134 out tokens · 106435 ms · 2026-08-02T01:56:04.159504+00:00 · methodology

0 comments
read the original abstract

Large language model (LLM)-based generative agents simulate human behavior through long-horizon decision-making processes that comprise intermediate steps such as planning, memory retrieval, reflection, and action selection. However, fine-grained human annotations of these intermediate steps remain scarce, and existing agents are not grounded in human preferences over such intermediate decisions. To address this gap, we introduce \method, an interactive simulation interface that enables us to collect step-level human preference supervision over agent decision trajectories, leading to a dataset of 57K fine-grained annotations. We conduct step-level preference learning on open-weight language models using supervised finetuning and direct preference optimization on this data, consistently improving simulation fidelity, coordination, and interaction quality, and inducing more socially effective agent behavior. Our results show that step-level human supervision is an effective training signal for improving both local decision quality and long-horizon agent behavior.

Figures

Figures reproduced from arXiv: 2607.14485 by Baicheng Chen, Jian Zhao, Kangda Wang, Lanlan Qiu, Pingyue Sheng, Shunqiang Mao, Tianxing He, Wenchang Gao, Yunfei Ma, Yuyang Tian.

Figure 1
Figure 1. Figure 1: Overview of our framework. (1) SimPref enables step-level human super￾vision. At each triggered decision step, the LLM backend of the agent generates three candidate outputs from the current decision context, and a human annota￾tor selects a preferred output or provides a customized alternative. This yields one accepted output and multiple rejected outputs, stored as pairwise prefer￾ence tuples in the data… view at source ↗
Figure 2
Figure 2. Figure 2: Distribution of agent time allocation across behavioral categories before [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Case study of event information propagation before and after preference [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Overview of the SimPref interface. The left panel shows controlled-agent observations, activity, and goal, while the map view shows agents and their current activities; annotators can switch agents or inspect other agents during simulation [PITH_FULL_IMAGE:figures/full_fig_p015_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Screenshots of the SimPref annotation interface B Additional Training Details LoRA (rank 16, α = 16, dropout 0.05) is applied to the self-attention projection matrices (q, k, v, o) in both SFT and SFT+DPO stages of Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct experiments. For Qwen2.5-14B-Instruct experiments, we applied LoRA with the same hyperparameters to self-attention projection matrices q, v. We use … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

40 extracted references · 2 canonical work pages

  1. [1]

    Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., et al.: Training a helpful and harmless assistant with reinforcement learning from human feedback (2022), https://arxiv.org/abs/2204.05862

    Pith/arXiv arXiv 2022

  2. [2]

    Bougie, N., Watanabe, N.: Citysim: Modeling urban behaviors and city dynamics with large-scale llm-driven agent simulation (2025), https://arxiv.org/abs/2506. 21805

    2025

  3. [3]

    Cau, E., Pansanella, V., Pedreschi, D., Rossetti, G.: Language-driven opinion dy- namics in agent-based simulations with llms (2025), https://arxiv.org/abs/2502. 19098

    2025

  4. [4]

    Chen, W., Su, Y., Zuo, J., Yang, C., Yuan, C., Chan, C.M., Yu, H., Lu, Y., Hung, Y.H., Qian, C., Qin, Y., Cong, X., Xie, R., Liu, Z., Sun, M., Zhou, J.: Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors (2023), https://arxiv.org/abs/2308.10848

    Pith/arXiv arXiv 2023

  5. [5]

    In: NeurIPS 2024 Workshop on Open-World Agents (2024), https://openreview.net/forum?id=irKHAnZjvc

    Chen, X., Zhang, S., Song, G.: Towards humanoid: Value-driven agent model- ing based on large language models. In: NeurIPS 2024 Workshop on Open-World Agents (2024), https://openreview.net/forum?id=irKHAnZjvc

    2024

  6. [6]

    arXiv preprint arXiv:2512.02556 (2025)

    DeepSeek-AI, et al.: Deepseek-v3.2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556 (2025)

    Pith/arXiv arXiv 2025

  7. [7]

    Dong, Y., Wang, Z., Sreedhar, M.N., Wu, X., Kuchaiev, O.: Steerlm: Attribute conditioned sft as an (user-steerable) alternative to rlhf (2023), https://arxiv.org/ abs/2310.05344

    Pith/arXiv arXiv 2023

  8. [8]

    Feng, Y., Lu, Y., Su, H., He, T.: Simcity: Multi-agent urban development simula- tion with rich interactions (2025), https://arxiv.org/abs/2510.01297

    Pith/arXiv arXiv 2025

  9. [9]

    Grattafiori, A., Dubey, A., Jauhri, A., et al.: The llama 3 herd of models (2024), https://arxiv.org/abs/2407.21783

    Pith/arXiv arXiv 2024

  10. [10]

    Hou, A.B., Du, H., Wang, Y., Zhang, J., Wang, Z., Liang, P.P., Khashabi, D., Gardner, L., He, T.: Can a society of generative agents simulate human behavior and inform public health policy? a case study on vaccine hesitancy (2025), https: //arxiv.org/abs/2503.09639

    Pith/arXiv arXiv 2025

  11. [11]

    Ji, J., Hong, D., Zhang, B., Chen, B., Dai, J., Zheng, B., Qiu, T., Zhou, J., Wang, K., Li, B., Han, S., Guo, Y., Yang, Y.: Pku-saferlhf: Towards multi-level safety alignment for llms with human preference (2025), https://arxiv.org/abs/ 2406.15513

    Pith/arXiv arXiv 2025

  12. [12]

    Gao et al

    Ji, J., Zhou, J., Lou, H., Chen, B., Hong, D., Wang, X., Chen, W., Wang, K., Pan, R., Li, J., Wang, M., Dai, J., Qiu, T., Xu, H., Li, D., Chen, W., Song, J., Zheng, B., Yang, Y.: Align anything: Training all-modality models to follow instructions with language feedback (2024), https://arxiv.org/abs/2412.15838 14 W. Gao et al

    Pith/arXiv arXiv 2024

  13. [13]

    Li, J., Li, J., Chen, J., Li, Y., Wang, S., Zhou, H., Ye, M., Su, Y.: Evolving agents: Interactive simulation of dynamic and diverse human personalities (2024), https://arxiv.org/abs/2404.02718

    Pith/arXiv arXiv 2024

  14. [14]

    org/abs/2310.10436

    Li, N., Gao, C., Li, M., Li, Y., Liao, Q.: Econagent: Large language model- empowered agents for simulating macroeconomic activities (2024), https://arxiv. org/abs/2310.10436

    Pith/arXiv arXiv 2024

  15. [15]

    Proceedings of the ACM on Human-Computer In- teraction 9(2), 1–27 (May 2025)

    Li, Y., Sun, L., Zhang, Y.: MetaAgents: Large language model based agents for decision-making on teaming. Proceedings of the ACM on Human-Computer In- teraction 9(2), 1–27 (May 2025). https://doi.org/10.1145/3711032, http://dx.doi. org/10.1145/3711032

    work page doi:10.1145/3711032 2025

  16. [16]

    Liu, R., Yang, R., Jia, C., Zhang, G., Zhou, D., Dai, A.M., Yang, D., Vosoughi, S.: Training socially aligned language models on simulated social interactions (2023), https://arxiv.org/abs/2305.16960

    Pith/arXiv arXiv 2023

  17. [17]

    Mou, X., Qian, C., Liu, W., Huang, X., Wei, Z.: Ecolang: Efficient and effec- tive agent communication language induction for social simulation (2025), https: //arxiv.org/abs/2505.06904

    Pith/arXiv arXiv 2025

  18. [18]

    Movva, R., Milli, S., Min, S., Pierson, E.: What’s in my human feedback? learning interpretable descriptions of preference data (2025), https://arxiv.org/abs/2510. 26202

    2025

  19. [19]

    https://openai.com/index/hello-gpt-4o/ (2024)

    OpenAI: Gpt-4o: Omni model. https://openai.com/index/hello-gpt-4o/ (2024)

    2024

  20. [20]

    https://openai.com/index/ new-embedding-models-and-api-updates/ (2024), accessed: 2026-01-18

    OpenAI: New embedding models and api updates. https://openai.com/index/ new-embedding-models-and-api-updates/ (2024), accessed: 2026-01-18

    2024

  21. [21]

    OpenAI: Update to GPT-5 system card: GPT-5.2. Tech. rep., OpenAI (Decem- ber 2025), https://cdn.openai.com/pdf/3a4153c8-c748-4b71-8e31-aecbde944f8d/ oai_5_2_system-card.pdf, accessed: 2026-01-18

    2025

  22. [22]

    org/abs/2203.02155

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C.L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., Lowe, R.: Training language models to follow instructions with human feedback (2022), https://arxiv. org/abs/2203.02155

    Pith/arXiv arXiv 2022

  23. [23]

    Park, J.S., O’Brien, J.C., Cai, C.J., Morris, M.R., Liang, P., Bernstein, M.S.: Gen- erative agents: Interactive simulacra of human behavior (2023), https://arxiv.org/ abs/2304.03442

    Pith/arXiv arXiv 2023

  24. [24]

    Park, J.S., Zou, C.Q., Shaw, A., Hill, B.M., Cai, C., Morris, M.R., Willer, R., Liang, P., Bernstein, M.S.: Generative agent simulations of 1,000 people (2024), https://arxiv.org/abs/2411.10109

    Pith/arXiv arXiv 2024

  25. [25]

    Piao, J., Yan, Y., Zhang, J., Li, N., Yan, J., Lan, X., Lu, Z., Zheng, Z., Wang, J.Y., Zhou, D., Gao, C., Xu, F., Zhang, F., Rong, K., Su, J., Li, Y.: Agentsociety: Large-scale simulation of llm-driven generative agents advances understanding of human behaviors and society (2025), https://arxiv.org/abs/2502.08691

    Pith/arXiv arXiv 2025

  26. [26]

    Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C.D., Finn, C.: Direct preference optimization: Your language model is secretly a reward model (2024), https://arxiv.org/abs/2305.18290

    Pith/arXiv arXiv 2024

  27. [27]

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y.K., Wu, Y., Guo, D.: Deepseekmath: Pushing the limits of mathematical reasoning in open language models (2024), https://arxiv.org/abs/2402.03300

    Pith/arXiv arXiv 2024

  28. [28]

    Social Science Computer Review 32(3), 279–294 (2014), https: //doi.org/10.1177/0894439313512975 Step-Level Preference Learning for Generative Agents 15

    Squazzoni, F., Jager, W., Edmonds, B.: Social simulation in the social sciences: A brief overview. Social Science Computer Review 32(3), 279–294 (2014), https: //doi.org/10.1177/0894439313512975 Step-Level Preference Learning for Generative Agents 15

    work page doi:10.1177/0894439313512975 2014

  29. [29]

    Tan, Y., Jiang, Y., Li, Y., Liu, J., Bu, X., Su, W., Yue, X., Zhu, X., Zheng, B.: Equilibrate rlhf: Towards balancing helpfulness-safety trade-off in large language models (2025), https://arxiv.org/abs/2502.11555

    Pith/arXiv arXiv 2025

  30. [30]

    Tian, Y., Mao, S., Gao, W., Qiu, L., He, T.: A visualized framework for event cooperation with generative agents (2025), https://arxiv.org/abs/2509.13011

    arXiv 2025

  31. [31]

    Tomasev, N., Franklin, M., Leibo, J.Z., Jacobs, J., Cunningham, W.A., Gabriel, I., Osindero, S.: Virtual agent economies (2025), https://arxiv.org/abs/2509.10147

    Pith/arXiv arXiv 2025

  32. [32]

    org/abs/2505.07581

    Wang, L., Gao, H., Bo, X., Chen, X., Wen, J.R.: Yulan-onesim: Towards the next generation of social simulator with large language models (2025), https://arxiv. org/abs/2505.07581

    Pith/arXiv arXiv 2025

  33. [33]

    Wang, Y., Chen, Y., Zhong, F., Ma, L., Wang, Y.: Simulating human-like daily activities with desire-driven autonomy (2025), https://arxiv.org/abs/2412.06435

    Pith/arXiv arXiv 2025

  34. [34]

    Wang, Z., Chiu, Y.Y., Chiu, Y.C.: Humanoid agents: Platform for simulating human-like generative agents (2023), https://arxiv.org/abs/2310.05418

    Pith/arXiv arXiv 2023

  35. [35]

    Wang, Z., Dong, Y., Delalleau, O., Zeng, J., Shen, G., Egert, D., Zhang, J.J., Sreedhar, M.N., Kuchaiev, O.: Helpsteer2: Open-source dataset for training top- performing reward models (2024), https://arxiv.org/abs/2406.08673

    Pith/arXiv arXiv 2024

  36. [36]

    Wang, Z., Zeng, J., Delalleau, O., Shin, H.C., Soares, F., Bukharin, A., Evans, E., Dong, Y., Kuchaiev, O.: Helpsteer3-preference: Open human-annotated preference data across diverse tasks and languages (2025), https://arxiv.org/abs/2505.11475

    arXiv 2025

  37. [37]

    Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q., Men, R., Lin, R., Li, T., Tang, T., Xia, T., Ren, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Wan, Y., Li...

    Pith/arXiv arXiv 2025

  38. [38]

    Ye, X., Bougie, N., Yamasaki, T., Watanabe, N.: Mobilecity: An efficient framework for large-scale urban behavior simulation (2025), https://arxiv.org/abs/2504.16946

    arXiv 2025

  39. [39]

    Yu, Y., Zhang, Q., Li, J., Fu, Q., Ye, D.: Affordable generative agents (2024), https://arxiv.org/abs/2402.02053

    Pith/arXiv arXiv 2024

  40. [40]

    org/abs/2406.14228 Appendix A Interface Illustration Fig

    Yuan, S., Song, K., Chen, J., Tan, X., Li, D., Yang, D.: Evoagent: Towards au- tomatic multi-agent generation via evolutionary algorithms (2025), https://arxiv. org/abs/2406.14228 Appendix A Interface Illustration Fig. 4: Overview of the SimPref interface. The left panel shows controlled-agent observations, activity, and goal, while the map view shows age...

    Pith/arXiv arXiv 2025