Pith. sign in

REVIEW 4 major objections 5 minor 3 cited by

LiPUP-MA: A Residential Experience-centric Multi-Agent Framework for Living-in-the-loop Participatory Urban Planning

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A closed loop of planning, simulated living, and judging raises residents' reported experience score to 69.03% across three cycles.

desk verdict Genuinely new closed-loop planning-living-judging framework, but the main evidence is self-referential and the abstract overstates the static-metric results. read the letter →

arxiv 2412.20505 v2 pith:FT3O3R6Z submitted 2024-12-29 cs.AI cs.CLcs.LG

Hang Ni , Yuzhi Wang , Yizhi Song , Hao Liu This is my paper · ORCID
classification cs.AIcs.CLcs.LG
keywords participatoryurbanplanninglargelanguagemodelsmulti-agentsimulationclosed-loopqualityofliferegenerationagent-basedmodelingHuilongguan
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper seeks to establish that urban plans get better if they are revised in a closed loop: LLM agents draft a land-use plan, simulated residents 'live' under it and report how it feels, and the plan is revised using that feedback, over and over. The authors claim this living-in-the-loop paradigm beats random, deep-reinforcement-learning, and single-iteration multi-agent baselines on residents' living experience, and that the experience score keeps rising with each cycle, reaching 69.03% after three iterations in a Huilongguan, Beijing case study. A reader should care because urban regeneration decisions are currently made largely as one-shot interventions; if this holds, planners could pre-test how a neighborhood feels before building it. The authors also observe that the experiential gains come with declining static Accessibility and Ecology scores, suggesting a trade-off between subjective well-being and conventional spatial-efficiency metrics.

What carries the argument

Central machinery is the closed loop itself, instantiated as three interacting agent groups: a planner who revises land-use assignments; resident agents whose dynamic memory pools accumulate mobility trajectories, social-media posts, and reflections; and a judger who combines quantitative statistics with qualitative interviews into a quality-of-life score and suggestions for the next cycle. In the LiPUP-MA formulation, scattered resident feedback is organized into a Plan-centric Graph-based Experience Bank that grounds experiences in concrete urban contexts, and the planner is a Spatially-constrained Skill-augmented Planner that turns subjective feedback into spatially coherent land-use changes. The loop carries the argument because the same resident experiences that feed the revision are also what measures improvement.

What would settle it

Run the full loop on Huilongguan but evaluate each revised plan with a held-out panel of resident agents who never participated in discussions or living; if the held-out panel's experience scores do not rise with iterations and do not beat the one-shot baseline, the reported 65.03-to-69.03 improvement is an artifact of the closed self-evaluation loop rather than genuine plan improvement.

Watch

Extended reading notes

Core claim

The central claim is that participatory urban planning should be a cyclical process rather than a one-time event, and that LLM-based agents can run that cycle. Given a region partitioned into areas with land-use assignments, the framework iterates: the planner agent revises the plan after expert knowledge, prior suggestions, and a resident discussion; resident agents then simulate a day of mobility and social-media behavior under the plan, storing experiences in memory; the judger computes quantitative metrics (accessibility, ecology) and interviews residents for a qualitative Experience score, then feeds suggestions back into planning. On the Huilongguan dataset the Experience score rises from 65.03 to 69.03 across three iterations and generally exceeds the baselines, while Accessibility and Ecology decline, which the authors interpret as a sacrifice of static spatial efficiency for subjective well-being.

Load-bearing premise

The whole result depends on trusting the simulated residents' interview answers as a real measure of a plan's quality, even though those same simulated residents took part in changing the plan they are judging.

Editorial extensions

If this is right

  • Running more planning–living–judging cycles keeps raising the residents' average experience score (65.03, then 66.6, then 69.03) in the reported runs.
  • The framework reports higher Experience scores than random, deep-reinforcement-learning, and single-iteration multi-agent baselines in the Huilongguan case.
  • Improvements in experiential quality can come at the cost of lower Accessibility and Ecology scores, so evaluating plans on static metrics alone can miss what residents actually feel.
  • Because the loop needs no human surveys or field observation, the same cycle can be re-run at scale to explore alternative plans before any physical change.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the closed-loop gains are real rather than self-confirmation, the same cycle could be applied to roads, infrastructure, or zoning by adding new resident voices and watching which plans survive many iterations.
  • A direct test would be evaluating the final plan with a fresh panel of resident agents who never joined discussions; if the score still beats baselines, the improvement is more than the loop talking to itself.
  • The decline in Accessibility and Ecology may be a quirk of the current balance rather than a rule; adding an explicit constraint to preserve quantitative targets could yield plans that win on both axes.
  • The abstract's claim of consistent outperformance on static metrics is stronger than Table 1, where the deep-reinforcement-learning method leads Accessibility and the single-iteration version leads Ecology; the concrete advantage shown is on the Experience metric.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes an LLM-based multi-agent framework for cyclical urban planning. The framework alternates three phases: Planning, where planner and resident agents generate and revise a land-use plan; Living, where resident agents simulate mobility and social behavior in the current plan; and Judging, where a judge agent evaluates the plan via quantitative metrics and qualitative interviews and produces suggestions for the next cycle. The authors test the framework on the Huilongguan community in Beijing over three iterations and report that iterative cycles improve a resident Experience score, with the third iteration reaching 69.03%. The paper claims the framework 'consistently outperforms baselines on both conventional static planning metrics and living-based metrics,' although the full text later qualifies this as 'generally outperforms baselines in Experience metric.'

Significance. If the closed-loop planning-living-judging paradigm were shown to produce reliable, independently verified improvements in plan quality, it would be a meaningful contribution to LLM-based participatory urban planning. The paper addresses a real problem—static one-shot planning versus the inherently cyclical nature of urban regeneration—and the use of LLM agents for simulated living is a plausible direction. However, the significance is currently not established: the central evidence of improvement comes from a metric computed by interviewing the same agents whose feedback drove the plan revisions, no external validation is provided, and the abstract overstates the quantitative results. The paper also ships no code or detailed prompts, so the experiments are not reproducible from the manuscript alone.

major comments (4)
  1. [Title and Abstract vs. Full Text] The submitted title and abstract describe 'LiPUP-MA' with a 'Plan-centric Graph-based Experience Bank' and a 'Spatially-constrained Skill-augmented Planner agent,' but the full-text abstract, Methodology, and Experiments describe only the Planning/Living/Judging (CUP) framework and contain none of these components. The evaluation in Table 1 tests the CUP framework, not the framework claimed in the title and abstract. This mismatch makes the stated contribution unverifiable from the submitted manuscript.
  2. [Abstract, Table 1] The abstract claims LiPUP-MA 'consistently outperforms baselines on both conventional static planning metrics and living-based metrics,' but Table 1 contradicts this. Ours-3rd scores 64.17 on Accessibility versus 66.25 for DRL, and 53.33 on Ecology versus 56.67 for DRL and 73.33 for MA-LLM (Ours-1st). The full text itself acknowledges that 'the DRL-based method achieves the best performance in Accessibility' and that 'the outcome decline of our framework during cycling in Accessibility and Ecology metrics.' The only metric with consistent improvement is Experience, so the abstract's claim of consistent outperformance is not supported.
  3. [Methodology, Eqs. (2)-(3) and (6)-(9); Experiment, Table 1] The central evidence for iterative improvement is the Experience score, but this score is circular. The resident agents produce discussion D_k (Eq. 2) that the planner uses to revise plan P_k (Eq. 3); the same residents are then interviewed in Eq. (7) to compute Q_k,2, which is averaged into the overall score R_k (Eq. 8). There is no held-out resident cohort, no independent evaluator, and no external ground truth for resident well-being. The monotone rise from 65.03% to 69.03% across three iterations is therefore compatible with the agents repeating their earlier preferences rather than with an independent measure of plan quality. Without variance estimates or a control condition that breaks this dependency, the paper's main claim is not supported.
  4. [Experiment, Table 1 and 'Results'] The paper reports a single run with temperature 0, 30 resident agents, and 3 iterations, and gives no error bars, multiple seeds, or significance tests. The observed increases of about 1.6 and 2.4 percentage points in Experience could be within the noise of the LLM simulation. The conclusion that 'the effectiveness gradually increases with the number of cycles' is based on three unverified points. The authors also state, 'we speculate that the boost of subjective human well-being ought to undertake the sacrifice of the valid spatial layout of land uses,' which is speculation, not a demonstrated trade-off, and does not rescue the circular evaluation.
minor comments (5)
  1. [Experiment, first paragraph] The sentence 'The four evaluation metrics originate from the judging procedure, including Accessibility, Ecology, and Experience' lists only three metrics; either a fourth metric is missing or the count should be 'three.'
  2. [Methodology, Eq. (7)] The 'interview({Ri})' operation is underspecified: no questionnaire, question list, scoring rubric, or aggregation scheme is provided, so a reader cannot reproduce the Experience score.
  3. [Case study, Figure 2] The case study of resident R19 is purely qualitative and reports only a single agent's trajectory; it does not provide quantitative evidence that resident agents' behaviors or feedback are reliable or representative.
  4. [Overall] The manuscript provides no code, prompts, or detailed simulation configuration, making the experiments difficult to reproduce; consider a supplementary appendix with these materials.
  5. [Terminology] The full text uses 'cyclical urban planning' and 'CUP' throughout, while the submission title uses 'LiPUP-MA' and 'Living-in-the-loop Participatory Urban Planning'; the terminology should be unified to avoid confusion about what is being proposed and evaluated.

Circularity Check

2 steps flagged · score 8.0 of 10

The iterative-improvement claim reduces to self-confirmation: the Experience metric is generated by the same resident agents whose discussion and suggestions drive plan revision.

  1. fitted input called prediction [Methodology, Planning (Eqs. 2–3), Judging (Eq. 7), and Results (Table 1)]
    "Then, the resident agents are nudged into a discussion to balance the diverging interests, motivated by their profiles and previous living experiences (if k ≥ 2). Finally, the discussion result Dk is summarized and leveraged by the planner to polish the plan further. ... Additionally, we implement qualitative interviews for the residents through a questionnaire, to gather the living experience scores (percentage scale)."

    In Eq. (2), the same resident agents {R_i} produce discussion D_k that Eq. (3) uses to revise the plan; in Eq. (7), the same set {R_i} is interviewed to produce the Experience score Q_{k,2}. The reported monotone rise from 65.03% to 69.03% is therefore the plan being evaluated by the very agents whose stated preferences were used to alter the plan. No held-out cohort, external survey, or independent metric certifies the improvement; the 'prediction' that cycles improve living experience is structurally dependent on the input it claims to validate.

  2. self definitional [Methodology, Judging (Eq. 9) and Planning (Eq. 3)]
    "Sk = J.suggest(Qk,1, Qk,2). ... Pk = P.plan(K, ˜Pk, Sk−1, Dk)."

    The judging stage's suggestions S_k are computed from the interview-derived qualitative score Q_{k,2} (Eq. 9), and those suggestions become S_{k-1} in the next planning revision (Eq. 3). Thus the objective driving the plan change is the same resident-report measure later used to declare improvement. The improvement claim is closed by construction: Q_{k,2} both guides the modification and certifies it as beneficial.

full rationale

The central evidence for the paper's headline claim is the Experience metric, which is defined by Eq. (7) as an interview of the resident agents. Those same resident agents produce the discussion D_k used in Eqs. (2)–(3) to revise the plan, and the judger's suggestions S_k come directly from the interview scores via Eq. (9). The paper's own result section states that 'our framework generally outperforms baselines in Experience metric, and the effectiveness gradually increases with the number of cycles, which arrives at 69.03% after 3 iterations.' This is the loop rating its own preference-driven changes, not an independent measurement of plan quality. Setting temperature to 0 removes sampling noise but not the systematic dependence of the judge on the planner's inputs. The paper also reports no variance across runs and no fresh resident cohort. The abstract's additional claim that LiPUP-MA 'consistently outperforms baselines on both conventional static planning metrics and living-based metrics' is contradicted by Table 1, where DRL beats Ours-3rd on Accessibility (66.25 vs 64.17) and Ecology (56.67 vs 53.33), and MA-LLM beats Ours-3rd on Ecology (73.33 vs 53.33). The paper itself acknowledges that 'the outcome decline of our framework during cycling in Accessibility and Ecology metrics' suggests a trade-off, and it notes that 'we only use part of QOL metrics in this ongoing work.' Thus the only consistent advantage is the self-referential Experience metric, and the iterative-improvement claim reduces to self-confirmation rather than to independent validation.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The framework rests on several domain assumptions about LLM agents faithfully representing residents and about self-reported experience being a valid outcome. The equal weighting in Eq. 8 is a design choice, and the experimental hyperparameters, such as 30 agents, 3 iterations, and one-day simulation, are unverified choices. No external benchmark or human validation is used, so the ledger is heavy with assumptions rather than free numerical fits.

free parameters (5)
  • number_of_resident_agents = 30
    Chosen for the experiment; experience scores and discussion dynamics could change with population size, and no sensitivity analysis is provided.
  • number_of_CUP_iterations = 3
    The claim of increasing effectiveness rests on a trend across just three cycles, with no convergence or longer-horizon evidence.
  • simulation_duration_and_granularity = 1 day at 1-minute steps
    Hand-selected daily horizon; longer or finer simulations may change mobility, social posts, and judged experience.
  • LLM_temperature = 0
    Set to zero for stability, but this choice affects diversity of resident behavior and discussion, and no ablations are reported.
  • qualitative_quantitative_weight = 0.5/0.5 average in Eq. 8
    The final plan score is an equal average of automatic statistics and resident interviews; different weights would change reported plan quality.
assumptions (5)
  • domain assumption LLM-simulated residents are faithful proxies for real residents.
    The Living phase equates simulated mobility, social posts, and reflections with actual residential experience; no validation against real residents is provided.
  • domain assumption Resident interview scores are an unbiased measure of plan quality.
    Judging Eq. 7 uses interview responses as a quality signal, but the same agents produced the suggestions used to revise the plan.
  • domain assumption The LLM judger produces valid suggestions and evaluations.
    Eq. 9 relies on J.suggest without checking whether LLM suggestions are correct, feasible, or actionable.
  • ad hoc to paper Three iterations are sufficient to demonstrate a trend.
    Only 3 cycles are run; the increasing Experience values may be noise or short-horizon effects.
  • domain assumption Quantitative metrics and qualitative interviews capture plan quality.
    The paper admits only part of QOL metrics are used and no benchmark ground truth is defined.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LiPUP-MA: A Residential Experience-centric Multi-Agent Framework for Living-in-the-loop Participatory Urban Planning." pith.science (2026). https://pith.science/paper/FT3O3R6Z

@misc{pith2026241220505,
  author       = {Pith},
  title        = {Pith review of: LiPUP-MA: A Residential Experience-centric Multi-Agent Framework for Living-in-the-loop Participatory Urban Planning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FT3O3R6Z}},
  note         = {Machine review of arXiv:2412.20505}
}
read the original abstract

Participatory Urban Planning (PUP) is increasingly supported by LLM-based agents, yet existing methods largely rely on static preference elicitation and one-shot stakeholder discussions, overlooking the cyclical nature of real-world planning, where residential life, experience collection, and plan adjustment continually interact. We propose Living-in-the-loop Participatory Urban Planning (LiPUP), a closed-loop paradigm that alternates between simulated residential living and experience-driven plan revision, while posing two key challenges: grounding scattered living experience in concrete urban contexts and translating subjective feedback into spatially coherent planning actions. To instantiate LiPUP, we introduce LiPUP-MA, an LLM-based multi-agent framework that constructs a Plan-centric Graph-based Experience Bank to organize urban-grounded residential feedback from living simulation and equips a Spatially-constrained Skill-augmented Planner agent to revise plans by harmonizing experiential, visual, and geospatial evidence. Experiments show that LiPUP-MA consistently outperforms baselines on both conventional static planning metrics and living-based metrics, while iterative LiPUP cycles further improve plan quality.

Figures

Figures reproduced from arXiv: 2412.20505 by the authors.

Figure 1
Figure 1. Framework overview. and improving scalability (Zheng et al. 2023; Wang et al. 2023; Qian et al. 2023). The rise of large language models (LLMs) has further advanced urban planning capabilities, enabling complex multi-agent collaboration for consensus￾building (Zhou et al. 2024). However, existing approaches focus primarily on one-time planning efforts, lacking the it￾erative, closed-loop nature needed for adaptive u… view at source ↗
Figure 2
Figure 2. Case study of R19. Case study. To assess whether our resident agents gen￾erate reliable behaviors and feedback and contribute to plan renovations, we study the participation of R19 during the cy￾cling of our framework ( [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Large Language Model Powered Intelligent Urban Agents: Concepts, Capabilities, and Applications

    cs.MA 2025-07 conditional novelty 4.0 of 10

    The paper defines urban LLM agents, surveys their sensing, memory, reasoning, execution, and learning workflows, and organizes their applications across planning, transportation, environment, safety, and society.

  2. Earth Science Foundation Models: From Perception to Reasoning and Discovery

    astro-ph.IM 2026-05 unverdicted novelty 3.0 of 10

    The paper delivers a unified review and roadmap of Earth science foundation models, structured by capability depth from perception to agentic reasoning and by application breadth across atmosphere, hydrosphere, lithos...

  3. A Survey on Agent Workflow -- Status and Future

    cs.AI 2025-08 conditional novelty 3.0 of 10

    A review that classifies 24 agent workflow systems along functional and architectural axes and argues for standardization, optimization, and security work.

Reference graph

Works this paper leans on

21 extracted references · 13 canonical work pages · cited by 3 Pith papers

  1. [1]

    Arnstein, S. R. 1969. A ladder of citizen participation. Journal of the American Institute of planners, 35(4): 216--224

    work page 1969

  2. [2]

    E.; Haub, O.; Mabikke, S.; Antonio, D.; and Espinoza, J

    Chigbu, U. E.; Haub, O.; Mabikke, S.; Antonio, D.; and Espinoza, J. 2016. Tenure responsive land use planning: a guide for country level implementation. UN-Habitat: Nairobi, Kenya

    work page 2016

  3. [3]

    Gao, C.; Lan, X.; Li, N.; Yuan, Y.; Ding, J.; Zhou, Z.; Xu, F.; and Li, Y. 2024. Large language models empowered agent-based modeling and simulation: A survey and perspectives. Humanities and Social Sciences Communications, 11(1): 1--24

    work page 2024

  4. [4]

    Gonz \'a lez-M \'e ndez, M.; Olaya, C.; Fasolino, I.; Grimaldi, M.; and Obreg \'o n, N. 2021. Agent-based modeling for urban development planning based on human needs. Conceptual basis and model formulation. Land Use Policy, 101: 105110

    work page 2021

  5. [5]

    A.; and Nourian, F

    Hosseinali, F.; Alesheikh, A. A.; and Nourian, F. 2013. Agent-based modeling of urban land-use development, case study: Simulating future scenarios of Qazvin city. Cities, 31: 105--113

    work page 2013

  6. [6]

    Lane, M. B. 2005. Public participation in planning: an intellectual history. Australian geographer, 36(3): 283--299

    work page 2005

  7. [7]

    C.-m.; and Lang, W

    Li, X.; Zhang, F.; Hui, E. C.-m.; and Lang, W. 2020. Collaborative workshop and community participation: A new approach to urban regeneration in China. Cities, 102: 102743

    work page 2020

  8. [8]

    Oliveira, V.; and Pinho, P. 2010. Evaluation in urban planning: Advances and prospects. Journal of planning literature, 24(4): 343--361

    work page 2010

Show all 21 references
  1. [9]

    S.; O'Brien, J.; Cai, C

    Park, J. S.; O'Brien, J.; Cai, C. J.; Morris, M. R.; Liang, P.; and Bernstein, M. S. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, 1--22

    2023

  2. [10]

    S.; Popowski, L.; Cai, C.; Morris, M

    Park, J. S.; Popowski, L.; Cai, C.; Morris, M. R.; Liang, P.; and Bernstein, M. S. 2022. Social simulacra: Creating populated prototypes for social computing systems. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, 1--18

    2022

  3. [11]

    Qian, K.; Mao, L.; Liang, X.; Ding, Y.; Gao, J.; Wei, X.; Guo, Z.; and Li, J. 2023. AI Agent as Urban Planner: Steering Stakeholder Dynamics in Urban Planning via Consensus-based Multi-Agent Reinforcement Learning. arXiv preprint arXiv:2310.16772

    2023 arXiv

  4. [12]

    Ramezani, S.; and Nooraie, H. 2023. A comparative study of participatory land-use planning and the theory of the right to the city. GeoJournal, 88(5): 5173--5203

    2023

  5. [13]

    D.; and Martins, I

    Santos, L. D.; and Martins, I. 2007. Monitoring urban quality of life: The Porto experience. Social Indicators Research, 80: 411--425

    2007

  6. [14]

    Taylor, N. 1998. Urban planning theory since 1945

    1998

  7. [15]

    Wang, D.; Fu, Y.; Liu, K.; Chen, F.; Wang, P.; and Lu, C.-T. 2023. Automated urban planning for reimagining city configuration via adversarial learning: quantification, generation, and evaluation. ACM Transactions on Spatial Algorithms and Systems, 9(1): 1--24

    2023

  8. [16]

    Wang, J.; Jiang, R.; Yang, C.; Wu, Z.; Onizuka, M.; Shibasaki, R.; Koshizuka, N.; and Xiao, C. 2024. Large language models as urban residents: An llm agent framework for personal mobility generation. arXiv preprint arXiv:2402.14744

    2024 arXiv

  9. [17]

    Yan, Y.; Zeng, Q.; Zheng, Z.; Yuan, J.; Feng, J.; Zhang, J.; Xu, F.; and Li, Y. 2024. OpenCity: A Scalable Platform to Simulate Urban Activities with Massive LLM Agents. arXiv preprint arXiv:2410.21286

    2024 arXiv

  10. [18]

    Zheng, Y.; Lin, Y.; Zhao, L.; Wu, T.; Jin, D.; and Li, Y. 2023. Spatial planning of urban communities via deep reinforcement learning. Nature Computational Science, 3(9): 748--762

    2023

  11. [19]

    Zhou, Z.; Lin, Y.; Jin, D.; and Li, Y. 2024. Large language model for participatory urban planning. arXiv preprint arXiv:2402.17161

    2024 arXiv

  12. [20]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  13. [21]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.