{"id":"2bbe0402-c400-4d7b-b06f-b7844e89482f","arxiv_id":"2506.13599","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"CAMS, an agentic framework built on CityGPT, generates urban mobility trajectories from user profiles and claims superior JSD-based performance over prior simulators.","lead":"CAMS is a three-module AI system that uses the CityGPT urban language model to simulate realistic human mobility trajectories in Beijing. The authors claim it beats existing simulators while using only user profiles, though the experiments rely on external map data despite claiming otherwise.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported spatial gains may be inflated because the reflection stage's ground-truth distribution is never shown to exclude evaluation users, and DPO may train on test trajectories.","rationale":"The reader's weakest assumption correctly identifies the reflection stage's use of the ground-truth distribution as a potential leak. I agree that this is load-bearing, and I add a second, related channel: iterated DPO may train on collected test outputs. Both channels share the same failure mode: the model is given information about the evaluation users, either as an aggregate anchor-point distribution or as individual real trajectories, before generating their trajectories. If either channel uses evaluation users, the strong claims in the abstract are unsupported. The paper's own ablations (e.g., Table 6, w/o C) show that the reflection component improves results, which makes the split question decisive: if the ground-truth distribution includes test users, that improvement is a fitting artifact. The evaluation also suffers from a very small test sample (50 generation users) and no error bars, but those are secondary to the leakage risk. I do not see a reason to reject outright, because the paper may have used a proper training-only split that is simply under-reported. The conditional verdict is therefore appropriate: the authors should disclose the exact composition of the ground-truth distribution and the DPO training set, and rerun the key comparison with a clean split.","tokens_in":17201,"tokens_out":7246,"duration_ms":65634,"concrete_test":"Re-run the generation experiment with the reflection stage's ground-truth home/work distribution computed exclusively from the 100 recovery users (or another split disjoint from the 50 generation users), and with DPO training restricted to the 300 training users only, then report Distance, Radius, and SD JSD on the 50 generation users. If the spatial JSD values increase substantially (e.g., Radius JSD exceeds 0.1) or the CMRR ranking changes, the reported superiority is partly an artifact of test-set distribution fitting.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that CAMS generates realistic trajectories from user profiles alone. The most load-bearing weakness is that the evaluation pipeline appears to admit test-set information. In Section 2.2.1, the reflection stage of Anchor Location Extractor 'does comparative analysis against ground-truth distribution and adjusts generation strategies for subsequent iterations.' The paper never states that this ground-truth distribution is computed only from training users. Appendix A.1 reports that collective-knowledge experiments use a 44,313-user subset and then select 100 users for recovery and 50 for generation; if the ground-truth home/work distribution is aggregated over that subset, it includes the 50 generation users' anchor points. Because the reflection loop adjusts individual generated locations to reduce distributional divergence, the reported Distance, Radius, and SD JSD values (e.g., Radius 0.0517 on Tencent generation) can be artificially low. A second channel is DPO: Section 2.3 says training data are 'the corpus output by CAMS and corresponding individuals' real trajectories' and the loop is 'training → deployment → testing → data collection → retraining'; Appendix A.1's '300 training users and 1200 test users' is not reconciled with the 150-user evaluation, so it is unclear whether positive DPO examples are real trajectories of the evaluation users. If either channel uses evaluation users, the comparison against baselines is not a fair zero-shot test.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CAMS, an agentic framework for urban human mobility simulation built on the urban LLM CityGPT. The framework combines three modules: MobExtractor, which extracts and synthesizes linguistic mobility patterns from user profiles; GeoGenerator, which generates home/work anchor points through a macro-to-micro cascaded reasoning loop with collective-distribution reflection and maps activity sequences to real urban locations using an enhanced CityGPT; and TrajEnhancer, which applies iterative DPO training to align generated trajectories with real trajectories. Experiments on Tencent and ChinaMobile Beijing datasets compare CAMS against mechanistic, deep-learning, and LLM baselines using JSD-based individual, collective, semantic, and hallucination metrics, and report that CAMS achieves the highest CMRR on both trajectory generation and recovery. The abstract claims that CAMS achieves superior performance without relying on externally provided geospatial information and that it establishes a new paradigm for integrating agentic frameworks with urban-knowledgeable LLMs.","tokens_in":17627,"tokens_out":5162,"duration_ms":49418,"significance":"If the central claim holds, CAMS would be a meaningful advance in LLM-based mobility simulation: it explicitly models hierarchical urban structure, uses collective distribution feedback to condition individual generation, and applies DPO to improve spatiotemporal continuity. The paper also provides a broad set of ablations and compares multiple LLM backbones, which is useful for the community. However, the empirical support is currently undermined by unresolved questions about test-set information leakage through the collective-distribution reflection and the DPO training data, and by the small, density-biased evaluation sample. These issues bear directly on the zero-shot generalization claim and must be resolved before the main conclusion can be accepted.","major_comments":[{"comment":"The reflection stage of the Anchor Location Extractor adjusts generated home and workplace assignments by comparing against the 'ground-truth distribution,' but the paper never states that this distribution is computed exclusively from training users. Appendix A.1 says the collective-knowledge experiment uses a 44,313-user subset and that 100 recovery and 50 generation users are subsequently selected from the trajectory-dense users; the ground-truth home/work distribution may therefore include the evaluation users themselves. If it does, the reflection loop is directly fitting the test users' anchor-point distribution, and the reported spatial JSD values in Table 2 (e.g., Tencent Radius 0.0517 and SD 0.0457; ChinaMobile Radius 0.0507 and SD 0.0473) are inflated and do not support the zero-shot generalization claim. Please state explicitly whether the ground-truth distribution excludes all 150 evaluation users, and if it does not, rerun the evaluation using a distribution computed only from training users.","section":"Section 2.2.1 and Appendix A.1"},{"comment":"The DPO training data are described as constructed from 'the corpus output by CAMS and corresponding individuals' real trajectories,' with '300 training users and 1200 test users' reported in Appendix A.1, but this is never reconciled with the 100 recovery and 50 generation users used in Tables 2 and 3. If any of the 150 evaluation users' real trajectories serve as positive DPO examples, then TrajEnhancer is trained on test data and the comparison against the baselines is not a fair zero-shot test. Please define the exact disjoint sets of users used for DPO training, evaluation, and the collective distribution, and confirm that evaluation users are excluded from all positive DPO samples.","section":"Section 2.3 and Appendix A.1"},{"comment":"The evaluation uses only 100 recovery users and 50 generation users, selected as the top users by average daily trajectory points, and Tables 2 and 3 report single point estimates with no error bars, confidence intervals, or significance tests. Given the stochastic nature of LLM generation and the biased sampling toward data-rich users, the claimed 'superior performance on 11 out of 16 metrics' is not yet statistically supported. Please report repeated-generation variance and significance tests, and evaluate on a more representative sample or justify why the dense-trajectory subsample is sufficient for the generalization claim.","section":"Section 3.1 and Appendix A.1"},{"comment":"The abstract and Section 3.2.1 state that CAMS achieves superior performance without relying on externally provided geospatial information, but GeoGenerator fine-tunes CityGPT on 10,000 POI question-answer pairs constructed from urban spatial data aligned to OSM and Foursquare, and Section 3.1 states that OSM road/AOI data and Foursquare POI data jointly represent urban space. Please clarify what counts as 'external' in this claim; as written, the claim is difficult to interpret because the method appears to use the same city's geospatial data during fine-tuning and evaluation, even if no map tool is queried at inference time.","section":"Section 2.2.2 and Section 3.1"},{"comment":"The Gemma3-27B row in Table 4 has identical values for Tencent and ChinaMobile (0.3994, 0.3903, 0.4160, 0.0695, 0.1717, 0.0506, 0.3265, 0.6570, 0.8252), while all other models differ across the two datasets, which strongly suggests a copy-paste error. This table supports the claim that enhanced CityGPT outperforms other LLM backbones, so the error must be corrected and the affected conclusions rechecked before the comparison can be interpreted.","section":"Table 4"}],"minor_comments":[{"comment":"Figure 1 contains garbled placeholder text ('?????????') and an anomalous '240 Duration 4h' fragment; the figure should be cleaned before publication.","section":"Figure 1"},{"comment":"The name of the LLMob baseline is inconsistent: the introduction refers to 'LLM-Mob' while the tables and text use 'LLMob'; please unify the name and ensure the cited reference matches the method.","section":"Introduction and Tables 2-3"},{"comment":"The description of the deep-learning baseline setup says the training set is reduced to '3× test set,' but the exact number of users and trajectories is not specified; please state the concrete training sizes for each baseline.","section":"Section 3.1"},{"comment":"There are several typos, including 'accross' in Section 3.1 and 'it's ability' in Section 2.3; the manuscript should be proofread carefully.","section":"Throughout"},{"comment":"The relationship between the 44,313-user subset, the top 1,500 users for the Individual Knowledge Extractor, the 300 training users and 1,200 test users for DPO, and the 150 evaluation users is not clearly described; presenting a single diagram of the data splits would remove ambiguity.","section":"Appendix A.1"}],"recommendation":"major_revision","confidential_remarks":"The central issue is the unresolved possibility of test-set leakage in two channels: the collective-distribution reflection and the DPO training data. If the authors cannot demonstrate strict disjointness of the evaluation users from the ground-truth distribution and from the DPO positive samples, the main zero-shot generalization claim would collapse, which would make the paper unsuitable for publication in its current form. The identical Gemma3-27B row in Table 4 also suggests table-quality problems that should be checked across all reported numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, what's interesting: CAMS is a clean three-module agentic design—MobExtractor, GeoGenerator, TrajEnhancer—that puts an urban-knowledgeable LLM (CityGPT) at the center, uses ReAct-style iterative reasoning for anchor points, and uses DPO to align generated trajectories with real ones. The novelty is in the integration and in showing that a small set of template users can be used to synthesize patterns for new users. The ablation does show that the reflection module and DPO iterations move the metrics, and the paper is upfront about comparing against a reasonable set of baselines. That's real work.\n\nBut the evaluation has a problem that undercuts the central claim of zero-shot generation from profiles alone. Section 2.2.1 describes a reflection stage that compares generated home/work assignments against 'ground-truth distribution' and adjusts subsequent iterations. The paper never says that this distribution is computed only from training users. Appendix A.1 says collective-knowledge experiments use a 44,313-user subset, and the reported evaluation uses the top 100 users for recovery and 50 for generation. If that ground-truth distribution is aggregated over the whole subset—which includes the 50 generation users—then the spatial metrics (Distance, Radius, SD) are being directly fit to the test distribution. That's not a minor technicality; it's the difference between reporting generalization and reporting a curve-fit.\n\nThe DPO section has the same ambiguity. Section 2.3 says training data are 'the corpus output by CAMS and corresponding individuals' real trajectories' and describes a loop of train->deploy->test->collect->retrain. Appendix A.1 says 300 training and 1200 test users but doesn't reconcile those numbers with the 100/50 evaluation split. If real trajectories of the evaluation users appear as positive DPO examples, the model has seen the test trajectories during training. The paper needs to state explicitly that evaluation users are excluded from both the reflection ground-truth distribution and the DPO training data.\n\nThere are also smaller but real issues: only 100 and 50 users, selected as the top by trajectory density (biased toward heavy users), no error bars or significance tests, and a claim that CAMS works 'without relying on externally provided geospatial information' while the method section says it fine-tunes CityGPT using OSM and Foursquare POI data. That last point may be defensible if the claim is about inference-time inputs only, but as written it's an overstatement.\n\nSo my take: the architecture is worth publishing and discussing, and the evaluation could be fixed with experiments that cleanly separate training and test. As it stands, the paper's central empirical claim is not yet supported. I'd send it to peer review—this is exactly the kind of paper a good reviewer can push to either confirm or kill the leakage hypothesis—but I'd advise the editor to treat the empirical claims with suspicion until the data-handling questions are answered. I wouldn't cite the empirical results until then, though I might cite the framework.","headline":"A well-built LLM-agent pipeline for mobility simulation whose central zero-shot claim is currently undercut by ambiguous test/train separation in the reflection and DPO stages.","tokens_in":17996,"tokens_out":2765,"would_cite":false,"duration_ms":25493,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CAMS claims that an agentic framework built on an urban-knowledgeable LLM can generate realistic daily trajectories in a real city from only a user profile, without external geospatial data.","keywords":["human mobility simulation","LLM agents","CityGPT","urban spatial knowledge","trajectory generation","direct preference optimization","agentic framework","mobility pattern extraction"],"falsifier":"Re-run the Anchor Location Extractor with the reflection stage restricted to training-user home and work distributions only, then compare the resulting Distance, Radius, and SD JSDs to the values in Table 2; if the gap shrinks materially, the reported zero-shot generalization was partly built on peeking at evaluation users.","tokens_in":17012,"feed_emoji":"🏙️","tokens_out":5763,"duration_ms":61152,"temperature":0.7,"pith_summary":"The paper proposes CAMS, an agentic framework for human mobility simulation that combines a large language model pre-trained on urban spatial knowledge with three cooperating modules. It claims that, starting from a short user profile such as age, income, occupation, and education, CAMS can generate a full day of realistic movement through a real city, including plausible home and work locations and semantically sensible activity sequences. This matters because traditional trajectory simulators need large travel-diary datasets or external map services, while CAMS aims to produce synthetic populations cheaply and interpretably from profiles alone. Experiments on two real-world datasets show CAMS outperforming existing deep-learning and LLM-based baselines on most metrics, especially spatial distribution and daily routine measures.","feed_headline":"One user profile in, one realistic urban day out","feed_subtitle":"Three cooperating modules draw on built-in urban-geography knowledge, no map API needed at generation time.","key_machinery":"The central object is a three-module agentic pipeline built on CityGPT, an LLM post-trained on urban spatial knowledge expressed in language. Mobility patterns flow from MobExtractor to GeoGenerator to TrajEnhancer, with urban structure encoded as a hierarchical address system (administrative area to subdistrict to street to point of interest). The load-bearing mechanism is the macro-to-micro cascaded generator for anchors, which uses reasoning-execution-reflection cycles and compares generated home and work locations against the collective ground-truth distribution, together with DPO training that progressively reinforces spatiotemporal continuity in generated trajectories.","core_discovery":"The paper contends that human mobility simulation can be recast as a language-driven generation problem, and that an LLM endowed with urban geospatial knowledge can carry the whole pipeline. CAMS decomposes the task into three stages: MobExtractor condenses raw trajectories of a small set of template users into natural-language mobility patterns and synthesizes new patterns for unseen users; GeoGenerator places each user's home and workplace through a macro-to-micro cascaded generator that reflects against the collective spatial distribution, then maps the rest of the day onto real points of interest; TrajEnhancer generates the final trajectory and aligns it with real trajectories through iterated direct preference optimization. The paper claims this yields superior performance without relying on externally provided geospatial information at generation time, and that the framework establishes a new paradigm for integrating urban-knowledgeable LLMs with agentic reasoning in mobility simulation.","pith_inferences":["Not tested in the paper: withholding evaluation users' home and work coordinates from the collective distribution used in the reflection stage would directly test whether the reported spatial-metric advantages come from genuine zero-shot generalization or from fitting the test set.","The modular split between pattern extraction and geometry generation suggests the pipeline could transfer to other spatiotemporal simulation tasks, such as epidemic contact networks or retail footfall, by swapping the urban geometry source.","Because the reflection stage pushes individual outputs toward aggregate distributions, the framework may underrepresent rare or extreme mobility profiles; a subgroup analysis by occupation and income would quantify this flattening.","The paper's experiments use open map data and Foursquare points of interest to align the dataset, so the 'no external geospatial information' claim should be read as applying at query time, not to the model's training and alignment data."],"forward_implications":["A city planner could generate synthetic daily trajectories for hypothetical residents defined only by profiles, without collecting travel surveys, and use them to test interventions like 15-minute-city layouts or transport policies.","Because new-user trajectories are generated without retraining, the framework could scale profile-to-trajectory simulation to large synthetic populations.","Generated trajectories come out as interpretable toponyms in a hierarchical address system, making them auditable against known urban facts rather than opaque coordinate sequences.","Repeated DPO iterations progressively reduce spatial-temporal discontinuity, so the same framework can be continuously refined as more real trajectories become available."],"supporting_citations":[{"why":"Supplies CityGPT, the urban-knowledgeable LLM that CAMS fine-tunes and uses as its foundation model.","marker":"[10]"},{"why":"The CoPB baseline that CAMS compares against, which also motivated the need to map mobility patterns onto real urban space.","marker":"[31]"},{"why":"The LLMob baseline that derives mobility patterns from trajectory features; CAMS credits it for collective-distribution strength.","marker":"[37]"},{"why":"The MoveSim deep-learning baseline whose generative framework and evaluation metrics CAMS adopts in comparisons.","marker":"[9]"},{"why":"The TimeGeo mechanistic baseline used as a comparison point for mobility simulation without travel surveys.","marker":"[16]"},{"why":"The ActSTD deep-learning baseline for activity trajectory generation compared in the experiments.","marker":"[43]"},{"why":"The DSTPP diffusion-based baseline for spatiotemporal point processes compared in the experiments.","marker":"[44]"},{"why":"Provides the reasoning-execution-reflection cycle used in the anchor location extractor.","marker":"[41]"},{"why":"Supplies the embedding-based similar-user retrieval method used to synthesize mobility patterns for new users.","marker":"[42]"}],"fun_headline_variants":["LLM agentic framework simulates realistic urban mobility","CAMS: CityGPT agentic framework for urban human mobility","No map needed: LLM-only agentic urban mobility","Agentic LLM with built-in city knowledge for mobility","From user profile to realistic urban day, via LLM agent"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the ground-truth spatial distribution used in GeoGenerator's reflection stage is computed only from training users; the paper never states that, and the zero-shot claim collapses if evaluation users helped shape it.","fun_headline_variants_meta":{"raw":{"variants":["LLM agentic framework simulates realistic urban mobility","CAMS: CityGPT agentic framework for urban human mobility","No map needed: LLM-only agentic urban mobility","Agentic LLM with built-in city knowledge for mobility","From user profile to realistic urban day, via LLM agent"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000907,"raw_usage":{"total_tokens":3917,"prompt_tokens":981,"completion_tokens":2936,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":597,"completion_tokens_details":{"reasoning_tokens":2868}},"tokens_in":597,"tokens_out":2936,"duration_ms":20383,"temperature":1.0,"reasoning_tokens":2868,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:57:54.432041+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the Anchor Location Extractor with the reflection stage restricted to training-user home and work distributions only, then compare the resulting Distance, Radius, and SD JSDs to the values in Table 2; if the gap shrinks materially, the reported zero-shot generalization was partly built on peeking at evaluation users.","supporting_citations":[{"cited_title":"Beyond imitation: Generating human mobility from context-aware reasoning with large language models","cited_arxiv_id":null,"evidence_quote":"The CoPB baseline that CAMS compares against, which also motivated the need to map mobility patterns onto real urban space."},{"cited_title":"Learning to simulate human mobility","cited_arxiv_id":null,"evidence_quote":"The MoveSim deep-learning baseline whose generative framework and evaluation metrics CAMS adopts in comparisons."},{"cited_title":"The timegeo modeling framework for urban mobility without travel surveys","cited_arxiv_id":null,"evidence_quote":"The TimeGeo mechanistic baseline used as a comparison point for mobility simulation without travel surveys."},{"cited_title":"Activity trajectory generation via modeling spatiotemporal dynamics","cited_arxiv_id":null,"evidence_quote":"The ActSTD deep-learning baseline for activity trajectory generation compared in the experiments."},{"cited_title":"Spatio-temporal diffusion point processes","cited_arxiv_id":null,"evidence_quote":"The DSTPP diffusion-based baseline for spatiotemporal point processes compared in the experiments."}],"review_version":2}