{"id":"a13ee56c-5f8a-4607-b023-24ebef81dd9f","arxiv_id":"2607.14542","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"CityLLM, an LLM agent over PostGIS and Neo4j, answers 85–100% of 54 curated Rotterdam city-model queries correctly.","lead":"CityLLM lets users ask natural-language questions about 3D city models by combining a spatial database for buildings with a graph database for streets and amenities. On 54 self-curated Rotterdam questions, it answered 85–100% correctly, but the evaluation is small, not externally benchmarked, and lacks released code.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The evaluation's only quantitative evidence rests on 54 self-curated queries labeled by the authors with no inter-coder reliability check or released rubric; if those labels are biased, the reported 85–100% accuracy claims do not stand.","rationale":"The reader's weakest_assumption is essentially the same as mine: the evaluation depends on self-curated queries and author manual checks. I agree with that assessment. The framework description is clear, and the sample outputs in Tables 3–5 provide some evidence that chaining works in selected instances; the explicit failure table also adds transparency. However, independent support is thin: no code or data release, no external test set, no repeated runs, no external baseline, and no formal verification. The correct disposition is therefore conditional: the architecture is plausible, but the quantitative claim needs an independent evaluation before acceptance. Since the reader already returned CONDITIONAL, my read does not change the verdict.","tokens_in":10840,"tokens_out":3510,"duration_ms":44195,"concrete_test":"Release the full evaluation pack: the 54 natural-language queries, expected answers, per-query scoring rubric, raw model outputs/transcripts, and generated SQL/Cypher. Then have two independent annotators who are not authors and are blind to the expected answers score every response and visualization, and report Cohen's kappa plus per-model accuracy using their labels. If kappa is below 0.8 or if the independently scored accuracies fall outside the reported ranges, the empirical headline should be revised; if they match, the manual-evaluation concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim — answer correctness 85.2–100%, visualization correctness 92.9–100% — rests entirely on the §4 protocol in which the authors themselves scored each response and visualization against their own expected answers, with partial matches counted as incorrect. The 54 queries, expected answers, scoring rubric, and raw transcripts are unreleased, no inter-annotator agreement is reported, and each query was run once. This matters because the labels are not just a measurement detail: the paper uses them to rank LLMs and to conclude the framework works. If the queries were selected or phrased after seeing what the system handles well, or if the manual judgments are lenient/biased, all reported percentages could shift substantially. This is not an accusation of fraud; it is an unaddressed threat to reproducibility. The architecture and chaining examples in Tables 3–5 are useful, but they do not support the headline accuracy numbers without an independent scoring process.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents CityLLM, an LLM-based agent framework for querying semantic 3D city models. CityJSON data are stored in PostgreSQL/PostGIS using a compact relational schema, while OpenStreetMap street networks and amenities are stored in a Neo4j graph database. An LLM agent translates natural-language queries into SQL or Cypher, supports iterative retry on execution errors, and chains across databases when needed; results are returned conversationally and visualized on an interactive map. The authors evaluate the framework on a Rotterdam CityJSON dataset (853 LoD2 buildings) using 54 self-curated queries grouped into spatial, graph, cross-database, and conversational scenarios, with GPT-OSS (120B), Gemini 3.1, and GPT-5.4. Reported results are high: answer correctness 85.2–100%, visualization correctness 92.9–100%, 100% query success, and few retries. The paper includes worked examples of generated SQL/Cypher and documented failure cases.","tokens_in":11134,"tokens_out":8576,"duration_ms":91942,"significance":"If the evaluation were independently reproducible, CityLLM would be a useful and pragmatic contribution to lowering the barrier for non-expert querying of semantic 3D city models. The architecture is clearly described, and Tables 3–5 provide valuable concrete examples of the generated SQL/Cypher and cross-database chaining. The paper's candor in reporting failure cases and in re-evaluating them with model variants is a strength. However, the quantitative claims currently rest on an internal, non-released evaluation protocol: 54 author-curated queries, author-defined expected answers, and author-performed manual scoring, with no inter-annotator agreement, no repeated runs, and no release of the benchmark artifacts. This is a serious reproducibility and validity concern that should be addressed before the headline accuracy figures can be taken as robust evidence for the framework's performance.","major_comments":[{"comment":"The headline quantitative claims (answer correctness 85.2–100%, visualization correctness 92.9–100%) are supported only by the §4 protocol: 54 queries curated by the authors, expected answers defined by the authors, and all criteria 'checked manually by the authors' against those expected answers. No query list, expected answers, rubric, transcripts, or code are released; no inter-annotator agreement is reported; and each query was run once. This design leaves substantial room for selection and judgment bias, so the reported percentages are not reproducible. I recommend releasing the full benchmark (queries, expected answers, raw outputs, and generated SQL/Cypher), running each query multiple times, and obtaining independent human annotations with agreement statistics (e.g., Cohen's κ). At minimum, the results should be presented as a pilot study rather than as definitive comparative acc","section":"§4, Tables 1–2"},{"comment":"The paper attributes the strong results to the framework's components — schema introspection, iterative retry, and cross-database chaining — but provides no ablation or comparison against a direct LLM baseline (e.g., a single-prompt, no-retry, no-schema-context setting). Because 'query success' is defined as successful execution within the retry limit, that metric is saturated by construction and carries little discriminative power; the meaningful metric is answer correctness. Without a baseline, it is unclear whether the orchestration adds value beyond the raw LLM's native ability to generate SQL/Cypher from a prompt. I recommend adding an ablation study that removes each framework component, or at least one no-framework baseline, to substantiate the contribution.","section":"§3.2, §5"}],"minor_comments":[{"comment":"Table 4 is very hard to parse. The caption says '-' denotes a correct response, but the cell alignment in the rendered table is ambiguous: for example, the PostGIS Q1 row appears to show 'Building -[Failed to return distinct types.]' in the GPT-OSS column, which conflicts with GPT-OSS's perfect score in Table 2. Please restructure the table (e.g., separate sub-tables per model, color-coding, or explicit cell rows) so that each failure description is clearly attributable to the correct model.","section":"Table 4"},{"comment":"Model naming is inconsistent: 'GPT-OSS (120B)' in Table 1 vs 'GPT-OSS' in the text; 'Gemini 3.1' vs 'Gemini 3.1 (Flash Lite)' in §5; 'GPT-5.x' in §3.2 vs 'GPT-5.4' elsewhere. Please harmonize the model names and versions.","section":"Throughout"},{"comment":"The statement 'Conversational queries (5 interactions; N=17)' is unclear. Please specify whether N counts individual user turns across the five interactions, and how multi-turn sequences are scored in the per-query metrics (e.g., whether each turn is an independent query or whether the sequence is scored holistically).","section":"§4"},{"comment":"No data or code availability statement is provided. Given the paper's emphasis on reproducibility and the current non-release of the benchmark, a clear statement on artifact availability (even if 'available upon request') should be added.","section":"§3.2 / §5"},{"comment":"Example responses such as 'Found 10 results. Showing first 5: [list]' suggest a truncation policy. If the system limits the number of displayed results, please explain how the framework decides what to show and whether truncation affects answer correctness when a user asks for a specific number of items.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The framework idea is sound and the architecture description is solid, but the evaluation is currently not at the level expected for publication in a serious venue. The authors should be strongly encouraged to release the benchmark artifacts and to add an external or independent scoring component. If the reproducibility issues are addressed, the paper could be acceptable; as it stands, the quantitative claims are not yet verified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the honest take: CityLLM is a tidy systems paper whose genuinely new piece is the two-backend design—PostGIS for the CityJSON buildings and Neo4j for OSM street network and amenities—with the LLM chaining across them. The worked examples in Tables 3–5 show that this is real and plausible. If you work on LLM interfaces to urban data, you should know this exists.\n\nThe paper does a fair job on the architecture: the relational schema is deliberately simple, the system prompt is large but the constraints are sensible, and exposing the generated SQL/Cypher is a nice transparency choice. The literature review is honest and correctly identifies prior single-backend systems (KCityChatBot, Kanna-Kolbe). The re-evaluation of failure cases with model variants is a step many papers skip.\n\nThe soft spot is exactly what the reader's report flagged: the evaluation is one manual pass over 54 self-curated queries on one city, with the authors scoring their own answers, no released code, no inter-annotator check, no repeated runs. Those accuracy numbers (85–100%) are therefore illustrative, not established. The stress-test note is right that the labels are doing real work in ranking models. It doesn't look like fraud, but the protocol is too unconstrained to support strong comparative claims like 'GPT-OSS outperforms GPT-5.4.' That could easily be an artifact of query phrasing or scoring leniency. Also no external baseline; we don't know how this compares to a simpler text-to-SQL wrapper or to the cited prior systems.\n\nProportionately, these are addressable weaknesses. The architecture and chaining behavior don't rest on the exact percentages. The paper is also honest about its scope: it calls the graph backend a proof of concept and lists extensions. That limitation statement should count in its favor.\n\nWho is this for? Researchers building LLM-based geospatial query tools, and people in urban digital twin tooling. It's a solid related-work anchor and a useful system description. It deserves peer review, but the eval needs independent scoring, a released benchmark, and ideally another city before publication.\n\nMy recommendation: send it for review, but push hard on the evaluation protocol.","headline":"CityLLM is a clean two-backend LLM-query system for CityJSON, with the cross-database chaining genuinely new and the evaluation too self-contained to trust the headline percentages.","tokens_in":11577,"tokens_out":2016,"would_cite":true,"duration_ms":20845,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CityLLM turns plain-language questions into accurate queries over 3D city models, with a perfect score on a 54-question test.","keywords":["semantic 3D city models","natural language querying","large language models","CityJSON","conversational interface","cross-database querying","spatial database","graph database"],"falsifier":"Run the same pipeline on a different city's CityJSON dataset with a new set of, say, 100 natural-language questions written by independent users, and measure answer correctness; if accuracy falls well below the reported 85–100% range, or the open-source model no longer leads, the central claim fails.","tokens_in":10802,"feed_emoji":"🏙️","tokens_out":3711,"duration_ms":40142,"temperature":0.7,"pith_summary":"This paper claims that a lightweight LLM-based agent, grounded in a deliberately simple relational schema for 3D city data plus a graph schema for street networks, can serve as a reliable conversational interface for non-experts. On a real city dataset of 853 buildings, the framework translated 54 natural-language questions into executable database queries, achieving a 100% query-success rate and answer correctness between 85% and 100% depending on the underlying model. An open-source model reached 100% correctness, outperforming two larger proprietary models, which the authors attribute to the constraining effect of schema context, tool selection, retries, and cross-database chaining. If this holds, natural-language access to urban digital twins no longer requires heavyweight knowledge graphs or top-tier commercial models.","feed_headline":"Open-source LLM posts perfect score on 54 city-model queries","feed_subtitle":"A lightweight spatial-plus-graph setup lets non-experts ask 3D city data questions in plain language.","key_machinery":"The load-bearing component is the combination of a purpose-built relational schema for the city model (object attributes separated from geometries, with precomputed derived properties and a simplified per-object view) and a graph schema for street networks and amenities, plus the LLM agent's constrained workflow: explicit tool selection, structured query generation, iterative error-fed retries, and cross-database chaining. This machinery does the work of turning natural language into reliable database queries.","core_discovery":"The central claim is that strong conversational querying of semantic 3D city models can be achieved through a modular architecture that maps CityJSON geometry and attributes into a compact relational schema, stores street-network and amenity data in a graph database, and lets an LLM agent choose the backend, generate SQL or the graph query language, chain results across backends when needed, and repair failed queries by feeding database errors back into the model. The authors demonstrate this with a 54-query evaluation on a Rotterdam CityJSON dataset, reporting 100% query success across all models, answer correctness of 100%, 94.4%, and 85.2% for three LLMs, and visualization correctness of","pith_inferences":["The evaluation is small and self-curated; transferring the framework to other cities and larger, user-generated question sets would test whether the reported accuracy generalizes.","The finding that schema design and prompt constraints may matter more than raw model capability could shift how practitioners build LLM-based data interfaces.","A testable extension is to apply the same framework to other semantic 3D formats (e.g., CityGML or IFC) and see if the compact-schema approach holds up.","Reducing the large system prompt via dynamic context retrieval, as the authors note, could make the framework lighter and cheaper without losing accuracy."],"forward_implications":["Non-expert users can interrogate 3D city models conversationally without learning SQL, graph query languages, or GIS software.","Cross-database questions that combine building data with street networks and amenities can be answered through chained queries.","Iterative query refinement with database error feedback reliably produces executable queries, even when initial attempts fail.","Smaller, open-source LLMs can match or exceed larger proprietary models when given structured schema context and clear tool boundaries.","The architecture is modular, so adding new data backends (e.g., sensor data, regulatory documents) is a natural extension."],"fun_headline_variants":["CityLLM nails all 54 city-model queries","CityLLM turns plain language into 3D city answers","CityLLM: 100% success on 54 city data queries","CityLLM lets non-experts query 3D city models","CityLLM: ask your 3D city anything"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The reported accuracy rests on the authors' own manual judgment of 54 self-curated queries against expected answers, so the results are only as representative and unbiased as that question set and those judgments.","fun_headline_variants_meta":{"raw":{"variants":["CityLLM nails all 54 city-model queries","CityLLM turns plain language into 3D city answers","CityLLM: 100% success on 54 city data queries","CityLLM lets non-experts query 3D city models","CityLLM: ask your 3D city anything"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001082,"raw_usage":{"total_tokens":4369,"prompt_tokens":762,"completion_tokens":3607,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":506,"completion_tokens_details":{"reasoning_tokens":3521}},"tokens_in":506,"tokens_out":3607,"duration_ms":27604,"temperature":1.0,"reasoning_tokens":3521,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T01:46:07.484694+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same pipeline on a different city's CityJSON dataset with a new set of, say, 100 natural-language questions written by independent users, and measure answer correctness; if accuracy falls well below the reported 85–100% range, or the open-source model no longer leads, the central claim fails.","supporting_citations":[],"review_version":1}