Pith. sign in

REVIEW 5 major objections 5 minor 10 references

AI vs Human Expert Reasoning: Assessing Agreements in Building Typology Predictions based on Street View Imagery

T0 review · 5 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Pith's one-line read This paper claims that off-the-shelf vision-language models can label building construction, current use, and number of storeys from street-view images with roughly 70% agreement with expert labels, without task-specific training.

desk verdict Useful applied benchmark for VLM-based building typology prediction, but the 70% agreement figure is only as solid as the uncharacterized expert labels behind it. read the letter →

arxiv 2607.14756 v1 pith:P7EIFDPL submitted 2026-07-16 cs.AI

classification cs.AI
keywords vision-languagemodelsbuildingtypologystreetviewimageryzero-shotclassificationChain-of-ThoughtpromptingexpertagreementurbananalysisJakarta
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Zero-shot vision-language models, no fine-tuning, can produce building-typology labels from street-view imagery that agree with expert civil-engineer and architect labels roughly 70% of the time. The authors tested three commercial models on 6,000 images in Penjaringan, Jakarta, and found storeys prediction most accurate (about 80% agreement) and current use least, while Chain-of-Thought prompting gave the most stable results across models and typologies. They also compared the models' written reasoning with experts' reasoning, finding AI relies on visible features like walls and entrances while experts add context and domain knowledge. If correct, this gives data-scarce cities a fast, cheap route to granular building data for planning, flood-risk, and energy modelling, with the caveat that the ground-truth labels themselves are not independently validated.

What carries the argument

The load-bearing machinery is the evaluation protocol: zero-shot prompting (straight vs Chain-of-Thought, divided vs merged) of commercial VLMs, with expert labels as reference; the merged Chain-of-Thought prompt — asking the model to reason step by step about building features, surroundings, and validation before answering all three typologies together — proved most stable. A second piece is the word-probability analysis of the models' 200-word justifications, which extracts the dominant cues (e.g., wall, structure, entrance) and is used as the basis for comparing AI and expert reasoning. These two parts together let the authors both quantify agreement and diagnose why disagreements occur.

What would settle it

Have a second independent team of experts label the same 6,000 images and compute expert-expert agreement; if expert-expert agreement is close to the VLM-expert agreement (~70%), the claim that VLMs 'approximate expert capability' is confirmed as matching human consensus, but if expert-expert agreement is much higher (e.g., 90%), the VLM's 70% is a real gap. Alternatively, compare VLM labels to verified field-survey or building-record data for a sample of buildings; if accuracy there is far below 70%, the benchmark itself is the source of the number.

Watch

Extended reading notes

Core claim

The central claim is that off-the-shelf vision-language models approximate expert human judgement in classifying construction materials, current use, and number of storeys from street-view images, with average expert agreement of about 70% on a 6,000-image test set (Gemini 2.0 Flash 71.6%, GPT-4o 70.6%, Claude 3.5 Sonnet 69.7%). Storeys prediction is the strongest (~80% agreement) and current use the weakest (60-70%), because function is hard to infer from appearance alone. The authors also show that the models' reasoning differs systematically from experts': keyword probabilities in the models' 200-word rationales emphasize visual indicators, while experts emphasize condition, neighbourhood

Load-bearing premise

The expert labels are treated as ground truth, but the paper never shows that experts reliably agree with each other; if different expert teams label the same buildings differently, then the reported ~70% measures agreement with one subjective opinion stream rather than true classification accuracy.

Editorial extensions

If this is right

  • A city-district map of construction material, current use, and storeys for ~30,000 buildings in Penjaringan can be produced automatically, with mostly concrete/brick low-rise residential buildings and more diverse materials in informal coastal settlements.
  • The same zero-shot setup can be applied to other Global South cities without model retraining, lowering the cost of building-level data for planning and disaster-risk studies.
  • Storeys predictions are reliable enough for first-pass use; current-use predictions should be treated as draft labels needing verification, ideally with signage or on-site data.
  • The reasoning comparison suggests a practical division of labour: AI drafts labels from visual cues, human experts review ambiguous or mixed-use cases where context matters.
  • Using CoT merged prompting and batch processing cuts inference cost by about 50%, making city-scale application feasible on modest budgets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 70% figure is agreement with one expert annotation stream, not accuracy against an independent ground truth; measuring inter-expert agreement on the same images would show whether 70% is a human-consensus ceiling or a real shortfall.
  • A concrete test: have a second team of experts label a random sample of the 6,000 images; if expert-expert agreement is also around 70%, the VLM is effectively performing at the human-consensus level for these ambiguous categories.
  • The word-probability findings imply a tunable lever: instructing the model to weigh 'neighbourhood', 'condition', and 'design intent' — cues experts use — could plausibly raise current-use agreement, a hypothesis the paper does not test.
  • The method could be extended to an active-learning loop: use VLM confidence or reasoning-keyword agreement to flag low-confidence images for expert review, but the paper stops at full-scale prediction and does not evaluate such a mixed workflow.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper evaluates three commercial Vision-Language Models (GPT-4o, Claude 3.5 Sonnet, Gemini 2.0 Flash) for zero-shot prediction of building construction, current use, and storeys from Google Street View images in Penjaringan, Jakarta. Predictions are compared against manual labels by civil engineers and architects across a 2,000-image validation set and a 6,000-image test set, under four prompt configurations (straight/CoT, divided/merged). The central claim is that VLMs approximate expert capability at scale, with an average agreement of approximately 70% (Table 3). The paper also analyzes word probabilities in VLM reasoning and compares these with expert reflections on the same keywords, then maps predictions for about 30,000 buildings.

Significance. If the 70% figure is robust, this is a practically important result for data-scarce urban contexts: it would show that off-the-shelf VLMs can generate first-pass building-typology labels without task-specific fine-tuning. The paper has clear strengths: it compares multiple commercial VLMs, systematically varies prompting strategies, provides open code, and addresses a relevant Global South case study. However, the headline claim rests on expert labels whose reliability is not established, and the reasoning analysis is partly self-referential. With additional inter-annotator reliability evidence and an independent reasoning elicitation, the paper could become a useful benchmark for automated urban building characterization.

major comments (5)
  1. [§3.2 and Table 3] The 'accuracy' label is not supported by evidence of expert-label reliability. The paper defines accuracy as agreement with human expert labels, yet no inter-annotator reliability statistic (e.g., Cohen's or Fleiss' kappa) is reported, and only 20% of the initial 2,000 images were 'examined and labelled by consensus among experts.' The 6,000-image labels are not described at all: who labelled them, how many experts, and what consensus process. Since §4.1 itself states that current-use categorization is subjective and prone to 'greater variability in expert labelling,' the 70% VLM-expert agreement could be near or below the expert-expert agreement ceiling. Without expert-expert agreement, the headline figure is uninterpretable; the authors should report inter-annotator reliability for each typology and compare VLM-expert agreement against expert-expert agreement.
  2. [§3.2 and Fig 6] The provenance and independence of the 6,000-image expert labels are not specified. The text says 'expert agreement with model outputs was systematically assessed,' which suggests experts may have been asked to adjudicate VLM predictions rather than independently label the images. If experts were shown model outputs, the reported 71.6%, 70.6%, and 69.7% figures are human-acceptance rates, not accuracy. If labels were produced independently, the protocol (number of experts, training, consensus rule, blinding to model output) must be reported. This information is required to interpret Table 3.
  3. [§4.2 and Figs 7–8] The reasoning comparison is partly circular. The 10 keywords per typology are selected from the VLM's own highest-probability explanation words; the same words are then presented to experts, who are asked whether they 'reflected the primary factors considered in their own decision-making.' This elicitation restricts experts to the model's vocabulary and cannot independently establish expert reasoning. The conclusion that 'AI tends to focus on visual indicators, whereas human experts place greater emphasis on broader contextual cues' is therefore not tested. An independent expert reasoning elicitation (e.g., free-listing or think-aloud coding) is needed, with the resulting themes compared directly to the AI-derived keywords.
  4. [§4.1 and Table 3] Comparative and average accuracy claims lack uncertainty quantification. The text states that Gemini 2.0 Flash 'achieved the highest expert agreement' (71.6% vs 70.6% for GPT-4o) without confidence intervals or significance tests; with n=6,000 the difference may be within sampling error. Additionally, hallucination/invalid outputs are only reported for the 2,000-image Gemini straight-divided case; no invalid-output counts are given for the scaled CoT-m predictions in Table 3. If invalid outputs were discarded, the reported accuracies are conditional on the model producing a valid response. Report confidence intervals, formal model comparisons if ranking is claimed, and invalid-output counts for all models and configurations.
  5. [§4.2 and Fig 9] The 30,000-image map is described as providing 'reliable building-level information,' but the paper validates only a 6,000-image subset, and that validation is VLM-expert agreement, not an accuracy benchmark. The claim goes beyond the evidence. Please soften the wording or propagate prediction uncertainty to the full map; otherwise 'reliable' should be replaced by 'predicted.'
minor comments (5)
  1. [Abstract] Grammar: 'AI have the potential' should be 'AI has the potential.' Also, 'Storeys' is capitalized inconsistently throughout.
  2. [Tables 2–3] Model names are inconsistent: 'Anthropic (Claude Sonnet)', 'Google (Gemini)', and 'OpenAI (GPT)' are used in table headers, while the text refers to Claude 3.5 Sonnet, Gemini 2.0 Flash, and GPT-4o. Please harmonize.
  3. [References] The reference list is incomplete: several in-text citations are missing (e.g., Belaroussi 2025; DeepSeek-AI et al. 2025; Maharana et al. 2025; Wang et al. 2023; Husin 2025; Englhardt et al. 2019; Godfrey et al. 2015; Balasbaneh et al. 2020), and some listed entries are not cited in the text. Please reconcile the bibliography.
  4. [Fig 8 and §3.2] Fig 8 is titled 'AI vs Human Precursor (n=6,000)', but the method section describes keyword selection based on the 2,000-image validation set. Clarify whether the precursor was regenerated on the 6,000-image reasoning outputs or applied from the 2,000-image set; the current text is ambiguous.
  5. [Figs 5 and 9] Figure 5 heatmaps would benefit from numeric cell annotations; the colour legend is coarse and many cells are not readable in print. Figure 9's caption says 'Construction' but the text refers to three typologies; update the caption.

Circularity Check

1 steps flagged · score 4.0 of 10

Main 70% accuracy is externally benchmarked, but the reasoning-alignment analysis is self-referential: VLM-generated high-probability words are used as the items experts are asked to ratify.

  1. self definitional [Section 3.2 (Methods, paragraph on 10 keywords) and Section 4.2 (Experts’ Agreements in AI predictions)]
    "We then selected 10 keywords based on their probability scores and frequencies in the VLMs’ reasoning outputs. These keywords represent the most prominent concepts that influenced the models’ predictions. To assess alignment with expert reasoning, we presented these keywords to domain experts and asked whether they accurately reflected the primary factors considered in their own decision-making processes. All experts have identified that the presented keywords were reasonable and can serve as a precursor to further analysis."

    The 'AI reasoning' construct is defined by the VLM's own highest-probability explanation words, and 'expert alignment' is measured by asking experts to ratify those same AI-generated words. Thus the comparison is not between independent human reasoning and AI reasoning; it is between AI's self-selected vocabulary and experts' assent to that vocabulary. The later claim that 'VLM-generated reasoning aligns with expert interpretative frameworks' (Section 3.2) is therefore partly guaranteed by construction, because the instrument was generated by the system under test. This does not affect the external 70% accuracy benchmark, but it makes the reasoning-alignment results self-referential.

full rationale

The headline claim (VLMs reach ~70% agreement/accuracy against expert labels) is grounded in external human labels, so it is not circular; precision/recall/F1 are computed against expert-labelled sets (n=2,000 and n=6,000). The paper's self-citations (Law et al. 2019; Sakti et al. 2022) appear only as background and are not load-bearing. However, the reasoning-analysis component contains a self-referential step: the 10 keywords used to characterise AI reasoning are drawn from the VLM's own explanation outputs, and experts were then asked whether those same words were 'reasonable'. Using the model's own top words as the test items means the measured 'alignment' between AI and expert reasoning is at least partly an artefact of the evaluation design rather than an independent finding. This warrants a moderate circularity score, though the central accuracy result remains intact.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

This is an empirical benchmark paper with no mathematical derivation. The main implicit burdens are ground-truth reliability, the representativeness of filtered GSV data, and the assumption that VLM text rationales faithfully encode reasoning. All three are acknowledged partially in the paper but remain load-bearing.

free parameters (5)
  • Keyword selection threshold (top 10) = 10
    The reasoning analysis uses the 10 words with highest probability in VLM explanations; no sensitivity analysis is reported. Section 3.2.
  • Road buffer radius for visibility filtering = 10 m (95th percentile road width)
    Buildings not intersecting a 10-m buffer around roads are excluded, changing the building sample. Section 3.1.
  • Minimum valid image file size threshold = not specified
    Invalid images are removed by 'low file sizes' but the threshold is not disclosed. Section 3.1.
  • Model hyperparameters (temperature, max tokens) = defaults except max tokens
    Hyperparameters affect outputs and vary across models; the paper acknowledges this limits reproducibility. Section 3.2.
  • Scaling prompt selection = CoT merged
    After validation, only CoT merged prompts were used for the 6,000 and 30,000 image runs; this is a design choice based on observed stability. Section 4.1.
assumptions (4)
  • domain assumption Expert labels constitute reliable ground truth for building typology.
    The central 'accuracy' is expert agreement; if expert labels are noisy or biased, the accuracy estimate is not a stable benchmark. Section 3.2.
  • domain assumption Google Street View imagery and the LLaVA-filtered sample are representative of the building stock.
    The paper acknowledges GSV coverage biases but still generalizes to Penjaringan. Sections 2.1 and 3.1.
  • domain assumption The VLM's 200-word rationales reflect its actual decision process.
    The authors cite Turpin et al. (2023) showing CoT explanations can be post-hoc rationalizations, yet they still interpret word probabilities in those rationales as reasoning evidence. Sections 2.2 and 3.2.
  • domain assumption The predefined typology categories are complete and mutually exclusive.
    Hallucinations are flagged as invalid, indicating the predefined categories may not capture all valid building states. Section 4.1 and Fig A1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AI vs Human Expert Reasoning: Assessing Agreements in Building Typology Predictions based on Street View Imagery." pith.science (2026). https://pith.science/paper/P7EIFDPL

@misc{pith2026260714756,
  author       = {Pith},
  title        = {Pith review of: AI vs Human Expert Reasoning: Assessing Agreements in Building Typology Predictions based on Street View Imagery},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P7EIFDPL}},
  note         = {Machine review of arXiv:2607.14756}
}
read the original abstract

This research investigates the potential of Vision-Language Models (VLMs) to infer building typologies: Construction, Current Use, and Storeys from Google Street View (GSV) images. Predictions generated by VLMs are compared with inference by human experts (civil engineers and architects) as a source of manually labelled ground-truth data. We evaluate several state-of-the-art VLMs, including GPT-4o, Claude 3.5 Sonnet, and Gemini 2.0 Flash. By applying different scaling strategies and prompting techniques, we found that Chain-of-Thought prompts provide an overall more stable model performance. We also investigate the reasoning behind VLMs' building-typology predictions by examining the probabilities of keywords appearing in AI explanations. This enabled us to analyse patterns in these reasonings and identify key themes driving both agreements and disagreements between VLM and expert labels. We find that AI tends to focus on visual indicators, whereas human experts place greater emphasis on broader contextual cues and domain knowledge, in addition to visual cues. Overall, VLM can approximate experts' capability in building-typology classification at scale, with an average accuracy of approximately 70%. The study demonstrates the VLM's potential for AI automation in tasks that require pattern recognition and object identification in an urban context. AI have the potential to serve as complementary and collaborative tools for urban analysis, leveraging their strengths in understanding visual patterns. This study contributes to the exploration of the efficiency and scalability of AI visual prediction and provides insights into the reasoning processes that could support automation processes in urban analysis and prediction.

Figures

Figures reproduced from arXiv: 2607.14756 by the authors.

Figure 1
Figure 1. The Locations of Buildings with Available and Valid GSV data in Penjaringan District, North Jakarta A total of 63,729 street-view images were collected from the API, of which 29,937 were considered valid for predictions after a series of preprocessing steps, including data screening using LLaVA-v1.6- Mistral-7B. The locations of buildings with available and valid GSV data are shown in [PITH_FULL_IMAGE:figures/full_… view at source ↗
Figure 2
Figure 2. Flowchart of the Methodology Initially, the VLM was tested on a small validation dataset of 2,000 images in a zero-shot setting to evaluate baseline reasoning and output quality, identifying potential limitations and areas for refinement as can be seen in [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Multi-stage Framework Scaling To optimise the performance of VLM models, a series of prompting strategies was employed to improve prompt structure, including zero-shot and Chain-of-Thought (CoT) prompting (Wei et al., 2022). This experiment also explores the implications of using both divided and merged prompts of the four building typologies predicted. The approaches for prompting are [PITH_FULL_IMAGE:figures/full… view at source ↗
Figures from the paper (2 more)
Figure 8
Figure 8. Figure 8: AI vs Human Precursor (n=6,000) [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: visualises the spatial distribution of predicted building typology in North Jakarta, Indonesia, using the full, cleaned dataset. Key areas of interest are magnified in inset panels, emphasising regions with diverse building compositions and visually communicating the a…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

10 extracted references · 1 linked inside Pith

  1. [1]

    Allen, G. C. (2020). Understanding AI Technology. Joint Artificial Intelligence Center (JAIC), (April). Ananthram, A., Stengel-Eskin, E., Bansal, M., & McKeown, K. (2025). See It from My Perspective: How Language Affects Cultural Bias in Image Understanding. http://arxiv.org/abs/2406.11665 Anguelov, D., Dulong, C., Filip, D., Frueh, C., Lafon, S., Lyon, R...

  2. [15]

    https://doi.org/10.1016/j.jobe.2017.11.022 Lartey, D., & Law, K. M. Y . (2025). Artificial intelligence adoption in urban planning governance: A systematic review of advancements in decision-making, and policy making. Landscape and Urban Planning, 258, 105337. https://doi.org/10.1016/J.LANDURBPLAN.2025.105337 Law, S., Paige, B., & Russell, C. (2019). Take...

  3. [35]

    18 Xiao, J., Wu, W., Zhao, J., Fang, M., & Wang, J. (2025). Enhancing long-form question answering via reflection with question decomposition. Information Processing & Management, 62(6), 104274. https://doi.org/10.1016/J.IPM.2025.104274 Yin, L., & Wang, Z. (2016). Measuring visual enclosure for street walkability: Using machine learning algorithms and Goo...

  4. [76]

    https://doi.org/10.1016/j.apgeog.2016.09.024 Yin, Z., & Wang, S. (2025). Enhancing scientific table understanding with type-guided chain-of- thought. Information Processing & Management, 62(4), 104159. https://doi.org/10.1016/J.IPM.2025.104159 Zhang, Y ., & Dong, R. (2018). Impacts of street-visible greenery on housing prices: Evidence from a hedonic pric...

  5. [131]

    https://doi.org/10.1016/j.engappai.2023.107824 Fischer, G. (2022). A Research Framework Focused on AI and Humans instead of AI versus Humans. CEUR Workshop Proceedings,

  6. [139]

    P., & Kumar, V

    Ramalingam, S. P., & Kumar, V . (2025). Building usage prediction in complex urban scenes by fusing text and facade features from street view images using deep learning. Building and Environment, 267, 112174. https://doi.org/10.1016/J.BUILDENV .2024.112174 Reinhart, C. F., & Cerezo Davila, C. (2016). Urban building energy modeling - A review of a nascent ...

  7. [154]

    https://doi.org/10.1016/j.enbuild.2017.08.029 Colven, E. (2020). Thinking beyond success and failure: Dutch water expertise and friction in postcolonial Jakarta. Environment and Planning C: Politics and Space, 38(6). https://doi.org/10.1177/2399654420911947 Fan, Z., Feng, C. C., & Biljecki, F. (2025). Coverage and bias of street view imagery in mapping th...

  8. [177]

    https://doi.org/10.1016/j.buildenv.2020.106805 Hecht, R., Meinel, G., & Buchroithner, M. (2015). Automatic identification of building types based on topographic databases–a comparison of different data sources. International Journal of Cartography, 1(1). https://doi.org/10.1080/23729333.2015.1055644 16 Hoffmann, E. J., Wang, Y ., Werner, M., Kang, J., & Z...

Show all 10 references
  1. [2021]

    https://doi.org/10.1109/IC-NIDC54101.2021.9660554 Turpin, M., Michael, J., Perez, E., & Bowman, S. R. (2023). Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. Advances in Neural Information Processing Systems., 36, 74952–...

  2. [3136]

    Goel, R., Garcia, L. M. T., Goodman, A., Johnson, R., Aldred, R., Murugesan, M., Brage, S., Bhalla, K., & Woodcock, J. (2018). Estimating city-level travel patterns using street imagery: A case study of using Google Street View in Britain. PLoS ONE, 13(5). https://doi.org/10.1...

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.