Pith. sign in

REVIEW 4 major objections 6 minor 58 references

Geo-Spatial Concept Probing of Large Language Models: Abstraction, Compositionality, and Grounding

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that concept understanding in LLMs should be measured through abstraction, compositionality, and grounding in internal representations, and reports that most tested LLMs show abstraction and partial compositionality but…

desk verdict A genuinely useful new benchmark and a robust-looking Mistral deficit, but the central abstraction claim is confounded by surface word identity until the authors add the obvious control experiments. read the letter →

arxiv 2608.07353 v1 pith:SHYCQ7BV submitted 2026-08-07 cs.CL cs.AIcs.IRcs.LG

Karim Radouane , Jose G Moreno , Lynda Tamine This is my paper · ORCID
classification cs.CLcs.AIcs.IRcs.LG
keywords conceptprobinglargelanguagemodelsgeo-spatialreasoningabstractioncompositionalitygroundinglinearout-of-distributiongeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the question 'do LLMs understand concepts?' can be answered by probing three core concept properties—abstraction, compositionality, and grounding—inside the models' internal representations, rather than by looking only at question-answering accuracy. Using a generated benchmark of geo-spatial questions about direction, distance, and topology between UK wards, the authors report that layer-wise linear probes classify concept types with 99.95–99.98% accuracy on a random split and keep 75–83% accuracy under out-of-distribution token and geographic splits for most models. The Mistral-family models are a clear exception, staying near chance on these splits. The paper also reports that compositional structure in the representations (logit correlations above 0.85) goes with stronger QA performance, and that grounding—answering the same questions when all coordinates and distances are given in the prompt—remains near chance. The overall point is that current LLMs show genuine abstraction and partial compositionality of geo-spatial concepts in their latent spaces while lacking a solid connection to real-world numeric referents.

What carries the argument

The load-bearing mechanism is a layer-wise linear probe applied to the average token embedding of a question: for each layer, a softmax classifier predicts one of seven concept classes (direction, distance, topology, and their conjunctions). The benchmark supplies the controlled variation: relational triplets between UK wards generated from geometric measurements, question templates in binary and MCQ form, and deliberately constructed splits—token-level OOD (train on north/south, test on east/west), compositional OOD (unseen conjunction combinations), and geographic OOD (train on middle region, test on upper region). The same testbed supports compositionality checks through cosine similarity between composite and summed atomic embeddings and through correlations among logits, embedding sums, and probability averages, plus grounding tests that insert coordinates, distances, and bearings into the prompt.

What would settle it

Train the same linear probes on questions whose concept words (for example, 'north', 'far', 'bordering') are replaced by arbitrary placeholder tokens while preserving the underlying relation structure; if out-of-distribution accuracy stays near the 75–99% range, the conclusion that LLMs represent concepts abstractly would be corroborated, whereas a collapse to chance would show the probes were reading lexical templates rather than concepts.

Watch

Extended reading notes

Core claim

The central claim is that, judged by internal-representation probes, most text-only LLMs encode geo-spatial concepts as abstract, generalizable types, compose those types approximately additively, but fail to ground them. Concretely, linear probes trained on the average token embedding of a question reach 99.95–99.98% accuracy in classifying the underlying concept type on random splits, 75–83% on a geographic out-of-distribution split, and above 99% on compositional token-level splits, for Llama and Qwen models; Mistral models stay at 52.8–58.0% (random), 37.8–42.2% (geographic), and 22.5–29.4% (single token-level). The paper interprets this as evidence that most models clearly recognize concepts via their latent representations. For compositionality, cosine similarity between composite-question embeddings and sums of atomic embeddings is high for the successful models, and their logit correlations are ≳0.85, whereas Mistral models show weak, layer-unstable correlations. For grounding, accuracy in a binary task with explicit coordinates and distances hovers around chance (≈50%), and even the best MCQ grounding (66.7% on distance) comes with low consistency, so the paper concludes that concepts are not naturally grounded in current LLMs.

Load-bearing premise

The abstraction and compositionality results rest on the assumption that a linear classifier trained on the averaged token embedding of a full question measures the concept representation rather than the surface wording of the question; the experiments include no control that removes word identity or question-template overlap between composite and atomic items.

Editorial extensions

If this is right

  • QA accuracy alone is not a reliable proxy for concept understanding: even moderate MCQ accuracy coexists with low consistency, and the distance-threshold analysis shows MCQ answers track the anchor options rather than the model's own perception of closeness.
  • Most current LLMs (Llama, Qwen) store geo-spatial concepts as abstract, generalizable representations that survive out-of-distribution token and region shifts; Mistral-family architectures appear to lack this property.
  • Compositional structure in latent representations is associated with better question answering: models with logit correlations ≳0.85 across layers outperform models whose logits lose correlation, such as the Mistral family.
  • Explicit thinking/reasoning modes do not repair conceptual understanding and often degrade accuracy and consistency, so the deficit is representational rather than a decoding artifact.
  • Because grounding remains weak even when all numerical information is supplied in context, improving conceptual understanding in LLMs likely requires external grounding mechanisms rather than more training data in the same format.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to ask whether the near-perfect random-split probe scores partly reflect lexical template identity; replacing concept words with abstract symbols while preserving relational structure would separate surface encoding from genuine abstraction.
  • The same triplet-based test could be transferred to non-spatial concepts such as relevance, truth, or clinical attributes, giving information retrieval and healthcare a property-level audit of what LLMs represent.
  • The finding that MCQ distance answers re-calibrate to the option scale suggests geo-QA benchmarks should report consistency and threshold-revealed measures alongside accuracy to avoid overstating spatial understanding.
  • If the Mistral deficit is architectural rather than scale-dependent, testing intermediate-size variants or changing positional/attention schemes could pinpoint which design choice blocks concept abstraction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a concept-centric probing benchmark for geo-spatial concepts (direction, distance, topology, and their conjunctions) in text-only LLMs. It evaluates eight LLMs through binary and multiple-choice QA tasks and layer-wise linear probing, measuring three concept properties: abstraction, compositionality, and grounding. The central claims are that most models encode concepts almost perfectly in random splits (99.95–99.98%) and generalize to OOD splits (75–83%), that high compositional logit correlations (≥0.85) accompany higher QA accuracy, and that grounding is weak, with accuracy near chance when numerical context is provided. The Mistral family is reported to underperform on all abstraction and compositionality tests.

Significance. If the findings were fully supported, the paper would provide a reusable, controlled benchmark for testing concept properties in LLMs, a layer-wise analysis across model families and scales, and a clear negative result on grounding. The authors also introduce useful consistency metrics and a revealed-threshold analysis for the distance concept, and they explicitly separate task performance from representation-level probing. However, the central positive claims rest on linear probes over average token embeddings of full questions, a design that is highly vulnerable to lexical surface-form confounds, and the grounding test is partially tautological. With additional control conditions, the benchmark and results could become valuable to the community, but as presented the evidence for the abstraction and compositionality conclusions is not yet convincing.

major comments (4)
  1. [§6.1, Table 9] The abstraction results are confounded by surface lexical cues. The probe input is the average token embedding of the full question h=f_l(q), and the question templates in Table 3 and Figure 2 explicitly contain the concept word (north/south/east/west, close/far, within/borders). A linear probe can achieve near-perfect random-split accuracy by reading word identity, and the token-level OOD splits in Table 8 (e.g., train {N,S}, test {E,W}) test extrapolation along pretrained lexical-embedding axes rather than abstraction from linguistic tokens to a semantic type. The paper does not report any control that masks or removes the concept word, compares against a bag-of-words lexical probe, or evaluates on templates with random placeholders. Without such a control, the claim that 'most models clearly recognize concepts via their latent representations' (end of §6.1) is not established; the Mistral deficit may be real, but the positive abstraction claim is unsupported.
  2. [§6.2.2, Table 10] The compositionality metrics are subject to the same lexical-overlap problem. Cosine similarity between L(q) and L_add = L(q1)+L(q2) is high whenever q and its subquestions share surface tokens, and the logit/embedding/probability correlations use a classifier trained on the same average-token-embedding representation, so they can reflect template and word overlap rather than compositional structure. The claim that models with logit correlations above 0.85 have better QA accuracy is therefore not a clean test of compositionality. I recommend adding baselines with non-compositional control phrases (e.g., composed embeddings of unrelated atomic pairs, or of atomics whose words are permuted) and reporting whether the observed correlations exceed those baselines.
  3. [§6.3, grounding] The grounding test is operationally circular. The context provides the exact distance N, the threshold T0, and the rule 'X and Y are considered close if d(X,Y) ≤ T0', while the ground-truth labels were generated by the same phi_d,th with the same T0 in §4.2. A model that performs the arithmetic comparison should be at ceiling; near-chance accuracy therefore conflates failure to ground the concept with failure in numerical comparison or instruction following under a long context. The conclusion that 'concepts are not naturally grounded' is not supported unless control tasks isolate numerical comparison (e.g., direct comparison questions without the words close/far) or the threshold is varied across contexts. The revealed-threshold analysis in §5.2 also uses T0-derived labels, so the estimated T* is not an independent measure of the model's own closeness notion.
  4. [§5.2, Table 6] The revealed-threshold estimation is fitted to the model's predictions on data whose labels were generated from T0, so the KDE intersection can reflect the labeling scheme rather than a model-intrinsic boundary. Furthermore, the reported deltas (e.g., +7.6, +14.5, +30.8) are given without variance, confidence intervals, or significance testing, making it difficult to assess whether the differences between T* and T0 are meaningful. At minimum, the authors should report bootstrap or per-region variability and clarify that T* is a data-dependent estimate, not a free model parameter.
minor comments (6)
  1. [§4.2 and §5.2] The threshold is described as the mean in §4.2 ('setting th as the mean of the distribution, resulting in th=47.76km') but as the pairwise distance median in §5.2 ('a fixed threshold T0 set to the pairwise distance median'). Please clarify which statistic generated the labels and whether the two thresholds differ.
  2. [Table 3] The composition example 'Is X far from Y and close to Y?' is logically contradictory, since far and close are negations of each other in Table 2; this appears to be a typo and should be corrected.
  3. [Algorithm 2] In lines 8–10, the variable x is used both for the correct target ward and for a sampled negative ward; the notation x ∼ {z∈W | (z,y)∈¬R} is undefined because x was already fixed. Please use distinct symbols and define the sampling distribution.
  4. [§6.2.1] The abbreviation CGA is used both for the per-question compositional accuracy and for the aggregate Compositional Gap Accuracy; these should be renamed to avoid ambiguity.
  5. [Tables 9 and 10] The layer numbers are inconsistent: Table 9 reports final-layer indices L=36, 32, 28, while Table 10 lists '# layers' as 33 or 37. Please clarify which layer index is used for the final-layer results and whether the two tables refer to the same depth.
  6. [Abstract] The abstract uses the term 'groundness'; the standard term used throughout the paper is 'grounding'. Please correct this.

Circularity Check

1 steps flagged · score 4.0 of 10

Section 5.2's revealed-threshold accuracy is self-referential, but the central probing claims rest on independent splits.

  1. fitted input called prediction [Section 5.2 (Distance threshold analysis), Table 6]
    "For each region, and task, we estimate each model’s revealed threshold 𝑇∗ as the intersection of the kernel density estimation (KDE) curves fitted to the close- and far-labelled distance distributions extracted from existing predictions. ... Acc (%) and Cons (%) are evaluated at 𝑇∗; deltas in parentheses show gain over evaluation at 𝑇0."

    T* is fit to the very predictions being scored. Acc/Cons at T* relabel each question using a boundary inferred from the model's own close/far answers, so high accuracy only shows that the model agrees with a threshold extracted from itself. The deltas over T0 are self-consistency gains, not independent evidence of closeness perception or distance concept understanding. No held-out threshold or external behavioral anchor validates T*, so the performance numbers reported at T* reduce by construction to the fitted parameter. The T* values themselves are descriptive statistics, but the Acc/Cons columns presented alongside them are not independent measurements. This circular step is confined to Section 5.2 and does not support the main abstraction, compositionality, or grounding conclusions.

full rationale

The paper is a probing study rather than a derivation, and most of its pipeline is self-contained. Concept labels come from geometric measurements (bearings, distances, and GeoSPARQL topological predicates) via phi_theta and phi_d,th, not from model outputs. The abstraction, compositionality, and grounding tests evaluate held-out generalization of linear probes and QA accuracy; no load-bearing self-citation chain exists, and the authors cite no prior work of their own. The only place where an input is refit and then re-scored is the revealed-threshold analysis in Section 5.2: T* is estimated from the model's own close/far predictions, and then Acc/Cons are evaluated at T*, so those performance numbers measure self-consistency rather than independent accuracy. This is a genuine but localized circular step; it does not drive the main RQ2-RQ4 conclusions, which rest on the probe splits and grounding experiments. Surface-lexical confounds in the abstraction probe are a validity threat, but not a circularity by the criteria used here. I therefore assign a score of 4.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The central claims rest on the authors' operational definitions of spatial concepts. The close/far boundary is an arbitrary threshold (described once as mean, once as median), the direction bins are hand-set, and the probing tests assume average question embeddings and OOD splits measure concept abstraction. No external benchmark validates the concept definitions. The only entities introduced are the generated QA and probing datasets, which are not theoretical inventions.

free parameters (3)
  • Distance threshold T0 for close/far labeling = 47.76 km (UK); 1636.66 km (US)
    Defines ground-truth close/far for all distance instances. Chosen as the mean in Section 4.2 and as the median in Section 5.2, and it is the boundary against which all distance abstraction, compositionality, and grounding results are evaluated.
  • Revealed threshold T* per model, task, and region = e.g., Llama-3.1-8B UK Binary 16.80 km; Mistral-7B-v0.3 US Binary 344.80 km
    Inferred by fitting KDE curves to model predictions and taking their intersection (Section 5.2). Used to claim models compress closeness and to report performance deltas; it is fit to the same data it evaluates, with no uncertainty estimate.
  • Cardinal direction bin boundaries = 45, 135, 225, 315 degrees
    phi_theta maps bearings to N/S/E/W with fixed quadrant boundaries. These boundaries are chosen by hand and define the direction concept instances; no sensitivity analysis is reported.
assumptions (6)
  • domain assumption Concepts can be represented by finite sets of lexical instances and relational triplets (x, r, y).
    Section 3.1 identifies a concept C with its instance set I(C) and represents each instance as a positive triplet, which is a standard knowledge-representation assumption but not validated for the spatial concepts under study.
  • domain assumption Abstraction, compositionality, and grounding are the three core properties that define conceptual understanding.
    Sections 1 and 3.1 adopt this tripartite characterization from cognitive science [10, 24]; if the properties are not exhaustive or are misspecified, the benchmark's scope is misaligned.
  • ad hoc to paper Average token embedding of the full question is a valid layer representation for probing concept instances.
    Section 6.1 pools embeddings across all tokens of the question. This choice can smooth away or surface lexical cues; no comparison to other pooling strategies is made.
  • domain assumption Token-level and region-based OOD splits measure abstraction and systematicity.
    Section 6.1 equates generalization from N/S to E/W and from mid-region to upper-region with abstraction. This requires that the held-out tokens and regions are not learnable from surface statistics of the training split.
  • ad hoc to paper The distance threshold T0 (mean or median of pairwise distances) defines the correct close/far label.
    Section 4.2 sets T0 to the distribution mean (47.76 km), Section 5.2 calls it the median, and a different T0 is used for the US. All distance-concept results depend on this arbitrary boundary.
  • ad hoc to paper Providing coordinates, distances, and the threshold in context is a valid operationalization of grounding.
    Section 6.3 treats correct answers to numeric-context questions as evidence of grounding. This reduces grounding to arithmetic threshold comparison, which may not capture concept-to-world mapping.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Geo-Spatial Concept Probing of Large Language Models: Abstraction, Compositionality, and Grounding." pith.science (2026). https://pith.science/paper/SHYCQ7BV

@misc{pith2026260807353,
  author       = {Pith},
  title        = {Pith review of: Geo-Spatial Concept Probing of Large Language Models: Abstraction, Compositionality, and Grounding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SHYCQ7BV}},
  note         = {Machine review of arXiv:2608.07353}
}
read the original abstract

Understanding concepts is fundamental to generalization. Despite their impressive performance on a wide range of tasks, Large Language Models (LLMs) still struggle with genuine concept understanding. Prior work has evaluated conceptual understanding in LLMs using natural-language benchmarks or narrowly scoped synthetic tasks, but these settings often conflate multiple skills or lack precise control over the underlying concepts and their properties. To support controlled probing of concepts in LLMs, we design tests on their core properties: abstraction, compositionality, and groundness. We set up a concept-centric benchmark, targeting spatial concepts such as direction, distance, topology, and their compositions, and use question answering tasks serving as a proxy. We conduct extensive experiments across multiple LLM architectures and training regimes to analyze how model scale and design impact conceptual understanding. The results reveal clear limitations in current LLMs and provide insights into the factors shaping their ability to acquire and compose structured concepts. Our findings shed light on how concept-based LLMs can be redesigned for improved information access and knowledge management. The code will be available at https://github.com/rd20karim/concept-probing.

Figures

Figures reproduced from arXiv: 2608.07353 by the authors.

Figure 1
Figure 1. Direction function 𝜙𝜃 mapping angles to cardinal di￾rections. Distance threshold function 𝜙𝑑,th mapping distances to ‘close’ or ‘far’. For example, 𝜙𝜃 (𝑥, 𝑦) = E yields the triplet (𝑥, east_of, 𝑦), while 𝜙𝑑,th (𝑥, 𝑦) = close yields (𝑥, close_to, 𝑦); analogous triplets are generated for all remaining atomic distance and direction relations. We select the distance threshold based on the pairwise distance distribution … view at source ↗
Figure 2
Figure 2. Prompt templates for Binary-QA and MCQ tasks. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Performance trends per concept type and composi [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Binary Task: Concepts classification accuracy per [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Compositional gap analysis (in blue rectangle), highlighting differences between composite and atomic question [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Binary-QA Task: Cosine similarity per layer between each compositional question embedding [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Logits-based Pearson correlation coefficient across LLM layers for both tasks. [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Effect of grounding information on model performance compared to settings without grounding information per [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 38 canonical work pages

  1. [1]

    Hiba Ahsan, Arnab Sen Sharma, Silvio Amir, David Bau, and Byron C Wallace

    work page

  2. [2]

    Amos Azaria and Tom Mitchell. 2023. The Internal State of an LLM Knows When It’s Lying. InFindings of the Association for Computational Linguistics: EMNLP 2023, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 967–976. https://doi.org/10.18653/v1/ 2023.findings-emnlp.68

    doi:10.18653/v1/ 2023

  3. [3]

    Lisa Beinborn, Teresa Botschen, and Iryna Gurevych. 2018. Multimodal Ground- ing for Language Processing. InProceedings of the 27th International Conference on Computational Linguistics, Emily M. Bender, Leon Derczynski, and Pierre Isabelle (Eds.). Association for Computational Linguistics, Santa Fe, New Mexico, USA, 2325–2339. https://aclanthology.org/C18-1197/

    work page 2018

  4. [4]

    Yonatan Belinkov. 2022. Probing Classifiers: Promises, Shortcomings, and Advances.Computational Linguistics48, 1 (March 2022), 207–219. https: //doi.org/10.1162/coli_a_00422

    doi:10.1162/coli_a_00422 2022

  5. [5]

    Brachman

    Ronald J. Brachman. 1979. Taxonomy, Descriptions, and Individuals in Natural Language Understanding. In17th Annual Meeting of the Association for Computa- tional Linguistics. Association for Computational Linguistics, La Jolla, California, USA, 33–37. https://doi.org/10.3115/982163.982174

    work page arXiv 1979

  6. [6]

    Anthony G Cohn and Robert E Blackwell. 2025. Evaluating the Ability of Large Language Models to Reason about Cardinal Directions, Revisited.arXiv preprint arXiv:2507.12059(2025)

    arXiv 2025

  7. [7]

    Cheng Deng, Tianhang Zhang, Zhongmou He, Yi Xu, Qiyuan Chen, Yuanyuan Shi, Luoyi Fu, Weinan Zhang, Xinbing Wang, Chenghu Zhou, Zhouhan Lin, and Junxian He. 2024. K2: A Foundation Language Model for Geoscience Knowledge Understanding and Utilization. InProceedings of the 17th ACM International Conference on Web Search and Data Mining (WSDM ’24). 161–170. h...

    work page arXiv 2024

  8. [8]

    Mahir Labib Dihan, MD Tanvir Hassan, MD TANVIR PARVEZ, Md Hasebul Hasan, Md Almash Alam, Muhammad Aamir Cheema, Mohammed Eunus Ali, and Md Rizwan Parvez. 2025. MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models. InForty-second International Conference on Machine Learning. https://openreview.net/forum?id=hS2Ed5XYRq

    work page 2025

Show all 58 references
  1. [9]

    Alexandru Dumitru, Venktesh V, Adam Jatowt, and Avishek Anand. 2025. Eval- uating List Construction and Temporal Understanding capabilities of Large Language Models. InProceedings of the 2025 International ACM SIGIR Conference on Innovative Concepts and Theories in Information...

    2025

  2. [10]

    Fodor and Zenon W

    Jerry A. Fodor and Zenon W. Pylyshyn. 1988. Connectionism and cognitive architecture: A critical analysis. , 3-71 pages. https://doi.org/10.1016/0010- 0277(88)90031-5

    1988 doi

  3. [11]

    Ruth Fong and Andrea Vedaldi. 2018. Net2Vec: Quantifying and Explaining How Concepts are Encoded by Filters in Deep Neural Networks.2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition(2018), 8730–8738. https: //api.semanticscholar.org/CorpusID:2738204

    2018

  4. [12]

    Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. 2022. Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Yoav Goldberg, Zornitsa Koza...

    2022 doi

  5. [13]

    Goddu, Alva Noë, and Evan Thompson

    Mariel K. Goddu, Alva Noë, and Evan Thompson. 2024. LLMs don’t know anything: reply to Yildirim and Paul.Trends in Cognitive Sciences28, 11 (2024), 963–964

    2024

  6. [14]

    Joseph Goguen. 2005. What Is a Concept?. InConceptual Structures: Common Semantics for Sharing Knowledge, Frithjof Dau, Marie-Laure Mugnier, and Gerd Stumme (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 52–77

    2005

  7. [15]

    Wes Gurnee and Max Tegmark. 2024. Language Models Represent Space and Time. InThe Twelfth International Conference on Learning Representations. https: //openreview.net/forum?id=jE8xbmvFin

    2024

  8. [16]

    Jianchao Ji, Zelong Li, Shuyuan Xu, Max Xiong, Juntao Tan, Yingqiang Ge, Hao Wang, and Yongfeng Zhang. 2023. Counterfactual Collaborative Reasoning. In Proceedings of the 16th ACM International Conference on Web Search and Data Mining (WSDM ’23). 249–257. https://doi.org/10.11...

    2023

  9. [17]

    Yuhan Ji, Song Gao, Ying Nie, Ivan Majić, and Krzysztof Janowicz. 2025. Founda- tion models for geospatial reasoning: assessing the capabilities of large language models in understanding geometries and topological spatial relations.Interna- tional Journal of Geographical Infor...

    2025

  10. [18]

    Mingyu Jin, Qinkai Yu, Jingyuan Huang, Qingcheng Zeng, Zhenting Wang, Wenyue Hua, Haiyan Zhao, Kai Mei, Yanda Meng, Kaize Ding, Fan Yang, Meng- nan Du, and Yongfeng Zhang. 2025. Exploring Concept Depth: How Large Language Models Acquire Knowledge and Concept at Different Layer...

    2025

  11. [19]

    Ekaterina Khramtsova, Shengyao Zhuang, Mahsa Baktashmotlagh, and Guido Zuccon. 2024. Leveraging llms for unsupervised dense retriever ranking. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1307–1317

    2024

  12. [20]

    Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pier- son, Been Kim, and Percy Liang. 2020. Concept bottleneck models. InInternational conference on machine learning. PMLR, 5338–5348

    2020

  13. [21]

    Lake and Marco Baroni

    Brenden M. Lake and Marco Baroni. 2018. Generalization without Systematicity: On the Compositional Skills of Sequence-to-Sequence Recurrent Networks. In Proceedings of the 35th International Conference on Machine Learning (ICML). 2873–2882. Proc. of ML Research Vol. 80

    2018

  14. [22]

    Martha Lewis, Nihal Nayak, Peilin Yu, Jack Merullo, Qinan Yu, Stephen Bach, and Ellie Pavlick. 2024. Does CLIP Bind Concepts? Probing Compositional- ity in Large Image Models. InFindings of the Association for Computational Linguistics: EACL 2024, Yvette Graham and Matthew Pur...

    2024

  15. [23]

    Lei Li, Jingjing Xu, Qingxiu Dong, Ce Zheng, Xu Sun, Lingpeng Kong, and Qi Liu. 2023. Can Language Models Understand Physical Concepts?. InThe 2023 Conference on Empirical Methods in Natural Language Processing. https: //openreview.net/forum?id=HaSS8a3Oe7

    2023

  16. [24]

    Charles Lovering and Ellie Pavlick. 2022. Unit Testing for Concepts in Neural Networks.Transactions of the Association for Computational Linguistics10 (2022), 1193–1208. https://doi.org/10.1162/tacl_a_00514

    2022 doi

  17. [25]

    Gengchen Mai, Weiming Huang, Jin Sun, Suhang Song, Deepak Mishra, Ninghao Liu, Song Gao, Tianming Liu, Gao Cong, Yingjie Hu, Chris Cundy, Ziyuan Li, Rui Zhu, and Ni Lao. 2024. On the Opportunities and Challenges of Foundation Models for GeoAI (Vision Paper).ACM Trans. Spatial ...

    2024 doi

  18. [26]

    Gengchen Mai, Krzysztof Janowicz, Rui Zhu, Ling Cai, and Ni Lao. 2021. Geo- graphic Question Answering: Challenges, Uniqueness, Classification, and Future Directions. InAGILE Conf.8. https://doi.org/10.5194/agile-giss-2-8-2021

    2021 doi

  19. [27]

    Rohin Manvi, Samar Khanna, Marshall Burke, David Lobell, and Stefano Ermon

  20. [28]

    Lobell, and Stefano Ermon

    Rohin Manvi, Samar Khanna, Gengchen Mai, Marshall Burke, David B. Lobell, and Stefano Ermon. 2024. GeoLLM: Extracting Geospatial Knowledge from Large Language Models. InThe Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=TqL2xBwXP3

    2024

  21. [29]

    Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed Representations of Words and Phrases and their Compositionality. InAdvances in Neural Information Processing Systems. 3111–3119. NeurIPS 2013

    2013

  22. [30]

    Jatin Nainani. 2024. Evaluating Brain-Inspired Modular Training in Automated Circuit Discovery for Mechanistic Interpretability.ArXivabs/2401.03646 (2024). https://api.semanticscholar.org/CorpusID:266843932 Karim Radouane, Jose G Moreno, and Lynda Tamine

    2024 arXiv

  23. [31]

    Masahiro Naito, Sho Yokoi, Geewook Kim, and Hidetoshi Shimodaira. 2021. Revisiting Additive Compositionality: AND, OR and NOT Operations with Word Embeddings.Proceedings of the ACL-IJCNLP 2021 Student Research Workshop (2021)

    2021

  24. [32]

    Core Francisco Park, Andrew Lee, Ekdeep Singh Lubana, Yongyi Yang, Maya Okawa, Kento Nishi, Martin Wattenberg, and Hidenori Tanaka. 2025. ICLR: In- Context Learning of Representations. InThe Thirteenth International Conference on Learning Representations. https://openreview.ne...

    2025

  25. [33]

    Andrew Parry, Catherine Chen, Carsten Eickhoff, and Sean MacAvaney. 2025. MechIR: A Mechanistic Interpretability Framework for Information Retrieval. InAdvances in Information Retrieval: 47th European Conference on Information Retrieval, ECIR 2025, Lucca, Italy, April 6–10, 20...

    2025 doi

  26. [34]

    Roma Patel and Ellie Pavlick. 2022. Mapping Language Models to Grounded Conceptual Spaces. InInternational Conference on Learning Representations. https: //openreview.net/forum?id=gJcEM8sxHK

    2022

  27. [35]

    Ellie Pavlick. 2023. Symbols and grounding in large language models.Philosoph- ical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences381, 2251 (2023), 20220041

    2023

  28. [36]

    Eleonora Poeta, Gabriele Ciravegna, Eliana Pastor, Tania Cerquitelli, and Elena Baralis. 2025. Concept-based Explainable Artificial Intelligence: A Survey.ACM Comput. Surv.(Nov. 2025). https://doi.org/10.1145/3774643 Just Accepted

    2025 doi

  29. [37]

    Sayantan Polley, Subhajit Mondal, Venkata Srinath Mannam, Kushagra Kumar, Subhankar Patra, and Andreas Nürnberger. 2022. X-Vision: Explainable Image Retrieval by Re-Ranking in Semantic Space. InProceedings of the 31st ACM International Conference on Information & Knowledge Man...

    2022

  30. [38]

    Ronak Pradeep, Sahel Sharifymoghaddam, and Jimmy Lin. 2023. Rankvicuna: Zero-shot listwise document reranking with open-source large language models. arXiv preprint arXiv:2309.15088(2023)

    2023 arXiv

  31. [39]

    Smith, and Mike Lewis

    Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. 2023. Measuring and Narrowing the Compositionality Gap in Language Models. InFindings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, Houda Bouamor, ...

    2023 doi

  32. [40]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Mod- els From Natural Language Supervision. InProceeding...

    2021

  33. [41]

    Nitin Ramrakhiyani, Vasudeva Varma, Girish Keshav Palshikar, and Sachin Pawar

  34. [42]

    Kenneth H. Rosen. 2011.Discrete Mathematics and Its Applications(7 ed.). McGraw–Hill Education

    2011

  35. [43]

    Gesina Schwalbe. 2022. Concept embedding analysis: A review.arXiv preprint arXiv:2203.13909(2022)

    2022 arXiv

  36. [44]

    Gauging, enriching and applying geography knowledge in Pre-trained Language Models.Inf. Process. Manage.62, 1 (Jan. 2025), 23 pages. https: //doi.org/10.1016/j.ipm.2024.103892

    2025

  37. [45]

    Weihang Su, Yichen Tang, Qingyao Ai, Junxi Yan, Changyue Wang, Hongning Wang, Ziyi Ye, Yujia Zhou, and Yiqun Liu. 2025. Parametric Retrieval Augmented Generation. Association for Computing Machinery, New York, NY, USA. https: //doi.org/10.1145/3726302.3729957

    2025

  38. [46]

    Matthew Trager, Pramuditha Perera, Luca Zancato, Alessandro Achille, Parmin- der Bhatia, and Stefan 0 Soatto. 2023. Linear Spaces of Meanings: Compositional Structures in Vision-Language Models.2023 IEEE/CVF International Conference on Computer Vision (ICCV)(2023), 15349–15358...

    2023

  39. [47]

    Adam Stein, Aaditya Naik, Yinjun Wu, Mayur Naik, and Eric Wong. 2024. To- wards compositionality in concept learning. InProceedings of the 41st Interna- tional Conference on Machine Learning(Vienna, Austria)(ICML’24). JMLR.org, Article 1893, 26 pages

    2024

  40. [48]

    Michael Völske, Alexander Bondarenko, Maik Fröbe, Benno Stein, Jaspreet Singh, Matthias Hagen, and Avishek Anand. 2021. Towards Axiomatic Explanations for Neural Ranking Models(ICTIR ’21). Association for Computing Machinery, New York, NY, USA, 13–22. https://doi.org/10.1145/3...

    2021

  41. [49]

    Mengru Wang, Yunzhi Yao, Ziwen Xu, Shuofei Qiao, Shumin Deng, Peng Wang, Xiang Chen, Jia-Chen Gu, Yong Jiang, Pengjun Xie, Fei Huang, Huajun Chen, and Ningyu Zhang. 2024. Knowledge Mechanisms in Large Language Models: A Survey and Perspective. InFindings of the Association for...

    2024 doi

  42. [50]

    Nico Van de Weghe, Lars De Sloover, Anthony Cohn, Haosheng Huang, Simon Scheider, Renée Sieber, Sabine Timpf, and Christophe Claramunt. 2025. Oppor- tunities and challenges of integrating geographic information science and large language models.Journal of Spatial Information S...

    2025

  43. [51]

    Yutaro Yamada, Yihan Bao, Andrew Kyle Lampinen, Jungo Kasai, and Ilker Yildirim. 2024. Evaluating Spatial Understanding of Large Language Models. Transactions on Machine Learning Research(2024). https://openreview.net/ forum?id=xkiflfKCw3

    2024

  44. [52]

    Ilker Yildirim and L.A. Paul. 2024. From task structures to world models: what do LLMs know?Trends in Cognitive Sciences28, 5 (2024), 404–415. https: //doi.org/10.1016/j.tics.2024.02.008

    2024 doi

  45. [53]

    Yuzhang Xie, Jiaying Lu, Joyce Ho, Fadi Nahab, Xiao Hu, and Carl Yang. 2024. PromptLink: Leveraging Large Language Models for Cross-Source Biomedical Concept Linking. InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrie...

    2024

  46. [54]

    Qianheng Zhang, Song Gao, Chen Wei, Yibo Zhao, Ying Nie, Ziru Chen, Shijie Chen, Yu Su, and Huan Sun. 2025. GeoAnalystBench: A GeoAI benchmark for assessing large language models for spatial analysis workflow and code generation.Transactions in GIS29, 7 (2025), e70135

    2025

  47. [55]

    Shengyao Zhuang, Honglei Zhuang, Bevan Koopman, and Guido Zuccon. 2024. A setwise approach for effective and highly efficient zero-shot ranking with large language models. InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information R...

    2024

  48. [56]

    Zeping Yu and Sophia Ananiadou. 2025. Understanding and mitigating gender bias in llms via interpretable neuron editing.arXiv preprint arXiv:2501.14457 (2025)

    2025 arXiv

  49. [2024]

    InProceedings of the 41st International Conference on Machine Learning(Vienna, Austria)(ICML’24)

    Large language models are geographically biased. InProceedings of the 41st International Conference on Machine Learning(Vienna, Austria)(ICML’24). JMLR.org, Article 1409, 16 pages

  50. [2025]

    In Findings of the Association for Computational Linguistics: EMNLP 2025, Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (Eds.)

    Elucidating Mechanisms of Demographic Bias in LLMs for Healthcare. In Findings of the Association for Computational Linguistics: EMNLP 2025, Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (Eds.). Association for Computational Linguistics, Suzhou...

    2025 doi

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.