Pith. sign in

REVIEW 3 major objections 5 minor 21 references

DistRAG: Towards Distance-Based Spatial Reasoning in LLMs

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DistRAG gives LLMs exact distance reasoning by retrieving city graphs.

desk verdict DistRAG is a clean, honest application of RAG to distance questions with real Easy/Medium gains, but the abstract's 'enables' claim overreaches: Difficult questions are only solved after handing the model a query template that encodes the answer structure. read the letter →

arxiv 2506.03424 v1 pith:NFUYD5VI submitted 2025-06-03 cs.CL cs.IR

Nicole R Schneider , Nandini Ramachandran , Kent O'Sullivan , Hanan Samet This is my paper
classification cs.CLcs.IR
keywords retrieval-augmentedgenerationspatialreasoningdistancegeodesicgraphretrievalSPARQLqueryLLMgroundingknowledgegraphs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the reason large language models fail at distance-based spatial reasoning is not the reasoning itself but missing spatial knowledge, and that the gap can be closed by retrieval. DistRAG stores geodesic distances between places as a graph and, for each question, retrieves a small subgraph to place in the prompt. Evaluated on 60 Australian distance questions, the method drives error to zero on direct and nearest-city questions where the same LLM without retrieval makes large errors. On harder comparison questions, retrieval alone does not suffice; the paper shows the difficulty shifts to constructing the right query structure.

What carries the argument

The central mechanism is a spatial knowledge store: an attributed graph whose nodes are place-name strings and whose edges carry geodesic distances, built from public map data. Around this graph, DistRAG wraps two retrievers: a vector-similarity index over graph triples that selects the top k=10 triples as natural-language context, and a SPARQL-query generator that writes and executes a query against the RDF form of the graph. The prompt then instructs the LLM to extract and return the distance or city name from that context alone. The graph-lookup step is what carries the argument: the model's task collapses to reading one edge or scanning the neighborhood of one node, which is why Easy and Medium questions reach zero error.

What would settle it

Run DistRAG on a distance question where the needed edge is deliberately excluded from the top-k retrieved triples or absent from the graph. If the model still returns the correct distance, the retrieval-completeness assumption is not load-bearing; if it answers wrongly rather than abstaining, the claim that retrieval fixes distance reasoning would need qualification. A second test: take the 20 Difficult questions and withhold the query template; if a different prompt ordering or a different base LLM solves them without the template, the claim that complex comparison reasoning is the bottleneck would be weakened.

Watch

Extended reading notes

Core claim

The paper claims that an LLM augmented with a retrieved spatial graph can answer distance-based reasoning questions it otherwise cannot answer. Concretely, encoding geodesic distances as graph triples and retrieving the top-k most relevant triples (by vector similarity or by executing a generated SPARQL query) lets the LLM produce exact answers for direct distance questions and for closest-city questions on a 60-question Australian benchmark, with zero mean squared error for the SPARQL variant and near-zero for vector retrieval. The same base LLM, prompted without the graph, reports errors up to $10^{5}$ in squared-kilometer terms. The claim therefore is that distance reasoning reduces to a retrieval problem plus a simple graph-lookup step, not to stronger parametric reasoning. Complex questions requiring comparing two distances remain unsolved by all methods unless the SPARQL query template is supplied, at which point error drops to zero.

Load-bearing premise

The method depends on the retriever putting every edge needed to answer the question into the top-k context it passes to the model, and on the graph actually containing those edges; the ablation measures only how often the model abstains, not whether wrong answers appear when edges are missing.

Editorial extensions

If this is right

  • For direct 'distance between A and B' questions, DistRAG achieves zero error, so a retrieval-augmented LLM can be trusted for exact lookups it was not trained on.
  • For 'closest city to A' questions, the method reduces error by 90-100% compared with the bare LLM, showing one-hop spatial reasoning is attainable through retrieved graph context.
  • Because the graph can be updated without retraining the model, DistRAG offers a route to answering questions about obscure places that never appear in training data.
  • Difficult questions involving comparing two distances remain unsolved by all methods without additional query-structure scaffolding, so the paper does not claim full spatial reasoning.
  • The SPARQL variant answers correctly every time it answers, at the cost of abstaining more often, especially as the knowledge store becomes sparser.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the retrieval-completeness assumption holds at scale, the same graph-lookup trick could extend to other edge-attribute relations stored in the same graph, such as travel time, bearing, or road distance, without changing the prompt.
  • Editorial inference: a hierarchical or region-indexed graph might push the approach past one-hop questions, since the Difficult failure appears to be a query-construction problem and a planner that decomposes comparisons into staged lookups could inherit the zero-error property.
  • Editorial inference: a testable extension is to measure accuracy, not just abstention, under edge sparsity; the current ablation reports only response rate, so it is unknown whether missing edges produce silent wrong answers when retrieval is incomplete.
  • Editorial inference: the 60-question Australian benchmark is small, so reproducing the result on a larger, multilingual set of place names would clarify whether the 'unpopular place' generalization is robust.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes DistRAG, a retrieval-augmented generation method for distance-based spatial reasoning. It constructs a graph of cities and towns with geodesic distances, retrieves a context subgraph either by vector similarity or by generating a SPARQL query, and prompts an LLM (GPT-4-0613) to answer distance questions. The authors evaluate on 60 self-constructed questions in three difficulty levels (Easy, Medium, Difficult) and report MSE against a baseline. They report 0 MSE for Easy and for the SPARQL variant on Medium, but Difficult questions are not solved by either variant; a templated SPARQL variant reaches 0 MSE only after receiving the query structure. An ablation studies abstention under graph sparsity.

Significance. If the Easy/Medium results are reproducible, DistRAG is a useful demonstration that a spatial graph can serve as a RAG knowledge store for simple distance lookup and nearest-neighbor questions, with the SPARQL variant providing exact answers. The paper includes a public code link, uses openly available OSM data, and includes an ablation for store sparsity. However, the claimed generality—'answers distance-based reasoning questions it otherwise cannot answer'—is not established for the Difficult subset, and the retrieval contribution is not quantified. The significance is therefore conditional on a narrowed claim and additional retrieval-recall analysis.

major comments (3)
  1. [Section 4, Table 1] The abstract and Section 1 claim that DistRAG 'enables an LLM to answer distance-based reasoning questions that it otherwise cannot answer.' On the Difficult dataset, neither variant supports this claim: DistRAG-SPARQL abstains on all 20 questions, DistRAG-Vector achieves MSE 8.34e7 (worse than the baseline's 4.11e7), and the text itself concedes that 'no method was able to achieve consistently.' The 0 MSE for Difficult is obtained only after supplying a template of the query structure, which is not part of DistRAG as defined in Section 2.2, is not shown in Table 1, and is not described in enough detail to reproduce. The supported claim is limited to Easy and Medium questions with the SPARQL retriever, and the paper should be revised accordingly.
  2. [Section 2.2 and Section 3.4] The method's success depends on the retriever placing every edge needed for the exact answer into the top-k context (for Easy, edge (A,B); for Medium, all edges incident to A; for Difficult, edge (A,B) and all edges incident to C), but no recall@k or accuracy-conditional-on-retrieval is reported. The hyperparameter k=10 is stated to be 'the best-performing setting' on the same 60 questions used for the main evaluation, so it is effectively selected on the test set. The ablation in Figure 2 reports only abstention rate as the graph is sparsified, which cannot distinguish retrieval failure from reasoning failure. The Medium vector-similarity result (MSE 3.04e3, about 55 km RMSE) shows that non-abstaining errors occur, but the cause is not analyzed. Without a retrieval-recall measurement, the method is indistinguishable from 'retrieve the answer into context.'
  3. [Section 3.1 and Section 2.2] The dataset consists of only 60 questions constructed by the authors, with one template per difficulty level, but no details are given on how the place pairs were sampled, how many unique cities are involved, whether the same cities appear across difficulty levels, or whether the graph is guaranteed to contain all needed edges. In addition, the size and average degree of the spatial graph are not reported, which is necessary to interpret the k=10 choice and the difficulty of the retrieval task. These omissions limit external validity and make it impossible to assess whether the graph contains the required edges by construction.
minor comments (5)
  1. [Table 1] The Difficult row for DistRAG-SPARQL reports a dash in the MSE column; define the dash (e.g., 'not applicable because the model abstained on all questions') in the table caption.
  2. [Figure 1 caption] The caption states 'Abstention is binned with errors > 700km', which conflates refusal to answer with large numeric errors; consider displaying abstentions as a separate category outside the residual histogram.
  3. [Prompt Template 2] The example RDF block appears truncated after 'Mount_Isa ;' and is not a syntactically complete RDF/Turtle snippet; ensure the template as printed is the exact template used in the experiments.
  4. [Section 4] The templated Difficult result described in prose should be reported as an explicit row in Table 1 with the template included in an appendix, or the claim should be removed from the abstract and conclusion.
  5. [Section 2.2] Reporting the number of nodes and edges in the spatial graph and the average node degree would help the reader judge whether k=10 is a reasonable context size for the retrieval task.

Circularity Check

1 steps flagged · score 3.0 of 10

The Difficult-item success is circular: the supplied template encodes the query structure, yet the paper credits the model with constructing the queries; Easy/Medium results are independent and non-circular.

  1. self definitional [Section 4 (Results), first paragraph; cf. Section 3.1 Difficult dataset definition]
    "After further providing DistRAG-SPARQL with a template of the query structure, it achieves 0 MSE, correctly answering every Difficult question, indicating that the model is capable of constructing the queries needed to answer the questions, but that it cannot perform the complex reasoning to arrive at the correct query structure on its own."

    The 0 MSE result is obtained only after the authors supply a template encoding the query structure, i.e., the two-hop reasoning (find distance A-B; compare C's neighbor distances) that Difficult questions require. The conclusion drawn — that the model is capable of constructing the queries needed to answer the questions — is the very capability the template supplies, so the measured success is the template's content by construction. The same sentence disclaims that capability ('cannot perform the complex reasoning to arrive at the correct query structure on its own'), confirming the structure was externally provided. The success thus cannot support the claimed query-construction capability.

full rationale

The central contribution is an empirical RAG comparison: a retriever selects distance triples from a geodesic-distance graph and supplies them to an LLM, and the LLM's answers are checked against ground-truth geodesic distances. For Easy and Medium questions the evidence is self-contained: Table 1 reports the base model failing (Easy MSE 1.01e5, Medium MSE 3.07e4) while both DistRAG variants achieve 0 MSE on Easy and DistRAG-SPARQL achieves 0 MSE on Medium. No fitted quantity is renamed as a prediction at this level; the method honestly places the answer or candidate set in context, so these results are not circular. The one genuine circularity is the Difficult templated result: the 0 MSE and the claimed query-construction capability are obtained by providing the template that encodes the reasoning the task requires, so the success reduces to the input (the template) by construction, and the paper's own sentence disclaims the capability it attributes to the model. This is a secondary, interpretive claim; the central Easy/Medium results do not depend on it. The self-citations to the authors' prior work ([11], [15], [16]) are present but not load-bearing, since [11]'s premise that LLMs fail at distance inference is corroborated by the paper's own baseline in Table 1, and [15]/[16] are motivational. Separate correctness risks, which are not derivation circularity, include the fact that k=10 was selected as the best-performing setting on the evaluation set, that retrieval recall@k is never reported so retrieval failure and reasoning failure are confounded, that the sparsity ablation measures abstention rate rather than accuracy, and that the templated Difficult run omits the template details entirely.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central method depends on four domain assumptions: the coordinate and distance source is accurate, vector retrieval finds relevant triples, the LLM follows the graph context, and SPARQL generation succeeds. There is one fitted quantity, k=10, tuned on the evaluation set. No new physical or conceptual entities are introduced.

free parameters (1)
  • k (number of retrieved triples) = 10
    Section 2.2 states k=10 was chosen as the best-performing setting, i.e., tuned on the same evaluation questions with no separate validation split.
assumptions (4)
  • domain assumption Distances in the OSM/OSMnx-derived graph are treated as ground truth geodesic distances.
    Section 2.2 constructs the spatial graph from OpenStreetMap and OSMnx; accuracy of the source is assumed, not independently verified.
  • domain assumption FAISS vector similarity between the prompt and graph triples retrieves the triples needed to answer the question.
    Section 2.2 Retriever; no retrieval recall analysis is reported, so this assumption is untested.
  • domain assumption Given a natural-language graph context, the LLM will extract the correct distance or city name.
    Prompt Templates 1 and 3; this is the core RAG assumption and is only indirectly tested.
  • domain assumption The SPARQL retriever can convert the natural-language question into an executable query for Easy and Medium cases, and with an added template for Difficult cases.
    Section 2.2 and Section 4; the Difficult case only works after the query structure is supplied.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DistRAG: Towards Distance-Based Spatial Reasoning in LLMs." pith.science (2026). https://pith.science/paper/NFUYD5VI

@misc{pith2026250603424,
  author       = {Pith},
  title        = {Pith review of: DistRAG: Towards Distance-Based Spatial Reasoning in LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NFUYD5VI}},
  note         = {Machine review of arXiv:2506.03424}
}
read the original abstract

Many real world tasks where Large Language Models (LLMs) can be used require spatial reasoning, like Point of Interest (POI) recommendation and itinerary planning. However, on their own LLMs lack reliable spatial reasoning capabilities, especially about distances. To address this problem, we develop a novel approach, DistRAG, that enables an LLM to retrieve relevant spatial information not explicitly learned during training. Our method encodes the geodesic distances between cities and towns in a graph and retrieves a context subgraph relevant to the question. Using this technique, our method enables an LLM to answer distance-based reasoning questions that it otherwise cannot answer. Given the vast array of possible places an LLM could be asked about, DistRAG offers a flexible first step towards providing a rudimentary `world model' to complement the linguistic knowledge held in LLMs.

Figures

Figures reproduced from arXiv: 2506.03424 by the authors.

Figure 1
Figure 1. Residual errors of DistRAG and baseline method [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Ablation Results. Response rate by question type for [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

21 extracted references · 11 canonical work pages

  1. [2]

    G. Boeing. 2024. Modeling and Analyzing Urban Networks and Amenities with OSMnx. (2024)

    work page 2024

  2. [3]

    Douze, A

    M. Douze, A. Guzhva, C. Deng, J. Johnson, G. Szilvasy, P.-E. Mazaré, M. Lomeli, L. Hosseini, and H. Jégou. 2024. The Faiss library. (2024). arXiv:2401.08281 [cs.LG]

    arXiv 2024

  3. [4]

    Dubey, S

    M. Dubey, S. Dasgupta, A. Sharma, K. Höffner, and J. Lehmann. 2016. AskNow: A Framework for Natural Language Query Formalization in SPARQL. InThe Seman- tic Web. Latest Advances and New Domains. Springer International Publishing, Cham, 300–316

    work page 2016

  4. [5]

    Fatemi, J

    B. Fatemi, J. Halcrow, and B. Perozzi. 2023. Talk like a Graph: Encoding Graphs for Large Language Models. arXiv:2310.04560 [cs.LG] https://arxiv.org/abs/2310. 04560

    arXiv 2023

  5. [6]

    Haklay and P

    M. Haklay and P. Weber. 2008. OpenStreetMap: User-Generated Street Maps.IEEE Pervasive Computing7, 4 (2008), 12–18. https://doi.org/10.1109/MPRV.2008.80

    doi:10.1109/mprv.2008.80 2008

  6. [7]

    Lewis, E

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t Yih, T. Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks.NeurIPS33 (2020), 9459–9474

    work page 2020

  7. [8]

    S. Li, Y. Gao, H. Jiang, Q. Yin, Z. Li, X. Yan, C. Zhang, and B. Yin. 2023. Graph reasoning for question answering with triplet retrieval.arXiv preprint arXiv:2305.18742(2023)

    arXiv 2023

  8. [9]

    M. Liu, X. Wang, J. Xu, and H. Lu. 2023. NALSpatial: An Effective Natural Language Transformation Framework for Queries over Spatial Data. InProc. of SIGSPATIAL(Hamburg, Germany). New York, NY, USA, Article 57, 4 pages. https://doi.org/10.1145/3589132.3625600

    arXiv 2023

Show all 21 references
  1. [10]

    Manvi, S

    R. Manvi, S. Khanna, G. Mai, M. Burke, D. B. Lobell, and S. Ermon. 2024. GeoLLM: Extracting Geospatial Knowledge from Large Language Models. InProc. of ICLR

    2024

  2. [11]

    O’Sullivan, N

    K. O’Sullivan, N. R. Schneider, and H. Samet. 2024. Metric Reasoning in Large Language Models. InProc. of SIGSPATIAL(Atlanta, USA)

    2024

  3. [12]

    B. Peng, Y. Zhu, Y. Liu, X. Bo, H. Shi, C. Hong, Y. Zhang, and S. Tang. 2024. Graph retrieval-augmented generation: A survey.arXiv preprint arXiv:2408.08921 (2024)

    2024 arXiv

  4. [13]

    T. T. Procko and O. Ochoa. 2024. Graph retrieval-augmented generation for large language models: A survey. In2024 Conference on AI, Science, Engineering, and Technology (AIxSET). IEEE, 166–169

    2024

  5. [14]

    J. Qi, Z. Li, and E. Tanin. 2023. MaaSDB: Spatial Databases in the Era of Large Language Models (Vision Paper). InProc. of SIGSPATIAL(Hamburg, Germany). New York, NY, USA, Article 54, 4 pages. https://doi.org/10.1145/3589132.3625597

    2023

  6. [15]

    N. R. Schneider, K. O’Sullivan, and H. Samet. 2024. The Future of Graph-based Spatial Pattern Matching (Vision Paper). In40th IEEE International Conference on Data Engineering, ICDE 2024 – SEAGraph Workshop. Utrecht, Netherlands

    2024

  7. [16]

    N. R. Schneider, N. Ramachandran, K. O’Sullivan, and H. Samet. 2025. Engineering Prompts for Spatial Questions. InCompanion Proc. of WWW(Sydney, NSW, Australia). 2 pages. https://doi.org/10.1145/3701716.3717807

    2025

  8. [17]

    P. Sen, S. Mavadia, and A. Saffari. 2023. Knowledge graph-augmented language models for complex question answering. InProc. of the 1st Workshop on Natural Language Reasoning and Structured Explanations (NLRSE). 1–8

    2023

  9. [18]

    Shaik, P

    S. Shaik, P. Kanakam, S. M. Hussain, and D. Suryanarayana. 2016. Transforming natural language query to SPARQL for semantic information retrieval.Interna- tional Journal of Engineering Trends and Technology7 (2016), 347–350

    2016

  10. [19]

    Y. Wu, N. Hu, S. Bi, G. Qi, J. Ren, A. Xie, and W. Song. 2023. Retrieve-rewrite- answer: A kg-to-text enhanced llms framework for knowledge graph question answering.arXiv preprint arXiv:2309.11206(2023)

    2023 arXiv

  11. [20]

    R. Ye, C. Zhang, R. Wang, S. Xu, Y. Zhang, et al. 2023. Natural language is all a graph needs.arXiv preprint arXiv:2308.071344, 5 (2023), 7

    2023 arXiv

  12. [21]

    X. Yin, D. Gromann, and S. Rudolph. 2021. Neural machine translating from natural language to SPARQL.Future Generation Computer Systems117 (2021), 510–519

    2021

  13. [22]

    D. Yu, R. Bao, G. Mai, and L. Zhao. 2025. Spatial-rag: Spatial retrieval aug- mented generation for real-world spatial reasoning questions.arXiv preprint arXiv:2502.18470(2025)

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.