Pith. sign in

REVIEW 3 major objections 5 minor 17 references

MultiGlobeQA: A Multilingual and Globally Diverse Benchmark for Geospatial Reasoning

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Given gold triples and compute tools, no evaluated large language model exceeds 61.6% exact match on MultiGlobeQA's geospatial questions, and grid indexing stays below 13%.

desk verdict MultiGlobeQA is a genuinely useful, well-built benchmark, but the headline 'computation is the bottleneck' claim is stronger than the oracle design and the income-gap analysis actually support. read the letter →

arxiv 2608.03882 v1 pith:MLSXHPW6 submitted 2026-08-04 cs.CL cs.AIcs.IR

Martin Böckling , Elizaveta Nosova , Heiko Paulheim , Andreea Iana This is my paper
classification cs.CLcs.AIcs.IR
keywords geospatialreasoningbenchmarkmultilingualevaluationknowledgegraphsspatialindexingexecution-basedgroundtruthLLMagentic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MultiGlobeQA is a multilingual benchmark of 46,060 question-answer pairs whose answers are computed by executing query templates over three knowledge graphs, so correctness is verified by construction rather than judged by a model. The paper uses it to ask whether large language models fail geospatial questions because they lack geographic knowledge or because they cannot compute over it. Across four models in parametric, reasoning, and agentic settings, the answer is computation: injecting the exact gold triples and giving the model a Python interpreter caps exact match at 61.6%, with grid-indexing questions below 13%. Retrieval and tools produce large gains, but they cannot close the gap, and low-income regions still lag even under perfect retrieval.

What carries the argument

The load-bearing mechanism is the template-executor pipeline: 65 typed templates, expanded into 129 sub-templates and 315 natural-language paraphrases, are each paired with a hand-written query executed over WorldKG, KnowWhereGraph, or OSMH3KG, so every gold answer is produced by a query engine and every QA pair is execution-verified. The second mechanism is the oracle condition—injecting the supporting gold triples into the prompt or into a tool-using agent's context—which cleanly separates whether a failure comes from missing knowledge or from computation over the evidence.

What would settle it

Re-run MultiGlobeQA's count, set-enumeration, and nearest-entity queries against the same knowledge graphs updated by one year, and score models that retrieve live data against the new gold. If accuracy rises substantially on those families, the plateau partly reflects stale or incomplete snapshots rather than pure computational failure.

Watch

Extended reading notes

Core claim

The paper's central finding is that access to the right facts is not the limiting factor in geospatial reasoning for current LLMs. In oracle conditions where the supporting gold triples are injected directly, no evaluated model exceeds 61.6% exact match even when it can write and execute Python; most models fall well short, and H3/S2 grid-indexing questions stay below 13%. Because these conditions eliminate retrieval failure, the residual errors are attributed to executing geometric and topological computation over the evidence—distances, containment, shape, and spatial indexing—rather than to missing parametric knowledge. The benchmark itself is constructed so that every gold answer is prod

Load-bearing premise

The result assumes the gold triples injected in oracle conditions are complete and correct enough to compute each answer, yet the paper notes snapshots can miss entities or contain wrong geometries, so some answers marked wrong may actually be right about the real world.

Editorial extensions

If this is right

  • Geospatial QA systems should stop expecting token prediction to perform spatial arithmetic and instead delegate to deterministic geographic libraries, treating the LLM as planner and parser.
  • Agentic geospatial evaluation should report oracle-condition accuracy alongside live retrieval, or gains from better search will be conflated with gains from better computation.
  • Grid-indexing and shape families are the diagnostic floor: models that cannot handle H3/S2 cell arithmetic should not be assumed ready for navigation, logistics, or disaster-response deployment.
  • Perfect retrieval does not fix regional equity: lower-income regions remain harder even with gold facts, so benchmark scores need per-region reporting rather than aggregate claims.
  • Accuracy is stable across the 17 languages, indicating that for this benchmark cross-lingual performance is mostly a function of English tooling and entity names, not of the question language itself.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension of the computation-bottleneck claim: replace raw Python with a fixed GIS function library (pre-bound distance, geohash, and H3 APIs) and measure whether the 61.6% ceiling rises; if it does, the bottleneck is partly code generation, not arithmetic.
  • The sharp contrast between geohash (54.5) and H3/S2 (16.2) under gold facts suggests a predictable ordering of failure by spatial-index complexity, which could be probed with a smaller set of discrete encodings.
  • Because the injected oracle contexts carry about twice as many gold triples for high-income questions, the widening low-income gap may partly reflect evidence volume rather than regional knowledge; controlling triple count per question would test this.
  • The 61.6% ceiling under perfect retrieval is a natural calibration target: if models trained on synthetic spatial-computation traces exceed it, that would confirm the bottleneck is trainable rather than architectural.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces MultiGlobeQA, a multilingual benchmark of 46,060 geospatial question-answer pairs derived from three knowledge graphs via 65 templates and 14 spatial-function families, with execution-based ground truth, stratified sampling over income and density tiers, 17 language versions, and false-premise and multimodal slices. The authors evaluate four LLMs under parametric (T1), reasoning (T2), agentic retrieval (T3a/T3b/T3c), and oracle conditions with injected gold triples (T1-o/T3-o). The central claims are that retrieval and tools help substantially but that performance plateaus below two thirds even under gold facts, indicating a computation bottleneck, and that model accuracy is worse for low-income regions, a gap that gold facts widen. The construction is careful in many respects, but the headline oracle and income-gap results are currently entangled with confounds that the manuscript itself acknowledges.

Significance. If the central claim is established, the paper makes a timely and important contribution: it provides a large, multilingual, execution-grounded benchmark and shows that current LLMs cannot reliably perform geospatial computation even under near-perfect retrieval with code execution. The benchmark construction has notable strengths: execution-verified ground truth, human post-edited translations with inter-annotator agreement reported, released gold triples for auditability, per-seed variance, and honest reporting of non-replications such as the density-tier result. The per-family findings—especially the collapse on grid indexing and shape computation—are likely robust and valuable. However, the headline quantitative ceiling ('no model exceeds 61.6 EM', 'plateaus below two thirds') and the income-gap claim are not cleanly supported as stated, because the oracle is not closed-world for exhaustive-answer templates, and the income comparison is confounded by triple-count imbalance. These are fixable with additional analysis or rephrasing, so the paper merits major revision rather than rejection.

major comments (3)
  1. [§4 (oracle conditions) and Limitations] The oracle tiers inject only positive gold triples that 'support' each answer. For set-enumeration (759 items in the small split), count (289), and nearest/optimization/ranking templates under SFIDs A/F/G, correctness requires knowing that no unlisted entity satisfies the query. The prompts in Figs. 29/33 assert 'all the information you need is already provided' but never state a closure or exhaustiveness condition, and refusals are scored incorrect (§4). The Limitations explicitly concede that incompleteness affects counts, set enumerations, and nearest-entity questions. Consequently, the 61.6 EM ceiling conflates computational failure with under-specification for roughly a fifth of the small-split questions. Please either add explicit closure statements to the oracle contexts (e.g., 'the listed candidates are exhaustive') or recompute the ceiling on the subset of templates for which th
  2. [§5.3 (Fig. 3) and Abstract] The abstract's claim that gold facts widen the low-income gap is not identifiable from the presented comparison. §5.3 states that the injected oracle contexts carry on average about twice as many gold triples for high-income as for low-income questions. Since T1-o/T3-o accuracy can depend on evidence volume and context length, the widening under oracle conditions may be an artifact of triple-count imbalance rather than of computation or regional familiarity. Please control for the number and length of injected triples (e.g., matched triple-count subsets or per-triple accuracy) or explicitly rephrase the headline to acknowledge this confound. As written, a central advertising result is undermined by the paper's own analysis.
  3. [§5.1 and Table 14] The statement that 'no model exceeds 61.6 EM' and the abstract's 'plateaus below two thirds' are contradicted within the same section: excluding context-window overflow, Qwen3.5-35B reaches 68.4 at T3-o. Because the oracle condition overflows the context window on 8.9–19.6% of open-weight questions, the intent-to-treat ceiling partly measures context capacity, not computational ability. The text should clearly separate intent-to-treat from per-protocol results, or the headline should be re-estimated on questions whose contexts fit. The qualitative conclusion may survive, but the specific 61.6 / two-thirds bound does not follow without this caveat.
minor comments (5)
  1. [§3.5 / Appendix B.2 (Table 9)] The negative Krippendorff alpha for French is explained only in a table footnote; given its relevance to translation reliability, a one-sentence note in the main text would be appropriate.
  2. [Fig. 1 caption] The caption contains 'hatched-obars', which appears to be a typo for 'hatched oracle bars'; please clarify the oracle-condition symbology.
  3. [Table 4] The note says 'EM above baseline is in bold', but the table as rendered does not show bold formatting. Please ensure the final PDF/HTML preserves the intended emphasis.
  4. [§5.3] The phrase 'the highest is a middle tier' is vague; please name the specific tier(s) per model or refer more precisely to Fig. 3 so readers can verify the ordering.
  5. [Table 12] The tolerance floors (e.g., δ = 0.5/5/10 km for distances, δ = 5 km/200 km for coordinates) are central to EM, but their motivation is not discussed. A sentence or a small sensitivity check would strengthen the metric's credibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central result is execution-grounded and the oracle controls are not tautological.

full rationale

MultiGlobeQA's central claim—that LLMs plateau below two-thirds EM even with gold triples and compute tools—is an empirical evaluation result, not a derived prediction. Ground truth is generated by executing query templates over three knowledge graphs (§3.3), so answers are not fitted to model outputs. The oracle conditions inject 'the gold KG triples supporting each answer' (§4) while withholding retrieval, which is a designed control for perfect retrieval, not an instance of renaming the target as an input. No parameter is fitted to the benchmark, no target quantity is defined in terms of the measured outcome, and no uniqueness theorem or ansatz is imported from the authors' prior work. The only self-citation is the use of OSMH3KG (Böckling et al., 2024) as one of three data sources; it is a published external KG and is not the sole or load-bearing source for the headline result, which is averaged across all three KGs and replicated per model and per spatial family. The paper's own Limitations explicitly concede that 'incompleteness affects questions whose answer ranges over a set of entities, such as counts, set enumerations, and nearest-entity questions' and that ground truth is 'correct with respect to the snapshot rather than the world.' This is a validity caveat about whether the oracle is truly perfect for exhaustive-set questions; it does not make the evaluation circular, because the gold answer is still computed from the injected triples rather than being equivalent to them by construction. Accordingly, no circular step can be exhibited with a quoted reduction, and the honest finding is no significant circularity.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claims rest on hand-chosen scoring tolerances, on the assumption that execution over KG snapshots gives correct ground truth, and on the sufficiency of injected oracle triples. No new physical or scientific entities are introduced.

free parameters (2)
  • Scoring tolerances tau_i = alpha=0.05; delta=0.5/5/10 km for distances, 0.25/25/100 km^2 for areas; IoU>=0.5; coordinate delta=5 with d_scale=200 k
    Hand-chosen thresholds in the EM/NE scoring functions (Table 12). They define what counts as correct and directly set the reported accuracy numbers; different tolerances would change all headline EM values.
  • Sampling caps for strata cells and small split = 30 entity tuples per (template, KG, stratum); 200 per (KG, SFID) cell in the small split
    These caps shape benchmark composition. The paper itself shows the small split's density composition differs from the large split and density-tier gaps fail to replicate (Appendix G.6). They are design choices, not external constraints.
assumptions (3)
  • domain assumption Gold answers computed by executing queries over fixed KG snapshots are the correct ground truth
    The benchmark validates itself by execution over snapshots; if a snapshot is incomplete or a geometry is wrong, gold answers propagate errors. The paper states this in the Limitations section.
  • domain assumption The injected gold triples in oracle conditions are sufficient to compute the answer
    T1-o and T3-o prompts say 'all the information you need is already provided above as structured triples'. If any template requires a fact not in the injected triples (e.g., country borders in nested-country templates), failures in the oracle would be misattributed to computation. The executor built the triples, so this is plausible but not proven for every template.
  • domain assumption LLM post-editing and human-verified translations preserve spatial meaning across the 17 languages
    Cross-lingual comparisons are only valid if translations preserve the spatial relation. Annotator agreement is 69.7% with LLM arbitration; translation errors would introduce language-specific noise.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MultiGlobeQA: A Multilingual and Globally Diverse Benchmark for Geospatial Reasoning." pith.science (2026). https://pith.science/paper/MLSXHPW6

@misc{pith2026260803882,
  author       = {Pith},
  title        = {Pith review of: MultiGlobeQA: A Multilingual and Globally Diverse Benchmark for Geospatial Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MLSXHPW6}},
  note         = {Machine review of arXiv:2608.03882}
}
read the original abstract

Geospatial reasoning, i.e., computing distances, containment, and other spatial relations over real-world entities, is central to navigation and logistics, yet large language models (LLMs) struggle with the required geometric and topological computation despite storing considerable geographic knowledge. Existing benchmarks localize these failures only partially: they are synthetic or smallscale, largely monolingual, and offer limited control over geographic coverage. We introduce MultiGlobeQA, a multilingual benchmark of 46,060 question-answer pairs spanning 14 spatial-function families and 15 answer formats, with execution-based ground truth over three knowledge graphs. It covers 201 countries and territories via income- and density-stratified sampling, with parallel questions in English and 16 additional high- and low-resource languages. Across parametric, reasoning, and agentic settings, LLMs collapse on tasks requiring grid indexing and shape computation, while topological relations and directions fare best. Retrieval and tool use yield considerable gains, yet performance plateaus below two thirds even when gold facts are supplied, indicating that computation, not access to knowledge, is the bottleneck. Models also underperform on low-income regions, a gap that gold facts widen rather than close.

Figures

Figures reproduced from arXiv: 2608.03882 by the authors.

Figure 1
Figure 1. LLMs systematically fail on questions requiring geospatial reasoning and computation. Top: three example items (distance, direction, geohash), each with its executable gold answer and the models’ predictions (✔ correct, ✘ wrong). + gold triples denotes an oracle setting with perfect retrieval, where the relevant facts are injected into the prompt. Bottom: exact match (%) averaged over four LLMs per category, across … view at source ↗
Figure 2
Figure 2. Accuracy (EM) by spatial function (a) and answer format (b) across evaluation conditions (small split, English, text modality; averaged over the four models). Parentheses give the number of true-premise questions in each family; the T3-o value is printed at the right of each row. We omit Date as it covers only 8 questions (b). (Qwen3.5-35B) at T3a, as the same query can suc￾ceed or exhaust its budget depending on wh… view at source ↗
Figure 3
Figure 3. Accuracy by income tier (small split, En￾glish, text modality; standardized EM). under gold context, H3 and S2 cell indices only 16.2, and their composition is never correct in any condition. Geohash interleaves latitude and lon￾gitude bits deterministically, whereas H3 and S2 require projection onto an icosahedral or spherical￾cube grid and hierarchical cell arithmetic. Grid indexing ranks last for the same reason:… view at source ↗
Figures from the paper (30 more)
Figure 4
Figure 4. Figure 4: Geographic density of geometries over (a) KnowWhereGraph (KWG), (b) WorldKG, and (c) OSMH3KG. [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]
Figure 5
Figure 5. Figure 5: Global distribution of MultiGlobeQA entities. 290,086 distinct entity sites across 201 countries and territories, colored by World Bank income tier. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Distribution of MultiGlobeQA questions over the 15 answer formats. Annotation Workflow. We use the Potato anno￾tation tool (Pei et al., 2022). The annotation task comprises six phases. (1) Welcome and Consent. The welcome page sum￾marizes the project, task, and expecte…
Figure 7
Figure 7. Figure 7: Resolution pipeline outcomes across languages. Left: resolution source over all items. Right: outcomes for LLM-arbitrated cases only. reason from the fixed taxonomy in [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Pairwise human-annotator agreement on translation post-editing, per target language. Two annotators independently labeled every translated question as Correct (no edit needed) or non-Correct, supplying an edited string in the latter case. comma-below distinction in Rom…
Figure 9
Figure 9. Figure 9: Normalized multi-select correction￾reason distributions per language. Phrasing-level reasons (left block): Word=word choice/vocabu￾lary; Gram.=grammar/syntax; Reg.=register/formality; W.O.=word order/sentence structure. Incorrect-level rea￾sons (right block): Mean.=mea…
Figure 10
Figure 10. Figure 10: False-refusal rate against spatial-function family difficulty (small split, English, true-premise questions, three seeds). Difficulty is measured as the accuracy that family permits under gold triples (EMcov at T3-o, averaged over models). Lines are least-squares fits…
Figure 11
Figure 11. Figure 11: Oracle ablation (T1) (small split, English, text modality; EM over true-premise questions, refusals counted incorrect; mean ± std. dev. over three seeds). weights LLMs, by 8.6 EM points over structured JSON for Qwen3.5-27B and 4.1 for Gemma-3-27B￾Instruct. The choice …
Figure 12
Figure 12. Figure 12: Accuracy vs. compute cost (small split, English, text modality; EM over true-premise questions against median tokens per question, log scale). Color denotes model, shape denotes tier. Arrows run from T3a to T3-o. source at 30k-88k median tokens, and adding it to the K…
Figure 13
Figure 13. Figure 13: Normalized error (NE) by spatial function (a) and answer format (b) across evaluation conditions (small split, English, text modality; averaged over the four models). Parentheses give the number of true-premise questions in each family. G.6 Income & Density Equity [P…
Figure 14
Figure 14. Figure 14: Accuracy (EM) by spatial function and model across evaluation conditions (small split, English, text modality). Parentheses give the number of true-premise questions in each family. Model T1 T2 T1-o T3a T3-o Qwen3.5-27B 7.7 5.0 19.2 25.2 50.6 Qwen3.5-35B 3.5 0.0 2.6 1…
Figure 15
Figure 15. Figure 15: Normalized error (NE) by spatial function and model across evaluation conditions (small split, English, text modality). Parentheses give the number of true-premise questions in each family. • $triple_context: the injected gold KG facts, for the oracle conditions; • $m…
Figure 16
Figure 16. Figure 16: Accuracy (EM) by answer format and model across evaluation conditions (small split, English, text modality). Parentheses give the number of true-premise questions in each family. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_16.png]
Figure 17
Figure 17. Figure 17: Normalized error (NE) by answer format and model across evaluation conditions (small split, English, text modality). Parentheses give the number of true-premise questions in each family 29 [PITH_FULL_IMAGE:figures/full_fig_p029_17.png]
Figure 18
Figure 18. Figure 18 [PITH_FULL_IMAGE:figures/full_fig_p030_18.png]
Figure 19
Figure 19. Figure 19: Accuracy (EM) by income and density tier on the large split (English, text modality, T1, one seed). 30 [PITH_FULL_IMAGE:figures/full_fig_p030_19.png]
Figure 20
Figure 20. Figure 20: Premise detection against indiscriminate abstention (small split, English, text modality). Color denotes model, shape denotes tier. False refusal is measured on answerable questions from the 13 templates that carry false premises, not on the whole benchmark. The dotte…
Figure 21
Figure 21. Figure 21: Per-language accuracy relative to English (small split, text modality; EM over true-premise questions). 31 [PITH_FULL_IMAGE:figures/full_fig_p031_21.png]
Figure 22
Figure 22. Figure 22: System prompt shared across the post-editing LLM ensemble. The prompt is abbreviated: we omit the [PITH_FULL_IMAGE:figures/full_fig_p032_22.png]
Figure 23
Figure 23. Figure 23: System prompt shared across the post-editing LLM ensemble. [PITH_FULL_IMAGE:figures/full_fig_p033_23.png]
Figure 24
Figure 24. Figure 24: User prompt for the post-editing LLM ensemble. [PITH_FULL_IMAGE:figures/full_fig_p034_24.png]
Figure 25
Figure 25. Figure 25: Expected JSON output schema for the LLM ensemble models. [PITH_FULL_IMAGE:figures/full_fig_p034_25.png]
Figure 26
Figure 26. Figure 26: Tier 1 (parametric) prompt. Prompt: Tier 1 oracle raw You are answering a spatial question about real-world locations and entities. You have been given the relevant RDF triples from a spatial knowledge graph. Relevant facts: $triple_context Question: $question Expecte…
Figure 27
Figure 27. Figure 27: Tier 1 oracle (raw) prompt. 34 [PITH_FULL_IMAGE:figures/full_fig_p034_27.png]
Figure 28
Figure 28. Figure 28: Tier 1 oracle (structured) prompt. Prompt: Tier 1 oracle verbalized You are answering a spatial question about real-world locations and entities. You have been given the relevant facts from a spatial knowledge graph. Relevant facts: $triple_context Question: $question…
Figure 29
Figure 29. Figure 29: Tier 1 oracle (verbalized) prompt. Prompt: Tier 2 (reasoning) You are answering a spatial question about real-world locations and entities. Question: $question Expected answer: $answer_type_description$unit_clause $distance_clause Think step by step. Reason through th…
Figure 30
Figure 30. Figure 30: Tier 2 (reasoning) prompt. 35 [PITH_FULL_IMAGE:figures/full_fig_p035_30.png]
Figure 31
Figure 31. Figure 31: Tier 3a (agentic) prompt. T3b/T3c differ only in the first-sentence retrieval source: [PITH_FULL_IMAGE:figures/full_fig_p036_31.png]
Figure 32
Figure 32. Figure 32: Agent tools for Tier 3. 37 [PITH_FULL_IMAGE:figures/full_fig_p037_32.png]
Figure 33
Figure 33. Figure 33: Tier 3 oracle prompt. 38 [PITH_FULL_IMAGE:figures/full_fig_p038_33.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

17 extracted references · 14 canonical work pages

  1. [1]

    A correction that renames, merges, splits, or loses a placeholder is worse than one that keeps them intact

    Placeholder integrity:every placeholder must appear verbatim, matching the count and iden- tity in the English template. A correction that renames, merges, splits, or loses a placeholder is worse than one that keeps them intact. 12We choose a frontier closed-source model to reduce training-data overlap with the models under evaluation

    work page

  2. [2]

    what/where) must match the English; the cor- rection preserving meaning more faithfully is preferred

    Semantic fidelity:the spatial relation, quantifier, and question type (yes/no vs. what/where) must match the English; the cor- rection preserving meaning more faithfully is preferred

    work page

  3. [3]

    Grammatical correctness:agreement, case, verb form, and word order must be valid in the target language; the correction with fewer grammar errors is preferred

    work page

  4. [4]

    Technical terminology:for domain terms (e.g., centroid, variance, fractal dimension), the cor- rection using established vocabulary over ap- proximations is preferred

    work page

  5. [5]

    GS-QA: A Benchmark for Geospatial Question Answering

    ‘smolagents‘: a smol library to build great agentic systems. https://github.com/ huggingface/smolagents. Majid Saeedan, Muhammad Shihab Rashid, Ahmed El- dawy, and Vagelis Hristidis. 2026. GS-QA: A Bench- mark for Geospatial Question Answering.arXiv preprint arXiv:2605.22811. Zhengxiang Shi, Qiang Zhang, and Aldo Lipani. 2022. StepGame: A New Benchmark fo...

    work page Pith review arXiv 2026

  6. [6]

    Earthquake from USGS Earthquake Catalog with ID pr2020052043

    STaRK: Benchmarking LLM Retrieval on Tex- tual and Relational Knowledge Bases.Advances in Neural Information Processing Systems, 37:127129– 127153. Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. 2024. TravelPlanner: A Benchmark for Real-World Plan- ning with Language Agents. InProceedings of the 41st In...

    work page 2024

  7. [10]

    User Prompt Template.Fig

    Placeholders such as {{entity_A}} mark proper nouns to be preserved verbatim. User Prompt Template.Fig. 24 shows the per-question user prompt, specifying the target- language ISO 639-3 code, the original English tem- plate, and the instantiated question. Expected Output Scheme.Each model returns strict JSON formatted to the required number of ed- its (25)...

    work page 2023

  8. [11]

    Natural phrasing:among otherwise-equal options, the more idiomatic phrasing is pre- ferred. To prevent spurious choices, the prompt disallows four dimensions as decision drivers: (i) gender and case agreement hard-coded on a placeholder (these vary with the entity filler and are handled downstream by the post-editing LLM ensemble), (ii) whitespace and tri...

    work page

Show all 17 references
  1. [13]

    A correction that renames, merges, splits, or loses a placeholder is worse than one that keeps them intact

    **Placeholder integrity.** Placeholders must be present verbatim, in matching quantity to the English template. A correction that renames, merges, splits, or loses a placeholder is worse than one that keeps them intact

  2. [14]

    Prefer the correction that preserves meaning more faithfully

    **Semantic fidelity.** The spatial relation, quantifier, and question type (yes/no vs wh-) must match the English. Prefer the correction that preserves meaning more faithfully

  3. [15]

    The correction with fewer grammar errors wins

    **Grammatical correctness.** Agreement, case, verb form, and word order must be valid {target_language}. The correction with fewer grammar errors wins

  4. [16]

    **Technical terminology.** For domain terms (centroid, variance, fractal dimension, perimeter, etc.), prefer the correction that uses established technical vocabulary over approximations

  5. [17]

    Both are bad

    **Natural phrasing.** Among otherwise-equal options, prefer the more idiomatic phrasing a native speaker would write. # WHAT TO IGNORE - **Gender or case agreement that depends on the entity that fills a placeholder.** Do not prefer one correction solely because it adds a gend...

  6. [197]

    Sergios-Anestis Kefalidis, Dharmen Punjani, Eleni Tsalapati, Konstantinos Plas, Maria-Aggeliki Pol- lali, Pierre Maret, and Manolis Koubarakis

    Springer. Sergios-Anestis Kefalidis, Dharmen Punjani, Eleni Tsalapati, Konstantinos Plas, Maria-Aggeliki Pol- lali, Pierre Maret, and Manolis Koubarakis. 2024. The question answering system GeoQA2 and a new benchmark for its evaluation.International Journal of Applied Earth Ob...

    2024 arXiv

  7. [2023]

    Johannes Hoffart, Fabian M Suchanek, Klaus Berberich, and Gerhard Weikum

    A spatio-temporal analysis investigating com- pleteness and inequalities of global urban building data in OpenStreetMap.Nature Communications, 14(1):3985. Johannes Hoffart, Fabian M Suchanek, Klaus Berberich, and Gerhard Weikum. 2013. YAGO2: A spa- tially and temporally enhanc...

    2013

  8. [2024]

    In GeoLD2024: 6th Geospatial Linked Data Workshop

    A Planet Scale Spatial-Temporal Knowledge Graph Based On OpenStreetMap And H3 Grid. In GeoLD2024: 6th Geospatial Linked Data Workshop. Isaac Brodsky. 2018. H3: Uber’s Hexagonal Hierarchi- cal Spatial Index. https://www.uber.com/blog/ h3/. Uber Engineering Blog. Zijian Chen, Xu...

    2018 arXiv

  9. [2025]

    InPro- ceedings of the 2025 Conference on Empirical Meth- ods in Natural Language Processing, pages 28797– 28809

    XLQA: A Benchmark for Locale-Aware Mul- tilingual Open-Domain Question Answering. InPro- ceedings of the 2025 Conference on Empirical Meth- ods in Natural Language Processing, pages 28797– 28809. Aymeric Roucher, Albert Villanova del Moral, Thomas Wolf, Leandro von Werra, and ...

    2025

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.