Pith. sign in

REVIEW 2 major objections 4 minor 66 references

A Vision for Geo-Temporal Deep Research Systems: Towards Comprehensive, Transparent, and Reproducible Geo-Temporal Information Synthesis

T0 review · 2 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that deep research systems must be rebuilt to handle geo-temporal constraints.

desk verdict A clear, well-written vision for geo-temporal deep research, but the boundary between text synthesis and spatial computation is undefined and untested. read the letter →

arxiv 2506.14345 v1 pith:KE22SPZU submitted 2025-06-17 cs.CL cs.IR

Bruno Martins , Piotr Szymański , Piotr Gramacki This is my paper
classification cs.CLcs.IR
keywords AgenticAIDeepResearchGeo-TemporalInformationRetrievalTemporalGeographicEvaluationProtocolsReproducibility
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that today's deep research systems — LLM agents that plan iterative searches and write report-style answers — cannot properly answer questions constrained by place and time, and that fixing this is a distinct research agenda. The authors claim that adding geo-temporal capabilities requires changes across query generation, retrieval, reranking, and synthesis, plus open search infrastructure and evaluation protocols that measure geo-temporal relevance and diversity. If the vision is right, future deep research systems will need specialized geo-temporal components instead of treating geography and time as incidental keywords.

What carries the argument

The architectural proposal is a pipeline in which query generation constructs search queries with explicit location and time constraints; retrieval and reranking prioritize documents by spatial proximity and containment and by temporal relations; and synthesis parses and disambiguates geo-temporal cues through geocoding, gazetteers, and temporal-expression normalization, optionally calling external tools. The whole system can be tuned end-to-end with reinforcement learning using task-specific metrics. The boundary condition carrying the argument is that the system is a text-based synthesis engine, not an automated GIS workbench.

What would settle it

A concrete test: assemble a set of realistic geo-temporal questions whose answers require quantitative spatial operations, such as estimating pollution exposure at an unmonitored point from nearby station readings or computing area overlap between a flood polygon and a neighborhood boundary. If a text-only deep research pipeline with the proposed architecture produces measurably wrong or unsupported answers on such a set while a GIS-augmented baseline succeeds, the paper's scope boundary fails.

Watch

Extended reading notes

Core claim

The central claim is that current deep research systems lack the geo-temporal capabilities essential for context-rich questions involving geographic or temporal constraints, which occur frequently in public health, environmental science, and socio-economic analysis. The paper proposes that augmenting retrieval and synthesis with geo-temporal constraint handling, supported by open and reproducible infrastructure and rigorous evaluation, is the necessary path toward more advanced systems. It explicitly distinguishes this goal from automating GIS-style spatial computation: the aim is orchestrated retrieval, reasoning, and synthesis from textual documents, producing report-style answers that could later feed external GIS workflows.

Load-bearing premise

The load-bearing premise is that geo-temporal deep research can be delivered through text retrieval and synthesis alone, without automating quantitative spatial computation; if real questions require interpolation, map-based reasoning, or spatial statistics as core capabilities, the proposed architecture is incomplete.

Editorial extensions

If this is right

  • Deep research systems will need query generation that plans around place names, geographic footprints, dates, intervals, and diversity across geo-temporal dimensions.
  • Retrieval and reranking must become geo-temporally aware, using proximity, containment, and temporal ordering rather than topical similarity alone.
  • Synthesis components must parse and disambiguate geo-temporal cues, likely with external tools such as geocoders, gazetteers, and temporal reasoning libraries.
  • Progress depends on open, stable search infrastructures, because commercial Web search APIs are opaque and their evolving results undermine reproducibility.
  • Evaluation needs new benchmarks and adapted metrics, for example LLM-as-a-Judge extended to score geo-temporal relevance, coherence, and diversity.
  • Relying on text-only synthesis means the system cannot produce quantitative spatial answers, such as interpolations or map-based measurements, on its own.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If this vision is adopted, it would merge geographic and temporal information retrieval research with agentic LLM systems, giving a clear use case for retrieval models that follow instructions about location and time.
  • The text-only boundary may not hold: a testable extension would compare answer quality on questions whose evidence lives in maps, satellite imagery, or tabular statistics rather than prose.
  • Benchmark design could start by filtering existing complex-question datasets for geo-temporal constraints, but long-form geo-temporal ground truth will require new annotation schemes; a small pilot on health-policy questions across European regions would be a concrete first step.
  • Using LLMs as judges of geo-temporal relevance introduces the judge's own geo-temporal biases as a confound, so judge calibration against human geo-temporal judgments needs separate study.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. This position paper argues that current LLM-powered deep research systems lack the geo-temporal capabilities needed for many real-world information needs, and it outlines a research vision for integrating geographic and temporal constraint handling into agentic retrieval and synthesis pipelines. The paper contributes a taxonomy of complex geo-temporal question types, a high-level architecture (Figure 1) comprising query generation, retrieval, re-ranking, and summarization modules, a set of infrastructure requirements favoring open and reproducible search/retrieval environments, and a discussion of evaluation challenges and possible adaptations of existing benchmarks and metrics. The authors explicitly delimit their scope to text-based retrieval and synthesis, excluding automation of GIS-style spatial computation, and they frame the paper as a call for a research agenda rather than as a report of implemented systems or experiments.

Significance. If the vision is adopted, the paper could serve as a useful organizing framework for a community effort to make LLM-based research systems spatially and temporally aware. Its strengths are the concrete typology of geo-temporal questions, the attention to reproducibility problems introduced by commercial search APIs, the connection of established GIR/TIR techniques to recent agentic deep research systems, and the identification of evaluation gaps. The paper is an agenda-setting work with no empirical validation; it contains no circular reasoning, no fitted parameters, and no invented entities. Its main value is in articulating research directions, not in demonstrating feasibility, and the load-bearing scope decision regarding quantitative spatial computation is internally inconsistent, as detailed below.

major comments (2)
  1. [§2.1, §2.2, Introduction] The paper's central scope decision is internally inconsistent and load-bearing. Section 2.1 states that the focus 'is not on automating GIS-style spatial analysis – including code generation or tool execution for complex spatial computations,' while Section 2.2 requires summarization components to perform 'lightweight GIS operations—like aggregating counts or interpolating values over spatial regions,' and the Introduction lists 'computation of geo-spatial statistics, aggregations, and interpolations' as part of the required infrastructure. No criterion is given to distinguish 'lightweight' from 'complex' GIS operations, so the exclusion is vacuous. Moreover, several of the paper's own exemplar questions in Section 2.1 are quantitative and demand computation over data: cumulative economic loss from natural disasters, comparison of air quality levels between urban and rural areas, and the count of endangered species across continents and phyla. The architecture in Figure 1 contains no module for tool execution, structured data access, or numerical aggregation, so the proposed design lacks a principled way to answer these questions. The authors should either extend the architecture to include a computation/tool-use component for geo-spatial statistics, or they must define and justify a precise boundary that explains how the textual report generator can fully delegate or internalize such quantitative operations. As written, the boundary between text synthesis and GIS automation is both undefined and contradicted by the paper's own examples.
  2. [§2.3] The evaluation agenda, which the paper calls 'equally crucial' to the technical and infrastructural challenges, is left too under-specified to be actionable. The text proposes extending LLM-as-a-Judge methods to assess 'geo-temporal relevance, coherence, and diversity' and suggests adapting datasets like researchy questions and InfoDeepSeek, but it does not define these criteria operationally, does not specify how ground truth for long-form geo-temporal outputs would be constructed, and does not indicate what a stable open retrieval environment would need to contain (e.g., corpora with geographic and temporal annotations, relevance judgments, or baseline systems). Since the paper positions rigorous evaluation as a central pillar of its vision, the absence of at least one concrete evaluation protocol or metric definition makes the proposed research agenda difficult to test or compare against. The authors should sketch a minimal concrete evaluation framework, even at a programmatic level, to make the vision falsifiable.
minor comments (4)
  1. [Author affiliations] The first author's affiliation contains the typo 'Univesrity of Lisbon'; it should read 'University of Lisbon.'
  2. [References] Several references are listed twice: [36] appears twice in the reference list, and [48] is also duplicated; the in-text citation ranges such as '[36, 36–38]' and '[48, 48]' should be cleaned up.
  3. [Figure 1] The text never refers to Figure 1 in the body, although the caption describes an architectural overview; please add an explicit reference when the architecture is discussed in Section 2.2.
  4. [Throughout] Minor stylistic issues: 'e.g' is missing a period in several places (e.g., 'e.g LLM-as-a-Judge'), and the sentence in Section 2.1 beginning 'The range of question types that was given as example...' is grammatically awkward and should be revised.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the paper is a vision/position piece with no derived predictions; its two self-citations are background references and never load-bearing.

full rationale

This paper makes no formal derivation, fits no parameters, and reports no experimental prediction. Its central claim—that deep research systems should incorporate geo-temporal constraint handling—is presented as a research program and argued from general observations about LLM-based information access, not derived from any input quantity. The two self-citations ([3] Coelho et al. 2021 and [4] DeepResearchGym) are used only as examples of geographic passage retrieval and of open evaluation infrastructure respectively; neither is invoked as a uniqueness theorem, ansatz authority, or fitted input. Section 2.1's explicit scoping decision to exclude full GIS automation is a stated assumption about the problem boundary, and any tension with Section 2.2's mention of 'lightweight GIS operations' is a coherence/correctness concern, not a circular reduction. No equation, definition, or fitted parameter is shown to be equivalent to the paper's own inputs. Accordingly no circular step is identified; the minor self-citations are not load-bearing, so the score is low.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

No free parameters or invented entities. The paper's argument rests on two domain assumptions about scope and research infrastructure.

assumptions (2)
  • domain assumption Geo-temporal deep research should focus on text-based synthesis rather than automated GIS spatial computation.
    Stated in Section 2.1 as a scope decision. If spatial computation becomes essential, the proposed architecture would be incomplete.
  • domain assumption Open and reproducible search infrastructures are necessary for progress.
    Argued in Sections 2.2 and 2.3. This is a normative claim about how research should be conducted.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Vision for Geo-Temporal Deep Research Systems: Towards Comprehensive, Transparent, and Reproducible Geo-Temporal Information Synthesis." pith.science (2026). https://pith.science/paper/KE22SPZU

@misc{pith2026250614345,
  author       = {Pith},
  title        = {Pith review of: A Vision for Geo-Temporal Deep Research Systems: Towards Comprehensive, Transparent, and Reproducible Geo-Temporal Information Synthesis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KE22SPZU}},
  note         = {Machine review of arXiv:2506.14345}
}
read the original abstract

The emergence of Large Language Models (LLMs) has transformed information access, with current LLMs also powering deep research systems that can generate comprehensive report-style answers, through planned iterative search, retrieval, and reasoning. Still, current deep research systems lack the geo-temporal capabilities that are essential for answering context-rich questions involving geographic and/or temporal constraints, frequently occurring in domains like public health, environmental science, or socio-economic analysis. This paper reports our vision towards next generation systems, identifying important technical, infrastructural, and evaluative challenges in integrating geo-temporal reasoning into deep research pipelines. We argue for augmenting retrieval and synthesis processes with the ability to handle geo-temporal constraints, supported by open and reproducible infrastructures and rigorous evaluation protocols. Our vision outlines a path towards more advanced and geo-temporally aware deep research systems, of potential impact to the future of AI-driven information access.

Figures

Figures reproduced from arXiv: 2506.14345 by the authors.

Figure 1
Figure 1. Overview on the possible architecture of a geo-temporal deep research system. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

66 extracted references · 25 canonical work pages

  1. [1]

    Omid Reza Abbasi, Franz Welscher, Georg Weinberger, and Johannes Scholz

    work page

  2. [2]

    Abdelrahman Abdallah, Bhawna Piryani, Jonas Wallat, Avishek Anand, and Adam Jatowt. 2025. TempRetriever: Fusion-based Temporal Dense Passage Retrieval for Time-Sensitive Questions.arXiv preprint arXiv:2502.21024(2025)

    arXiv 2025

  3. [3]

    João Coelho, João Magalhães, and Bruno Martins. 2021. Improving neural models for the retrieval of relevant passages to geographical queries. InProceedings of the International Conference on Advances in Geographic Information Systems

    work page 2021

  4. [4]

    João Coelho, Jingjie Ning, Jingyuan He, Kangrui Mao, Abhijay Paladugu, Pranav Setlur, Jiahe Jin, Jamie Callan, João Magalhães, Bruno Martins, et al. 2025. Deep- ResearchGym: A Free, Transparent, and Reproducible Evaluation Sandbox for Deep Research.arXiv preprint arXiv:2505.19253(2025)

    arXiv 2025

  5. [5]

    Debrup Das, Sam O’ Nuallain, and Razieh Rahimi. 2025. RaDeR: Reasoning-aware Dense Retrieval Models.arXiv preprint arXiv:2505.18405(2025)

    arXiv 2025

  6. [6]

    Cheng Deng, Tianhang Zhang, Zhongmou He, Qiyuan Chen, Yuanyuan Shi, Yi Xu, Luoyi Fu, Weinan Zhang, Xinbing Wang, Chenghu Zhou, et al. 2024. K2: A foundation language model for geoscience knowledge understanding and utilization. InProceedings of the ACM International Conference on Web Search and Data Mining

    work page 2024

  7. [7]

    Akash Kumar Gautam, Lukas Lange, and Jannik Strötgen. 2024. Discourse-Aware In-Context Learning for Temporal Expression Normalization.arXiv preprint arXiv:2404.07775(2024)

    work page Pith review arXiv 2024

  8. [8]

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al . 2024. A Survey on LLM-as-a-Judge.arXiv preprint arXiv:2411.15594(2024)

    arXiv 2024

Show all 66 references
  1. [9]

    Chenghua Huang, Shisong Chen, Zhixu Li, Jianfeng Qu, Yanghua Xiao, Jiaxin Liu, and Zhigang Chen. 2024. GeoAgent: To empower LLMs using geospatial tools for address standardization. InFindings of the Association for Computational Linguistics

    2024

  2. [10]

    Intology. 2025. Zochi Technical Report.arXiv(2025)

    2025

  3. [11]

    Yuhan Ji, Song Gao, Ying Nie, Ivan Majić, and Krzysztof Janowicz. 2025. Founda- tion Models for Geospatial Reasoning: Assessing Capabilities of Large Language Models in Understanding Geometries and Topological Spatial Relations.arXiv preprint arXiv:2505.17136(2025)

    2025 arXiv

  4. [12]

    Bowen Jin, Jinsung Yoon, Priyanka Kargupta, Sercan O Arik, and Jiawei Han

  5. [13]

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. 2025. Search-R1: Training LLMs to rea- son and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516(2025)

    2025 arXiv

  6. [14]

    An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agents.arXiv preprint arXiv:2505.15117(2025)

    2025 arXiv

  7. [15]

    Satyapriya Krishna, Kalpesh Krishna, Anhad Mohananey, Steven Schwarcz, Adam Stambler, Shyam Upadhyay, and Manaal Faruqui. 2024. Fact, fetch, and reason: A unified evaluation of retrieval-augmented generation.arXiv preprint arXiv:2409.12941(2024)

    2024 arXiv

  8. [16]

    Mehmet Kardan, Bhawna Piryani, and Adam Jatowt. 2025. Evaluating Answer Reranking Strategies in Time-sensitive Question Answering.arXiv preprint arXiv:2503.04972(2025)

    2025 arXiv

  9. [17]

    Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. 2024. LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods.arXiv preprint arXiv:2412.05579(2024)

    2024 arXiv

  10. [18]

    Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al

  11. [19]

    Yuan Li, Qi Luo, Xiaonan Li, Bufan Li, Qinyuan Cheng, Bo Wang, Yining Zheng, Yuxin Wang, Zhangyue Yin, and Xipeng Qiu. 2025. R3-RAG: Learning Step- by-Step Reasoning and Retrieval for LLMs via Reinforcement Learning.arXiv preprint arXiv:2505.23794(2025)

    2025

  12. [20]

    Zhenlong Li and Huan Ning. 2023. Autonomous GIS: The next-generation AI-powered GIS.International Journal of Digital Earth16, 2 (2023)

    2023

  13. [21]

    Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. 2025. Search-o1: Agentic search-enhanced large reasoning models.arXiv preprint arXiv:2501.05366(2025)

    2025 arXiv

  14. [22]

    Z. Li, W. Zhou, Y. Y. Chiang, and M. Chen. 2023. GeoLM: Empowering Language Models for Geospatially Grounded Language Understanding.arXiv preprint arXiv:2310.14478(2023)

    2023 arXiv

  15. [23]

    Zhouhan Lin, Cheng Deng, Le Zhou, Tianhang Zhang, Yi Xu, Yutong Xu, Zhong- mou He, Yuanyuan Shi, Beiya Dai, Yunchong Song, et al. 2023. GeoGalactica: A scientific large language model in geoscience.arXiv preprint arXiv:2401.00434 (2023)

    2023 arXiv

  16. [24]

    Zhenlong Li, Huan Ning, Song Gao, Krzysztof Janowicz, Wenwen Li, Samantha T Arundel, Chaowei Yang, Budhendra Bhaduri, Shaowen Wang, A Zhu, et al

  17. [25]

    GIScience in the era of artificial intelligence: A research agenda towards autonomous GIS.arXiv preprint arXiv:2503.23633(2025)

    2025 arXiv

  18. [26]

    Zilong Liu, Krzysztof Janowicz, Ivan Majic, Meilin Shi, Alexandra Fortacz, Mina Karimi, Gengchen Mai, and Kitty Currier. 2025. Operationalizing Geographic Diversity for the Evaluation of AI-Generated Content.Transactions in GIS29, 3 (2025), e70057

    2025

  19. [27]

    Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha

  20. [28]

    Zilong Liu, Krzysztof Janowicz, Ling Cai, Rui Zhu, Gengchen Mai, and Meilin Shi. 2022. Geoparsing: Solved or biased? An evaluation of geographic biases in geoparsing.AGILE: GIScience Series3 (2022)

    2022

  21. [29]

    Zilong Liu, Krzysztof Janowicz, Kitty Currier, and Meilin Shi. 2024. Measuring geographic diversity of foundation models with a natural language-based geo- guessing experiment on GPT-4.AGILE: GIScience Series5 (2024)

    2024

  22. [30]

    Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al . 2025. Humanity’s last exam.arXiv preprint arXiv:2501.14249(2025)

    2025 arXiv

  23. [31]

    Bhawna Piryani, Abdelrahman Abdullah, Jamshid Mozafari, Avishek Anand, and Adam Jatowt. 2025. It’s High Time: A Survey of Temporal Information Retrieval and Question Answering.arXiv preprint arXiv:2505.20243(2025)

    2025 arXiv

  24. [32]

    arXiv preprint arXiv:2408.06292(2024)

    The AI scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292(2024)

    2024 arXiv

  25. [33]

    Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2023. GAIA: A benchmark for general ai assistants. InProceedings of the International Conference on Learning Representations

    2023

  26. [34]

    Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. arXiv preprint arXiv:2305.14251(2023)

    2023 arXiv

  27. [35]

    Zehan Qi, Rongwu Xu, Zhijiang Guo, Cunxiang Wang, Hao Zhang, and Wei Xu

  28. [36]

    Corby Rosset, Ho-Lam Chung, Guanghui Qin, Ethan C Chau, Zhuo Feng, Ahmed Awadallah, Jennifer Neville, and Nikhil Rao. 2024. Researchy questions: A dataset of multi-perspective, decompositional questions for LLM Web agents.arXiv preprint arXiv:2402.17896(2024)

    2024 arXiv

  29. [37]

    Ronak Pradeep, Nandan Thakur, Sahel Sharifymoghaddam, Eric Zhang, Ryan Nguyen, Daniel Campos, Nick Craswell, and Jimmy Lin. 2025. Ragnarök: A reusable RAG framework and baselines for TREC 2024 retrieval-augmented gen- eration track. InProceedings of the European Conference on ...

    2025

  30. [38]

    Ronak Pradeep, Nandan Thakur, Shivani Upadhyay, Daniel Campos, Nick Craswell, and Jimmy Lin. 2024. Initial nugget evaluation results for the TREC 2024 RAG track with the autonuggetizer framework.arXiv preprint arXiv:2411.09607 (2024)

    2024 arXiv

  31. [39]

    Ross S Purves, Paul Clough, Christopher B Jones, Mark H Hall, Vanessa Murdock, et al. 2018. Geographic information retrieval: Progress and challenges in spatial search of text.Foundations and Trends®in Information Retrieval12, 2-3 (2018)

    2018

  32. [40]

    Rulin Shao, Rui Qiao, Varsha Kishore, Niklas Muennighoff, Xi Victoria Lin, Daniela Rus, Bryan Kian Hsiang Low, Sewon Min, Wen-tau Yih, Pang Wei Koh, et al. 2025. ReasonIR: Training Retrievers for Reasoning Tasks.arXiv preprint arXiv:2504.20595(2025)

    2025 arXiv

  33. [41]

    Long 2RAG: Evaluating Long-Context & Long-Form Retrieval-Augmented Generation with Key Point Recall.arXiv preprint arXiv:2410.23000(2024)

    2024 arXiv

  34. [42]

    Huatong Song, Jinhao Jiang, Wenqing Tian, Zhipeng Chen, Yuhuan Wu, Jiahao Zhao, Yingqian Min, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. 2025. R1- Searcher++: Incentivizing the Dynamic Knowledge Acquisition of LLMs via Reinforcement Learning.arXiv preprint arXiv:2505.17005(2025)

    2025 arXiv

  35. [43]

    Dongyu Ru, Lin Qiu, Xiangkun Hu, Tianhang Zhang, Peng Shi, Shuaichen Chang, Cheng Jiayang, Cunxiang Wang, Shichao Sun, Huanyu Li, et al . 2024. RAGChecker: A fine-grained framework for diagnosing retrieval-augmented generation.Proceedings of the Anual Meeting on Neural Informa...

    2024

  36. [44]

    Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. 2023. ARES: An automated evaluation framework for retrieval-augmented generation systems.arXiv preprint arXiv:2311.09476(2023)

    2023 arXiv

  37. [45]

    Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Zicheng Liu, and Emad Barsoum. 2025. Agent laboratory: Using LLM agents as research assistants.arXiv preprint arXiv:2501.04227(2025)

    2025 arXiv

  38. [46]

    Nandan Thakur, Ronak Pradeep, Shivani Upadhyay, Daniel Campos, Nick Craswell, and Jimmy Lin. 2025. Support Evaluation for the TREC 2024 RAG Track: Comparing Human versus LLM Judges.arXiv preprint arXiv:2504.15205 (2025)

    2025 arXiv

  39. [47]

    Shangeetha Sivasothy, Scott Barnett, Stefanus Kurniawan, Zafaryab Rasool, and Rajesh Vasa. 2024. RAGProbe: An Automated Approach for Evaluating RAG Applications.arXiv preprint arXiv:2409.19019(2024)

    2024 arXiv

  40. [48]

    Jonas Wallat, Abdelrahman Abdallah, Adam Jatowt, and Avishek Anand. 2025. A Study into Investigating Temporal Robustness of LLMs.arXiv preprint arXiv:2503.17073(2025)

    2025 arXiv

  41. [49]

    Shuang Sun, Huatong Song, Yuhao Wang, Ruiyang Ren, Jinhao Jiang, Jun- jie Zhang, Fei Bai, Jia Deng, Wayne Xin Zhao, Zheng Liu, et al . 2025. Sim- pleDeepSearcher: Deep information seeking via web-powered reasoning trajec- tory synthesis.arXiv preprint arXiv:2505.16834(2025)

    2025

  42. [50]

    Jiabin Tang, Lianghao Xia, Zhonghang Li, and Chao Huang. 2025. AI-Researcher: Autonomous Scientific Innovation.arXiv preprint arXiv:2505.18705(2025)

    2025 arXiv

  43. [51]

    NovelSeek Team, Bo Zhang, Shiyang Feng, Xiangchao Yan, Jiakang Yuan, Zhiyin Yu, Xiaohan He, Songtao Huang, Shaowei Hou, Zheng Nie, et al. 2025. Novel- Seek: When Agent Becomes the Scientist–Building Closed-Loop System from Hypothesis to Verification.arXiv preprint arXiv:2505.1...

    2025 arXiv

  44. [52]

    Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. 2025. The AI scientist-v2: Workshop-level auto- mated scientific discovery via agentic tree search.arXiv preprint arXiv:2504.08066 (2025)

    2025 arXiv

  45. [53]

    Nico Van de Weghe, Lars De Sloover, Anthony Cohn, Haosheng Huang, Simon Scheider, Renée Sieber, Sabine Timpf, and Christophe Claramunt. 2025. Oppor- tunities and challenges of integrating geographic information science and large language models.Journal of Spatial Information S...

    2025

  46. [54]

    Le Zhang, Bo Wang, Xipeng Qiu, Siva Reddy, and Aishwarya Agrawal. 2025. REARANK: Reasoning Re-ranking Agent via Reinforcement Learning.arXiv preprint arXiv:2505.20046(2025)

    2025 arXiv

  47. [55]

    Orion Weller, Benjamin Chang, Sean MacAvaney, Kyle Lo, Arman Cohan, Ben- jamin Van Durme, Dawn Lawrie, and Luca Soldaini. 2024. FollowIR: Evaluating and teaching information retrieval models to follow instructions.arXiv preprint arXiv:2403.15246(2024)

    2024 arXiv

  48. [56]

    Yunjia Xi, Jianghao Lin, Menghui Zhu, Yongzhao Xiao, Zhuoying Ou, Jiaqi Liu, Tong Wan, Bo Chen, Weiwen Liu, Yasheng Wang, et al. 2025. InfoDeepSeek: Benchmarking Agentic Information Seeking for Retrieval-Augmented Genera- tion.arXiv preprint arXiv:2505.15872(2025)

    2025 arXiv

  49. [57]

    Wanghan Xu, Xiangyu Zhao, Yuhao Zhou, Xiaoyu Yue, Ben Fei, Fenghua Ling, Wenlong Zhang, and Lei Bai. 2025. EarthSE: A Benchmark Evaluating Earth Scientific Exploration Capability for Large Language Models.arXiv preprint arXiv:2505.17139(2025)

    2025 arXiv

  50. [58]

    Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. 2025. DeepResearcher: Scaling deep research via rein- forcement learning in real-world environments.arXiv preprint arXiv:2504.03160 (2025)

    2025 arXiv

  51. [59]

    Dazhou Yu, Riyang Bao, Gengchen Mai, and Liang Zhao. 2025. Spatial-RAG: Spatial retrieval augmented generation for real-world spatial reasoning questions. arXiv preprint arXiv:2502.18470(2025)

    2025 arXiv

  52. [60]

    Yuchen Zhuang, Aaron Trinh, Rushi Qiang, Haotian Sun, Chao Zhang, Hanjun Dai, and Bo Dai. 2025. Towards Better Instruction Following Retrieval Models. arXiv preprint arXiv:2505.21439(2025)

    2025 arXiv

  53. [61]

    Yifan Zhang, Zhiyun Wang, Zhengting He, Jingxuan Li, Gengchen Mai, Jianfeng Lin, Cheng Wei, and Wenhao Yu. 2024. BB-GeoGPT: A framework for learning a large language model for geographic information science.Information Processing & Management61, 5 (2024), 103808

    2024

  54. [62]

    Yifan Zhang, Cheng Wei, Zhengting He, and Wenhao Yu. 2024. GeoGPT: An assistant for understanding and processing geospatial tasks.International Journal of Applied Earth Observation and Geoinformation131 (2024)

    2024

  55. [63]

    Zeyu Zhang and Steven Bethard. 2024. A survey on geocoding: algorithms and datasets for toponym resolution.Language Resources and Evaluation(2024)

    2024

  56. [65]

    Jianqun Zhou, Yuanlei Zheng, Wei Chen, Qianqian Zheng, Hui Su, Wei Zhang, Rui Meng, and Xiaoyu Shen. 2024. Beyond content relevance: Evaluating instruction following in retrieval models.arXiv preprint arXiv:2410.23841(2024)

    2024 arXiv

  57. [2024]

    From generation to judgment: Opportunities and challenges of LLM-as-a- Judge.arXiv preprint arXiv:2411.16594(2024)

    2024

  58. [2025]

    The World As Large Language Models See It: Exploring the reliability of LLMs in representing geographical features.arXiv preprint arXiv:2506.00203 (2025)

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.