arxiv: 2604.20144 · v1 · submitted 2026-04-22 · 💻 cs.DB

Recognition: unknown

An Agentic Approach to Metadata Reasoning

Alon Halevy, Cosmin Arad, Fatma Ozcan, Jiani Zhang, Sercan O. Arik

Pith reviewed 2026-05-09 23:26 UTC · model grok-4.3

classification 💻 cs.DB

keywords data source selectionmetadata reasoningLLM agentsdata lakesagentic systemstable retrievalanalytical tasks

0 comments

The pith

The Metadata Reasoner agent uses LLM reasoning on metadata to select sufficient and minimal data sources for tasks.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper introduces the Metadata Reasoner, an agentic system that helps autonomous LLM agents discover the right datasets from large collections by evaluating metadata. Data selection is a bottleneck for multi-step analysis because semantics are nuanced and volumes are high. The system first retrieves candidate tables, then has the agent consult multiple metadata aspects to confirm the set meets the task needs exactly. If successful, agents could perform complex data work with less human help and fewer errors from irrelevant sources. The approach shows strong results on real-world benchmarks and stays effective when extra or poor tables are present.

Core claim

The Metadata Reasoner leverages a table-search engine to retrieve candidate tables, and then autonomously consults various aspects of the available metadata to determine whether the candidates fit the requirements of the task, identifying a small set of data sources that are both sufficient and minimal.

What carries the argument

An LLM agent that reasons over multiple metadata aspects to judge whether retrieved candidate tables are sufficient and minimal for a given analytical task.

Load-bearing premise

Available metadata is rich, accurate, and complete enough for the agent to judge sufficiency and minimality without errors or hallucinations.

What would settle it

Apply the Metadata Reasoner to a benchmark with sparse or inaccurate metadata and measure whether its F1-score falls to or below baseline levels due to wrong sufficiency decisions.

Figures

Figures reproduced from arXiv: 2604.20144 by Alon Halevy, Cosmin Arad, Fatma Ozcan, Jiani Zhang, Sercan O. Arik.

**Figure 2.** Figure 2: Illustrative Example of the Metadata Reasoner Workflow. Given an analytical task, the Metadata Reasoner first decomposes the query into multi-faceted search plans (e.g., reports vs. population) to retrieve candidate datasets. It then uses specialized tools to confirm entity existence (e.g., "Puerto Rico/DC") and to validate relational paths. The final output provides a justified selection of tables, ens… view at source ↗

**Figure 3.** Figure 3: An example that Metadata Reasoner selects the right tables with low retrieval ranks. The two ground truth tables rank 5 and 11 in the returned list in vector search. The Metadata Reasoner breaks down the complex analytic task (Step 1) into searchable and computable variables for search (Step 2). It then uses tools to verify the data presence (Step 3), ensuring the precise and complete set of tables is sele… view at source ↗

**Figure 4.** Figure 4: Distribution of selected table types from [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗

read the original abstract

As LLM-driven autonomous agents evolve to perform complex, multi-step tasks that require integrating multiple datasets, the problem of discovering relevant data sources becomes a key bottleneck. Beyond the challenge posed by the sheer volume of available data sources, data-source selection is difficult because the semantics of data are extremely nuanced and require considering many aspects of the data. To address this, we introduce the Metadata Reasoner, an agentic approach to metadata reasoning, designed to identify a small set of data sources that are both sufficient and minimal for a given analytical task. The Metadata Reasoner leverages a table-search engine to retrieve candidate tables, and then autonomously consults various aspects of the available metadata to determine whether the candidates fit the requirements of the task. We demonstrate the effectiveness of the Metadata Reasoner through a series of empirical studies. Evaluated on the real-world KramaBench datasets for data selection, our approach achieves an average F1-score of 83.16%, outperforming state-of-the-art baselines by a substantial margin of 32 percentage points. Furthermore, evaluations on a newly-created synthetic benchmark based on the BIRD data lake reveal that the Metadata Reasoner is highly robust against redundant and low-quality tables that may be in the data lake. In this noisy environment, it maintains an average of 85.5% F1-score for selecting the right datasets and demonstrates a 99% success rate in avoiding low-quality data.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The Metadata Reasoner adds a two-stage agent that retrieves tables then lets an LLM autonomously judge metadata for sufficiency and minimality, delivering reported F1 gains of 32 points on KramaBench, but the gains rest on unverified reliability of those LLM judgments.

read the letter

The main point is that this paper builds a Metadata Reasoner that first pulls candidate tables with a search engine and then has an LLM agent check multiple metadata aspects to decide if the set is both sufficient and minimal for the task. That architecture is the concrete new piece, and it targets a clear bottleneck in LLM agents that need to pull from data lakes for multi-dataset work.

Referee Report

3 major / 2 minor

Summary. The paper introduces the Metadata Reasoner, an agentic LLM-based system that first retrieves candidate tables via a search engine and then autonomously reasons over multiple metadata aspects to select a small set of data sources that are both sufficient and minimal for a given analytical task. On the real-world KramaBench data-selection datasets it reports an average F1-score of 83.16% (32 pp above SOTA baselines); on a newly constructed synthetic benchmark derived from BIRD it reports 85.5% F1 together with a 99% success rate at avoiding low-quality tables.

Significance. If the performance margins are reproducible and attributable to the agentic reasoning step, the work would represent a meaningful advance in automated data-source discovery for data lakes, where semantic nuance across metadata makes simple retrieval insufficient. The dual evaluation on a real-world benchmark and a controlled noisy synthetic setting is a positive feature. The central claim nevertheless rests on the unverified reliability of LLM semantic judgments, so the practical significance remains conditional on stronger empirical grounding.

major comments (3)

[Abstract and Evaluation] Abstract and Evaluation section: the headline claims (83.16% F1 on KramaBench, +32 pp; 85.5% F1 and 99% low-quality avoidance on the synthetic set) are presented without baseline implementation details, statistical significance tests, error bars, or inter-run variance. These omissions prevent assessment of whether the reported margins are driven by the agentic component or by upstream retrieval and post-hoc choices.
[Method (§3)] Method (§3): no ablation is reported that compares the full agentic Metadata Reasoner against a non-agentic LLM baseline given identical metadata and asked to judge sufficiency and minimality. Without this control it is impossible to isolate the contribution of the autonomous, multi-step reasoning process from the quality of the table-search engine alone.
[Experiments (§5.2)] Experiments (§5.2): the robustness claims (99% success avoiding low-quality tables, high F1 under redundancy) are not accompanied by error analysis, case studies of agent decisions, or ablation on LLM choice. The core mechanism—LLM judgment of nuanced semantic fit—therefore remains untested for hallucinations or systematic bias, directly undermining the central agentic contribution.

minor comments (2)

A workflow diagram of the agent’s interaction with the table-search engine and metadata aspects would improve clarity of the architecture.
The precise operational definitions of “sufficient” and “minimal” with respect to an analytical task should be stated more formally, ideally with examples.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive comments. We address each major comment point by point below and indicate the revisions we will make.

read point-by-point responses

Referee: [Abstract and Evaluation] Abstract and Evaluation section: the headline claims (83.16% F1 on KramaBench, +32 pp; 85.5% F1 and 99% low-quality avoidance on the synthetic set) are presented without baseline implementation details, statistical significance tests, error bars, or inter-run variance. These omissions prevent assessment of whether the reported margins are driven by the agentic component or by upstream retrieval and post-hoc choices.

Authors: We agree that the evaluation section would benefit from greater transparency. In the revised manuscript we will add full baseline implementation details (including retrieval parameters and any post-processing), report statistical significance tests against each baseline, include error bars, and document inter-run variance from repeated executions. These additions will allow readers to better evaluate the source of the observed margins. revision: yes
Referee: [Method (§3)] Method (§3): no ablation is reported that compares the full agentic Metadata Reasoner against a non-agentic LLM baseline given identical metadata and asked to judge sufficiency and minimality. Without this control it is impossible to isolate the contribution of the autonomous, multi-step reasoning process from the quality of the table-search engine alone.

Authors: We acknowledge that a direct comparison to a non-agentic LLM judge using the same metadata inputs would more cleanly isolate the value of multi-step autonomous reasoning. While our existing baselines already include several non-agentic approaches, we did not report this specific control. We will implement and evaluate the requested non-agentic LLM baseline in the revised version. revision: yes
Referee: [Experiments (§5.2)] Experiments (§5.2): the robustness claims (99% success avoiding low-quality tables, high F1 under redundancy) are not accompanied by error analysis, case studies of agent decisions, or ablation on LLM choice. The core mechanism—LLM judgment of nuanced semantic fit—therefore remains untested for hallucinations or systematic bias, directly undermining the central agentic contribution.

Authors: We agree that additional diagnostics would strengthen confidence in the LLM-based judgments. In the revision we will add an error analysis of selection mistakes, case studies that display the agent's intermediate reasoning steps, and an ablation across different LLM backbones to examine sensitivity to hallucinations or bias. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical results on external benchmarks

full rationale

The paper's core contribution is an empirical evaluation of the Metadata Reasoner agent on the external KramaBench dataset (real-world) and a BIRD-derived synthetic benchmark. Reported F1 scores (83.16% average, +32 pp over baselines; 85.5% on synthetic) are measured against independent baselines and datasets with no equations, fitted parameters renamed as predictions, or self-citation chains that reduce the performance claims to the paper's own inputs by construction. The method description relies on standard LLM agent components and table-search retrieval without any load-bearing self-referential definitions or uniqueness theorems imported from prior author work.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the unproven domain assumption that LLMs can perform reliable metadata-based sufficiency and minimality judgments; no free parameters or invented entities are introduced in the abstract.

axioms (1)

domain assumption LLMs can autonomously consult various aspects of metadata to determine whether candidate tables fit analytical task requirements.
This is the core mechanism of the Metadata Reasoner described in the abstract.

pith-pipeline@v0.9.0 · 5561 in / 1291 out tokens · 26134 ms · 2026-05-09T23:26:28.693336+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

46 extracted references · 21 canonical work pages · 2 internal anchors

[1]

Anthropic. 2026. Agent Skills.Claud API Docs(2026). https://platform.claude. com/docs/en/agents-and-tools/agent-skills/overview

2026
[2]

Muhammad Imam Luthfi Balaka, David Alexander, Qiming Wang, Yue Gong, Adila Krisnadhi, and Raul Castro Fernandez. 2025. Pneuma: Leveraging llms for tabular data representation and retrieval in an end-to-end system.Proceedings of the ACM on Management of Data3, 3 (2025), 1–28

2025
[3]

Alex Bogatu, Alvaro AA Fernandes, Norman W Paton, and Nikolaos Konstanti- nou. 2020. Dataset discovery in data lakes. In2020 ieee 36th international confer- ence on data engineering (icde). IEEE, 709–720

2020
[4]

Dan Brickley, Matthew Burgess, and Natasha Noy. 2019. Google dataset search: Building a search engine for datasets in an open web ecosystem. InThe world wide web conference. 1365–1375

2019
[5]

Adriane Chapman, Elena Simperl, Laura Koesten, George Konstantinidis, Luis- Daniel Ibáñez, Emilia Kacprzak, and Paul Groth. 2020. Dataset search: a survey. The VLDB Journal29, 1 (2020), 251–272

2020
[6]

Yeounoh Chung, Gaurav T Kakkar, Yu Gan, Brenton Milne, and Fatma Ozcan
[7]

Is long context all you need? leveraging LLM’s extended context for NL2SQL.arXiv preprint arXiv:2501.12372(2025)

work page arXiv 2025
[8]

Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multi- modality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261(2025)

work page internal anchor Pith review Pith/arXiv arXiv 2025
[9]

Debrup Das, Sam O’ Nuallain, and Razieh Rahimi. 2025. RaDeR: Reasoning-aware Dense Retrieval Models.arXiv preprint arXiv:2505.18405(2025)

work page arXiv 2025
[10]

Halevy, and Zachary G

AnHai Doan, Alon Y. Halevy, and Zachary G. Ives. 2012.Principles of Data Integration. Morgan Kaufmann. http://research.cs.wisc.edu/dibook/

2012
[11]

Grace Fan, Jin Wang, Yuliang Li, and Renée J. Miller. 2023. Table Discovery in Data Lakes: State-of-the-art and Future Directions. InCompanion of the 2023 International Conference on Management of Data(Seattle, WA, USA)(SIGMOD ’23). Association for Computing Machinery, New York, NY, USA, 69–75. doi:10. 1145/3555041.3589409

work page arXiv 2023
[12]

Grace Fan, Jin Wang, Yuliang Li, Dan Zhang, and Renée J. Miller. 2023. Semantics- Aware Dataset Discovery from Data Lakes with Contextualized Column-Based Representation Learning.Proc. VLDB Endow.16, 7 (March 2023), 1726–1739. doi:10.14778/3587136.3587146

work page doi:10.14778/3587136.3587146 2023
[13]

Raul Castro Fernandez, Ziawasch Abedjan, Famien Koko, Gina Yuan, Samuel Madden, and Michael Stonebraker. 2018. Aurum: A data discovery system. In2018 IEEE 34th International Conference on Data Engineering (ICDE). IEEE, 1001–1012

2018
[14]

Yanjie Fu, Dongjie Wang, Wangyang Ying, Xinyuan Wang, Xiangliang Zhang, Huan Liu, and Jian Pei. 2025. Autonomous data agents: A new opportunity for smart data.arXiv preprint arXiv:2509.18710(2025)

work page arXiv 2025
[15]

Alon Y. Halevy. 2001. Answering queries using views: A survey.VLDB J.10, 4 (2001), 270–294. doi:10.1007/S007780100054

work page doi:10.1007/s007780100054 2001
[16]

Madelon Hulsebos, Wenjing Lin, Shreya Shankar, and Aditya Parameswaran
[17]

In Proceedings of the 2024 Workshop on Human-In-the-Loop Data Analytics

It took longer than i was expecting: Why is dataset search still so hard?. In Proceedings of the 2024 Workshop on Human-In-the-Loop Data Analytics. 1–4

2024
[18]

Tengjun Jin, Yuxuan Zhu, and Daniel Kang. 2025. Elt-bench: An end-to-end bench- mark for evaluating ai agents on elt pipelines.arXiv preprint arXiv:2504.04808 (2025)

work page arXiv 2025
[19]

Wojciech Kosiuk, Xingyu Ji, Yeounoh Chung, Fatma Özcan, and Madelon Hulse- bos. 2026. Fine-Grained Table Retrieval Through the Lens of Complex Queries. arXiv preprint arXiv:2603.07146(2026)

work page arXiv 2026
[20]

Christos Koutras, George Siachamis, Andra Ionescu, Kyriakos Psarakis, Jerry Brons, Marios Fragkoulis, Christoph Lofi, Angela Bonifati, and Asterios Katsi- fodimos. 2021. Valentine: Evaluating matching techniques for dataset discovery. In2021 IEEE 37th International Conference on Data Engineering (ICDE). IEEE, 468–479

2021
[21]

Christos Koutras, Jiani Zhang, Xiao Qin, Chuan Lei, Vasileios Ioannidis, Christos Faloutsos, George Karypis, and Asterios Katsifodimos. 2025. OmniMatch: Join- ability Discovery in Data Products.Proceedings of the VLDB Endowment18, 11 (2025), 4588–4601

2025
[22]

Eugenie Lai, Gerardo Vitagliano, Ziyu Zhang, Sivaprasad Sudhir, Om Chabra, Anna Zeng, Anton A Zabreyko, Chenning Li, Ferdi Kossmann, Jialin Ding, et al
[23]

KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakes.arXiv preprint arXiv:2506.06541(2025)

work page arXiv 2025
[24]

Aristotelis Leventidis, Martin Pekár Christensen, Matteo Lissandrini, Laura Di Rocco, Katja Hose, and Renée J Miller. 2024. A large scale test corpus for se- mantic table search. InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1142–1151

2024
[25]

Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et al . 2023. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. Advances in Neural Information Processing Systems36 (2023), 42330–42357

2023
[26]

Zhuoming Li, Yichen Gong, Yelong Shen, and Xing Xie Zhang. 2024. Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach.arXiv preprint arXiv:2402.11193(2024)

work page arXiv 2024
[27]

Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the Middle: How Language Models Use Long Contexts. InTransactions of the Association for Computational Linguis- tics, Vol. 12. 157–173

2024
[28]

Zhaoyang Liu, Zezheng Lai, Gaojie Zhang, Renjie Zhang, Keqing Chen, Xiao Wang, Yujie Zhu, Shaogang Cao, Jiacheng Chen, Yixiao Ge, et al. 2024. Control- LLM: Augmenting Language Models with Tools by Planning, Customization, and Interaction. InProceedings of the 2024 International Conference on Machine Learning (ICML)

2024
[29]

Meixiu Long, Duolin Sun, Dan Yang, Junjie Wang, Yue Shen, Jian Wang, Peng Wei, Jinjie Gu, and Jiahai Wang. 2025. DIVER: A Multi-Stage Approach for Reasoning- intensive Information Retrieval.arXiv preprint arXiv:2508.07995(2025)

work page arXiv 2025
[30]

Yu A Malkov and Dmitry A Yashunin. 2018. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs.IEEE transactions on pattern analysis and machine intelligence42, 4 (2018), 824–836

2018
[31]

Fatemeh Nargesian, Erkang Zhu, Ken Q Pu, and Renée J Miller. 2018. Table union search on open data.Proceedings of the VLDB Endowment11, 7 (2018), 813–825

2018
[32]

Dataset discovery and exploration: A survey,

Norman W. Paton, Jiaoyan Chen, and Zhenyu Wu. 2023. Dataset Discovery and Exploration: A Survey.ACM Comput. Surv.56, 4, Article 102 (Nov. 2023), 37 pages. doi:10.1145/3626521

work page doi:10.1145/3626521 2023
[33]

2009.The probabilistic relevance frame- work: BM25 and beyond

Stephen Robertson and Hugo Zaragoza. 2009.The probabilistic relevance frame- work: BM25 and beyond. Vol. 4. Now Publishers Inc

2009
[34]

Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Bhardwaj, Naman Goyal, et al. 2023. Toolformer: Lan- guage models can teach themselves to use tools.Advances in Neural Information Processing Systems36 (2023)

2023
[35]

Rulin Shao, Rui Qiao, Varsha Kishore, Niklas Muennighoff, Xi Victoria Lin, Daniela Rus, Bryan Kian Hsiang Low, Sewon Min, Wen-tau Yih, Pang Wei Koh, et al. 2025. ReasonIR: Training Retrievers for Reasoning Tasks.arXiv preprint arXiv:2504.20595(2025)

work page arXiv 2025
[36]

Chi, Nathanael Schärli, and Denny Zhou

Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Schärli, and Denny Zhou. 2023. Large Language Models are Easily Distracted by Irrelevant Context. InInternational Conference on Machine Learning (ICML). PMLR

2023
[37]

Aditi Singh, Abul Ehtesham, Saket Kumar, and Tala Talaei Khoei. 2025. Agen- tic retrieval-augmented generation: A survey on agentic rag.arXiv preprint arXiv:2501.09136(2025)

work page internal anchor Pith review arXiv 2025
[38]

Hongjin Su, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han-yu Wang, Haisu Liu, Quan Shi, Zachary S Siegel, Michael Tang, et al. 2024. Bright: A realistic and challenging benchmark for reasoning-intensive retrieval.arXiv preprint arXiv:2407.12883(2024)

work page arXiv 2024
[39]

Zixin Wei, Yucan Guo, Jinyang Li, Xiaolin Han, Xiaolong Jin, and Chenhao Ma
[40]

Revisiting Task-Oriented Dataset Search in the Era of Large Language Models: Challenges, Benchmark, and Solution.arXiv preprint arXiv:2512.15363 (2025)

work page arXiv 2025
[41]

Hongli Yu, Tinghong Chen, Jiangtao Feng, Jiangjie Chen, Weinan Dai, Qiying Yu, Ya-Qin Zhang, Wei-Ying Ma, Jingjing Liu, Mingxuan Wang, et al. 2025. Memagent: Reshaping long-context llm with multi-conv rl-based memory agent.arXiv preprint arXiv:2507.02259(2025)

work page arXiv 2025
[42]

Siyu Yuan, Kaitao Song, Jiangjie Chen, Xu Tan, Yongliang Shen, Kan Ren, Dong- sheng Li, and Deqing Yang. 2025. Easytool: Enhancing llm-based agents with concise tool instruction. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 951–972

2025
[43]

Weinan Zhang, Junwei Liao, Ning Li, Kounianhua Du, and Jianghao Lin. 2024. Agentic information retrieval.arXiv preprint arXiv:2410.09713(2024)

work page arXiv 2024
[44]

Erkang Zhu, Dong Deng, Fatemeh Nargesian, and Renée J Miller. 2019. Josie: Overlap set similarity search for finding joinable tables in data lakes. InProceed- ings of the 2019 International Conference on Management of Data. 847–864

2019
[45]

Yizhang Zhu, Liangwei Wang, Chenyu Yang, Xiaotian Lin, Boyan Li, Wei Zhou, Xinyu Liu, Zhangyang Peng, Tianqi Luo, Yu Li, et al . 2025. A Survey of Data Agents: Emerging Paradigm or Overstated Hype?arXiv preprint arXiv:2510.23587 (2025)

work page arXiv 2025
[46]

Yuan Zhuang, Yifei Li, Ling Chen, and Wei Wang. 2024. ToolNet: Connect- ing LLMs with Massive Tools via Graph-based Propagation.arXiv preprint arXiv:2403.00839(2024)

work page arXiv 2024