Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Interactive Data Harmonization with LLM Agents: Opportunities and Challenges

T0 review · 3 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that an LLM-based agent can orchestrate data integration primitives and interactive user feedback to synthesize data harmonization pipelines that outperform the underlying algorithms alone.

desk verdict A solid, honest vision paper with a working prototype; take the evaluation numbers as illustrative, not evidence. read the letter →

arxiv 2502.07132 v3 pith:DUG7ZKYN submitted 2025-02-10 cs.AI cs.DB

Aécio Santos , Eduardo H. M. Pena , Roque Lopez , Juliana Freire This is my paper · ORCID
classification cs.AIcs.DB
keywords dataharmonizationLLMagentsschemamatchingvaluemappingintegrationinteractivesystemsreproduciblepipelinesclinical
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Data harmonization — reconciling tables from different sources into one standard format — is usually a slow, manual process. This paper argues that an LLM-based agent can make it interactive and partly automated: the agent chooses and runs integration algorithms, inspects their output, spots wrong column and value matches, corrects them or asks the user, and finally emits a reusable pipeline specification. The authors demonstrate this with Harmonia, an agentic prototype that maps a clinical dataset to a standard cancer-research vocabulary. In their evaluation, agent-assisted schema matching reached an accuracy of 1.00 and value mapping rose from 0.58 to 0.68 against the same algorithms used without the agent. The point is not that the LLM replaces the algorithms, but that it acts as an evaluator and orchestrator that catches the errors the algorithms miss.

What carries the argument

The key machinery is the agentic loop: an LLM is given descriptions of the task and of available integration primitives, and repeatedly returns actions — tool calls to those primitives, generated code, or questions to the user — with each action's output fed back into the LLM until the task is complete. Around this loop sits a library of composable primitives (schema matching, value mapping, materialization) that encodes efficient, well-known algorithms, so the LLM does not do heavy computation itself but rather evaluates and orchestrates. The loop is what lets the agent catch mistakes: the LLM inspects a primitive's output, detects a wrong match like 'Histologic_type' mapped to 'roots', and triggers a secondary primitive to find alternatives. This combination — LLM as evaluator and planner, primitives as workers, and the user as final authority for context-dependent calls — is the mechanism that carries the paper's claim.

What would settle it

Repeatedly run the same harmonization task with identical prompts and count how often the agent fails to correct a wrong mapping, as the paper admits can happen. A concrete test: inject known-incorrect matches into a held-out set of clinical attributes and measure the correction rate — if it is not substantially above the no-agent baseline, the central claim is falsified.

Watch

Extended reading notes

Core claim

The central claim is that agentic data harmonization — a system that combines LLM reasoning, a composable library of data integration primitives, and user interaction — can produce harmonization pipelines whose accuracy exceeds that of the underlying integration algorithms alone. In the reported clinical use case, the agent detects that the column 'Histologic_type' was wrongly matched to 'roots', asks the primitive library for alternatives, and proposes 'primary_diagnosis' to the user; likewise it corrects value mappings such as 'FIGO grade 1' to 'G1' rather than 'Low Grade'. The agent runs a loop: parse the user request, call a primitive (schema matching, value mapping, materialization), feed the result back to the LLM, and iterate until the task is done, asking the user only when context is needed. When no primitive fits, the LLM writes custom Python code on demand. The result is a declarative mapping specification that can be stored and re-executed, so the harmonization process becomes reproducible without re-running the LLM.

Load-bearing premise

The demonstration assumes that the LLM's general knowledge reliably detects and corrects errors in domain-specific vocabularies without fine-tuning, yet the paper itself notes the LLM occasionally failed to do so even with identical prompts.

Editorial extensions

If this is right

  • Domain experts without programming experience could harmonize their data by conversing with an agent, with the agent doing the heavy lifting and escalating only context-dependent judgment calls.
  • Harmonization pipelines become reproducible artifacts: the declarative mapping specification can be published with the data, so results can be re-derived without re-running model interactions.
  • The agent's ability to catch primitive mistakes suggests a general pattern: LLMs as evaluator layers over deterministic data-integration tools, applicable beyond harmonization to cleaning and entity resolution.
  • Agent-assisted harmonization still needs end-to-end benchmarks; the paper's evaluation is a single clinical use case, so measured gains may not transfer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the pattern holds, the same 'LLM as inspector' loop could extend to other data preparation steps — deduplication, normalization, missing-value imputation — where deterministic algorithms are imperfect and user context is occasionally required.
  • A testable extension: compare task completion time and user effort with and without the agent, since the paper reports accuracy but not the interaction cost of asking questions.
  • The declarative pipeline artifact could enable harmonization as a service: organizations publish their standards as target schemas, and agents map incoming data to those schemas on demand.
  • The reported value-mapping gain (0.58 to 0.68) is modest; a stronger demonstration would show that corrections generalize across multiple cohorts, not just one dataset.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper argues for agentic data harmonization, in which an LLM-based agent orchestrates data integration primitives, interacts with a user, and generates code, with the goal of producing reusable harmonization pipelines. The authors introduce Harmonia, a prototype built on the bdi-kit library and the Archytas/Beaker frameworks, and demonstrate it on a clinical dataset mapped to the GDC standard. The evaluation reports that Harmonia improves schema matching accuracy from 0.88 to 1.00 and value mapping accuracy from 0.58 to 0.68 relative to baseline bdi-kit methods. The remainder of the paper frames open problems in agent evaluation, primitive uncertainty, robustness, interaction, provenance, and pipeline optimization.

Significance. If the quantitative claims were supported, the paper would offer a useful proof of concept for combining LLM reasoning with classical data integration primitives, and the declarative pipeline specification would be a step toward reproducible harmonization. The paper is transparent in listing limitations and makes its code and demonstration available, which is a strength. However, the single use case and underspecified evaluation protocol mean that the empirical contribution is not yet established; the principal value at present is the vision and the design lessons rather than the reported performance numbers.

major comments (3)
  1. [Section 4, Evaluation paragraph and Table 1] The evaluation is underspecified and does not support the stated conclusion that "Harmonia achieved the best performance across both tasks." The authors do not define the ground truth against which accuracy, precision, recall, and F1 were computed; they do not state how many columns or value pairs were scored; they do not report the number of repeated runs or any variance; and they do not name which bdi-kit methods served as baselines. Please provide a complete protocol, report statistics over multiple runs, and, ideally, include additional datasets or at least a description of how the single dataset was selected.
  2. [Section 5, Robustness and Reliability] This section notes that the LLM "occasionally failed to do so (even when provided with the same prompts)," yet Section 4 presents a single successful trace with no failure analysis or success rate. Because the central mechanism is LLM-based error detection and correction, the absence of repeated-trial data leaves open the possibility that the reported gains are a selection artifact. Please add repeated-run statistics and characterize the observed failure modes and their frequency.
  3. [Section 4, Evaluation and Section 1, Contributions] The comparison against bdi-kit baselines is potentially circular because the primitives and the baseline methods come from the same library, and the LLM's "corrections" are not checked against an independent gold standard. The paper should either use an established schema-matching or value-mapping benchmark with documented ground truth or explicitly reframe Table 1 as an illustrative trace rather than a performance measurement. As written, the claim "achieved the best performance" overstates what the evidence shows.
minor comments (6)
  1. [Section 4, Use Case] The transcript uses the symbols "/user" and "♂robot," which may not render consistently in all PDF viewers; consider using plain text labels such as "User:" and "Agent:".
  2. [Section 5, Data Harmonization Pipelines] "maximining" should be "maximizing," and "a non-trivial tasks" should be "a non-trivial task."
  3. [Section 2, Preliminaries] "direct acyclic graphs" should be "directed acyclic graphs."
  4. [References] References [2] and [24] do not include complete publication information or URLs; please add the repository or DOI links to make the code and tooling easier to verify.
  5. [Section 4, Data Integration Primitives] The sentence "Thetarget parameter can either be..." contains a missing space; it should read "The target parameter can either be...".
  6. [Figure 2(a) and Section 3] The caption states that solid lines represent components implemented in Harmonia, but the provenance DB is described in the architecture and is not clearly identified as implemented in the evaluated prototype; please clarify which components were actually exercised in the use case.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular dependency: Harmonia's reported gains are empirical and not forced by construction; self-citations are disclosed system components, not load-bearing evidence.

full rationale

The paper's central claim is a vision and a prototype demonstration, not a formal derivation. Harmonia's evaluation compares the full agent against bdi-kit baselines executed without agent support, and the reported improvements (schema matching accuracy 0.88 to 1.00, value mapping accuracy 0.58 to 0.68) are empirical outcomes of running the system, not identities derived from the definitions. The LLM corrections are made against the external GDC standard vocabulary rather than against the LLM's own output as ground truth; the paper describes the LLM querying 'the list of acceptable values in a GDC attribute' to select the correct value. The self-citations to bdi-kit and Harmonia are engineering dependencies that are explicitly disclosed as the authors' own open-source components; they do not function as an unverified authority invoked to force a conclusion. The paper also acknowledges in Section 5 that the LLM 'occasionally failed' to fix incorrect mappings even with the same prompts, which undercuts any suggestion that the single successful trace is a tautological guarantee. The evaluation is underspecified and preliminary, and the reported numbers would need a larger, clearly ground-truthed study to be convincing, but these are correctness and reproducibility concerns, not circularity. No equation, definition, or evaluation step reduces to its own input, so the circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no fitted numerical parameters. The load-bearing assumptions are about the reliability of external components: the bdi-kit primitives, GPT-4o's domain knowledge, and the GDC standard as the target schema. These are stated or implicit in the system description and use case. Harmonia is an assembled software system, not a new physical entity.

assumptions (3)
  • domain assumption bdi-kit primitives for schema matching and value mapping are correctly implemented and composable.
    The paper relies on bdi-kit for all underlying integration algorithms; if these are incorrect, Harmonia inherits their errors (Section 4, Data Integration Primitives).
  • domain assumption GPT-4o can evaluate and correct mapping outputs from its general knowledge without fine-tuning.
    The prototype's error correction depends on the LLM's latent knowledge, as in the use case where it corrects a wrong match (Section 4, Use Case). The authors themselves flag brittleness in Section 5.
  • domain assumption The GDC standard is the appropriate target canonical schema.
    Definition 1 assumes an external canonical representation; the use case treats GDC as given (Section 2, Definition 1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Interactive Data Harmonization with LLM Agents: Opportunities and Challenges." pith.science (2026). https://pith.science/paper/DUG7ZKYN

@misc{pith2026250207132,
  author       = {Pith},
  title        = {Pith review of: Interactive Data Harmonization with LLM Agents: Opportunities and Challenges},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DUG7ZKYN}},
  note         = {Machine review of arXiv:2502.07132}
}
read the original abstract

Data harmonization is an essential task that entails integrating datasets from diverse sources. Despite years of research in this area, it remains a time-consuming and challenging task due to schema mismatches, varying terminologies, and differences in data collection methodologies. This paper presents the case for agentic data harmonization as a means to both empower experts to harmonize their data and to streamline the process. We introduce Harmonia, a system that combines LLM-based reasoning, an interactive user interface, and a library of data harmonization primitives to automate the synthesis of data harmonization pipelines. We demonstrate Harmonia in a clinical data harmonization scenario, where it helps to interactively create reusable pipelines that map datasets to a standard format. Finally, we discuss challenges and open problems, and suggest research directions for advancing our vision.

Figures

Figures reproduced from arXiv: 2502.07132 by the authors.

Figure 1
Figure 1. Domain of attributes in different data sources. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. (a) Components of an interactive agentic data harmonization system. Solid lines represent components implemented [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Some bdi-kit functions integrated in Harmonia. To implement tool calling, we used the Archytas [2], an open￾source library for building AI agents based on the ReAct frame￾work [73], and used the GPT-4o model (i.e., no fine-tuning is in￾volved). Instead of using a fixed pipeline that invokes the LLM, we implement a dynamic agentic loop, as described in Figure 2b. The LLM has the flexibility to break down the task int… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: A snippet from the mapping specification generated [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Ontology- and LLM-based Data Harmonization for Federated Learning in Healthcare

    cs.LG 2025-05 conditional novelty 4.0 of 10

    An ontology-retrieval plus LLM-adjudication pipeline maps EHR outcomes to MONDO/HPO codes with 78% to 92% agreement against a human expert reviewer.

Reference graph

Works this paper leans on

82 extracted references · 60 canonical work pages · cited by 1 Pith paper

  1. [1]

    Hidir Aras, Norbert Fuhr, Seung-won Hwang, Ander de Keijzer, Friederike Klan, Hans-Joachim Lenz, Tom Matthé, Heinz Schweppe, Mirco Stern, and Guy De Tré

  2. [2]

    Archytas: A Tools Interface for AI Agents

    archytas [n.d.]. Archytas: A Tools Interface for AI Agents

  3. [3]

    Christopher Barrie, Elli Palaiologou, and Petter Törnberg. 2024. Prompt sta- bility scoring for text annotation with large language models. arXiv preprint arXiv:2407.02039

    arXiv 2024

  4. [4]

    The bdi-kit data harmonization library

    bdi-kit [n.d.]. The bdi-kit data harmonization library. https://github.com/VIDA- NYU/bdi-kit

    work page

  5. [5]

    Beaker-Kernel: Contextually-aware notebooks with built-in AI assistant

    beaker [n.d.]. Beaker-Kernel: Contextually-aware notebooks with built-in AI assistant. https://github.com/jataware/beaker-kernel

    work page

  6. [6]

    Laure Berti-Equille. 2019. Learn2clean: Optimizing the sequence of tasks for web data preparation. In The world wide web conference . 2580–2586

    work page 2019

  7. [7]

    Liwei Cao, Chen Huang, Daniel Cui Zhou, Yingwei Hu, T Mamie Lih, Sara R Savage, Karsten Krug, David J Clark, Michael Schnaubelt, Lijun Chen, et al. 2021. Proteogenomic characterization of pancreatic ductal adenocarcinoma. Cell 184, 19 (2021), 5031–5052

    work page 2021

  8. [8]

    Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. 2024. Mle- bench: Evaluating machine learning agents on machine learning engineering. arXiv preprint arXiv:2410.07095

    arXiv 2024

Show all 82 references
  1. [9]

    Adriane Chapman, Paolo Missier, Giulia Simonelli, and Riccardo Torlone. 2020. Capturing and querying fine-grained provenance of preprocessing pipelines in data science. Proc. VLDB Endow. 14, 4 (Dec. 2020), 507–520. https://doi.org/10. 14778/3436905.3436911

    2020

  2. [10]

    Sibei Chen, Yeye He, Weiwei Cui, Ju Fan, Song Ge, Haidong Zhang, Dongmei Zhang, and Surajit Chaudhuri. 2024. Auto-Formula: Recommend Formulas in Spreadsheets using Contrastive Learning for Table Representations. Proc. ACM Manag. Data 2, 3, Article 122 (May 2024), 27 pages. htt...

    2024

  3. [11]

    Yibin Chen, Yifu Yuan, Zeyu Zhang, Yan Zheng, Jinyi Liu, Fei Ni, and Jianye Hao. 2024. SheetAgent: A Generalist Agent for Spreadsheet Reasoning and Manipulation via Large Language Models. CoRR abs/2403.03636 (2024). https: //doi.org/10.48550/ARXIV.2403.03636

    2024 doi arXiv

  4. [12]

    Cindy Cheng, Luca Messerschmidt, Isaac Bravo, Marco Waldbauer, Rohan Bhavikatti, Caress Schenk, Vanja Grujic, Tim Model, Robert Kubinec, and Joan Barceló. 2024. A general primer for data harmonization. Scientific data 11, 1 (2024), 152

    2024

  5. [13]

    Vassilis Christophides, Vasilis Efthymiou, Themis Palpanas, George Papadakis, and Kostas Stefanidis. 2020. An overview of end-to-end entity resolution for big data. ACM Computing Surveys (CSUR) 53, 6 (2020), 1–42

    2020

  6. [14]

    David J Clark, Saravana M Dhanasekaran, Francesca Petralia, Jianbo Pan, Xiaoyu Song, Yingwei Hu, Felipe da Veiga Leprevost, Boris Reva, Tung-Shing M Lih, Hui-Yin Chang, et al. 2019. Integrated proteogenomic characterization of clear cell renal cell carcinoma. Cell 179, 4 (2019...

    2019

  7. [15]

    Arash Dargahi Nobari and Davood Rafiei. 2024. DTT: An Example-Driven Tabular Transformer for Joinability by Leveraging Large Language Models. Proc. ACM Manag. Data 2, 1, Article 24 (March 2024), 24 pages. https://doi.org/10.1145/ 3639279

    2024

  8. [16]

    Xin Luna Dong, Alon Halevy, and Cong Yu. 2009. Data integration with uncer- tainty. The VLDB Journal 18, 2 (April 2009), 469–500. https://doi.org/10.1007/ s00778-008-0119-9

    2009

  9. [17]

    Yongchao Dou, Lizabeth Katsnelson, Marina A Gritsenko, Yingwei Hu, Boris Reva, Runyu Hong, Yi-Ting Wang, Iga Kolodziejczak, Rita Jui-Hsien Lu, Chia- Feng Tsai, et al. 2023. Proteogenomic insights suggest druggable pathways in endometrial carcinoma. Cancer cell 41, 9 (2023), 1586–1605

    2023

  10. [18]

    Yongchao Dou, Emily A Kawaler, Daniel Cui Zhou, Marina A Gritsenko, Chen Huang, Lili Blumenberg, Alla Karpova, Vladislav A Petyuk, Sara R Savage, Shankha Satpathy, et al. 2020. Proteogenomic characterization of endometrial carcinoma. Cell 180, 4 (2020), 729–748

    2020

  11. [19]

    Reza Esfandiarpoor and Stephen Bach. 2024. Follow-Up Differential Descriptions: Language Models Resolve Ambiguities for Image Classification. In International Conference on Learning Representations (ICLR) . https://openreview.net/forum? id=g6rZtxaXRm

    2024

  12. [20]

    Meihao Fan, Xiaoyue Han, Ju Fan, Chengliang Chai, Nan Tang, Guoliang Li, and Xiaoyong Du. 2024. Cost-effective in-context learning for entity resolution: A design space exploration. In 2024 IEEE 40th International Conference on Data Engineering (ICDE). IEEE, 3696–3709

    2024

  13. [22]

    Benjamin Feuer, Yurong Liu, Chinmay Hegde, and Juliana Freire. 2024. ArcheType: A Novel Framework for Open-Source Column Type Annotation using Large Language Models. Proc. VLDB Endow. 17, 9 (2024), 2279–2292. https://www.vldb. org/pvldb/vol17/p2279-freire.pdf

    2024

  14. [23]

    Michael A Gillette, Shankha Satpathy, Song Cao, Saravana M Dhanasekaran, Suhas V Vasaikar, Karsten Krug, Francesca Petralia, Yize Li, Wen-Wei Liang, Boris Reva, et al . 2020. Proteogenomic characterization reveals therapeutic vulnerabilities in lung adenocarcinoma. Cell 182, 1...

    2020

  15. [24]

    Harmonia: An Interactive Data Harmonization Agent

    Harmonia [n.d.]. Harmonia: An Interactive Data Harmonization Agent. https: //github.com/VIDA-NYU/harmonia/

  16. [25]

    Harmonia Demonstration. [n.d.]. Harmonia: Interactive Data Harmonization with LLM Agents (YouTube). https://www.youtube.com/watch?v=D25x0B_xs3c

  17. [26]

    Cheng-Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023. Distill- ing Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes. In Findings of the ...

    2023 doi

  18. [27]

    Xueyu Hu, Ziyu Zhao, Shuang Wei, Ziwei Chai, Qianli Ma, Guoyin Wang, Xuwu Wang, Jing Su, Jingjing Xu, Ming Zhu, et al. 2024. Infiagent-dabench: Evaluating agents on data analysis tasks. arXiv preprint arXiv:2401.05507

    2024 arXiv

  19. [28]

    Chen Huang, Lijun Chen, Sara R Savage, Rodrigo Vargas Eguez, Yongchao Dou, Yize Li, Felipe da Veiga Leprevost, Eric J Jaehnig, Jonathan T Lei, Bo Wen, et al

  20. [29]

    Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. 2024. MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation. In Interna- tional Conference on Machine Learning

    2024

  21. [30]

    Yue Huang, Lichao Sun, Haoran Wang, Siyuan Wu, Qihui Zhang, Yuan Li, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, Hanchi Sun, Zhengliang Liu, Yixin Liu, Yijue Wang, Zhikun Zhang, Bertie Vidgen, Bhavya Kailkhura, Caiming Xiong, Chaowei Xiao, Chunyuan Li, Eric P....

    2024

  22. [31]

    National Cancer Institute. 2024. GDC Data Model. https://gdc.cancer.gov/ developers/gdc-data-model. Accessed: 2024-09-20

    2024

  23. [32]

    Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen, and Xia Hu. 2024. Llm maybe longlm: Self-extend llm context window without tuning. arXiv preprint arXiv:2401.01325

    2024 arXiv

  24. [33]

    Project Jupyter

    jupyter [n.d.]. Project Jupyter. https://jupyter.org/

  25. [34]

    Moe Kayali, Anton Lykov, Ilias Fountalis, Nikolaos Vasiloglou, Dan Olteanu, and Dan Suciu. 2024. CHORUS: Foundation Models for Unified Data Discovery and Exploration. Proc. VLDB Endow. 17, 8 (2024), 2104–2114. https://www.vldb.org/ pvldb/vol17/p2104-kayali.pdf

    2024

  26. [35]

    Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan A, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts

  27. [36]

    Kolaitis

    Phokion G. Kolaitis. 2018. Reflections on Schema Mappings, Data Exchange, and Metadata Management. In Proceedings of the 37th ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems (Houston, TX, USA) (PODS ’18). Association for Computing Machinery, New York, NY, U...

    2018

  28. [37]

    David Koop, Carlos E Scheidegger, Steven P Callahan, Juliana Freire, and Cláu- dio T Silva. 2008. Viscomplete: Automating suggestions for visualization pipelines. IEEE Transactions on Visualization and Computer Graphics14, 6 (2008), 1691–1698

    2008

  29. [38]

    Christos Koutras, George Siachamis, Andra Ionescu, Kyriakos Psarakis, Jerry Brons, Marios Fragkoulis, Christoph Lofi, Angela Bonifati, and Asterios Katsi- fodimos. 2021. Valentine: Evaluating matching techniques for dataset discovery. In 2021 IEEE 37th International Conference...

    2021

  30. [39]

    Karsten Krug, Eric J Jaehnig, Shankha Satpathy, Lili Blumenberg, Alla Karpova, Meenakshi Anurag, George Miles, Philipp Mertins, Yifat Geffen, Lauren C Tang, et al. 2020. Proteogenomic landscape of breast cancer tumorigenesis and targeted NOVAS ’25, June 22–27, 2025, Berlin, Ge...

    2020

  31. [40]

    LangChain

    langchain [n.d.]. LangChain. https://www.langchain.com/

  32. [41]

    Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. Camel: Communicative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems 36 (2023), 51991–52008

    2023

  33. [42]

    Peng Li, Yeye He, Cong Yan, Yue Wang, and Surajit Chaudhuri. 2023. Auto- Tables: Synthesizing Multi-Step Transformations to Relationalize Tables without Using Examples. PVLDB 16, 11 (July 2023), 3391–3403. https://doi.org/10.14778/ 3611479.3611534

    2023

  34. [43]

    Yize Li, Yongchao Dou, Felipe Da Veiga Leprevost, Yifat Geffen, Anna P Calinawan, François Aguet, Yo Akiyama, Shankara Anand, Chet Birger, Song Cao, et al. 2023. Proteogenomic data and resources for pan-cancer analysis. Cancer cell 41, 8 (2023), 1397–1406

    2023

  35. [44]

    Yurong Liu, Eduardo Pena, Aecio Santos, Eden Wu, and Juliana Freire. 2025. Magneto: Combining Small and Large Language Models for Schema Matching. Proceedings of the VLDB Endowment 18, 8 (2025), 2681–2694. https://doi.org/10. 14778/3742728.3742757

    2025

  36. [45]

    Yurong Liu, Aécio Santos, Eduardo HM Pena, Roque Lopez, Eden Wu, and Juliana Freire. 2024. Enhancing Biomedical Schema Matching with LLM-based Training Data Generation. In NeurIPS 2024 Third Table Representation Learning Workshop

    2024

  37. [46]

    Roque Lopez, Raoni Lourenço, Remi Rampin, Sonia Castelo, Aécio SR Santos, Jorge Henrique Piazentin Ono, Claudio Silva, and Juliana Freire. 2023. AlphaD3M: An Open-Source AutoML Library for Multiple ML Tasks. In International Confer- ence on Automated Machine Learning . PMLR, 22–1

    2023

  38. [47]

    Xuezhe Ma, Xiaomeng Yang, Wenhan Xiong, Beidi Chen, Lili Yu, Hao Zhang, Jonathan May, Luke Zettlemoyer, Omer Levy, and Chunting Zhou. 2024. Mega- lodon: Efficient llm pretraining and inference with unlimited context length. arXiv preprint arXiv:2404.08801

    2024 arXiv

  39. [48]

    Zeyao Ma, Bohan Zhang, Jing Zhang, Jifan Yu, Xiaokang Zhang, Xiaohan Zhang, Sijia Luo, Xi Wang, and Jie Tang. 2024. SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation. In The Thirty-eight Confer- ence on Neural Information Processing Systems Datasets and...

    2024

  40. [49]

    Jason E McDermott, Osama A Arshad, Vladislav A Petyuk, Yi Fu, Marina A Gritsenko, Therese R Clauss, Ronald J Moore, Athena A Schepmoes, Rui Zhao, Matthew E Monroe, et al. 2020. Proteogenomic characterization of ovarian HGSC implicates mitotic kinases, replication stress in obs...

    2020

  41. [50]

    Aída Muñoz Monjas, David Rubio Ruiz, David Pérez del Rey, and Matvey B. Palchuk. 2025. Enhancing real world data interoperability in healthcare: A methodological approach to laboratory unit harmonization. International Journal of Medical Informatics 193 (2025), 105665. https:/...

    2025 doi

  42. [51]

    Orr, and Christopher Ré

    Avanika Narayan, Ines Chami, Laurel J. Orr, and Christopher Ré. 2022. Can Foundation Models Wrangle Your Data? Proc. VLDB Endow. 16, 4 (2022), 738– 746

    2022

  43. [52]

    Society of Automotive Engineers (SAE). 2018. Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles (J3016_201806)

    2018

  44. [53]

    Tiernan Ray. [n.d.]. Microsoft has over a million paying Github Copilot users: CEO Nadella. https://www.zdnet.com/article/microsoft-has-over-a-million-paying- github-copilot-users-ceo-nadella/

  45. [54]

    Davis, Constantine Arnold, Yaniv Gur, and Deepavali Bhagwat

    Lukas Rupprecht, James C. Davis, Constantine Arnold, Yaniv Gur, and Deepavali Bhagwat. 2020. Improving reproducibility of data science pipelines through transparent provenance capture. Proc. VLDB Endow. 13, 12 (Aug. 2020), 3354–3368. https://doi.org/10.14778/3415478.3415556

    2020

  46. [55]

    Stuart J Russell and Peter Norvig. 2016. Artificial intelligence: a modern approach . Pearson

    2016

  47. [56]

    Aécio Santos, Sonia Castelo, Cristian Felix, Jorge Piazentin Ono, Bowen Yu, Sungsoo Ray Hong, Cláudio T Silva, Enrico Bertini, and Juliana Freire. 2019. Visus: An interactive system for automatic machine learning model building and curation. In Proceedings of the Workshop on H...

    2019

  48. [57]

    Shankha Satpathy, Karsten Krug, Pierre M Jean Beltran, Sara R Savage, Francesca Petralia, Chandan Kumar-Sinha, Yongchao Dou, Boris Reva, M Harry Kane, Shayan C Avanessian, et al. 2021. A proteogenomic portrait of lung squamous cell carcinoma. Cell 184, 16 (2021), 4348–4371

    2021

  49. [58]

    Zeyuan Shang, Emanuel Zgraggen, Benedetto Buratti, Ferdinand Kossmann, Philipp Eichmann, Yeounoh Chung, Carsten Binnig, Eli Upfal, and Tim Kraska

  50. [59]

    scikit-learn: Pipelines and composite estimators

    sklearn-pipelines [n.d.]. scikit-learn: Pipelines and composite estimators. https: //scikit-learn.org/1.6/modules/compose.html

  51. [60]

    Rickard Stureborg, Dimitris Alikaniotis, and Yoshi Suhara. 2024. Large language models are inconsistent and biased evaluators. arXiv preprint arXiv:2405.01724 (2024)

    2024 arXiv

  52. [61]

    Yuan Sui, Mengyu Zhou, Mingjie Zhou, Shi Han, and Dongmei Zhang. 2024. Ta- ble Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Study (WSDM ’24). Association for Computing Ma- chinery, New York, NY, USA, 645–654. https://doi.org/...

    2024

  53. [62]

    Immanuel Trummer. 2022. CodexDB: Synthesizing code for query processing from natural language instructions using GPT-3 Codex. Proceedings of the VLDB Endowment 15, 11 (2022), 2921–2928

    2022

  54. [63]

    Jianhong Tu, Ju Fan, Nan Tang, Peng Wang, Guoliang Li, Xiaoyong Du, Xiaofeng Jia, and Song Gao. 2023. Unicorn: A unified multi-tasking model for supporting matching tasks in data integration. Proceedings of the ACM on Management of Data 1, 1 (2023), 1–26

    2023

  55. [64]

    Suhas Vasaikar, Chen Huang, Xiaojing Wang, Vladislav A Petyuk, Sara R Sav- age, Bo Wen, Yongchao Dou, Yun Zhang, Zhiao Shi, Osama A Arshad, et al

  56. [65]

    Jin Wang, Yuliang Li, and Wataru Hirota. 2021. Machamp: A generalized entity matching benchmark. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management . 4633–4642

    2021

  57. [66]

    Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. 2024. A survey on large language model based autonomous agents. Frontiers of Computer Science 18, 6 (2024), 186345

    2024

  58. [67]

    Liang-Bo Wang, Alla Karpova, Marina A Gritsenko, Jennifer E Kyle, Song Cao, Yize Li, Dmitry Rykunov, Antonio Colaprico, Joseph H Rothstein, Runyu Hong, et al. 2021. Proteogenomic and metabolomic characterization of human glioblas- toma. Cancer cell 39, 4 (2021), 509–528

    2021

  59. [68]

    Xiaolan Wang, Laura Haas, and Alexandra Meliou. 2018. Explaining Data Inte- gration. IEEE Data Engineering Bulletin 41, 2 (June 2018), 47–58

    2018

  60. [69]

    Cell 177, 4 (2019), 1035–1049

    Proteogenomic analysis of human colon cancer reveals new therapeutic opportunities. Cell 177, 4 (2019), 1035–1049

    2019

  61. [70]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824–24837

    2022

  62. [71]

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. 2023. Autogen: Enabling next-gen llm applications via multi-agent conversation framework.arXiv preprint arXiv:2308.08155 (2023)

    2023 arXiv

  63. [72]

    Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. 2023. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.07864 (2023)

    2023 arXiv

  64. [73]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2022. ReAct: Synergizing Reasoning and Acting in Language Models. In NeurIPS 2022 Foundation Models for Decision Making Workshop . https:// openreview.net/forum?id=tvI4u1ylcqs

    2022

  65. [74]

    Zilong Wang, Hao Zhang, Chun-Liang Li, Julian Martin Eisenschlos, Vincent Perot, Zifeng Wang, Lesly Miculicich, Yasuhisa Fujii, Jingbo Shang, Chen-Yu Lee, and Tomas Pfister. 2024. Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understanding. In The Twelfth In...

    2024

  66. [75]

    Michael JQ Zhang and Eunsol Choi. 2023. Clarify when necessary: Resolving ambiguity through interaction with lms. arXiv preprint arXiv:2311.09469 (2023)

    2023 arXiv

  67. [76]

    Yuge Zhang, Qiyang Jiang, Xingyu Han, Nan Chen, Yuqing Yang, and Kan Ren

  68. [77]

    Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. 2024. Explainability for Large Language Models: A Survey. ACM Trans. Intell. Syst. Technol. 15, 2, Article 20 (Feb. 2024), 38 pages. https://doi.org/10.1145/3639372

    2024 doi

  69. [79]

    Bowen Yu and Cláudio T Silva. 2019. FlowSense: A natural language inter- face for visual data exploration within a dataflow system. IEEE transactions on visualization and computer graphics 26, 1 (2019), 1–11

    2019

  70. [82]

    arXiv preprint arXiv:2402.17168 (2024)

    Benchmarking Data Science Agents. arXiv preprint arXiv:2402.17168 (2024)

    2024 arXiv

  71. [2009]

    08421 Working Group: Explanation. In Uncertainty Management in Infor- mation Systems (Dagstuhl Seminar Proceedings (DagSemProc)) , Christoph Koch, Birgitta König-Ries, Volker Markl, and Maurice van Keulen (Eds.), Vol. 8421. Schloss Dagstuhl – Leibniz-Zentrum für Informatik, Da...

    doi

  72. [2019]

    In Proceedings of the 2019 international conference on management of data

    Democratizing data science through interactive curation of ml pipelines. In Proceedings of the 2019 international conference on management of data . 1171– 1188

    2019

  73. [2021]

    Cancer cell 39, 3 (2021), 361–379

    Proteogenomic insights into the biology and treatment of HPV-negative head and neck squamous cell carcinoma. Cancer cell 39, 3 (2021), 361–379

    2021

  74. [2024]

    In The Twelfth International Conference on Learning Representations

    DSPy: Compiling Declarative Language Model Calls into State-of-the-Art Pipelines. In The Twelfth International Conference on Learning Representations . https://openreview.net/forum?id=sY5N0zY5Od

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.