REVIEW 4 major objections 5 minor 35 references
Something's Fishy In The Data Lake: A Critical Re-evaluation of Table Union Search Benchmarks
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Simple baselines match or beat specialized table union search models on all five major benchmarks, showing that reported scores reward dataset artifacts rather than semantic reasoning.
desk verdict A useful benchmark critique whose overlap and baseline findings are solid, but whose ground truth audit is more provisional than the conclusion suggests. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central instrument is a battery of deliberately simple baselines used as diagnostic probes: bag-of-words vectorizers (hashing, TF-IDF, and count-based) and pre-trained Sentence-BERT embeddings over column names and sampled values. The overlap coefficient, computed with exact column-name and data-value set intersection normalized by the smaller set, quantifies surface similarity between ground-truth unionable pairs. The ground-truth false-positive rate (GTFP@k) and ground-truth false-negative rate (GTFN@k) measure how often a strong method such as Sentence-BERT disagrees with the ground truth at top ranks, flagging likely label incompleteness or error. These tools together expose which information source—lexical overlap, general vocabulary, or semantic reasoning—actually drives a benchmark’s scores.
What would settle it
Manually audit a large random sample (several hundred pairs) of Sentence-BERT-versus-ground-truth disagreements in the LB-OpenData and LB-Webtable benchmarks at top-2 and top-3 ranks, using multiple independent human annotators who do not know the retrieval rankings; if fewer than half of those pairs are judged unionable, the claim of non-negligible ground-truth incompleteness collapses.
Extended reading notes
Core claim
The central claim is that current table union search benchmark scores are heavily influenced by dataset-specific characteristics and fail to isolate gains from semantic understanding. Empirically, the paper shows that on partitioning-based benchmarks over 90% of ground-truth unionable pairs share at least half of their exact column names, and 45–60% share at least half of their string tokens, which lets bag-of-words methods reach near-perfect precision. On corpus-derived benchmarks such as PYLON, Sentence-BERT variants outperform specialized models like Starmie, indicating the vocabulary is general rather than domain-specific. For UGEN and LAKE BENCH, the paper’s ground-truth false-positive and false-negative rates and an LLM-based audit find substantial label noise: 24.8% of UGENV1 pairs labeled unionable were judged non-unionable by the LLM adjudicator, and manual inspection of 20 LAKE BENCH disagreements found 19 genuinely unionable pairs missing from the ground truth. The authors conclude that benchmark scores reward adaptation to artifacts and propose benchmarks with reduced artificial overlap, controlled semantic heterogeneity, and richer adjudicated ground-truth labels.
Load-bearing premise
The conclusion that UGEN and LAKE BENCH ground truths are incomplete rests on trusting the strong search method’s top-ranked disagreements with the labels as genuinely unionable pairs, so if those high-ranked candidates are not actually unionable, the measured disagreement rates say nothing about ground-truth quality.
Editorial extensions
If this is right
- Reported precision and recall numbers on these benchmarks should not be read as evidence of semantic understanding, because they can be reproduced by surface matching and pre-trained vocabulary coverage.
- The maximum achievable recall on UGEN and LAKE BENCH is capped by missing ground-truth positives, so absolute scores and cross-benchmark comparisons are unreliable indicators of model quality.
- Future benchmarks should reduce artificial schema and value overlap, introduce domain vocabularies absent from pre-training corpora, and replace binary labels with multi-stage adjudicated ground truth.
- A low-cost baseline suite of lexical and general embedding methods should be reported alongside any new TUS benchmark as a sanity check for whether the benchmark isolates semantic gains.
- Specialized models should be evaluated in terms of their improvement over such baselines rather than by absolute ranking, to make progress claims meaningful.
Reading between the lines
- If the paper’s diagnosis generalizes beyond these five benchmarks, progress claims in related data-discovery tasks such as join search and dataset search may be similarly inflated by benchmark artifacts, and simple baselines should be routinely added to those evaluations as controls.
- The disagreement-based auditing procedure described for LAKE BENCH—using a strong retrieval method to propose candidate label errors and an LLM plus spot-checked human review to adjudicate—could serve as a scalable cleaning pipeline for noisy unionability labels in other benchmark collections.
- A concrete testable prediction follows: a benchmark constructed according to the paper’s criteria (domain-specific vocabulary, schema diversity, and adjudicated ground truth) would produce a large performance gap between specialized models and the simple baselines, reversing the pattern observed here.
- The paper leaves open an operational definition of unionability; a task-oriented definition, in which a pair counts as unionable only if a downstream analysis user would benefit from combining the tables, would make ground truth less arbitrary and better aligned with practical data-lake use.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper critically re-examines five families of table union search (TUS) benchmarks (TUSSmall/TUSLarge, SANTOS, PYLON, UGEN, and LAKE BENCH derivatives) and argues that reported scores are inflated by benchmark artifacts rather than by genuine semantic understanding. The authors measure exact column-name and value overlap among ground-truth unionable pairs, run simple lexical baselines and pre-trained SBERT embeddings against specialized methods (Starmie, HEARTS, TabSketchFM), and investigate ground-truth reliability using SBERT as a reference, an LLM-as-a-judge for UGEN, and a 20-pair manual inspection for LAKE BENCH. They find high overlap in partitioning-based benchmarks, strong performance of lexical and general embeddings, and evidence of label noise/incompleteness in UGEN and LAKE BENCH. The paper concludes that current benchmarks fail to isolate semantic gains and proposes design criteria for future benchmarks, including reduced artificial overlap, domain complexity, and richer ground-truth representations. The authors release code and preprocessed benchmark data.
Significance. If the main findings hold, the paper is a timely and useful service to the TUS community: it provides reproducible evidence that several widely used benchmarks are partially solvable by surface-level features, and it offers concrete design principles for more discriminative benchmarks. The overlap analysis (Figure 1) and the consistent performance of simple baselines across diverse benchmarks are convincing and are backed by released code and data. The ground-truth investigation is the weakest pillar: it relies on a small manual sample and an LLM judge conditioned on the very benchmark it audits, so the quantitative claims of 'non-negligible noise and incompleteness' are provisional. The authors are appropriately candid about this in the Limitations section. Overall the paper is likely to influence benchmark design and evaluation practice in table discovery, but the ground-truth claims need stronger external validation before they can be accepted at face value.
major comments (4)
- [Section 5.2, Table 4] The GTFP/GTFN rates treat SBERT(V+C) as a reliable relevance oracle: high disagreement between SBERT's top-k results and the ground truth is interpreted as evidence of ground-truth incompleteness or incorrectness. This assumption is load-bearing for the claim of 'non-negligible incompleteness' in LAKE BENCH, yet the paper itself notes that SBERT can be misled (one of the 20 manual pairs was correctly non-unionable despite SBERT ranking it highly). The direct manual evidence consists of only 20 pairs, with no confidence interval or inter-annotator agreement. To support the quantitative conclusion, the authors should report a larger blinded manual audit (e.g., 200 pairs), provide the precision of the reference method on a labeled sample, and show that disagreements are not dominated by reference-method errors.
- [Section 5.2, Table 5 and Appendix D] The LLM-as-a-judge audit of UGEN is partially circular: the prompt conditions the LLM using hand-selected positive and negative examples taken from UGEN itself, so the adjudicator inherits the same unionability criteria that the audit is meant to check. The LLM can still reveal internal inconsistency, but it cannot validate the criteria. In addition, Table 5 as printed is uninterpretable: it contains two rows labeled 'Unionable | Non-unionable' (24.8% and 25.2% for UGEN V1) and no row for the agreement case 'Unionable | Unionable'. The row percentages only sum to 100% if the last row is actually 'Unionable | Unionable', so the table likely has a labeling typo that must be corrected before the 'notable noise' claim can be evaluated.
- [Section 4.1, Table 1] The LAKE BENCH preprocessing removes approximately 5,300 unreferenced files from LB-OpenData and more than 2.7 million from LB-Webtable. The authors state that this simplifies the benchmark because unreferenced tables would otherwise be false positives if retrieved, but this exclusion raises the recall ceiling and removes a substantial fraction of the retrieval corpus, changing the task and preventing direct comparison with original LAKE BENCH results. The conclusion that simple baselines are 'surprisingly competitive' on LAKE BENCH is therefore partly a property of the preprocessed variant. The paper should re-run or clearly qualify the scale claim, and at minimum report the number of remaining candidate tables per query after preprocessing so readers can gauge how much of the original benchmark difficulty remains.
- [Section 5.1, Table 2] The effectiveness results are reported as point estimates with no variance or significance testing. Several comparisons that the narrative describes as 'outperform' are within 0.01-0.02 (e.g., SANTOS P@10: TFIDF 0.99 vs. Starmie 0.98; LB-WebTable P@20: SBERT(V+C) 0.26 vs. Starmie 0.25). Since Starmie is retrained and HEARTS uses clustering with stochastic components, the ranking of methods could easily flip across seeds. The authors should provide standard deviations over multiple runs or at least bootstrap confidence intervals over query subsets to substantiate the claim that simple baselines match or exceed specialized methods.
minor comments (5)
- [Abstract] The abstract says simple baselines 'often outperforming more sophisticated approaches,' but Table 2 mostly shows ties or small differences on the partitioning-based benchmarks; consider saying 'matching or sometimes exceeding' to be more precise.
- [Section 3.1.b] The sentence 'which have with a high, however unverifiable, probability encountered similar content during pre-training' is grammatically awkward; suggest rewriting for clarity.
- [Section 5.2] The phrase '2 < k < 5' appears to mean k = 2,3,4 rather than a strict inequality; use '2 ≤ k ≤ 4' or simply 'for k = 2 to 4'.
- [Appendix C.1] The example captions in Figures 3 and 4 refer to 'UGENV1' and 'UGENV2', but the main text uses the names 'UGENV1' and 'UGENV2'; please unify the notation.
- [Table 3] The table formatting for the runtime results is very dense and the columns are difficult to align with the benchmark names; consider splitting into separate tables or using a landscape layout.
Circularity Check
No significant circularity: the benchmark-limitation claims rest on independent overlap measurements and off-the-shelf baselines, with external manual validation of the ground-truth audit.
full rationale
The paper's central claims are supported by direct measurements of overlap (Eqs. 1-2, Figure 1) and by comparing off-the-shelf lexical and SBERT baselines with specialized TUS methods; no fitted parameter is renamed as a prediction, and the main baseline evidence does not depend on the authors' own prior work. The ground-truth audit in Section 5.2 is admittedly heuristic: GTFP/GTFN assume a strong search method can localize label flaws, and the UGEN LLM judge is conditioned on examples drawn from UGEN itself, which is a self-referential audit design rather than an independent oracle. However, the paper supplements these probes with manual inspection of 20 LLM outputs and 20 LAKE BENCH disagreement pairs, providing external grounding, and it explicitly acknowledges the investigation is provisional in the Limitations section. The citation of HEARTS, the authors' own method, is only as one compared system, not as the justification for the benchmark-limitation conclusion. No load-bearing step reduces mathematically or by definition to its own inputs, so there is no significant circularity.
Assumptions & free parameters
assumptions (4)
- domain assumption Unionability is defined as vertical concatenation of tables representing the same information type (Nargesian et al., 2018).
- ad hoc to paper High performance of simple baselines indicates benchmark artifacts rather than genuine baseline capability.
- domain assumption The LLM (gemini-2.0-flash-thinking) provides reliable unionability judgments after few-shot conditioning.
- domain assumption Preprocessing LAKE BENCH (removing unreferenced tables, truncation to 1,000 rows) does not bias the relative performance comparison.
Cite this review
Pith. "Pith review of Something's Fishy In The Data Lake: A Critical Re-evaluation of Table Union Search Benchmarks." pith.science (2026). https://pith.science/paper/5DVUHPZX
@misc{pith2026250521329,
author = {Pith},
title = {Pith review of: Something's Fishy In The Data Lake: A Critical Re-evaluation of Table Union Search Benchmarks},
year = {2026},
howpublished = {\url{https://pith.science/paper/5DVUHPZX}},
note = {Machine review of arXiv:2505.21329}
}
read the original abstract
Recent table representation learning and data discovery methods tackle table union search (TUS) within data lakes, which involves identifying tables that can be unioned with a given query table to enrich its content. These methods are commonly evaluated using benchmarks that aim to assess semantic understanding in real-world TUS tasks. However, our analysis of prominent TUS benchmarks reveals several limitations that allow simple baselines to perform surprisingly well, often outperforming more sophisticated approaches. This suggests that current benchmark scores are heavily influenced by dataset-specific characteristics and fail to effectively isolate the gains from semantic understanding. To address this, we propose essential criteria for future benchmarks to enable a more realistic and reliable evaluation of progress in semantic table union search.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Jan-Micha Bodensohn, Ulf Brackmann, Liane Vogel, Anupam Sanghi, and Carsten Binnig. 2025. https://arxiv.org/abs/2504.10950 Unveiling challenges for llms in enterprise data engineering . Preprint, arXiv:2504.10950
arXiv 2025
-
[4]
Alex Bogatu, Alvaro A. A. Fernandes, Norman W. Paton, and Nikolaos Konstantinou. 2020. https://doi.org/10.1109/ICDE48307.2020.00067 D3L : Dataset Discovery in Data Lakes . In 2020 IEEE 36th International Conference on Data Engineering ( ICDE ) , pages 709--720. ArXiv:2011.10427 [cs]
arXiv 2020
-
[5]
Allaa Boutaleb, Alaa Almutawa, Bernd Amann, Rafael Angarita, and Hubert Naacke. 2025. https://openreview.net/forum?id=XgRbxO9pLJ HEARTS : Hypergraph-based related table search . In ELLIS workshop on Representation Learning and Generative Models for Structured Data
work page 2025
-
[6]
Riccardo Cappuzzo, Gaël Varoquaux, Aimee Coelho, and Paolo Papotti. 2024. https://doi.org/10.48550/arXiv.2402.06282 Retrieve, merge, predict: Augmenting tables with data lakes . CoRR, abs/2402.06282
-
[7]
Sonia Castelo, Rémi Rampin, Aécio Santos, Aline Bessa, Fernando Chirigati, and Juliana Freire. 2021. https://doi.org/10.14778/3476311.3476346 Auctus: a dataset search engine for data discovery and augmentation . Proceedings of the VLDB Endowment, 14(12):2791--2794
arXiv 2021
-
[8]
Pei Chen, Soumajyoti Sarkar, Leonard Lausen, Balasubramaniam Srinivasan, Sheng Zha, Ruihong Huang, and George Karypis. 2023. Hytrel: Hypergraph-enhanced tabular data representation learning. Advances in Neural Information Processing Systems, 36:32173--32193
work page 2023
Show all 35 references
-
[9]
Tianji Cong, Fatemeh Nargesian, and H. V. Jagadish. 2023. Pylon: Semantic table union search in data lakes. CoRR, abs/2301.04901
2023 arXiv
- [10]
-
[11]
Yuhao Deng, Chengliang Chai, Lei Cao, Qin Yuan, Siyuan Chen, Yanrui Yu, Zhaoze Sun, Junyi Wang, Jiajun Li, Ziqi Cao, Kaisen Jin, Chi Zhang, Yuqing Jiang, Yuanfang Zhang, Yuping Wang, Ye Yuan, Guoren Wang, and Nan Tang. 2024. https://doi.org/10.14778/3659437.3659448 LakeBench :...
2024
-
[12]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associat...
2019 doi
-
[13]
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. 2024. https://arxiv.org/abs/2401.08281 The faiss library
2024 arXiv
-
[14]
Grace Fan, Jin Wang, Yuliang Li, and Renée J. Miller. 2023 a . https://doi.org/10.1145/3555041.3589409 Table Discovery in Data Lakes : State -of-the-art and Future Directions . In Companion of the 2023 International Conference on Management of Data , pages 69--75, Seattle WA USA. ACM
2023
-
[15]
Grace Fan, Jin Wang, Yuliang Li, Dan Zhang, and Ren \' e e J. Miller. 2023 b . https://doi.org/10.14778/3587136.3587146 Semantics-aware dataset discovery from data lakes with contextualized column-based representation learning . Proc. VLDB Endow. , 16(7):1726--1739
2023
-
[16]
Daniel Gomm and Madelon Hulsebos. 2025. https://openreview.net/forum?id=rELWIvq2Qy Metadata matters in dense table retrieval . In ELLIS workshop on Representation Learning and Generative Models for Structured Data
2025
- [17]
-
[18]
Xuming Hu, Shen Wang, Xiao Qin, Chuan Lei, Zhengyuan Shen, Christos Faloutsos, Asterios Katsifodimos, George Karypis, Lijie Wen, and Philip S. Yu. 2023. https://doi.org/10.18653/v1/2023.findings-acl.233 AUTOTUS : Automatic Table Union Search with Tabular Representation Learnin...
2023 doi
- [19]
-
[20]
Madelon Hulsebos, C agatay Demiralp, and Paul Groth. 2023. https://doi.org/10.1145/3588710 Gittables: A large-scale corpus of relational tables . Proc. ACM Manag. Data , 1(1):30:1--30:17
2023 doi
-
[21]
Madelon Hulsebos, Wenjing Lin, Shreya Shankar, and Aditya Parameswaran. 2024. https://doi.org/10.1145/3665939.3665959 It Took Longer than I was Expecting : Why is Dataset Search Still so Hard ? In Proceedings of the 2024 Workshop on Human - In -the- Loop Data Analytics , pages...
2024
-
[22]
Miller, and Mirek Riedewald
Aamod Khatiwada, Grace Fan, Roee Shraga, Zixuan Chen, Wolfgang Gatterbauer, Renée J. Miller, and Mirek Riedewald. 2023. https://doi.org/10.1145/3588689 SANTOS : Relationship -based Semantic Table Union Search . Proceedings of the ACM on Management of Data, 1(1):1--25
2023 doi
-
[23]
Aamod Khatiwada, Harsha Kokel, Ibrahim Abdelaziz, Subhajit Chaudhury, Julian Dolby, Oktie Hassanzadeh, Zhenhan Huang, Tejaswini Pedapati, Horst Samulowitz, and Kavitha Srinivas. 2025. Tabsketchfm: Sketch-based tabular representation learning for data discovery over data lakes....
2025
- [24]
-
[25]
Leland McInnes, John Healy, Steve Astels, and 1 others. 2017. hdbscan: Hierarchical density based clustering. J. Open Source Softw., 2(11):205
2017
-
[26]
Leland McInnes, John Healy, Nathaniel Saul, and Lukas Gro berger. 2018. https://doi.org/10.21105/JOSS.00861 UMAP: uniform manifold approximation and projection . J. Open Source Softw., 3(29):861
2018 doi
-
[27]
Hamed Mirzaei and Davood Rafiei. 2023. https://ceur-ws.org/Vol-3462/TADA2.pdf Table union search with preferences . In Joint Proceedings of Workshops at the 49th International Conference on Very Large Data Bases (VLDB 2023), Vancouver, Canada, August 28 - September 1, 2023 , v...
2023
-
[28]
Pu, and Renée J
Fatemeh Nargesian, Erkang Zhu, Ken Q. Pu, and Renée J. Miller. 2018. https://doi.org/10.14778/3192965.3192973 TUS : Table union search on open data . Proceedings of the VLDB Endowment, 11(7):813--825
2018
-
[29]
Koyena Pal, Aamod Khatiwada, Roee Shraga, and Ren \'e e J Miller. 2024. Alt-gen: Benchmarking table union search using large language models. Proceedings of the VLDB Endowment. ISSN, 2150:8097
2024
-
[30]
Thomas Pellissier Tanon, Gerhard Weikum, and Fabian Suchanek. 2020. Yago 4: A reason-able knowledge base. In The Semantic Web, pages 583--596, Cham. Springer International Publishing
2020
-
[31]
Nils Reimers and Iryna Gurevych. 2019. https://doi.org/10.18653/v1/D19-1410 Sentence- BERT : Sentence embeddings using S iamese BERT -networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference...
2019 doi
-
[32]
Anish Das Sarma, Lujun Fang, Nitin Gupta, Alon Y Halevy, Hongrae Lee, Fei Wu, Reynold Xin, and Cong Yu. 2012. Finding related tables. In SIGMOD Conference, volume 10, pages 2213836--2213962
2012
-
[33]
Nandan Thakur, Nils Reimers, Andreas R \"u ckl \'e , Abhishek Srivastava, and Iryna Gurevych. 2021. https://openreview.net/forum?id=wCu6T5xFjeJ BEIR : A heterogeneous benchmark for zero-shot evaluation of information retrieval models . In Thirty-fifth Conference on Neural Info...
2021
-
[34]
Cornelius Wolff and Madelon Hulsebos. 2025. How well do llms reason over tabular data, really? arXiv preprint arXiv:2505.07453
2025
-
[35]
Pu, and Ren \' e e J
Erkang Zhu, Fatemeh Nargesian, Ken Q. Pu, and Ren \' e e J. Miller. 2016. https://doi.org/10.14778/2994509.2994534 LSH ensemble: Internet-scale domain search . Proc. VLDB Endow. , 9(12):1185--1196
2016
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.