{"id":"9b3c989a-ab92-4ddd-a713-9ce51cec989f","arxiv_id":"2602.10441","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"LakeMLB is a new six-dataset benchmark for multi-table machine learning in data lakes; experiments find pretraining helps in Union scenarios and feature augmentation helps in Join scenarios.","lead":"A new benchmark, LakeMLB, gives machine-learning researchers six real-world table pairs (union and join scenarios) plus code to test models that learn from multiple tables in data lakes. It runs standard tabular models under pre-training, data-augmentation, and feature-augmentation strategies to find which strategy helps most.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Potential label leakage in Join auxiliary tables: unremoved Spotify genre / Wikipedia industry fields may fully explain the reported +2.97pp FA gains.","rationale":"The reader's weakest assumption focused on the reliability of BERT+cosine 1-NN entity matching. That is a valid concern, but the paper supplies a negative control—Figure 4 shows degrading matching quality lowers accuracy—so alignment noise alone is unlikely to fully explain the positive FA gains. A more direct threat is label leakage: the auxiliary tables are drawn from sources that naturally contain a field semantically identical to the target label, and the paper nowhere states that this field was excluded. The omission is conspicuous because the authors explicitly removed the Discogs 'styles' column to prevent leakage. If the auxiliary genre/industry column is retained, the FA strategy concatenates the label as a feature, making the benchmark's headline Join result (FA best) an artifact. This would also contaminate PT/DA comparisons for those datasets if leakage propagates through the auxiliary table. The concern is concrete and falsifiable by inspecting the released artifact. Since the paper's contribution rests on the validity of the datasets and strategy rankings, the current version should not be accepted as a reliable benchmark; hence the verdict moves from CONDITIONAL to REJECT (or at minimum CONDITIONAL with a mandatory artifact audit before acceptance).","tokens_in":18501,"tokens_out":6018,"duration_ms":64920,"concrete_test":"Inspect the released artifact (https://github.com/zhengwang100/LakeMLB): (1) list the columns of NNWiki, LHWiki, and Spotify auxiliary tables and check for columns whose names or semantics match the target label (e.g., 'Industry', 'Sector', 'genre', 'track_genre'); (2) if such columns exist, rerun the FA strategy on NNStocks, LHStocks, and DSMusic with those columns removed from the auxiliary table before concatenation; (3) compare FA accuracy to the T_tgt baseline in Table 3. If removing these columns reduces FA gains to near zero or negative, the +2.97pp/91.7% finding is a leakage artifact. Even if the current release has already removed them, the paper should explicitly document this preprocessing step to permit verification.","verdict_should_be":"REJECT","load_bearing_attack":"The Join-scenario auxiliary tables are sourced from Wikipedia infoboxes (NNStocks, LHStocks) and the public Spotify tracks dataset (DSMusic). These sources natively contain a field semantically equivalent to the target label: Wikipedia infoboxes often include 'Industry' (close to sector labels), and the standard Spotify tracks dataset includes a 'genre' column (close to Discogs genre labels). The paper explicitly mentions removing the Discogs 'styles' column to avoid label leakage (Appendix A.2), but it never states that the analogous auxiliary label field was dropped before constructing FA feature vectors. Under FA, 1-NN matched auxiliary rows are concatenated as features (Appendix B.3). If the auxiliary label column remains in those features, the classifier can directly observe the target label at training and test time, trivially producing the observed average FA gain of +2.97 pp and 91.7% win rate (Section 5.4.1, Table 4). This concern is distinct from row-alignment quality: even perfect matching would leak; imperfect matching would leak partially. The central conclusion that FA is best for Join would be an artifact, and the benchmark datasets as released would require correction before any downstream use.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LakeMLB, a benchmark for multi-table machine learning in data lakes, with two scenarios (Union and Join), six real-world datasets, fixed 70/10/20 splits, and three integration strategies: pre-training (PT), data augmentation (DA), and feature augmentation (FA). The authors evaluate tree-based, deep tabular, transfer-learning, and tabular foundation models, reporting that PT is best in Union scenarios, FA is best in Join scenarios, and transfer-learning models are generally robust. Code and datasets are released. If the construction is sound, LakeMLB fills a real gap between data-discovery benchmarks and single-table tabular benchmarks.","tokens_in":18833,"tokens_out":5488,"duration_ms":63393,"significance":"The benchmark addresses an important gap: most tabular benchmarks are single-table, while data-discovery benchmarks stop at table retrieval. LakeMLB provides a concrete, reproducible evaluation platform for the model-learning stage over unionable and joinable tables, using real-world public data with fixed splits and a unified protocol. The breadth of baselines (tree ensembles, single-table transformers, transfer learners, and foundation models) is a strength, and the release of code and datasets is valuable to the community. However, the empirical conclusions currently rest on two fragile pillars: possible label leakage via auxiliary-table columns in the Join scenarios and the absence of any variance/error-bar reporting for comparisons whose differences are sometimes only one or two percentage points on test sets of a few hundred rows. These issues need to be resolved before the benchmark and its findings can be relied upon.","major_comments":[{"comment":"The Join auxiliary tables are described as containing raw Wikipedia infobox attributes (NNWiki, LHWiki) and Spotify track metadata (DSMusic), while the target labels are company sector and music genre. For DSMusic, the paper explicitly removes the Discogs 'styles' column to avoid label leakage, but it is silent on whether the Spotify auxiliary table's 'genre' column was removed; the named public Spotify dataset commonly contains such a column. For NNWiki/LHWiki, no analogous removal of infobox fields such as 'Industry' or 'Type' is documented. Under FA, B.3 concatenates the auxiliary attributes into the target rows for both training and test, so if any label-equivalent column remains, the model can observe the target label directly. This would trivially explain the +2.97 pp average FA gain and 91.7% win rate in Table 4. The authors must audit the released auxiliary tables, remove or othe","section":"Appendix A.2 and B.3"},{"comment":"Appendix C.2 states that deep baselines are re-trained for 10 runs and that mean and standard deviation are reported. Tables 2 and 3, however, contain bare accuracy numbers with no variance or confidence intervals. This matters for the paper's central ranking: NNStocks and LHStocks have about 216–224 test rows, so a 1–3 pp accuracy difference corresponds to only a handful of instances. The observed FA advantage in Join scenarios is of exactly this size. The authors should report means ± std or confidence intervals across seeds (and, ideally, across split variability), or explicitly state that the numbers are single-run results and temper the conclusions accordingly.","section":"Tables 2–3 and Appendix C.2"},{"comment":"The experiment varies the k-th nearest neighbor (20, 21, 22, 24, and random) and interprets the associated accuracy decline as 'matching quality deteriorates.' But changing k changes the retrieved row, not a measured notion of match quality; the paper never reports whether the 1-NN match is actually correct or how match accuracy degrades with k. The conclusion that FA 'critically relies on high-quality one-to-one row matching' is therefore only weakly supported. Reporting entity-match accuracy on a sample of hand-labeled or exact-match pairs, or comparing against an exact-key join, would make the claim concrete.","section":"Section 5.4.2 / Figure 4"}],"minor_comments":[{"comment":"The GACars auxiliary table is listed as 3,600 rows in Table 1 but as 36,000 samples in Appendix A.1. Please correct this inconsistency; it affects reproducibility and the DA cap of 30%.","section":"Table 1 vs. Appendix A.1"},{"comment":"The x-axis labels '20 21 22 24' in Figure 4 are ambiguous; they should be typeset as 2^0, 2^1, 2^2, 2^4. Also, the caption of Table 3 should indicate whether the reported numbers are means over runs or a single representative run.","section":"Figure 4 and Table 3"},{"comment":"The paper would benefit from a data-card-style appendix listing, for each auxiliary table, the full set of columns and which columns were dropped and why. This is standard practice for benchmarks and would also have made the label-leakage situation self-evident.","section":"General documentation"}],"recommendation":"major_revision","confidential_remarks":"The leakage concern is the most serious issue. I did not inspect the GitHub repository, but the manuscript's own silence about auxiliary-table label columns, combined with the known contents of the named public Spotify and Wikipedia sources, makes the risk material. If the leakage is confirmed, the benchmark datasets and all Join experiments must be corrected; if it is not, an explicit column-level audit in the paper should settle the matter. The missing variance reporting is also a substantive issue for a benchmark paper whose headline ranking rests on very small margins."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—\n\nThe headline is that this is a genuinely useful benchmark resource with a potentially load-bearing flaw in its headline result. The six dataset pairs with fixed splits and the PT/DA/FA protocol are a real contribution; nothing else covers ML over unionable/joinable lake tables. But the paper's central claim—feature augmentation wins in Join—looks like it could be an artifact of label leakage in the auxiliary tables.\n\nWhat's good: dataset construction is documented in unusual detail (sources, filtering, splits, chronological splits), the benchmark code is promised, and the paper is honest about limitations (Appendix 3.3.2). The fixed splits and balanced classes are right for a benchmark. The fuzzy 1NN matching is described clearly enough to reproduce.\n\nThe soft spot: the stress-test note about the Spotify 'genre' column and Wikipedia 'Industry' fields is not addressed. The paper explicitly removes the Discogs 'styles' column to avoid leakage, but never mentions dropping the analogous label columns from the auxiliary tables. If those remain in the concatenated FA features, the classifier sees the label at train and test time. That would explain the suspicious 91.7% win rate and +2.97 pp average gain for FA. Without the released code/data, we can't confirm, but the burden shifts to the authors. This is not a minor quibble—it's the paper's main empirical conclusion.\n\nAlso, Tables 2–3 have no error bars, even though Appendix C.2 says deep baselines should report mean/std. Several test sets are only ~200 rows. The 'first benchmark' claim rests on absence but is plausible. These are secondary.\n\nIf the leakage is real, the benchmark assets may still be salvageable: the Union part and the fixed splits survive; the Join part needs corrected auxiliary tables and re-run experiments. The paper deserves peer review because a referee would catch this, and the benchmark fills a real gap. I'd tell the editor to send it out, with the leakage issue as the first question.","headline":"Useful benchmark resource whose headline FA-in-Join result may be a label-leakage artifact.","tokens_in":19296,"tokens_out":1849,"would_cite":false,"duration_ms":20731,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LakeMLB claims to be the first benchmark for multi-table machine learning in data lakes, reporting that pre-training excels on union tables and feature augmentation on join tables.","keywords":["Data Lake","Tabular Learning","Multi-table ML Benchmark","Union","Join","Feature Augmentation","Pre-training","Transfer Learning"],"falsifier":"Manually label a random sample of the 1-NN entity matches in NNStocks or DSMusic and measure matching precision; or rerun the feature augmentation experiments using ground-truth join keys instead of the fuzzy 1-NN alignments. If matching precision is low, or if FA with true keys does not outperform FA with the benchmark's alignments, the paper's central claim about feature augmentation in joins is not supported.","tokens_in":18428,"feed_emoji":"🧪","tokens_out":4310,"duration_ms":45172,"temperature":0.7,"pith_summary":"This paper introduces LakeMLB, a benchmark for evaluating machine learning models in data lake environments, which it claims is the first of its kind. It covers two fundamental multi-table scenarios, Union and Join, with six real-world datasets, fixed splits, and balanced classification tasks. The paper compares pre-training, data augmentation, and feature augmentation strategies across classical and deep tabular models. Its central finding is that the best way to use an auxiliary table depends on the relationship: pre-training works best for unionable tables, feature augmentation works best for joinable tables, and transfer-learning models are the most robust overall. If these results hold, LakeMLB gives the community a standard testbed and practical guidance for data-lake machine learning.","feed_headline":"Pre-training wins for union tables; feature augmentation for joins","feed_subtitle":"Six real datasets and three integration strategies show how to use auxiliary tables in data-lake ML.","key_machinery":"The load-bearing component is the benchmark design itself: each instance is a target table paired with one auxiliary table, with join alignments constructed by embedding entity-name columns with BERT-base-uncased and retrieving the cosine-similarity 1-nearest neighbor. That weak entity matching converts joinable tables into horizontally enriched feature tables, and the same 1-NN machinery (over row-text embeddings) is used to attach auxiliary features in union settings. The three integration strategies—pre-training (PT), data augmentation (DA), and feature augmentation (FA)—are defined on top of this pairing, and fixed train/validation/test splits plus balanced labels make results comparable","core_discovery":"LakeMLB is a benchmark suite built around pairs of target and auxiliary tables, one pair per scenario, with three datasets for Union (traffic collisions, building violations, used cars) and three for Join (stock sectors twice, music genres). The paper's claim is that this is the first standardized benchmark for the model-learning stage in data lakes. Across experiments with tree ensembles, tabular neural networks, transfer-learning models, and tabular foundation models, the paper finds that pre-training on the auxiliary table yields the most consistent gains in Union scenarios (83.3% win rate), feature augmentation via 1-NN concatenation yields the largest gains in Join scenarios (91.7% win","pith_inferences":["The join-scenario results likely overstate feature augmentation's real-world value, because the fuzzy BERT-based alignments may be tuned to entity-name matching; testing FA with true foreign keys or a manually validated join would isolate how much of the +2.97 pp gain is due to alignment quality rather than the strategy itself.","A natural extension is to measure the precision of the 1-NN entity matching on a labeled sample; if precision is low, the benchmark's join tasks become matching-noise benchmarks, and FA gains should be re-examined.","The paper's strategy rankings suggest a practical rule of thumb for data-lake pipelines: when auxiliary tables share a label space, pre-train; when they share entities, augment features; avoid blind row-level data augmentation across sources.","The benchmark's two-table simplification likely underestimates the value of data augmentation in lakes with many overlapping sources; a version with multiple auxiliary tables could change the DA conclusions."],"forward_implications":["Pre-training on a unionable auxiliary table is a dependable way to boost target accuracy, while data augmentation in union settings often hurts because label taxonomies differ across sources.","Feature augmentation in join settings pays off only when row matching is accurate; deliberately deteriorating the match (farther k-NN or random) degrades accuracy, so entity-resolution quality gates the benefit.","Transfer-learning tabular models, especially those that exploit text in schemas and cell values, are the strongest overall family in both scenarios, suggesting they are promising for real lake workloads.","LakeMLB's fixed splits and public code provide a reproducible reference point for comparing future multi-table tabular methods.","The benchmark's scope is limited to two-table, balanced classification tasks; extending to more tables, regression, and imbalanced labels is the stated next step."],"fun_headline_variants":["Pre-training beats augmentation for union tables in new data-lake benchmark","For joins, feature augmentation wins in LakeMLB benchmark","First data-lake ML benchmark reveals best table integration strategies","LakeMLB: pre-training for unions, feature augmentation for joins"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The join scenarios (and the FA/DA strategies that exploit them) depend on the assumption that BERT-base-uncased embeddings plus cosine 1-NN retrieval create row/entity alignments that are good enough to be useful; if the alignments are wrong, the join tasks are dominated by matching noise and the reported gains would be artifacts of benchmark construction rather than properties of multi-table learning.","fun_headline_variants_meta":{"raw":{"variants":["Pre-training beats augmentation for union tables in new data-lake benchmark","For joins, feature augmentation wins in LakeMLB benchmark","First data-lake ML benchmark reveals best table integration strategies","LakeMLB: pre-training for unions, feature augmentation for joins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000251,"raw_usage":{"total_tokens":1356,"prompt_tokens":670,"completion_tokens":686,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":414,"completion_tokens_details":{"reasoning_tokens":615}},"tokens_in":414,"tokens_out":686,"duration_ms":7389,"temperature":1.0,"reasoning_tokens":615,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T06:02:29.658570+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Manually label a random sample of the 1-NN entity matches in NNStocks or DSMusic and measure matching precision; or rerun the feature augmentation experiments using ground-truth join keys instead of the fuzzy 1-NN alignments. If matching precision is low, or if FA with true keys does not outperform FA with the benchmark's alignments, the paper's central claim about feature augmentation in joins is not supported.","supporting_citations":[],"review_version":1}