Pith. sign in

REVIEW 2 major objections 5 minor 3 cited by

Tackling prediction tasks in relational databases with LLMs

T0 review · 2 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A frozen pretrained LLM that reads relational databases as nested JSON documents matches or slightly exceeds the relational deep learning baseline on RelBench classification, and comes within about 0.5 MAE on regression.

desk verdict Useful first LLM baseline for RelBench, but the headline competitiveness claim hinges on rel-f1 tasks where the model uses memorized knowledge; still deserves peer review with revisions. read the letter →

arxiv 2411.11829 v1 pith:M3ZIGKLC submitted 2024-11-18 cs.LG cs.CLcs.DB

Marek Wydmuch , {L}ukasz Borchmann , Filip Graliński This is my paper · ORCID
classification cs.LGcs.CLcs.DB
keywords largelanguagemodelsrelationaldatabasesRelBenchdocumentconstructionin-contextlearningmetric-awareinferencedeeptabulardata
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that predictive tasks in relational databases can be solved by turning each prediction into a text document and letting a pretrained large language model read it. The authors construct documents by following foreign-key links between tables, nesting related rows as JSON, and adding in-context examples that respect prediction time. On the RelBench benchmark, frozen Llama 3.2 models with a small trained MLP head reach an average classification AUROC of 76.8 and 77.1, above the 75.8 of the relational deep learning baseline, and come within roughly half a point of MAE on regression. If right, this gives the field a much simpler baseline for machine learning on relational data, one that avoids end-to-end training of graph neural networks.

What carries the argument

The load-bearing object is the document construction procedure (Algorithm 1): for each entity, it recursively joins related tables via foreign keys, selects up to $n_{\mathrm{nest}}$ most recent rows from linked tables before the prediction timestamp, and nests them in JSON so the LLM does not need multi-hop reasoning to connect an entity to its history. Around this sits a metric-aware readout: for AUROC the positive-token probability is used directly, and for MAE a one-hidden-layer MLP head is trained on the frozen LLM's last-token embedding using at most $10^5$ documents. The procedure's role is to convert a heterogeneous relational database into a uniform text input that a pretrained model can score without any parameter updates to the LLM itself.

What would settle it

Recompute RelBench averages after removing the three rel-f1 tasks or after replacing driver, team, and race names with anonymous IDs before document construction; if the anonymized rel-f1 scores fall to near chance while the original documents score around 78–90 AUROC, or if the remaining average drops below RDL, the leakage channel is confirmed and the headline comparison must be revised.

Watch

Extended reading notes

Core claim

The central claim is that denormalizing a relational database into nested text documents is enough to make LLMs competitive with dedicated relational deep learning. The paper's procedure starts from a task-table row, follows foreign keys to primary keys, recursively pulls in up to $n_{\mathrm{nest}}$ related rows from linked tables up to depth $d$, prepends task and database descriptions plus in-context examples, and serializes everything as JSON with the target last. For binary classification the authors use the probability of the token "1" as the score (justified by AUROC's invariance to monotone transforms of $P(y=1\mid x)$), and for regression they train a small MLP on the LLM's token embeddings rather than using the median of sampled token probabilities, which performs poorly. The resulting frozen-model system matches or slightly exceeds the relational deep learning baseline on RelBench classification while remaining within about 0.5 MAE on regression, establishing LLM-based prediction as a new simple baseline.

Load-bearing premise

The comparison assumes the RelBench temporal split is the only gate on information: that test labels are not already inside the LLM's pretraining knowledge, which the paper itself undercuts by noting that on rel-f1 the models appear to lean on memorized Formula-1 facts.

Editorial extensions

If this is right

  • A frozen pretrained LLM plus a small readout becomes a credible baseline for relational database prediction, competing with end-to-end relational deep learning.
  • Because the MLP head needs only a subset of training documents (up to $10^5$) while several RelBench tasks have over a million training rows, the approach may pay off in low-data regimes.
  • Which document parameters matter varies by task: some need many related examples, some need deeper nesting into linked tables, and zero-shot documents almost always fail, so context selection is the main lever.
  • The modest gap between 1B and 3B models suggests document content, not model capacity, drives most of the gain.
  • The same document construction extends in principle to non-text columns (images, audio) as multimodal foundation models mature.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If test labels from rel-f1 are already memorized by the pretrained model, the headline average is inflated; a fair comparison would re-run the benchmark with entity names anonymized or rel-f1 excluded, and the gap over RDL may shrink or reverse.
  • Because the MLP head reads a single hidden layer of embeddings, a cheaper linear-probe version of the method is a natural testable variant that would isolate how much relational signal survives in the frozen representations.
  • Future benchmarks of this kind should include contamination checks, for example probing the base model on task rows stripped of all context, since a temporal split alone does not bound what a pretrained LLM already knows.
  • A testable extension: use the LLM's own allocation of probability mass to select which in-context examples and nested rows to keep, reducing document length and cost instead of the current grid search.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. This paper proposes a simple LLM-based approach to RelBench relational prediction tasks. For each test entity, it constructs a JSON document containing a task description, up to ninc in-context examples, up to nrel related examples, and nested rows from linked tables up to depth d; a frozen Llama 3.2 base model then either scores the probability of the positive token (classification) or feeds its token embeddings to a small MLP head (regression and classification). The authors compare with LightGBM and Relational Deep Learning (RDL) on 21 RelBench tasks and report average AUROC 76.83/77.09 for the two LLM sizes with MLP heads versus 75.83 for RDL, and MAE 13.76/13.16 versus 12.63 for RDL.

Significance. If the competitiveness claim held, this would be a valuable new baseline: it avoids end-to-end graph training, can exploit a small training subset, and provides a clear algorithmic recipe. The paper is also unusually transparent in reporting the full per-configuration tables (Appendix D) and the algorithmic pseudocode. However, the headline result is contingent on the rel-f1 tasks, where the authors themselves suspect reliance on memorized factual knowledge; the contribution is therefore not yet established. The method is not presented with machine-checked proofs or code, but the algorithmic description is complete enough to replicate with reasonable effort.

major comments (2)
  1. [Section 4, Table 1] The rel-f1 results are not a fair test of the document-construction method because the models appear to rely on memorized Formula-1 knowledge. The authors write in Section 4 that the models 'seem to be mostly relying on their pre-existing factual knowledge of Formula-1'; the RelBench temporal split filters database rows (tv < tp) but cannot remove information from the LLM pretraining corpus. Recomputing Table 1 without driver-dnf and driver-top3 gives average AUROC 75.6 (1B+MLP) and 75.3 (3B+MLP) versus 76.2 for RDL, reversing the reported 76.8/77.1 versus 75.8 advantage. For regression, excluding driver-position changes the MAE gap from 1.13 to 1.33 (1B) and from 0.52 to 0.71 (3B) relative to RDL. On the remaining tasks the LLM+MLP variant is below RDL on 7 of 10 classification tasks and on most regression tasks, so the headline 'competitive to RDL' claim is not supported once the contaminated tasks are removed. The paper should either exclude rel-f1 from the headline averages or provide a leakage-controlled analysis (e.g., results on seasons or entities unlikely to be in pretraining, or a discussion of how many test labels are plausibly memorized).
  2. [Section 4 and Appendix D] The evaluation protocol selects, per task, the best of the document-parameter combinations on the validation set and then reports the test score of that single selection; Appendix D states 'For each set of document generation parameters, we conducted a single run.' Because there are no repeated seeds, bootstrap intervals, or other variance estimates, and because the test metric is computed on a random 10,000-example sample (Section 4) without a stated seed, the 0.5-1.3 point average differences in Table 1 cannot be distinguished from selection noise. The authors should report confidence intervals (e.g., repeated runs or bootstrap over test samples) and either fix the document parameters a priori or correct for selection over the grid.
minor comments (5)
  1. [Table 1] The header is ambiguous: each LLM column contains two numbers (metric-aware inference and +MLP) but the header only says 'Llama 3.2 1B + MLP' and 'Llama 3.2 3B + MLP'; please label the two subcolumns explicitly.
  2. [Algorithm 1] Line 13 of ADD_RELATED_ENTITIES returns X' rather than the updated X; this appears to be a typo, and the return value is never used by the caller.
  3. [Section 4] The grid over ninc, nrel, nnest, and d is described only in Appendix D; a compact statement of the grid in the main text would make the selection process easier to follow.
  4. [Limitations] The Limitations section should explicitly acknowledge the pretraining-leakage risk for rel-f1; currently it only mentions the single benchmark and context-length constraints.
  5. [Appendix C] The statement 'We will publish the code for replicating all the results' is not fulfilled in the manuscript; please provide a code link or state availability explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the LLM document-construction pipeline is benchmarked against external RelBench baselines, and no prediction is defined in terms of its fitted inputs.

full rationale

The paper's central claim—that serialized relational documents let LLMs perform competitively with Relational Deep Learning—is evaluated against external RelBench baselines from Robinson et al. (2024), not against a quantity derived from its own fitted parameters. The document-generation procedure constructs test inputs from database rows with tv < tp and appends the target only to training in-context examples, so the test prediction is not built from the target by construction. The MLP head is trained on a subset of generated training documents and selected via a validation set, which is standard supervised evaluation rather than a fitted parameter renamed as a prediction. The metric-aware inference argument relies on external statistical results (cross-entropy minimization, AUROC monotone transforms, MAE median), not on a self-citation. The paper's own observation that on rel-f1 the models 'seem to be mostly relying on their pre-existing factual knowledge of Formula-1' (Section 4) identifies a pretraining-data contamination risk for benchmark validity, but that is an evaluation confound, not a circular derivation: the test labels are not fed into the document construction or the model's inference procedure. No load-bearing step reduces to its own input or to a self-citation chain, so the circularity score is 0.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim rests mainly on empirical evaluation rather than derivation, so the ledger is dominated by document-generation hyperparameters and domain assumptions about serialization and leakage. The free parameters are the four document-construction knobs plus MLP training choices; each is selected per task on validation, which means the reported numbers are best-case over a small grid. The most fragile assumption is temporal isolation of test labels, which the rel-f1 results show is violated for LLM pretraining. No new physical or learned entities are introduced.

free parameters (6)
  • ninc (number of in-context examples) = 0, 8, or 16, selected per task on validation
    Document generation parameter; best value varies by task (Appendix D), central to method's performance.
  • nrel (number of related examples) = 0, 8, or 16, selected per task on validation
    Related examples from the train table linked to the same primary keys; selected via validation and has a large effect on results.
  • nnest (number of nested rows) = 0, 4, or 8, selected per task on validation
    Rows pulled from tables linked by primary-to-foreign keys; per-task validation choice.
  • d (relation graph traversal depth) = 0 or 1, selected per task on validation
    Depth of recursive denormalization; best value found by search.
  • ntrain (MLP training subset size) = 1e4 or 1e5 examples, selected per task
    MLP head trained on a subset; the choice affects the low-data claim.
  • MLP hyperparameters = hidden size 10, lr 1e-4, weight decay 1e-3, 100 epochs
    Fixed architecture and optimizer; not tuned per task but still free choices that affect the results.
assumptions (5)
  • domain assumption A base LLM trained only with next-token prediction, under an unrestricted hypothesis class, models the true conditional distribution P(y|x).
    Invoked in Section 3.2 to justify reading token probabilities as scores; real LLMs are not unconstrained minimizers. The authors cite Gneiting and Raftery (2007).
  • domain assumption JSON serialization of denormalized related rows preserves enough information for prediction and reduces multi-hop reasoning failures.
    Core representational choice in Section 3.1; the paper does not compare JSON against other serializations, and later notes some tasks are hurt by extra context.
  • domain assumption The RelBench temporal split (only rows with tv < tp may be used) prevents information leakage during LLM evaluation.
    Underlies all comparisons; the F1 paragraph in Section 4 shows the assumption is violated because pretraining already contains future race outcomes.
  • standard math Optimal decision rules for AUROC and MAE justify metric-aware inference: any monotone transform of P(y=1|x) for AUROC, and the median for MAE.
    Standard statistical decision theory cited in Section 3.2 (Clémençon et al., 2008; Uematsu and Lee, 2014; Bishop and Nasrabadi, 2006).
  • domain assumption The random 10,000-example test sample is representative, and single runs per parameter configuration are sufficient for the reported comparisons.
    Stated in Section 4; no confidence intervals or repeated-seed results are given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Tackling prediction tasks in relational databases with LLMs." pith.science (2026). https://pith.science/paper/M3ZIGKLC

@misc{pith2026241111829,
  author       = {Pith},
  title        = {Pith review of: Tackling prediction tasks in relational databases with LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M3ZIGKLC}},
  note         = {Machine review of arXiv:2411.11829}
}
read the original abstract

Though large language models (LLMs) have demonstrated exceptional performance across numerous problems, their application to predictive tasks in relational databases remains largely unexplored. In this work, we address the notion that LLMs cannot yield satisfactory results on relational databases due to their interconnected tables, complex relationships, and heterogeneous data types. Using the recently introduced RelBench benchmark, we demonstrate that even a straightforward application of LLMs achieves competitive performance on these tasks. These findings establish LLMs as a promising new baseline for ML on relational databases and encourage further research in this direction.

Figures

Figures reproduced from arXiv: 2411.11829 by the authors.

Figure 1
Figure 1. Process of constructing a single example for LLM-based inference. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Parameter-Free Encoders Remain Viable for RDB Foundation Models

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Trainable RDB encoders cannot robustly exploit neighborhood labels as fixed foundation-model features or feature-importance signals, so simple parameter-free encoders stay near-SOTA.

  2. No Need to Train Your RDB Foundation Model

    cs.AI 2026-02 conditional novelty 6.0 of 10

    Column-wise, parameter-free JUICE encodings let single-table ICL models solve multi-table RDB prediction tasks with no training or fine-tuning.

  3. RelGNN: Composite Message Passing for Relational Deep Learning

    cs.LG 2025-02 conditional novelty 6.0 of 10

    RelGNN passes messages along schema-derived atomic routes through junction tables and improves over heterogeneous GNN baselines on most RelBench tasks, with the largest gain on one trial-database regression task.

Reference graph

Works this paper leans on

31 extracted references · 12 canonical work pages · cited by 3 Pith papers

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Christopher M Bishop and Nasser M Nasrabadi. 2006. Pattern recognition and machine learning, volume 4. Springer

    2006

  4. [4]

    St \'e phan Cl \'e men c on, G \'a bor Lugosi, and Nicolas Vayatis. 2008. Ranking and empirical minimization of u-statistics

    work page 2008

  5. [5]

    Jacob Devlin, Ming - Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/V1/N19-1423 BERT: pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, ...

    doi:10.18653/v1/n19-1423 2019

  6. [6]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

  7. [7]

    Gus Eggert, Kevin Huo, Mike Biven, and Justin Waugh. 2023. https://arxiv.org/abs/2310.07875 TabLib : A dataset of 627m tables with context . Preprint, arXiv:2310.07875

    arXiv 2023

  8. [8]

    Matthias Fey, Weihua Hu, Kexin Huang, Jan Eric Lenssen, Rishabh Ranjan, Joshua Robinson, Rex Ying, Jiaxuan You, and Jure Leskovec. 2023. Relational deep learning: Graph representation learning on relational databases. arXiv preprint arXiv:2312.04615

    arXiv 2023

Show all 31 references
  1. [9]

    Jerome H Friedman. 2001. Greedy function approximation: a gradient boosting machine. Annals of statistics, pages 1189--1232

    2001

  2. [10]

    Ullman, and Jennifer Widom

    Hector Garcia-Molina, Jeffrey D. Ullman, and Jennifer Widom. 2008. Database Systems: The Complete Book, 2 edition. Prentice Hall Press, USA

    2008

  3. [11]

    Josh Gardner, Juan C Perdomo, and Ludwig Schmidt. 2024. Large scale transfer learning for tabular data via language modeling. arXiv preprint arXiv:2406.12031

    2024 arXiv

  4. [12]

    Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. 2017. Neural message passing for quantum chemistry. In International conference on machine learning, pages 1263--1272. PMLR

    2017

  5. [13]

    Tilmann Gneiting and Adrian E Raftery. 2007. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477):359--378

    2007

  6. [14]

    Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. 2021. Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems, 34:18932--18943

    2021

  7. [15]

    Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs. Advances in neural information processing systems, 30

    2017

  8. [16]

    Madelon Hulsebos, Çagatay Demiralp, and Paul Groth. 2023. https://doi.org/10.1145/3588710 GitTables : A large-scale corpus of relational tables . Proceedings of the ACM on Management of Data, 1(1):1–17

    2023 doi

  9. [17]

    Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017. LightGBM : A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30

    2017

  10. [18]

    Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et al. 2024 a . Can LLM already serve as a database interface? a Big Bench for large-scale database grounded text-to-sqls. Advances in Neural Information Processing ...

    2024

  11. [19]

    Weichen Li, Xiaotong Huang, Jianwu Zheng, Zheng Wang, Chaokun Wang, Li Pan, and Jianhua Li. 2024 b . rLLM : Relational table learning with LLMs . arXiv preprint arXiv:2407.20157

    2024

  12. [20]

    Michal Lukasik, Harikrishna Narasimhan, Aditya Krishna Menon, Felix Yu, and Sanjiv Kumar. 2024. Metric-aware LLM inference. arXiv preprint arXiv:2403.04182

    2024 arXiv

  13. [21]

    Barbara J McNeil and James A Hanley. 1984. Statistical approaches to the analysis of receiver operating characteristic ( ROC ) curves. Medical decision making, 4(2):137--150

    1984

  14. [22]

    Jan Motl and Oliver Schulte. 2024. https://arxiv.org/abs/1511.03086 The CTU Prague Relational Learning Repository . Preprint, arXiv:1511.03086

    2024 arXiv

  15. [23]

    Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. http://www.aclweb.org/anthology/D14-1162 GloVe : Global vectors for word representation . In Empirical Methods in Natural Language Processing (EMNLP), pages 1532--1543

    2014

  16. [24]

    Joshua Robinson, Rishabh Ranjan, Weihua Hu, Kexin Huang, Jiaqi Han, Alejandro Dobles, Matthias Fey, Jan E Lenssen, Yiwen Yuan, Zecheng Zhang, et al. 2024. RelBench : A benchmark for deep learning on relational databases. arXiv preprint arXiv:2407.20060

    2024 arXiv

  17. [25]

    Ananya Singha, Jos \'e Cambronero, Sumit Gulwani, Vu Le, and Chris Parnin. 2023. Tabular representation, noisy operators, and impacts on table structure understanding tasks in LLMs . arXiv preprint arXiv:2310.10358

    2023 arXiv

  18. [26]

    Quan M Tran, Suong N Hoang, Lam M Nguyen, Dzung Phan, and Hoang Thanh Lam. 2024. TabularFM : An open framework for tabular foundational models. arXiv preprint arXiv:2406.09837

    2024 arXiv

  19. [27]

    Kazuki Uematsu and Yoonkyung Lee. 2014. Statistical optimality in multipartite ranking and ordinal regression. IEEE transactions on pattern analysis and machine intelligence, 37(5):1080--1094

    2014

  20. [28]

    Jian Wu, Linyi Yang, Manabu Okumura, and Yue Zhang. 2024 a . MRKE : The multi-hop reasoning evaluation of LLMs by knowledge edition. arXiv preprint arXiv:2402.11924

    2024 arXiv

  21. [29]

    Jian Wu, Linyi Yang, Zhen Wang, Manabu Okumura, and Yue Zhang. 2024 b . https://api.semanticscholar.org/CorpusID:267751383 Evaluating LLMs ' inherent multi-hop reasoning ability

    2024

  22. [30]

    Jiaxuan You, Jonathan M Gomes-Selman, Rex Ying, and Jure Leskovec. 2021. Identity-aware graph neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 10737--10745

    2021

  23. [31]

    Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. https://doi.org/10.18653/v1/D18-1425 S pider: A large-scale human-labeled dataset for complex and cross-domain seman...

    2018 doi

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.