{"id":"37f62bf2-c547-4cb0-b792-9cc933fbfa67","arxiv_id":"2411.11829","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Pre-trained LLMs, fed serialized relational rows with related examples, achieve competitive AUROC/MAE on RelBench without fine-tuning, but the headline comparison is weakened by pretraining contamination on Formula 1 tasks.","lead":"This paper applies off-the-shelf LLMs to prediction tasks in relational databases by turning database rows into JSON documents and reading out probabilities or a small trained head. On the RelBench benchmark the method lands close to, and sometimes above, a graph neural network baseline, though one dataset's results appear to be inflated by the model's memorized knowledge of Formula 1 results.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Rel-f1 label leakage inflates headline averages; excluding rel-f1, LLM+MLP no longer beats RDL on classification, so the central competitiveness claim hinges on contaminated tasks.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: the evaluation assumes the RelBench temporal split gates information flow, but the authors' own observation about rel-f1 shows that the LLM's pretraining corpus contains the test labels. This is not a peripheral issue—the aggregate numbers that support 'competitive to RDL' are sensitive to whether rel-f1 is included. The back-of-envelope recomputation (excluding rel-f1 from Table 1) moves the 1B+MLP average from 76.8 to about 75.6, and the 3B+MLP from 77.1 to about 75.3, while RDL stays at about 76.2. Thus the apparent superiority on classification flips to a small deficit. On regression, the gap also widens. The paper is otherwise thorough and honest: it reports detailed per-configuration tables, acknowledges the rel-f1 anomaly, and uses a sensible temporal filtering procedure. The flaw is in the interpretation of the aggregate results, not in the method's mechanics. Because the issue is concrete and addressable (exclude or separately report rel-f1; add a memorization probe), the conditional verdict is appropriate. I do not see a reason to reject the paper outright, nor to accept it without revision, so the reader's CONDITIONAL verdict stands unchanged.","tokens_in":15105,"tokens_out":7355,"duration_ms":61075,"concrete_test":"Recompute the aggregate classification and regression results in Tables 1 and 2 after excluding all rel-f1 tasks (driver-dnf, driver-top3, driver-position). If the LLM+MLP averages drop below RDL on classification and the regression gap grows substantially, the headline claim depends on pretraining contamination. To directly verify memorization, rerun rel-f1 predictions with driver names, race names, and dates anonymized or replaced by generic identifiers while preserving the relational structure; if AUROC falls to near chance (or the best results no longer occur without related rows), the original scores are not attributable to the constructed documents.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that serialized documents let LLMs match or beat Relational Deep Learning (RDL) on RelBench—rests on the aggregate AUROC averages in Table 1: Llama 3.2 1B+MLP reaches 76.83 and 3B+MLP 77.09 versus RDL's 75.83. But the authors themselves note that on rel-f1 the models 'seem to be mostly relying on their pre-existing factual knowledge of Formula-1' (Section 4), meaning the temporal split does not prevent label leakage from pretraining. Recomputing Table 1 without the two rel-f1 classification tasks (driver-dnf and driver-top3) yields approximate averages of 75.6 for 1B+MLP and 75.3 for 3B+MLP versus 76.2 for RDL—the LLM advantage disappears and reverses. The same pattern holds for regression: removing driver-position widens the MAE gap from 0.5 to about 1.3–1.7. Because rel-f1 contributes three of the tasks and is the only domain where the method appears to beat RDL clearly, the headline result is not a fair test of the proposed document-construction approach. The concern is not that the method is useless—it remains competitive on several tasks—but that the stated central claim is not supported once contaminated tasks are excluded.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a simple LLM-based approach to RelBench relational prediction tasks. For each test entity, it constructs a JSON document containing a task description, up to ninc in-context examples, up to nrel related examples, and nested rows from linked tables up to depth d; a frozen Llama 3.2 base model then either scores the probability of the positive token (classification) or feeds its token embeddings to a small MLP head (regression and classification). The authors compare with LightGBM and Relational Deep Learning (RDL) on 21 RelBench tasks and report average AUROC 76.83/77.09 for the two LLM sizes with MLP heads versus 75.83 for RDL, and MAE 13.76/13.16 versus 12.63 for RDL.","tokens_in":15363,"tokens_out":10967,"duration_ms":100843,"significance":"If the competitiveness claim held, this would be a valuable new baseline: it avoids end-to-end graph training, can exploit a small training subset, and provides a clear algorithmic recipe. The paper is also unusually transparent in reporting the full per-configuration tables (Appendix D) and the algorithmic pseudocode. However, the headline result is contingent on the rel-f1 tasks, where the authors themselves suspect reliance on memorized factual knowledge; the contribution is therefore not yet established. The method is not presented with machine-checked proofs or code, but the algorithmic description is complete enough to replicate with reasonable effort.","major_comments":[{"comment":"The rel-f1 results are not a fair test of the document-construction method because the models appear to rely on memorized Formula-1 knowledge. The authors write in Section 4 that the models 'seem to be mostly relying on their pre-existing factual knowledge of Formula-1'; the RelBench temporal split filters database rows (tv < tp) but cannot remove information from the LLM pretraining corpus. Recomputing Table 1 without driver-dnf and driver-top3 gives average AUROC 75.6 (1B+MLP) and 75.3 (3B+MLP) versus 76.2 for RDL, reversing the reported 76.8/77.1 versus 75.8 advantage. For regression, excluding driver-position changes the MAE gap from 1.13 to 1.33 (1B) and from 0.52 to 0.71 (3B) relative to RDL. On the remaining tasks the LLM+MLP variant is below RDL on 7 of 10 classification tasks and on most regression tasks, so the headline 'competitive to RDL' claim is not supported once the contaminated tasks are removed. The paper should either exclude rel-f1 from the headline averages or provide a leakage-controlled analysis (e.g., results on seasons or entities unlikely to be in pretraining, or a discussion of how many test labels are plausibly memorized).","section":"Section 4, Table 1"},{"comment":"The evaluation protocol selects, per task, the best of the document-parameter combinations on the validation set and then reports the test score of that single selection; Appendix D states 'For each set of document generation parameters, we conducted a single run.' Because there are no repeated seeds, bootstrap intervals, or other variance estimates, and because the test metric is computed on a random 10,000-example sample (Section 4) without a stated seed, the 0.5-1.3 point average differences in Table 1 cannot be distinguished from selection noise. The authors should report confidence intervals (e.g., repeated runs or bootstrap over test samples) and either fix the document parameters a priori or correct for selection over the grid.","section":"Section 4 and Appendix D"}],"minor_comments":[{"comment":"The header is ambiguous: each LLM column contains two numbers (metric-aware inference and +MLP) but the header only says 'Llama 3.2 1B + MLP' and 'Llama 3.2 3B + MLP'; please label the two subcolumns explicitly.","section":"Table 1"},{"comment":"Line 13 of ADD_RELATED_ENTITIES returns X' rather than the updated X; this appears to be a typo, and the return value is never used by the caller.","section":"Algorithm 1"},{"comment":"The grid over ninc, nrel, nnest, and d is described only in Appendix D; a compact statement of the grid in the main text would make the selection process easier to follow.","section":"Section 4"},{"comment":"The Limitations section should explicitly acknowledge the pretraining-leakage risk for rel-f1; currently it only mentions the single benchmark and context-length constraints.","section":"Limitations"},{"comment":"The statement 'We will publish the code for replicating all the results' is not fulfilled in the manuscript; please provide a code link or state availability explicitly.","section":"Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The paper is a straightforward empirical study with a modest but useful contribution if the rel-f1 issue is resolved. The main gate is whether the authors can reanalyze the results without the contaminated tasks and with variance estimates; otherwise the central claim is not supported. The fit with the journal is appropriate for a machine learning venue that values reproducible baselines."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know about this paper: the headline claim—that serialized documents let frozen LLMs match Relational Deep Learning on RelBench—only holds because of the rel-f1 tasks, where the models are doing something closer to memorized trivia than reasoning from the document. The authors say this themselves in Section 4. Recompute Table 1 without the two rel-f1 classification tasks and the LLM+MLP average drops below RDL; the regression gap also widens. So the central \"competitive\" claim is not supported in the way the abstract suggests.\n\nThat said, this is a genuinely useful study. It's the first LLM evaluation on RelBench, and the document-construction recipe—nested related rows, in-context examples, metric-aware inference, MLP head—is a practical baseline that others will cite. The appendix is unusually detailed, with token counts and per-parameter results. The authors are honest about the rel-f1 issue and about the preliminary nature of the work. The method also clearly works on several individual tasks (e.g., rel-stack, rel-amazon), even if the average is flattered.\n\nThe soft spots, in proportion: the rel-f1 contamination is load-bearing, not minor. It undermines the abstract's general claim. The other issues—single runs, best parameters selected per task, evaluation on a 10k sample, no released code—are real but addressable in a revision and don't change the main story once rel-f1 is handled. The lack of error bars is a bit annoying for a benchmark comparison, but the per-parameter tables give some sense of variance.\n\nWho is this for? Anyone building a baseline for relational prediction or studying LLMs on structured data. The recipe is simple and reproducible in principle, and the limitations section is clear. I'd send it to a serious reviewer, but with a request to either exclude rel-f1 from the headline averages or run a leakage check on pretraining corpora, report repeated runs, and release the code. The paper earns a conditional accept, not a desk reject.","headline":"Useful first LLM baseline for RelBench, but the headline competitiveness claim hinges on rel-f1 tasks where the model uses memorized knowledge; still deserves peer review with revisions.","tokens_in":15922,"tokens_out":2227,"would_cite":true,"duration_ms":21013,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A frozen pretrained LLM that reads relational databases as nested JSON documents matches or slightly exceeds the relational deep learning baseline on RelBench classification, and comes within about 0.5 MAE on regression.","keywords":["large language models","relational databases","RelBench","document construction","in-context learning","metric-aware inference","relational deep learning","tabular data"],"falsifier":"Recompute RelBench averages after removing the three rel-f1 tasks or after replacing driver, team, and race names with anonymous IDs before document construction; if the anonymized rel-f1 scores fall to near chance while the original documents score around 78–90 AUROC, or if the remaining average drops below RDL, the leakage channel is confirmed and the headline comparison must be revised.","tokens_in":14861,"feed_emoji":"🗄️","tokens_out":6887,"duration_ms":61469,"temperature":0.7,"pith_summary":"The paper tries to establish that predictive tasks in relational databases can be solved by turning each prediction into a text document and letting a pretrained large language model read it. The authors construct documents by following foreign-key links between tables, nesting related rows as JSON, and adding in-context examples that respect prediction time. On the RelBench benchmark, frozen Llama 3.2 models with a small trained MLP head reach an average classification AUROC of 76.8 and 77.1, above the 75.8 of the relational deep learning baseline, and come within roughly half a point of MAE on regression. If right, this gives the field a much simpler baseline for machine learning on relational data, one that avoids end-to-end training of graph neural networks.","feed_headline":"Flattening databases into text lets small LLMs match graph models","feed_subtitle":"On RelBench classification, frozen Llama 3.2 1B and 3B with a small MLP head reach 76.8/77.1 AUROC vs 75.8 for relational deep learning.","key_machinery":"The load-bearing object is the document construction procedure (Algorithm 1): for each entity, it recursively joins related tables via foreign keys, selects up to $n_{\\mathrm{nest}}$ most recent rows from linked tables before the prediction timestamp, and nests them in JSON so the LLM does not need multi-hop reasoning to connect an entity to its history. Around this sits a metric-aware readout: for AUROC the positive-token probability is used directly, and for MAE a one-hidden-layer MLP head is trained on the frozen LLM's last-token embedding using at most $10^5$ documents. The procedure's role is to convert a heterogeneous relational database into a uniform text input that a pretrained model can score without any parameter updates to the LLM itself.","core_discovery":"The central claim is that denormalizing a relational database into nested text documents is enough to make LLMs competitive with dedicated relational deep learning. The paper's procedure starts from a task-table row, follows foreign keys to primary keys, recursively pulls in up to $n_{\\mathrm{nest}}$ related rows from linked tables up to depth $d$, prepends task and database descriptions plus in-context examples, and serializes everything as JSON with the target last. For binary classification the authors use the probability of the token \"1\" as the score (justified by AUROC's invariance to monotone transforms of $P(y=1\\mid x)$), and for regression they train a small MLP on the LLM's token embeddings rather than using the median of sampled token probabilities, which performs poorly. The resulting frozen-model system matches or slightly exceeds the relational deep learning baseline on RelBench classification while remaining within about 0.5 MAE on regression, establishing LLM-based prediction as a new simple baseline.","pith_inferences":["If test labels from rel-f1 are already memorized by the pretrained model, the headline average is inflated; a fair comparison would re-run the benchmark with entity names anonymized or rel-f1 excluded, and the gap over RDL may shrink or reverse.","Because the MLP head reads a single hidden layer of embeddings, a cheaper linear-probe version of the method is a natural testable variant that would isolate how much relational signal survives in the frozen representations.","Future benchmarks of this kind should include contamination checks, for example probing the base model on task rows stripped of all context, since a temporal split alone does not bound what a pretrained LLM already knows.","A testable extension: use the LLM's own allocation of probability mass to select which in-context examples and nested rows to keep, reducing document length and cost instead of the current grid search."],"forward_implications":["A frozen pretrained LLM plus a small readout becomes a credible baseline for relational database prediction, competing with end-to-end relational deep learning.","Because the MLP head needs only a subset of training documents (up to $10^5$) while several RelBench tasks have over a million training rows, the approach may pay off in low-data regimes.","Which document parameters matter varies by task: some need many related examples, some need deeper nesting into linked tables, and zero-shot documents almost always fail, so context selection is the main lever.","The modest gap between 1B and 3B models suggests document content, not model capacity, drives most of the gain.","The same document construction extends in principle to non-text columns (images, audio) as multimodal foundation models mature."],"supporting_citations":[{"why":"Supplies the RelBench benchmark, the task definitions, the temporal-split protocol, and the RDL and LightGBM numbers the paper compares against.","marker":"[Robinson et al., 2024]"},{"why":"Defines relational deep learning, the graph-neural-network approach that is the paper's main baseline.","marker":"[Fey et al., 2023]"},{"why":"Provides the metric-aware token-probability inference used for classification.","marker":"[Lukasik et al., 2024]"},{"why":"Supports JSON as an effective text representation of tabular data for LLMs.","marker":"[Singha et al., 2023]"},{"why":"Supplies the frozen Llama 3.2 1B and 3B backbones used in all experiments.","marker":"[Dubey et al., 2024]"},{"why":"Supplies the LightGBM gradient-boosted-tree baseline on naively normalized tables.","marker":"[Ke et al., 2017]"},{"why":"Gives the proper scoring rule argument behind treating next-token probability as the conditional distribution $P(y\\mid x)$.","marker":"[Gneiting and Raftery, 2007]"},{"why":"Justifies the median of $P(y\\mid x)$ as the optimal MAE prediction, the route the paper tries for regression before switching to an MLP head.","marker":"[Bishop and Nasrabadi, 2006]"}],"fun_headline_variants":["Small LLMs match graph models after flattening relational DBs","Flattening databases to text lifts small LLMs to parity with graph models","LLMs become competitive on relational tasks via text flattening","RelBench: small LLMs compete with relational deep learning after JSON flattening","Frozen Llama 3.2 matches graph models on relational DB predictions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes the RelBench temporal split is the only gate on information: that test labels are not already inside the LLM's pretraining knowledge, which the paper itself undercuts by noting that on rel-f1 the models appear to lean on memorized Formula-1 facts.","fun_headline_variants_meta":{"raw":{"variants":["Small LLMs match graph models after flattening relational DBs","Flattening databases to text lifts small LLMs to parity with graph models","LLMs become competitive on relational tasks via text flattening","RelBench: small LLMs compete with relational deep learning after JSON flattening","Frozen Llama 3.2 matches graph models on relational DB predictions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000571,"raw_usage":{"total_tokens":2652,"prompt_tokens":846,"completion_tokens":1806,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":462,"completion_tokens_details":{"reasoning_tokens":1712}},"tokens_in":462,"tokens_out":1806,"duration_ms":13654,"temperature":1.0,"reasoning_tokens":1712,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:05:46.364331+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute RelBench averages after removing the three rel-f1 tasks or after replacing driver, team, and race names with anonymous IDs before document construction; if the anonymized rel-f1 scores fall to near chance while the original documents score around 78–90 AUROC, or if the remaining average drops below RDL, the leakage channel is confirmed and the headline comparison must be revised.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the proper scoring rule argument behind treating next-token probability as the conditional distribution $P(y\\mid x)$."}],"review_version":1}