{"id":"c5aa7f21-20dc-4a73-9a3c-158705517692","arxiv_id":"2608.13023","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Relational deep learning models can be evaluated and updated incrementally over time, and incremental fine-tuning matches or beats from-scratch retraining on most standard benchmark tasks.","lead":"Databases grow over time, but current evaluation of relational deep learning models uses one static snapshot; this paper proposes evaluating and updating the same model across many time steps. On standard benchmarks, fine-tuning the model from step to step is often as accurate as retraining from scratch and much cheaper, though it does not win every task.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'consistently outperform' claim rests on selecting checkpoints with the same future labels used for scoring, and the paper's own Tables 3–4 show from-scratch winning several tasks.","rationale":"The reader correctly flags the checkpoint-selection issue in Section 5.3; that is the strongest methodological soft spot. I agree with that concern, but the paper itself contains a more direct contradiction: Tables 3 and 4, which do not use subsequent-increment selection, show from-scratch beating fine-tuning on five of twelve tasks, so the abstract's 'consistently outperform' is overstated even before the selection issue. The fine-tuning advantage in Figure 5 may be an artifact of (i) selecting checkpoints on the exact future increment being reported, and (ii) the from-scratch baseline being starved by the 2000-step/2-hour cap. The paradigm itself—multi-episodic evaluation, drift detection, time-weighted metrics—remains a useful contribution, and the few-shot result (Figure 10) may survive; that is why conditional, not reject. A future-free selection protocol with converged baselines would settle the transfer claim.","tokens_in":13709,"tokens_out":4684,"duration_ms":49055,"concrete_test":"Rerun Figure 5 under a future-free selection rule: for each episode, split the newly available increment into a checkpoint-selection sub-interval (used for early stopping) and a reporting sub-interval; train each regime with a step budget large enough for the from-scratch model to converge (or report training curves), and then compare final held-out scores. If the fine-tuning advantage disappears or reverses in the majority of tasks, revise the abstract's 'consistently outperform' claim. A complementary check is to remove the 2-hour/2000-step cap and verify that each from-scratch baseline reaches its validation plateau before comparison.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is the abstract's assertion that incrementally fine-tuned models consistently outperform the standard, expensive, from-scratch trained baselines. The cleanest evidence is Figure 5, but the protocol in Section 5.3 states that each model is trained up to t_i with the best checkpoint selected via validation on the subsequent increment (up to t_{i+1}), and Figure 5 then reports the validation score for that same subsequent increment. This uses the labels of the very increment being scored to pick the checkpoint and to early-stop. In real deployment, t_{i+1} labels are unavailable at update time, so the reported advantage may reflect selection toward the next increment rather than transfer. The issue is not hypothetical: under the more standard full-training evaluation in Tables 3–4, from-scratch wins on driver-position (3.106 vs 3.256 MAE), site-success (0.393 vs 0.395), user-clicks (0.648 vs 0.659 AUROC), driver-dnf (0.814 vs 0.763), and study-outcome (0.657 vs 0.633). Thus the abstract's 'consistently' is unsupported even by the paper's own main results, independent of the selection subtlety. A second confound is the 2000-step / 2-hour cap in Appendix A.1, which may stop from-scratch training before convergence. These issues make the central performance claim optimistic, although the incremental evaluation paradigm and drift analysis remain useful.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an incremental, multi-episode evaluation and training paradigm for Relational Deep Learning (RDL). It formalizes episode-wise data increments, three fine-tuning regimes (cumulative, incremental, and upsampled) plus a from-scratch baseline, and an exponential-decay time-weighted evaluation metric. Using 12 tasks from RelBench, the paper reports temporal concept drift in most tasks and claims that incrementally fine-tuned models consistently outperform from-scratch baselines, with transfer learning happening in very few training steps. The contributions are framed as a move from static snapshot evaluation to continuous, horizon-shifting evaluation for RDL.","tokens_in":14025,"tokens_out":5875,"duration_ms":60279,"significance":"The multi-episodic evaluation paradigm is a conceptually useful contribution: standard RDL benchmarks evaluate a single chronological split, while real databases grow continuously. The paper's drift analysis over 12 tasks and the proposal of a near-future-weighted metric are also valuable for the community. If the central performance claim were robust, the work would justify a shift toward incremental fine-tuning in RDL practice. However, the claim of consistent superiority of fine-tuning is not supported by the paper's own tables, and the checkpoint-selection protocol in the incremental experiments uses future labels. The framework and analysis remain useful, but the headline empirical conclusion needs substantial rework before the paper can be recommended for publication.","major_comments":[{"comment":"The abstract and conclusion claim that incrementally fine-tuned models 'consistently outperform' from-scratch baselines. This is contradicted by the paper's own full results. In Table 3, from-scratch wins on driver-position at decay 0.0 (3.106 vs. best fine-tuned 3.256) and at decay 0.3 (2.827 vs. 2.858); on site-success, from-scratch achieves 0.393 vs. 0.428 for cumulative fine-tuning. In Table 4, from-scratch wins on user-clicks at decay 0.0 (0.648 vs. 0.659), on driver-dnf at decay 0.0 (0.814 vs. 0.763), and on study-outcome at decay 0.0 (0.657 vs. 0.633). The evidence supports task-dependent gains, not 'consistent' superiority; the claim should be substantially weakened or the analysis should identify which regime helps when.","section":"Abstract, Tables 3-4"},{"comment":"In Section 5.3, the checkpoint for each episode is selected 'via validation on the subsequent increment (up to t_{i+1})', and Figures 5 and 9 then report the validation score for that same subsequent increment. This uses the labels of the very data being scored to choose the early-stopped checkpoint. In a real deployment, t_{i+1} labels are unavailable at the time of the update. This protocol gives the fine-tuning regimes an oracle advantage that the from-scratch baseline does not receive in the same way. Please redo the comparison with checkpoint selection based only on information available at t_i (for example, a held-out portion of the already-observed increment), or report results for fixed training budgets without early stopping on the next increment.","section":"5.3, Figures 5 and 9"},{"comment":"The 2000-step and 2-hour caps per incremental step may under-train the from-scratch baseline, especially for large tasks such as post-votes (2.45M training samples). A fine-tuned model starts from a previously trained initialization, so it needs fewer steps to reach good performance; the comparison is then confounded by compute budget rather than measuring generalization. The paper should report whether from-scratch models converge within the cap, e.g., by showing training curves or using a convergence-based stopping rule, or should use a compute-matched protocol in which from-scratch receives the cumulative number of steps spent by all fine-tuning episodes.","section":"Appendix A.1"}],"minor_comments":[{"comment":"Tasks on rel-avito and rel-f1 driver-top3 have only two training episodes. For these tasks, the claimed transfer ability rests on a single fine-tuning step and should be explicitly reported as a caveat in the text.","section":"5.1, Table 1"},{"comment":"Equation (1) uses (1-alpha)^{i-1}, which is undefined at alpha = 1 for i = 1. Please state that alpha is in [0,1) and treat alpha close to 1 as the limit that isolates the first window.","section":"4.3, Eq. (1)"},{"comment":"The symbol T is used both for the set of relations' tables and for the time domain in the mapping tau: V -> T. This overloaded notation makes the definition of tau confusing; please use different symbols, e.g., T_t for the time domain.","section":"3.1"},{"comment":"References [13] and [14] are the same paper (Gorishniy et al., 'Revisiting deep learning models for tabular data'). Please merge them.","section":"References"},{"comment":"There are several formatting issues: Figure 2 shows 'TRAIN V AL' with an extra space; Figure 3's caption says 'full range off the task windows' instead of 'full range of the task windows'; Table 1's header 'Total Nof' appears to be missing a symbol or space. Please correct these in a final polish.","section":"Figures 2-3, Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper makes a useful framework contribution, but its headline claim needs to be brought in line with its own empirical results and the evaluation protocol must be made leak-free with respect to future labels. The self-citations to REDELEX are appropriate for the implementation. I expect that a revision can address these issues, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this paper gives the RDL field something it genuinely lacks—a multi-episode, incremental evaluation protocol—and then oversells the main finding. The abstract says fine-tuned models \"consistently outperform\" from-scratch baselines, but their own Tables 3 and 4 show from-scratch winning driver-position, site-success, user-clicks, driver-dnf, and study-outcome under several decay settings. So \"consistently\" is contradicted by their own results, even before you get to the checkpoint-selection issue.\n\nWhat's genuinely new and good: the multi-episodic setting is a real gap; RelBench and REDELEX evaluate single snapshots. The drift analysis across time windows is informative. The exponential-decay metric is simple and sensible, and the finding that fine-tuned models get strong performance after 100 steps is a useful, plausible practical takeaway.\n\nSoft spots, in order of severity. First, Section 5.3 selects each episode's best checkpoint via validation on the subsequent increment, and Figure 5 reports the score for that same increment. That's using future labels for model selection, which isn't deployable; you can't validate on t_{i+1} before making predictions for that window. This makes the Figure 5 advantage optimistic. Second, the full-training tables don't have that selection issue and they show mixed results, so the abstract's central claim is overbroad. Third, the 2-hour / 2000-step cap in Appendix A.1 likely under-trains from-scratch on larger rel-stack tasks, another thumb on the scale.\n\nNone of this kills the paper. The protocol is worth having, and the transfer results are still suggestive. But the claims need recalibration: incremental fine-tuning \"can match or beat\" from-scratch, not \"consistently outperforms.\" The authors should also adopt a stricter protocol where checkpoint selection uses only past validation data, or at least be explicit about the optimistic selection.\n\nThis deserves a serious referee. It's a solid benchmark contribution in need of honest framing. I'd bring it to reading group.","headline":"A useful incremental evaluation protocol for RDL, but the 'consistently outperform' claim is not supported by the paper's own full-training tables.","tokens_in":14537,"tokens_out":2204,"would_cite":true,"duration_ms":22911,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Incrementally fine-tuned relational deep learning models consistently outperform expensive from-scratch retraining, so static snapshot benchmarks understate temporal degradation.","keywords":["relational deep learning","multi-episode evaluation","temporal concept drift","incremental fine-tuning","time-weighted evaluation","graph neural networks","transfer learning","temporal benchmark"],"falsifier":"Re-run one classification task with early stopping and checkpoint selection restricted to labels available at or before the current episode, then compare fine-tuned and from-scratch models on the next increment. If fine-tuning no longer matches or beats from-scratch, the paper's deployment-oriented claim would not survive realistic operation.","tokens_in":13516,"feed_emoji":"🕒","tokens_out":4621,"duration_ms":48819,"temperature":0.7,"pith_summary":"Relational deep learning (RDL) models turn multi-table databases into heterogeneous graphs and learn on them end to end, but standard evaluations freeze a database at one moment in time. This paper argues that such snapshot evaluation hides how models go stale as new rows arrive, and it proposes testing and training models over successive increments of data instead. Across twelve predictive tasks on four large-scale databases, the paper finds temporal concept drift in most tasks and shows that fine-tuning from the previous episode consistently matches or beats the much more expensive practice of retraining from scratch. The paper also introduces a time-weighted metric that gives more weight to near-future accuracy, matching operational priorities. If correct, static benchmarks should be replaced by multi-episode, lifecycle-aware evaluation.","feed_headline":"Fine-tuning beats from-scratch retraining as relational databases grow","feed_subtitle":"A multi-episode benchmark shows concept drift is common and weight transfer adapts models faster and at least as accurately.","key_machinery":"The load-bearing mechanism is the multi-episodic evaluation loop. A task is defined by a future-pointing query $Q_{\\Delta w}(t)$ that produces labels in a window of size $\\Delta w$ after each anchor time $t$; consecutive anchors are spaced by a fixed increment $\\Delta I$, and each episode appends the newly revealed rows and links to the historical graph while a leakage barrier keeps validation and test horizons strictly ahead. This loop supports four training regimes, from scratch, cumulative fine-tuning, incremental fine-tuning, and upsampled fine-tuning, and the evaluation metric assigns sample weight $(1-\\alpha)^{i-1}$ to the $i$-th future window, reducing to a uniform mean at $\\alpha=0$ and isolating the next window as $\\alpha\\to 1$. The incremental protocol is what turns an otherwise static benchmark into a measure of temporal robustness.","core_discovery":"The paper's central claim is that standard single-episode RDL benchmarks are inadequate because relational databases grow continuously, and that a multi-episode protocol is both necessary and practical. On its own terms, the paper demonstrates that temporal concept drift, the distribution of target values shifting as time advances, occurs in the majority of the twelve tasks it studies, that models trained from scratch at each episode degrade further from the horizon they start from, and that three fine-tuning regimes (cumulative, incremental, and upsampled) transfer knowledge well enough to match or beat from-scratch retraining, sometimes within a hundred training steps. The newly proposed exponential-decay metric weights each future window by $(1-\\alpha)^{i-1}$, so a decay near 1 isolates performance on the immediate next window; varying the decay preserves the relative ranking of regimes in most tasks. The paper concludes that snapshot evaluation should give way to incremental evaluation and that incremental fine-tuning should replace repeated from-scratch training as the default strategy for RDL models deployed on growing databases.","pith_inferences":["If checkpoints cannot be selected using next-increment labels, the reported fine-tuning advantage is optimistic; testing with past-only validation is a direct check.","The same protocol could serve as a continual-learning benchmark for relational data, bridging RDL and continual graph learning.","The time-weighted metric could be generalized to arbitrary operational cost curves, not just exponential decay.","Extending from append-only growth to updates and deletions is the next stress test; the paper itself flags CRUD operations as open."],"forward_implications":["Static single-episode scores likely overstate long-horizon performance; multi-episode evaluation is needed to see how quickly a model obsolesces.","Incremental fine-tuning becomes a strong and cheap default, so repeated from-scratch retraining can be dropped in most continuous-deployment settings.","Near-future-weighted evaluation can change which regime looks best in some tasks, so benchmark reports should state the decay factor alongside the metric.","Few-step adaptation suggests pretrained RDL weights are a practical starting point for continual deployment, reducing per-update compute."],"supporting_citations":[{"why":"Supplies the four databases and twelve predictive tasks whose chronological splits the incremental protocol reorganizes.","marker":"[33]"},{"why":"Provides the RDL exploration framework and evaluation libraries that the incremental infrastructure builds on.","marker":"[29]"},{"why":"Defines the RDL graph representation, task tables, and chronological split conventions that the paper extends.","marker":"[11]"},{"why":"Supplies the message-passing architecture used as the model backbone in the experiments.","marker":"[16]"},{"why":"Provides the newer benchmark repository version used for large-scale relational data and tasks.","marker":"[15]"}],"fun_headline_variants":["Fine-tuning adapts relational deep learning to growing data","Multi-episode training beats from-scratch as databases evolve","Incremental fine-tuning outperforms retraining on growing databases","Relational models drift: fine-tuning beats retraining over time","Temporal concept drift calls for incremental RDL training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported advantage assumes it is fair to pick each episode's best checkpoint by validating on the next increment of labels, because the same future labels also appear in the reported score; if a deployed model must act before seeing those labels, the fine-tuning advantage may be optimistic.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuning adapts relational deep learning to growing data","Multi-episode training beats from-scratch as databases evolve","Incremental fine-tuning outperforms retraining on growing databases","Relational models drift: fine-tuning beats retraining over time","Temporal concept drift calls for incremental RDL training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000185,"raw_usage":{"total_tokens":1314,"prompt_tokens":930,"completion_tokens":384,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":302}},"tokens_in":546,"tokens_out":384,"duration_ms":3826,"temperature":1.0,"reasoning_tokens":302,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:19:40.096628+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run one classification task with early stopping and checkpoint selection restricted to labels available at or before the current episode, then compare fine-tuned and from-scratch models on the next increment. If fine-tuning no longer matches or beats from-scratch, the paper's deployment-oriented claim would not survive realistic operation.","supporting_citations":[{"cited_title":"Robinson, R","cited_arxiv_id":null,"evidence_quote":"Supplies the four databases and twelve predictive tasks whose chronological splits the incremental protocol reorganizes."},{"cited_title":"Peleška and G","cited_arxiv_id":null,"evidence_quote":"Provides the RDL exploration framework and evaluation libraries that the incremental infrastructure builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the RDL graph representation, task tables, and chronological split conventions that the paper extends."},{"cited_title":"Hamilton, Z","cited_arxiv_id":null,"evidence_quote":"Supplies the message-passing architecture used as the model backbone in the experiments."}],"review_version":1}