REVIEW 4 major objections 5 minor 1 cited by
Not All Features Deserve Attention: Graph-Guided Dependency Learning for Tabular Data Generation with Language Models
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Graph-guided attention steers LLMs to focus on real feature dependencies in tabular data.
desk verdict A promising direction undermined by a mis-specified core mechanism: Eq. 6 inverts the intended gating, but the underlying idea and thorough evaluation merit referee attention. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is a per-head dynamic dependency graph $W$ with entries $w_{ij} = G_\phi(q_i, k_j)$, computed by a two-layer MLP over query-key pairs, which is fused into attention through the multiplicative gating form $\tilde{a}_{ij} = a_{ij} \cdot \log(w_{ij} + \epsilon)$. The graph is supervised at feature level by functional dependencies extracted with the HyFD algorithm, through a soft constraint $\phi(w, \alpha) = \log(1 + \exp(\alpha - w))$ applied to the averaged connection strength $\bar{w}_{Y,X}$ between the token sets of dependent feature pairs.
What would settle it
Train GraDe on the Diabetes dataset with the same hyperparameters but with the gating term replaced by $\tilde{a}_{ij} = a_{ij} \cdot \log(1 - w_{ij} + \epsilon)$; if the reported gains persist, the logarithmic gate itself is not the cause. Also rerun the low-data experiments with FDs extracted only from the training subset and compare the 250-sample accuracy.
Extended reading notes
Core claim
GraDe claims that the structural mismatch between the dense attention of language models and the sparse dependency structure of tabular data can be closed by injecting a learnable, sparse token-level dependency graph directly into self-attention. The core mechanism is a logarithmic gating term: modulated scores take the form $\tilde{a}_{ij} = a_{ij} \cdot \log(w_{ij} + \epsilon)$, so learned edge weights near one preserve attention while weights near zero suppress it. This dynamic graph is trained with a composite objective that combines autoregressive language modeling, an L1 sparsity penalty, and a softplus constraint that pushes average connection strength between tokens of functionally dependent features above a threshold. The paper reports that this design outperforms the GReaT LLM baseline by up to 12% on complex medical datasets (Diabetes, Sick), halves the correlation error on those datasets, and preserves logical constraints such as state-code-to-bird and education-to-education-num mappings at a 1.12% average violation rate.
Load-bearing premise
The central premise is that multiplying attention scores by $\log(w_{ij} + \epsilon)$ really makes the model focus on the learned strong dependencies and suppress the weak ones, and that the functional dependencies extracted from the full dataset remain valid when training on small subsets.
Editorial extensions
If this is right
- LLM-based tabular generators can preserve logical consistency in synthetic data when dependency structure is encoded as an inductive bias rather than left to implicit learning.
- A parameter-efficient variant that updates only the graph-attention modules retains most of the utility gain while training roughly 100 million fewer parameters on GPT-2.
- The approach transfers to a larger backbone and to high-dimensional data, where the sparsity regularizer contributes more on a 124-feature medical dataset.
- In low-data regimes, the structural guidance yields larger gains: up to roughly 15% over the baseline at 250 training examples.
- Privacy metrics improve alongside fidelity, suggesting that explicit dependency modeling reduces both correlation error and distance-to-closest-record scores.
Reading between the lines
- The claim implies that attention sparsity itself, not just the external FD labels, is responsible for the gains; a testable extension would be to run GraDe with random or corrupted FD sets and check how much of the improvement survives.
- The gating formula suggests a continuum: a perfectly learned graph should recover standard attention when all weights are near one, so the method subsumes the baseline at its least sparse extreme.
- Because FD extraction is performed on the full dataset, the low-data advantage could partially reflect information leakage; recomputing FDs from the training subset alone would test whether the gain is real in data-scarce settings.
- The intuition that sparse dependencies help most when features have ambiguous names or many numerical attributes could be turned into a diagnostic: dataset-level FD counts or graph sparsity could predict where the method beats dense-attention baselines.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes GraDe, a method that enhances LLM-based tabular data generation by learning a dynamic token-level dependency graph and using it to modulate self-attention. Graph edge weights are computed from query/key vectors, and a logarithmic gating function is applied to attention scores (Eq. 6). The model is trained with a composite loss combining language modeling, graph sparsity, and a functional-dependency alignment term (Eqs. 10-12). Experiments on five main datasets plus two additional ones compare GraDe and a parameter-efficient variant GraDe-Light against GAN-, VAE-, diffusion-, and LLM-based baselines, reporting machine learning efficiency, correlation fidelity, privacy risk, and constraint-violation metrics.
Significance. If the mechanism worked as described, the paper would address a real and recognized limitation: the mismatch between dense transformer attention and the sparse feature dependencies of tabular data. The parameter-efficient GraDe-Light variant is a useful practical contribution, and the manuscript ships a code link and a fairly broad evaluation suite. However, the central gating equation, as written, does not implement the stated behavior and is mathematically inconsistent with the formalization of attention in Eq. (2). Because the main claim rests on this mechanism, the current results cannot be attributed to the described method without substantial correction and rerunning of experiments.
major comments (4)
- [§4.2, Eq. (6)] The stated logarithmic gating does not implement the claimed behavior. For epsilon in (0,1), log(w_ij + epsilon) is negative for w_ij < 1 - epsilon and is at most log(1 + epsilon), i.e., approximately epsilon, when w_ij = 1. Thus a strong dependency (w≈1) is multiplied by a factor near zero, while a weak dependency (w≈0) is multiplied by a large negative factor. Since the raw dot-product scores a_ij can be negative, this can amplify and sign-flip weak edges rather than suppressing them. The mechanism is therefore the opposite of the description in the same paragraph, and the reported gains cannot be attributed to the described gating. The authors must correct Eq. (6) with a monotone nonnegative gate and rerun the experiments, or, if the released code implements a different formula, rewrite the method description to match the code.
- [§3, Eq. (2) and §4.2, Eq. (6)] There is an internal inconsistency in the definition of a_ij. Eq. (2) defines a_ij as the softmax-normalized attention weight, but the text after Eq. (6) calls a_ij the unnormalized attention score and then states that the modulated scores are subsequently normalized via softmax. This ambiguity prevents the reader from reproducing the exact computation, and it changes the mathematical analysis in the previous comment. Please define the input to the modulation precisely and remove the double normalization or justify it explicitly.
- [§A.2, Table 5 and §4.3] The intrinsic-constraint evaluation in Table 5 uses the same functional dependencies that are fed into the training objective via L_FD in Eq. (8). The low violation rates are therefore partly a check that the model learned the supervised dependencies, not an independent test of structural generalization. To support the claim that GraDe improves dependency preservation, the authors should report violation rates for held-out functional dependencies, or for a variant trained without L_FD, and ideally relate these results to the ablation variants in Table 3.
- [§6.2 and §B.4] The paper does not state whether the functional dependencies used in the low-data experiments were extracted from each training subset or from the full dataset. If HyFD was run on the full data before the 80/20 split, then the experiments with 250-4000 training samples inherit supervision from the held-out test set, making the comparison with GReaT unfair. Please clarify the extraction protocol and, if necessary, rerun the low-data experiments using FDs extracted only from the corresponding training subset.
minor comments (5)
- [§2 and §5] There are several inconsistent notation and spacing artifacts, such as "V AEs" and "TV AE", which should be unified and corrected.
- [§6.1, Figure 3] The correlation-error histograms lack axis labels and sample-size information, making the claim that GraDe's error distribution is 'more concentrated near zero' difficult to verify from the figure alone.
- [§B.3, Table 11] For Bird and Housing, the table does not report class distribution or target-type details that would be useful for interpreting the MLE results; please add the missing information or state why it is omitted.
- [§B.2, Table 10] The reduction percentages are mentioned in the text but not derived in the table; add a column for the percentage of trainable parameters or explicitly state the computation.
- [§A.2] The constraint description for Income says 'education level matching education code', while the FD list uses 'education' and 'education-num'; align the terminology for clarity.
Circularity Check
FD-alignment supervision makes the Appendix A.2 constraint-fidelity result partly a check of the training objective; the main MLE comparison is independent.
-
fitted input called prediction
[Section 4.4, Eq. (8)/(12); Appendix A.2, Table 5]
"LFD = Σ_{X→Y∈F} φ(w̄_Y,X, α) ... GraDe achieves the lowest average violation rate with 1.12% ... This superior performance stems from our explicit modeling of functional dependencies."
The FD alignment loss directly raises the average graph connection strength w̄_Y,X for each known FD in F. The Appendix A.2 'intrinsic constraint' metrics (state bird, education_num, ocean_proximity) are exactly the functional dependencies extracted by HyFD and then supervised by LFD. Reporting low violation rates for these same constraints is therefore partly a measurement of whether the training objective was satisfied, not an independent prediction of dependency preservation. The main ML efficiency result in Table 1 is not affected because it evaluates downstream classifiers on generated rows without using the FD objective.
full rationale
The paper's central derivation is not circular: the graph-guided attention mechanism (Eq. 4-6), the FD alignment loss (Eq. 7-9), and the composite training loss (Eq. 12) are all defined from the paper's own equations, and the headline MLE improvements over GReaT and other baselines (Table 1) are empirical comparisons against external methods, not quantities forced by construction. The self-citations to the authors' own prior work (GReaT, P-TA, SPADA) are descriptive related-work mentions and are not load-bearing. The only materially circular element is the Appendix A.2 constraint-fidelity evaluation, which measures violations of the same functional dependencies that LFD explicitly optimizes; that sub-claim is partly a re-statement of the training objective. Two further caveats are correctness or leakage issues rather than circularity: the Eq. 6 log-gating formula log(w+ε) is non-positive for w in [0,1] and so does not literally implement the stated 'preserve when w≈1, suppress when w≈0' behavior, and the low-data experiments do not state whether FDs were re-extracted from each 250-4000 sample subset or taken from the full dataset, which would be a data-leakage concern. Neither affects the circularity score as defined here, but both should be checked in the release code and experimental logs.
Assumptions & free parameters
free parameters (6)
- lambda_sparse =
0.001
- lambda_FD =
0.1
- alpha (FD threshold) =
not specified
- epsilon (stability constant) =
not specified
- temperature =
0.7
- nucleus p =
0.95
assumptions (4)
- domain assumption Tabular data has sparse dependency structure with |E'| << |V'|^2
- domain assumption Functional dependencies discovered by HyFD on the training set are valid priors for the generative task
- ad hoc to paper The gating function log(w+epsilon) preserves strong dependencies and suppresses weak ones
- domain assumption Soft constraints with softplus penalty effectively guide the model without over-constraining
Cite this review
Pith. "Pith review of Not All Features Deserve Attention: Graph-Guided Dependency Learning for Tabular Data Generation with Language Models." pith.science (2026). https://pith.science/paper/A327T5X5
@misc{pith2026250718504,
author = {Pith},
title = {Pith review of: Not All Features Deserve Attention: Graph-Guided Dependency Learning for Tabular Data Generation with Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/A327T5X5}},
note = {Machine review of arXiv:2507.18504}
}
read the original abstract
Large Language Models (LLMs) have shown strong potential for tabular data generation by modeling textualized feature-value pairs. However, tabular data inherently exhibits sparse feature-level dependencies, where many feature interactions are structurally insignificant. This creates a fundamental mismatch as LLMs' self-attention mechanism inevitably distributes focus across all pairs, diluting attention on critical relationships, particularly in datasets with complex dependencies or semantically ambiguous features. To address this limitation, we propose GraDe (Graph-Guided Dependency Learning), a novel method that explicitly integrates sparse dependency graphs into LLMs' attention mechanism. GraDe employs a lightweight dynamic graph learning module guided by externally extracted functional dependencies, prioritizing key feature interactions while suppressing irrelevant ones. Our experiments across diverse real-world datasets demonstrate that GraDe outperforms existing LLM-based approaches by up to 12% on complex datasets while achieving competitive results with state-of-the-art approaches in synthetic data quality. Our method is minimally intrusive yet effective, offering a practical solution for structure-aware tabular data modeling with LLMs.
Forward citations
Cited by 1 Pith paper
-
Attributes as Textual Genes: Leveraging LLMs as Genetic Algorithm Simulators for Conditional Synthetic Data Generation
Genetic Prompt uses LLMs to run semantic-level crossover and mutation on text attributes from two far-apart parent examples, producing synthetic data that improves downstream NLP performance, especially for rare classes.
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.