Deep Semantic Graph Learning via LLM based Node Enhancement

Chuanqi Shi; Hang Zhang; Lun Wang; Shaoshuai Du; Yanxin Shen; Yixian Shen; Yiyi Tao

REVIEW 4 major objections 6 minor 17 references

Deep Semantic Graph Learning via LLM based Node Enhancement

T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLM-generated text features consistently outperform traditional word embeddings for node classification on text-attributed graphs, with Graph Transformers leading when labels are abundant and GCNs proving more robust under low label ratios.

desk verdict Useful empirical sweep of LLM embeddings as node features, but the abstract overclaims what the authors' own low-label tables show, and missing code/seeds/statistics keeps me from relying on the numbers. read the letter →

arxiv 2502.07982 v1 pith:NAQUYQNH submitted 2025-02-11 cs.AI

Chuanqi Shi , Yiyi Tao , Hang Zhang , Lun Wang , Shaoshuai Du , Yixian Shen , Yanxin Shen This is my paper · ORCID

classification cs.AI

keywords largelanguagemodelsgraphneuralnetworksnodeclassificationtext-attributedgraphstransformersentenceembeddingsfeatureenhancement

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether replacing traditional word-count features with large-language-model (LLM) embeddings improves node classification in text-attributed graphs. Using Cora and Pubmed, the authors compare six feature encoders—from TF-IDF-like word vectors to LLM-generated embeddings—across GCN, MLP, and Graph Transformer architectures. They find that LLM-based features consistently outperform shallow embeddings, by up to 12.54 percentage points on Pubmed under a high-label split. They also conclude that Graph Transformers are the best-performing graph architecture when labels are abundant, while simpler GCNs hold up better when training labels are scarce. The significance is that feature quality appears to be the main lever, and the choice of graph architecture should depend on data availability.

What carries the argument

The central mechanism is the LLM-generated node embedding matrix, obtained by passing each node's text through a pretrained language model to produce a dense vector that replaces the original bag-of-words features. These embeddings are then fed into standard graph models that perform neighborhood aggregation: GCNs use local message passing, Graph Transformers use multi-head self-attention over the graph to combine local and global structure, and MLPs serve as a no-aggregation baseline. The work relies on the linguistic richness of LLM embeddings—contextual, sense-aware representations learned from large corpora—to supply semantic information that shallow word embeddings lack.

What would settle it

A concrete check: rerun the Pubmed high-label experiment with per-architecture and per-embedding hyperparameter search (for example, learning rate in {0.001, 0.005, 0.01} and depth in {2, 4, 8}); if the Word-feature baseline closes the gap to within one standard deviation of the best LLM-feature result, the paper's claim that LLM features improve graph learning would be substantially weakened. Alternatively, evaluating on a held-out text-attributed graph outside citation networks where LLM embeddings fail to beat TF-IDF would contradict the general claim of improvement.

Watch

Extended reading notes

Core claim

The central claim is that using LLMs as feature-level enhancers for node text—generating dense semantic embeddings for each node's text before any graph aggregation—substantially improves node classification accuracy over traditional non-contextual embeddings such as word-frequency vectors. In the high-label regime (60-20-20 split), Graph Transformers paired with LLM features achieve the highest accuracies, for example 81.38% on Pubmed with the PaLM-Cortex-001 embedding and 82.31% on Cora with Sentence-BERT. In the low-label regime (20 nodes per class), however, the advantage of Graph Transformers diminishes and GCNs with LLM features become more competitive, as the 65.90% of SBERT-GCN on Pubmed versus 64.32% of its Graph Transformer counterpart illustrates. The authors interpret this as evidence that complex attention architectures need enough labeled data to leverage their capacity.

Load-bearing premise

The comparison assumes that one fixed training recipe—4-layer network, 64 hidden units, learning rate 0.01, weight decay 5e-4, dropout 0.5—is equally suitable for every embedding type and every graph architecture, so accuracy differences reflect feature and architecture quality rather than hyperparameter mismatch.

Editorial extensions

If this is right

Replacing shallow word features with LLM embeddings can yield double-digit accuracy gains on node classification, as the 12.54-point gap between the PaLM-Cortex-001 Transformer (81.38%) and Word Transformer (68.84%) on Pubmed demonstrates.
Graph Transformers appear to be the strongest GNN architecture when labels are plentiful, but this advantage reverses or disappears under low label ratios.
MLPs consistently underperform their graph counterparts, showing that graph structure remains informative even when text features are semantically rich.
The best LLM choice depends on data availability: the PaLM-Cortex-001 embedding excels with abundant labels, whereas SBERT offers more consistent performance across label ratios.
A fixed, untuned training recipe can still reveal clear ordering among features, implying that embedding quality is a primary driver of downstream accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The fixed hyperparameter recipe may systematically disadvantage Graph Transformers in low-label settings, so per-architecture tuning could change the reported ordering between GCNs and Graph Transformers.
Because the two benchmarks are both citation networks, the generality of the findings to other text-attributed graphs (e.g., product co-purchase or social networks) remains untested.
The framework suggests that simple feature substitution may capture most of the benefit of LLMs in graph learning, raising the question of whether more complex model modifications add value beyond tuned features.
One testable extension is to combine LLM-generated features with edge-level semantics, for example by using pairwise text similarity to re-weight graph edges, which the current framework does not exploit.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a benchmark paper, not a method paper. It swaps TF-IDF or word features for pretrained LLM embeddings (DeBERTa, LLaMA, SBERT, E5, Ada, Google) and runs GCN, MLP, and Graph Transformer on Cora and PubMed under two label ratios. That systematic sweep is the useful part. The high-label results are striking: on PubMed, Google embeddings plus Graph Transformer get 81.38% versus 68.84% for TF-IDF, a 12.5-point gain. The authors report standard deviations and include all four tables, so the main empirical pattern is visible and, in principle, checkable.

The soft spots are real. The abstract and the contribution bullets claim LLM features "significantly improve" node classification and that Graph Transformers "perform optimally" with LLM features, but the paper's own low-label tables contradict both. On Cora low-label, TF-IDF beats every LLM embedding for the Graph Transformer (44.73 vs. the next best 44.48 for SBERT), and the best overall is SBERT-GCN (45.10), not any Graph Transformer. On PubMed low-label, TF-IDF-GCN beats both LLAMA-GCN and Google-GCN by wide margins. The authors do hedge in the conclusion—they note that simpler GCNs are often better under limited labels—so the contradiction is mostly between the abstract and the body, but that mismatch is exactly what a reader trips over.

There are also reproducibility issues. The text says both "5 random seeds" and "three independent runs." No code or seed values are released. There is no significance testing, so "significantly improve" is doing no statistical work. The fixed training recipe (4-layer, 64 hidden, lr 0.01, dropout 0.5) may not suit every embedding and architecture; the paper itself suspects Graph Transformers struggle with limited data, but with one recipe you cannot cleanly separate feature quality from hyperparameter mismatch. The citation list also has some odd entries—the PubMed dataset is credited to the GAT paper, and the sentence-embedding cite is an LSTM paper.

Bottom line: the empirical sweep is worth knowing about, and the high-label result is probably real, but as printed the central claims are not supported by the paper's own tables, and the missing artifacts make it hard to verify. I would not cite it yet, and I would not publish it in its current form. But I would send it to a referee, because a cleaned-up version with narrowed claims and released code could be a useful practical reference for practitioners choosing embeddings and architectures. If you have students working on LLM features for graphs, the tables are worth a look, but caution them to read the low-label columns.

Referee Report

4 major / 6 minor

Summary. The paper proposes a feature-level framework in which LLM-generated text embeddings (SBERT, E5, ADA, Google, LLaMA, DeBERTa) are used as initial node features for GCN, MLP, and Graph Transformer models, and evaluates these combinations on Cora and PubMed under high-label (60-20-20) and low-label (20 nodes per class) splits. The authors claim that LLM-enhanced features significantly improve node classification and that Graph Transformer models perform optimally among GNNs when combined with LLM features, while also noting that low-label settings favor simpler architectures. All experiments use a single fixed training configuration, and results are reported as accuracy with standard deviations across four tables.

Significance. If the claims were supported, the paper would provide a useful practical benchmark for combining LLM embeddings with graph architectures for text-attributed node classification. The manuscript has some strengths: it evaluates multiple externally pretrained embeddings on standard benchmark datasets, reports standard deviations, and avoids circular derivations. However, the central conclusions are contradicted by the paper's own tabulated results in the low-label regime, and the absence of significance tests and the use of a single fixed hyperparameter recipe undermine the empirical ordering claims. As presented, the work does not establish its headline contributions.

major comments (4)

[Abstract and Section I (contributions)] The unqualified claims that 'LLM-enhanced node features significantly improve the performance of graph learning models' and that 'Graph Transformer-based models perform optimally among GNN-based methods when combined with LLM-enhanced features' are contradicted by the paper's own tables. In Table IV (Cora, low-label), the TF-IDF baseline (TIFI) achieves 44.73 with Graph Transformer, while every LLM embedding is lower (SBERT 44.48, ADA 40.46, Google 38.58, LLAMA 29.30); in Table III (PubMed, low-label), LLAMA and Google embeddings underperform the TF-IDF baseline in both the GCN column (48.51 and 52.06 vs. 57.36) and the Graph Transformer column (48.88 and 53.40 vs. 58.09). Additionally, in the low-label setting SBERT-GCN outperforms SBERT-Graph Transformer on both datasets (45.10 vs. 44.48 on Cora; 65.90 vs. 64.32 on PubMed). These results directly falsify the stated contributions as written.
[Section III, experimental setup] All models are trained with one fixed hyperparameter configuration (Adam lr=0.01, weight-decay 5e-4, 4-layer architecture, 64 hidden dimensions, dropout=0.5, 300 epochs, early stopping patience 10) with no per-embedding or per-architecture tuning. Because the paper's central contribution is an ordering of embeddings and architectures, differences in accuracy could be attributable to hyperparameter mismatch rather than to feature quality. The abstract's use of 'significantly improve' is also unsupported because no significance tests are reported. Furthermore, Section III states both that experiments were repeated across '5 random seeds' and that results represent 'the average performance across three independent runs'; this inconsistency must be resolved, and the standard deviations should be accompanied by clarified seed and splitting details.
[Section IV, analysis text and Tables III-IV] The assertion in Section IV that 'Traditional Word embeddings consistently underperform' is not supported by the reported numbers. In the low-label Cora Graph Transformer column, the TF-IDF baseline (TIFI) is the best entry (44.73), and in the low-label PubMed Graph Transformer column, the Word baseline (55.17) exceeds the LLAMA embedding (48.88). The conclusion section's more measured statement that 'their optimal utilization depends on data availability' directly contradicts the unqualified contributions in the abstract and introduction. The paper should either restrict its claims to the high-label regime or reconcile these statements with a consistent interpretation of the tables.
[Section III and Section IV, reproducibility] The Graph Transformer architecture is not described beyond the generic attention equation in Eq. (3); the number of attention heads, any use of positional encodings, normalization layers, and the mechanism for injecting graph adjacency into the attention are unspecified. No code or seed values are released, and the LLaMA implementation is described only as 'via LLaMA-cpp3' with '[EOS] token embeddings' but no version or pooling details. Given that the conclusions rest entirely on the empirical ordering in Tables I-IV, this lack of architectural and implementation detail prevents verification or reproduction.

minor comments (6)

[Section IV, opening] 'As show in Table 1 to 4' should be 'As shown in Tables I to IV'; the text should also use a consistent table-numbering style (roman numerals in captions versus arabic numerals in the text).
[Tables I-IV] The acronym 'TIFI' appears without definition; presumably it denotes TF-IDF features, but the term should be introduced in Section III where TF-IDF is described.
[Throughout] The dataset name is spelled 'Pubmed' in Section II but 'PubMed' elsewhere; standardize to 'PubMed'.
[Author list] The author line contains 'Y anxin Shen' with an extra space; it should read 'Yanxin Shen'.
[Section II, Eq. (4)] Equation (4), the scaling law N = N0 · alpha^S, is not used anywhere in the paper; either connect it to the evaluation or remove it to avoid an unexplained formalism.
[References] Some references appear tangential (e.g., [2] on coreference resolution and [6] on blackbox attacks); the related-work grounding would be strengthened by citing standard text-attributed graph benchmarks and Graph Transformer baselines more directly.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper reports empirical comparisons of external embeddings and benchmark datasets, with no fitted parameter renamed as a prediction and no load-bearing self-citation.

full rationale

The paper's claims are empirical orderings obtained by combining off-the-shelf text embeddings (TF-IDF, SBERT, ADA, Google, LLAMA, Word) with fixed GNN architectures (GCN, MLP, Graph Transformer) on the public Cora and PubMed benchmarks. No quantity is derived from an assumption that already contains the conclusion. The LLM embeddings are externally pretrained, not fitted to the target labels or to the GNN outputs, so the accuracy differences reported in Tables I-IV are measured outcomes rather than identities. The contribution list and abstract assert that LLM-enhanced features improve performance and that Graph Transformers perform optimally with such features, but these are empirical generalizations that can be checked against the reported tables; even if the low-label tables contradict the unqualified phrasing, that is an internal-consistency or correctness concern, not a circularity concern. There is no derivation chain, no fitted input called a prediction, and no self-citation used to justify the central empirical result. Consequently, the circularity score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No new entities, parameters, or architectural components are invented. The central empirical claim depends on a fixed hyperparameter recipe, two benchmark datasets, and transfer from pretrained embeddings, none of which is independently validated within the paper.

free parameters (2)

Fixed hyperparameter configuration = lr=0.01, weight_decay=5e-4, hidden=64, layers=4, dropout=0.5, epochs=300, patience=10
Hand-chosen and held fixed across all models; if suboptimal for some embedding or architecture pairs, reported accuracy gaps and the claimed ordering may be artifacts.
Label split protocol = High: 60/20/20; low: 20 per class train, 500 validation, 1000 test
Chosen by the authors; results are conditional on these splits, and no split seeds or variance across splits are reported.

assumptions (4)

domain assumption Cora and PubMed are representative text-attributed graph benchmarks and the fixed split procedures are sufficient to support general conclusions.
The paper generalizes from two citation graphs to graph learning broadly without additional datasets or domain variation (Section III).
domain assumption Pretrained embeddings used as fixed feature extractors capture the semantic information needed for node labels.
No fine-tuning or prompt engineering is described; the claimed semantic benefit relies on transfer from pretraining (Sections II and III).
standard math Standard GNN message passing, Transformer attention, and InfoNCE equations are accepted as correct from prior literature.
Architecture definitions are invoked from references and not re-derived (Section II, Equations 1 to 3 and 5).
domain assumption Reported standard deviations over three to five runs support the claim of significant improvement without a formal significance test.
The paper says 'significantly improve' but does not report p-values, confidence intervals, or paired tests across seeds (Sections III and IV).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Deep Semantic Graph Learning via LLM based Node Enhancement." pith.science (2026). https://pith.science/paper/NAQUYQNH

@misc{pith2026250207982,
  author       = {Pith},
  title        = {Pith review of: Deep Semantic Graph Learning via LLM based Node Enhancement},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NAQUYQNH}},
  note         = {Machine review of arXiv:2502.07982}
}

read the original abstract

Graph learning has attracted significant attention due to its widespread real-world applications. Current mainstream approaches rely on text node features and obtain initial node embeddings through shallow embedding learning using GNNs, which shows limitations in capturing deep textual semantics. Recent advances in Large Language Models (LLMs) have demonstrated superior capabilities in understanding text semantics, transforming traditional text feature processing. This paper proposes a novel framework that combines Graph Transformer architecture with LLM-enhanced node features. Specifically, we leverage LLMs to generate rich semantic representations of text nodes, which are then processed by a multi-head self-attention mechanism in the Graph Transformer to capture both local and global graph structural information. Our model utilizes the Transformer's attention mechanism to dynamically aggregate neighborhood information while preserving the semantic richness provided by LLM embeddings. Experimental results demonstrate that the LLM-enhanced node features significantly improve the performance of graph learning models on node classification tasks. This approach shows promising results across multiple graph learning tasks, offering a practical direction for combining graph networks with language models.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 12 canonical work pages

[1]

A comprehensive survey on gr aph neural networks,

Z. Wu, S. Pan, F. Chen et al., “A comprehensive survey on gr aph neural networks,” IEEE Transactions on Neural Networks and Learning Systems, vol. 32, no. 1, pp. 4–24, 2020

work page 2020
[2]

Bridging context gaps: Lev eraging coreference resolution for long contextual understanding ,

Y . Liu, X. Peng, J. Cao et al., “Bridging context gaps: Lev eraging coreference resolution for long contextual understanding ,” arXiv preprint arXiv:2410.01671, 2024

arXiv 2024
[3]

BERT: a review of applications in natura l language processing and understanding,

M. V . Koroteev, “BERT: a review of applications in natura l language processing and understanding,” arXiv preprint arXiv:2103.11943 , 2021

arXiv 2021
[4]

Deberta: Decoding-enhanced bert with disentangled attention,

P . He, X. Liu, J. Gao et al., “Deberta: Decoding-enhanced bert with disentangled attention,” arXiv preprint arXiv:2006.03654 , 2020

arXiv 2006
[5]

A survey on large language mo del (llm) security and privacy: The good, the bad, and the ugly,

Y . Y ao, J. Duan, K. Xu et al., “A survey on large language mo del (llm) security and privacy: The good, the bad, and the ugly,” High-Conﬁdence Computing, p. 100211, 2024

work page 2024
[6]

SQBA: sequential query-based blackbox attack,

Y . Tao, “SQBA: sequential query-based blackbox attack, ” in Fifth In- ternational Conference on Artiﬁcial Intelligence and Comp uter Science (AICS 2023) , vol. 12803, pp. 721–729, SPIE, 2023

work page 2023
[7]

The FacT: Taming late nt factor models for explainability with factorization trees ,

Y . Tao, Y . Jia, N. Wang, and H. Wang, “The FacT: Taming late nt factor models for explainability with factorization trees ,” in Proceedings of the 42nd International ACM SIGIR Conference on Research a nd Development in Information Retrieval (SIGIR’19) , pp. 295–304, ACM, 2019

work page 2019
[8]

A comprehensive study on tex t-attributed graphs: Benchmarking and rethinking,

H. Y an, C. Li, R. Long et al., “A comprehensive study on tex t-attributed graphs: Benchmarking and rethinking,” Advances in Neural Information Processing Systems, vol. 36, pp. 17238–17264, 2023

work page 2023

Show all 17 references

[9]

Pre-trained language models and their applications,

H. Wang, J. Li, H. Wu et al., “Pre-trained language models and their applications,” Engineering, vol. 25, pp. 51–65, 2023

2023
[10]

Deep sentence embed ding using long short-term memory networks: Analysis and application to informa- tion retrieval,

H. Palangi, L. Deng, Y . Shen et al., “Deep sentence embed ding using long short-term memory networks: Analysis and application to informa- tion retrieval,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 24, no. 4, pp. 694–707, 2016

2016
[11]

Role of chat GPT in public health,

S. S. Biswas, “Role of chat GPT in public health,” Annals of Biomedical Engineering, vol. 51, no. 5, pp. 868–869, 2023

2023
[12]

Grap h attention net- works,

P . V eliˇ ckovi´ c, G. Cucurull, A. Casanova et al., “Grap h attention net- works,” arXiv preprint arXiv:1710.10903 , 2017

2017 arXiv
[13]

Code llama: Open foundation models for code,

B. Roziere, J. Gehring, F. Gloeckle et al., “Code llama: Open foundation models for code,” arXiv preprint arXiv:2308.12950 , 2023

2023 arXiv
[14]

TA-SBERT: token attention sentence- BERT for improving sentence representation,

J. Seo, S. Lee, L. Liu et al., “TA-SBERT: token attention sentence- BERT for improving sentence representation,” IEEE Access , vol. 10, pp. 39119–39128, 2022

2022
[15]

Integrating BERT opic and large language models for thematic identiﬁcation of Indonesian l egal docu- ments,

M. Ananta, R. Utari, A. Akhyar et al., “Integrating BERT opic and large language models for thematic identiﬁcation of Indonesian l egal docu- ments,” in 2024 11th International Conference on Advanced Informatic s: Concept, Theory and Application (ICAICTA) , pp. 1–6, IEEE, 2024

2024
[16]

Hierarchical multi-label text classiﬁcati on in Earth science datasets,

R. Dahal, “Hierarchical multi-label text classiﬁcati on in Earth science datasets,” 2024

2024
[17]

Exploring the potential of large language models (LLMs) in learning on graph,

Z. Chen, H. Mao, H. Li et al., “Exploring the potential of large language models (LLMs) in learning on graph,” in NeurIPS 2023 W orkshop: New Frontiers in Graph Learning , 2023

2023

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [1]

A comprehensive survey on gr aph neural networks,

Z. Wu, S. Pan, F. Chen et al., “A comprehensive survey on gr aph neural networks,” IEEE Transactions on Neural Networks and Learning Systems, vol. 32, no. 1, pp. 4–24, 2020

work page 2020

[2] [2]

Bridging context gaps: Lev eraging coreference resolution for long contextual understanding ,

Y . Liu, X. Peng, J. Cao et al., “Bridging context gaps: Lev eraging coreference resolution for long contextual understanding ,” arXiv preprint arXiv:2410.01671, 2024

arXiv 2024

[3] [3]

BERT: a review of applications in natura l language processing and understanding,

M. V . Koroteev, “BERT: a review of applications in natura l language processing and understanding,” arXiv preprint arXiv:2103.11943 , 2021

arXiv 2021

[4] [4]

Deberta: Decoding-enhanced bert with disentangled attention,

P . He, X. Liu, J. Gao et al., “Deberta: Decoding-enhanced bert with disentangled attention,” arXiv preprint arXiv:2006.03654 , 2020

arXiv 2006

[5] [5]

A survey on large language mo del (llm) security and privacy: The good, the bad, and the ugly,

Y . Y ao, J. Duan, K. Xu et al., “A survey on large language mo del (llm) security and privacy: The good, the bad, and the ugly,” High-Conﬁdence Computing, p. 100211, 2024

work page 2024

[6] [6]

SQBA: sequential query-based blackbox attack,

Y . Tao, “SQBA: sequential query-based blackbox attack, ” in Fifth In- ternational Conference on Artiﬁcial Intelligence and Comp uter Science (AICS 2023) , vol. 12803, pp. 721–729, SPIE, 2023

work page 2023

[7] [7]

The FacT: Taming late nt factor models for explainability with factorization trees ,

Y . Tao, Y . Jia, N. Wang, and H. Wang, “The FacT: Taming late nt factor models for explainability with factorization trees ,” in Proceedings of the 42nd International ACM SIGIR Conference on Research a nd Development in Information Retrieval (SIGIR’19) , pp. 295–304, ACM, 2019

work page 2019

[8] [8]

A comprehensive study on tex t-attributed graphs: Benchmarking and rethinking,

H. Y an, C. Li, R. Long et al., “A comprehensive study on tex t-attributed graphs: Benchmarking and rethinking,” Advances in Neural Information Processing Systems, vol. 36, pp. 17238–17264, 2023

work page 2023