Automated Decision-Making on Networks with LLMs through Knowledge-Guided Evolution

Lanning Wei; Quanming Yao; Xiaohan Zheng; Yong Li

REVIEW 3 major objections 6 minor 14 references

Automated Decision-Making on Networks with LLMs through Knowledge-Guided Evolution

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Knowledge-guided LLM agents can design GNNs that outperform all baselines across twelve datasets.

desk verdict Useful demo of LLM-driven GNN design, but the headline result is undercut by what looks like test-set leakage through the knowledge bases. read the letter →

arxiv 2506.14529 v1 pith:KB53D5O3 submitted 2025-06-17 cs.LG

Xiaohan Zheng , Lanning Wei , Yong Li , Quanming Yao This is my paper · ORCID

classification cs.LG

keywords GraphNeuralNetworksAutomatedMachineLearningLargeLanguageModelsRetrieval-AugmentedGenerationKnowledgeBaseNodeClassificationItemRanking

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LLMNet is a system that automates the design of graph neural network architectures using a team of LLM-based agents. The paper's central claim is that LLMNet consistently outperforms all baselines—hand-designed GNNs, AutoML methods, and LLM-only generation—on twelve datasets spanning node classification, graph classification, and item ranking. This matters because configuring a GNN currently requires expert graph knowledge and manual tuning; if the claim is right, LLMNet offers a path to automated, knowledge-guided model design for networked data.

What carries the argument

The load-bearing mechanism is the two-tier knowledge base managed by a Knowledge Agent: a prior knowledge base of task-specific information from sources such as OGB leaderboards and PyG documentation, and an experiment knowledge base that archives each evaluated model's setup and result. Retrieval is goal-aware: agent queries and stored knowledge are embedded with all-MiniLM-L6-v2, ranked by cosine similarity, and post-ranked by resource type before being placed in prompts. A planning agent then decides whether the results require another design-and-evaluate loop, which is the knowledge-guided evolution that the paper argues drives LLMNet's advantage.

What would settle it

Re-run the twelve-dataset benchmark against a strong, fully specified LLM-GNN baseline and per-task AutoML algorithms under equal search budgets; if the average-rank gap over LLMNet disappears, the claim of consistent superiority would be falsified.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that explicit graph-learning knowledge, retrieved and injected into an LLM agent pipeline, produces better GNN designs than pure LLM generation or conventional AutoML within a fixed search space. LLMNet constructs two knowledge bases—prior knowledge extracted from benchmark leaderboards, library documentation, and conference papers, plus experiment knowledge accumulated from its own evaluations—and uses retrieval-augmented generation to feed them into planning, data, configuration, and evaluation agents. In Table 1, LLMNet takes the best average rank across twelve datasets, and the full system beats its knowledge-free variant LLMNet (GL), which the paper reads as evidence that the knowledge bases, not the LLM alone, carry the gain.

Load-bearing premise

The load-bearing premise is that the comparison methods are chosen and run fairly; if the AutoML row hides weak configurations or the LLM-GNN baseline is unrepresentative, the reported average-rank advantage would not establish consistent superiority.

Editorial extensions

If this is right

Automated GNN design would no longer require a hand-crafted search space; the LLM agents expand the space using retrieved task knowledge.
The gap between LLMNet and LLMNet (GL) implies that external knowledge bases, not model capacity alone, produce the performance gain.
Because the experiment knowledge base stores each run, the system should improve on later tasks by reusing its own prior outcomes.
Winning over F2GNN, LRGNN, and Prof-CF would mean knowledge-guided LLM search can match or beat dedicated AutoML search algorithms on graph tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper leaves untested which knowledge source drives the gain; ablating the prior and experiment knowledge bases separately would isolate the contribution of benchmark-derived versus self-accumulated knowledge.
A fair stress test would pit LLMNet against a fully specified, cited LLM-GNN baseline under equal inference budgets, since the paper's LLM-GNN baseline is neither described nor cited.
If the average-rank advantage transfers to unseen datasets, graph AutoML could shift from search-space topology design to knowledge-base curation and retrieval quality.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Xiaohan — quick take on arXiv:2506.14529. The core idea is reasonable: use LLM agents with RAG over curated knowledge bases to automate GNN architecture search, and maintain an experiment memory that improves future designs. That combination is not in the cited literature, and the demo is clearly written. The multi-agent decomposition (planning, data, configuration, evaluation) is a sensible engineering pattern, and reporting means and standard deviations is better than nothing. The ablation against LLMNet (GL) is also the right thing to include.

The problem is the evidence for "consistently outperforms all baselines." Section 2.1 says the prior knowledge base is built from OGB leaderboards and benchmark evaluation results, including model setups and their performance on specific datasets, and the experiment knowledge base stores the results of every evaluated model. Table 1 is exactly those standard benchmarks. So when LLMNet runs on ogbn-arxiv or Cora, the design loop can retrieve published test scores and its own previous test results for the same test sets. That is transductive model selection with test-set memory, not knowledge-guided generalization. The baselines don't get that. Unless the authors can show the knowledge bases are reset or never contain test metrics for the target dataset, the 1 vs 2.63 average rank does not support the claim. This is a load-bearing flaw, not a style complaint.

Other issues are smaller but real: the AutoML row aggregates F2GNN, LRGNN, and Prof-CF without per-dataset attribution; LLM-GNN is never described or cited; there are no significance tests; and the appendix with ablations and costs is promised but missing. The code link is a plus, but there is no reproduction metadata, and a code dump doesn't fix the leakage question.

Who should read this: people building LLM-based AutoML systems and anyone designing evaluation protocols for agents that retrieve external knowledge. It is a useful cautionary example. I would not cite its empirical claims as they stand, but I'd send it to a serious referee, because the system idea is timely and the methodological issue is exactly what peer review should catch. The authors should be asked to clarify the knowledge base contents, show a version where no test performance is retrievable, and report the baselines transparently.

Referee Report

3 major / 6 minor

Summary. This paper presents LLMNet, a system that automates GNN architecture design via LLM-based agents and retrieval-augmented generation. Two knowledge bases are constructed: a 'prior knowledge base' derived from sources such as OGB leaderboards, PyG documentation, and arXiv papers, and an 'experiment knowledge base' that stores outcomes of the system's own previous evaluations. A pipeline of agents (Planning, Data, Configuration, Evaluation) designs and iteratively refines GNN architectures, with retrieval from these knowledge bases guiding each step. The paper claims that LLMNet consistently outperforms GCN, GraphSAGE, AutoML baselines, LLM-generated GNNs, and its own ablation without external knowledge (LLMNet (GL)) across twelve datasets for node classification, graph classification, and item ranking.

Significance. If validated, LLMNet would be a practical demonstration of using LLM agents and knowledge bases for automated GNN design, and the code release supports reproducibility. The underlying idea of knowledge-guided evolution with a self-updating knowledge base is interesting and potentially of use to the AutoML-for-graphs community. However, the empirical support for the central claim is currently undermined by potential information leakage from the knowledge bases and by the lack of a controlled evaluation protocol. The reader cannot currently verify the claim of consistent superiority from Table 1 alone.

major comments (3)

[2.1] Section 2.1 (Knowledge Bases Construction and Utilization): The construction of the prior knowledge base explicitly includes 'OGB leaderboards' and 'benchmark evaluation results, including models setups and their performance on specific datasets', and the experiment knowledge base stores 'the experimental summary... after the evaluation of a GNN model' for 'future pipeline runs'. Since Table 1 evaluates on standard benchmark datasets (ogbn-arxiv, ogbg-molhiv, Cora, DD, Proteins, Amazon-Sports, etc.), the retrieval mechanism can, as described, surface test-set performance information from published leaderboards and from LLMNet's own earlier runs on the same test sets. The baselines do not have access to this information. The manuscript does not state that the memory is reset between repeated runs or seeds, nor does it specify any filter that excludes the target dataset's test performance from retrieval. Consequently, the reported average rank of 1 (vs. 2.63 for AutoML) may reflect test-set memorization rather than knowledge-guided generalization. Please clarify (a) whether leaderboard/test performances on the target dataset are excluded from the knowledge bases during design, (b) whether each seed or fold starts with an empty experiment knowledge base or with one built only from training data of that fold, and (c) whether any retrieved knowledge item can contain the test metric for the exact dataset being evaluated.
[3.1] Section 3.1 (Baselines) and Table 1: The 'AutoML' row aggregates three different algorithms (F2GNN, LRGNN, Prof-CF) across the three tasks without per-dataset attribution, so the reader cannot tell which baseline produced the reported value on each dataset or whether each baseline was tuned with a comparable budget. The 'LLM-GNN' baseline is not described or cited in Section 3.1; the phrase 'GNNs generated by LLMs' is the only specification. This makes it impossible to assess whether LLMNet's advantage reflects a fair comparison with similarly powered LLM-based generators. Please provide per-dataset baseline identity, the protocol for LLM-GNN (including the LLM model, number of generation attempts, and whether it has access to the same graph-analysis tools), and the computational budget allocated to each baseline.
[3.2/Table 1] Table 1 and Section 3.2 (Performance Comparisons): The central claim 'LLMNet consistently outperforms all baselines' is supported only by point estimates of mean and standard deviation; no paired significance tests are reported. Several differences are smaller than or comparable to the reported standard deviations (e.g., Cora: 87.10±0.36 vs. AutoML 86.57±0.32; ogbg-molhiv: 75.44±0.93 vs. AutoML 74.58±2.61; Amazon-Sports: 0.9298±0.0071 vs. AutoML 0.9327±0.0006). Please report pairwise significance tests (e.g., Wilcoxon signed-rank test across datasets, or per-dataset corrected t-tests) to justify the 'consistently' claim.

minor comments (6)

[Abstract] The abstract contains a typo: 'knowlege bases' should read 'knowledge bases'.
[Section 1] Section 1 contains a stray period before 'and learning subgraph patterns'; the sentence should be rephrased.
[Section 3] The manuscript repeatedly states that experiments cover 'twelve datasets' (abstract and Section 3), but Table 1 lists only nine; the other three are deferred to an extended version that is not part of the submitted text, which makes the consistency of the claim unverifiable from the paper alone.
[Section 2] Figure 1 is referenced throughout Section 2 but is not included in the arXiv submission; please include the figure or ensure it appears in the demo track version.
[Section 3.1] The paper does not specify the LLM model and decoding settings used by the agents; for a demo claiming reproducibility via the provided code, state the default LLM and key generation parameters.
[Table 1] In Table 1, the column header 'Amazon-Sports(↓)' indicates a lower-is-better metric, but the caption does not label task types for each block of columns; please add task groupings (node classification, graph classification, link/ranking) to make the table self-contained.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the empirical benchmark claim is self-contained against external fixed datasets; the knowledge-base leakage concern is an evaluation fairness risk, not a circular derivation.

full rationale

The paper makes no formal derivation and does not fit a parameter that is then renamed as a prediction. Its central claim, that LLMNet consistently outperforms baselines, is an empirical comparison on external benchmark datasets. The knowledge bases ingest published benchmark results and LLMNet's own past evaluations, which may create an information asymmetry, but that is a potential data-leakage / evaluation-protocol threat, not a case where the output is equivalent to the input by construction. The self-citations (F2GNN, LRGNN, Prof-CF, SANE) are used as published AutoML baselines, not as load-bearing evidence for LLMNet's correctness, so they do not make the argument circular. The missing citation or specification of the 'LLM-GNN' baseline and the aggregation of three AutoML methods into one baseline row are comparability and completeness weaknesses, but they are not circularity. No step in the paper reduces, by equation or by self-citation, to its own inputs, so the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim is an empirical system demonstration, so the free parameters are the unstated design choices that control the system's behavior. The axioms are the background beliefs about LLM capabilities, knowledge source quality, and retrieval effectiveness on which the system's success depends. No new theoretical entities are introduced.

free parameters (3)

top-k retrieval count per resource type
The number of knowledge items retrieved per resource type and re-ranked for each prompt is a hand-chosen system parameter that directly affects the LLM's context and the designed GNN. The value is not reported or ablated in the main text.
number of design and refinement iterations
The planning agent decides when to stop revising the GNN, but the stopping criterion, iteration budget, and compute limits are not specified, even though they determine the final architecture and reported performance.
LLM model and decoding settings
The underlying LLM, temperature, prompt templates, and token budgets are never stated. The empirical results depend on these choices, and the paper does not disclose them.

assumptions (4)

domain assumption LLMs can generate competitive GNN architectures when prompted with retrieved knowledge.
The method relies on the premise that an LLM, given RAG context, can propose architecture choices that rival human-designed or searched ones; this is tested only indirectly through the final benchmark numbers.
domain assumption Knowledge sources (OGB leaderboards, PyG documentation, arXiv papers) are accurate and relevant.
The knowledge agent extracts and later retrieves this content, so any noise, redundancy, or outdated information in the sources could mislead the design process. The paper states it reduces noise but does not measure residual contamination.
domain assumption The all-MiniLM-L6-v2 embedding model yields adequate retrieval alignment.
Cosine similarity in this embedding space is the sole retrieval mechanism; the paper provides no comparison with other embedding or retrieval methods, so this choice is an unverified premise.
domain assumption Graph visualization and automatic feature engineering by the Data Agent help performance.
These steps are part of the pipeline, but the main text does not ablate them, so their contribution to the reported gains is not established.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Automated Decision-Making on Networks with LLMs through Knowledge-Guided Evolution." pith.science (2026). https://pith.science/paper/KB53D5O3

@misc{pith2026250614529,
  author       = {Pith},
  title        = {Pith review of: Automated Decision-Making on Networks with LLMs through Knowledge-Guided Evolution},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KB53D5O3}},
  note         = {Machine review of arXiv:2506.14529}
}

read the original abstract

Effective decision-making on networks often relies on learning from graph-structured data, where Graph Neural Networks (GNNs) play a central role, but they take efforts to configure and tune. In this demo, we propose LLMNet, showing how to design GNN automated through Large Language Models. Our system develops a set of agents that construct graph-related knowlege bases and then leverages Retrieval-Augmented Generation (RAG) to support automated configuration and refinement of GNN models through a knowledge-guided evolution process. These agents, equipped with specialized knowledge bases, extract insights into tasks and graph structures by interacting with the knowledge bases. Empirical results show LLMNet excels in twelve datasets across three graph learning tasks, validating its effectiveness of GNN model designing.

Figures

Figures reproduced from arXiv: 2506.14529 by the authors.

**Figure 1.** System architecture of LLMNet. LLMNet is designed to automate the design of GNN model architectures through a knowledge [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

**Figure 2.** The detailed steps and output of LLMNet. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 14 canonical work pages

[1]

Neural message passing for quantum chemistry

[Gilmer et al., 2017] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In ICML, pages 1263–1272,

work page 2017
[3]

Lightgcn: Simplifying and powering graph convolution network for recommendation

[He et al., 2020] Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. Lightgcn: Simplifying and powering graph convolution network for recommendation. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pages 639–648,

work page 2020
[8]

Designing the topology of graph neural networks: A novel feature fusion perspective

[Wei et al., 2022] Lanning Wei, Huan Zhao, and Zhiqiang He. Designing the topology of graph neural networks: A novel feature fusion perspective. In The WebConf, pages 1381–1391,

work page 2022
[9]

Search to capture long-range depen- dency with stacking gnns for graph classification

[Wei et al., 2023] Lanning Wei, Zhiqiang He, Huan Zhao, and Quanming Yao. Search to capture long-range depen- dency with stacking gnns for graph classification. In Pro- ceedings of the ACM Web Conference 2023 , pages 588– 598,

work page 2023
[11]

Hierarchical graph representation learning with differen- tiable pooling

[Ying et al., 2018] Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. Hierarchical graph representation learning with differen- tiable pooling. In NeurIPS, pages 4800–4810,

work page 2018
[13]

An end-to-end deep learning architecture for graph classification

[Zhang et al., 2018] Muhan Zhang, Zhicheng Cui, Marion Neumann, and Yixin Chen. An end-to-end deep learning architecture for graph classification. In AAAI,

work page 2018
[14]

Search to aggregate neighborhood for graph neural network

[Zhao et al., 2021] Huan Zhao, Quanming Yao, and Weiwei Tu. Search to aggregate neighborhood for graph neural network. In ICDE, 2021

work page 2021
[1995]

Semi-supervised classification with graph convolutional networks

[Kipf and Welling, 2016] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. ICLR,

work page 2016

Show all 14 references

[2016]

Revisiting heterophily for graph neural networks

[Luan et al., 2022] Sitao Luan, Chenqing Hua, Qincheng Lu, Jiaqi Zhu, Mingde Zhao, Shuyuan Zhang, Xiao-Wen Chang, and Doina Precup. Revisiting heterophily for graph neural networks. In NeurIPS,

2022
[2017]

Inductive representation learning on large graphs

[Hamilton et al., 2017] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NeurIPS, pages 1024–1034,

2017
[2018]

Link prediction based on graph neural networks.Advances in neural information processing systems , 31,

[Zhang and Chen, 2018] Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks.Advances in neural information processing systems , 31,

2018
[2020]

Eliciting knowledge from experts: A methodological analysis

[Hoffman et al., 1995] Robert R Hoffman, Nigel R Shadbolt, A Mike Burton, and Gary Klein. Eliciting knowledge from experts: A methodological analysis. Organizational behavior and human decision processes , 62(2):129–158,

1995
[2022]

Automated graph machine learning: Approaches, li- braries and directions

[Wang et al., 2022] Xin Wang, Ziwei Zhang, and Wenwu Zhu. Automated graph machine learning: Approaches, li- braries and directions. arXiv preprint arXiv:2201.01288 ,

2022 arXiv
[2023]

A comprehensive survey on graph neural networks

[Wu et al., 2020] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and Learning Systems (TNNLS),

2020

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Neural message passing for quantum chemistry

[Gilmer et al., 2017] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In ICML, pages 1263–1272,

work page 2017

[2] [3]

Lightgcn: Simplifying and powering graph convolution network for recommendation

[He et al., 2020] Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. Lightgcn: Simplifying and powering graph convolution network for recommendation. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pages 639–648,

work page 2020

[3] [8]

Designing the topology of graph neural networks: A novel feature fusion perspective

[Wei et al., 2022] Lanning Wei, Huan Zhao, and Zhiqiang He. Designing the topology of graph neural networks: A novel feature fusion perspective. In The WebConf, pages 1381–1391,

work page 2022

[4] [9]

Search to capture long-range depen- dency with stacking gnns for graph classification

[Wei et al., 2023] Lanning Wei, Zhiqiang He, Huan Zhao, and Quanming Yao. Search to capture long-range depen- dency with stacking gnns for graph classification. In Pro- ceedings of the ACM Web Conference 2023 , pages 588– 598,

work page 2023

[5] [11]

Hierarchical graph representation learning with differen- tiable pooling

[Ying et al., 2018] Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. Hierarchical graph representation learning with differen- tiable pooling. In NeurIPS, pages 4800–4810,

work page 2018

[6] [13]

An end-to-end deep learning architecture for graph classification

[Zhang et al., 2018] Muhan Zhang, Zhicheng Cui, Marion Neumann, and Yixin Chen. An end-to-end deep learning architecture for graph classification. In AAAI,

work page 2018

[7] [14]

Search to aggregate neighborhood for graph neural network

[Zhao et al., 2021] Huan Zhao, Quanming Yao, and Weiwei Tu. Search to aggregate neighborhood for graph neural network. In ICDE, 2021

work page 2021

[8] [1995]

Semi-supervised classification with graph convolutional networks

[Kipf and Welling, 2016] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. ICLR,

work page 2016