{"id":"5a2f9df5-716e-4023-917e-4ed35b2d9095","arxiv_id":"2506.14529","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LLMNet automates GNN architecture design with LLM agents that retrieve knowledge from curated knowledge bases, and it reports top average rank on twelve graph learning benchmarks.","lead":"This paper introduces LLMNet, a system that uses large language model agents and retrieval-augmented generation to automatically design graph neural network architectures. The authors report that LLMNet ranks first on average across twelve graph learning datasets, beating standard GNNs and AutoML baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central superiority claim may rest on target leakage: LLMNet's knowledge bases ingest published benchmark results and its own prior evaluations on the same test datasets, so the design loop can retrieve test-performance information that baselines cannot access.","rationale":"The reader's weakest assumption concerns fairness of the evaluation protocol, specifically the aggregated AutoML row and the undescribed LLM-GNN baseline. I agree that protocol fairness is the key risk, but the more load-bearing and testable mechanism is target leakage through the knowledge bases. Section 2.1 describes retrieving published benchmark performance and storing LLMNet's own evaluation outcomes, with no stated reset between seeds or exclusion of the evaluation datasets. For a benchmark with public leaderboards, this is not a hypothetical: an LLM agent that retrieves the test accuracy of many architectures on ogbn-arxiv or ogbg-molhiv has access to signal that the comparison methods lack. The strongest counter-evidence is LLMNet(GL), which reaches average rank 2.5 without external knowledge; that suggests the agent pipeline itself has some value, but it does not rescue the headline claim that knowledge integration yields consistent superiority, because the external-knowledge version is the one exposed to the suspected leakage. The proposed scrub test is decisive: it removes exactly the information channel that makes the comparison unfair while keeping the rest of the system intact. If the advantage survives the scrub, the central claim stands; if not, the paper needs to either defend the use of target benchmark results as legitimate prior knowledge or weaken the claim. Since the verdict was already conditional on evaluation-protocol evidence, my read does not change the requested verdict category, but it sharpens the condition that must be met.","tokens_in":4911,"tokens_out":5911,"duration_ms":62524,"concrete_test":"Run a scrubbed ablation: build LLMNet's knowledge bases only from resources that predate the twelve evaluation datasets or that explicitly exclude those datasets and their reported test scores, and clear the experiment knowledge base after every single run so no within-dataset memory accumulates; then rerun the full Table 1 protocol with identical compute budgets. If scrubbed LLMNet no longer achieves average rank 1 or loses to AutoML/LLMNet(GL), the headline superiority claim is an artifact of target leakage; if it still holds rank 1, the concern is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that Table 1 compares LLMNet with baselines under the same information constraints. Section 2.1 conflicts with this. The prior knowledge base is built from 'OGB leaderboards' and 'benchmark evaluation results, including models setups and their performance on specific datasets,' and the experiment knowledge base stores 'the experimental summary... after the evaluation of a GNN model' for future pipeline runs. On standard benchmarks in Table 1 (ogbn-arxiv, ogbg-molhiv, Cora, DD, Proteins, Amazon-Sports), published leaderboard/test scores and LLMNet's own earlier runs on the same test sets are therefore retrievable during design. That turns automated architecture search into transductive model selection with test-set memory, which GCN, SAGE, AutoML, and LLM-GNN do not have. The paper does not state that memory is reset between the repeated seeds used to compute the reported means and standard deviations. The average rank of 1 vs. 2.63 may thus reflect retrieval of known-good configurations rather than knowledge-guided generalization. This is a correctness risk independent of the baseline-aggregation issue already noted by the reader; it directly undermines the claim that LLMNet 'consistently outperforms all baselines.'","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents LLMNet, a system that automates GNN architecture design via LLM-based agents and retrieval-augmented generation. Two knowledge bases are constructed: a 'prior knowledge base' derived from sources such as OGB leaderboards, PyG documentation, and arXiv papers, and an 'experiment knowledge base' that stores outcomes of the system's own previous evaluations. A pipeline of agents (Planning, Data, Configuration, Evaluation) designs and iteratively refines GNN architectures, with retrieval from these knowledge bases guiding each step. The paper claims that LLMNet consistently outperforms GCN, GraphSAGE, AutoML baselines, LLM-generated GNNs, and its own ablation without external knowledge (LLMNet (GL)) across twelve datasets for node classification, graph classification, and item ranking.","tokens_in":5160,"tokens_out":5803,"duration_ms":53055,"significance":"If validated, LLMNet would be a practical demonstration of using LLM agents and knowledge bases for automated GNN design, and the code release supports reproducibility. The underlying idea of knowledge-guided evolution with a self-updating knowledge base is interesting and potentially of use to the AutoML-for-graphs community. However, the empirical support for the central claim is currently undermined by potential information leakage from the knowledge bases and by the lack of a controlled evaluation protocol. The reader cannot currently verify the claim of consistent superiority from Table 1 alone.","major_comments":[{"comment":"Section 2.1 (Knowledge Bases Construction and Utilization): The construction of the prior knowledge base explicitly includes 'OGB leaderboards' and 'benchmark evaluation results, including models setups and their performance on specific datasets', and the experiment knowledge base stores 'the experimental summary... after the evaluation of a GNN model' for 'future pipeline runs'. Since Table 1 evaluates on standard benchmark datasets (ogbn-arxiv, ogbg-molhiv, Cora, DD, Proteins, Amazon-Sports, etc.), the retrieval mechanism can, as described, surface test-set performance information from published leaderboards and from LLMNet's own earlier runs on the same test sets. The baselines do not have access to this information. The manuscript does not state that the memory is reset between repeated runs or seeds, nor does it specify any filter that excludes the target dataset's test performance from retrieval. Consequently, the reported average rank of 1 (vs. 2.63 for AutoML) may reflect test-set memorization rather than knowledge-guided generalization. Please clarify (a) whether leaderboard/test performances on the target dataset are excluded from the knowledge bases during design, (b) whether each seed or fold starts with an empty experiment knowledge base or with one built only from training data of that fold, and (c) whether any retrieved knowledge item can contain the test metric for the exact dataset being evaluated.","section":"2.1"},{"comment":"Section 3.1 (Baselines) and Table 1: The 'AutoML' row aggregates three different algorithms (F2GNN, LRGNN, Prof-CF) across the three tasks without per-dataset attribution, so the reader cannot tell which baseline produced the reported value on each dataset or whether each baseline was tuned with a comparable budget. The 'LLM-GNN' baseline is not described or cited in Section 3.1; the phrase 'GNNs generated by LLMs' is the only specification. This makes it impossible to assess whether LLMNet's advantage reflects a fair comparison with similarly powered LLM-based generators. Please provide per-dataset baseline identity, the protocol for LLM-GNN (including the LLM model, number of generation attempts, and whether it has access to the same graph-analysis tools), and the computational budget allocated to each baseline.","section":"3.1"},{"comment":"Table 1 and Section 3.2 (Performance Comparisons): The central claim 'LLMNet consistently outperforms all baselines' is supported only by point estimates of mean and standard deviation; no paired significance tests are reported. Several differences are smaller than or comparable to the reported standard deviations (e.g., Cora: 87.10±0.36 vs. AutoML 86.57±0.32; ogbg-molhiv: 75.44±0.93 vs. AutoML 74.58±2.61; Amazon-Sports: 0.9298±0.0071 vs. AutoML 0.9327±0.0006). Please report pairwise significance tests (e.g., Wilcoxon signed-rank test across datasets, or per-dataset corrected t-tests) to justify the 'consistently' claim.","section":"3.2/Table 1"}],"minor_comments":[{"comment":"The abstract contains a typo: 'knowlege bases' should read 'knowledge bases'.","section":"Abstract"},{"comment":"Section 1 contains a stray period before 'and learning subgraph patterns'; the sentence should be rephrased.","section":"Section 1"},{"comment":"The manuscript repeatedly states that experiments cover 'twelve datasets' (abstract and Section 3), but Table 1 lists only nine; the other three are deferred to an extended version that is not part of the submitted text, which makes the consistency of the claim unverifiable from the paper alone.","section":"Section 3"},{"comment":"Figure 1 is referenced throughout Section 2 but is not included in the arXiv submission; please include the figure or ensure it appears in the demo track version.","section":"Section 2"},{"comment":"The paper does not specify the LLM model and decoding settings used by the agents; for a demo claiming reproducibility via the provided code, state the default LLM and key generation parameters.","section":"Section 3.1"},{"comment":"In Table 1, the column header 'Amazon-Sports(↓)' indicates a lower-is-better metric, but the caption does not label task types for each block of columns; please add task groupings (node classification, graph classification, link/ranking) to make the table self-contained.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is positioned as a demo, and its contribution is a system description plus an empirical claim. The stress-test concern about information leakage in the knowledge bases is the most serious issue; it strikes at the validity of the headline result. If the authors can demonstrate a clean protocol with memory reset and no test-set-derived knowledge in retrieval, the paper may be acceptable as a demo. The missing baseline details and significance tests are fixable. I recommend major revision rather than rejection because the architecture idea remains interesting and the code is promised, but the current evaluation as written does not support the central claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Xiaohan — quick take on arXiv:2506.14529. The core idea is reasonable: use LLM agents with RAG over curated knowledge bases to automate GNN architecture search, and maintain an experiment memory that improves future designs. That combination is not in the cited literature, and the demo is clearly written. The multi-agent decomposition (planning, data, configuration, evaluation) is a sensible engineering pattern, and reporting means and standard deviations is better than nothing. The ablation against LLMNet (GL) is also the right thing to include.\n\nThe problem is the evidence for \"consistently outperforms all baselines.\" Section 2.1 says the prior knowledge base is built from OGB leaderboards and benchmark evaluation results, including model setups and their performance on specific datasets, and the experiment knowledge base stores the results of every evaluated model. Table 1 is exactly those standard benchmarks. So when LLMNet runs on ogbn-arxiv or Cora, the design loop can retrieve published test scores and its own previous test results for the same test sets. That is transductive model selection with test-set memory, not knowledge-guided generalization. The baselines don't get that. Unless the authors can show the knowledge bases are reset or never contain test metrics for the target dataset, the 1 vs 2.63 average rank does not support the claim. This is a load-bearing flaw, not a style complaint.\n\nOther issues are smaller but real: the AutoML row aggregates F2GNN, LRGNN, and Prof-CF without per-dataset attribution; LLM-GNN is never described or cited; there are no significance tests; and the appendix with ablations and costs is promised but missing. The code link is a plus, but there is no reproduction metadata, and a code dump doesn't fix the leakage question.\n\nWho should read this: people building LLM-based AutoML systems and anyone designing evaluation protocols for agents that retrieve external knowledge. It is a useful cautionary example. I would not cite its empirical claims as they stand, but I'd send it to a serious referee, because the system idea is timely and the methodological issue is exactly what peer review should catch. The authors should be asked to clarify the knowledge base contents, show a version where no test performance is retrievable, and report the baselines transparently.","headline":"Useful demo of LLM-driven GNN design, but the headline result is undercut by what looks like test-set leakage through the knowledge bases.","tokens_in":5711,"tokens_out":3435,"would_cite":false,"duration_ms":34517,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Knowledge-guided LLM agents can design GNNs that outperform all baselines across twelve datasets.","keywords":["Graph Neural Networks","Automated Machine Learning","Large Language Models","Retrieval-Augmented Generation","Knowledge Base","Node Classification","Graph Classification","Item Ranking"],"falsifier":"Re-run the twelve-dataset benchmark against a strong, fully specified LLM-GNN baseline and per-task AutoML algorithms under equal search budgets; if the average-rank gap over LLMNet disappears, the claim of consistent superiority would be falsified.","tokens_in":4714,"feed_emoji":"🤖","tokens_out":6644,"duration_ms":58545,"temperature":0.7,"pith_summary":"LLMNet is a system that automates the design of graph neural network architectures using a team of LLM-based agents. The paper's central claim is that LLMNet consistently outperforms all baselines—hand-designed GNNs, AutoML methods, and LLM-only generation—on twelve datasets spanning node classification, graph classification, and item ranking. This matters because configuring a GNN currently requires expert graph knowledge and manual tuning; if the claim is right, LLMNet offers a path to automated, knowledge-guided model design for networked data.","feed_headline":"LLMNet's knowledge-guided agents top baselines on 12 graph datasets","feed_subtitle":"Retrieval-augmented LLM agents design GNN architectures automatically, winning rank one across node, graph, and ranking tasks.","key_machinery":"The load-bearing mechanism is the two-tier knowledge base managed by a Knowledge Agent: a prior knowledge base of task-specific information from sources such as OGB leaderboards and PyG documentation, and an experiment knowledge base that archives each evaluated model's setup and result. Retrieval is goal-aware: agent queries and stored knowledge are embedded with all-MiniLM-L6-v2, ranked by cosine similarity, and post-ranked by resource type before being placed in prompts. A planning agent then decides whether the results require another design-and-evaluate loop, which is the knowledge-guided evolution that the paper argues drives LLMNet's advantage.","core_discovery":"On the paper's own terms, the discovery is that explicit graph-learning knowledge, retrieved and injected into an LLM agent pipeline, produces better GNN designs than pure LLM generation or conventional AutoML within a fixed search space. LLMNet constructs two knowledge bases—prior knowledge extracted from benchmark leaderboards, library documentation, and conference papers, plus experiment knowledge accumulated from its own evaluations—and uses retrieval-augmented generation to feed them into planning, data, configuration, and evaluation agents. In Table 1, LLMNet takes the best average rank across twelve datasets, and the full system beats its knowledge-free variant LLMNet (GL), which the paper reads as evidence that the knowledge bases, not the LLM alone, carry the gain.","pith_inferences":["The paper leaves untested which knowledge source drives the gain; ablating the prior and experiment knowledge bases separately would isolate the contribution of benchmark-derived versus self-accumulated knowledge.","A fair stress test would pit LLMNet against a fully specified, cited LLM-GNN baseline under equal inference budgets, since the paper's LLM-GNN baseline is neither described nor cited.","If the average-rank advantage transfers to unseen datasets, graph AutoML could shift from search-space topology design to knowledge-base curation and retrieval quality."],"forward_implications":["Automated GNN design would no longer require a hand-crafted search space; the LLM agents expand the space using retrieved task knowledge.","The gap between LLMNet and LLMNet (GL) implies that external knowledge bases, not model capacity alone, produce the performance gain.","Because the experiment knowledge base stores each run, the system should improve on later tasks by reusing its own prior outcomes.","Winning over F2GNN, LRGNN, and Prof-CF would mean knowledge-guided LLM search can match or beat dedicated AutoML search algorithms on graph tasks."],"supporting_citations":[{"why":"Supplies the GCN baseline whose Cora accuracy of 0.8568 LLMNet compares against in the demonstration.","marker":"[Kipf and Welling, 2016]"},{"why":"Supplies the GraphSAGE baseline and the inductive representation learning setting for node-level tasks.","marker":"[Hamilton et al., 2017]"},{"why":"Supplies the F2GNN AutoML baseline used for one of the three tasks.","marker":"[Wei et al., 2022]"},{"why":"Supplies the LRGNN AutoML baseline used for graph classification.","marker":"[Wei et al., 2023]"},{"why":"Supplies the Prof-CF AutoML baseline used for the item ranking task.","marker":"[Wang et al., 2022]"},{"why":"Supplies the SANE AutoML baseline whose Cora accuracy of 0.8640 is cited in the demonstration.","marker":"[Zhao et al., 2021]"},{"why":"Supplies the ACM-GCN baseline whose Cora accuracy of 0.8667 is compared in the demonstration.","marker":"[Luan et al., 2022]"}],"fun_headline_variants":["RAG-powered agents evolve GNNs, top 12 graph benchmarks","Knowledge-guided LLM agents beat AutoML on 12 graph datasets","LLMNet's retrieval-augmented agents outrank AutoML on 12 datasets","Knowledge bases power LLM agents to design better GNNs","LLMNet uses knowledge-guided evolution to auto-design GNNs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the comparison methods are chosen and run fairly; if the AutoML row hides weak configurations or the LLM-GNN baseline is unrepresentative, the reported average-rank advantage would not establish consistent superiority.","fun_headline_variants_meta":{"raw":{"variants":["RAG-powered agents evolve GNNs, top 12 graph benchmarks","Knowledge-guided LLM agents beat AutoML on 12 graph datasets","LLMNet's retrieval-augmented agents outrank AutoML on 12 datasets","Knowledge bases power LLM agents to design better GNNs","LLMNet uses knowledge-guided evolution to auto-design GNNs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000524,"raw_usage":{"total_tokens":2472,"prompt_tokens":823,"completion_tokens":1649,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":439,"completion_tokens_details":{"reasoning_tokens":1554}},"tokens_in":439,"tokens_out":1649,"duration_ms":13251,"temperature":1.0,"reasoning_tokens":1554,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:16:53.716776+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the twelve-dataset benchmark against a strong, fully specified LLM-GNN baseline and per-task AutoML algorithms under equal search budgets; if the average-rank gap over LLMNet disappears, the claim of consistent superiority would be falsified.","supporting_citations":[{"cited_title":"Semi-supervised classification with graph convolutional networks","cited_arxiv_id":null,"evidence_quote":"Supplies the GCN baseline whose Cora accuracy of 0.8568 LLMNet compares against in the demonstration."},{"cited_title":"Inductive representation learning on large graphs","cited_arxiv_id":null,"evidence_quote":"Supplies the GraphSAGE baseline and the inductive representation learning setting for node-level tasks."},{"cited_title":"Designing the topology of graph neural networks: A novel feature fusion perspective","cited_arxiv_id":null,"evidence_quote":"Supplies the F2GNN AutoML baseline used for one of the three tasks."},{"cited_title":"Search to capture long-range depen- dency with stacking gnns for graph classification","cited_arxiv_id":null,"evidence_quote":"Supplies the LRGNN AutoML baseline used for graph classification."},{"cited_title":"Automated Graph Machine Learning: Approaches, Libraries, Benchmarks and Directions","cited_arxiv_id":"2201.01288","evidence_quote":"Supplies the Prof-CF AutoML baseline used for the item ranking task."},{"cited_title":"Search to aggregate neighborhood for graph neural network","cited_arxiv_id":null,"evidence_quote":"Supplies the SANE AutoML baseline whose Cora accuracy of 0.8640 is cited in the demonstration."},{"cited_title":"Revisiting heterophily for graph neural networks","cited_arxiv_id":null,"evidence_quote":"Supplies the ACM-GCN baseline whose Cora accuracy of 0.8667 is compared in the demonstration."}],"review_version":1}