{"id":"84a7c17a-9ba9-4081-bef9-60a67545b6bd","arxiv_id":"2412.17029","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"GraphAgent links an LLM agent pipeline with graph tokenization, automatically building semantic knowledge graphs and using them for both graph prediction and open-ended text generation.","lead":"GraphAgent is a multi-agent system that turns text and graph data into natural-language questions a language model can answer, using three cooperating agents to build knowledge graphs, plan tasks, and execute them with graph-aware tokens. The paper reports large gains over prior graph language models and over much larger LLMs on prediction and text generation benchmarks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Table 3 zero-shot results are unverified because Section 3.3 never states whether GraphAgent-Zero-Shot was trained on the Arxiv-Papers or ICLR-Peer Reviews splits that it is evaluated on.","rationale":"The reader identified the zero-shot training/evaluation protocol as the weakest assumption, and I agree this is the most load-bearing concern because it directly affects the paper's key generalization claim. However, the reader's specific citation is factually off: Section 3.1.2 and Table 1 give 3,141 training samples for ICLR-Peer Reviews, while 4,155 corresponds to the Related Work Generation dataset. The core concern still stands: the paper says ICLR-Peer Reviews is 'used for both training and testing purposes,' and Section 3.3 does not state what data GraphAgent-Zero-Shot was trained on. Without this specification, the Table 3 zero-shot results cannot be interpreted as measuring cross-dataset generalization. This is a fixable reporting gap, so the verdict should remain CONDITIONAL pending the specified check. I also note the generation evaluation relies on same-family perplexity and an LLM judge without human validation, which is a secondary weakness, but the zero-shot protocol is the more decisive issue because if the data overlaps, the predictive results are not evidence of generalization at all.","tokens_in":23928,"tokens_out":8384,"duration_ms":69524,"concrete_test":"Check the released training configuration of GraphAgent-Zero-Shot (or request it from the authors) and verify that its fine-tuning data has zero overlap with the Table 3 evaluation sets: intersect the document IDs in the zero-shot model's training split with the 500 Arxiv-Papers evaluation IDs and the 500 ICLR-Peer Reviews evaluation IDs. If the intersection is non-empty, the zero-shot claim is invalid; if empty, re-run the Table 3 evaluation with this verified-disjoint model to confirm the reported numbers are reproducible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is that GraphAgent-Zero-Shot in Table 3 is truly zero-shot with respect to the Arxiv-Papers and ICLR-Peer Reviews evaluation sets. Section 3.2 defines the zero-shot protocol for IMDB→ACM ('training on IMDB dataset... evaluating node classification performance on ACM dataset'), but Section 3.3 never specifies the training data for the three GraphAgent variants. Table 1 lists 5,175 training samples for Arxiv-Papers and 3,141 for ICLR-Peer Reviews, and Section 3.1.2 explicitly says of ICLR-Peer Reviews: 'This dataset is used for both training and testing purposes.' If GraphAgent-Zero-Shot was fine-tuned on either training split, or if the 500 evaluation samples overlap with the training set, then the Table 3 zero-shot results are not zero-shot at all; they would be in-distribution supervised results, making the comparison against non-fine-tuned LLMs misleading. The paper provides no sentence establishing that GraphAgent-Zero-Shot's fine-tuning data excludes Arxiv-Papers and ICLR-Peer Reviews, in contrast to the clear statement for the IMDB→ACM transfer. This is a verification gap in the evaluation protocol, not the architecture, and it directly undermines the headline claim of zero-shot generalization.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GraphAgent, an agentic pipeline that combines a Graph Generator Agent (which constructs semantic knowledge graphs from text), a Task Planning Agent (which parses user queries and grounds explicit or implicit graphs into graph tokens), and a Graph Action Agent (a Llama3-8B-based graph language model trained with graph-instruction alignment and curriculum multi-task finetuning). The system is evaluated on predictive tasks (node classification on IMDB, ACM, Arxiv-Papers, ICLR-Peer Reviews) and generative tasks (related-work generation and GovReport summarization). The headline claims are an average improvement of over 28% over HiGPT on IMDB-to-ACM zero-shot transfer, consistent outperformance of larger open-source LLMs on implicit-semantic prediction, and lower perplexity than several commercial and open-source LLMs on graph-enhanced generation, with additional LLM-as-judge preference results.","tokens_in":24126,"tokens_out":3394,"duration_ms":29438,"significance":"If the empirical results hold, the paper makes a useful contribution by showing that a relatively small open LLM, when embedded in a multi-agent pipeline with automatic semantic knowledge graph construction and graph tokenization, can match or exceed much larger models on graph-related predictive and generative tasks. The ablations (Figure 7) provide evidence that each of the three main components contributes, and the release of code at https://github.com/HKUDS/GraphAgent supports reproducibility. The main scientific value lies in the system-level integration and the corresponding evidence that the design choices matter; however, the zero-shot generalization claim, which is central to the paper's framing, rests on an under-specified evaluation protocol that must be clarified before the results can be accepted at face value.","major_comments":[{"comment":"The zero-shot configuration of GraphAgent-Zero-Shot is not specified. Unlike Section 3.2, which explicitly states that the model is trained on IMDB and evaluated on ACM, Section 3.3 never states what data GraphAgent-Zero-Shot was trained on. Table 1 lists 5,175 training samples for Arxiv-Papers and 3,141 for ICLR-Peer Reviews, and Section 3.1.2 explicitly says the ICLR-Peer Reviews dataset 'is used for both training and testing purposes.' If GraphAgent-Zero-Shot was fine-tuned on either of those training splits, or if the 500 evaluation samples overlap with the training data, then the Table 3 numbers for this variant are in-distribution supervised results, not zero-shot transfer, and the comparison against non-fine-tuned LLMs would be misleading. The authors must clearly report the training data for each variant (Task Expert, General, Zero-Shot) and, if the zero-shot model was trained on other data, re-run the evaluation on a truly disjoint split to substantiate the zero-shot claim.","section":"Section 3.3, Table 3, Section 3.1.2"},{"comment":"The generation-quality comparison relies primarily on perplexity computed with Llama3-70b and Qwen2-72b. The paper acknowledges same-family bias only by shading those cells lightly grey, but the central claims of lower perplexity than GPT-4o-mini, Gemini-1.5-Flash, and other evaluators of different families are still based on a metric that is known to favor models that are close to the evaluating LM. The authors should either add a third evaluator from a different model family, supplement with human judgments (even a small sample), or report significance/confidence intervals for the PPL differences; as written, a large part of RQ3's support rests on a single, potentially biased metric.","section":"Section 3.4, Tables 4 and 5, Figure 5"},{"comment":"The dataset statistics for the related-work generation task are inconsistent between the table and the text. Table 1 reports 4,155 training samples for 'Related Work Generation' but no evaluation count, while the text states 'approximately 5,000 topic-content pairs' and Section 3.4 mentions '200 samples from the text generation test set' for the judge-based evaluation. Please reconcile these numbers and provide an exact train/test split for the generative datasets so that the experimental results are reproducible and the evaluation scale is clear.","section":"Section 3.1.2, Table 1"}],"minor_comments":[{"comment":"Typo: 'anthor' should be 'another'.","section":"Section 2.4.2, Eq. (17)"},{"comment":"The reference list contains duplicate entries, e.g., Wang et al. 2019a and 2019b, and Velickovic et al. 2018 appears twice in different formats; these should be consolidated.","section":"References"},{"comment":"Table 8 is referenced as both the curriculum training table and a qualitative comparison table; the numbering is confusing. Renumber the appendix tables to avoid ambiguity.","section":"Appendix A.1.2 and Section 3.6"},{"comment":"The claim of '31.9% improvement across all metrics on both datasets' is not supported by the full table: GPT-4o-mini achieves a higher Mi-F1 on ICLR-Peer Reviews (0.692 vs 0.686 for GraphAgent-Task Expert), and several commercial models have comparable or higher AUC on one dataset. The claim should either be scoped to open-source LLMs or supported with a table-level significance test.","section":"Section 3.3, Table 3"},{"comment":"The system uses GPT-3.5-Turbo for the task planning and graph generation agents, so the open-source release cannot reproduce the full pipeline without relying on a proprietary API. The GitHub repository should clearly document which components can run entirely with open-weight models and which require external API access.","section":"Section 3.1.1, Implementation Details"}],"recommendation":"major_revision","confidential_remarks":"The central issue is the unverified zero-shot protocol for Table 3. This is fixable with a clear statement of training data and a truly disjoint evaluation, so I recommend major revision rather than rejection. The self-citation pattern (HiGPT, GraphGPT) is present but not egregious; the evaluations do compare against external baselines. One additional concern for the editor: the paper's headline '31.9% improvement' and 'over 28%' averages are computed across heterogeneous metrics and baselines, and the manuscript would benefit from either a pre-registered evaluation protocol or a more conservative presentation of these aggregate numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: a genuinely integrated agentic pipeline for graph-plus-text tasks, with an evaluation hole in the exact place the headline zero-shot claim sits. The architecture is sensible, the ablation is honest, and the IMDB-to-ACM transfer is well specified. But Section 3.3 never states what GraphAgent-Zero-Shot trained on, and one dataset is explicitly said to be used for both training and testing; until that is clarified, the Table 3 zero-shot column is unverified.\n\nWhat is actually new: not the individual pieces—automatic knowledge graph construction, graph tokenization, LLM agents all exist—but the end-to-end assembly with a curriculum-trained 8B action model that handles predictive and generative tasks from natural-language queries. The ablation study earns credit: removing the SKG component drops predictive Mi-F1 by roughly 15%, removing alignment hurts generative PPL by a large margin, and curriculum helps both. That is real evidence the components do work.\n\nWhere it gets soft: the zero-shot protocol. Table 1 lists 5,175 training samples for Arxiv-Papers and 3,141 for ICLR-Peer Reviews, and Section 3.1.2 says the ICLR dataset is used for both training and testing. Section 3.3 introduces GraphAgent-Zero-Shot but never says what it was trained on. If it used those splits, the zero-shot results are in-distribution supervised results and the comparison against non-finetuned LLMs is misleading. This is a verification gap in the evaluation protocol, not an architectural flaw, and it is fixable. The authors need to state the training data for each variant, and ideally run a truly held-out transfer.\n\nThe generation evaluation is softer. Perplexity is computed by Llama3-70b and Qwen2-72b, which are same-family with the judged outputs, and the GPT-4o judge results cover 200 samples with no human validation or agreement stats. That is not enough to support the claimed quality improvements. Also, the newly curated datasets are not released even though the paper says open-source; the GitHub link may contain code, but the data needs shipping.\n\nMinor: the intro's \"28% improvement\" refers only to the IMDB-to-ACM zero-shot table, which is fine, but it is phrased like a general result; an editor should ask for precise scope.\n\nWho this is for: people building graph-LLM agents and anyone benchmarking zero-shot graph transfer. It deserves a serious referee, because the core claim is plausible and the ablations are informative. Send it to review with a request to pin down the training/eval split, release the datasets, and add human or at least inter-annotator evaluation on generation. That is a conditional accept path, not a desk reject.","headline":"Solid integration paper for graph-LLM agents, but the zero-shot evaluation protocol is unspecified in exactly the place the headline claim depends on.","tokens_in":24733,"tokens_out":3001,"would_cite":false,"duration_ms":27540,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GraphAgent claims that an 8B open-source model, using three collaborating agents that build a semantic knowledge graph from text, can exceed prior graph language models by 28% on average and larger LLMs by 31.9% on graph-aware prediction…","keywords":["graph language models","semantic knowledge graphs","agentic pipeline","zero-shot node classification","graph tokenization","LLM-as-judge","text generation","heterogeneous graphs"],"falsifier":"Audit the training and evaluation splits: enumerate the exact ICLR-Peer Reviews samples (Table 1 lists 4,155 training samples) and Arxiv-Papers samples used for GraphAgent-Zero-Shot, and check them against the 500-sample evaluation sets; any overlap would invalidate the zero-shot claim. Separately, human rating of 200 generated related-work passages would check whether the lower perplexity and GPT-4o preferences match human judgment.","tokens_in":23662,"feed_emoji":"🕸️","tokens_out":9084,"duration_ms":75995,"temperature":0.7,"pith_summary":"GraphAgent is an automated pipeline that turns text into a semantic knowledge graph and uses that graph, plus any user-supplied graph, to answer questions and generate text. The paper argues that this lets a small open 8B model achieve stronger zero-shot node classification than the previous best graph language model, by over 28% on average, and beat much larger language models on prediction tasks that require implicit semantic relationships. In generation, it reports lower perplexity than GPT-4o-mini, Gemini-1.5-Flash, and Llama3-70B, with GPT-4o as judge preferring its related-work text in most comparisons. A sympathetic reader would care because the claim is that automatic relational structure extraction can compensate for model size, making capable graph analysis available in an open model.","feed_headline":"8B graph agent beats 70B models and GPT-4o-mini","feed_subtitle":"Auto-built knowledge graphs lift a small open model to top scores on zero-shot node classification and generation.","key_machinery":"The load-bearing component is the Semantic Knowledge Graph (SKG), produced by an iterative two-phase extraction where an LLM first identifies high-level scaffold entities, then augments each with a detailed description, and repeats so that fine-grained nodes hang off coarse ones; edges connect each derived node to the parent whose description generated it. The SKG is encoded with a pre-trained text encoder and a heterogeneous GNN into graph tokens, projected into the language model's embedding space through a learnable linear layer, and interleaved with language tokens. Training rests on graph-instruction alignment (intra-type and inter-type matching between graph tokens and text) followed by curriculum-ordered multi-task fine-tuning over predictive and generative agent tasks.","core_discovery":"The central discovery is that a semantic knowledge graph constructed automatically from plain text can serve as a compact, tokenizable representation that lets a graph-language model reason about both explicit links and implicit semantic ties. On node classification with explicit graphs, the paper reports an average improvement of more than 28% over the previous state-of-the-art graph language model across settings, with specific gains of 12.8% to 48.5% depending on shot count and metric. On implicit graph prediction, the 8B GraphAgent outperforms Llama3-70B and Qwen2-72B by a 31.9% average across all metrics on Arxiv-Papers and ICLR-Peer Reviews. For generation, GraphAgent achieves lower perplexity than several much larger or closed models and GPT-4o judges its related-work text better than Llama3-8B in 67% of cases and better than Llama3-70B in 58%.","pith_inferences":["The paper does not test whether the quality of the SKG is the bottleneck; a natural extension is to vary the number of extraction iterations or entity granularity and measure how prediction and generation performance respond, which would tell where the gains saturate.","Because the zero-shot claim depends on training/evaluation split disjointness that the paper does not explicitly document, an audit of the sample overlap is the first check a reader should perform before relying on the Table 3 numbers.","The task planning and graph generation agents currently rely on GPT-3.5-Turbo, so the full pipeline is not entirely open; replacing those two agents with an open model would test how much of the benefit is due to the architecture versus the proprietary planner.","If the pipeline transfers to other text-heavy domains, a likely consequence is that scientific literature, legal documents, and clinical notes could get graph-aware assistants built on small models plus automated knowledge graphs, without hand-curated ontologies."],"forward_implications":["If the results hold, graph-aware prediction no longer requires a task-specific GNN or a very large proprietary model: an 8B open model with an automatically built SKG can carry zero-shot node classification and implicit semantic prediction.","Graph-enhanced generation benefits more from structured knowledge injection than from retrieval prompting: GraphAgent beats vanilla SFT and GraphRAG baselines of the same size on perplexity and judged quality.","A user who can upload a graph file or paste text, then ask in natural language, can get predictions with reasoning and generated text, because task planning and graph construction are automated inside the pipeline.","The ablation results imply that semantic knowledge graphs contribute most to predictive gains, while graph-instruction alignment is the key to generation quality, and curriculum training helps both."],"supporting_citations":[{"why":"Supplies the heterogeneous graph-language model architecture and graph-tokenization approach that the Task Execution Agent builds on, and is the main baseline GraphAgent outperforms in zero-shot node classification.","marker":"Tang et al. (2024b)"},{"why":"GraphGPT contributes the graph instruction tuning and encoder–LLM alignment paradigm that GraphAgent extends from graph-only tasks to agentic predictive and generative tasks.","marker":"Tang et al. (2024a)"},{"why":"LLaGA provides a structure-aware graph-to-sequence baseline for graph LLMs, one of the comparative methods for predictive tasks.","marker":"Chen et al. (2024a)"},{"why":"Llama 3-8B is the foundational LLM of the Graph Action Agent and the base for the fine-tuned, GraphRAG, and perplexity-judge baselines.","marker":"Llama Team (2024)"},{"why":"Validates the LLM-as-judge methodology used to claim that GraphAgent's generated related work is preferred over stronger baselines.","marker":"Zheng et al. (2024)"},{"why":"Supplies the Arxiv-Papers dataset used to evaluate implicit-graph prediction of paper acceptance from titles and abstracts.","marker":"He et al. (2023)"},{"why":"Supplies the IMDB benchmark used for few-shot training in the zero-shot transfer evaluation on node classification.","marker":"Fu et al. (2020)"},{"why":"Supplies the ACM dataset used as the zero-shot target domain for evaluating node-classification transfer.","marker":"Wang et al. (2019a)"},{"why":"PyG is the graph-building and wrapping tool used to ground arbitrary uploaded graphs into Python objects before tokenization.","marker":"Fey & Lenssen (2019)"}],"fun_headline_variants":["8B agent with auto knowledge graphs beats 70B models","Auto-built graphs boost 8B model beyond 70B rivals","Knowledge graphs turn 8B model into 70B-class performer","Small model, auto knowledge graphs: 8B bests 70B and GPT-4o-mini"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline zero-shot numbers stand on the premise that GraphAgent-Zero-Shot was trained only on data that is fully disjoint from the Arxiv-Papers and ICLR-Peer Reviews evaluation samples, so those results measure true transfer rather than memorized overlap.","fun_headline_variants_meta":{"raw":{"variants":["8B agent with auto knowledge graphs beats 70B models","Auto-built graphs boost 8B model beyond 70B rivals","Knowledge graphs turn 8B model into 70B-class performer","Small model, auto knowledge graphs: 8B bests 70B and GPT-4o-mini"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000188,"raw_usage":{"total_tokens":1340,"prompt_tokens":960,"completion_tokens":380,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":297}},"tokens_in":576,"tokens_out":380,"duration_ms":3782,"temperature":1.0,"reasoning_tokens":297,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:51:13.276430+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Audit the training and evaluation splits: enumerate the exact ICLR-Peer Reviews samples (Table 1 lists 4,155 training samples) and Arxiv-Papers samples used for GraphAgent-Zero-Shot, and check them against the 500-sample evaluation sets; any overlap would invalidate the zero-shot claim. Separately, human rating of 200 generated related-work passages would check whether the lower perplexity and GPT-4o preferences match human judgment.","supporting_citations":[],"review_version":1}