{"id":"4bede90a-a1e1-4b5d-8062-086d4a7aefc2","arxiv_id":"2501.14300","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"FastToG lets LLMs reason 'community by community' over knowledge graphs, reporting higher accuracy and faster reasoning than Think-on-Graph.","lead":"FastToG is a retrieval method that groups knowledge-graph facts into communities and has a language model reason from one community to the next when answering questions. The paper reports better accuracy and fewer AI calls than the previous Think-on-Graph method on six question-answering benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unspecified subgraph sampling probability ρ leaves retrieval recall unmeasured; the accuracy and speed gains over ToG are not yet established.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing concern I find: the exponential-decay sampling parameter ρ is never specified, and the paper provides no experiment measuring whether the sampled local subgraph preserves the evidence needed to answer. That is not a minor implementation detail. The LCS procedure builds every reasoning chain from these sampled subgraphs, so the entire 'community-by-community' mechanism is downstream of an uncontrolled random choice. I agree with the reader's CONDITIONAL verdict: the paper's positive results are coherent and the code repository is a useful step, but the missing ρ and lack of recall/sensitivity analysis prevent the central accuracy and speed claims from being fully established. I would not escalate to REJECT because the concern is addressable by a straightforward experimental addition, and the reported gains, while small in places, are consistent across two LLMs. I would not downgrade to ACCEPT because without the ρ specification and recall check, the main contribution is not reproducible enough. The conditional verdict, with requests for the sampling parameter, sensitivity analysis, and prompt release, is the right call.","tokens_in":1050,"tokens_out":755,"duration_ms":62005,"concrete_test":"Run FastToG and ToG on the same 1k subsets with ρ in {0.5, 0.7, 0.9, 1.0} (or the value used in the released code), and for each question record (a) whether the gold answer entity or relation lies inside the sampled subgraph and (b) exact-match accuracy and AvgDepth. If accuracy varies by more than the reported ToG gap (roughly 1-6 points), or if retrieval recall drops substantially while the ρ=1.0 full-subgraph baseline reaches higher accuracy, then the claimed community-based advantage is not robust and the speed gain conflates sampling sparsity with method design. Also check the repository's default ρ to confirm the reported numbers correspond to a disclosed setting.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that FastToG improves accuracy and speed rests on Local Community Search (LCS), whose first step (Section 'Community Detection on Subgraph') builds a local subgraph by sampling each n-hop neighbor with probability ρ^(n-1). The paper never states ρ, reports no sensitivity analysis, and provides no recall experiment measuring whether the sampled subgraph retains the entities and relations needed to answer a question. If ρ is small, the subgraph is aggressively truncated; communities computed on that subgraph can omit the decisive evidence, and the subsequent coarse/fine pruning cannot recover it. This makes the observed gains over ToG hard to attribute to 'community-by-community' reasoning rather than to pruning noise or LLM prior knowledge. The efficiency claim is likewise entangled: the reported reduction in average depth (Fig. 3) may partly reflect a sparse, incomplete subgraph rather than a genuine advantage of community units. Since the differences over ToG are small on several benchmarks (e.g., +0.2-1.0 points on ZSRE, QALD, and Creak in Table 1), an uncontrolled sampling parameter is a serious threat to the headline result.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FastToG, a GraphRAG method that replaces single-entity reasoning steps with community-level units. After extracting the query entity, FastToG repeatedly applies Local Community Search: it samples a local subgraph, runs community detection, coarse-prunes communities by modularity, fine-prunes by an LLM, and appends the selected communities to W parallel reasoning chains. The communities are converted to text either by rule-based Triple2Text or by a fine-tuned T5-based Graph2Text module. Experiments on six datasets (CWQ, WebQSP, QALD, ZSRE, TREx, Creak) with two LLMs report exact-match accuracy, reasoning depth, and ablations over community-detection algorithms, pruning method, and community-size limit. The central claims are higher accuracy, faster reasoning, and better explainability than ToG and other GRAG baselines.","tokens_in":14478,"tokens_out":4146,"duration_ms":40034,"significance":"If the results are substantiated, treating communities rather than individual entities as the unit of LLM reasoning is a genuinely useful idea for GraphRAG: it can shorten the number of LLM calls and provide structurally richer context. The paper also contains useful design probes, including a modularity-vs-random pruning comparison, a comparison of four community-detection algorithms plus random partitioning, and an analysis of Graph2Text hallucination types. Public code and pseudocode are supplied, which aids reproducibility. However, the empirical support is currently incomplete: the neighbor-sampling rate is unspecified and unmeasured, no confidence intervals or significance tests are reported, and the community-size hyperparameter is selected on the evaluation datasets, so the headline accuracy and efficiency claims are not yet firmly established.","major_comments":[{"comment":"The exponential neighbor-sampling probability P r(x = 1) = rho^(n-1) is a free parameter that is never given a value anywhere in the manuscript. This sampling step produces the subgraph on which all subsequent community detection and pruning operate, so if rho is small, the omitted nodes and edges can never be recovered by coarse or fine pruning. The paper reports no sensitivity analysis for rho and no recall experiment measuring whether the sampled subgraph retains the entities and relations needed to answer the query. I consider this a load-bearing gap: the reported accuracy gains over ToG could partly reflect the luck of the subgraph sample rather than the community-based reasoning. Please report the value of rho used, study its effect on accuracy and depth, and measure retrieval recall of the sampled subgraph against the full local graph.","section":"Community Detection on Subgraph"},{"comment":"The headline improvements \"4.4% in Tab. 1 and 5.9% in Tab. 2\" are selected from single datasets: 4.4% is the TREx improvement for gpt-4o-mini (68.6 vs. 64.2), and 5.9% is the CWQ improvement for llama-3-70b (46.2 vs. 40.3). Most other differences are much smaller (often 0.2 to 2.2 percentage points), and a few are negative, e.g., QALD g2t underperforms t2t by 0.2 points in Table 1. No confidence intervals, standard deviations, repeated runs, or significance tests are reported, so the reader cannot assess whether the differences are stable. Please report per-dataset variance or bootstrap confidence intervals, and give the average improvement across datasets rather than only the maximum single-dataset gain.","section":"Performance on Accuracy (Tables 1 and 2)"},{"comment":"The maximum community size M = 4 is selected after inspecting accuracy on the same evaluation datasets that are later used for the headline comparison, and the same tuning is not performed for ToG or the other baselines. This is effectively selection on the test data and biases the comparison in favor of FastToG. Please define a validation protocol (e.g., a held-out split or cross-validation) for choosing M, or report results as a function of M for both FastToG and ToG so the comparison is not confounded by asymmetric hyperparameter selection.","section":"Ablation Study (Fig. 4)"},{"comment":"The efficiency claim is based on average reasoning depth, not wall-clock time or measured LLM calls, and the depth reduction may be entangled with the subgraph sampling in Local Community Search. If the sampled subgraph is aggressively truncated, shorter chains can simply mean that less evidence is available, which could also explain some accuracy differences. Please report actual end-to-end latency or measured LLM call counts together with accuracy, and show how the depth reduction varies with rho in the sensitivity analysis requested above.","section":"Performance on Efficiency (Fig. 3)"}],"minor_comments":[{"comment":"The sentence \"All the prompts are open source and publicly available at xxx\" contains a placeholder URL; the link must be completed before publication.","section":"Appendix C (Prompt Design)"},{"comment":"The sentence \"Not that the length of each chain p may not be the same\" should read \"Note that the length of each chain p may not be the same.\"","section":"Reasoning"},{"comment":"The phrase \"literately detect the communities\" should be \"iteratively detect the communities.\"","section":"The Method (Overview)"},{"comment":"Please clarify how the random community detection (\"Rand\") is constructed, in particular whether it preserves the number and size constraints used by the non-random algorithms; this would make the comparison more interpretable.","section":"Table 3"},{"comment":"The rows labeled \"1-d n-w\" are described in the text as representing KGP, but the table caption does not include the citation; please make the mapping from table rows to cited baselines explicit in the table caption.","section":"Experiments (Accuracy)"},{"comment":"The hallucination statistics in Fig. 7 are based on manual analysis of 100 samples per dataset and per community size, but no inter-annotator agreement or confidence intervals are reported; adding this information would strengthen the G2T analysis.","section":"Appendix B2 (Errors Analysis)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a reasonable candidate after revision, but the omitted rho value and the absence of any uncertainty quantification are serious empirical gaps. I would also draw the editor's attention to the placeholder 'xxx' for the prompt repository and the test-set-based selection of M, both of which reduce the reliability of the headline claims. The distinction between internal consistency and external validity is not at issue here; the method is benchmarked externally, so the required fix is additional experiments and reporting, not a change of claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"FastToG is a coherent extension of ToG: you swap entity-level hops for community-level hops, and that shortens the reasoning chains. The specific combination—local community search, modularity-based coarse pruning, LLM fine pruning, and two community-to-text converters—is new to me, and the ablations support the design choices. Community detection beats random partitioning, modularity pruning beats random pruning, and the case study gives a concrete sense of why a community unit is easier for an LLM to score than a single node. The authors are also honest about the G2T hallucination problem, which is a good sign.\n\nThe soft spots are real and mostly about measurement. The sampling probability ρ for the local subgraph is never stated, and there is no recall experiment showing the sampled subgraph retains query-relevant entities. If ρ is set aggressively, the depth reduction in Fig. 3 could come from a truncated subgraph rather than from the community mechanism. The maximum community size M is tuned on the evaluation datasets (Fig. 4), so there's a selection effect. The reported gains are single numbers without confidence intervals, and the headline '4.4% and 5.9%' improvements are the largest cell-wise differences; on several datasets the gap over ToG is under a point. Efficiency is reported only as average depth, not wall-clock time or token usage, so we don't know whether longer community texts eat into the savings. The prompt link in the appendix is still 'xxx'.\n\nThe central idea is sound and the execution is generally careful. I'd send this to review for a venue that cares about GraphRAG, but I'd insist on revisions that specify ρ, add error bars, report latency/cost, and either fix the prompt link or remove it. It's an incremental but useful contribution, not a field-changer.","headline":"A useful community-level extension of ToG, but the missing sampling parameter and lack of error bars keep the empirical claims from being fully established.","tokens_in":15042,"tokens_out":4047,"would_cite":true,"duration_ms":36667,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FastToG claims that reasoning from one knowledge-graph community to the next is more accurate, faster, and more explainable than stepping through single nodes.","keywords":["FastToG","graph retrieval-augmented generation","knowledge graph reasoning","community detection","local community search","modularity pruning","community-to-text conversion","chain-of-thought reasoning"],"falsifier":"Take a set of multi-hop knowledge-base QA questions whose gold answers sit at known distances (for example 3–5 hops) from the seed entity, run FastToG while varying the decay factor $\\rho$, and measure whether the sampled subgraph still contains the gold answer at each setting; if answer entities vanish from the subgraph before the number of LLM calls drops, the claimed efficiency-accuracy trade-off fails.","tokens_in":14043,"feed_emoji":"🧩","tokens_out":11587,"duration_ms":89853,"temperature":0.7,"pith_summary":"The paper argues that a graph-RAG system can get both wider and deeper evidence for an LLM by making the unit of reasoning a community of nodes rather than a single entity or triple. FastToG builds a fixed number W of parallel reasoning chains whose steps are communities found by local community detection, prunes candidate communities first by modularity and then by LLM selection, and feeds the chains to the LLM as text. On six question-answering, slot-filling, and commonsense benchmarks, the paper reports that FastToG beats node-step baselines such as Think-on-Graph, and that community-sized steps shorten the chains enough to reduce the number of LLM calls. If this is right, dense knowledge graphs stop being a cost bottleneck for retrieval-augmented generation.","feed_headline":"Community-sized steps speed up graph LLM reasoning","feed_subtitle":"FastToG beats node-by-node Think-on-Graph on six benchmarks while cutting LLM calls","key_machinery":"The load-bearing object is the community, used as the step of a reasoning chain. Instead of expanding one node at a time and asking the LLM to choose among many neighbors, FastToG partitions a locally sampled subgraph into dense groups, keeps only the top communities by modularity $Q(c)=\\sum_{\\rm in} - \\left(\\sum_{\\rm tot}\\right)^2/(2m)$, then lets the LLM choose among the survivors. This compresses many graph paths into a single step, which is why the paper claims fewer LLM calls and shorter chains; the two Community-to-Text converters (Triple2Text and Graph2Text) are what let the LLM read the community's internal triples at all.","core_discovery":"FastToG's central claim is that 'thinking community by community' gives an LLM the same or better answer quality as stepping through graph nodes one by one, while needing fewer steps. The method starts by having the LLM extract the subject entity of the query as a single-node community, then repeatedly runs Local Community Search: it samples a local subgraph within a hop radius (dropping nodes with probability $\\rho^{n-1}$ with hop $n$), detects communities on that subgraph with algorithms such as Louvain, prunes low-modularity communities in a coarse stage, asks the LLM to pick the most relevant community in a fine stage, and grows $W$ parallel chains of communities. Communities are converted to text either by joining triples (Triple2Text) or by a fine-tuned T5 summarizer (Graph2Text) before the LLM reads them. The paper reports accuracy gains over all baseline families on CWQ, WebQSP, QALD, ZSRE, TREx, and Creak, with the largest margins over Think-on-Graph (ToG), and shows in ablation that the average chain depth drops as the maximum community size grows.","pith_inferences":["A testable extension not explored in the paper: the speed advantage should grow with graph density, because community compression collapses more paths per step on dense graphs; running FastToG on synthetic graphs with controlled density would isolate this effect from dataset difficulty.","The paper leaves the decay factor $\\rho$ of the hop-sampling probability unspecified and does not measure whether the sampled subgraph still contains answer entities; a cheap diagnostic is to record answer-recall of the sampled subgraph against full retrieval, which would reveal the real upper bound on accuracy.","If community-based chains make reasoning traces shorter and more human-readable, the same macro-step idea could transfer to other structured data such as relational databases or code dependency graphs, where dense local structure can be grouped before an LLM reads it."],"forward_implications":["On the six reported benchmarks, community-based chains beat node-based Think-on-Graph by 4.4 points with gpt-4o-mini and 5.9 points with Llama-3-70b, so the community step is not only a speed trick but also improves answer accuracy.","Increasing the maximum community size from 1 to 2 already cuts average reasoning depth noticeably (about 0.2–0.4 on CWQ and about 1.0 on WebQSP), and the reduction translates directly into fewer calls to the LLM per question.","Modularity-based coarse pruning outperforms random pruning on CWQ and WebQSP in every tested configuration, so the structural quality of the retained communities matters for accuracy.","Setting the maximum community size too high (8 in the ablation) starts to hurt accuracy, meaning the method has an internal accuracy–efficiency trade-off rather than a monotone gain."],"supporting_citations":[{"why":"The Think-on-Graph baseline FastToG extends, supplying the W reasoning chains and per-step LLM pruning that FastToG reworks into community steps.","marker":"Sun et al. 2023"},{"why":"Provides Louvain community detection and the modularity objective used for coarse pruning.","marker":"Blondel et al. 2008"},{"why":"Supplies the 'Let's think step by step' chain-of-thought idea that FastToG converts into community-by-community reasoning.","marker":"Kojima et al. 2022"},{"why":"The community-based GraphRAG method that establishes densely connected groups as retrieval units for LLMs.","marker":"Edge et al. 2024"},{"why":"Contributes the Girvan-Newman divisive community detection algorithm used in the comparison.","marker":"Girvan and Newman 2002"},{"why":"Chain of Knowledge, a 1-d 1-w knowledge-graph retrieval baseline FastToG is compared against.","marker":"Li et al. 2023"},{"why":"StructGPT, an n-w knowledge-graph retrieval baseline in the comparison set.","marker":"Jiang et al. 2023"},{"why":"T5-base, the model fine-tuned to build the Graph2Text community-to-text converter.","marker":"Raffel et al. 2020"}],"fun_headline_variants":["Graph LLMs think faster in community-sized chunks","Divide KGs into communities for faster LLM reasoning","Community detection trims graph steps for LLM QA","FastToG: community hopping beats node hopping in KGs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that sampling neighbor nodes with probability $\\rho^{n-1}$ still leaves every entity and relation needed to answer the query inside the local subgraph, because any answer entity dropped here cannot be recovered by later pruning or reasoning.","fun_headline_variants_meta":{"raw":{"variants":["Graph LLMs think faster in community-sized chunks","Divide KGs into communities for faster LLM reasoning","Community detection trims graph steps for LLM QA","FastToG: community hopping beats node hopping in KGs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000452,"raw_usage":{"total_tokens":2293,"prompt_tokens":983,"completion_tokens":1310,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":1246}},"tokens_in":599,"tokens_out":1310,"duration_ms":9149,"temperature":1.0,"reasoning_tokens":1246,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:14:29.677814+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of multi-hop knowledge-base QA questions whose gold answers sit at known distances (for example 3–5 hops) from the seed entity, run FastToG while varying the decay factor $\\rho$, and measure whether the sampled subgraph still contains the gold answer at each setting; if answer entities vanish from the subgraph before the number of LLM calls drops, the claimed efficiency-accuracy trade-off fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the Girvan-Newman divisive community detection algorithm used in the comparison."}],"review_version":1}