{"id":"8a82a3e6-0aa2-4005-8751-4d24300b788b","arxiv_id":"2505.02130","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Fine-tuning LLMs on graph tasks with a restricted attention window that includes partial graph links, then switching to a fully connected window at inference, improves node classification over standard full-attention fine-tuning.","lead":"This paper studies how large language models pay attention when reading graph data written as text. It finds that models do not use the connections between nodes well, and that training with a limited attention window then switching to a full window improves accuracy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Attention-window experiments lack a random-mask control, so the claim that graph topology, rather than reduced context, drives the transfer gain is not established.","rationale":"The reader's weakest-assumption concern targets the disruption experiment in A2 and the missing GNN baseline; that concern is valid but diagnostic. The more load-bearing issue is the attention-window experiment behind the paper's positive recommendation: the comparison in Table 4 conflates the amount of visible context with whether the visible context is topologically selected. A random-mask control would isolate the proposed mechanism. This does not falsify the paper, but it strengthens the case for keeping the verdict CONDITIONAL, so no verdict change is recommended. Credit is due for releasing code, testing multiple base models in appendices, and for an honest limitation statement; those do not resolve the missing control.","tokens_in":23261,"tokens_out":5196,"duration_ms":77652,"concrete_test":"Run the Table 3/4 pipeline on all four datasets (Llama-2-7B, 8x8 sampling) with random attention masks matched per row to the same number of visible tokens as the k-hop topological mask (k=1,2,3, unidirectional and bidirectional), and evaluate the identical transfer grid to k=4. If random-mask training yields accuracy within noise of the topological-mask results, the 'graph topology' interpretation of Finding 3 is refuted; if topology masks are significantly better (paired test across seeds), the concern is settled.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's positive recommendation (Finding 3) is that training under intermediate attention windows with graph topological links and then transferring to a fully connected window is superior. The evidence in Section C (Tables 3 and 4) varies two quantities at the same time: the set of visible nodes is constrained by graph distance (k), and the number of visible tokens is much smaller than at k=4. The paper never compares against random attention masks matched to the same per-row visibility count or to the same context size. Its own explanation for large-to-small transfer is 'reduced level of complexity and fewer distractions' - a context-size/regularization account, not a graph-topology account. Without a control that holds visible-set size fixed while removing topological information, the data support only 'training with limited windows helps', not 'training with graph-hop windows helps because of topology'. This is the load-bearing step for the headline recommendation, separate from the disruption analysis in A2, which is diagnostic and already mixed across datasets.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an empirical, attention-centric study of how LLMs process graph-structured data. Using LLaMA2-7B fine-tuned with LLaGA-style instructions on four node-classification datasets, it analyzes attention score distributions before and after fine-tuning, performs connectivity-disruption experiments, visualizes node-node and text-node attention matrices, and varies a 'global linkage horizon' k to restrict attention windows. The paper claims three main findings: (1) LLMs recognize graph data but do not effectively use connectivity information; (2) LLM attention does not match an ideal structural pattern, exhibiting Attention Sinks and a graph-specific 'Skewed Line Sink'; and (3) neither fully connected attention nor fixed connectivity is optimal, with intermediate graph-hop attention windows being best for training and transferring well to fully connected inference. The paper also includes supplementary experiments with Vicuna-7B and LLaMA3-7B, template comparisons, and per-layer attention visualizations.","tokens_in":23409,"tokens_out":5853,"duration_ms":72137,"significance":"If the conclusions hold, this is a useful empirical contribution: it is one of the first studies to inspect LLM attention behavior on graph-structured data at the level of node-text and node-node interactions, it ships source code, and it tests multiple datasets and base models. The disruption methodology and the separation of text-node from node-node attention are valuable lenses for the LLM-for-graphs community, and the central qualitative observation that connectivity disruption often does not change LLM accuracy is a credible, falsifiable claim. However, the prescriptive Finding 3 rests on a comparison that confounds graph-topology restriction with context-size reduction, and the diagnostic Finding 1 lacks the GNN control needed to interpret null disruption results as 'failure to use connectivity.' These gaps are fixable with additional experiments, but they are load-bearing for the headline claims.","major_comments":[{"comment":"The conclusion that LLMs 'fail to effectively utilize connectivity information' relies on the premise that a model genuinely using graph connectivity would show monotonically decreasing accuracy as disruption level increases. The paper asserts this premise in the sentence 'GNNs that passed the WL-test have been continuously decreasing' but provides neither a GNN baseline experiment nor a citation for this behavior, and the WL-test is not defined. Without such a control, the null results on Wikics, Pubmed, and Amazon-Ratings can be explained by the task being largely solvable from node text alone, or by the higher disruption levels (III-IV) damaging position/format rather than topology. A fair test requires running GNN baselines through the same disruption protocol and demonstrating that they degrade monotonically under levels I-IV; otherwise the interpretation of the null result is unsupported.","section":"Section A2, Table 2"},{"comment":"The attention-window manipulation varies two quantities simultaneously: the set of visible nodes is constrained by graph distance (k), and the number of visible tokens is much smaller at low k. The paper reports no random-mask control that fixes the per-row visibility count or the total context size while removing topological information. Consequently, the experiments support only the weaker conclusion that 'training with limited windows helps,' not the stronger claim that 'training with graph-hop windows helps because of topology.' The authors' own explanation for large-to-small transfer ('reduced level of complexity and fewer distractions') is a context-size/regularization account rather than a graph-topology account. A control with random attention masks matched to the same visible-set size is needed to establish that topology, not reduced context, drives the improvements.","section":"Section C, Tables 3 and 4, Figure 6"},{"comment":"The claimed small-to-large transfer advantage is not consistently supported by the reported numbers. On Wikics (Table 4), training at k3_unidi and testing at k4_unidi gives 77.82 versus 77.49 for direct k4 training, a 0.33-point gain with overlapping standard deviations, while training at k2_unidi and testing at k4 gives 74.97, which is worse. Appendix L shows a larger gain on Roman-Empire (k2_unidi to k4_unidi: 82.66 versus 80.73), but the effect is absent or reversed on Amazon-Ratings, and no Pubmed transfer table is provided. The statement that 'we can achieve better performance at inference time with k=4 by training at k=2 or k=3' is too strong given the mixed evidence; the claim should be qualified per dataset with significance testing.","section":"Table 4 and Appendix L"},{"comment":"The diagnosis of a 'failure to adapt to graph topology' assumes an ideal attention pattern in which central/highly-connected nodes receive higher attention than peripheral nodes and attention decays hierarchically. This ideal is asserted rather than validated; the paper does not show that models with attention closer to this pattern achieve better task performance. The observed U-shaped and positional attention curves are interesting as observations, but labeling them as deficient requires an independent argument connecting the normative pattern to graph-task performance. In addition, the paper does not report a quantitative check that the fixed-position and shuffling procedure fully eliminates position-type confounds between first-order and second-order nodes.","section":"Section B1, Figure 3"}],"minor_comments":[{"comment":"The t-tests and KS tests are run on very large collections of attention scores; p-values alone are not informative. Please report effect sizes, and interpret the small JS divergence values (0.006 to 0.079) when claiming that attention distributions changed 'significantly.'","section":"Table 1 and Appendix E"},{"comment":"There is a typo: 'use embedding Ture' should be 'True.' The appendix also does not describe how the attention-window mask is implemented (which layers are masked, how bidirectional masking is realized in a causal decoder-only model, and how k=0 would be defined). These details are needed for reproducibility.","section":"Appendix A, Table 5"},{"comment":"The phrase 'slash trend' is unclear; specify whether it means a decreasing trend, a diagonal pattern, or something else. Also, 'caucal' in Figure 6 should be 'causal.'","section":"Figure 3 caption and Figure 6 caption"},{"comment":"The prompts for Pubmed and Wikics contain stray '::' characters and an extra question mark. The dataset name is also written inconsistently as 'Wikics' and 'WikiCS' across the paper and the dataset appendix.","section":"Appendix J"},{"comment":"The sentence 'When shifting to a narrower perspective, the model benefits from a reduced level of complexity and fewer distractions, leading to better.' is incomplete and should be finished.","section":"Section 4, Findings"},{"comment":"The text says k ranges from 0 to 2L, but Table 3 reports only k=1 to 4. Please explain why k=0 is not included in the main results, or include it for completeness.","section":"Section C1"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the reader's concern about circularity is not, in my view, a case of the paper fitting free parameters and then re-deriving them; the empirical outcomes are observed. The deeper problem is missing controls: a GNN baseline for the disruption protocol and a random-mask control for the attention-window experiments. Both are feasible extensions within the scope of the paper, so I recommend major revision rather than rejection. The paper also leans on the authors' own prior works in places, but that is not by itself disqualifying. The topic fits the conference's machine-learning scope, though the contribution is more empirical-diagnostic than methodological."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a solid empirical first pass at what LLM attention actually does when the input is a graph. It contributes three things worth knowing: (1) a visualization-and-statistics showing that after fine-tuning, attention to node tokens changes but the model still doesn't strongly use connectivity; (2) two named phenomena, an attention sink and a \"Skewed Line Sink,\" that are new in the graph-LLM context; (3) a training recipe: restrict the attention window during training (k=2/3) and use full attention at inference, which does better than training with full attention. The code is released, which is a real plus.\n\nThe attention analysis is the strongest part. The distribution plots and the statistical tests support the claim that node-node attention doesn't match a structural prior, and the disruption experiments on Roman-Empire show progressive degradation. The authors deserve credit for being the first to look systematically at attention in this setting.\n\nThe soft spots are in the transfer experiments (Section C, Tables 3-4), and they matter because the paper's headline recommendation is \"train small, infer large.\" The problem, as the stress-test note correctly identifies, is confounding: changing the global linkage horizon k changes both the graph-topological information available and the total number of visible tokens. There is no random-mask control matched for visible-set size. Without one, the data only support \"limiting the attention window during training helps,\" not \"graph-hop windows help because of topology.\" The authors' own explanation for large-to-small transfer is \"reduced complexity and fewer distractions,\" which is a context-size account, not a topology account. That's a load-bearing gap.\n\nSecond, the disruption experiments lack the GNN baseline they invoke. The text says \"GNNs that passed the WL-test have been continuously decreasing,\" but no such curve is shown or cited. On WikiCS and Pubmed, low-level disruption has essentially no effect on the LLM; that could mean the model isn't using connectivity, or it could mean the task is solvable without it. A real GNN comparator would settle that.\n\nThird, the accuracy comparisons across k values have no significance tests, and the transfer gain is small and inconsistent across datasets. I'd want validation-based selection rather than arguing from the best cell.\n\nNone of this sinks the core observation that LLMs underuse graph connectivity. But the stronger conclusion needs the control experiment and baselines before I'd trust it. The paper deserves a serious referee; I'd send it to review with the expectation of a revision that adds a random-mask control, a GNN disruption baseline, and significance tests.","headline":"Real attention-level observations for LLMs on graphs, but the headline transfer recipe lacks the control needed to separate topology from context size.","tokens_in":23980,"tokens_out":2271,"would_cite":true,"duration_ms":24486,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that neither fully connected attention nor fixed-link attention is optimal for LLMs on graphs, and that training with an intermediate attention window that includes some topology transfers cleanly to full attention at…","keywords":["attention mechanisms","graph-structured data","large language models","attention window","Global Linkage Horizon","node classification","attention sink","graph topology"],"falsifier":"Run the same four disruption levels on a GNN baseline with identical node sampling: if the GNN's accuracy is flat across levels I and II on Wikics or Pubmed, the yardstick for \"ignoring topology\" fails. Alternatively, on a dataset with random node text but real links, if an LLM's accuracy drops at levels I or II, then LLMs do use connectivity, contradicting Finding 1.","tokens_in":22980,"feed_emoji":"🔗","tokens_out":6365,"duration_ms":69368,"temperature":0.7,"pith_summary":"This paper is an empirical study of how transformer attention in large language models behaves when the input is graph-structured text: a target node plus its neighbors written out as sentences. It argues that LLMs learn to notice nodes and to attend from text to nodes, but fail to model edges between nodes, so their attention is poorly matched to graph topology. The paper proposes the Global Linkage Horizon, a mask that controls how far a node token can attend to other node tokens, and shows that neither the fully connected view of LLMs nor the fixed-neighbor view of GNNs is best. Training with an intermediate window that contains some topology improves node classification, and a model trained at a small window can be deployed at the full window without retraining.","feed_headline":"Train graph LLMs on narrow attention, deploy on full","feed_subtitle":"An intermediate linkage horizon that exposes real links beats both GNN-style and fully connected windows.","key_machinery":"The central object is the Global Linkage Horizon, a masking parameter $k \\in \\{0,\\dots,2L\\}$ that defines, within one attention layer, which node tokens may attend to which other node tokens. It interpolates between the fixed adjacency of a GNN ($k=1$) and the fully connected window of an LLM ($k=4$ for $L=2$), turning the model's visible perspective on the graph into an experimental variable. The argument is carried by two measurement tools: four disruption levels (I--IV) that scramble links with increasing severity, and density and heatmap statistics of attention scores before and after fine-tuning. The named phenomena \"Attention Sink\" and \"Skewed Line Sink\" are used to explain why node-to-node attention fails to follow graph structure.","core_discovery":"Using disruption experiments and attention-score statistics on four node-classification datasets, the paper finds that when link structure is progressively scrambled, most fine-tuned LLMs barely change accuracy at low disruption levels, indicating that they do not effectively use connectivity. Attention heatmaps show sink positions that attract disproportionate attention, plus a newly named \"Skewed Line Sink,\" rather than topology-aware patterns. The constructive result is that the attention window, defined by the Global Linkage Horizon $k$ (from $0$ to $2L$ for $L$ sampled hops), is tunable: intermediate $k$ values that expose some real links outperform both full connectivity $k=4$ and the GNN-style $k=1$ window. Small-to-large transfer also works: training at $k=2$ or $k=3$ and evaluating at $k=4$ beats training and evaluating at $k=4$.","pith_inferences":["A direct testable extension would be to use uninformative node text, such as random descriptions, while keeping real links: if a fine-tuned LLM still solves the task, the paper's text-driven reading is supported, and if it fails, the null disruption results may be an artifact of redundant text.","The small-to-large transfer result may generalize to other structured inputs, such as tables or code, where restricting early-layer attention to local dependencies during training could act as a regularizer.","The \"Skewed Line Sink\" suggests the model latches onto sequence position or path order; an explicit test that randomizes node order without changing topology would show whether this artifact comes from the decoder's causal bias rather than from graph structure."],"forward_implications":["If LLMs are trained with a Global Linkage Horizon of $k=2$ or $k=3$, node-classification accuracy on these benchmarks can exceed both the standard fully connected setting and the GNN-style fixed-link setting.","Small-to-large transfer means the method is deployment-friendly: training with a masked attention window costs nothing extra at inference, because the model then runs with normal full attention.","The two named attention artifacts give concrete targets for future corrections: suppressing attention sinks and skewed-line sinks on graph inputs may close part of the performance gap to GNNs.","The weak response to low-level connectivity disruption on most datasets implies that current instruction-tuned LLM gains on graphs are driven mostly by node text, not by links."],"supporting_citations":[{"why":"Supplies the instruction-tuned graph LLM whose attention distributions and disruption behavior are analyzed.","marker":"Chen et al. (2024b)"},{"why":"Defines the attention-sink phenomenon that the paper detects in its node-attention matrices.","marker":"Xiao et al. (2024)"},{"why":"Provides the fixed-link message-passing baseline that marks one end of the attention-window spectrum.","marker":"Kipf & Welling (2022)"},{"why":"Establishes the view of transformers as fully connected graph attention, the other end of the spectrum.","marker":"Kim et al. (2022)"},{"why":"Supplies the base LLM used for fine-tuning and attention analysis.","marker":"Touvron et al. (2023)"},{"why":"Provides the heterophilic Amazon-Ratings and Roman-Empire datasets used in the experiments.","marker":"Platonov et al. (2023)"},{"why":"Provides the WikiCS dataset used in the experiments.","marker":"Mernyei & Cangea (2020)"},{"why":"Gives an instruction-construction approach the paper mimics when serializing graphs into text.","marker":"Ye et al. (2023)"}],"fun_headline_variants":["Intermediate attention windows beat full and sparse for graph LLMs","Graph LLMs profit from partial link exposure in attention","Tuning attention horizon improves graph LLM performance","Middle-ground attention window optimal for graph LLM training","Partial connectivity beats extremes in graph attention"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The disruption experiments assume that a model genuinely using graph links will lose accuracy monotonically as the links are increasingly scrambled, and that GNNs behave this way; the paper asserts this without demonstrating it, and if the assumption is false, the conclusion that LLMs ignore connectivity is unsupported.","fun_headline_variants_meta":{"raw":{"variants":["Intermediate attention windows beat full and sparse for graph LLMs","Graph LLMs profit from partial link exposure in attention","Tuning attention horizon improves graph LLM performance","Middle-ground attention window optimal for graph LLM training","Partial connectivity beats extremes in graph attention"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00015,"raw_usage":{"total_tokens":1216,"prompt_tokens":985,"completion_tokens":231,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":158}},"tokens_in":601,"tokens_out":231,"duration_ms":3383,"temperature":1.0,"reasoning_tokens":158,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T01:01:51.466852+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same four disruption levels on a GNN baseline with identical node sampling: if the GNN's accuracy is flat across levels I and II on Wikics or Pubmed, the yardstick for \"ignoring topology\" fails. Alternatively, on a dataset with random node text but real links, if an LLM's accuracy drops at levels I or II, then LLMs do use connectivity, contradicting Finding 1.","supporting_citations":[{"cited_title":"The statistical metrics of each dataset are shown in the following Table","cited_arxiv_id":null,"evidence_quote":"Provides the heterophilic Amazon-Ratings and Roman-Empire datasets used in the experiments."}],"review_version":1}