{"id":"5d85170b-9bb4-4e79-9e37-86361efdaf91","arxiv_id":"2501.00244","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A nine-task benchmark shows structural knowledge prompting gives LLMs coarse-grained understanding but not fine-grained entity precision, and simple MLP adapters outperform complex ones.","lead":"This paper introduces SUBARU, a benchmark with nine tasks that tests how well structural knowledge prompts let language models use knowledge graph information. The results show the approach works for coarse-grained reasoning but fails to capture fine-grained entity details, and simple adapters beat complex ones.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Main quantitative support for 'SKP excels at coarse-grained reasoning' relies on MC prompts that include textual relation names and option names; the w/o-text control in Appendix B.3 is partial and absent from Table 2.","rationale":"The reader's weakest-assumption is the right load-bearing concern: the MC tasks, especially TG-MC, include textual relation names and textual option names, so high accuracy may reflect parametric knowledge rather than the structural prompt. I partially agree rather than fully agree because Appendix B.3 already shows that for DistMult, RotatE, and R-GCN, accuracy remains well above chance even without text, so the leakage is not complete. The defect is that the control is limited to one task with the MLP adapter and is absent from the main tables. The negative half of the paper—EG DESC at 0.00 EM and the case study showing incorrect central entities—is robust evidence for the fine-grained-limitation claim. Thus the concern does not overturn the verdict; it supports keeping the conditional recommendation and requiring a systematic w/o-text control before Observation 2 and the coarse-grained half of the central claim are stated as general conclusions.","tokens_in":16596,"tokens_out":5884,"duration_ms":61945,"concrete_test":"Extend the Appendix B.3 control to the full main table: for every structural encoder x adapter cell, re-run TG-MC and SG-MC with the textual query relation and the textual option names removed or masked, keeping only the SKP token sequence, and report 3-seed means with the 25% random baseline. If accuracy for a cell collapses toward chance, that cell does not support Observation 2; if it remains well above baseline, it does. Adding this w/o-text column to Table 2 would settle whether SKP itself drives the coarse-grained MC results.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Observation 2 and the coarse-grained half of the central claim rest primarily on TG-MC and SG-MC accuracy. In the TG-MC template (Fig. 14), the query contains a textual relation name ('occupation') and the options are entity-name strings, so the frozen LLM can answer from parametric knowledge without using the SKP tokens. This conflicts with the protocol stated in Section 4.2.2, which says important textual information is removed so that tasks are solved 'using mainly the SKPs rather than the texts.' The authors' own control (Appendix B.3, Table 4) shows the effect: with text removed, TransE drops from 86.51 to 49.09 (near the 25% random baseline), R-GCN drops from 94.59 to 78.45, while DistMult (97.91 to 90.94) and RotatE (83.74 to 83.51) retain most of their accuracy. The leakage is therefore real but not uniform. Because Table 4 covers only TG-MC with the MLP adapter and is not included in the main results, the headline claim that SKP 'effectively provides coarse-grained information across different granularities and task types' is over-generalized as presented: it may hold for some encoder/adapter combinations and possibly for SG-MC, but the main tables do not cleanly demonstrate that the MC scores measure SKP rather than textual shortcuts.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SUBARU, a benchmark of nine tasks spanning three granularities (entity, triple, subgraph) and three difficulty levels (classification, multiple choice, description), and uses it to evaluate structural knowledge prompting (SKP) across four dimensions: granularity, transferability, scalability, and universality. The evaluation combines four structural encoders (TransE, DistMult, RotatE, R-GCN), four adapters (FC, MLP, MoE, Qformer), and a frozen Llama3-8B-Instruct, with additional experiments on other backbones. The authors report three main conclusions: simple MLP adapters are surprisingly effective; SKP excels at coarse-grained reasoning tasks (TG/SG MC) but struggles with fine-grained entity-level understanding; and SKP shows limited transferability and modest scalability. The central claim is that SKP effectively provides LLMs with coarse-grained information but cannot achieve precise factual awareness.","tokens_in":16912,"tokens_out":3658,"duration_ms":38319,"significance":"If the conclusions hold, the paper provides a useful empirical map of where current SKP methods succeed and fail, and the SUBARU benchmark could serve as a reusable testbed for future structural prompting research. The authors ship code and data, evaluate 16 SKP configurations, and include a case study that illustrates the coarse-vs-fine distinction. The paper is honest about its limitations (benchmark scale, LLM sizes, lack of mechanistic analysis). However, the central claim about coarse-grained reasoning is weakened by textual leakage in the MC prompts and by the absence of variance estimates, so the significance is conditional on correcting these issues.","major_comments":[{"comment":"The protocol states that \"important textual information\" is removed so that models solve tasks \"using mainly the SKPs rather than the texts,\" but the TG-MC template in Figure 14 includes a textual relation name (e.g., \"occupation\") and textual entity-name options. The authors' own control in Table 4 shows this matters: removing text drops TransE TG-MC accuracy from 86.51 to 49.09, near the 25% random baseline, and R-GCN from 94.59 to 78.45, while DistMult and RotatE are less affected. Since Observation 2 and the coarse-grained half of the central claim rely on TG-MC and SG-MC, the main results in Table 2 do not cleanly establish that SKP itself drives the strong MC performance. The w/o-text control should be extended to all MC tasks and adapter/encoder combinations, and the coarse-grained claim should be conditioned on the textual shortcut or removed from the headline.","section":"§4.2.2 and Appendix B.3 (Table 4)"},{"comment":"No error bars, confidence intervals, or multiple-seed results are reported anywhere. Observation 1 (\"the simple MLP architecture achieves the best performance on most tasks\") and the cross-adapter comparisons in Table 2 rely on single runs; some differences, such as TransE 91.01 vs R-GCN 44.46 on Entity-MC with MLP, are large, but others are small and could be noise. Similarly, the transferability and scalability analyses (Figures 4-6) are presented as qualitative plots without numerical support. The paper should report mean and standard deviation over at least three seeds (or otherwise justify that the evaluation is deterministic) before drawing relative-comparison conclusions.","section":"Table 2 and all experiments"},{"comment":"The claim that \"SKP excels in coarse-grained reasoning tasks\" conflates task granularity with the presence of textual cues. EG-MC requires selecting an entity type from an entity embedding alone, while TG-MC provides a textual relation name and SG-MC includes a subgraph with a missing core. The Appendix B.3 result for TransE shows that the TG-MC accuracy is largely attributable to text. The observation should be re-framed as an interaction between SKP and textual hints, or the w/o-text results should be integrated into the main comparison; otherwise the coarse-grained superiority claim is over-generalized.","section":"§5.2, Observation 2"},{"comment":"The transferability conclusion (\"SKP does not exhibit strong transferability\") and the scalability conclusion (\"an MLP with 3-4 layers is strong enough\") are based on figures that report no quantitative values, no variance, and no statistical tests. For a paper whose contribution is a systematic evaluation, these RQ2/RQ3 claims need to be backed by numbers (e.g., delta tables with significance tests) rather than visual inspection of line charts.","section":"§5.3.1 and §5.4"}],"minor_comments":[{"comment":"The sentence \"current approaches often this paradigm directly to specific downstream tasks\" is missing the verb \"apply\"; it should read \"often apply this paradigm directly.\"","section":"§3, sentence after Eq. (3)"},{"comment":"\"In this section, we we briefly introduce SUBARU\" contains a duplicated \"we.\"","section":"§4.2.2, first sentence"},{"comment":"The legend lists \"Entity MC\" twice; one of these should presumably be \"Subgraph MC\" to match the figure labels.","section":"Figure 2"},{"comment":"The text says \"As shown in Figure 3, we present a simple case\" but the case study is presented in Table 3; the cross-reference should be corrected.","section":"§5.6, text above Figure 3"},{"comment":"\"GPT-3.5-tubor\" is a typo for \"GPT-3.5-turbo.\"","section":"§5.1, Evaluation Protocol"},{"comment":"\"with a 3000 epoch training until coverage\" should likely read \"until convergence\" rather than \"coverage.\"","section":"Appendix B.1"},{"comment":"\"dominate in most of the colored cells\" is vague; please state the exact counts (e.g., \"MLP is the best adapter on 6 of 9 tasks\") to make the observation precise.","section":"§5.2, Observation 1"}],"recommendation":"major_revision","confidential_remarks":"The SUBARU benchmark and the four-RQ evaluation are a reasonable empirical contribution, and the authors' own limitations section is refreshingly candid. The main barrier to acceptance is the textual-leakage control being confined to Appendix B.3 and only for TG-MC with MLP; integrating it into the main results and qualifying Observation 2 would address the core correctness risk. The absence of variance reporting is a second issue that should be fixed by adding multiple seeds. I see this as a major-revision rather than a reject because the leakage is partial (DistMult and RotatE retain much of their accuracy without text) and the benchmark itself remains useful even if the coarse-grained claim is weakened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—quick take on arXiv:2501.00244. The paper's real contribution is SUBARU, a 9-task benchmark that crosses three granularities (entity/triple/subgraph) with three difficulty levels (CLS/MC/DESC). That resource plus the four-axis evaluation frame (granularity, transferability, scalability, universality) is genuinely useful. The best-supported finding is negative: SKP cannot deliver fine-grained entity-level facts. EG DESC exact-match is 0.00 for every encoder/adapter combination, and the case study shows models can't even name the central entity of a subgraph. That result looks robust.\n\nThe softer spot is the positive half of the headline. Observation 2 says SKP 'excels in coarse-grained reasoning tasks,' but the TG-MC template in Figure 14 includes the textual relation name ('occupation') and entity-name options, so a frozen LLM can answer from parametric knowledge without touching the structural tokens. The paper's own control (Appendix B.3) shows the effect is real but uneven: with text removed, TransE TG-MC drops from 86.51 to 49.09 near random, R-GCN drops from 94.59 to 78.45, while DistMult and RotatE largely retain accuracy. So structural signal is present for some encoders, but Table 2's MC numbers do not cleanly separate it from text. The protocol in Section 4.2.2 says text is removed—it isn't. That overstates the evidence for the coarse-grained claim.\n\nOther issues are minor. No error bars or multiple seeds, which matters for a paper that ranks adapters, though the MLP-vs-complex-adapter gaps are large enough that the ranking is probably real. Using GPT-3.5-turbo to both generate golden subgraph descriptions and score predicted ones couples the reference and the judge. Both are fixable.\n\nWho benefits: anyone building SKP-style knowledge injection into LLMs. The benchmark is a real asset, and the negative fine-grained result is a useful correction to the field's optimism. The coarse-grained claim needs a rewrite and a w/o-text control in the main analysis. I'd send this to review—the benchmark deserves referee time—but I'd want the leakage fixed and variance reported before publication.","headline":"Worth engaging: SUBARU is a genuinely useful benchmark and the fine-grained negative result is solid, but the coarse-grained success claim is partly confounded by textual leakage in the MC tasks.","tokens_in":17464,"tokens_out":3479,"would_cite":true,"duration_ms":32778,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Structural knowledge prompting reliably delivers coarse-grained facts to LLMs across task types, but it fails at fine-grained entity identification, according to a systematic nine-task benchmark.","keywords":["structural knowledge prompting","knowledge graph","large language models","adapter","benchmark","granularity","knowledge graph completion","multi-task transfer"],"falsifier":"Run the triple and subgraph multiple-choice tasks with all textual names removed from both query and options, keeping only structural embeddings and arbitrary option labels, and check whether accuracy stays above chance; the paper's own text-removal control on triple multiple-choice predicts it will not for several encoders. A clean version across all tasks would settle whether SKP itself, rather than the LLM's parametric knowledge, drives the coarse-grained results.","tokens_in":16432,"feed_emoji":"🧩","tokens_out":7081,"duration_ms":62536,"temperature":0.7,"pith_summary":"This paper asks whether structural knowledge prompting (SKP) — feeding a frozen LLM embeddings of entities, triples, or subgraphs through an adapter — is a genuinely generalizable way to improve factual accuracy. To answer it, the authors build SUBARU, a benchmark of nine tasks crossing three granularities (entity, triple, subgraph) with three difficulty levels (classification, multiple choice, description), and test 16 combinations of structural encoders and adapters. They find that SKP consistently helps on coarse-grained reasoning such as triple and subgraph multiple choice, but fails on fine-grained tasks like naming an entity from its embedding, where exact-match accuracy is zero. They also find that a simple MLP adapter beats more complex adapters on most tasks, and that performance peaks at three to four adapter layers. The paper's conclusion is that SKP is a useful coarse-grained knowledge channel but not yet a source of precise factual awareness.","feed_headline":"SKP aids coarse reasoning but fails at precise facts","feed_subtitle":"A new 9-task benchmark finds simple MLP adapters win and entity-level exact match stays at zero.","key_machinery":"The central machinery is the SKP pipeline itself: a pretrained structural encoder maps KG elements to embeddings, an adapter projects those embeddings into the LLM's token-embedding space, and the frozen LLM performs next-token prediction conditioned on the structural prompt plus a task instruction. The evaluation instrument is SUBARU, a 3x3 benchmark whose entity, triple, and subgraph granularities are crossed with classification, multiple-choice, and description levels, letting the authors separate granularity from difficulty. The benchmark also masks textual names in the prompt templates and includes a text-removal control, which together test whether the LLM is actually using the structural tokens rather than the words in the query.","core_discovery":"On its own terms, the paper establishes a capability boundary for structural knowledge prompting: SKP can transmit coarse-grained relational and attribute information to LLMs at entity, triple, and subgraph granularities, but it cannot transmit fine-grained, precisely identifying facts. The 16-setting sweep shows the simple MLP adapter dominates, complex adapters such as Qformer and MoE do not help and often hurt on subgraph tasks, and the paradigm transfers across 7B–8B LLM backbones with only small variation. The benchmark's most telling result is the EG-DESC task, where no configuration can recover an entity's name from its structural embedding (0.00 exact match), and the case study in which every subgraph description misses the central entity. From this the paper concludes that future SKP design should add more prompt tokens aimed at activating precise, detailed information.","pith_inferences":["If the text-removal results generalize, a large share of the reported multiple-choice gains may come from the LLM's own parametric knowledge activated by the textual query, not from structural tokens; the paper's coarse-grained conclusion would then describe the combined prompt rather than SKP alone.","A natural testable extension is to add entity-description or entity-name tokens to the SKP sequence; the paper's own closing recommendation predicts this should close part of the entity-description gap, and SUBARU makes that measurable.","The lack of transfer across tasks suggests the trained adapters may be memorizing task-specific mappings rather than learning a general structural semantics; layer-wise attention probing or zero-shot task switches could test this directly."],"forward_implications":["On triple and subgraph multiple-choice tasks, SKP lifts accuracy well above random choice, so the paradigm is a usable coarse-grained interface between knowledge graphs and LLMs.","For applications that need exact entity identification, such as entity linking or fact verification, current SKP provides almost nothing: entity-level description exact match is 0.00 across configurations.","Adapter complexity is not buying accuracy; a two-layer MLP is the strongest bridge, and Qformer or MoE can reduce performance on subgraph tasks.","Training on additional tasks or granularities does not reliably transfer to the target task, so SKP modules currently need task-specific training; the exception is induction to new entities in triple multiple choice, which works almost as well as seen entities.","Adapter depth follows an inverted-U curve, with three to four layer MLPs optimal, so scaling adapter capacity alone is not the path to better structural prompting."],"supporting_citations":[{"why":"Defines the graph-neural-prompting SKP recipe with a structural encoder, adapter, and frozen LLM that SUBARU evaluates.","marker":"(Tian et al., 2024)"},{"why":"Introduces structural-embedding prompting with a projection layer for knowledge graph completion, the origin of the FC adapter and triple-level task format.","marker":"(Zhang et al., 2024b)"},{"why":"Supplies the Qformer adapter design and the bridging-representation-gap idea used by SKP.","marker":"(Li et al., 2023)"},{"why":"Provides CoDeX, the knowledge graph from which all SUBARU entity, triple, and subgraph instances are sampled.","marker":"(Safavi and Koutra, 2020)"},{"why":"Llama3-8B-Instruct is the main frozen backbone for the 16-setting evaluation.","marker":"(Dubey et al., 2024)"},{"why":"TransE is one of the four structural encoders and the clearest case where text removal collapses triple multiple-choice accuracy.","marker":"(Bordes et al., 2013)"},{"why":"RotatE supplies one encoder and the self-adversarial training objective used to pretrain all structural encoders.","marker":"(Sun et al., 2019)"},{"why":"DistMult is another structural encoder in the sweep; it often drives the best triple multiple-choice results.","marker":"(Yang et al., 2015)"},{"why":"R-GCN is the graph-neural-network structural encoder used to test whether message passing changes SKP behavior.","marker":"(Schlichtkrull et al., 2018)"}],"fun_headline_variants":["SKP aids coarse reasoning but fails at precise facts","Structural knowledge prompting hits zero exact match on entity recall","Simple MLP adapter outperforms complex Qformer in SKP benchmarks","New 9-task benchmark tests generalization of structural knowledge prompts","SKP transfers across LLMs but cannot recover entity names precisely"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that strong multiple-choice accuracy reflects what the model learned from the structural prompt tokens, but the triple multiple-choice template includes a textual relation name and textual options, and the paper's own text-removal control shows TransE dropping from 86.51 to 49.09 accuracy when the text is removed, so the non-textual signal alone is much weaker.","fun_headline_variants_meta":{"raw":{"variants":["SKP aids coarse reasoning but fails at precise facts","Structural knowledge prompting hits zero exact match on entity recall","Simple MLP adapter outperforms complex Qformer in SKP benchmarks","New 9-task benchmark tests generalization of structural knowledge prompts","SKP transfers across LLMs but cannot recover entity names precisely"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000255,"raw_usage":{"total_tokens":1529,"prompt_tokens":858,"completion_tokens":671,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":474,"completion_tokens_details":{"reasoning_tokens":589}},"tokens_in":474,"tokens_out":671,"duration_ms":6809,"temperature":1.0,"reasoning_tokens":589,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:54:40.903048+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the triple and subgraph multiple-choice tasks with all textual names removed from both query and options, keeping only structural embeddings and arbitrary option labels, and check whether accuracy stays above chance; the paper's own text-removal control on triple multiple-choice predicts it will not for several encoders. A clean version across all tasks would settle whether SKP itself, rather than the LLM's parametric knowledge, drives the coarse-grained results.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides CoDeX, the knowledge graph from which all SUBARU entity, triple, and subgraph instances are sampled."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"TransE is one of the four structural encoders and the clearest case where text removal collapses triple multiple-choice accuracy."},{"cited_title":"Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling","cited_arxiv_id":null,"evidence_quote":"R-GCN is the graph-neural-network structural encoder used to test whether message passing changes SKP behavior."}],"review_version":1}