pith. machine review for the scientific record. sign in

arxiv: 2605.03689 · v1 · submitted 2026-05-05 · 💻 cs.SE

Recognition: unknown

Deep Graph-Language Fusion for Structure-Aware Code Generation

Mert Tiftikci , Amir Molzam Sharifloo , Mira Mezini
Authors on Pith no claims yet

Pith reviewed 2026-05-07 15:47 UTC · model grok-4.3

classification 💻 cs.SE
keywords code generationgraph neural networkspre-trained language modelsstructure-aware code modelsdeep fusionabstract syntax treesdata-flow graphs
0
0 comments X

The pith

Fusing graph neural network features directly into language model intermediate layers improves code generation by preserving control flow and data dependencies.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that transformer-based language models for code generation fail to capture relational structure such as control flow and data dependencies because they treat input as token sequences. CGFuse addresses this by running a graph neural network on abstract syntax trees and data-flow graphs, then infusing the resulting features token-by-token into the model's hidden layers rather than only at the input or output. This deep integration is shown to yield measurable gains over prior extraction, retrieval, or prompt-based graph methods. A reader who accepts the premise would conclude that explicit structural fusion can make pre-trained models more faithful to code semantics without retraining from scratch.

Core claim

CGFuse combines a graph neural network with a pre-trained language model to enable token-level infusion of learned graph representations into the model's intermediate layers, thereby preserving fine-grained structural information from code graphs including abstract syntax trees and data-flow graphs and producing up to 10-16% BLEU and 6-11% CodeBLEU gains in generation tasks across multiple language models.

What carries the argument

CGFuse, the token-level graph-feature infusion mechanism that injects GNN outputs into PLM intermediate layers to retain relational attributes of code graphs.

If this is right

  • Code generation systems can retain control-flow and data-dependency information that sequential transformers normally discard.
  • The same deep-fusion pattern can be applied to other pre-trained models without requiring full retraining.
  • Performance improvements hold across multiple base language models, indicating the fusion is not tied to one architecture.
  • Tasks beyond generation, such as code repair or summarization, become candidates for the same structural infusion.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the layer-wise fusion works, similar structural signals could be added to models for non-code sequence tasks that have latent graph structure.
  • The approach implies that information loss in graph-to-sequence conversion is the dominant bottleneck rather than model capacity.
  • Future variants might test whether fusion depth or graph type can be tuned per task without changing the base language model.

Load-bearing premise

Infusing learned graph features directly into the intermediate layers of pre-trained language models captures structured and relational attributes of code without the information loss of prior extraction or prompt methods.

What would settle it

An ablation that adds the same graph features only to the input embeddings or final output instead of intermediate layers and measures whether the BLEU and CodeBLEU gains disappear.

Figures

Figures reproduced from arXiv: 2605.03689 by Amir Molzam Sharifloo, Mert Tiftikci, Mira Mezini.

Figure 1
Figure 1. Figure 1: A sample Java snippet with documentation (a), code (b), and its augmented code graph (c). view at source ↗
Figure 2
Figure 2. Figure 2: Fusion of GNN features into a PLM layer. Only view at source ↗
read the original abstract

Pre-trained Language Models (PLMs) have the potential to transform software development tasks. However, despite significant advances, current PLMs struggle to capture the structured and relational attributes of code, such as control flow and data dependencies. This limitation is rooted in an architectural mismatch: whereas code structure is best represented by graphs, transformer-based LLMs process input as sequential token patterns and therefore lack explicit structural awareness. While recent research has explored integrating graph-based code representations using techniques like graph feature extraction, retrieval-augmented generation, and prompt engineering, existing approaches suffer from information loss during dense feature extraction or prompt encoding; notably, the potential of deep, token-level fusion of graph features within model internals has not been systematically explored. In this paper, we initiate such an exploration by introducing CGFuse, a novel framework that enables token-level integration of graph-derived representations by infusing learned graph features directly into the intermediate layers of pre-trained language models. CGFuse combines a graph neural network (GNN) with a language model to explicitly preserve and exploit fine-grained structural information from code graphs, including abstract syntax trees and data-flow graphs. We systematically evaluate CGFuse across multiple LLMs, demonstrating up to 10-16% BLEU and 6-11% CodeBLEU improvements in code generation performance. These results highlight the potential of deep graph-PLM integration to advance the field toward more robust, capable AI-driven software development.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper introduces CGFuse, a novel framework that fuses a graph neural network (GNN) with pre-trained language models (PLMs) for code generation. It enables deep, token-level infusion of learned graph features derived from abstract syntax trees (ASTs) and data-flow graphs directly into the intermediate layers of the PLMs. The central claim is that this approach explicitly preserves fine-grained structural and relational information (control flow, data dependencies) that sequential transformers otherwise miss, outperforming prior methods based on feature extraction, retrieval, or prompting. Empirical results across multiple LLMs are reported as up to 10-16% BLEU and 6-11% CodeBLEU gains.

Significance. If the reported gains prove robust under rigorous controls, the work would meaningfully advance structure-aware code generation by demonstrating the viability of deep internal fusion rather than shallow external integration. This could influence future PLM architectures for code and related structured domains. The paper's strength lies in its systematic exploration of fusion points, though significance hinges on the quality and transparency of the experimental evidence.

major comments (2)
  1. [§4] §4 (Evaluation protocol): The abstract and manuscript claim consistent gains across multiple LLMs but provide no information on the specific datasets used, the choice of baselines, number of runs, statistical significance tests, or controls for confounding variables such as model scale and training details. These omissions make it impossible to assess whether the 10-16% BLEU deltas are attributable to the token-level fusion design or to other factors.
  2. [§3.2] §3.2 (Fusion mechanism): The central architectural claim that direct infusion into intermediate layers avoids the information loss of prior extraction/prompt methods is presented without supporting ablations on fusion depth, GNN layer count, or quantitative measures of structural information retention (e.g., graph reconstruction accuracy or dependency coverage). This leaves the weakest assumption untested.
minor comments (2)
  1. [Abstract] The abstract would be clearer if it named the concrete code generation tasks (e.g., function-level completion, full program synthesis) and the exact LLMs evaluated.
  2. [§3] Notation for the infusion operation (how graph embeddings are aligned to token positions) should be formalized with an equation in §3 to aid reproducibility.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We appreciate the referee's thorough review and constructive feedback on our manuscript. We address each of the major comments below and outline the revisions we plan to make to strengthen the paper.

read point-by-point responses

  1. Referee: [§4] §4 (Evaluation protocol): The abstract and manuscript claim consistent gains across multiple LLMs but provide no information on the specific datasets used, the choice of baselines, number of runs, statistical significance tests, or controls for confounding variables such as model scale and training details. These omissions make it impossible to assess whether the 10-16% BLEU deltas are attributable to the token-level fusion design or to other factors.

    Authors: We agree with the referee that the evaluation protocol requires more detailed reporting to allow proper assessment of the results. The manuscript in Section 4 describes the overall setup but does not provide exhaustive information on datasets, baselines, run counts, statistical tests, or confounding controls. We will revise the paper to include these details explicitly, ensuring that the reported gains can be properly attributed to the proposed token-level fusion approach. revision: yes

  2. Referee: [§3.2] §3.2 (Fusion mechanism): The central architectural claim that direct infusion into intermediate layers avoids the information loss of prior extraction/prompt methods is presented without supporting ablations on fusion depth, GNN layer count, or quantitative measures of structural information retention (e.g., graph reconstruction accuracy or dependency coverage). This leaves the weakest assumption untested.

    Authors: We agree that additional ablations and quantitative measures are needed to support the architectural claims. Section 3.2 introduces the fusion mechanism but lacks the suggested ablations on fusion depth, GNN layers, and structural retention metrics. We will incorporate these analyses in the revised manuscript to test the assumption that deep infusion avoids information loss. revision: yes

Circularity Check

0 steps flagged

No significant circularity; empirical architecture evaluation

full rationale

The manuscript proposes CGFuse as a novel GNN-PLM fusion architecture for token-level graph feature infusion and supports its claims solely through empirical evaluation on code generation benchmarks, reporting BLEU and CodeBLEU gains. No equations, derivations, or load-bearing steps reduce by construction to fitted inputs, self-definitions, or self-citation chains; the architecture description and performance deltas are presented as outcomes of the design choice rather than tautological restatements of inputs. The work is self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 1 invented entities

The claim rests on the domain assumption that graphs best represent code structure and that deep internal fusion avoids information loss; no free parameters or external benchmarks are specified in the abstract.

axioms (2)
  • domain assumption Code structure is best represented by graphs such as abstract syntax trees and data-flow graphs rather than sequential tokens
    Explicitly stated as the root limitation of current PLMs in the abstract.
  • ad hoc to paper Deep token-level infusion of graph features into intermediate model layers preserves fine-grained structural information
    Central premise of the CGFuse method described in the abstract.
invented entities (1)
  • CGFuse no independent evidence
    purpose: Framework for token-level integration of graph-derived representations into pre-trained language models
    Newly proposed architecture combining GNN and LM components.

pith-pipeline@v0.9.0 · 5557 in / 1470 out tokens · 64086 ms · 2026-05-07T15:47:14.507071+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

27 extracted references · 19 canonical work pages · 6 internal anchors

  1. [1]

    Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Uni- fied Pre-training for Program Understanding and Generation. InProceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. Association for Computational Linguistics, Online, 2655–2668. doi:10.186...

    work page doi:10.18653/v1/2021.naacl-main.211 2021

  2. [2]

    Abhinav Anand, Shweta Verma, Krishna Narasimhan, and Mira Mezini. 2024. A Critical Study of What Code-LLMs (Do Not) Learn. InFindings of the Association for Computational Linguistics ACL 2024. Association for Computational Linguis- tics, Bangkok, Thailand and virtual meeting, 15869–15889. doi:10.18653/v1/2024. findings-acl.939

    work page doi:10.18653/v1/2024 2024

  3. [3]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models.arXiv preprint arXiv:2108.07732(2021)

    work page internal anchor Pith review arXiv 2021

  4. [4]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    work page Pith review doi:10.48550/arxiv.2107.03374 2021

  5. [5]

    Nuo Chen, Qiushi Sun, Jianing Wang, Xiang Li, and Ming Gao. 2023. Pass-Tuning: Towards Structure-Aware Parameter-Efficient Tuning for Code Representation Learning. InFindings of the Association for Computational Linguistics: EMNLP

    2023

  6. [6]

    Association for Computational Linguistics, Singapore, 577–591. doi:10. 18653/v1/2023.findings-emnlp.42

    2023

  7. [7]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. CoRRabs/1810.04805 (2018). arXiv:1810.04805 http://arxiv.org/abs/1810.04805

    work page internal anchor Pith review arXiv 2018

  8. [8]

    Kounianhua Du, Jizheng Chen, Renting Rui, Huacan Chai, Lingyue Fu, Wei Xia, Yasheng Wang, Ruiming Tang, Yong Yu, and Weinan Zhang. 2025. CodeGRAG: Bridging the Gap between Natural Language and Programming Language via Graphical Retrieval Augmented Generation. arXiv:2405.02355 [cs] doi:10.48550/ arXiv.2405.02355

    work page arXiv 2025

  9. [9]

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. 2020. Codebert: A pre-trained model for programming and natural languages.arXiv preprint arXiv:2002.08155 (2020)

    work page internal anchor Pith review arXiv 2020

  10. [10]

    Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022. UniXcoder: Unified Cross-Modal Pre-training for Code Representation. InPro- ceedings of the 60th Annual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers). Association for Computational Linguistics, Dublin, Ireland, 7212–7225. doi:10.18653/v1/2022.acl...

    work page doi:10.18653/v1/2022.acl-long.499 2022

  11. [11]

    Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. 2020. GraphCodeBERT: Pre-training Code Representations with Data Flow. InInternational Conference on Learning Representations

    2020

  12. [12]

    Hamilton, Rex Ying, and Jure Leskovec

    William L. Hamilton, Rex Ying, and Jure Leskovec. 2017. Inductive Representation Learning on Large Graphs. InProceedings of the 31st International Conference on Neural Information Processing Systems (NIPS’17). Curran Associates Inc., Red Hook, NY, USA, 1025–1035

    2017

  13. [13]

    Donald E Knuth. 1968. Semantics of context-free languages.Mathematical systems theory2, 2 (1968), 127–145

    1968

  14. [14]

    Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2019. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension.CoRRabs/1910.13461 (2019). arXiv:1910.13461 http://arxiv.org/abs/1910.13461

    work page internal anchor Pith review arXiv 2019

  15. [15]

    Wei Liu, Ailun Yu, Daoguang Zan, Bo Shen, Wei Zhang, Haiyan Zhao, Zhi Jin, and Qianxiang Wang. 2024. GraphCoder: Enhancing Repository-Level Code Completion via Coarse-to-fine Retrieval Based on Code Context Graph. InPro- ceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (ASE ’24). Association for Computing Machinery,...

    work page doi:10.1145/3691620.3695054 2024

  16. [16]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach.CoRRabs/1907.11692 (2019). arXiv:1907.11692 http://arxiv.org/abs/1907.11692

    work page internal anchor Pith review arXiv 2019

  17. [17]

    Wei Ma, Shangqing Liu, Mengjie Zhao, Xiaofei Xie, Wenhang Wang, Qiang Hu, Jie Zhang, and Yang Liu. 2024. Unveiling Code Pre-Trained Models: Investigating Syntax and Semantics Capacities.ACM Trans. Softw. Eng. Methodol.33, 7, Article 169 (Aug. 2024), 29 pages. doi:10.1145/3664606

    work page doi:10.1145/3664606 2024

  18. [18]

    Yingwei Ma, Rongyu Cao, Yongchang Cao, Yue Zhang, Jue Chen, Yibo Liu, Yuchen Liu, Binhua Li, Fei Huang, and Yongbin Li. 2025. SWE-GPT: A Process-Centric Language Model for Automated Software Improvement.Proceedings of the ACM on Software Engineering2, ISSTA (June 2025), 2362–2383. doi:10.1145/3728981

    work page doi:10.1145/3728981 2025

  19. [19]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2001. BLEU: A Method for Automatic Evaluation of Machine Translation. InProceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL ’02. Association for Computational Linguistics, Philadelphia, Pennsylvania, 311. doi:10.3115/1073083.1073135

    work page doi:10.3115/1073083.1073135 2001

  20. [20]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.Journal of Machine Learning Research21, 140 (2020), 1–67

    2020

  21. [21]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer.The Journal of Machine Learning Research21, 1 (2020), 5485–5551

    2020

  22. [22]

    Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundare- san, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. CodeBLEU: A Method for Automatic Evaluation of Code Synthesis. arXiv:2009.10297 [cs]

    work page internal anchor Pith review arXiv 2020

  23. [23]

    Hongyuan Tao, Ying Zhang, Zhenhao Tang, Hongen Peng, Xukun Zhu, Bingchang Liu, Yingguang Yang, Ziyin Zhang, Zhaogui Xu, Haipeng Zhang, Linchao Zhu, Rui Wang, Hang Yu, Jianguo Li, and Peng Di. 2025. Code Graph Model (CGM): A Graph-Integrated Large Language Model for Repository-Level Software Engineering Tasks. arXiv:2505.16901 [cs] doi:10.48550/arXiv.2505.16901

    work page doi:10.48550/arxiv.2505.16901 2025

  24. [24]

    Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. 2021. CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Under- standing and Generation. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguis- tics, Online and Punta Cana, Dominican Republic, 8696–87...

    work page doi:10.18653/v1/ 2021

  25. [25]

    Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2025. De- mystifying LLM-Based Software Engineering Agents.Proceedings of the ACM on Software Engineering2, FSE (June 2025), 801–824. doi:10.1145/3715754

    work page doi:10.1145/3715754 2025

  26. [26]

    Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. 2018. How Powerful Are Graph Neural Networks?. InInternational Conference on Learning Represen- tations

    2018

  27. [27]

    Ziyin Zhang, Hang Yu, Shijie Li, Peng Di, Jianguo Li, and Rui Wang. 2024. GALLa: Graph Aligned Large Language Models for Improved Source Code Understanding. arXiv:2409.04183 [cs]

    work page arXiv 2024