Pith. sign in

REVIEW 4 major objections 4 minor 77 references

A causality-based analysis framework quantifies the causal contribution of each intermediate output to multi-agent code generation correctness, and its feature-responsibility rankings enable targeted failure repair and token-cost pruning.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 05:26 UTC pith:VOP7KGEP

load-bearing objection A useful empirical framework for ranking intermediate features in multi-agent code generation, but the search pruning rests on an unproven monotonicity assumption and the causality framing is looser than claimed. the 4 major comments →

arxiv 2602.02138 v3 pith:VOP7KGEP submitted 2026-02-02 cs.SE cs.AI

CAM: A Causality-based Analysis Framework for Multi-Agent Code Generation Systems

Zongyi Lyu , Zhenlan Ji , Songqiang Chen , Liwen Wang , Yuheng Huang , Shuai Wang , Shing-Chi Cheung This is my paper
classification cs.SE cs.AI
keywords multi-agent systemscode generationcausal analysiscounterfactual interventionfeature importanceLLM agentssoftware testingfeature responsibility
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Multi-agent code generation systems produce many intermediate outputs, but which of those outputs actually determine whether the final code is correct has been opaque. The paper proposes CAM, a causality-based framework that decomposes intermediate outputs into a small set of semantic features, simulates realistic errors on each feature, and measures the causal effect of these errors on final correctness. The resulting feature-responsibility rankings agree strongly with human annotation (Kendall tau 0.76–0.91) and reveal that some features matter mainly in combination with others. These rankings support two practical applications: repairing failures by improving the top-ranked features achieves a 73.3% success rate, and pruning low-ranked features saves up to 33.6% of intermediate tokens without performance loss. If valid, the framework turns an otherwise opaque multi-agent pipeline into a measurable, optimization-ready account of what truly matters.

Core claim

The paper establishes that intermediate outputs of a MACGS can be systematically mapped to twelve semantic features organized into four functional categories, and that the causal importance of each feature can be quantified through counterfactual interventions. Using an LLM to generate realistic errors on features, the method identifies minimal feature combinations whose simultaneous intervention causes failure, and aggregates these into a Feature Responsibility (FR) ranking. The ranking matches human judgment across four datasets and three backend LLMs, uncovers context-dependent features (e.g., Program_Lang is important in combination with other features in 78.8% of cases), and demonstrate

What carries the argument

The central machinery is a three-part causal-measurement pipeline. First, a function-oriented categorization maps intermediate outputs to twelve feature variables arranged in a directed acyclic graph, using each feature's final value to avoid cycles from iterative refinement. Second, the influence set E(S) — the set of downstream features whose semantics change when a feature set S is intervened — and its collective approximation are used to greedily prioritize feature-combination search and to prune unpromising combinations. Third, Feature Responsibility FR(f_i) = Σ over important sets S containing f_i of (1/|S|)^2 aggregates the importance of each feature, weighting shorter (more decisive)

Load-bearing premise

The search prunes combinations under an unproven monotonicity: if intervening on a feature set S does not cause failure, then no length-ℓ combination drawn from S's influence set E(S) can cause failure either (Sec. 4.3, Algorithm 1, lines 19–20); if individually weak errors interact into a failure, this pruning biases the entire feature-responsibility ranking.

What would settle it

Run Algorithm 1 on a sample of problems with the influence-set pruning disabled, exhaustively testing all length-ℓ feature combinations up to the same execution budget, and compare the discovered important feature sets. If the unpruned search finds any important combination that the pruned search missed — particularly a combination whose features all lie in E(S) for some non-failing set S — the monotonicity assumption is false and the FR rankings are systematically biased.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Quality assurance for MACGS should incorporate cross-feature consistency checks, because many failures stem from interactions between individually plausible features rather than from any single flaw.
  • Hybrid multi-backend architectures — assigning different LLMs to different stages according to relative feature strengths — can yield measurable Pass@1 gains, especially on design-heavy tasks.
  • Failure repair can be prioritized by improving the top-3 FR-ranked features rather than exhaustively inspecting all intermediate outputs, raising the repair success rate from 27–41% to 73.3%.
  • Pruning 2–4 low-FR features in production can reduce intermediate token consumption by up to 33.6% with no performance loss (or even improvement), while aggressive pruning degrades performance.
  • Feature importance is not static: it shifts with dataset difficulty and domain, so resource allocation and optimization should be adapted to the target problem distribution.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The entire ranking inherits the fidelity of the LLM used for counterfactual intervention: errors that the intervention model cannot render as plausible will be invisible to the analysis, so the method's output is bounded by the intervention engine's realism.
  • The FR rankings are computed on a fixed problem set; transferring them to a new domain would require re-running the analysis, but the method itself is designed to be re-instantiated.
  • The context-dependent feature finding suggests a testable extension: automatically checking pairwise consistency of high-FR features (e.g., Program_Lang × Data_Struct) could serve as an inexpensive pre-submission filter for MACGS.
  • The pruning application hints at a cost-geometry trade-off: the fact that pruning a few features sometimes helps suggests some low-FR features add noise rather than information, a hypothesis that could be probed on other architectures.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes CAM, a framework that categorizes intermediate outputs of multi-agent code generation systems (MACGS) into 12 features across four categories, constructs a causal graph, and uses LLM-generated counterfactual interventions to find minimal feature combinations whose simultaneous corruption leads to failure. It then aggregates a feature responsibility (FR) score via Eq. (4) and validates the resulting ranking against human annotations (Kendall 0.76–0.91). The paper reports several downstream findings: context-dependent features, hybrid-backend Pass@1 gains up to 7.2%, causality-guided failure repair at 73.3% success, and up to 66.8% token reduction via feature pruning.

Significance. If the causal ranking is trustworthy, CAM is a useful and timely contribution: intermediate-output analysis in MACGS is currently manual or coarse-grained, and the paper provides a concrete feature taxonomy, an intervention-generation method, and two practical applications with quantitative benefits. The human-agreement study is a genuine strength, and the hybrid-backend and pruning results are actionable. However, the central claim of formal causality is not actually instantiated, and the search algorithm relies on an unproven pruning property. These issues affect the correctness of the rankings and therefore of the empirical findings, so the paper cannot be accepted in its current form; the framework and data collection are substantial enough that a major revision is warranted.

major comments (4)
  1. [Sec. 2.2 and Sec. 4.3 (Def. 3, Eq. 4)] The paper invokes Halpern actual causality, but the method does not implement Definitions 2–3. Interventions are applied only to problems that originally succeed, so AC1 (actuality) is never satisfied for a failure outcome; the discovered combinations are failure-inducing counterfactuals, not actual causes of observed failures. Moreover, FR in Eq. (4) is a squared-inverse-length heuristic, not the responsibility 1/(1+|W|) from Eq. (1), and no contingency set W is computed. This is a substantive mismatch with the title and abstract claims. The authors should either instantiate the formal definitions or explicitly reframe CAM as intervention-based importance analysis rather than actual-causality analysis.
  2. [Sec. 4.3, Alg. 1 lines 19–20] The pruning step assumes an error-containment property: if intervening on S does not induce failure, then no length-ℓ combination drawn from E(S) can induce failure. This is not entailed by Eq. (2), because E(S) only records that downstream features have changed semantically after the S intervention; the specific corrupted values are generated by that intervention. A direct intervention on S' subset of E(S) uses independently generated errors, and there is no monotonicity guarantee that two individually weak errors cannot interact into a failure. In fact, the context-dependence finding in Sec. 7.1 presupposes such interactions. If the property fails, important combinations are silently discarded, biasing S_p, Eq. (4), Table 6, and all downstream applications. The pilot study cannot detect this because Table 4 validates only aggregate 12-feature rankings. Please provide a proof under stat
  3. [Sec. 6, Table 4 and Sec. 7.1, Table 6] The human-agreement study validates only the aggregate FR ranking over 12 features. The central new finding—context-dependent features, e.g., 78.8% of Program_Lang appearances occurring in combinations—is an interaction-level claim derived from the same search procedure that defines interactions. Because the search is incomplete (budget N, patience k, and the pruning in Alg. 1), the interaction distribution is not independently confirmed. The feature vocabulary is also partly constitutive: if importance is defined through minimal failure-inducing combinations, interaction effects are guaranteed by construction; the empirical content lies in which features interact and how often. This needs external validation, e.g., human annotation of a sample of discovered minimal combinations or a held-out comparison.
  4. [Sec. 5 Parameters and Sec. 9.2] FR rankings and all downstream results depend on several thresholds and design choices: θ in Eq. (2), L_max, k, N, and the weighting exponent in Eq. (4). Only k and L_max receive sensitivity analysis. In particular, θ determines the influence set and therefore the search and pruning behavior, and the exponent determines the FR aggregation; the stability of Tables 5–10 with respect to these choices should be reported. Without this, the reader cannot tell whether the main findings are robust or artifacts of parameter settings.
minor comments (4)
  1. [Abstract and Section 7/8] The abstract reports 73.6% repair success, 33.6% token reduction, and 7.3% Pass@1 improvement, while the introduction and tables report 73.3%, 66.8%, and 7.2%. Please harmonize these numbers.
  2. [Sec. 9.1] The generalizability claim for PairCoder and MapCoder is prose-only; a table mapping each system's intermediate outputs to the four categories would make the extensibility argument concrete.
  3. [Eq. (3)] The collective influence set uses union over previously computed subsets; if some subsets were pruned or never executed, its definition should state how missing E(S') values are handled (e.g., treated as empty).
  4. [Sec. 4.2] The manual inspection of 100 counterfactual interventions confirms semantic coherence, but not that the interventions represent realistic errors rather than merely plausible paraphrases; a small set of examples in the text would help the reader assess this.

Circularity Check

0 steps flagged

No significant circularity: FR rankings have independent human-validation support; the unproven Alg. 1 pruning property is a correctness risk, not a definitional reduction.

full rationale

CAM's derivation chain is: categorize intermediate outputs into 12 features, build a causal graph, generate LLM counterfactual interventions, search for minimal failure-inducing feature combinations (Alg. 1), and aggregate feature responsibility via Eq. 4. None of these steps defines importance in terms of the final FR ranking: FR is an aggregation of independently executed interventions, not a fitted parameter renamed as a result. The central claim—that FR rankings agree with human judgment—is checked against external manual annotations (Table 4, Kendall 0.76–0.91), so the ranking is not equivalent to its inputs by construction. The repair, pruning, and hybrid-backend results are in-sample demonstrations rather than out-of-sample predictions, which limits generalization claims, but they are not circular: the reported improvements (73.3% repair success, up to 66.8% token reduction, 7.2% Pass@1) could have failed and are not forced by the definitions. The main formal weakness is in Sec. 4.3 / Alg. 1 lines 19–20, where an unproven error-containment/monotonicity property is used to prune combinations drawn from the influence set E(S) (Eq. 2). This is an omitted proof and a correctness risk that could bias rankings if the property fails; however, it is not a definitional circularity, because the claim that a combination from E(S) is 'a less comprehensive perturbation' is an additional assumption and is not entailed by the definition of E(S). Self-citations such as [10] and [37] are motivational/contextual and are not load-bearing for the FR results, which have independent human-validation support. Verdict: no significant circularity.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 1 invented entities

The framework rests on a small number of hand-set thresholds (θ, k, L_max, N), a heuristic FR weighting, and several domain assumptions about error realism and graph acyclicity. The central claim does not reduce to a fitted equation, but the ranked output is sensitive to these choices, which are only partially justified.

free parameters (6)
  • Similarity threshold θ (Eq. 2) = 0.5
    Hand-set following prior similarity literature; determines which downstream features count as 'influenced' and directly controls influence-set sizes and pruning.
  • Maximum combination length L_max = 5
    Hand-set; later justified by showing length-5 contributes only 0.2% of FR, but the choice itself shapes the FR scores used in all findings.
  • Early-stop patience k = 10
    Hand-set; Table 12 shows a plateau for k=10-20, but the exact choice affects how many combinations are explored and therefore the FR ranking.
  • MACGS execution budget N = 100
    Per-problem query budget; caps the search and is a practical compromise rather than a value derived from the data.
  • FR weighting exponent (1/|S|)^2 = 2
    Chosen to amplify short combinations, citing Shepard; the final feature ranking depends on this nonlinear weighting.
  • Semantic similarity model = Sentence-Transformer
    Choice of embedding model and θ jointly determine influence sets; no robustness analysis is given across embedding models.
axioms (5)
  • domain assumption Causal graph can be made acyclic by using final feature values and ignoring mid-stage iterative refinements.
    Sec. 4.1 resolves MetaGPT's circular refinement loop by fiat; if iterative corrections carry causal weight, the graph is misspecified.
  • domain assumption LLM-generated counterfactual values are realistic, semantically coherent errors.
    Sec. 4.2 validates only coherence (99/100 samples), not that the errors resemble real failure modes encountered in practice.
  • ad hoc to paper Influence-set pruning property: if S does not cause failure, then combinations of features within E(S) cannot cause failure.
    Alg. 1 lines 19-20 assume an unproven error-containment/monotonicity property; a violation would directly bias FR rankings.
  • domain assumption Feature categorization (12 fields) is functional and generalizes across MACGS.
    The taxonomy is derived from MetaGPT; extension to Self-Collab/PairCoder/MapCoder is argued by mapping, with full instantiation shown only for Self-Collab.
  • domain assumption Kendall correlation with human annotation is a valid ground truth for causal importance.
    Human annotators may rely on surface plausibility; the pilot protocol in Sec. 6 is incompletely specified (failures vs. successes), so this ground truth is weakly established.
invented entities (1)
  • 12 semantic feature variables (e.g., Req_Stat, Data_Struct, Program_Lang) no independent evidence
    purpose: To decompose heterogeneous intermediate outputs into intervenable units for causal analysis.
    A new analytical abstraction introduced by the paper; its validity rests on the reported human-agreement results and downstream applications, but it has no independently measured external handle.

pith-pipeline@v1.3.0-alltime-deepseek · 25915 in / 12780 out tokens · 118182 ms · 2026-08-03T05:26:08.159260+00:00 · methodology

0 comments
read the original abstract

Despite the remarkable success that Multi-Agent Code Generation Systems (MACGS) have achieved, the inherent complexity of multi-agent architectures produces substantial volumes of intermediate outputs. To date, the individual importance of these intermediate outputs to the system correctness remains opaque, which impedes targeted optimization of MACGS designs. To address this challenge, we propose CAM, the first \textbf{C}ausality-based \textbf{A}nalysis framework for \textbf{M}ACGS that systematically quantifies the contribution of different intermediate features for system correctness. By comprehensively categorizing intermediate outputs and systematically simulating realistic errors on intermediate features, we identify the important features for system correctness and aggregate their importance rankings. We conduct extensive empirical analysis on the identified importance rankings. Our analysis reveals intriguing findings: first, we uncover context-dependent features\textemdash features whose importance emerges mainly through interactions with other features, revealing that quality assurance for MACGS should incorporate cross-feature consistency checks; second, we reveal that hybrid backend MACGS with different backend LLMs assigned according to their relative strength achieves up to 7.3\% Pass@1 improvement, underscoring hybrid architectures as a promising direction for future MACGS design. We further demonstrate CAM's practical utility through two applications: (1) failure repair which achieves a 73.6\% success rate by optimizing top-3 importance-ranked features and (2) feature pruning that reduces up to 33.6\% intermediate token consumption while maintaining generation performance. Our work provides actionable insights for MACGS design and deployment, establishing causality analysis as a powerful approach for understanding and improving MACGS.

Figures

Figures reproduced from arXiv: 2602.02138 by Liwen Wang, Shing-Chi Cheung, Shuai Wang, Songqiang Chen, Yuheng Huang, Zhenlan Ji, Zongyi Lyu.

Figure 1
Figure 1. Figure 1: Illustration of causality. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Feature responsibility (FR) distribution across different LLMs on CoderEval. [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Feature responsibility (FR) distribution across different datasets on Deepseek. [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Results of Self-Collab. First, as illustrated in [PITH_FULL_IMAGE:figures/full_fig_p017_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

77 extracted references · 14 linked inside Pith

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report.arXiv preprint arXiv:2303.08774 (2023)

    Pith/arXiv arXiv 2023

  2. [2]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models.arXiv preprint arXiv:2108.07732 (2021)

    Pith/arXiv arXiv 2021

  3. [3]

    Christel Baier, Clemens Dubslaff, Florian Funke, Simon Jantsch, Rupak Majumdar, Jakob Piribauer, and Robin Ziemek

  4. [4]

    2022.Team roles at work

    R Meredith Belbin and Victoria Brown. 2022.Team roles at work. Routledge

    2022

  5. [5]

    Guangyao Chen, Siwei Dong, Yu Shu, Ge Zhang, Jaward Sesay, Börje Karlsson, Jie Fu, and Yemin Shi. 2024. AutoAgents: a framework for automatic agent generation. InProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence. 22–30

    2024

  6. [6]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374(2021)

    Pith/arXiv arXiv 2021

  7. [7]

    Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi-Hsin Hung, Chen Qian, et al. 2023. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors. InThe Twelfth International Conference on Learning Representations

    2023

  8. [8]

    Philipp Chrszon, Clemens Dubslaff, Sascha Klüppelholz, and Christel Baier. 2018. ProFeat: feature-oriented engineering for family-based probabilistic model checking.Formal Aspects of Computing30, 1 (2018), 45–75

    2018

  9. [9]

    Jacob Cohen. 1960. A coefficient of agreement for nominal scales.Educational and psychological measurement20, 1 (1960), 37–46

    1960

  10. [10]

    Yanbo Dai, Zhenlan Ji, Zongjie Li, Kuan Li, and Shuai Wang. 2025. Disabling Self-Correction in Retrieval-Augmented Generation via Stealthy Retriever Poisoning.arXiv preprint arXiv:2508.20083(2025)

    Pith/arXiv arXiv 2025

  11. [11]

    Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. 2024. Self-collaboration code generation via chatgpt.ACM Transactions on Software Engineering and Methodology33, 7 (2024), 1–38

    2024

  12. [12]

    Clemens Dubslaff, Kallistos Weis, Christel Baier, and Sven Apel. 2022. Causality in Configurable Software Systems. In Proceedings of the 44th International Conference on Software Engineering (ICSE). ACM, 325–337. doi:10.1145/3510003. 3510200

    doi:10.1145/3510003 2022

  13. [13]

    Ábel Elekes, Martin Schäler, and Klemens Böhm. 2017. On the various semantics of similarity in word embedding models. In2017 ACM/IEEE Joint Conference on Digital Libraries (JCDL). IEEE, 1–10

    2017

  14. [14]

    Sarah Fakhoury, Aaditya Naik, Georgios Sakkas, Saikat Chakraborty, and Shuvendu K Lahiri. 2024. Llm-based test- driven interactive code generation: User study and empirical evaluation.IEEE Transactions on Software Engineering (2024)

    2024

  15. [15]

    Sarah Fakhoury, Aaditya Naik, Georgios Sakkas, Saikat Chakraborty, Madan Musuvathi, and Shuvendu Lahiri. 2024. Exploring the effectiveness of llm based test-driven interactive code generation: User study and empirical evaluation. InProceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings. 390–391

    2024

  16. [16]

    Anna Fariha, Suman Nath, and Alexandra Meliou. 2020. Causality-guided adaptive interventional debugging. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data. 431–446

    2020

  17. [17]

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. 2020. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. InFindings of the Association for Computational Linguistics: EMNLP 2020. 1536–1547

    2020

  18. [18]

    Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022. UniXcoder: Unified Cross-Modal Pre-training for Code Representation. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 7212–7225

    2022

  19. [19]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming–The Rise of Code Intelligence. arXiv preprint arXiv:2401.14196(2024)

    Pith/arXiv arXiv 2024

  20. [20]

    Joseph Y. Halpern. 2016.Actual Causality. MIT Press

    2016

  21. [21]

    Junda He, Christoph Treude, and David Lo. 2024. LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Ahead.ACM Transactions on Software Engineering and Methodology(2024)

    2024

  22. [22]

    Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber

  23. [23]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. 2024. Qwen2. 5-coder technical report.arXiv preprint arXiv:2409.12186(2024)

    Pith/arXiv arXiv 2024

  24. [24]

    Md Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez. 2024. MapCoder: Multi-Agent Code Generation for Competitive Problem Solving. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 4912–4944

    2024

  25. [25]

    Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024. A survey on large language models for code generation.arXiv preprint arXiv:2406.00515(2024)

    Pith/arXiv arXiv 2024

  26. [26]

    Zhijing Jin, Yuen Chen, Felix Leeb, Luigi Gresele, Ojasv Kamal, Zhiheng Lyu, Kevin Blin, Fernando Gonzalez Adauto, Max Kleiman-Weiner, Mrinmaya Sachan, et al. 2023. Cladder: Assessing causal reasoning in language models.Advances in Neural Information Processing Systems36 (2023), 31038–31065

    2023

  27. [27]

    Sathvik Joel, Jie Wu, and Fatemeh Fard. 2024. A survey on llm-based code generation for low-resource and domain- specific programming languages.ACM Transactions on Software Engineering and Methodology(2024)

    2024

  28. [28]

    Brittany Johnson, Yuriy Brun, and Alexandra Meliou. 2020. Causal testing: Understanding defects’ root causes. In Proceedings of the ACM/IEEE 42nd international conference on software engineering. 87–99

    2020

  29. [29]

    Maurice G Kendall. 1938. A new measure of rank correlation.Biometrika30, 1-2 (1938), 81–93

    1938

  30. [30]

    Emre Kiciman, Robert Ness, Amit Sharma, and Chenhao Tan. 2023. Causal reasoning and large language models: Opening a new frontier for causality.Transactions on Machine Learning Research(2023)

    2023

  31. [31]

    J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data.biometrics (1977), 159–174

    1977

  32. [32]

    Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. Camel: Communicative agents for" mind" exploration of large language model society.Advances in Neural Information Processing Systems36 (2023), 51991–52008

    2023

  33. [33]

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022. Competition-level code generation with alphacode.Science378, 6624 (2022), 1092–1097

    2022

  34. [34]

    Fulin Lin, Shaowen Chen, Ruishan Fang, Hongwei Wang, and Tao Lin. 2025. Stop wasting your tokens: Towards efficient runtime multi-agent systems.arXiv preprint arXiv:2510.26585(2025)

    arXiv 2025

  35. [35]

    Feng Lin, Dong Jae Kim, and Tse-Hsun Chen. 2025. SOEN-101: Code Generation by Emulating Software Process Models Using Large Language Model Agents. In2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 1527–1539

    2025

  36. [36]

    Junwei Liu, Kaixin Wang, Yixuan Chen, Xin Peng, Zhenpeng Chen, Lingming Zhang, and Yiling Lou. 2024. Large language model-based agents for software engineering: A survey.arXiv preprint arXiv:2409.02977(2024)

    Pith/arXiv arXiv 2024

  37. [37]

    Zongyi Lyu, Songqiang Chen, Zhenlan Ji, Liwen Wang, Shuai Wang, Daoyuan Wu, Wenxuan Wang, and Shing-Chi Cheung. 2025. Testing and Enhancing Multi-Agent Systems for Robust Code Generation.arXiv preprint arXiv:2510.10460 (2025)

    arXiv 2025

  38. [38]

    M. A. Islam, M. E. Ali, and M. R. Parvez. 2025. CodeSim: Multi-Agent Code Generation and Problem Solving through Simulation-Driven Planning and Debugging. InFindings of the Association for Computational Linguistics: NAACL 2025. 5113–5139

    2025

  39. [39]

    Herong Meng and Dalal Alrajeh. 2025. Causal Detection in Linear Temporal Counterexamples. InProceedings of the 33rd ACM International Conference on the Foundations of Software Engineering. 1418–1423

    2025

  40. [40]

    Ari Morcos, Maithra Raghu, and Samy Bengio. 2018. Insights on representational similarity in neural networks with canonical correlation.Advances in neural information processing systems31 (2018)

    2018

  41. [41]

    Changan Niu, Chuanyi Li, Vincent Ng, Jidong Ge, Liguo Huang, and Bin Luo. 2022. Spt-code: Sequence-to-sequence pre-training for learning source code representations. InProceedings of the 44th international conference on software engineering. 2006–2018

    2022

  42. [42]

    Melissa Z Pan, Mert Cemri, Lakshya A Agrawal, Shuyi Yang, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Kannan Ramchandran, Dan Klein, et al. 2025. Why do multiagent systems fail?. InICLR 2025 Workshop on Building Trust in Language Models and Applications

    2025

  43. [43]

    2009.Causality

    Judea Pearl. 2009.Causality. Cambridge university press

    2009

  44. [44]

    Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. 2023. Communicative agents for software development.arXiv preprint arXiv:2307.079246, 3 (2023)

    Pith/arXiv arXiv 2023

  45. [45]

    Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, et al. 2024. Chatdev: Communicative agents for software development. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 15174–15186

    2024

  46. [46]

    Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. 2017. Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability.Advances in neural information processing systems CAM: A Causality-based Analysis Framework for Multi-Agent Code Generation Systems 21 30 (2017)

    2017

  47. [47]

    Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 3982–3992

    2019

  48. [48]

    Navid Rekabsaz, Mihai Lupu, and Allan Hanbury. 2017. Exploration of a threshold for similarity based on uncertainty in word embedding. InEuropean conference on information retrieval. Springer, 396–409

    2017

  49. [49]

    Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 4902–4912

    2020

  50. [50]

    Cedric Richter and Heike Wehrheim. 2022. Learning realistic mutations: Bug creation for neural bug detectors. In2022 IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE Computer Society, 162–173

    2022

  51. [51]

    Mohamad Salim, Jasmine Latendresse, SayedHassan Khatoonabadi, and Emad Shihab. 2026. Tokenomics: Quantifying Where Tokens Are Used in Agentic Software Engineering.arXiv preprint arXiv:2601.14470(2026)

    arXiv 2026

  52. [52]

    Donald Shepard. 1968. A two-dimensional interpolation function for irregularly-spaced data. InProceedings of the 1968 23rd ACM National Conference. ACM, 517–524

    1968

  53. [53]

    Norbert Siegmund, Sergiy S Kolesnikov, Christian Kästner, Sven Apel, Don Batory, Marko Rosenmüller, and Gunter Saake. 2012. Predicting performance via automated feature-interaction detection. In2012 34th International Conference on Software Engineering (ICSE). IEEE, 167–177

    2012

  54. [54]

    Norbert Siegmund, Marko Rosenmüller, Christian Kästner, Paolo G Giarrusso, Sven Apel, and Sergiy S Kolesnikov

  55. [55]

    Carolin Strobl, Anne-Laure Boulesteix, Thomas Kneib, Thomas Augustin, and Achim Zeileis. 2008. Conditional variable importance for random forests.BMC bioinformatics9, 1 (2008), 1–11

    2008

  56. [56]

    Nandan Thakur, Nils Reimers, Johannes Daxenberger, and Iryna Gurevych. 2021. Augmented SBERT: Data augmenta- tion method for improving bi-encoders for pairwise sentence scoring tasks. InProceedings of the 2021 conference of the North American chapter of the association for computational linguistics: Human language technologies. 296–310

    2021

  57. [57]

    Frank Tip, Jonathan Bell, and Max Schäfer. 2025. Llmorpheus: Mutation testing using large language models.IEEE Transactions on Software Engineering(2025)

    2025

  58. [58]

    Gladys Tyen, Hassan Mansoor, Victor Cărbune, Yuanzhu Peter Chen, and Tony Mak. 2024. LLMs cannot find reasoning errors, but can correct them given the error location. InFindings of the Association for Computational Linguistics: ACL

    2024

  59. [59]

    Alexander Von Rhein, Alexander Grebhahn, Sven Apel, Norbert Siegmund, Dirk Beyer, and Thorsten Berger. 2015. Presence-condition simplification in highly configurable systems. In2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, Vol. 1. IEEE, 178–188

    2015

  60. [60]

    Ningning Wang, Xavier Hu, Pai Liu, He Zhu, Yue Hou, Heyuan Huang, Shengyu Zhang, Jian Yang, Jiaheng Liu, Ge Zhang, et al. 2025. Efficient agents: Building effective agents while reducing cost.arXiv preprint arXiv:2508.02694 (2025)

    Pith/arXiv arXiv 2025

  61. [61]

    Qian Wang, Tianyu Wang, Zhenheng Tang, Qinbin Li, Nuo Chen, Jingsheng Liang, and Bingsheng He. 2025. MegaAgent: A large-scale autonomous LLM-based multi-agent system without predefined SOPs. InFindings of the Association for Computational Linguistics: ACL 2025. 4998–5036

    2025

  62. [62]

    Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, et al. 2025. Thoughts are all over the place: On the underthinking of o1-like llms.arXiv preprint arXiv:2501.18585 (2025)

    Pith/arXiv arXiv 2025

  63. [63]

    Yue Wang, Weishi Wang, Shafiq Joty, and Steven C. H. Hoi. 2021. CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 8696–8708

    2021

  64. [64]

    Kallistos Weis, Leopoldo Motta Teixeira, Clemens Dubslaff, and Sven Apel. 2024. Blackbox Observability of Features and Feature Interactions. InProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. 1120–1132

    2024

  65. [65]

    Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. 2024. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. Association for Computational Linguistics

    2024

  66. [66]

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering.Advances in Neural Information Processing Systems37 (2024), 50528–50652

    2024

  67. [67]

    Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Qianxiang Wang, and Tao Xie. 2024. Codereval: A benchmark of pragmatic code generation with generative pre-trained models. InProceedings 22 Zongyi, Zhenlan, Songqiang, Liwen, Yuheng, Shuai, and Shing-Chi of the 46th IEEE/ACM International Conference on Software Engineer...

    2024

  68. [68]

    Huan Zhang, Wei Cheng, Yuhan Wu, and Wei Hu. 2024. A Pair Programming Framework for Code Generation via Multi-Plan Exploration and Feedback-Driven Refinement. InProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. 1319–1331

    2024

  69. [69]

    Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. 2024. CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 13643–13658

    2024

  70. [70]

    Shaokun Zhang, Ming Yin, Jieyu Zhang, Jiale Liu, Zhiguang Han, Jingyang Zhang, Beibin Li, Chi Wang, Huazheng Wang, Yiran Chen, et al. 2025. Which agent causes task failures and when? on automated failure attribution of llm multi-agent systems.arXiv preprint arXiv:2505.00212(2025)

    Pith/arXiv arXiv 2025

  71. [71]

    Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. Autocoderover: Autonomous program improvement. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1592–1604

    2024

  72. [72]

    Dewu Zheng, Yanlin Wang, Ensheng Shi, Ruikai Zhang, Yuchi Ma, Hongyu Zhang, and Zibin Zheng. 2024. Towards more realistic evaluation of LLM-based code generation: an experimental study and beyond.arXiv preprint arXiv:2406.06918 (2024)

    Pith/arXiv arXiv 2024

  73. [73]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in neural information processing systems36 (2023), 46595–46623

    2023

  74. [74]

    Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. 2024. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence.arXiv preprint arXiv:2406.11931(2024)

    Pith/arXiv arXiv 2024

  75. [2013]

    Information and Software Technology55, 3 (2013), 491–507

    Scalable prediction of non-functional properties in software product lines: Footprint and memory consumption. Information and Software Technology55, 3 (2013), 491–507

    2013

  76. [2021]

    In48th International Colloquium on Automata, Languages, and Programming, ICALP 2021

    From Verification to Causality-Based Explications. In48th International Colloquium on Automata, Languages, and Programming, ICALP 2021. 1–1

    2021

  77. [2024]

    InThe Twelfth International 20 Zongyi, Zhenlan, Songqiang, Liwen, Yuheng, Shuai, and Shing-Chi Conference on Learning Representations

    MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework. InThe Twelfth International 20 Zongyi, Zhenlan, Songqiang, Liwen, Yuheng, Shuai, and Shing-Chi Conference on Learning Representations