Pith. sign in

REVIEW 3 major objections 4 minor 33 references

ECLAIR: A Causally-Grounded AI Framework for Scientific Discovery in Empirical Software Engineering

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ECLAIR embeds LLMs in the causal scientific method for software engineering and finds that instruction-style, longer, and signature-augmented prompts each cause small reductions in code-generation accuracy for two 7B models.

desk verdict A useful methodology framework whose empirical case study is undermined by post-treatment conditioning; worth peer review but not the current ATE claims. read the letter →

arxiv 2608.02323 v2 pith:GXPGRJ73 submitted 2026-08-03 cs.SE

Alejandro Velasco , Daniel Rodriguez-Cardenas , Dipin Khati , David N. Palacio , Denys Poshyvanyk This is my paper
classification cs.SE
keywords causalinferencelargelanguagemodelssoftwareengineeringscientificmethodstructuralpromptcodegenerationhuman-in-the-loop
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes ECLAIR, an eight-phase, human-in-the-loop framework that treats large language models as scientific agents operating under Pearl's theory of causal inference, and applies it to empirical software engineering (SE). ECLAIR is claimed to be the first causally grounded structured methodology that embeds LLMs in every stage of the SE scientific method, from hypothesis generation through causal analysis and dissemination. In a demonstration case study on prompt design, the framework estimates that instruction-style prompts, longer few-shot demonstrations, and signature-augmented prompts each produce small negative average treatment effects on CodeBLEU accuracy for two 7B-parameter code models. The payoff, if the framework works as claimed, is a reusable pipeline that converts correlational LLM observations into causal claims with explicit confounders and human checkpoints.

What carries the argument

The load-bearing mechanism is the structural causal model (SCM) constructed per hypothesis: a graph specifying treatments, outcomes, mediators, and confounders, which the Scientific Agent (a configured LLM) generates and the human researcher approves before any experiment runs. In the case study the confounder set is $C = \{\text{complexity}, \#\text{ast\_levels}, \#\text{ast\_nodes}, \#\text{ast\_errors}, \#\text{whitespaces}, \#\text{tokens}, \text{vocab\_size}, \#\text{words}, \#\text{identifiers}, \#\text{loc}, \text{prompt\_length}, \#\text{prompt\_tokens}\}$, with complexity also modeled as an effect modifier in hypothesis $H_{13}$. The framework wraps this SCM in eight phases that pair LLM outputs (literature synthesis, hypothesis lists, SCM specifications, ATE estimates, interpretation drafts) with mandatory human checkpoints, and computes average treatment effects using the DoWhy causal-inference pipeline with robustness checks (placebo, random common cause, data subset, unobserved confounder tests).

What would settle it

Re-run the case study after adding a plausible unmeasured confounder (for example, dataset difficulty or snippet length beyond the current $C$) to the SCM and check whether the ATE sign changes; alternatively, conduct a fully randomized prompt assignment on the same dataset and compare the randomized ATE with the observational estimate—a mismatch would indicate residual confounding.

Watch

Extended reading notes

Core claim

On its own terms, the central discovery is that a structured, causally grounded scientific method can be operationalized for SE with LLM assistance, and that the resulting causal estimates are non-trivial: for both Qwen2.5-Coder-7B and CodeLlama-7b-Python, instruction-style prompts ($T_1$) reduce CodeBLEU relative to code-delimited prompts (ATE $-0.139$ and $-0.126$), each additional few-shot sample slightly reduces accuracy ($-0.001$ and $-0.028$), and adding function signatures lowers CodeBLEU by $-0.055$ in both models. The paper frames these small negative effects as challenging the expectation that adding structure or information to prompts improves model performance, and attributes the outcome to model architecture and training context rather than a universal prompt rule.

Load-bearing premise

The reported causal effects are only valid if each SCM correctly includes all important confounders, so that treatment assignment is independent of the outcome once the measured variables are accounted for; if a relevant unmeasured confounder exists, the estimated ATEs are biased.

Editorial extensions

If this is right

  • If ECLAIR works as claimed, empirical SE studies can move beyond reporting correlations and report causal effect estimates with explicit identification assumptions.
  • Prompt design effects on code generation can be estimated and compared across models, so future guidelines for prompt engineering can be grounded in causal evidence rather than anecdote.
  • The small negative ATEs imply that simply adding instructions, examples, or signatures to code-generation prompts is not a free improvement; effectiveness depends on model and context.
  • Because the framework fixes human checkpoints at each phase, it provides a template for auditing LLM-assisted research claims before publication.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper does not pursue: randomize prompt assignments across a broader model family (including larger or differently tuned models) to see whether the small negative ATEs persist or reverse.
  • The framework's value depends on the confounder set being sufficient; a sensitivity analysis that varies the candidate confounder list would show how fragile the reported ATEs are to model misspecification.
  • If the negative prompt effects replicate, they suggest that prompt 'enrichment' may add distracting tokens for code-focused models, a hypothesis that could be tested by ablating prompt components individually.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces ECLAIR (Empirical-Causal LLM-Augmented Inference for SE Research), an eight-phase, human-in-the-loop framework that integrates LLMs into the scientific method for empirical software engineering, grounded in Pearl's causal inference. The framework's phases range from phenomenon observation to dissemination, with LLM assistance in knowledge mapping, hypothesis generation, SCM construction, experimentation automation, causal effect estimation, and interpretation, each with mandatory human checkpoints. The paper demonstrates ECLAIR on a case study of prompt design effects on code generation accuracy (CodeBLEU) for two 7B code models (Qwen2.5-Coder-7B and CodeLlama-7b-Python), evaluating three hypotheses: instruction-style prompts (H1), longer few-shot prompts (H11), and signature-augmented prompts (H13). The reported ATEs are small and negative for both models, leading to the conclusion that these prompt interventions reduce accuracy.

Significance. If the framework is genuinely novel and the case study's estimates are unbiased, ECLAIR would be a useful, reusable pipeline for causally grounded LLM-assisted research in SE, addressing a recognized gap between associational and causal evidence. The paper includes a public replication package, uses standard causal inference tooling (DoWhy), and explicitly integrates human oversight, which are strengths. The case study also provides a transparent template for applying causal inference to prompt engineering questions. However, the current empirical demonstration has identification and reporting shortcomings that must be fixed before the quantitative conclusions can be accepted.

major comments (3)
  1. [Section 3, Table 1] The confounder set C includes prompt_length and #prompt_tokens. For each hypothesis these variables are deterministic or near-deterministic consequences of the treatment (adding an instruction, adding few-shot examples, or adding a function signature). They are post-treatment mediators, not pre-treatment confounders. Conditioning on them in the backdoor adjustment blocks the path from treatment to outcome through prompt length/token count, so the reported ATEs are not total causal effects of prompt design; for H11, adjusting for #prompt_tokens removes the primary manipulated variation. The paper should re-estimate the ATEs without these variables or explicitly redefine the estimand as a direct effect conditional on prompt length/token features.
  2. [Section 3, 'Following the experimental designs'] The ATEs in Table 1 are reported as bare point estimates with no confidence intervals, standard errors, or hypothesis tests, and the robustness checks (placebo, random common cause, data subset, unobserved confounder) are only asserted to have been run. Given that the effects are small (e.g., -0.001 for H11-M1), the reader cannot assess whether they are distinguishable from zero or stable under the specification changes. Please report the uncertainty of each ATE and include the robustness output, including the unobserved-confounder sensitivity analysis.
  3. [Section 3, Phase 4] The paper does not provide the assumed DAG for any hypothesis, nor a justification for why the same variable (complexity) is a confounder in H1/H11 but an effect modifier in H13. Since causal identification rests entirely on the correctness of this graph, the paper should include the SCM specification for each hypothesis, the rationale for variable classification, and any validation or sensitivity of the causal graph.
minor comments (4)
  1. [Section 3] The paper mentions sixteen hypotheses but only three are evaluated; clarify that the remaining hypotheses are listed in the replication package/appendix and describe the selection criteria more explicitly.
  2. [Figure 1] The phase labels in the figure are very small and the contrast between ECLAIR and the canonical method is hard to read; consider enlarging and simplifying.
  3. [Table 1] The legend 'bold: −correlation; purple: −causal effect' is unclear. Explain what 'correlation' refers to (the ρ column) and use a standard notation for causal effects.
  4. [References [15], [21], [23]] These references are from the same research group; the claim that ECLAIR is the 'first' causally grounded structured methodology should be tempered or supported by a broader comparison to independent frameworks.

Circularity Check

1 steps flagged · score 4.0 of 10

The confounder set in Table 1 includes post-treatment prompt-length features, making the reported ATEs biased as total effects; otherwise the framework's derivation is not circular, only self-cited in places.

  1. self definitional [Table 1 (footnote) and Section 3, paragraphs on selected hypotheses and causal effects]
    "H1 binary T0: "{incomplete_method}" T1: "{instruction} + {incomplete_method}" ... C={complexity, #ast_levels, #ast_nodes, #ast_errors, #whitespaces, #tokens, vocab_size, #words, #identifiers, #loc, prompt_length, #prompt_tokens}."

    Under each hypothesis the treatment is a modification of the prompt (instruction, few-shot count, or signature), so prompt_length and #prompt_tokens are deterministic descendants of T, not pre-treatment confounders. Conditioning on them in the backdoor adjustment blocks part of the causal pathway from T to CodeBLEU (or induces collider bias), so the estimated ATE is not the total causal effect claimed. The paper reports these as 'causal effects' and asserts robustness tests confirmed stability, but the construction of C itself changes the estimand: the effect of 'longer' prompts is adjusted for the very length the treatment creates.

full rationale

ECLAIR's central novelty claim and the empirical pipeline are not circular in the sense of a prediction reducing to a fitted value: the ATEs come from a controlled experiment analyzed with DoWhy, not from fitting parameters to the outcome and renaming them predictions. The paper does rely on self-citations — the CodeText-Galeras dataset [21], the causal pipeline paper [15], and the replication package [23] — but these are not load-bearing in a way that forces the conclusions; Pearl's theory [18] and DoWhy [24] are external, and the dataset is a fixed empirical resource. The one genuine construction issue is that the confounder set in Table 1 includes prompt_length and #prompt_tokens, which are post-treatment consequences of the prompt treatments, making the reported ATEs biased as estimates of the total causal effect. That is an identification flaw rather than a pure logical equivalence, so the score is moderate rather than extreme. The paper's statement that 'robustness tests confirmed that the effects were stable and not vulnerable to confounding bias' cannot repair this misspecification because the adjustment set itself changes the estimand; however, the absence of detailed robustness outputs is a reporting gap, not by itself a circularity. Overall, the framework's derivation chain is largely independent of its conclusions, apart from the self-referential confounder construction in the case study.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. The framework is a workflow, not a postulated mechanism. The main assumptions are about causal identification (backdoor criterion, ignorability), outcome validity (CodeBLEU), and the reliability of the LLM agent. No numeric free parameters are fitted in the paper; the ATE values are estimated from the experiment and are not parameters of a derived model.

assumptions (4)
  • domain assumption The backdoor criterion is satisfied: the measured set C of structural and lexical features blocks all backdoor paths from treatment to outcome.
    Invoked in Phase 4 and used in Table 1; if confounders are insufficient, the ATE estimates are biased.
  • domain assumption Treatment assignment is ignorable given C, i.e., no unmeasured confounding remains.
    Required for DoWhy to interpret the estimated association as causal; the paper mentions an unobserved confounder robustness test but does not report its results.
  • domain assumption CodeBLEU is an adequate measure of code generation accuracy.
    All hypotheses define the outcome as CodeBLEU; if CodeBLEU does not capture accuracy, the causal effects are not about accuracy.
  • domain assumption GPT-5 as the Scientific Agent produces a reliable and unbiased knowledge base and hypothesis set.
    The knowledge mapping and hypothesis generation phases depend on the LLM agent's outputs; the paper provides no evaluation of their quality or of potential hallucination.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ECLAIR: A Causally-Grounded AI Framework for Scientific Discovery in Empirical Software Engineering." pith.science (2026). https://pith.science/paper/GXPGRJ73

@misc{pith2026260802323,
  author       = {Pith},
  title        = {Pith review of: ECLAIR: A Causally-Grounded AI Framework for Scientific Discovery in Empirical Software Engineering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GXPGRJ73}},
  note         = {Machine review of arXiv:2608.02323}
}
read the original abstract

The scientific method has long guided empirical research in Software Engineering (SE), but the complexity of modern software systems often hinders its systematic application. This paper introduces ECLAIR, a causally grounded AI framework that integrates Large Language Models (LLMs) into every stage of the scientific process, from hypothesis generation to analysis and interpretation. ECLAIR treats LLMs as active scientific agents operating under the principles of causal inference, within a human-in-the-loop design that safeguards against the risks of unsound automated reasoning. We demonstrate the framework through a case study examining how prompt design influences code generation accuracy in two LLMs. Results show that, for both models, instruction-style, longer few-shot, and signature-augmented prompts yield small negative causal effects on accuracy, illustrating how causal reasoning provides a principled foundation for explaining why software phenomena occur. This study presents the first causally grounded structured methodology for embedding LLMs within the scientific method in SE, designed around the epistemological demands of empirical SE research, establishing a basis for rigorous AI-assisted research.

Figures

Figures reproduced from arXiv: 2608.02323 by the authors.

Figure 1
Figure 1. Our ECLAIR framework (left) contrasted with the canonical scientific method (right) causal effects (e.g., the Average Treatment Effect (ATE)) for each tested hypothesis while controlling for confounding bias, output: ATE estimates and robustness/refutation results per method, fol￾lowing the principles of valid, transparent, and logically consistent inference [4]. (7) Interpretation. In this phase, causal results bec… view at source ↗

Review history (2 revisions) →

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

33 extracted references · 7 canonical work pages

  1. [1]

    Introducing GPT-5

    2025. Introducing GPT-5. https://openai.com/index/introducing-gpt-5/

    2025

  2. [2]

    Collins, Elizabeth Bourne, Gareth W

    Abbi Abdel-Rehim, Hector Zenil, Oghenejokpeme Orhobor, Marie Fisher, Ross J. Collins, Elizabeth Bourne, Gareth W. Fearnley, Emma Tate, Holly X. Smith, Lar- isa N. Soldatova, and Ross King. 2025. Scientific hypothesis generation by large language models: laboratory validation in breast cancer treatment.Journal of The Royal Society Interface22, 227 (June 20...

    work page arXiv 2025

  3. [3]

    Toufique Ahmed, Premkumar Devanbu, Christoph Treude, and Michael Pradel

  4. [4]

    Scott Armstrong and Kesten C

    J. Scott Armstrong and Kesten C. Green. 2022.The Scientific Method: A Guide to Finding Useful Knowledge. Cambridge University Press. doi:10.1017/ 9781009092265

    work page 2022

  5. [5]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

    arXiv 2023

  6. [6]

    Sebastian Baltes, Florian Angermeir, Chetan Arora, Marvin Muñoz Barón, Chun- yang Chen, Lukas Böhme, Fabio Calefato, Neil Ernst, Davide Falessi, Brian Fitzgerald, Davide Fucci, Marcos Kalinowski, Stefano Lambiase, Daniel Russo, Mircea Lungu, Lutz Prechelt, Paul Ralph, Rijnard van Tonder, Christoph Treude, and Stefan Wagner. 2025. Guidelines for Empirical ...

    arXiv 2025

  7. [7]

    Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes

    Daniil A. Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes. 2023. Au- tonomous chemical research with large language models.Nature624, 7992 (Dec. 2023), 570–578. Publisher: Nature Publishing Group. doi:10.1038/s41586-023- 06792-0

    doi:10.1038/s41586-023- 2023

  8. [8]

    Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching Large Language Models to Self-Debug. arXiv:2304.05128 [cs.CL] https://arxiv. org/abs/2304.05128

    arXiv 2023

Show all 33 references
  1. [9]

    Giuseppe Crupi, Rosalia Tufano, Alejandro Velasco, Antonio Mastropaolo, Denys Poshyvanyk, and Gabriele Bavota. 2025. On the Effectiveness of LLM-as-a- Judge for Code Generation and Summarization.IEEE Transactions on Software Engineering51, 8 (2025), 2329–2345. doi:10.1109/TSE....

    2025

  2. [10]

    Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen tau Yih, Luke Zettlemoyer, and Mike Lewis. 2023. InCoder: A Generative Model for Code Infilling and Synthesis. arXiv:2204.05999 [cs.SE] https://arxiv.org/abs/2204.05999

    2023 arXiv

  3. [11]

    Fengqing Jiang, Yichen Feng, Yuetai Li, Luyao Niu, Basel Alomair, and Radha Poovendran. 2025. BadScientist: Can a Research Agent Write Convincing but Unsound Papers that Fool LLM Reviewers? arXiv:2510.18003 [cs]. doi:10.48550/ arXiv.2510.18003

    2025 doi arXiv

  4. [12]

    Polak, So Yeon Kim, MD Al Amin Shuvo, Hrishikesh Shridhar Deodhar, Jeongsoo Han, Dane Morgan, and Hyunseok Oh

    Quanliang Liu, Maciej P. Polak, So Yeon Kim, MD Al Amin Shuvo, Hrishikesh Shridhar Deodhar, Jeongsoo Han, Dane Morgan, and Hyunseok Oh. 2025. Beyond designer’s knowledge: Generating materials design hy- potheses via a large language model.Acta Materialia297 (Sept. 2025), 12130...

    2025

  5. [13]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-Refine: It...

    2023 arXiv

  6. [14]

    Landsness, Daniel L

    Ludovico Mitchener, Angela Yiu, Benjamin Chang, Mathieu Bourdenx, Tyler Nadolski, Arvis Sulovari, Eric C. Landsness, Daniel L. Barabasi, Siddharth Narayanan, Nicky Evans, Shriya Reddy, Martha Foiani, Aizad Kamal, Leah P. Shriver, Fang Cao, Asmamaw T. Wassie, Jon M. Laurent, Ed...

    2025 doi arXiv

  7. [15]

    David Nader Palacio, Alejandro Velasco, Nathan Cooper, Alvaro Rodriguez, Kevin Moran, and Denys Poshyvanyk. 2024. Toward a Theory of Causation for Inter- preting Neural Code Models.IEEE Transactions on Software Engineering50, 5 (May 2024), 1215–1243. doi:10.1109/tse.2024.3379943

    2024

  8. [16]

    Alexander Novikov, Ngân V ˜u, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco J. R. Ruiz, Abbas Mehrabian, M. Pawan Kumar, Abigail See, Swarat Chaudhuri, George Holland, Alex Davies, Sebastian Nowozin, Push...

    2025 doi arXiv

  9. [17]

    Yang Jeong Park, Daniel Kaplan, Zhichu Ren, Chia-Wei Hsu, Changhao Li, Haowei Xu, Sipei Li, and Ju Li. 2024. Can ChatGPT be used to generate scientific hypotheses?Journal of Materiomics10, 3 (May 2024), 578–584. doi:10.1016/j.jmat.2023.08.007

    2024 doi

  10. [18]

    2009.Causality: models, reasoning, and inference

    Judea Pearl. 2009.Causality: models, reasoning, and inference. doi:10.1017/ CBO9780511803161

    2009

  11. [19]

    Judea Pearl. 2018. Theoretical Impediments to Machine Learning With Seven Sparks from the Causal Revolution. arXiv:1801.04016 [cs]. doi:10.48550/arXiv. 1801.04016

    2018 doi arXiv

  12. [20]

    Reddy and Parshin Shojaee

    Chandan K. Reddy and Parshin Shojaee. 2025. Towards scientific discovery with generative AI: progress, opportunities, and challenges. InProceedings of the Thirty- Ninth AAAI Conference on Artificial Intelligence and Thirty-Seventh Conference on Innovative Applications of Artif...

    2025 doi

  13. [21]

    Palacio, Dipin Khati, Henry Burke, and Denys Poshyvanyk

    Daniel Rodriguez-Cardenas, David N. Palacio, Dipin Khati, Henry Burke, and Denys Poshyvanyk. 2023. Benchmarking Causal Study to Interpret Large Lan- guage Models for Source Code. arXiv:2308.12415 [cs.SE] https://arxiv.org/abs/ 2308.12415

    2023 arXiv

  14. [22]

    Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiao- qing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cris- tian Canton Ferrer, Aaron Grattafiori, Wenhan X...

    2024 arXiv

  15. [23]

    2025.ci4sesci: Causal Inference for Software Science

    SEMERU Lab. 2025.ci4sesci: Causal Inference for Software Science. GitHub repository. https://github.com/WM-SEMERU/ci4sesci

    2025

  16. [24]

    Amit Sharma and Emre Kiciman. 2020. DoWhy: An End-to-End Library for Causal Inference.arXiv preprint arXiv:2011.04216(2020). doi:10.48550/arXiv.2011.04216

    2020 doi arXiv

  17. [25]

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: language agents with verbal reinforcement learning. InProceedings of the 37th International Conference on Neural Information Processing Systems(New Orleans, LA, USA)(NIPS ’23). ...

    2023 doi

  18. [26]

    Song Tong, Kai Mao, Zhen Huang, Yukun Zhao, and Kaiping Peng. 2024. Au- tomating psychological hypothesis generation with AI: when large language models meet causal graph.Humanities and Social Sciences Communications11, 1 (July 2024). doi:10.1057/s41599-024-03407-5

    2024 doi

  19. [27]

    Bayer, Aidan Acquah, Chetana Amancharla, Almog Barzilay-Siegal, Pablo Bermejo, Camille Bilodeau, Pablo Cárdenas Ramírez, Miles Cranmer, Ur- bano L

    Francisco Villaescusa-Navarro, Boris Bolliet, Pablo Villanueva-Domingo, Adrian E. Bayer, Aidan Acquah, Chetana Amancharla, Almog Barzilay-Siegal, Pablo Bermejo, Camille Bilodeau, Pablo Cárdenas Ramírez, Miles Cranmer, Ur- bano L. França, ChangHoon Hahn, Yan-Fei Jiang, Raul Jim...

  20. [28]

    Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak, Shengchao Liu, Peter Van Katwyk, Andreea Deac, Anima Anandkumar, Karianne Bergen, Carla P. Gomes, Shirley Ho, Pushmeet Kohli, Joan Lasenby, Jure Leskovec, Tie-Yan Liu, Arjun Manrai, Debora...

    2023

  21. [29]

    arXiv:2510.26887 [cs.AI] https://arxiv.org/abs/2510.26887

    The Denario project: Deep knowledge AI agents for scientific discovery. arXiv:2510.26887 [cs.AI] https://arxiv.org/abs/2510.26887

  22. [30]

    Khan, Adnan Mahmud, Huck Yang, Alexander Lavin, Michael Levin, Jeremy Frey, Jared Dunnmon, James Evans, Alan Bundy, Saso Dzeroski, Jesper Tegner, and Hector Zenil

    Yanbo Zhang, Sumeer A. Khan, Adnan Mahmud, Huck Yang, Alexander Lavin, Michael Levin, Jeremy Frey, Jared Dunnmon, James Evans, Alan Bundy, Saso Dzeroski, Jesper Tegner, and Hector Zenil. 2025. Exploring the role of large language models in the scientific method: from hypothesi...

    2025 doi

  23. [31]

    Ohlsson, Bj ¨rn Regnell, and Anders Wesslén

    Claes Wohlin, Per Runeson, Martin Höst, Magnus C. Ohlsson, Bj ¨rn Regnell, and Anders Wesslén. 2012.Experimentation in Software Engineering. Springer. doi:10.1007/978-3-642-29044-2

    2012 doi

  24. [33]

    Yangqiaoyu Zhou, Haokun Liu, Tejes Srivastava, Hongyuan Mei, and Chenhao Tan. 2024. Hypothesis Generation with Large Language Models. InProceedings of the 1st Workshop on NLP for Science (NLP4Science), Lotem Peled-Cohen, Nitay ASE ’26, October 12–16, 2026, Munich, Germany Vela...

    2024 doi

  25. [2025]

    Can LLMs Replace Manual Annotation of Software Engineering Artifacts? arXiv:2408.05534 [cs.SE] https://arxiv.org/abs/2408.05534

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.