Pith. sign in

REVIEW 2 cited by

Automated DevOps Pipeline Generation for Code Repositories using Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2312.13225 v1 pith:F4BYYJ5T submitted 2023-12-20 cs.SE

Deep Mehta , Kartik Rawool , Subodh Gujar , Bowen Xu This is my paper · ORCID
classification cs.SE
keywords githubdevopsworkflowsactiongenerationlanguagelargemodels
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Automating software development processes through the orchestration of GitHub Action workflows has revolutionized the efficiency and agility of software delivery pipelines. This paper presents a detailed investigation into the use of Large Language Models (LLMs) specifically, GPT 3.5 and GPT 4 to generate and evaluate GitHub Action workflows for DevOps tasks. Our methodology involves data collection from public GitHub repositories, prompt engineering for LLM utilization, and evaluation metrics encompassing exact match scores, BLEU scores, and a novel DevOps Aware score. The research scrutinizes the proficiency of GPT 3.5 and GPT 4 in generating GitHub workflows, while assessing the influence of various prompt elements in constructing the most efficient pipeline. Results indicate substantial advancements in GPT 4, particularly in DevOps awareness and syntax correctness. The research introduces a GitHub App built on Probot, empowering users to automate workflow generation within GitHub ecosystem. This study contributes insights into the evolving landscape of AI-driven automation in DevOps practices.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Doc2CI: A Multi-Service Study of CI Configuration Generation Using Large Language Models

    cs.SE 2026-08 conditional novelty 6.0 of 10

    Across four CI services and 16 LLMs, exact reproduction of reference configs stays below 3.2%, while schema validity (about 71%) trails parse rate (97%), showing similarity and usability are different objectives.

  2. Can LLMs Write CI? A Study on Automatic Generation of GitHub Actions Configurations

    cs.SE 2025-07 conditional novelty 6.0 of 10

    Across six LLMs, zero-shot generation of GitHub Actions YAML configurations reaches 69% similarity to ground truth but only 3% exact matches, with code-pretrained models performing no better than general-purpose ones.

Pith tools