Pith. sign in

REVIEW 3 cited by

Language Models in Software Development Tasks: An Experimental Analysis of Energy and Accuracy

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2412.00329 v2 pith:AGX6555O submitted 2024-11-30 cs.SE

Negar Alizadeh , Boris Belchev , Nishant Saurabh , Patricia Kelbert , Fernando Castor This is my paper · ORCID
classification cs.SE
keywords accuracydevelopmentmodelssoftwareenergylanguagemodeltasks
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The use of generative AI-based coding assistants like ChatGPT and Github Copilot is a reality in contemporary software development. Many of these tools are provided as remote APIs. Using third-party APIs raises data privacy and security concerns for client companies, which motivates the use of locally-deployed language models. In this study, we explore the trade-off between model accuracy and energy consumption, aiming to provide valuable insights to help developers make informed decisions when selecting a language model. We investigate the performance of 18 families of LLMs in typical software development tasks on two real-world infrastructures, a commodity GPU and a powerful AI-specific GPU. Given that deploying LLMs locally requires powerful infrastructure which might not be affordable for everyone, we consider both full-precision and quantized models. Our findings reveal that employing a big LLM with a higher energy budget does not always translate to significantly improved accuracy. Additionally, quantized versions of large models generally offer better efficiency and accuracy compared to full-precision versions of medium-sized ones. Apart from that, not a single model is suitable for all types of software development tasks.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Smaller Models, Unexpected Costs: Trade-offs in LLM Quantization for Automated Program Repair

    cs.SE 2026-06 unverdicted novelty 6.0 of 10

    Empirical evaluation of 13 quantization configurations on 6 LLMs for APR shows reduced memory (up to 85%) but increased inference time/energy, different repaired problem sets with little overlap, and 48% of configs st...

  2. Which Prompting Technique Should I Use? An Empirical Investigation of Prompting Techniques for Software Engineering Tasks

    cs.SE 2025-06 conditional novelty 6.0 of 10

    Across ten software engineering tasks and four LLMs, no prompting technique wins consistently; ES-KNN is best on many tasks, some techniques underperform the baseline, and USC is best for code QA and code generation.

  3. Greening AI-enabled Systems with Software Engineering: A Research Agenda for Environmentally Sustainable AI Practices

    cs.SE 2025-06 accept novelty 4.0 of 10

    A 29-participant workshop synthesized a research agenda for reducing AI's environmental footprint through software engineering, covering measurement, benchmarking, architecture, empirical methods, and education.

Pith tools