Pith. sign in

REVIEW 4 cited by

LLMs as Evaluators: A Novel Approach to Evaluate Bug Report Summarization

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2409.00630 v1 pith:JUCDMNB2 submitted 2024-09-01 cs.SE

Abhishek Kumar , Sonia Haiduc , Partha Pratim Das , Partha Pratim Chakrabarti This is my paper · ORCID
classification cs.SE
keywords llmsreportevaluatorssummarizationsoftwarehumansapproachesartifacts
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Summarizing software artifacts is an important task that has been thoroughly researched. For evaluating software summarization approaches, human judgment is still the most trusted evaluation. However, it is time-consuming and fatiguing for evaluators, making it challenging to scale and reproduce. Large Language Models (LLMs) have demonstrated remarkable capabilities in various software engineering tasks, motivating us to explore their potential as automatic evaluators for approaches that aim to summarize software artifacts. In this study, we investigate whether LLMs can evaluate bug report summarization effectively. We conducted an experiment in which we presented the same set of bug summarization problems to humans and three LLMs (GPT-4o, LLaMA-3, and Gemini) for evaluation on two tasks: selecting the correct bug report title and bug report summary from a set of options. Our results show that LLMs performed generally well in evaluating bug report summaries, with GPT-4o outperforming the other LLMs. Additionally, both humans and LLMs showed consistent decision-making, but humans experienced fatigue, impacting their accuracy over time. Our results indicate that LLMs demonstrate potential for being considered as automated evaluators for bug report summarization, which could allow scaling up evaluations while reducing human evaluators effort and fatigue.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Beyond Text Matching: Towards Reference-Free Evaluation for Human-Oriented Binary Reverse Engineering

    cs.SE 2026-08 conditional novelty 6.0 of 10

    LLM judges match human expert opinions on binary reverse engineering outputs better than text-similarity metrics, and a learned router improves both accuracy and cost.

  2. A Case Study Investigating the Role of Generative AI in Quality Evaluations of Epics in Agile Software Development

    cs.SE 2025-05 conditional novelty 6.0 of 10

    An LLM-based evaluator for agile epics was built from a new rubric and tested with 17 product managers, who found it useful but limited by lack of domain knowledge and rigid scoring.

  3. Do Automatic Comment Generation Techniques Fall Short? Exploring the Influence of Method Dependencies on Code Understanding

    cs.SE 2025-04 conditional novelty 6.0 of 10

    Including the bodies of helper methods in the prompt enables large language models to generate better comments for dependent Java methods than standard comment generators.

  4. Developer Challenges on Large Language Models: A Study of Stack Overflow and OpenAI Developer Forum Posts

    cs.SE 2024-11 conditional novelty 5.0 of 10

    A BERTopic analysis of 8,593 Stack Overflow posts and 26,474 OpenAI Developer Forum posts yields 9 and 17 LLM developer challenge topics, with API usage dominant and high unresolved rates.

Pith tools