REVIEW 4 major objections 4 minor 1 cited by
SPRINT: An Assistant for Issue Report Management
T0 review · 4 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read SPRINT packages three issue-management tasks into one GitHub app and reports benchmark accuracies of 97.3%, 65.6%, and 34%, plus positive user-study feedback.
desk verdict A competent tool paper that genuinely integrates three existing issue-management models into an open-source GitHub app; the predictive evaluation is credible, but the 'usable and useful' claim rests on a five-participant study that may not have involved hands-on use of the live tool. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central machinery is SPRINT's plugin-based architecture: a GitHub Event Handler receives new issues and posts results, an Issue Indexer keeps a local SQLite mirror of repository issues, and three modular Issue Management Components run the predictive models. The predictive engines are the RTA classifier, a transformer-based bug-report representation model fine-tuned twice (once for duplicate detection, once for severity prediction), and a Llama-2-7b-chat model fine-tuned for bug localization. RTA learns a representation of bug reports through masked-language modeling and contrastive self-supervision, while the localization model compares the issue text against code-file paths and names supplied in a prompt.
What would settle it
Run a longitudinal deployment of SPRINT on several active open-source repositories and compare its duplicate flags, severity labels, and top-5 bug-localization suggestions for the next 100 new issues against independent maintainer verdicts; if agreement falls well below the reported benchmark numbers, the in-practice accuracy claim fails.
Extended reading notes
Core claim
On its own terms, the paper establishes that the three tasks can be served by one open-source GitHub app without requiring per-repository configuration. Installed on a repository, SPRINT listens for new issues, and within about 90 seconds posts a comment listing similar existing issues, attaches a severity label, and posts a ranked list of candidate code files. The reported predictive numbers are 97.3% accuracy on duplicate issue pairs, 65.6% accuracy on five-class severity, and 34% accuracy with a 29% MAP on bug localization, and the user-study participants found the suggestions easy to understand and the combination helpful. The authors present SPRINT as evidence that a comprehensive, extensible issue management assistant can be built from state-of-the-art deep learning models.
Load-bearing premise
The load-bearing premise is that the five professional developers and the hand-picked issue queries in the user study represent how developers in general would react to SPRINT; if that sample is unrepresentative, the usability and usefulness claims do not generalize.
Editorial extensions
If this is right
- A project that installs SPRINT gets automatic duplicate flags and severity labels on every new issue, reducing the manual triage a maintainer must do.
- Duplicate detection at 97.3% accuracy means most repeated reports can be linked to existing issues before a human reads them, though false positives remain.
- Severity labels at 65.6% accuracy can support prioritization but should be treated as advisory rather than authoritative.
- Bug localization's 34% top-1 accuracy is low, but the top-5 ranked list still narrows a developer's search to a handful of files.
- Because components are modular APIs, project owners can replace a model with a better fine-tuned transformer by editing a configuration file, so the tool can improve as models improve.
Reading between the lines
- The 90-second response time was measured on a repository with about 20 issues and 50 code files; on larger repositories the pairwise duplicate scan will take longer, so that responsiveness number is unlikely to transfer without larger process pools.
- Because the bug-localization prompt uses only file paths and names, feeding the model code file contents or repository structure is a direct, testable way to try to raise the 34% top-1 accuracy.
- The extensibility claim is architectural; a convincing test would be to add a fourth component using the documented plugin APIs and report the integration effort.
- The user study deliberately mixes successful and unsuccessful model predictions, which helps participants judge the interface but cannot estimate how often the tool is wrong in real use; only a deployment log can give that error rate.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents SPRINT, an open-source GitHub application that integrates three issue-management features: duplicate/similar issue detection using the RTA model, severity prediction also using RTA, and bug localization using a fine-tuned Llama-2-7b-chat model from LongCodeArena. The authors evaluate the underlying models by replicating the benchmark evaluations from the original papers (reporting 97.3% accuracy for duplicate detection, 65.6% for severity prediction, and 34% accuracy, 20% R@2, 31% P@2, 29% MAP for bug localization) and conduct a user study with five professional developers from Samsung Research Bangladesh to assess usability and usefulness. The paper concludes that SPRINT is accurate, usable, and useful.
Significance. If the central claims were fully supported, SPRINT would be a convenient open-source integration of several established models into a single GitHub app, and the replication of existing benchmark evaluations gives some confidence in the models' predictive performance. The modular architecture and open-source release are concrete strengths. However, the headline claim of usability and usefulness rests entirely on a five-participant user study with one company, hand-picked queries, and no hands-on use of the live tool, so the significance as presented is limited. It is a reasonable tool demonstration with preliminary evaluation rather than a rigorous empirical study.
major comments (4)
- [IV-B] The user study in Section IV-B does not support the abstract's general claim that SPRINT is 'usable and useful.' The study has only five participants from a single organization, no control or baseline condition, and no statistical analysis. More importantly, the methodology describes only that participants were 'introduced to the tool with detailed guidelines' and given ground-truth data for reference; it does not state that they installed SPRINT, filed issues, or interacted with the live application. Under these conditions, the uniformly positive RQ1/RQ2 responses may reflect agreement with pre-computed examples rather than a genuine assessment of usability in a real issue-management workflow. I recommend either conducting a more realistic study (e.g., hands-on tasks with the live tool, larger and more diverse sample, or at least a report of actual usage) or substantially tempering the usability/usefulness claims.
- [IV-B, RQ3] The RQ3 evaluation selects queries that are known to include one success and one failure per feature, and participants are given the ground-truth labels. For similar-issue detection, the study uses only two queries (with eight non-duplicate issues), and for bug localization two queries from one project. This design can check whether users can recognize correct and incorrect suggestions, but it cannot independently verify the predictive accuracy of SPRINT. The paper should report the precision and recall for the selected queries and explicitly state that this is a user-perception check, not an accuracy benchmark.
- [IV-A] The bug-localization metric '34% accuracy' is undefined: it is not clear whether this is accuracy@1, accuracy@5, or something else. The top-5 cutoff used in the user study is not justified. Given that the reported R@2 and P@2 are only 20% and 31%, the claim that SPRINT is 'accurate' for bug localization is not strongly supported. Please define all metrics, report standard rank-based variants (e.g., MRR, accuracy@k for k=1,5,10), and relate them to the user study's top-5 setting.
- [III-A / IV-A] The paper repeatedly describes the underlying models as 'state-of-the-art,' but RTA and LongCodeArena are prior work; SPRINT's contribution is the integration. The model evaluation replicates existing benchmark results and thus validates the choice of models, but it does not validate the end-to-end tool behavior (e.g., how suggestions are rendered as comments/labels, or performance under the stated concurrency goals). The paper should either add an end-to-end evaluation of the integrated tool or explicitly restrict the 'accurate' claim to the underlying models.
minor comments (4)
- [Abstract] The GitHub repository URL in the abstract is broken ('sprint issue report assistant tool' with spaces instead of hyphens); the same issue appears in reference [15].
- [IV-A] The severity dataset's class distribution is described only as 'between 15% and 24%' without giving the exact distribution or per-class performance; adding a confusion matrix or per-class precision/recall would improve interpretability.
- [IV-B] No response-time statistics (e.g., mean, min, max) are reported for the 'approximately 90 seconds' latency; given that only one repository with approximately 20 issues and 50 code files was used, the responsiveness claim is not generalizable.
- [Throughout] The paper has inconsistent spacing around 'SPRINT' in several places (e.g., 'S PRINT' in Section II), and some references list 'Severity & priority types of bug reports' for unrelated sources [21,25]; a careful copyedit and reference cleanup are needed.
Circularity Check
No significant circularity: model accuracy is benchmarked on external datasets, and the only self-citations are peripheral.
full rationale
SPRINT's claims are not circular. The predictive performance in Section IV-A is obtained by replicating the evaluations of existing models on their published test datasets (RTA duplicate/severity sets; Bogomolov et al.'s LongCodeArena bug-localization set). These test sets are external ground truth, not outputs or fitted parameters of this paper, so the reported accuracies are genuine benchmark results rather than predictions defined by their inputs. The bug-localization model was fine-tuned on the prior authors' dataset and evaluated on a separate fixed 150-issue test set. The user study in Section IV-B is a small qualitative evaluation with five developers; the statement that participants were 'provided ... ground truth data for reference' (Section IV-B.1) means their accuracy judgments are partly informed, but this is an evidence-validity limitation, not a circular reduction from output to input. Self-citations in the reference list (e.g., Refs. [2], [4], [33]-[37], [42], [46]) appear in related-work and motivation contexts and do not carry the central claim. No equation or fitted quantity is reused as its own prediction, so the circularity burden is very low.
Assumptions & free parameters
free parameters (1)
- Top-5 cutoff for bug localization suggestions
assumptions (3)
- domain assumption RTA and LongCodeArena benchmark datasets are representative of real-world issue reports and provide valid ground truth.
- domain assumption Five professional developers from Samsung Research Bangladesh are representative of the developer population that would use SPRINT.
- domain assumption The models' benchmark performance transfers to SPRINT's operational setting, including live GitHub repositories and latest-version code files.
Cite this review
Pith. "Pith review of SPRINT: An Assistant for Issue Report Management." pith.science (2026). https://pith.science/paper/WXN7LL2K
@misc{pith2026250204147,
author = {Pith},
title = {Pith review of: SPRINT: An Assistant for Issue Report Management},
year = {2026},
howpublished = {\url{https://pith.science/paper/WXN7LL2K}},
note = {Machine review of arXiv:2502.04147}
}
read the original abstract
Managing issue reports is essential for the evolution and maintenance of software systems. However, manual issue management tasks such as triaging, prioritizing, localizing, and resolving issues are highly resource-intensive for projects with large codebases and users. To address this challenge, we present SPRINT, a GitHub application that utilizes state-of-the-art deep learning techniques to streamline issue management tasks. SPRINT assists developers by: (i) identifying existing issues similar to newly reported ones, (ii) predicting issue severity, and (iii) suggesting code files that likely require modification to solve the issues. We evaluated SPRINT using existing datasets and methodologies, measuring its predictive performance, and conducted a user study with five professional developers to assess its usability and usefulness. The results show that SPRINT is accurate, usable, and useful, providing evidence of its effectiveness in assisting developers in managing issue reports. SPRINT is an open-source tool available at https://github.com/sea-lab-wm/sprint_issue_report_assistant_tool.
Figures
Forward citations
Cited by 1 Pith paper
-
Applying Large Language Models to Issue Classification: Revisiting with Extended Data and New Models
Fine-tuned GPT-4o classifies GitHub issue types with about 86% F1 on NLBSE 2024 data, while a much larger NLBSE 2023 dataset does not improve results.
Reference graph
Works this paper leans on
-
[1]
On the effectiveness of information retrieval based bug localization for c programs,
R. K. Saha, J. Lawall, S. Khurshid, and D. E. Perry, “On the effectiveness of information retrieval based bug localization for c programs,” in Pro- ceedings of the IEEE International Conference on Software Maintenance and Evolution, 2014, pp. 161–170
work page 2014
-
[2]
Detecting missing information in bug descriptions,
O. Chaparro, J. Lu, F. Zampetti, L. Moreno, M. Di Penta, A. Marcus, G. Bavota, and V . Ng, “Detecting missing information in bug descriptions,” in Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ser. ESEC/FSE 2017, 2017, p. 396–407
work page 2017
-
[3]
What makes a good bug report?
N. Bettenburg, S. Just, A. Schr ¨oter, C. Weiss, R. Premraj, and T. Zim- mermann, “What makes a good bug report?” in Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of software engineering, 2008, pp. 308–318
2008
-
[4]
A. Saha and O. Chaparro, “Decoding the issue resolution process in practice via issue report analysis: A case study of firefox,” in Proceedings of the IEEE/ACM 47th International Conference on Software Engineering (ICSE’25), 2025
work page 2025
-
[5]
The benefit of issue management: anticipating crises in the digital age,
N. Strauβ and J. Jonkman, “The benefit of issue management: anticipating crises in the digital age,” Journal of Communication Management, vol. 21, pp. 34–50, 02 2017
work page 2017
-
[6]
A survey on bug-report analysis,
J. Zhang, X. Wang, D. Hao, B. Xie, and Zhang, “A survey on bug-report analysis,” Sci. China Inf. Sci. , vol. 58, no. 2, pp. 1–24, 2015
work page 2015
-
[7]
Duplicate bug reports considered harmful . . . really?
N. Bettenburg, R. Premraj, T. Zimmermann, and . Sunghun Kim, “Duplicate bug reports considered harmful . . . really?” in Proceedings of the IEEE International Conference on Software Maintenance , 2008, pp. 337–345
work page 2008
-
[8]
Automated classification of software bug reports,
A. F. Otoom, S. Al-jdaeh, and M. Hammad, “Automated classification of software bug reports,” in proceedings of the 9th international conference on information communication and management , 2019, pp. 17–21
work page 2019
Show all 46 references
-
[9]
How practi- tioners perceive automated bug report management techniques,
W. Zou, D. Lo, Z. Chen, X. Xia, Y . Feng, and B. Xu, “How practi- tioners perceive automated bug report management techniques,” IEEE Transactions on Software Engineering, vol. 46, no. 8, pp. 836–862, 2020
2020
-
[10]
Find duplicates: Detect similar issues, find related issues,
“Find duplicates: Detect similar issues, find related issues,” https://marketplace.atlassian.com/apps/1212706/ find-duplicates-detect-similar-issues-find-related-issues?tab=overview& hosting=datacenter, 2024
2024
-
[11]
Probot/duplicate-issues,
“Probot/duplicate-issues,” https://github.com/probot/duplicate-issues, 2024
2024
-
[12]
Nextbug: a bugzilla extension for recommending similar bugs,
H. Rocha, G. d. Oliveira, H. Marques-Neto, and M. T. Valente, “Nextbug: a bugzilla extension for recommending similar bugs,” Journal of Software Engineering Research and Development , vol. 3, pp. 1–14, 2015
2015
-
[13]
Priority scheduler: Easy issue priority management: Change priority by date
“Priority scheduler: Easy issue priority management: Change priority by date.” https://marketplace.atlassian.com/apps/1233172/priority-scheduler? hosting=cloud&tab=overview
-
[14]
Pr-agent,
“Pr-agent,” https://github.com/Codium-ai/pr-agent, 2024
2024
-
[15]
Sprint github repository,
“Sprint github repository,” https://github.com/sea-lab-wm/sprint issue report assistant tool, 2025
2025
-
[16]
Sprint: Issue report assistant tool,
“Sprint: Issue report assistant tool,” https://github.com/apps/ sprint-issue-report-assistant, 2024
2024
-
[17]
Severity & priority types of bug reports,
“Severity & priority types of bug reports,” https://www.javatpoint.com/ severity-and-priority-in-testing
-
[18]
About writing code for a github application,
“About writing code for a github application,” https://docs.github. com/en/apps/creating-github-apps/writing-code-for-a-github-app/ about-writing-code-for-a-github-app
-
[19]
Repre- sentthemall: A universal learning representation of bug reports,
S. Fang, T. Zhang, Y . Tan, H. Jiang, X. Xia, and X. Sun, “Repre- sentthemall: A universal learning representation of bug reports,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023, pp. 602–614
2023
-
[20]
Duplicate bug report dataset by rta,
“Duplicate bug report dataset by rta,” https://drive.google.com/drive/ folders/1gPnZbgOO4XiBBsyF27jS--XwhHaInxlQ, 2023
2023
-
[21]
Severity & priority types of bug reports,
“Severity & priority types of bug reports,” https://www.geeksforgeeks. org/processpoolexecutor-class-in-python/
-
[22]
Long code arena: a set of benchmarks for long-context code models,
E. Bogomolov, A. Eliseeva, T. Galimzyanov, E. Glukhov, A. Shapkin, M. Tigina, Y . Golubev, A. Kovrigin, A. van Deursen, M. Izadi et al., “Long code arena: a set of benchmarks for long-context code models,” arXiv preprint arXiv:2406.11612 , 2024
2024 arXiv
-
[23]
Llama-7b-chat-hf,
“Llama-7b-chat-hf,” https://huggingface.co/meta-llama/ Llama-2-7b-chat-hf
-
[24]
Bug localization dataset of long code arena by jetbrains,
“Bug localization dataset of long code arena by jetbrains,” https: //huggingface.co/datasets/JetBrains-Research/lca-bug-localization, 2024
2024
-
[25]
Severity & priority types of bug reports,
“Severity & priority types of bug reports,” https://medium.com/ omarelgabrys-blog/plug-in-architecture-dec207291800
-
[26]
Welcome to flask’s documentation,
“Welcome to flask’s documentation,” https://flask.palletsprojects.com/en/ stable/
-
[27]
Apache openoffice github repository,
“Apache openoffice github repository,” https://github.com/apache/ openoffice
-
[28]
wso2/testgrid project repository,
“wso2/testgrid project repository,” https://github.com/wso2/testgrid
-
[29]
Jira: Issue & project tracking software,
“Jira: Issue & project tracking software,” https://www.atlassian.com/ software/jira
-
[30]
Bugzilla: The software solution designed to drive software development,
“Bugzilla: The software solution designed to drive software development,” https://www.bugzilla.org/
-
[31]
Buglocalizer: Integrated tool support for bug localization,
F. Thung, T.-D. B. Le, P. S. Kochhar, and D. Lo, “Buglocalizer: Integrated tool support for bug localization,” in Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, 2014, pp. 767–770
2014
-
[32]
Introducing chatgpt — openai,
“Introducing chatgpt — openai,” https://openai.com/index/chatgpt/
-
[33]
Burt: A chatbot for interactive bug reporting,
Y . Song, J. Mahmud, N. De Silva, Y . Zhou, O. Chaparro, K. Moran, A. Marcus, and D. Poshyvanyk, “Burt: A chatbot for interactive bug reporting,” in 2023 IEEE/ACM 45th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) . IEEE, 2023, pp. 170–174
2023
-
[34]
Toward interactive bug reporting for (android app) end-users,
Y . Song, J. Mahmud, Y . Zhou, O. Chaparro, K. Moran, A. Marcus, and D. Poshyvanyk, “Toward interactive bug reporting for (android app) end-users,” in Proceedings of the 30th ACM joint european software engineering conference and symposium on the foundations of software engine...
2022
-
[35]
Bee: A tool for structuring and analyzing bug reports,
Y . Song and O. Chaparro, “Bee: A tool for structuring and analyzing bug reports,” in Proceedings of the 28th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering, 2020, pp. 1551–1555
2020
-
[36]
Combining language and app ui analysis for the automated assessment of bug reproduction steps,
J. Mahmud, A. Saha, O. Chaparro, K. Moran, and A. Marcus, “Combining language and app ui analysis for the automated assessment of bug reproduction steps,” in Proceedings of the 33rd IEEE/ACM International Conference on Program Comprehension (ICPC’25) , 2025
2025
-
[37]
Assessing the quality of the steps to reproduce in bug reports,
O. Chaparro, C. Bernal-C ´ardenas, J. Lu, K. Moran, A. Marcus, M. Di Penta, D. Poshyvanyk, and V . Ng, “Assessing the quality of the steps to reproduce in bug reports,” in Proceedings of the 2019 27th ACM joint meeting on european software engineering conference and symposium ...
2019
-
[38]
Duplicate bug report detection: How far are we?
T. Zhang, D. Han, V . Vinayakarao, I. C. Irsan, B. Xu, F. Thung, D. Lo, and L. Jiang, “Duplicate bug report detection: How far are we?” ACM Transactions on Software Engineering and Methodology , vol. 32, no. 4, pp. 1–32, 2023
2023
-
[39]
A soft alignment model for bug deduplication,
I. M. Rodrigues, D. Aloise, E. R. Fernandes, and M. Dagenais, “A soft alignment model for bug deduplication,” in Proceedings of the 17th International Conference on Mining Software Repositories , 2020, pp. 43–53
2020
-
[40]
Bug severity prediction algorithm using topic- based feature selection and cnn-lstm algorithm,
J. Kim and G. Yang, “Bug severity prediction algorithm using topic- based feature selection and cnn-lstm algorithm,” IEEE Access, vol. 10, pp. 94 643–94 651, 2022
2022
-
[41]
Bert based severity prediction of bug reports for the maintenance of mobile applications,
A. Ali, Y . Xia, Q. Umer, and M. Osman, “Bert based severity prediction of bug reports for the maintenance of mobile applications,” Journal of Systems and Software , vol. 208, p. 111898, 2024
2024
-
[42]
On using gui interaction data to improve text retrieval-based bug localization,
J. Mahmud, N. De Silva, S. A. Khan, S. H. Mostafavi, S. H. Mansur, O. Chaparro, A. Marcus, and K. Moran, “On using gui interaction data to improve text retrieval-based bug localization,” in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering , 202...
2024
-
[43]
Automatic categorization of bug reports using latent dirichlet allocation,
K. Somasundaram and G. C. Murphy, “Automatic categorization of bug reports using latent dirichlet allocation,” in Proceedings of the 5th India software engineering conference , 2012, pp. 125–130
2012
-
[44]
Not all bugs are the same: Understanding, characterizing, and classifying bug types,
G. Catolino, F. Palomba, A. Zaidman, and F. Ferrucci, “Not all bugs are the same: Understanding, characterizing, and classifying bug types,” Journal of Systems and Software , vol. 152, pp. 165–181, 2019
2019
-
[45]
Bug triaging: right developer recom- mendation for bug resolution using data mining technique,
B. Chaitra and K. Swarnalatha, “Bug triaging: right developer recom- mendation for bug resolution using data mining technique,” in Emerging Research in Computing, Information, Communication and Applications: ERCICA 2020, Volume 2 . Springer, 2022, pp. 609–618
2020
-
[46]
Toward the automated localization of buggy mobile app uis from bug descriptions,
A. Saha, Y . Song, J. Mahmud, Y . Zhou, K. Moran, and O. Chaparro, “Toward the automated localization of buggy mobile app uis from bug descriptions,” in Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis , 2024, pp. 1249–1261
2024
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.