Pith. sign in

REVIEW 3 major objections 5 minor 24 references

SkillScope: A Tool to Predict Fine-Grained Skills Needed to Solve Issues on GitHub

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SkillScope introduces a fully automatic tool that predicts 217 fine-grained, two-level API-domain skills for open issues, with the random-forest classifier reaching 91% precision, 88% recall, and 89% F-measure and beating fine-tuned LLMs.

desk verdict Useful engineering extension to issue-skill labeling, but the 91/88/89 numbers measure self-consistency with the auto-labeler, not validated skill prediction. read the letter →

arxiv 2501.15922 v1 pith:IMNYJENU submitted 2025-01-27 cs.SE cs.LG

Benjamin C. Carter , Jonathan Rivas Contreras , Carlos A. Llanes Villegas , Pawan Acharya , Jack Utzerath , Adonijah O. Farner , Hunter Jenkins , Dylan Johnson
show 4 more authors
Jacob Penney Igor Steinmacher Marco A. Gerosa Fabio Santos
This is my paper · ORCID
classification cs.SEcs.LG
keywords softwareengineeringskillcategorizationopensourcemachinelearninglargelanguagemodelsissuelabelingAPIdomainsrandomforest
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SkillScope is a fully automatic tool that labels open issues with the fine-grained programming skills needed to resolve them. It builds a two-level taxonomy of 217 API-domain and subdomain labels from Java source code, then trains classifiers to predict which labels apply to each issue's text. In a case study on a Java project, the random-forest model with TF-IDF features achieved 91% precision, 88% recall, and 89% F-measure, outperforming prior flat-label approaches and fine-tuned GPT models. If these figures hold, the tool could help new contributors find tasks that match their specific expertise and help maintainers delegate work more effectively.

What carries the argument

The key machinery is the two-level API-domain taxonomy and the automated pipeline that derives it: the paper treats class names as API domains and method names as subdomains, parses Java source files into abstract syntax trees, and uses zero-shot prompting of a small LLM to assign each class and method to a domain and then a subdomain, with a similarity function constraining outputs to the known label set. The resulting 217 binary labels are used to train per-domain classifiers—random forest with TF-IDF and fine-tuned GPT-4o-mini—in a one-vs-all scheme, after MLSMOTE oversampling for the random forest and synthetic data generation for the LLM.

What would settle it

Take a random sample of, say, 100 issues labeled by SkillScope, have human annotators familiar with the codebases judge whether each predicted domain and subdomain is genuinely required to resolve the issue, and measure the agreement; if agreement falls substantially below the reported 91% precision, the reported metrics do not reflect real-world accuracy.

Watch

Extended reading notes

Core claim

The central discovery is that a relatively simple pipeline—parse Java source files into abstract syntax trees, use zero-shot GPT-4o-mini to map class and method names to a hierarchy of 31 API domains and 186 subdomains, then train one-vs-all binary classifiers per label—can predict the multilevel skills required by an issue. In a case study on a Java project, the random-forest + TF-IDF variant scored 0.908 precision, 0.876 recall, and 0.889 F-1, exceeding all previously published issue-labeling tools. The fine-tuned GPT-4o-mini and GPT-3.5 models lagged behind, showing that the LLM advantage is not automatic for this fine-grained, long-tailed classification task.

Load-bearing premise

The whole evaluation rests on the assumption that the LLM-generated domain and subdomain labels assigned to classes and methods are accurate enough to serve as ground truth for both training and testing; the paper does not report any human validation of those labels.

Editorial extensions

If this is right

  • If the reported accuracy generalizes beyond the case study, issue trackers could automatically expose the precise skills each task demands, lowering the barrier for newcomers.
  • The two-level labels give contributors more context than flat domains, such as seeing a database subdomain like query execution instead of just 'Database', which could reduce mismatches between contributor skills and task demands.
  • The finding that random forest with TF-IDF outperforms fine-tuned LLMs suggests lightweight classifiers remain competitive for issue labeling, with lower computational cost and faster predictions.
  • The pipeline is limited to Java, but the same class-as-domain, method-as-subdomain idea could be adapted to other languages by replacing the parser.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • In our reading, the reported metrics measure agreement with an LLM-generated label set rather than with human judgment, so the real-world precision could be lower; a human-annotated test set would be needed to learn the true gap.
  • The result that a classical model beats a fine-tuned LLM on this highly imbalanced, long-tailed classification task suggests that synthetic data augmentation and fine-tuning budgets may not be sufficient to overcome label noise or data scarcity in code-focused tasks.
  • The 217-label taxonomy built from class and method structure could serve as a reusable benchmark for issue labeling beyond this tool, independent of the prediction models.
  • A natural next step, not tested here, would be to use the predicted subdomain labels to match issues to contributors by mining their contribution history; the paper leaves this for future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents SkillScope, a tool that predicts fine-grained, multilevel API-domain and subdomain labels (217 total) for GitHub issues in Java projects. The tool mines closed pull requests and issue text, parses Java source files into ASTs, extracts class and method names, uses zero-shot GPT-4o-mini prompts with a Spacy similarity filter to assign domain/subdomain labels, and then trains Random Forest and fine-tuned GPT models on one-hot encoded labels. The evaluation on the JabRef project reports precision 0.908, recall 0.876, and F-measure 0.889 for SkillScope RF+TF-IDF, which the paper claims surpasses prior 31-domain approaches. A user survey with eight respondents is also reported.

Significance. If the reported accuracy were valid, SkillScope would be a useful contribution to newcomer onboarding in OSS, extending prior flat 31-domain skill labels to a 217-label two-level hierarchy. Strengths of the paper include a fully automated pipeline, a publicly available tool and replication packages, and a direct comparison with several prior methods on the same project. However, the central evaluation claim is undermined by the fact that the ground-truth labels used for both training and testing are generated by the same automated GPT-4o-mini pipeline that the tool itself uses, with no human gold standard. The reported metrics therefore primarily measure self-consistency with the label-generation process, not the actual skills needed to solve issues. Since the paper explicitly defers an empirical study with contributors to future work, the central claim of accurate skill prediction is not supported by the evidence presented.

major comments (3)
  1. [Section III-B, III-C, IV] The ground-truth labels for both training and evaluation are produced by the same automated pipeline that the tool uses for prediction. Section III-B describes how class and method names extracted via ASTs are sent through zero-shot GPT-4o-mini prompts, with a Spacy similarity filter selecting from the 217 labels. Section IV evaluates the tool on JabRef contributions processed through the same tool pipeline, and the automatically generated labels appear to serve as ground truth. Consequently, the precision, recall, and F-measure reported in Table IV measure agreement between the Random Forest predictions and the auto-labeler's outputs, not the actual skills needed to resolve issues. The paper reports no human validation of the 217 labels; Section V explicitly defers an empirical study with contributors to future work. This circularity is load-bearing because the central claim of the paper is the numerical accuracy of the predictions.
  2. [Section IV, Table IV] The comparison with prior work is not apples-to-apples. Section II states that Vargovich et al.'s approach involves expert review and manual categorization of APIs, and Section V states that Santos et al. evaluated their API domains with students and practitioners from industry. In contrast, SkillScope's labels are generated without any human validation. If the prior methods were evaluated against human-validated labels while SkillScope is evaluated against its own auto-generated labels, then the claim that SkillScope 'overcame the similar tools with a precision of 91%' (Section I) is not supported by the table. A shared gold standard, or at least a human-annotated subset, is needed to make the comparison meaningful.
  3. [Section IV, Fig. 2] The user survey has only eight respondents, and the paper provides no details about the recruitment, the survey instrument, or how respondents interacted with the tool. The statement that 'the majority finding that the tool either strongly or somewhat confident to contribute' is based on a very small, self-selected sample and, as reported, cannot support a general claim about user experience. Additionally, no statistical tests or confidence intervals are reported for the main precision/recall/F-measure results, so the observed differences between models (e.g., 0.908 vs. 0.864 for RF+TF-IDF) are not shown to be statistically significant.
minor comments (5)
  1. [Section III-B] The mapping that 'the class name corresponds to the API domain, while the function name corresponds to the subdomain' is a strong simplification that is likely to introduce noise, since many method names are not informative about skills (e.g., 'toString', 'get', 'set'). This assumption should be acknowledged and ideally checked against human intuition.
  2. [Section III-C] There is an inconsistency in the model names: the text first says 'fine-tune a GPT-4o model' and later says 'Both models are fine-tuned GPT-4o-mini models.' Please clarify which model was actually used.
  3. [Section IV, Table IV] The table footnote '*all classes' is ambiguous: it is unclear whether this refers to micro-averaging over all classes or to a specific subset. Please state explicitly how the metrics are averaged (the text says micro-averaging, but the table caption and footnotes should be consistent).
  4. [Section V] The name 'SkilScope' is a typo for 'SkillScope'.
  5. [Section II] The phrase 'Collavito et al. [7] investigated the extension to its LLMs' is grammatically awkward and should be rewritten for clarity.

Circularity Check

1 steps flagged · score 7.0 of 10

The reported 91/88/89 metrics are scored against GPT-4o-mini-generated domain/subdomain labels, so they measure self-consistency with the auto-labeler, not human-validated skills.

  1. self definitional [Section III-B (Parsing Engine) -> Section IV (Tool Evaluation), Table IV; Section V (Conclusion)]
    "The functions and class names are first sent through zero-shot prompting into a tiered GPT-4o-mini prompt, first prompting for the domain and then prompting for the subdomain given the domain. ... To prevent hallucinations, the output from the GPT-4o-mini model is fed into a similarity function using Spacy which evaluates the most-likely response given the list of domains and subdomains. ... The final dataset holds columns for each possible domain and subdomain in a one-hot encoded format per issue ..."

    The only labeling mechanism described in the paper is the Section III-B zero-shot GPT-4o-mini + Spacy pipeline. That pipeline creates the one-hot training labels for the RF and fine-tuned LLMs, and the same pipeline is the only described source of labels for the JabRef issues scored in Table IV. No human validation of the 217 labels is reported; Section V admits SkillScope 'will be evaluated in an empirical study with contributors' and cites prior work only for the coarse 31 domains. Hence the 0.908/0.876/0.889 precision/recall/F-measure quantify how well the classifiers reproduce the auto-labeler's assignments, not the skills actually required to solve the issues.

full rationale

This is partial, not total, circularity: the RF is a separate model and the JabRef evaluation is a held-out set, so the numbers are not mathematically forced by a fitted parameter. However, the central claim—predicting 217 multilevel skills—is operationalized entirely by GPT-4o-mini zero-shot labels. Because the same auto-labeler produces both training and evaluation targets, and because Section V explicitly defers human evaluation to future work, the reported metrics are self-consistency scores with the labeling pipeline. The tool's engineering and the algorithm comparison retain independent content, but the headline empirical claim is not anchored to an external ground truth.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim depends on a labeling pipeline whose output is used as both training target and evaluation ground truth. Key assumptions are that the auto-labeled categories are correct, the repositories are representative, and parsing is accurate. The main free parameters are the ML hyperparameters and the unspecified similarity threshold.

free parameters (4)
  • Fine-tuning hyperparameters = batch size=1, temperature=1.0, epochs=3
    Chosen by experimentation for the GPT-4o-mini models (Section III-C-b); these affect all LLM results.
  • MLSMOTE oversampling settings = not specified
    Used to balance minority classes for Random Forest (Section III-C-a); exact settings are not reported, yet the paper credits it with a 38% precision gain.
  • Spacy similarity threshold = not specified
    Zero-shot GPT outputs are accepted or rejected via a similarity function (Section III-B); the threshold is not given, yet it determines which labels enter the dataset.
  • Train/test splits = 80/20 for RF, 70/30 for GPT-4o-mini
    Standard splits chosen following prior work (Section III-C).
assumptions (5)
  • domain assumption API domains are a valid proxy for the skills needed to solve an issue
    Inherited from Santos et al. and used as the basis for all labels (Sections I and III).
  • ad hoc to paper GPT-4o-mini zero-shot classification of class and method names into the 217 labels is accurate enough to serve as ground truth
    The entire dataset and evaluation rest on this auto-labeling step (Section III-B), with only a Spacy similarity filter to reduce hallucinations.
  • domain assumption The 11 selected repositories are representative of Java OSS projects
    Training data comes from only 11 repositories with over 3,000 stars (Section III-A); the generalizability to other Java projects is assumed.
  • domain assumption Closed and merged pull requests' changed files represent the skills needed to solve the corresponding issue
    The tool uses PR-associated source code as the source of skill labels (Section III-A).
  • domain assumption tree-sitter-java correctly extracts functions and classes from Java source files
    The parsing engine depends on tree-sitter-java (Section III-B); any parsing errors propagate to labels.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SkillScope: A Tool to Predict Fine-Grained Skills Needed to Solve Issues on GitHub." pith.science (2026). https://pith.science/paper/IMNYJENU

@misc{pith2026250115922,
  author       = {Pith},
  title        = {Pith review of: SkillScope: A Tool to Predict Fine-Grained Skills Needed to Solve Issues on GitHub},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IMNYJENU}},
  note         = {Machine review of arXiv:2501.15922}
}
read the original abstract

New contributors often struggle to find tasks that they can tackle when onboarding onto a new Open Source Software (OSS) project. One reason for this difficulty is that issue trackers lack explanations about the knowledge or skills needed to complete a given task successfully. These explanations can be complex and time-consuming to produce. Past research has partially addressed this problem by labeling issues with issue types, issue difficulty level, and issue skills. However, current approaches are limited to a small set of labels and lack in-depth details about their semantics, which may not sufficiently help contributors identify suitable issues. To surmount this limitation, this paper explores large language models (LLMs) and Random Forest (RF) to predict the multilevel skills required to solve the open issues. We introduce a novel tool, SkillScope, which retrieves current issues from Java projects hosted on GitHub and predicts the multilevel programming skills required to resolve these issues. In a case study, we demonstrate that SkillScope could predict 217 multilevel skills for tasks with 91% precision, 88% recall, and 89% F-measure on average. Practitioners can use this tool to better delegate or choose tasks to solve in OSS projects.

Figures

Figures reproduced from arXiv: 2501.15922 by the authors.

Figure 1
Figure 1. SkillScope Architecture and then prompting for the subdomain given the domain. The class name corresponds to the API domain, while the function name corresponds to the subdomain. To prevent hallucinations, the output from the GPT-4o-mini model is fed into a simi￾larity function using Spacy2 which evaluates the most-likely response given the list of domains and subdomains. After, each source file in an issue is class… view at source ↗
Figure 2
Figure 2. Evaluation The RF model surpassed both LLM models in predicting the domains and subdomains, as detailed in Table IV. This performance also overcame Santos et al. [4], [6]. The pre￾diction speed of the RF model was also much faster than GPT-4o-mini, as it makes a single call per domain and then a secondary call for its subdomain. With RF, only one prediction is necessary. We also evaluated the user experience with co… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 18 canonical work pages

  1. [1]

    A systematic literature review on the barriers faced by newcomers to open source software projects,

    I. Steinmacher, M. A. G. Silva, M. A. Gerosa, and D. F. Redmiles, “A systematic literature review on the barriers faced by newcomers to open source software projects,” Information and Software Technology , 2015

    work page 2015

  2. [2]

    How to choose a task? mismatches in perspectives of newcomers and existing contributors,

    F. Santos, B. Trinkenreich, J. F. Nicolati Pimentel, I. Wiese, I. Stein- macher, A. Sarma, and M. Gerosa, “How to choose a task? mismatches in perspectives of newcomers and existing contributors,” Empirical Software Engineering and Measurement , 2022

    work page 2022

  3. [3]

    Tell me who are you talking to and i will tell you what issues need your skills,

    F. Santos, J. Penney, J. F. Pimentel, I. Wiese, B. Trinkenreich, I. Stein- macher, and M. A. Gerosa, “Tell me who are you talking to and i will tell you what issues need your skills,” in 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR) , 2023

    work page 2023

  4. [4]

    Tag that issue: Applying api-domain labels in issue tracking systems,

    F. Santos, J. Vargovich, B. Penney, Trinkenreich, I. Santos, Jacob, R. Britto, J. F. Pimentel, I. Wiese, I. Steinmacher, A. Sarma, and M. A. Gerosa, “Tag that issue: Applying api-domain labels in issue tracking systems,” in 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR) , 2023

    work page 2023

  5. [5]

    Givemelabeledissues: An open source issue recommen- dation system,

    J. Vargovich, F. Santos, J. Penney, B. Trinkenreich, I. Steinmacher, and M. A. Gerosa, “Givemelabeledissues: An open source issue recommen- dation system,” in 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR - Data and Tool Showcase) , 2023

    work page 2023

  6. [6]

    Can i solve it? identifying apis required to complete oss tasks,

    F. Santos, I. Wiese, B. Trinkenreich, I. Steinmacher, A. Sarma, and M. A. Gerosa, “Can i solve it? identifying apis required to complete oss tasks,” in 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR) , IEEE. Madrid, Spain: IEEE, 2021, pp. 346–257

    work page 2021

  7. [7]

    Leveraging gpt-like llms to automate issue labeling,

    G. Colavito, F. Lanubile, N. Novielli, and L. Quaranta, “Leveraging gpt-like llms to automate issue labeling,” in 2024 IEEE/ACM 21st In- ternational Conference on Mining Software Repositories (MSR) . IEEE, 2024, pp. 469–480

    2024

  8. [8]

    Applying large language models to issue classification,

    G. Aracena, K. Luster, F. Santos, I. Steinmacher, and M. A. Gerosa, “Applying large language models to issue classification,” in Proceedings of the Third ACM/IEEE International Workshop on NL-based Software Engineering, 2024, pp. 57–60

    work page 2024

Show all 24 references
  1. [9]

    Can chatgpt reproduce human-generated labels? a study of social computing tasks,

    Y . Zhu, P. Zhang, E.-U. Haq, P. Hui, and G. Tyson, “Can chatgpt reproduce human-generated labels? a study of social computing tasks,”

  2. [10]

    Can large language models transform computational social science?

    C. Ziems, W. Held, O. Shaikh, J. Chen, Z. Zhang, and D. Yang, “Can large language models transform computational social science?” Computational Linguistics , vol. 50, no. 1, pp. 237–291, 2024

    2024

  3. [11]

    Language mod- els are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language mod- els are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020

    1901

  4. [12]

    Lost in the middle: How language models use long contexts,

    N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, “Lost in the middle: How language models use long contexts,” Transactions of the Association for Computational Linguistics , 2024

    2024

  5. [13]

    Attracting and retaining oss contributors with a maintainer dashboard,

    M. Guizani, T. Zimmermann, A. Sarma, and D. Ford, “Attracting and retaining oss contributors with a maintainer dashboard,” in Proceedings of the 2022 ACM/IEEE 44th International Conference on Software Engineering: Software Engineering in Society , 2022, pp. 36–40

    2022

  6. [14]

    Ticket tagger: Machine learning driven issue classification,

    R. Kallis, A. Di Sorbo, G. Canfora, and S. Panichella, “Ticket tagger: Machine learning driven issue classification,” in2019 IEEE International Conference on Software Maintenance and Evolution (ICSME) . IEEE

  7. [15]

    A bug or a suggestion? an automatic way to label issues,

    Y . Zhu, M. Pan, Y . Pei, and T. Zhang, “A bug or a suggestion? an automatic way to label issues,” arXiv preprint arXiv:1909.00934 , 2019

    1909 arXiv

  8. [16]

    How to find my task? chatbot to assist newcomers in choosing tasks in oss projects,

    L. P. Serrano Alves, I. S. Wiese, A. P. Chaves, and I. Steinmacher, “How to find my task? chatbot to assist newcomers in choosing tasks in oss projects,” in International Workshop on Chatbot Research and Design . Springer, 2021, pp. 90–107

    2021

  9. [17]

    Understanding skills for oss communities on github,

    J. T. Liang, T. Zimmermann, and D. Ford, “Understanding skills for oss communities on github,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the F oundations of Software Engineering , 2022, pp. 170–182

    2022

  10. [18]

    One-vs-all for class imbalance learning,

    B. Mirza and Z. Lin, “One-vs-all for class imbalance learning,” in 2013 9th International Conference on Information, Communications & Signal Processing. IEEE, 2013, pp. 1–5

    2013

  11. [19]

    Generative llms for synthetic data generation: Methods, challenges and the future,

    X. Guoa and Y . Chenb, “Generative llms for synthetic data generation: Methods, challenges and the future,” International Journal of Informa- tion Technology, vol. 29, no. 1, 2023

    2023

  12. [20]

    Does synthetic data make large language models more efficient?

    S. Gholami and M. Omar, “Does synthetic data make large language models more efficient?” arXiv preprint arXiv:2310.07830 , 2023

    2023 arXiv

  13. [21]

    Efficacy of synthetic data as a benchmark,

    G. Maheshwari, D. Ivanov, and K. E. Haddad, “Efficacy of synthetic data as a benchmark,” arXiv preprint arXiv:2409.11968 , 2024

    2024 arXiv

  14. [22]

    Performance comparison of tf-idf and word2vec models for emotion text classification,

    D. Cahyani and I. Patasik, “Performance comparison of tf-idf and word2vec models for emotion text classification,” Bull. Electr . Eng. Inform., vol. 10, no. 5, 2021

    2021

  15. [23]

    A no-regret generalization of hierarchical softmax to extreme multi-label classification,

    M. Wydmuch, K. Jasinska, M. Kuznetsov, R. Busa-Fekete, and K. Dem- bczynski, “A no-regret generalization of hierarchical softmax to extreme multi-label classification,” Advances in neural information processing systems, vol. 31, 2018

    2018

  16. [2023]

    Available: https://arxiv.org/abs/2304.10145

    [Online]. Available: https://arxiv.org/abs/2304.10145

    arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.