Pith. sign in

REVIEW 5 major objections 6 minor 53 references

Addressing Popularity Bias in Third-Party Library Recommendations Using LLMs

T0 review · 5 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Open-source LLMs cannot overcome popularity bias in third-party library recommenders.

desk verdict The first LLM-based TPL debiasing study, but its central negative claim is undercut by an instruction/metric mismatch that caps recall by construction. read the letter →

arxiv 2501.10313 v1 pith:MZOVS2GA submitted 2025-01-17 cs.SE

Claudio Di Sipio , Juri Di Rocco , Davide Di Ruscio , Vladyslav Bulhakov This is my paper · ORCID
classification cs.SE
keywords popularitybiasthird-partylibraryrecommendationlargelanguagemodelsrecommendersystemsforsoftwareengineeringlong-taileffectfine-tuningablationstudyLlama
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether open-source large language models can counteract popularity bias when recommending third-party libraries to Java developers. Popularity bias is the tendency of recommenders to keep suggesting the same well-known libraries, such as junit or log4j, while ignoring less-used libraries that may fit the project better. The authors run an ablation study across six configurations that combine three Llama models, zero-shot and few-shot prompts, LoRA fine-tuning, and a popularity penalty score. Their central finding is that none of these configurations fixes the problem: recall stays between 0.08 and 0.20 in every configuration, even though precision, catalog coverage, and diversity improve. The practical consequence, if the result holds, is that prompting or lightly fine-tuning an LLM is not enough, and TPL recommenders need different mechanisms such as retrieval-augmented generation, post-hoc debiasing, or explicit user feedback.

What carries the argument

The argument is carried by an ablation study with six configurations, built on the CrossRec dataset of Java projects and their dependencies. Three variables are manipulated: prompt strategy (zero-shot, few-shot, few-shot with history), model (Llama-2-7b-chat, Llama-2-13b-chat, Llama-3-8b-instruct), and two debiasing modules. The first debiasing module is parameter-efficient fine-tuning via LoRA (rank 16, $\alpha$ 32, dropout 0.05) applied only in configuration C6. The second is a popularity penalty score defined as $1/(\text{popularity rank}+1)$, computed from Maven usage counts, which is used to lower the likelihood of recommending frequently used libraries. The outcome is measured with precision, recall, F1, catalog coverage, and Expected Popularity Complement (EPC), and the pattern across configurations is what supports the negative conclusion.

What would settle it

Find the held-out Java projects whose ground-truth dependencies include popular libraries such as junit or slf4j, rerun configurations C4 and C6 with instruction IST3 (avoid popular libraries) removed but all other settings identical, and compare recall. If recall rises substantially toward the level of a non-LLM recommender such as LibSeek on the same split, the paper's conclusion that LLMs cannot address popularity bias would be weakened, because the low recall would be an artifact of the instruction conflicting with the ground truth.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is a negative result: the considered open-source LLMs cannot address popularity bias in third-party library recommendation. Across the six ablation configurations (C1–C6), recall never exceeds 0.20, with the best recall achieved by Llama-3-8b-instruct with few-shot prompting alone (C4), while the configuration that adds both fine-tuning and the popularity penalty (C6) reaches only 0.17 recall but raises catalog coverage to 55% and EPC to 60%. The authors interpret this as evidence that the long-tail effect persists in LLM-generated recommendations and that fine-tuning and post-processing penalties mainly increase diversity rather than relevance. They therefore conclude that substantially different mitigation strategies are needed.

Load-bearing premise

The evaluation assumes that asking the model to avoid popular libraries (instruction IST3) is compatible with a ground truth made of the libraries that held-out projects actually use; when popular libraries appear in that ground truth, the instruction itself suppresses recall, so the measured failure may be partly forced by the experiment rather than by the model's inability to address popularity bias.

Editorial extensions

If this is right

  • Fine-tuning and the popularity penalty do not rescue recall; the best configuration reaches only 0.20 recall, so developers cannot rely on these cheap interventions alone.
  • The debiasing modules do improve diversity: catalog coverage rises from 26% in the baseline to 55% in C6, and EPC from 15% to 60%, so the modules are not inert.
  • Because the gains in precision and diversity do not translate into recall, the long-tail effect in TPL recommendations remains unresolved for open-source LLMs.
  • The authors' proposed next steps—retrieval-augmented generation, post-processing debiasing algorithms, and explicit user feedback—are the directions that would need to be tested next.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The negative result is probably stronger than the paper states in one way and weaker in another: the penalty mechanism's formula $1/(\text{rank}+1)$ is applied as a post-hoc reweighting that never changes the model's internal predictions, so a failure to reach high recall with it does not rule out better constrained-decoding or re-ranking approaches.
  • A direct testable extension would be to swap the popularity penalty for a calibrated log-popularity term or a learned re-ranker; if recall then approaches a traditional recommender's level, the bottleneck would be the debiasing method, not the LLM.
  • The same evaluation design could be applied to code-specific open models such as CodeLlama or CodeMistral, which the authors mention as future work; a comparative result there would show whether the failure is generic to LLMs or specific to chat-tuned general models.
  • The instruction-based avoidance of popular libraries may trade accuracy for novelty in a way that the metrics do not disentangle; a user study asking developers whether the diverse recommendations are actually useful would be the natural follow-up.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper investigates whether open-source LLMs (Llama-2-7b-chat, Llama-2-13b-chat, Llama-3-8b-instruct) can mitigate popularity bias in third-party library (TPL) recommendation. The authors design an ablation study with six configurations (C1-C6) that combine zero-shot/few-shot prompting, conversation history, LoRA fine-tuning, and a popularity penalty mechanism. They evaluate with P@N, R@N, F1, Coverage@N, and Expected Popularity Complement (EPC). The main finding is that recall remains low (0.08-0.20) across all configurations, leading the authors to conclude that the considered LLMs cannot address popularity bias in TPL recommenders, while fine-tuning and penalty mechanisms improve diversity. A replication package is provided.

Significance. If the central negative result were methodologically airtight, the paper would be a useful counterpoint to optimism about LLM-based TPL recommenders, and the ablation design would help researchers avoid ineffective mitigation strategies. The choice of a reproducible, openly available setup, the inclusion of diversity-oriented metrics (Coverage@N, EPC), and the public replication package are genuine strengths. However, the current evaluation has several load-bearing gaps: the value of N is never stated, the ground-truth construction is not described, the interaction between the anti-popularity instruction (IST3) and the recall metric is unexamined, and the reported numbers lack variance estimates. These gaps prevent the paper from supporting its main claim in its present form, though the claim is potentially salvageable with additional analysis.

major comments (5)
  1. [Section IV-A, Eqs. (2)-(6)] N is never specified for any of P@N, R@N, F1, Coverage@N, or EPC@N. Without a fixed N, the values in Table III cannot be interpreted or compared across configurations, and the study is not reproducible. Please state N explicitly and justify the choice, or report results for a range of N values.
  2. [Section III-B and Section IV-A] IST3 ("Avoid popular libraries") is present in every prompt, including the baseline C1 (Listing 1), while R@N (Eq. 3) is scored against ground-truth libraries actually used by held-out projects, which Fig. 3 shows are predominantly popular (junit, slf4j, log4j, guava). This creates a ceiling on obtainable recall: the model is instructed not to recommend the very libraries that constitute a large share of the ground truth. The paper never reports this ceiling, the fraction of ground-truth items in the excluded popular set, or the composition of the ground-truth set, so recall values of 0.08-0.20 cannot be unambiguously interpreted as "LLMs cannot address popularity bias." The paper's own EPC result (C6 reaches 60% vs. 15% for C1) also suggests that some mitigation is occurring. Please quantify the ceiling and either remove IST3 from the baseline or analyze recall on the non-popular subset of the ground truth.
  3. [Section V-A, Table I and Table III] The text identifies "C3" as the best configuration and describes it as "the Llama-3-8b-instruct model ... with only the few-shots technique," but Table I defines C3 as Llama-2-13b-chat with few-shots + history and C4 as Llama-3-8b-instruct with few-shots. This mislabeling makes the discussion of which technique drives the improvement, and the comparison with C5/C6, untrustworthy. Please correct the configuration numbering or the table.
  4. [Section III-D] Eq. (1) defines the penalty score, but the paper never describes how this score is applied to the generation process (e.g., logit adjustment, re-ranking, filtering, or sampling) nor how the "Popularity Rank" from Maven is computed and aligned with the dataset used for evaluation. Without this information, the C5 and C6 results are not reproducible, and the lack of an effect on recall cannot be attributed to the mechanism itself.
  5. [Table III] No confidence intervals, error bars, or statistical tests are reported for any metric. Several comparisons used to support the negative conclusion are small (e.g., C5 R@N=0.16 vs. C6 R@N=0.17; C5 EPC=10% vs. C1 EPC=15%), so it is unclear whether the observed differences are reliable. Please report variance across repeated runs or apply an appropriate significance test, and state the number of held-out projects in the ground-truth set.
minor comments (6)
  1. [Section III-C and Section III-D] The paper uses "LoRa" where "LoRA" is standard, and "dicussed" should be "discussed."
  2. [Section V-A] The text refers to "the prompts defined in Section III-D," but the prompt templates and instructions are presented in Section III-B; Section III-D covers the penalty mechanism. Please fix the cross-reference.
  3. [Section III-B] The paper uses "few-shots" in most places but "few-shot" in Section III-B; please standardize the terminology.
  4. [Listing 4] The example output contains "org.jsonwebtoken:jwt-simple" and "com.nimbusds:oauth2," which appear to be malformed or incomplete Maven coordinates; please verify or correct the example.
  5. [Section IV-A, Eq. (6)] In Eq. (6), REC_r(p) is described as reflecting the "popularity of library at position r," but the notation is not formally introduced and is easy to confuse with R@N. Please define all symbols either in the text or in a notation table.
  6. [Table II] Table II lists fine-tuning hyperparameters, but configurations C1-C4 do not use fine-tuning. Please clarify which configurations actually use these settings, since the current layout suggests they apply to all rows of the ablation study.

Circularity Check

0 steps flagged · score 1.0 of 10

No load-bearing circularity; central negative result is empirical, with only a non-formal evaluation-alignment threat.

full rationale

The central claim is an empirical negative result from a six-configuration ablation, not a quantity derived from a fitted parameter. The popularity penalty (Eq. 1) is a fixed deterministic function of popularity rank and is not fitted to the evaluation metrics; fine-tuning is evaluated on the held-out 20% test split, so the reported precision and recall are genuine out-of-sample measurements. Self-citations to the CrossRec dataset, the prior popularity-bias study, and the README summarizer provide inputs and motivation, but none functions as a uniqueness theorem or as the derivation of the paper's conclusion, so they are not load-bearing in the circularity sense. The strongest concern is that instruction IST3 ('Avoid popular libraries') is present even in the baseline prompts, while Recall@N is computed against ground-truth dependencies extracted from the same corpus used to define the top-20 popular libraries (Fig. 3), which could cap achievable recall if the ground truth is popularity-saturated. However, the paper does not report the popularity composition of the test ground truth or the resulting recall ceiling, and a model told to avoid popular items can still be scored against non-popular ground-truth items; the measured recall values are therefore not forced by the metric definition alone. This is a construct-validity threat, not a circular step that can be exhibited as Eq. X = Eq. Y by construction. No circular step meets the required evidentiary bar.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The evaluation rests on one dataset, one popularity proxy, and one set of hand-written instructions. The top-20 cutoff and fine-tuning hyperparameters are free choices, and the ground-truth definition is left implicit. None of these are fitted to the target metric, so the circularity burden is low, but the fragility of the conclusions is high.

free parameters (3)
  • Top-20 popularity threshold = 20
    The data encoding step targets the top 20 most used libraries when defining the penalty and the avoid-popular instruction. No sensitivity analysis is provided, and the cutoff directly shapes both the intervention and the EPC metric. Section III-A.
  • LoRA and fine-tuning hyperparameters = rank 16, alpha 32, dropout 0.05, learning rate 2e-5, epochs 3, batch size 4
    These values are chosen without sensitivity analysis and only one fine-tuned configuration, C6, is evaluated, so results may be fragile to these choices. Section III-C and Table II.
  • N in all @N metrics = not reported
    Precision, recall, coverage, and EPC are all computed for a top-N list, but N is never stated, which makes the reported values impossible to interpret or reproduce. Section IV-A.
assumptions (4)
  • domain assumption The CrossRec dataset of GitHub Java projects is a valid and representative benchmark for TPL recommendation.
    It is the sole dataset used for filtering, training, and evaluation, with no cross-dataset validation. Section III-A.
  • domain assumption Maven usage frequency is a valid proxy for library popularity.
    The penalty score and the EPC metric both rely on popularity rank, but the paper does not validate this proxy against developer perceptions or task relevance. Sections III-D and IV-A.
  • domain assumption The actual dependencies of held-out projects form a correct ground truth for relevant recommendations.
    The evaluation appears to match recommendations against real dependencies, but the paper never describes how ground truth is built. If the ground truth contains popular libraries, the instruction to avoid them artificially lowers recall. Sections III-B and IV-B.
  • domain assumption Llama 2 and Llama 3 variants are representative of open-source LLMs for this task.
    The paper generalizes its negative conclusion to open-source LLMs while testing only three Llama variants. Section III-C.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Addressing Popularity Bias in Third-Party Library Recommendations Using LLMs." pith.science (2026). https://pith.science/paper/MZOVS2GA

@misc{pith2026250110313,
  author       = {Pith},
  title        = {Pith review of: Addressing Popularity Bias in Third-Party Library Recommendations Using LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MZOVS2GA}},
  note         = {Machine review of arXiv:2501.10313}
}
read the original abstract

Recommender systems for software engineering (RSSE) play a crucial role in automating development tasks by providing relevant suggestions according to the developer's context. However, they suffer from the so-called popularity bias, i.e., the phenomenon of recommending popular items that might be irrelevant to the current task. In particular, the long-tail effect can hamper the system's performance in terms of accuracy, thus leading to false positives in the provided recommendations. Foundation models are the most advanced generative AI-based models that achieve relevant results in several SE tasks. This paper aims to investigate the capability of large language models (LLMs) to address the popularity bias in recommender systems of third-party libraries (TPLs). We conduct an ablation study experimenting with state-of-the-art techniques to mitigate the popularity bias, including fine-tuning and popularity penalty mechanisms. Our findings reveal that the considered LLMs cannot address the popularity bias in TPL recommenders, even though fine-tuning and post-processing penalty mechanism contributes to increasing the overall diversity of the provided recommendations. In addition, we discuss the limitations of LLMs in this context and suggest potential improvements to address the popularity bias in TPL recommenders, thus paving the way for additional experiments in this direction.

Figures

Figures reproduced from arXiv: 2501.10313 by the authors.

Figure 2
Figure 2. Overview of the proposed approach ones. On the one hand, project-specific requirements must be con￾sidered by the recommender systems to provide more accurate recommendations. On the other hand, libraries that are well￾documented and supported by an active community are more likely to be adopted by developers. This leads to a situation where the most popular libraries are recommended while the less popular ones are … view at source ↗
Figure 3
Figure 3. Most popular TPLs in the dataset. reinforcement learning with human feedback (RLHF) to align with human preferences for helpfulness and safety. It is worth mentioning that we opt for small, optimized open-source models to allow easy deployment in local ma￾chine. To support the fine-tuning process, we first tokenized the pre-processed datasets as mentioned in Section III-A. After the tokenization phase, we adopt the … view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 31 canonical work pages

  1. [1]

    Algorithmic fairness datasets: the story so far,

    A. Fabris, S. Messina, G. Silvello et al. , “Algorithmic fairness datasets: the story so far,” Data Mining and Knowledge Discovery , vol. 36, no. 6, pp. 2074–2152, Nov. 2022. [Online]. Available: https://doi.org/10.1007/s10618-022-00854-z

    doi:10.1007/s10618-022-00854-z 2022

  2. [2]

    Social data: Biases, method- ological pitfalls, and ethical boundaries,

    A. Olteanu, C. Castillo, F. Diaz et al. , “Social data: Biases, method- ological pitfalls, and ethical boundaries,” Frontiers in big data , vol. 2, p. 13, 2019

    work page 2019

  3. [3]

    Software Engineering for Fairness: A Case Study with Hyperparameter Optimization

    J. Chakraborty, T. Xia, F. M. Fahid et al. , “Software Engineering for Fairness: A Case Study with Hyperparameter Optimization,” Oct. 2019, arXiv:1905.05786 [cs]. [Online]. Available: http://arxiv.org/abs/ 1905.05786

    work page Pith review arXiv 2019

  4. [4]

    Fairness-aware machine learning engineering: how far are we?

    C. Ferrara, G. Sellitto, F. Ferrucci et al. , “Fairness-aware machine learning engineering: how far are we?” Empirical Software Engineering, vol. 29, no. 1, p. 9, Nov. 2023. [Online]. Available: https: //doi.org/10.1007/s10664-023-10402-y

    doi:10.1007/s10664-023-10402-y 2023

  5. [6]

    [Online]

    European Commission, “Regulation of the european parliament and of the council laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts,” 2021. [Online]. Available: https://eur-lex.europa.eu/resource.html?uri=cellar: e0649735-a372-11eb-9585-01aa75ed71a1.0001.02/DOC 1&format= PDF

    work page 2021

  6. [7]

    Development of recommendation systems for software engineering: the CROSSMINER experience,

    J. Di Rocco, D. Di Ruscio, C. Di Sipio et al. , “Development of recommendation systems for software engineering: the CROSSMINER experience,” Empirical Software Engineering , vol. 26, no. 4, p. 69, Jul. 2021, publisher: Springer US. [Online]. Available: https: //doi.org/10.1007/s10664-021-09963-7

    work page doi:10.1007/s10664-021-09963-7 2021

  7. [8]

    M. P. Robillard, W. Maalej, R. J. Walker, and T. Zimmermann, Eds., Recommendation Systems in Software Engineering . Berlin, Heidelberg: Springer Berlin Heidelberg, 2014, dOI: 10.1007/978- 3-642-45135-5. [Online]. Available: http://link.springer.com/10.1007/ 978-3-642-45135-5

    doi:10.1007/978- 2014

  8. [9]

    Diversified third-party library prediction for mobile app development,

    Q. He, B. Li, F. Chen et al., “Diversified third-party library prediction for mobile app development,” IEEE Transactions on Software Engineering , pp. 1–1, 2020

    work page 2020

Show all 53 references
  1. [10]

    Libd: Scalable and precise third- party library detection in android markets,

    M. Li, W. Wang, P. Wang et al. , “Libd: Scalable and precise third- party library detection in android markets,” in 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE) , 2017, pp. 335–346

    2017

  2. [11]

    CrossRec: Supporting Software Developers by Recommending Third-party Libraries,

    P. T. Nguyen, J. Di Rocco, D. Di Ruscio et al., “CrossRec: Supporting Software Developers by Recommending Third-party Libraries,” Journal of Systems and Software, vol. 161, p. 110460, 2019. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0164121219302341

    2019

  3. [12]

    Improving reusability of software libraries through usage pattern mining,

    M. A. Saied, A. Ouni, H. Sahraoui et al. , “Improving reusability of software libraries through usage pattern mining,” Journal of Systems and Software , vol. 145, pp. 164–179, 2018. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0164121218301699

    2018

  4. [13]

    Automated library recommendation,

    F. Thung, D. Lo, and J. Lawall, “Automated library recommendation,” in 2013 20th Working Conference on Reverse Engineering (WCRE), Oct 2013, pp. 182–191

    2013

  5. [14]

    Managing popularity bias in recommender systems with personalized re-ranking,

    H. Abdollahpouri, R. Burke, and B. Mobasher, “Managing popularity bias in recommender systems with personalized re-ranking,” in Proceedings of the Thirty-Second International Florida Artificial Intelligence Research Society Conference, Sarasota, Florida, USA, May 19-22 2019 , ...

    2019

  6. [15]

    The unfairness of popularity bias in recommendation,

    H. Abdollahpouri, M. Mansoury, R. Burke et al. , “The unfairness of popularity bias in recommendation,” in Proceedings of the Workshop on Recommendation in Multi-stakeholder Environments co-located with the 13th ACM Conference on Recommender Systems (RecSys 2019), Copenhagen, ...

    2019

  7. [16]

    Bias and debias in recommender system: A survey and future directions,

    J. Chen, H. Dong, X. Wang et al. , “Bias and debias in recommender system: A survey and future directions,” ACM Trans. Inf. Syst., oct 2022, just Accepted. [Online]. Available: https://doi.org/10.1145/3564284

    2022 doi

  8. [17]

    Fairness in recommender systems: Research landscape and future directions,

    Y . Deldjoo, D. Jannach, A. Bellogin et al. , “Fairness in recommender systems: Research landscape and future directions,” pp. 59–108, Apr

  9. [18]

    A survey on popularity bias in recommender systems,

    A. Klimashevskaia, D. Jannach, M. Elahi et al. , “A survey on popularity bias in recommender systems,” User Modeling and User-Adapted Interaction , Jul. 2024. [Online]. Available: https: //doi.org/10.1007/s11257-024-09406-0

    2024 doi

  10. [19]

    Dealing with popularity bias in recommender systems for third-party libraries: How far are we?

    P. T. Nguyen, R. Rubei, J. Di Rocco et al. , “Dealing with popularity bias in recommender systems for third-party libraries: How far are we?” in 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR). IEEE, 2023, pp. 12–24

    2023

  11. [20]

    Anderson, The Long Tail: Why the Future of Business Is Selling Less of More

    C. Anderson, The Long Tail: Why the Future of Business Is Selling Less of More. Hyperion, 2006

    2006

  12. [21]

    Evaluating recommender systems,

    Z. Zaier, R. Godin, and L. Faucher, “Evaluating recommender systems,” in 2008 International Conference on Automated Solutions for Cross Media Content and Multi-Channel Distribution , 2008, pp. 211–217

    2008

  13. [22]

    Improving sales diversity by recommending users to items,

    S. Vargas and P. Castells, “Improving sales diversity by recommending users to items,” in Eighth ACM Conference on Recommender Systems, RecSys ’14, Foster City, Silicon Valley, CA, USA - October 06 - 10, 2014 , 2014, pp. 145–152. [Online]. Available: http://doi.acm.org/10.1145...

    2014

  14. [23]

    Large language models for software engineering: A systematic literature review,

    X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engineering: A systematic literature review,” ACM Trans. Softw. Eng. Methodol. , Sep. 2024, just Accepted. [Online]. Available: https://doi-org.univaq...

    2024 doi

  15. [24]

    An empirical study on the usage of transformer models for code completion,

    M. Ciniselli, N. Cooper, L. Pascarella et al., “An empirical study on the usage of transformer models for code completion,”IEEE Trans. Software Eng., vol. 48, no. 12, pp. 4818–4837, 2022

    2022

  16. [25]

    Studying the Usage of Text-To-Text Transfer Transformer to Support Code-Related Tasks,

    A. Mastropaolo, S. Scalabrino, N. Cooper et al. , “Studying the Usage of Text-To-Text Transfer Transformer to Support Code-Related Tasks,” in 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) . Madrid, ES: IEEE, May 2021, pp. 336–347. [Online]. Availab...

    2021

  17. [26]

    The Devil is in the Tails: How Long-Tailed Code Distributions Impact Large Language Models ,

    X. Zhou, K. Kim, B. Xu et al. , “ The Devil is in the Tails: How Long-Tailed Code Distributions Impact Large Language Models ,” in 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) . Los Alamitos, CA, USA: IEEE Computer Society, Sep. 2023, pp....

    2023

  18. [27]

    Empirical study of transformers for source code,

    N. Chirkova and S. Troshin, “Empirical study of transformers for source code,” in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , ser. ESEC/FSE 2021. New York, NY , USA: Associatio...

    2021

  19. [28]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone et al., “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288 , 2023

    2023 arXiv

  20. [29]

    Retrieval-augmented generation for knowledge-intensive nlp tasks,

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K¨uttler, M. Lewis, W.-t. Yih, T. Rockt¨aschel, S. Riedel, and D. Kiela, “Retrieval-augmented generation for knowledge-intensive nlp tasks,” in Proceedings of the 34th International Conference on Neural Inf...

    2020

  21. [30]

    Endowing third-party libraries recommender systems with explicit user feedback mechanisms,

    R. Rubei, C. Di Sipio, J. Di Rocco et al., “Endowing third-party libraries recommender systems with explicit user feedback mechanisms,” in 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), 2022, pp. 817–821

    2022

  22. [31]

    A survey on bias and fairness in machine learning,

    N. Mehrabi, F. Morstatter, N. Saxena, K. Lerman, and A. Galstyan, “A survey on bias and fairness in machine learning,” ACM Comput. Surv. , vol. 54, no. 6, Jul. 2021. [Online]. Available: https://doi-org.univaq.idm.oclc.org/10.1145/3457607

    2021 doi

  23. [32]

    Beyond accuracy: Evaluating recommender systems by coverage and serendipity,

    M. Ge, C. Delgado-Battenfeld, and D. Jannach, “Beyond accuracy: Evaluating recommender systems by coverage and serendipity,” in Proceedings of the Fourth ACM Conference on Recommender Systems , ser. RecSys ’10. New York, NY , USA: ACM, 2010, pp. 257–260. [Online]. Available: h...

  24. [33]

    Too long; didn’t read: Automatic summarization of github readme. md with transformers,

    T. T. Doan, P. T. Nguyen, J. Di Rocco, and D. Di Ruscio, “Too long; didn’t read: Automatic summarization of github readme. md with transformers,” in Proceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering, 2023, pp. 267–272

    2023

  25. [34]

    Process Modeling with Large Language Models,

    H. Kourani, A. Berti, D. Schuster et al., “Process Modeling with Large Language Models,” in Enterprise, Business-Process and Information Systems Modeling, H. van der Aa, D. Bork, R. Schmidt, and A. Sturm, Eds. Cham: Springer Nature Switzerland, 2024, pp. 229–244

    2024

  26. [35]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child et al., “Language models are unsupervised multitask learners,” in Open Access Library Journal , 2019. [Online]. Available: https://api.semanticscholar.org/CorpusID:160025533

    2019

  27. [36]

    Language models are few-shot learners,

    T. B. Brown, B. Mann, N. Ryder et al., “Language models are few-shot learners,” arXiv preprint arXiv:2005.14165 , 2020

    2005 arXiv

  28. [37]

    LoRA: Low-rank adaptation of large language models,

    E. J. Hu, yelong shen, P. Wallis et al. , “LoRA: Low-rank adaptation of large language models,” in International Conference on Learning Representations , 2022. [Online]. Available: https: //openreview.net/forum?id=nZeVKeeFYf9

    2022

  29. [38]

    EqBal-RS: Mitigating popularity bias in recommender systems,

    S. Gupta, K. Kaur, and S. Jain, “EqBal-RS: Mitigating popularity bias in recommender systems,” Journal of Intelligent Information Systems, vol. 62, no. 2, pp. 509–534, Apr. 2024, read Status: New Read Status Date: 2024-10-14T09:12:42.047Z. [Online]. Available: https://doi.org/...

    2024 doi

  30. [39]

    Debiaser for multiple variables to enhance fairness in classification tasks,

    G. d’Aloisio, A. D’Angelo, A. Di Marco et al., “Debiaser for multiple variables to enhance fairness in classification tasks,” Information Processing & Management , vol. 60, no. 2, p. 103226, Mar. 2023. [Online]. Available: https://www.sciencedirect.com/science/article/pii/ S03...

    2023

  31. [40]

    Automatic fairness testing of machine learning models,

    A. Sharma and H. Wehrheim, “Automatic fairness testing of machine learning models,” in Testing Software and Systems: 32nd IFIP WG 6.1 International Conference, ICTSS 2020, Naples, Italy, December 9–11, 2020, Proceedings . Berlin, Heidelberg: Springer- Verlag, 2020, p. 255–271....

    2020

  32. [41]

    Fairway: a way to build fair ML software,

    J. Chakraborty, S. Majumder, Z. Yu et al., “Fairway: a way to build fair ML software,” in ESEC/FSE ’20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event, USA, November 8-13, 2020 , ser. ESEC/FSE

    2020

  33. [42]

    Beyond words: On large language models actionability in mission-critical risk analysis,

    M. Esposito, F. Palagiano, V . Lenarduzzi, and D. Taibi, “Beyond words: On large language models actionability in mission-critical risk analysis,” in Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement , ser. ESEM ’24. New...

    2024

  34. [43]

    BRAID: an API recommender supporting implicit user feedback,

    Y . Zhou, H. Jin, X. Yang et al. , “BRAID: an API recommender supporting implicit user feedback,” in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering . Athens Greece: ACM, Aug. 2021, ...

    2021

  35. [44]

    ELIXIR: Learning from User Feedback on Explanations to Improve Recommender Models,

    A. Ghazimatin, S. Pramanik, R. S. Roy et al. , “ELIXIR: Learning from User Feedback on Explanations to Improve Recommender Models,” Proceedings of the Web Conference 2021 , pp. 3850– 3860, Apr. 2021, 00001 arXiv: 2102.09388. [Online]. Available: http://arxiv.org/abs/2102.09388

    2021 arXiv

  36. [45]

    Avalanche: an End-to-End Library for Continual Learning,

    V . Lomonaco, L. Pellegrini, A. Cossu et al. , “Avalanche: an End-to-End Library for Continual Learning,” arXiv:2104.00405 [cs] , Apr. 2021, 00003 arXiv: 2104.00405 version: 1. [Online]. Available: http://arxiv.org/abs/2104.00405

    2021 arXiv

  37. [46]

    Req2lib: A semantic neural model for software library recommendation,

    Z. Sun, Y . Liu, Z. Cheng et al. , “Req2lib: A semantic neural model for software library recommendation,” in 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER). Los Alamitos, CA, USA: IEEE Computer Society, feb 2020, pp. 542–546....

    2020

  38. [47]

    Embedding app-library graph for neural third party library recommendation,

    B. Li, Q. He, F. Chen et al. , “Embedding app-library graph for neural third party library recommendation,” in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , ser. ESEC/FSE 2021. N...

    2021

  39. [48]

    Mining likely analogical apis across third-party libraries via large-scale unsupervised api semantics embed- ding,

    C. Chen, Z. Xing, Y . Liu et al. , “Mining likely analogical apis across third-party libraries via large-scale unsupervised api semantics embed- ding,” IEEE Transactions on Software Engineering , vol. 47, no. 3, pp. 432–447, 2021

    2021

  40. [49]

    What’s spain’s paris? mining analogical libraries from q&a discussions,

    C. Chen, Z. Xing, and Y . Liu, “What’s spain’s paris? mining analogical libraries from q&a discussions,” Empirical Softw. Engg. , vol. 24, no. 3, p. 1155–1194, jun 2019. [Online]. Available: https://doi.org/10.1007/s10664-018-9657-y

    2019 doi

  41. [50]

    How scale affects structure in java programs,

    C. V . Lopes and J. Ossher, “How scale affects structure in java programs,” SIGPLAN Not. , vol. 50, no. 10, p. 675–694, Oct

  42. [51]

    Understanding the factors that impact the popularity of github repositories,

    H. Borges, A. Hora, and M. T. Valente, “Understanding the factors that impact the popularity of github repositories,” in 2016 IEEE International Conference on Software Maintenance and Evolution (ICSME), 2016, pp. 334–344

    2016

  43. [2015]

    Available: https://doi-org.univaq.idm.oclc.org/10.1145/ 2858965.2814300

    [Online]. Available: https://doi-org.univaq.idm.oclc.org/10.1145/ 2858965.2814300

  44. [2020]

    2020, pp

    New York, NY , USA: ACM, Nov. 2020, pp. 654–665. [Online]. Available: https://dl.acm.org/doi/10.1145/3368089.3409697

    2020

  45. [2022]

    Available: https://doi.org/10.1007/s11257-023-09364-z

    [Online]. Available: https://doi.org/10.1007/s11257-023-09364-z

    doi

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.