Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Developer Challenges on Large Language Models: A Study of Stack Overflow and OpenAI Developer Forum Posts

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims to derive a nine-topic taxonomy for Stack Overflow and a 17-topic taxonomy for the OpenAI Developer Forum, with LLM questions unresolved 79.03% of the time on Stack Overflow.

desk verdict Useful first combined taxonomy of LLM developer challenges, but the unthresholded 'openai' tag lets in OpenAI Gym posts and contaminates the headline numbers; needs a data-cleaning pass before the claims can be trusted. read the letter →

arxiv 2411.10873 v2 pith:UZFFQEZB submitted 2024-11-16 cs.SE

Khairul Alam , Kartik Mittal , Banani Roy , Chanchal Roy This is my paper · ORCID
classification cs.SE
keywords LargeLanguageModelsStackOverflowOpenAIDeveloperForumBERTopictopicmodelingchallengesquestiondifficultyQ&Acommunities
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to map the recurring practical problems developers hit when building with large language models, by mining what they actually ask on Stack Overflow and the OpenAI Developer Forum. It claims the questions cluster into nine Stack Overflow topics and 17 forum topics, from API usage and fine-tuning to agent-tool interactions and GPU indexing. It further claims these questions are measurably harder than established software fields: 79.03% of the Stack Overflow posts have no accepted answer, and the median time to a first answer is 19.88 hours overall. A sympathetic reader would care because the resulting topic list and difficulty ranking suggest where documentation, tools, and community support are missing.

What carries the argument

The load-bearing machinery is a pipeline that turns community posts into labeled challenge topics. The paper expands a seed 'large-language-model' tag using co-occurrence thresholds (TRT and TST), then applies BERTopic with HDBSCAN clustering and a Multi-QA sentence transformer to embed and cluster post texts, then has three annotators label each cluster. Difficulty is measured by the percentage of posts without accepted answers and median response time on Stack Overflow, and by reply-count distribution on the OpenAI forum; the same tag-selection and difficulty recipe is reused across comparison fields.

What would settle it

Take a random sample of the 606 posts in the 'Environment Management' topic (keywords: gym, reward, game) and check how many actually discuss large language models rather than OpenAI Gym reinforcement-learning environments. If a substantial share are non-LLM posts, then the tag-expansion step contaminated the dataset, and the claimed topic list and 79.03% unresolved figure would not be a valid measure of LLM developer challenges.

Watch

Extended reading notes

Core claim

The paper's central claim is that LLM development has a recognizable, recurring challenge landscape that can be recovered from community Q&A data. Using BERTopic on 8,593 Stack Overflow posts and 26,474 OpenAI Developer Forum posts, it identifies nine topics on Stack Overflow and 17 on the forum; the most prominent are LLM Ecosystem and Challenges and API Usage and Error Handling, respectively. The paper also claims difficulty is high and uneven: Agents and Tool Interactions has 90.63% of posts without accepted answers, Llama Indexing and GPU Utilization has the longest median response time at 55.76 hours, and overall LLM questions lag mature fields such as mobile and security on resolution speed and acceptance rate.

Load-bearing premise

The taxonomy and difficulty ranking collapse if the tag-expansion step mixes non-LLM posts into the dataset, as the added 'openai' tag may do by capturing OpenAI Gym reinforcement-learning discussions.

Editorial extensions

If this is right

  • API usage and error handling is the dominant challenge on both platforms, so better API documentation and debugging guides would address the largest share of developer pain.
  • Specialized topics, especially Agents and Tool Interactions and Llama Indexing and GPU Utilization, will need targeted support because community answers are slowest and rarest there.
  • LLM questions are harder to resolve than mobile, security, web, and big-data questions, implying the field still lacks the accumulated expertise mature ecosystems have.
  • The nine and 17 topic taxonomies give educators and tool builders a concrete checklist of where LLM developers get stuck.
  • The strong negative correlation between a topic's average views and its unresolved rate suggests that drawing attention to a topic may help it get answered, so community visibility tools could reduce difficulty.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the dataset were cleaned to exclude OpenAI Gym posts, the Environment Management topic could shrink or disappear, and the remaining percentages might shift.
  • The forum's question-type distribution was produced with GPT-4o classification plus manual validation; re-running with a different classifier or annotation scheme could yield a different breakdown, especially for Ethical and Best Practices and Maintenance categories.
  • Because the two platforms use different difficulty metrics (accepted answers and timings versus reply counts), the paper's cross-platform difficulty comparison is indirect; a direct comparison would require applying the same metric to both.
  • A testable extension would be to track the same topic labels over the next year to see whether resolution rates improve as LLM tooling matures, which would test the claim that difficulty reflects the field's newness.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper presents an empirical study of the challenges faced by developers working with large language models (LLMs). The authors collect 8,593 Stack Overflow posts via an LLM-related tag set and 26,474 posts from the OpenAI Developer Forum, apply BERTopic to extract topics (9 on Stack Overflow, 17 on the forum), classify question types, and measure topic difficulty using percentage of posts without accepted answers and median response time (Stack Overflow) and reply counts (forum). They report that LLM questions are often difficult, with 79.03% of Stack Overflow posts unresolved, and compare these metrics with other software engineering fields. The central contribution is the topic taxonomy and the difficulty ranking.

Significance. The study addresses a timely and relevant question and combines two complementary data sources, which is a strength. It builds on established methods from prior MSR studies (TRT/TST tag selection, BERTopic, Cohen's Kappa for the Stack Overflow classification), and the data tables are detailed. If the corpus were clean and the classification validated, the resulting taxonomy would be a useful resource for researchers and practitioners. However, the trustworthiness of the central claims depends on the corpus construction, and the analysis currently has a contamination problem in the Stack Overflow data that directly affects the taxonomy and difficulty figures.

major comments (3)
  1. [3.1 Step 2, Table 2 (topic 3)] The 'openai' tag is added to the extraction tag set without applying the TRT and TST thresholds used for every other tag. On Stack Overflow, 'openai' also covers OpenAI Gym, a reinforcement-learning environment library, and the paper's own Table 2 topic 3 ('Environment Management') confirms this: its keywords include 'gym', 'reward', 'game', 'reinforcement', and the accompanying text discusses examples such as 'OpenAI Gym' and 'mujoco-py'. These 606 posts are not LLM development posts, yet they are included in the 8,593-post corpus that generates the nine-topic taxonomy, the difficulty metrics in Table 6, the 79.03% unresolved figure in Table 9, and the cross-field comparison in Section 5.2. Because the corpus is the input to every downstream analysis, the authors must either apply the same thresholds to 'openai' or explicitly filter out known non-LLM subtags such as 'openai-gym', and then re-run the analysis to show that the substantive conclusions are unchanged.
  2. [4.2, Table 5] The eight-type classification of OpenAI Developer Forum posts is performed with GPT-4o, but the manuscript reports no validation statistics for this step, in contrast to the Cohen's Kappa of 0.87 reported for the Stack Overflow question-type classification. The text only says that a manual validation process was used. Because Table 5's percentages support the claim that Troubleshooting dominates the OpenAI forum and are compared with the Stack Overflow question types, the authors should report the size of the validation sample and the agreement or accuracy of the GPT-4o labels against human annotation.
  3. [5.2, Table 9] The cross-field comparison relies on 'reconstructed' datasets from prior studies, but the reconstruction procedure is not described (e.g., which tags, date ranges, and filtering were used for each field), and the reported post counts appear inconsistent with the original studies' descriptions (for example, WebApp is listed as 14,596 posts while Mobile is 2,502,811, without explanation). The conclusion that LLM challenges are harder than mobile and security depends on this comparison, so the authors should make the reconstruction explicit and transparent, or soften the comparative claims.
minor comments (6)
  1. [3.1 Step 1 and Section 6] The tag name is inconsistently written as 'large-language-model' (Section 3.1) and 'large-language-models' (Section 6); the authors should use one consistent name.
  2. [3.1 Step 5] The sentence 'our main topic begins with the label -1' is confusing; the authors should clarify whether the BERTopic outlier topic (-1) is excluded from the nine reported topics and how many posts it contains.
  3. [4.1] The paragraph titled 'Function Calling' appears after the 'RQ1 Summary' and seems to be a misplaced portion of the OpenAI forum topic descriptions; it should be moved into the earlier topic list or integrated with Table 3.
  4. [4.2] The sample size for the OpenAI forum classification is not reported; the authors give the total of 1,918 posts for Stack Overflow but only say a representative sample was drawn for the forum, so the reader cannot assess the statistical reliability of Table 5.
  5. [Table 2] The keyword list for topic 1 ('LLM Ecosystem and Challenges') contains very generic terms ('use', 'model', 'try', 'error'), and the topic might be a catch-all; the authors should discuss the coherence of this large topic (4,344 posts) in more detail.
  6. [Abstract] The abstract contains a typo: 'Topic Modeling,, Developer Challenges' with a double comma.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: this is a descriptive empirical study whose difficulty and popularity metrics are external platform signals, not fitted or derived from the claim.

full rationale

The paper makes no mathematical derivation or prediction; it is a descriptive mining study. The central outputs are a topic taxonomy from BERTopic, question-type distributions from manual annotation, and difficulty measures (percentage of posts without accepted answers, median time to answer, reply counts, average views, average score). These metrics are independent external platform signals, not quantities fitted to the conclusions. The TRT/TST tag-selection thresholds in Section 3.1 are inputs to corpus construction and are not reused to define any reported difficulty or popularity result, so the headline 79.03% unresolved rate and topic-level metrics are not equal to the tag-selection criteria by construction. The addition of the 'openai' tag without the thresholds is a corpus-definition choice; it can contaminate the dataset (as the paper's own Table 2 topic 'Environment Management' suggests, with keywords 'gym', 'reward', 'game'), but contamination is an external-validity threat rather than a circular reasoning step. No load-bearing self-citations appear: the method citations (e.g., [22], [23], [25], [29]) are prior independent studies, and the paper does not invoke any author-derived uniqueness theorem or ansatz to force its results. Consequently, no step in the derivation chain reduces to its own inputs, and the appropriate circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on data collection and measurement choices: tag thresholds, the inclusion of the 'openai' tag, topic model hyperparameters, and the reply-count difficulty proxy. These are not derived or independently validated.

free parameters (3)
  • TRT threshold = 11%
    Tags with TRT greater than 11% are included; the threshold was determined by three reviewers after inspecting samples, not prespecified.
  • TST threshold = 0.11%
    Tags with TST greater than 0.11% are included; the threshold was chosen by manual review and affects dataset composition.
  • BERTopic/HDBSCAN parameters
    Parameters such as min_topic_size and cluster selection are not reported, yet they determine the number and granularity of topics (9 and 17).
assumptions (4)
  • domain assumption Posts tagged with the 'openai' tag on Stack Overflow are LLM-related
    Including 'openai' in the tag set captures OpenAI Gym and other non-LLM posts; Section 3.1 Step 2.
  • domain assumption TRT and TST thresholds correctly separate LLM-related from unrelated tags
    Thresholds were chosen by manual review with no held-out validation, so the separation is not independently verified.
  • domain assumption Reply count on the OpenAI Developer Forum is a valid proxy for topic difficulty
    Assumed in RQ3 and cited to [29]; no validation is provided for this dataset.
  • domain assumption BERTopic with HDBSCAN produces coherent, meaningful topics for short technical Q&A posts
    No topic coherence scores or external benchmarks are reported to support this assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Developer Challenges on Large Language Models: A Study of Stack Overflow and OpenAI Developer Forum Posts." pith.science (2026). https://pith.science/paper/UZFFQEZB

@misc{pith2026241110873,
  author       = {Pith},
  title        = {Pith review of: Developer Challenges on Large Language Models: A Study of Stack Overflow and OpenAI Developer Forum Posts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UZFFQEZB}},
  note         = {Machine review of arXiv:2411.10873}
}
read the original abstract

Large Language Models (LLMs) have gained widespread popularity due to their exceptional capabilities across various domains, including chatbots, healthcare, education, content generation, and automated support systems. However, developers encounter numerous challenges when implementing, fine-tuning, and integrating these models into real-world applications. This study investigates LLM developers' challenges by analyzing community interactions on Stack Overflow and OpenAI Developer Forum, employing BERTopic modeling to identify and categorize developer discussions. Our analysis yields nine challenges on Stack Overflow (e.g., LLM Ecosystem and Challenges, API Usage, LLM Training with Frameworks) and 17 on the OpenAI Developer Forum (e.g., API Usage and Error Handling, Fine-Tuning and Dataset Management). Results indicate that developers frequently turn to Stack Overflow for implementation guidance, while OpenAI's forum focuses on troubleshooting. Notably, API and functionality issues dominate discussions on the OpenAI forum, with many posts requiring multiple responses, reflecting the complexity of LLM-related problems. We find that LLM-related queries often exhibit great difficulty, with a substantial percentage of unresolved posts (e.g., 79.03\% on Stack Overflow) and prolonged response times, particularly for complex topics like 'Llama Indexing and GPU Utilization' and 'Agents and Tool Interactions'. In contrast, established fields like Mobile Development and Security enjoy quicker resolutions and stronger community engagement. These findings highlight the need for improved community support and targeted resources to assist LLM developers in overcoming the evolving challenges of this rapidly growing field. This study provides insights into areas of difficulty, paving the way for future research and tool development to better support the LLM developer community.

Figures

Figures reproduced from arXiv: 2411.10873 by the authors.

Figure 1
Figure 1. Overview of the methodology of our study [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Types Distribution of SO Posts domains, demonstrating a strong preference for practical and procedural inquiries. For example, ’How’ questions make up 61.8% of chatbot-related posts [23] and 59% of mobile development posts [22], closely aligning with the 59.95% observed in our domain. The second most common type in chatbot and mobile development topics is ’Why’ (25.4% and 29%, respectively), mirroring our findings, … view at source ↗
Figure 3
Figure 3. Relative Growth of LLM-related posts over time in Stack Overflow [PITH_FULL_IMAGE:figures/full_fig_p035_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: LLM Topic Evolution Over Time in Stack Overflow [PITH_FULL_IMAGE:figures/full_fig_p036_4.png]
Figure 5
Figure 5. Figure 5: Relative Growth of LLM-related posts over time in OpenAI Developer Forum [PITH_FULL_IMAGE:figures/full_fig_p037_5.png]
Figure 6
Figure 6. Figure 6: LLM Topic Evolution Over Time in OpenAI Developer Forum [PITH_FULL_IMAGE:figures/full_fig_p038_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. An Empirical Study of OpenAI API Discussions on Stack Overflow

    cs.SE 2025-05 conditional novelty 4.0 of 10

    A taxonomy of developer challenges for OpenAI APIs derived from 2,874 Stack Overflow discussions, with popularity trends, difficulty metrics, and implications for vendors and researchers.

Reference graph

Works this paper leans on

141 extracted references · 47 canonical work pages · cited by 1 Pith paper

  1. [1]

    Vaswani, Attention is all you need, Advances in Neural Information Processing Systems (2017)

    A. Vaswani, Attention is all you need, Advances in Neural Information Processing Systems (2017)

    2017

  2. [2]

    T. B. Brown, Language models are few-shot learners, arXiv preprint arXiv:2005.14165 (2020)

    arXiv 2020

  3. [3]

    W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, et al., A survey of large language models, arXiv preprint arXiv:2303.18223 (2023)

    arXiv 2023

  4. [4]

    J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, et al., Emergent abilities of large language models, arXiv preprint arXiv:2206.07682 (2022)

    arXiv 2022

  5. [5]

    X. Wang, G. H. Chen, D. Song, Z. Zhang, Z. Chen, Q. Xiao, F. Jiang, J. Li, X. Wan, B. Wang, et al., Cmb: A comprehensive medical benchmark in chinese, arXiv preprint arXiv:2308.08833 (2023)

    arXiv 2023

  6. [6]

    Bubeck, V

    S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, et al., Sparks of artificial general intelligence: Early experiments with gpt-4, arXiv preprint arXiv:2303.12712 (2023)

    arXiv 2023

  7. [7]

    Chang, X

    Y. Chang, X. Wang, J. Wang, Y. Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y. Wang, et al., A survey on evaluation of large language models, ACM Transactions on Intelligent Systems and Technology 15 (3) (2024) 1–45

    2024

  8. [8]

    Z. A. Nazi, W. Peng, Large language models in healthcare and medical domain: A review, in: Informatics, Vol. 11, MDPI, 2024, p. 57

    2024

Show all 141 references
  1. [9]

    H. Zhao, Z. Liu, Z. Wu, Y. Li, T. Yang, P. Shu, S. Xu, H. Dai, L. Zhao, G. Mai, et al., Revolutionizing finance with llms: An overview of applications and insights, arXiv preprint arXiv:2401.11641 (2024)

    2024

  2. [10]

    Kasneci, K

    E. Kasneci, K. Seßler, S. K¨ uchemann, M. Bannert, D. Dementieva, F. Fischer, U. Gasser, G. Groh, S. G¨ unnemann, E. H¨ ullermeier, et al., Chatgpt for good? on opportunities and challenges of large language models for education, Learning and individual differences 103 (2023) 102274

    2023

  3. [11]

    Floridi, M

    L. Floridi, M. Chiriatti, Gpt-3: Its nature, scope, limits, and consequences, Minds and Machines 30 (2020) 681–694

    2020

  4. [12]

    Zhang, Z

    S. Zhang, Z. Chen, Y. Shen, M. Ding, J. B. Tenenbaum, C. Gan, Planning with large language models for code generation, arXiv preprint arXiv:2303.05510 (2023)

    2023 arXiv

  5. [13]

    J. Liu, C. S. Xia, Y. Wang, L. Zhang, Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation, Advances in Neural Information Processing Systems 36 (2024)

    2024

  6. [14]

    J. Yu, P. Liang, Y. Fu, A. Tahir, M. Shahin, C. Wang, Y. Cai, Security code review by llms: A deep dive into responses, arXiv preprint arXiv:2401.16310 (2024). 42

    2024 arXiv

  7. [15]

    D. Nam, A. Macvean, V. Hellendoorn, B. Vasilescu, B. Myers, Using an llm to help with code understanding, in: Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, 2024, pp. 1–13

    2024

  8. [16]

    Y. Su, C. Wan, U. Sethi, S. Lu, M. Musuvathi, S. Nath, Hotgpt: How to make software documentation more useful with a large language model?, in: Proceedings of the 19th Workshop on Hot Topics in Operating Systems, 2023, pp. 87–93

    2023

  9. [17]

    R. Pan, A. R. Ibrahimzada, R. Krishna, D. Sankar, L. P. Wassi, M. Merler, B. Sobolev, R. Pavuluri, S. Sinha, R. Jabbarvand, Understanding the effective- ness of large language models in code translation, arXiv preprint arXiv:2308.03109 (2023)

    2023 arXiv

  10. [18]

    Z. Yang, F. Liu, Z. Yu, J. W. Keung, J. Li, S. Liu, Y. Hong, X. Ma, Z. Jin, G. Li, Exploring and unleashing the power of large language models in automated code translation, Proceedings of the ACM on Software Engineering 1 (FSE) (2024) 1585–1608

    2024

  11. [19]

    Mamykina, B

    L. Mamykina, B. Manoim, M. Mittal, G. Hripcsak, B. Hartmann, Design lessons from the fastest q&a site in the west, in: Proceedings of the SIGCHI conference on Human factors in computing systems, 2011, pp. 2857–2866

    2011

  12. [20]

    J. Son, B. Kim, Trend analysis of large language models through a developer com- munity: A focus on stack overflow, Information 14 (11) (2023) 602

    2023

  13. [21]

    Barua, S

    A. Barua, S. W. Thomas, A. E. Hassan, What are developers talking about? an analysis of topics and trends in stack overflow, Empirical software engineering 19 (2014) 619–654

    2014

  14. [22]

    Rosen, E

    C. Rosen, E. Shihab, What are mobile developers asking about? a large scale study using stack overflow, ESE 21 (2016)

    2016

  15. [23]

    Abdellatif, D

    A. Abdellatif, D. Costa, K. Badran, R. Abdalkareem, E. Shihab, Challenges in chatbot development: A study of stack overflow posts, in: MSR, 2020, pp. 174– 185

    2020

  16. [24]

    Treude, O

    C. Treude, O. Barzilay, M.-A. Storey, How do programmers ask and answer ques- tions on the web?(nier track), in: Proceedings of the 33rd international conference on software engineering, 2011, pp. 804–807

    2011

  17. [25]

    Bagherzadeh, R

    M. Bagherzadeh, R. Khatchadourian, Going big: a large-scale study on what big data developers ask, in: Proceedings of the 2019 27th ACM joint meeting on euro- pean software engineering conference and symposium on the foundations of software engineering, 2019, pp. 432–442

    2019

  18. [26]

    X.-L. Yang, D. Lo, X. Xia, Z.-Y. Wan, J.-L. Sun, What security questions do developers ask? a large-scale study of stack overflow posts, Journal of Computer Science and Technology 31 (2016) 910–924

    2016

  19. [27]

    H. Li, F. Khomh, M. Openja, et al., Understanding quantum software engineer- ing challenges an empirical study on stack exchange forums and github issues, in: ICSME, IEEE, IEEE, 2021, pp. 343–354. 43

    2021

  20. [28]

    G. L. Scoccia, P. Migliarini, M. Autili, Challenges in developing desktop web apps: a study of stack overflow and github, in: 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR), IEEE, 2021, pp. 271–282

    2021

  21. [29]

    X. Chen, C. Gao, C. Chen, G. Zhang, Y. Liu, An empirical study on challenges for llm developers, arXiv preprint arXiv:2408.05002 (2024)

    2024 arXiv

  22. [30]

    Kaplan, S

    J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, D. Amodei, Scaling laws for neural language models, arXiv preprint arXiv:2001.08361 (2020)

    2020 arXiv

  23. [31]

    Shanahan, Talking about large language models, Communications of the ACM 67 (2) (2024) 68–79

    M. Shanahan, Talking about large language models, Communications of the ACM 67 (2) (2024) 68–79

    2024

  24. [32]

    Achiam, S

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al., Gpt-4 technical report, arXiv preprint arXiv:2303.08774 (2023)

    2023 arXiv

  25. [33]

    Chowdhery, S

    A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al., Palm: Scaling language modeling with pathways, Journal of Machine Learning Research 24 (240) (2023) 1–113

    2023

  26. [34]

    Touvron, T

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi` ere, N. Goyal, E. Hambro, F. Azhar, et al., Llama: Open and efficient foundation language models, arXiv preprint arXiv:2302.13971 (2023)

    2023 arXiv

  27. [35]

    Naseem, I

    U. Naseem, I. Razzak, S. K. Khan, M. Prasad, A comprehensive survey on word rep- resentation models: From classical to state-of-the-art word representation language models, Transactions on Asian and Low-Resource Language Information Processing 20 (5) (2021) 1–35

    2021

  28. [36]

    B. Min, H. Ross, E. Sulem, A. P. B. Veyseh, T. H. Nguyen, O. Sainz, E. Agirre, I. Heintz, D. Roth, Recent advances in natural language processing via large pre- trained language models: A survey, ACM Computing Surveys 56 (2) (2023) 1–40

    2023

  29. [37]

    C. Zhou, Q. Li, C. Li, J. Yu, Y. Liu, G. Wang, K. Zhang, C. Ji, Q. Yan, L. He, et al., A comprehensive survey on pretrained foundation models: A history from bert to chatgpt, arXiv preprint arXiv:2302.09419 (2023)

    2023 arXiv

  30. [38]

    A. J. Thirunavukarasu, D. S. J. Ting, K. Elangovan, L. Gutierrez, T. F. Tan, D. S. W. Ting, Large language models in medicine, Nature medicine 29 (8) (2023) 1930–1940

    2023

  31. [39]

    Huang, S

    J. Huang, S. S. Gu, L. Hou, Y. Wu, X. Wang, H. Yu, J. Han, Large language models can self-improve, arXiv preprint arXiv:2210.11610 (2022)

    2022 arXiv

  32. [40]

    Kaddour, J

    J. Kaddour, J. Harris, M. Mozes, H. Bradley, R. Raileanu, R. McHardy, Challenges and applications of large language models, arXiv preprint arXiv:2307.10169 (2023)

    2023 arXiv

  33. [41]

    Brants, A

    T. Brants, A. Popat, P. Xu, F. J. Och, J. Dean, Large language models in machine translation, in: Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), 2007, pp. 858–867. 44

    2007

  34. [42]

    L. Wu, Z. Zheng, Z. Qiu, H. Wang, H. Gu, T. Shen, C. Qin, C. Zhu, H. Zhu, Q. Liu, et al., A survey on large language models for recommendation, World Wide Web 27 (5) (2024) 60

    2024

  35. [43]

    Vaithilingam, T

    P. Vaithilingam, T. Zhang, E. L. Glassman, Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models, in: Chi conference on human factors in computing systems extended abstracts, 2022, pp. 1–7

    2022

  36. [44]

    Jiang, Y

    X. Jiang, Y. Dong, L. Wang, Z. Fang, Q. Shang, G. Li, Z. Jin, W. Jiao, Self-planning code generation with large language models, ACM Transactions on Software Engi- neering and Methodology 33 (7) (2024) 1–30

    2024

  37. [45]

    Poesia, O

    G. Poesia, O. Polozov, V. Le, A. Tiwari, G. Soares, C. Meek, S. Gulwani, Syn- chromesh: Reliable code generation from pre-trained language models, arXiv preprint arXiv:2201.11227 (2022)

    2022 arXiv

  38. [46]

    Y. Liu, T. Le-Cong, R. Widyasari, C. Tantithamthavorn, L. Li, X.-B. D. Le, D. Lo, Refining chatgpt-generated code: Characterizing and mitigating code quality issues, ACM Transactions on Software Engineering and Methodology 33 (5) (2024) 1–26

    2024

  39. [47]

    G. Weng, A. Andrzejak, Automatic bug fixing via deliberate problem solving with large language models, in: 2023 IEEE 34th International Symposium on Software Reliability Engineering Workshops (ISSREW), IEEE, 2023, pp. 34–36

    2023

  40. [48]

    Bouzenia, P

    I. Bouzenia, P. Devanbu, M. Pradel, Repairagent: An autonomous, llm-based agent for program repair, arXiv preprint arXiv:2403.17134 (2024)

    2024 arXiv

  41. [49]

    M. Jin, S. Shahriar, M. Tufano, X. Shi, S. Lu, N. Sundaresan, A. Svyatkovskiy, In- ferfix: End-to-end program repair with llms, in: Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2023, pp. 1646–1656

    2023

  42. [50]

    C. S. Xia, L. Zhang, Keep the conversation going: Fixing 162 out of 337 bugs for $0.42 each using chatgpt, arXiv preprint arXiv:2304.00385 (2023)

    2023 arXiv

  43. [51]

    X. Du, Z. Liu, C. Li, X. Ma, Y. Li, X. Wang, Llm-brc: A large language model- based bug report classification framework, Software Quality Journal (2024) 1–21

    2024

  44. [52]

    Kumar, S

    A. Kumar, S. Haiduc, P. P. Das, P. P. Chakrabarti, Llms as evaluators: A novel approach to evaluate bug report summarization, arXiv preprint arXiv:2409.00630 (2024)

    2024 arXiv

  45. [53]

    S. Kang, J. Yoon, S. Yoo, Large language models are few-shot testers: Explor- ing llm-based general bug reproduction, in: 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), IEEE, 2023, pp. 2312–2323

    2023

  46. [54]

    H. Li, Y. Hao, Y. Zhai, Z. Qian, Enhancing static analysis for practical bug de- tection: An llm-integrated approach, Proceedings of the ACM on Programming Languages 8 (OOPSLA1) (2024) 474–499

    2024

  47. [55]

    Zhang, I

    T. Zhang, I. C. Irsan, F. Thung, D. Lo, Cupid: Leveraging chatgpt for more accu- rate duplicate bug report detection, arXiv preprint arXiv:2308.10022 (2023). 45

    2023 arXiv

  48. [56]

    Z. Liu, C. Chen, J. Wang, M. Chen, B. Wu, Z. Tian, Y. Huang, J. Hu, Q. Wang, Testing the limits: Unusual text inputs generation for mobile app crash detection with large language model, in: Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, 2024...

    2024

  49. [57]

    J. A. Pizzorno, E. D. Berger, Coverup: Coverage-guided llm-based test generation, arXiv preprint arXiv:2403.16218 (2024)

    2024 arXiv

  50. [58]

    Y. Chen, Z. Hu, C. Zhi, J. Han, S. Deng, J. Yin, Chatunitest: A framework for llm- based test generation, in: Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, 2024, pp. 572–576

    2024

  51. [59]

    G. Lu, X. Ju, X. Chen, W. Pei, Z. Cai, Grace: Empowering llm-based software vulnerability detection with graph structure and in-context learning, Journal of Systems and Software 212 (2024) 112031

    2024

  52. [60]

    Akuthota, R

    V. Akuthota, R. Kasula, S. T. Sumona, M. Mohiuddin, M. T. Reza, M. M. Rahman, Vulnerability detection and monitoring using llm, in: 2023 IEEE 9th International Women in Engineering (WIE) Conference on Electrical and Computer Engineering (WIECON-ECE), IEEE, 2023, pp. 309–314

    2023

  53. [61]

    C. Chen, J. Su, J. Chen, Y. Wang, T. Bi, J. Yu, Y. Wang, X. Lin, T. Chen, Z. Zheng, When chatgpt meets smart contract vulnerability detection: How far are we?, ACM Transactions on Software Engineering and Methodology (2023)

    2023

  54. [62]

    M. Fu, C. K. Tantithamthavorn, V. Nguyen, T. Le, Chatgpt for vulnerability detec- tion, classification, and repair: How far are we?, in: 2023 30th Asia-Pacific Software Engineering Conference (APSEC), IEEE, 2023, pp. 632–636

    2023

  55. [63]

    Ahmed, P

    T. Ahmed, P. Devanbu, Few-shot training llms for project-specific code- summarization, in: Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, 2022, pp. 1–5

    2022

  56. [64]

    Ahmed, K

    T. Ahmed, K. S. Pai, P. Devanbu, E. Barr, Automatic semantic augmentation of language model prompts (for code summarization), in: Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, 2024, pp. 1– 13

    2024

  57. [65]

    Ullah, A

    E. Ullah, A. Parwani, M. M. Baig, R. Singh, Challenges and barriers of using large language models (llm) such as chatgpt for diagnostic medicine with a focus on digital pathology–a recent scoping review, Diagnostic pathology 19 (1) (2024) 43

    2024

  58. [66]

    Patil, V

    R. Patil, V. Gudivada, A review of current trends, techniques, and challenges in large language models (llms), Applied Sciences 14 (5) (2024) 2074

    2024

  59. [67]

    Jelodar, Y

    H. Jelodar, Y. Wang, C. Yuan, X. Feng, X. Jiang, Y. Li, L. Zhao, Latent dirichlet allocation (lda) and topic modeling: models, applications, a survey, Multimedia Tools and Applications 78 (2019) 15169–15211

    2019

  60. [68]

    Isoaho, D

    K. Isoaho, D. Gritsenko, E. M¨ akel¨ a, Topic modeling and text analysis for qualitative policy research, Policy Studies Journal 49 (1) (2021) 300–324. 46

    2021

  61. [69]

    X. Yi, J. Allan, A comparative study of utilizing topic models for information retrieval, in: Advances in Information Retrieval: 31th European Conference on IR Research, ECIR 2009, Toulouse, France, April 6-9, 2009. Proceedings 31, Springer, 2009, pp. 29–41

    2009

  62. [70]

    Wei, Topic models in information retrieval, University of Massachusetts Amherst, 2007

    X. Wei, Topic models in information retrieval, University of Massachusetts Amherst, 2007

    2007

  63. [71]

    C.-K. Yau, A. Porter, N. Newman, A. Suominen, Clustering scientific documents with topic modeling, Scientometrics 100 (2014) 767–786

    2014

  64. [72]

    P. Xie, E. P. Xing, Integrating document clustering and topic modeling, arXiv preprint arXiv:1309.6874 (2013)

    2013 arXiv

  65. [73]

    Luostarinen, O

    T. Luostarinen, O. Kohonen, Using topic models in content-based news recom- mender systems, in: Proceedings of the 19th Nordic conference of computational linguistics (NODALIDA 2013), 2013, pp. 239–251

    2013

  66. [74]

    S. Choi, Y. Hyun, N. Kim, Improving performance of recommendation systems using topic modeling, Journal of Intelligence and Information systems 21 (3) (2015) 101–116

    2015

  67. [75]

    Bergamaschi, L

    S. Bergamaschi, L. Po, Comparing lda and lsa topic models for content-based movie recommendation systems, in: Web Information Systems and Technologies: 10th In- ternational Conference, WEBIST 2014, Barcelona, Spain, April 3-5, 2014, Revised Selected Papers 10, Springer, 2015, ...

    2014

  68. [76]

    R. C. Belwal, S. Rai, A. Gupta, Extractive text summarization using clustering- based topic modeling, Soft Computing 27 (7) (2023) 3965–3982

    2023

  69. [77]

    Eidelman, J

    V. Eidelman, J. Boyd-Graber, P. Resnik, Topic models for dynamic translation model adaptation, in: Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2012, pp. 115–119

    2012

  70. [78]

    H. U. Asuncion, A. U. Asuncion, R. N. Taylor, Software traceability with topic modeling, in: Proceedings of the 32nd ACM/IEEE international conference on Software Engineering-Volume 1, 2010, pp. 95–104

    2010

  71. [79]

    C. C. Silva, M. Galster, F. Gilson, Topic modeling in software engineering research, Empirical Software Engineering 26 (6) (2021) 120

    2021

  72. [80]

    Z. Zhai, B. Liu, H. Xu, P. Jia, Constrained lda for grouping product features in opinion mining, in: Advances in Knowledge Discovery and Data Mining: 15th Pacific-Asia Conference, PAKDD 2011, Shenzhen, China, May 24-27, 2011, Pro- ceedings, Part I 15, Springer, 2011, pp. 448–459

    2011

  73. [81]

    T.-H. Chen, S. W. Thomas, M. Nagappan, A. E. Hassan, Explaining software de- fects using topic models, in: 2012 9th IEEE working conference on mining software repositories (MSR), IEEE, 2012, pp. 189–198

    2012

  74. [82]

    T.-H. Chen, S. W. Thomas, A. E. Hassan, A survey on the use of topic models when mining software repositories, Empirical Software Engineering 21 (2016) 1843–1919. 47

    2016

  75. [83]

    S. W. Thomas, Mining software repositories using topic models, in: Proceedings of the 33rd International Conference on Software Engineering, 2011, pp. 1138–1139

    2011

  76. [84]

    S. W. Thomas, B. Adams, A. E. Hassan, D. Blostein, Modeling the evolution of topics in source code histories, in: Proceedings of the 8th working conference on mining software repositories, 2011, pp. 173–182

    2011

  77. [85]

    K. Tian, M. Revelle, D. Poshyvanyk, Using latent dirichlet allocation for automatic categorization of software, in: 2009 6th IEEE international working conference on mining software repositories, IEEE, 2009, pp. 163–166

    2009

  78. [86]

    Gethers, D

    M. Gethers, D. Poshyvanyk, Using relational topic models to capture coupling among classes in object-oriented software systems, in: 2010 IEEE international conference on software maintenance, IEEE, 2010, pp. 1–10

    2010

  79. [87]

    Linstead, C

    E. Linstead, C. Lopes, P. Baldi, An application of latent dirichlet allocation to analyzing software evolution, in: 2008 seventh international conference on machine learning and applications, IEEE, 2008, pp. 813–818

    2008

  80. [88]

    S. K. Lukins, N. A. Kraft, L. H. Etzkorn, Bug localization using latent dirichlet allocation, Information and Software Technology 52 (9) (2010) 972–990

    2010

  81. [89]

    Savage, B

    T. Savage, B. Dit, M. Gethers, D. Poshyvanyk, Topic xp: Exploring topics in source code using latent dirichlet allocation, in: 2010 IEEE International Conference on Software Maintenance, IEEE, 2010, pp. 1–6

    2010

  82. [90]

    Gokcimen, B

    T. Gokcimen, B. Das, Topic modelling using bertopic for robust spam detection, in: 2024 12th International Symposium on Digital Forensics and Security (ISDFS), 2024, pp. 1–5. doi:10.1109/ISDFS60797.2024.10527342

    2024

  83. [91]

    J. Li, C. Cardie, S. Li, Topicspam: a topic-model based approach for spam detec- tion, in: Proceedings of the 51st Annual Meeting of the Association for Computa- tional Linguistics (Volume 2: Short Papers), 2013, pp. 217–221

    2013

  84. [92]

    Cheng, J

    Z. Cheng, J. Shen, On effective location-aware music recommendation, ACM Trans- actions on Information Systems (TOIS) 34 (2) (2016) 1–32

    2016

  85. [93]

    Y. Kim, K. Shim, Twilite: A recommendation system for twitter using a proba- bilistic model based on latent dirichlet allocation, Information Systems 42 (2014) 59–77

    2014

  86. [94]

    Lu, C.-H

    H.-M. Lu, C.-H. Lee, A twitter hashtag recommendation model that accommodates for temporal clustering effects, IEEE Intelligent Systems 30 (3) (2015) 18–25

    2015

  87. [95]

    F. Zhao, Y. Zhu, H. Jin, L. T. Yang, A personalized hashtag recommendation approach using lda-based topic model in microblog environment, Future Generation Computer Systems 65 (2016) 196–206

    2016

  88. [96]

    Zoghbi, I

    S. Zoghbi, I. Vuli´ c, M.-F. Moens, Latent dirichlet allocation for linking user- generated content and e-commerce data, Information Sciences 367 (2016) 573–599

    2016

  89. [97]

    Grootendorst, Bertopic: Neural topic modeling with a class-based tf-idf proce- dure, arXiv preprint arXiv:2203.05794 (2022)

    M. Grootendorst, Bertopic: Neural topic modeling with a class-based tf-idf proce- dure, arXiv preprint arXiv:2203.05794 (2022). 48

    2022 arXiv

  90. [98]

    D. M. Blei, A. Y. Ng, M. I. Jordan, Latent dirichlet allocation, Journal of machine Learning research 3 (Jan) (2003) 993–1022

    2003

  91. [99]

    D. Blei, J. Lafferty, Correlated topic models, Advances in neural information pro- cessing systems 18 (2006) 147

    2006

  92. [100]

    D. M. Blei, J. D. Lafferty, Dynamic topic models, in: Proceedings of the 23rd international conference on Machine learning, 2006, pp. 113–120

    2006

  93. [101]

    D. Lee, H. S. Seung, Algorithms for non-negative matrix factorization, Advances in neural information processing systems 13 (2000)

    2000

  94. [102]

    X. Yan, J. Guo, Y. Lan, X. Cheng, A biterm topic model for short texts, in: Proceedings of the 22nd international conference on World Wide Web, 2013, pp. 1445–1456

    2013

  95. [103]

    Parlina, I

    A. Parlina, I. Maryati, Leveraging bertopic for the analysis of scientific papers on seaweed, in: 2023 International Conference on Computer, Control, Informatics and its Applications (IC3INA), 2023, pp. 279–283. doi:10.1109/IC3INA60834.2023. 10285737

    2023

  96. [104]

    W. Kang, Y. Kim, H. Kim, J. Lee, An analysis of research trends on language model using bertopic, in: 2023 Congress in Computer Science, Computer Engineering, and Applied Computing (CSCE), 2023, pp. 168–172. doi:10.1109/CSCE60160.2023. 00032

    2023

  97. [105]

    T. Doi, M. Isonuma, H. Yanaka, Topic modeling for short texts with large language models, in: Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 4: Student Research Workshop), 2024, pp. 21–33

    2024

  98. [106]

    E. Mankolli, Reducing the complexity of candidate selection using natural lan- guage processing, in: 2022 29th International Conference on Systems, Signals and Image Processing (IWSSIP), Vol. CFP2255E-ART, 2022, pp. 1–4. doi: 10.1109/IWSSIP55020.2022.9854488

    2022

  99. [107]

    Atzeni, D

    D. Atzeni, D. Bacciu, D. Mazzei, G. Prencipe, A systematic review of wi-fi and machine learning integration with topic modeling techniques, Sensors 22 (13) (2022) 4925

    2022

  100. [108]

    L. Chen, What do user experience professionals discuss online? topic modeling of a user experience q&a community, in: International Conference on Human-Computer Interaction, Springer, 2023, pp. 365–380

    2023

  101. [109]

    A. Daud, J. Li, L. Zhou, F. Muhammad, Knowledge discovery through directed probabilistic topic models: a survey, Frontiers of computer science in China 4 (2010) 280–301

    2010

  102. [110]

    L. Nie, X. Wei, D. Zhang, X. Wang, Z. Gao, Y. Yang, Data-driven answer selection in community qa systems, IEEE transactions on knowledge and data engineering 29 (6) (2017) 1186–1198. 49

    2017

  103. [111]

    J. G. Fiscus, G. R. Doddington, Topic detection and tracking evaluation overview, in: Topic detection and tracking: event-based information organization, Springer, 2002, pp. 17–31

    2002

  104. [112]

    Allamanis, C

    M. Allamanis, C. Sutton, Why, when, and what: analyzing stack overflow questions by topic, type, and code, in: 2013 10th Working conference on mining software repositories (MSR), IEEE, 2013, pp. 53–56

    2013

  105. [113]

    Uddin, F

    G. Uddin, F. Sabir, Y.-G. Gu´ eh´ eneuc, O. Alam, F. Khomh, An empirical study of iot topics in iot developer discussions on stack overflow, Empirical Software Engineering 26 (2021) 1–45

    2021

  106. [114]

    L. S. Barbosa, Software engineering for’quantum advantage’, in: Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops, 2020, pp. 427–429

    2020

  107. [115]

    S. Wang, D. Lo, L. Jiang, An empirical study on developer interactions in stackover- flow, in: Proceedings of the 28th annual ACM symposium on applied computing, 2013, pp. 1019–1024

    2013

  108. [116]

    H. Chen, J. Coogle, K. Damevski, Modeling stack overflow tags and topics as a hierarchy of concepts, Journal of Systems and Software 156 (2019) 283–299

    2019

  109. [117]

    J. Zou, L. Xu, W. Guo, M. Yan, D. Yang, X. Zhang, Which non-functional re- quirements do developers focus on? an empirical study on stack overflow using topic analysis, in: 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories, IEEE, 2015, pp. 446–449

    2015

  110. [118]

    Alshangiti, H

    M. Alshangiti, H. Sapkota, P. K. Murukannaiah, X. Liu, Q. Yu, Why is developing machine learning applications challenging? a study on stack overflow posts, in: 2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), IEEE, 2019, pp. 1–11

    2019

  111. [119]

    Ahmed, M

    S. Ahmed, M. Bagherzadeh, What do concurrency developers ask about? a large- scale study using stack overflow, in: Proceedings of the 12th ACM/IEEE interna- tional symposium on empirical software engineering and measurement, 2018, pp. 1–10

    2018

  112. [120]

    Q. Mi, Q. Bao, L. Cui, Identifying topics and trends in devops: A study of stack overflow posts, in: 2023 49th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), 2023, pp. 402–409. doi:10.1109/SEAA60479. 2023.00067

    2023

  113. [121]

    Suwonchoochit, T

    N. Suwonchoochit, T. Senivongse, Classification of database technology problems on stack overflow, in: 2021 IEEE/ACIS 19th International Conference on Software Engineering Research, Management and Applications (SERA), 2021, pp. 21–26. doi:10.1109/SERA51205.2021.9509047

    2021 arXiv

  114. [122]

    J. Zou, L. Xu, M. Yang, X. Zhang, D. Yang, Towards comprehending the non- functional requirements through developers’ eyes: An exploration of stack overflow using topic analysis, Information and Software Technology 84 (2017) 19–32. 50

    2017

  115. [123]

    URL https://community.openai.com/

    OpenAI, Openai developer forum, online; last accessed November, 2024 (2024). URL https://community.openai.com/

    2024

  116. [124]

    J. Han, E. Shihab, Z. Wan, S. Deng, X. Xia, What do programmers discuss about deep learning frameworks, Empirical Software Engineering 25 (2020) 2694–2747

    2020

  117. [125]

    Y. Lou, Z. Chen, Y. Cao, D. Hao, L. Zhang, Understanding build issue resolution in practice: symptoms and fix patterns, in: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2020, pp. 617–628

    2020

  118. [126]

    Zhang, C

    T. Zhang, C. Gao, L. Ma, M. Lyu, M. Kim, An empirical study of common chal- lenges in developing deep learning applications, in: 2019 IEEE 30th International Symposium on Software Reliability Engineering (ISSRE), IEEE, 2019, pp. 104–115

    2019

  119. [127]

    S. Nadi, S. Kr¨ uger, M. Mezini, E. Bodden, Jumping through hoops: Why do java developers struggle with cryptography apis?, in: Proceedings of the 38th Interna- tional Conference on Software Engineering, 2016, pp. 935–946

    2016

  120. [128]

    Exchange, Stack exchange data explorer, online; last accessed September, 2024 (2024)

    S. Exchange, Stack exchange data explorer, online; last accessed September, 2024 (2024). URL https://data.stackexchange.com/

    2024

  121. [129]

    Z. Wan, X. Xia, A. E. Hassan, What do programmers discuss about blockchain? a case study on the use of balanced lda and the reference architecture of a domain to capture online discussions about blockchain platforms across stack exchange communities, IEEE Transactions on Softw...

    2019

  122. [130]

    Bird, Nltk: the natural language toolkit, in: Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions, 2006, pp

    S. Bird, Nltk: the natural language toolkit, in: Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions, 2006, pp. 69–72

    2006

  123. [131]

    Face, Hugging Face Model Hub, online; last accessed August, 2024 (2024)

    H. Face, Hugging Face Model Hub, online; last accessed August, 2024 (2024). URL https://huggingface.co/docs/hub/en/models-the-hub

    2024

  124. [132]

    McInnes, J

    L. McInnes, J. Healy, S. Astels, et al., hdbscan: Hierarchical density based cluster- ing., J. Open Source Softw. 2 (11) (2017) 205

    2017

  125. [133]

    Bajaj, K

    K. Bajaj, K. Pattabiraman, A. Mesbah, Mining questions asked by web developers, in: Proceedings of the 11th Working conference on mining software repositories, 2014, pp. 112–121

    2014

  126. [134]

    M. L. McHugh, Interrater reliability: the kappa statistic, Biochemia medica 22 (3) (2012) 276–282

    2012

  127. [135]

    Spearman Rank Correlation Coefficient, Springer New York, New York, NY, 2008, pp. 502–505. doi:10.1007/978-0-387-32833-1_379 . URL https://doi.org/10.1007/978-0-387-32833-1_379

    2008 doi

  128. [136]

    J. Li, B. Hui, G. Qu, J. Yang, B. Li, B. Li, B. Wang, B. Qin, R. Geng, N. Huo, et al., Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls, Advances in Neural Information Processing Sys- tems 36 (2024). 51

    2024

  129. [137]

    J. K. Kim, M. Chua, M. Rickard, A. Lorenzo, Chatgpt and large language model (llm) chatbots: The current state of acceptability and a proposal for guidelines on utilization in academic medicine, Journal of Pediatric Urology 19 (5) (2023) 598–604

    2023

  130. [138]

    X. Hou, Y. Zhao, Y. Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, H. Wang, Large language models for software engineering: A systematic literature review, ACM Transactions on Software Engineering and Methodology (2023)

    2023

  131. [139]

    C. J. Bean, Qualitative research design: An interactive approach, Organizational Research Methods 10 (2) (2007) 393

    2007

  132. [140]

    Wohlin, P

    C. Wohlin, P. Runeson, M. H¨ ost, M. C. Ohlsson, B. Regnell, A. Wessl´ en, Experi- mentation in software engineering, Springer Science & Business Media (2012)

    2012

  133. [141]

    Ferguson, External validity, generalizability, and knowledge utilization, Journal of Nursing Scholarship 36 (1) (2004) 16–22

    L. Ferguson, External validity, generalizability, and knowledge utilization, Journal of Nursing Scholarship 36 (1) (2004) 16–22. 52

    2004

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.