Pith. sign in

REVIEW 4 major objections 6 minor 49 references

An Empirical Study of OpenAI API Discussions on Stack Overflow

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Analyzing 2,874 Stack Overflow posts, this paper maps the distinctive challenges of OpenAI API development — prompt design, token cost, deprecation, third-party integration — and ranks GPT Actions as the hardest category.

desk verdict A solid descriptive map of OpenAI API pain points that overclaims novelty against prior work and needs statistical and reproducibility polish before it can be called comprehensive. read the letter →

arxiv 2505.04084 v1 pith:W62HIC6K submitted 2025-05-07 cs.SE cs.AI

Xiang Chen , Jibin Wang , Chaoyang Gao , Xiaolin Ju , Zhanqi Cui This is my paper · ORCID
classification cs.SEcs.AI
keywords StackOverflowminingOpenAIAPIempiricalstudytopicmodelingdeveloperchallengeslargelanguagemodelsusabilitycommunityQ&A
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks what actually goes wrong when developers build on OpenAI's APIs, and answers with a systematic read of 2,874 Stack Overflow discussions collected through January 2025. It claims to be the first comprehensive empirical study of OpenAI API challenges, classifying posts into nine API categories and using topic modeling to surface the most common problems. The result is a concrete map: prompt engineering, token-based cost control, model and SDK deprecation, context management, and third-party integration dominate developer struggles, with GPT Actions API questions proving the hardest to get answered. If the findings hold, they give LLM vendors a prioritized list of documentation and compatibility gaps and give researchers a starting point for API-misuse and deprecation tools.

What carries the argument

The load-bearing mechanism is a five-step Stack Overflow mining pipeline: tag-based filtering that expands from the seed tag `openai-api` to a six-tag set using significance ($\alpha$) and relevance ($\beta$) thresholds; manual inspection of every candidate post with a chance-corrected inter-rater agreement statistic; manual labeling into the nine API categories; and Latent Dirichlet Allocation (a statistical topic model) run on post titles only, with coherence scores to choose the number of topics and open card sorting to name them. The pipeline is what turns 2,874 unstructured forum posts into a reproducible taxonomy of developer challenges. Its two most consequential design choices — title-only preprocessing and the specific tag thresholds $\alpha=0.1$ and $\beta=0.01$ — do the work of keeping the taxonomy focused, and they are also the assumptions that the findings rest on.

What would settle it

Re-run the pipeline on the same date range with an expanded tag set (for example, adding `gpt-4`, `whisper`, `dalle`, `function-calling`, or `chatgpt`) and compare the nine-category distribution and the per-category topic lists; a material shift would show that tag selection under-samples the true space of OpenAI API questions. A second check: sample posts whose titles mention OpenAI but carry none of the six tags and see whether they contain challenges the current taxonomy misses.

Watch

Extended reading notes

Core claim

The central claim is that the difficulties of working with OpenAI's APIs are distinct from those of traditional APIs and can be systematically catalogued from developer questions. Concretely, the paper identifies nine API categories — Chat, Image Generation, Fine-tuning, Embeddings, Audio, Code Generation, Assistants, GPT Actions, and Others — and shows that the Chat API alone accounts for 44.2% of all posts. Difficulty metrics (the percentage of questions without accepted answers and the median time to receive an accepted answer) rank GPT Actions as the hardest category, with 94.9% of its questions unanswered, while general-purpose APIs such as Assistants, Fine-tuning, and Embeddings turn out harder than specialized ones. Topic modeling yields per-category challenge topics — nine for Chat, seven for Embeddings, six for Audio, and so on — that repeatedly point to prompt design, cost optimization, deprecation and compatibility, and third-party tool integration. The paper concludes with actionable implications for developers, LLM vendors, and researchers.

Load-bearing premise

The entire taxonomy and difficulty ranking rest on the assumption that the six selected Stack Overflow tags, plus manual filtering, capture a representative sample of OpenAI API questions, and that topic modeling on post titles alone accurately reflects what developers are actually asking.

Editorial extensions

If this is right

  • If the taxonomy is right, documentation and tutorial efforts should target prompt design, context management, and token-cost control before more exotic API features.
  • LLM vendors need explicit deprecation policies and version pinning, because SDK and model upgrades are a recurring source of developer failures.
  • GPT Actions API deserves dedicated debugging support: 94.9% of its questions receive no accepted answer, and the median wait for an answer is 583 hours.
  • Researchers can build API-misuse detectors, deprecated-API detectors, and API recommendation tools that target the patterns in this corpus.
  • The 2024 decline in posts suggests that AI-assisted coding and community dynamics are shifting where developers get help, which any future Stack Overflow mining study should account for.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same pipeline could be applied to other LLM API vendors to test whether this challenge taxonomy is specific to OpenAI or general to LLM APIs; the paper's external-validity discussion names vendor expansion as future work.
  • Title-only topic modeling is testable: re-running the analysis on post bodies or on title-plus-body and comparing the resulting topics would show whether titles compress developer concerns or distort them.
  • The difficulty ranking could become a live benchmark: if GPT Actions remains the most-unanswered category after documentation improvements, that would suggest its difficulty is intrinsic to third-party integration, not just a documentation gap.
  • The attribution of the 2024 decline to community tensions and AI tools could be checked by measuring deleted-post rates and answer-acceptance activity around the OpenAI–Stack Overflow partnership announcement.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents what it claims is the first comprehensive empirical study of developer challenges with OpenAI APIs, based on 2,874 Stack Overflow posts. The authors collect posts using a tag-based filter plus manual inspection, categorize them into nine OpenAI API categories, analyze popularity trends and difficulty metrics, and apply topic modeling to post titles to identify per-category challenges. They then derive implications for developers, LLM vendors, and researchers, and state that the dataset and scripts are available on GitHub. The central contribution is a reproducible map of common OpenAI API difficulties, including prompt engineering, cost management, deprecation, streaming, and third-party integration.

Significance. If the findings hold, this is a useful and timely contribution for the software engineering community. The strengths of the manuscript are the manual labeling effort with high inter-rater agreement (Cohen's Kappa 0.832 for filtering and 0.822 for categorization), the use of concrete Stack Overflow posts to support each topic, and the stated intention to share the dataset and analysis scripts for reproducibility. The paper also connects its taxonomy to actionable implications for different stakeholders. However, the significance of the central 'comprehensive map' claim depends on two testable assumptions that are not yet validated: the representativeness of the tag-based sample and the validity of title-only topic modeling as a proxy for post-body challenges. Because these assumptions are load-bearing for RQ3 and the overall novelty claim, the paper needs additional validation before it can be accepted.

major comments (4)
  1. [§3.5 Step 1] The challenge taxonomy for RQ3 is built from topic modeling of post titles only, with body content excluded during preprocessing. The paper justifies this by citing [14], [21] for noise reduction and [29], [30] for title representativeness, but those studies are not about OpenAI API posts, and the manuscript provides no internal check that the titles of the 2,874 collected posts align with their bodies. Since Findings 3–11 and the Section 7 implications are derived from these topics, a title/body mismatch would change the central map of developer challenges. I recommend adding a validation step, for example manually comparing a random sample of titles with their bodies, or running the same topic modeling on body text as a robustness check, and reporting agreement between title-derived and body-derived topic assignments.
  2. [§5, Table 1] The difficulty ranking in Table 1 is presented without confidence intervals or significance tests. The category sizes differ greatly: GPT Actions API has roughly 40 posts (1.4% of 2,874) while Chat API has roughly 1,270 posts, so the difference between 94.9% and 83.0% unresolved questions may not be statistically meaningful. I recommend adding bootstrap confidence intervals for the percentages and median times, and appropriate significance tests such as a chi-square or Fisher exact test for proportions and a permutation test for medians. In addition, the definition of 'median time to receive an accepted answer' needs clarification: the text states that the time is the creation time of the answer rather than the time the answer was marked accepted, which is ambiguous.
  3. [§1 and §8] The abstract and introduction claim that this is 'the first comprehensive empirical study' of OpenAI API challenges, yet reference [36] (Alam et al., 'Developer challenges on large language models: A study of stack overflow and openai developer forum posts') is a closely related study that is cited in Section 6 but not discussed in Related Work. Reference [10] (Chen et al., 'An empirical study on challenges for LLM application developers') is also directly relevant. The paper should position its contribution explicitly against these works and qualify the 'first' claim to its specific scope, or demonstrate concretely what is novel beyond them.
  4. [§3.1 Step 2] The representativeness of the sample depends on the six selected tags plus manual filtering, but no recall check is reported for OpenAI API posts that do not carry any of these tags, such as posts tagged only gpt, chatgpt, dall-e, or whisper. Since the paper's central claim is a 'comprehensive' map of developer challenges, I recommend quantifying tag coverage, for example by searching a random sample of posts containing known OpenAI-related terms and measuring how many are missed by the chosen tag set.
minor comments (6)
  1. [§3.1 Step 2] The threshold ranges contain formatting errors, such as '0.15 0.2 0.25', '0 .015', '0 .25', and '0 .25 0 .03'; please fix the notation and ensure all numeric values are properly comma-separated.
  2. [§4] The 2025 data point is based on 42 posts through January 22, 2025; describing this as 'relatively stable' is too strong for a partial year of data.
  3. [§5] The phrase 'median time to receive an accepted answer' should state explicitly whether the interval is question creation to answer creation, question creation to answer acceptance, or another interval; the current wording is confusing.
  4. [§6.1, Table 2] The keywords listed for topic A2 ('token, exceed, llm, modul, attributeerror, quota, valid, paramet') do not appear to match the narrative about model migration and SDK deprecation; please verify the keyword assignment.
  5. [§3.2] The labeling process is described as following 'descriptions and definitions of OpenAI APIs,' but the manuscript does not provide the labeling instructions or a codebook; adding the codebook to the online appendix would improve reproducibility.
  6. [§6.4, Finding 6] The phrase 'issues related to threat management' appears to be a typo; the surrounding text discusses permission, configuration, and tool-integration issues rather than threat management.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the study's findings are descriptive aggregates of Stack Overflow data, not predictions reduced from fitted inputs.

full rationale

The paper's derivation chain is: collect posts via tag filtering and manual inspection, label them into nine API categories, compute popularity and difficulty metrics, and topic-model post titles to identify challenges. Each step uses external data and manual annotation; no equation defines a target quantity in terms of the same quantity. The tag set in Section 3.1 is selected using significance/relevance heuristics, and the manual inspection is independent of the later topic naming. Section 3.5's topic modeling is applied to titles and then manually validated on sampled posts; the title-only choice is a validity assumption about title representativeness, not a circular reduction. Reference [10], by overlapping authors, appears among the supporting citations [8]-[11] for the nine-category taxonomy, but the taxonomy is also grounded in the other cited sources and is a descriptive organizational choice, not a predicted result. No fitted constant or hidden parameter is renamed as an outcome; RQ2 metrics are direct metadata calculations, and RQ3 topics are derived from the corpus rather than imposed by a self-citation. The Section 7 implications restate the findings as recommendations, which is normal translation of results into guidance rather than a derivation. A title/body mismatch or tag-coverage gap would affect the validity of the taxonomy, but that is an external-validity concern, not circularity. Therefore no circular step is exhibited.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The nine API categories are a classification scheme, not invented entities; no new physical or conceptual objects are postulated. The only ad hoc elements are analysis parameters captured above.

free parameters (2)
  • Tag filtering thresholds alpha and beta = alpha=0.1, beta=0.01
    Chosen after experiments with ranges; they determine the final tag set and therefore the initial post pool (Section 3.1 Step 2).
  • Per-category topic count K = not fixed; selected from 2 to 20 by coherence plus manual validation
    Topic number selection affects which topics and challenges are reported (Section 3.5 Step 2).
assumptions (3)
  • domain assumption Stack Overflow posts with the selected tags are a representative source of OpenAI API developer challenges.
    The study relies on Stack Overflow mining to generalize about developer challenges; stated in Section 3.1 and external threats in Section 7.2.
  • domain assumption Post titles are representative enough of body content for topic modeling.
    Section 3.5 Step 1 excludes bodies and relies on prior claims [29], [30] that titles represent content.
  • domain assumption Percentage without accepted answers and median time to answer are valid difficulty proxies.
    Section 3.4 uses these metrics; they can be confounded by question specificity, community size, and off-site solutions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of An Empirical Study of OpenAI API Discussions on Stack Overflow." pith.science (2026). https://pith.science/paper/W62HIC6K

@misc{pith2026250504084,
  author       = {Pith},
  title        = {Pith review of: An Empirical Study of OpenAI API Discussions on Stack Overflow},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W62HIC6K}},
  note         = {Machine review of arXiv:2505.04084}
}
read the original abstract

The rapid advancement of large language models (LLMs), represented by OpenAI's GPT series, has significantly impacted various domains such as natural language processing, software development, education, healthcare, finance, and scientific research. However, OpenAI APIs introduce unique challenges that differ from traditional APIs, such as the complexities of prompt engineering, token-based cost management, non-deterministic outputs, and operation as black boxes. To the best of our knowledge, the challenges developers encounter when using OpenAI APIs have not been explored in previous empirical studies. To fill this gap, we conduct the first comprehensive empirical study by analyzing 2,874 OpenAI API-related discussions from the popular Q&A forum Stack Overflow. We first examine the popularity and difficulty of these posts. After manually categorizing them into nine OpenAI API-related categories, we identify specific challenges associated with each category through topic modeling analysis. Based on our empirical findings, we finally propose actionable implications for developers, LLM vendors, and researchers.

Figures

Figures reproduced from arXiv: 2505.04084 by the authors.

Figure 1
Figure 1. Methodology overview of our empirical study. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Distribution of posts in nine OpenAI API categories. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The number of posts and users related to OpenAI APIs over time. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (8 more)
Figure 5
Figure 5. Figure 5: A sample post in the topic of model migration. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: A sample post in the topic of streaming and asynchronous [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 8
Figure 8. Figure 8: A sample post in the topic of vector database integration and [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: A sample post in the topic of attribute and query Errors. [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 10
Figure 10. Figure 10: A sample post in the topic of model errors and file loading. [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]
Figure 11
Figure 11. Figure 11: A sample post in the topic of core functions and fundamental [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]
Figure 12
Figure 12. Figure 12: A sample post in the topic of API basic usage. [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]
Figure 13
Figure 13. Figure 13: A sample post in the topic of fundamental API usage and [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

49 extracted references · 40 canonical work pages

  1. [36]

    Developer Challenges on Large Language Models: A Study of Stack Overflow and OpenAI Developer Forum Posts

    K. Alam, K. Mittal, B. Roy, and C. Roy, “Developer challenges on large language models: A study of stack overflow and openai developer forum posts,” arXiv preprint arXiv:2411.10873, 2024

    work page Pith review arXiv 2024

  2. [14]

    Challenges in devel- oping desktop web apps: a study of stack overflow and github,

    G. L. Scoccia, P . Migliarini, and M. Autili, “Challenges in devel- oping desktop web apps: a study of stack overflow and github,” in 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR). IEEE, 2021, pp. 271–282

    work page 2021

  3. [21]

    Challenges in chatbot development: A study of stack overflow posts,

    A. Abdellatif, D. Costa, K. Badran, R. Abdalkareem, and E. Shihab, “Challenges in chatbot development: A study of stack overflow posts,” in Proceedings of the 17th international conference on mining software repositories, 2020, pp. 174–185

    work page 2020

  4. [29]

    Learning a dual-language vector space for domain-specific cross-lingual question retrieval,

    G. Chen, C. Chen, Z. Xing, and B. Xu, “Learning a dual-language vector space for domain-specific cross-lingual question retrieval,” in Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering, 2016, pp. 744–755

    work page 2016

  5. [30]

    Answerbot: Automated gener- ation of answer summary to developers’ technical questions,

    B. Xu, Z. Xing, X. Xia, and D. Lo, “Answerbot: Automated gener- ation of answer summary to developers’ technical questions,” in 2017 32nd IEEE/ACM international conference on automated software engineering (ASE). IEEE, 2017, pp. 706–716

    work page 2017

  6. [10]

    An empirical study on challenges for llm application developers,

    X. Chen, C. Gao, C. Chen, G. Zhang, and Y. Liu, “An empirical study on challenges for llm application developers,” ACM Trans- actions on Software Engineering and Methodology , 2025

    work page 2025

  7. [1]

    Large language models for software engineering: Survey and open problems,

    A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, and J. M. Zhang, “Large language models for software engineering: Survey and open problems,” in 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE). IEEE, 2023, pp. 31–53

    work page 2023

  8. [2]

    Large language models for software engineering: A systematic literature review,

    X. Hou, Y. Zhao, Y. Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engineering: A systematic literature review,” ACM Transactions on Software Engineering and Methodology, vol. 33, no. 8, pp. 1–79, 2024

    work page 2024

Show all 49 references
  1. [3]

    Software testing with large language models: Survey, landscape, and vision,

    J. Wang, Y. Huang, C. Chen, Z. Liu, S. Wang, and Q. Wang, “Software testing with large language models: Survey, landscape, and vision,” IEEE Transactions on Software Engineering , vol. 50, no. 04, pp. 911–936, 2024

    2024

  2. [4]

    A survey on large language models for software engineering,

    Q. Zhang, C. Fang, Y. Xie, Y. Zhang, Y. Yang, W. Sun, S. Yu, and Z. Chen, “A survey on large language models for software engineering,” arXiv preprint arXiv:2312.15223, 2023

    2023 arXiv

  3. [5]

    Chatgpt utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. healthcare (basel). 2023; 11 (6): 887,

    M. Sallam, “Chatgpt utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. healthcare (basel). 2023; 11 (6): 887,” Nature, vol. 616, no. 7956, pp. 259–265, 2023. IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VO...

    2023

  4. [6]

    An empirical study of unanswered python-related questions on stack overflow,

    Y. S. Nugroho, S. A. A. Halim, S. Islam, Y. I. Kurniawan, and T. Erlina, “An empirical study of unanswered python-related questions on stack overflow,” in 2024 International Conference on Information Technology Research and Innovation (ICITRI) . IEEE, 2024, pp. 230–235

    2024

  5. [7]

    A survey on mining stack overflow: question and answering (q&a) community,

    A. Ahmad, C. Feng, S. Ge, and A. Yousif, “A survey on mining stack overflow: question and answering (q&a) community,” Data Technologies and Applications, vol. 52, no. 2, pp. 190–247, 2018

    2018

  6. [8]

    Eval- uating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P . D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al. , “Eval- uating large language models trained on code,” arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  7. [9]

    Chatgpt and open-ai models: A preliminary review,

    K. I. Roumeliotis and N. D. Tselikas, “Chatgpt and open-ai models: A preliminary review,” Future Internet, vol. 15, no. 6, p. 192, 2023

    2023

  8. [11]

    Overview of the openai apis,

    T. Auger and E. Saroyan, “Overview of the openai apis,” in Gener- ative AI for Web Development: Building Web Applications Powered by OpenAI APIs and Next. js. Springer, 2024, pp. 87–116

    2024

  9. [12]

    An empirical study of web services topics in web developer discussions on stack overflow,

    K. Mahmood, G. Rasool, F. Sabir, and A. Athar, “An empirical study of web services topics in web developer discussions on stack overflow,” IEEE Access, vol. 11, pp. 9627–9655, 2023

    2023

  10. [13]

    Understanding privacy- related questions on stack overflow,

    M. Tahaei, K. Vaniea, and N. Saphra, “Understanding privacy- related questions on stack overflow,” in Proceedings of the 2020 CHI conference on human factors in computing systems , 2020, pp. 1–14

    2020

  11. [15]

    What do practitioners discuss about iot and industry 4.0 related technologies? characterization and identification of iot and industry 4.0 categories in stack overflow discussions,

    M. Aly, F. Khomh, and S. Yacout, “What do practitioners discuss about iot and industry 4.0 related technologies? characterization and identification of iot and industry 4.0 categories in stack overflow discussions,” Internet of Things, vol. 14, p. 100364, 2021

    2021

  12. [16]

    What do client developers concern when using web apis? an empirical study on developer forums and stack overflow,

    P . K. Venkatesh, S. Wang, F. Zhang, Y. Zou, and A. E. Hassan, “What do client developers concern when using web apis? an empirical study on developer forums and stack overflow,” in 2016 IEEE International Conference on Web Services (ICWS) . IEEE, 2016, pp. 131–138

    2016

  13. [17]

    What are mobile developers asking about? a large scale study using stack overflow,

    C. Rosen and E. Shihab, “What are mobile developers asking about? a large scale study using stack overflow,”Empirical Software Engineering, vol. 21, pp. 1192–1223, 2016

    2016

  14. [18]

    Why is developing machine learning applications challenging? a study on stack overflow posts,

    M. Alshangiti, H. Sapkota, P . K. Murukannaiah, X. Liu, and Q. Yu, “Why is developing machine learning applications challenging? a study on stack overflow posts,” in 2019 ACM/IEEE Interna- tional Symposium on Empirical Software Engineering and Measurement (ESEM). IEEE, 2019, pp. 1–11

    2019

  15. [19]

    Common challenges of deep reinforcement learning applications development: an empirical study,

    M. M. Morovati, F. Tambon, M. Taraghi, A. Nikanjam, and F. Khomh, “Common challenges of deep reinforcement learning applications development: an empirical study,” Empirical Software Engineering, vol. 29, no. 4, p. 95, 2024

    2024

  16. [20]

    Towards enhancing the reproducibility of deep learning bugs: an empirical study,

    M. B. Shah, M. M. Rahman, and F. Khomh, “Towards enhancing the reproducibility of deep learning bugs: an empirical study,” Empirical Software Engineering, vol. 30, no. 1, p. 23, 2025

    2025

  17. [22]

    What do concurrency developers ask about? a large-scale study using stack overflow,

    S. Ahmed and M. Bagherzadeh, “What do concurrency developers ask about? a large-scale study using stack overflow,” inProceedings of the 12th ACM/IEEE international symposium on empirical software engineering and measurement, 2018, pp. 1–10

    2018

  18. [23]

    An empirical study of iot topics in iot developer discussions on stack overflow,

    G. Uddin, F. Sabir, Y.-G. Gu ´eh´eneuc, O. Alam, and F. Khomh, “An empirical study of iot topics in iot developer discussions on stack overflow,” Empirical Software Engineering, vol. 26, pp. 1–45, 2021

    2021

  19. [24]

    Challenges in docker development: A large-scale study using stack overflow,

    M. U. Haque, L. H. Iwaya, and M. A. Babar, “Challenges in docker development: A large-scale study using stack overflow,” in Proceedings of the 14th ACM/IEEE international symposium on empirical software engineering and measurement (ESEM) , 2020, pp. 1–11

    2020

  20. [25]

    Going big: a large-scale study on what big data developers ask,

    M. Bagherzadeh and R. Khatchadourian, “Going big: a large-scale study on what big data developers ask,” in Proceedings of the 2019 27th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering , 2019, pp. 432–442

    2019

  21. [26]

    What security questions do developers ask? a large-scale study of stack overflow posts,

    X.-L. Yang, D. Lo, X. Xia, Z.-Y. Wan, and J.-L. Sun, “What security questions do developers ask? a large-scale study of stack overflow posts,” Journal of Computer Science and Technology, vol. 31, pp. 910– 924, 2016

    2016

  22. [27]

    A coefficient of agreement for nominal scales,

    J. Cohen, “A coefficient of agreement for nominal scales,” Educa- tional and psychological measurement, vol. 20, no. 1, pp. 37–46, 1960

    1960

  23. [28]

    The measurement of observer agreement for categorical data,

    J. R. Landis and G. G. Koch, “The measurement of observer agreement for categorical data,” biometrics, pp. 159–174, 1977

    1977

  24. [31]

    Exploring the space of topic coherence measures,

    M. R ¨oder, A. Both, and A. Hinneburg, “Exploring the space of topic coherence measures,” in Proceedings of the eighth ACM international conference on Web search and data mining, 2015, pp. 399– 408

    2015

  25. [32]

    Mining the usage of reactive programming apis: a study on github and stack overflow,

    C. Zimmerle, K. Gama, F. Castor, and J. M. M. Filho, “Mining the usage of reactive programming apis: a study on github and stack overflow,” in Proceedings of the 19th International Conference on Mining Software Repositories, 2022, pp. 203–214

    2022

  26. [33]

    A com- prehensive study on challenges in deploying deep learning based software,

    Z. Chen, Y. Cao, Y. Liu, H. Wang, T. Xie, and X. Liu, “A com- prehensive study on challenges in deploying deep learning based software,” in Proceedings of the 28th ACM joint meeting on European software engineering conference and symposium on the foundations of software engine...

    2020

  27. [34]

    Making sense of card sorting data,

    S. Fincher and J. Tenenberg, “Making sense of card sorting data,” Expert Systems, vol. 22, no. 3, pp. 89–93, 2005

    2005

  28. [35]

    Large language models reduce public knowledge sharing on online q&a platforms,

    R. M. del Rio-Chanona, N. Laurentsyeva, and J. Wachs, “Large language models reduce public knowledge sharing on online q&a platforms,” PNAS nexus, vol. 3, no. 9, p. pgae400, 2024

    2024

  29. [37]

    Unveiling the impact of multi-modal interactions on user engagement: A comprehensive evaluation in ai-driven conversations,

    L. Zhang, J. Yu, S. Zhang, L. Li, Y. Zhong, G. Liang, Y. Yan, Q. Ma, F. Weng, F. Panet al., “Unveiling the impact of multi-modal interactions on user engagement: A comprehensive evaluation in ai-driven conversations,” arXiv preprint arXiv:2406.15000, 2024

    2024 arXiv

  30. [38]

    Prompt engineering in large language models,

    G. Marvin, N. Hellen, D. Jjingo, and J. Nakatumba-Nabende, “Prompt engineering in large language models,” in International conference on data intelligence and cognitive informatics . Springer, 2023, pp. 387–402

    2023

  31. [39]

    C5: toward better conversation comprehension and contextual continuity for chatgpt,

    P . Liang, D. Ye, Z. Zhu, Y. Wang, W. Xia, R. Liang, and G. Sun, “C5: toward better conversation comprehension and contextual continuity for chatgpt,” Journal of Visualization , vol. 27, no. 4, pp. 713–730, 2024

    2024

  32. [40]

    The what, why, and how of context length extension tech- niques in large language models–a detailed survey,

    S. Pawar, S. Tonmoy, S. Zaman, V . Jain, A. Chadha, and A. Das, “The what, why, and how of context length extension tech- niques in large language models–a detailed survey,” arXiv preprint arXiv:2401.07872, 2024

    2024 arXiv

  33. [41]

    Llm as a system service on mobile devices,

    W. Yin, M. Xu, Y. Li, and X. Liu, “Llm as a system service on mobile devices,” arXiv preprint arXiv:2403.11805, 2024

    2024 arXiv

  34. [42]

    Mapo: Mining and recommending api usage patterns,

    H. Zhong, T. Xie, L. Zhang, J. Pei, and H. Mei, “Mapo: Mining and recommending api usage patterns,” in ECOOP 2009–Object- Oriented Programming: 23rd European Conference, Genoa, Italy, July 6-10, 2009. Proceedings 23. Springer, 2009, pp. 318–343

    2009

  35. [43]

    Charac- terization and automatic updates of deprecated machine-learning api usages,

    S. A. Haryono, F. Thung, D. Lo, J. Lawall, and L. Jiang, “Charac- terization and automatic updates of deprecated machine-learning api usages,” in 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 2021, pp. 137–147

    2021

  36. [44]

    Llms meet library evolution: Evaluating deprecated api usage in llm-based code completion,

    C. Wang, K. Huang, J. Zhang, Y. Feng, L. Zhang, Y. Liu, and X. Peng, “Llms meet library evolution: Evaluating deprecated api usage in llm-based code completion,” in 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE) . IEEE Computer Society, 2025, pp. 781–781

    2025

  37. [45]

    Demystifying api misuses in deep learning applications,

    D. Yang, K. Liu, Y. Lei, L. Li, H. Xie, C. Liu, Z. Wang, X. Mao, and T. F. Bissyand ´e, “Demystifying api misuses in deep learning applications,” Empirical Software Engineering , vol. 29, no. 2, p. 45, 2024

    2024

  38. [46]

    Api recommendation for machine learning libraries: how far are we?

    M. Wei, Y. Huang, J. Wang, J. Shin, N. S. Harzevili, and S. Wang, “Api recommendation for machine learning libraries: how far are we?” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2022, ...

    2022

  39. [47]

    The symptoms, causes, and repairs of bugs inside a deep learning library,

    L. Jia, H. Zhong, X. Wang, L. Huang, and X. Lu, “The symptoms, causes, and repairs of bugs inside a deep learning library,” Journal of Systems and Software, vol. 177, p. 110935, 2021

    2021

  40. [48]

    Classifica- tion of android apis posts: An analysis of developer’s discussions on stack overflow,

    C. Beddiar, I. E. Khelili, N. Bounour, and A.-D. Seriai, “Classifica- tion of android apis posts: An analysis of developer’s discussions on stack overflow,” in 2020 International Conference on Advanced Aspects of Software Engineering (ICAASE). IEEE, 2020, pp. 1–5

    2020

  41. [49]

    What do developers ask about ml libraries? a large-scale study using stack overflow,

    M. J. Islam, H. A. Nguyen, R. Pan, and H. Rajan, “What do developers ask about ml libraries? a large-scale study using stack overflow,” arXiv preprint arXiv:1906.11940, 2019

    1906 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.