Uncovering Intention through LLM-Driven Code Snippet Description Generation

Brittany Reid; Farah Danisha Salam; Kazumasa Shimari; Kenichi Matsumoto; Raula Gaikovina Kula; Yusuf Sulistyo Nugroho

REVIEW 3 major objections 6 minor 25 references

Uncovering Intention through LLM-Driven Code Snippet Description Generation

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Most README code snippet descriptions are examples, and Llama overgeneralizes them.

desk verdict Useful taxonomy of README code snippet descriptions, but the LLM 'alignment' claim rests on aggregate proportions, not per-item agreement. read the letter →

arxiv 2506.15453 v1 pith:EEWIA467 submitted 2025-06-18 cs.SE cs.AI

Yusuf Sulistyo Nugroho , Farah Danisha Salam , Brittany Reid , Raula Gaikovina Kula , Kazumasa Shimari , Kenichi Matsumoto This is my paper · ORCID

classification cs.SEcs.AI

keywords codesnippetsREADMEdocumentationLLMclassificationdescriptiongenerationBERTsimilaritynpmpackagessoftwareLlama

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to learn what developers are actually doing when they describe code snippets in README files, and whether a general large language model can reproduce that behavior. Using 400 snippet-description pairs sampled from a dataset of over a million npm snippets, the authors manually classified the descriptions into instruction, example, and unclear categories and found that 55.5% are example-based. They then asked Llama to classify the same descriptions and found it also favors the example category, but more strongly at 79.75%, which they read as overgeneralization. Finally, they had Llama rewrite descriptions from the code alone and measured a BERT-based similarity of 0.7173 against the originals, concluding the model is helpful for drafting but needs oversight and prompt refinement. If these measurements hold, they describe ordinary documentation practice and a concrete gap between LLM and human labeling behavior.

What carries the argument

The central machinery is a three-way description taxonomy with subcategories: Instruction (installation or usage) and Example (usage example, feature explanation, or code example), plus Unclear. The authors operationalize it with a structured prompt that defines each category and demands one-line outputs from Llama running through Ollama, and with a manual coding procedure in which two annotators classified the first 30 items with a Cohen's Kappa of 80% and one annotator finished the remaining 370. BERT-based similarity between original and generated descriptions is then used as the quantitative measure of how well Llama can support description generation. The categories themselves carry the argument: the claim that developers mostly write examples and that Llama overgeneralizes is entirely a statement about how these defined buckets are filled.

What would settle it

Have two independent annotators re-label all 400 descriptions with the paper's rubric and compute the example share; if it is far from 55.5%, the manual baseline used to judge Llama is unstable. Separately, rerun Llama's classification with a prompt that removes the example subcategory definitions; if the 79.75% share persists, the overgeneralization is a model property, whereas if it moves toward 55.5%, the prompt is driving the divergence.

Watch

Extended reading notes

Core claim

Developers predominantly write README code snippet descriptions as examples rather than step-by-step instructions: manual annotation of 400 descriptions gives 55.5% example, 44.0% instruction, and 0.5% unclear. When the same rubric is given to Llama, it classifies 79.75% as example and 20.25% as instruction, agreeing with the human majority at the aggregate level while overgeneralizing; the paper attributes the gap to LLM bias toward common patterns in training data. When Llama generates fresh descriptions from the code snippets alone, the average BERT-based similarity to the original developer-written descriptions is 0.7173, and the paper reports that scores below about 0.9 tend to coincide with a mismatch in classification type. The authors conclude that the intended message of a snippet description varies with the snippet's task, that LLMs can assist documentation writing, and that outputs require careful validation. The paper presents these as early results, acknowledging in Section V that it did not run full accuracy or F1 metrics or external human judgment and that data leakage and hallucination remain threats.

Load-bearing premise

The load-bearing premise is that the 400 manual labels are a trustworthy ground truth: two annotators agreed on the first 30 descriptions, and then one author labeled the remaining 370; if external developers would categorize those descriptions differently, the claimed alignment between Llama and human classification has no reference point.

Editorial extensions

If this is right

LLM-based documentation tools should treat generated snippet descriptions as drafts, because the observed 0.7173 average similarity leaves room for divergence from the developer's intended wording.
Prompt design is a lever: the same rubric produced a 24-percentage-point gap between human and LLM example shares, so refining category definitions could reduce overgeneralization.
A similarity score below roughly 0.9 is a practical warning sign that the generated description may fall into a different category than the original.
Model output should not be trusted to detect ambiguity: Llama labeled nothing as unclear, while humans found two unclear descriptions, pointing to an overconfidence bias.
Documentation automation could be tuned per snippet task, since the paper finds that the intention behind a description differs between instructions and examples.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the 55.5% versus 79.75% gap is consistent with the paper's taxonomy granting Llama a broad feature-explanation bucket; a prompt that forces a choice among installation, usage, and code example might shrink the gap.
Editorial inference: if the pattern generalizes, LLM-based documentation assistants should expose a confidence or cannot-decide output, because the model here never used the unclear category despite two ambiguous human labels.
Editorial inference: a testable extension is to repeat the classification and generation pipeline with instruction-tuned or code-specialized models and with two independent human annotators on the full 400 samples, which would separate prompt effects from model effects.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a reasonable empirical study of README code snippet descriptions, with a useful first taxonomy, but the central RQ2 claim of LLM-human alignment is not supported by the analysis they actually run. The paper is honest about this in Section V, but the abstract overstates it.

What's new: they manually categorize 400 descriptions from an existing NPM dataset and report that 55.5% are example-style, 44% instruction. That's the first such breakdown for code snippet descriptions specifically, and documentation-tool designers might care. The LLM classification using a structured prompt is a straightforward application, and the finding that Llama over-predicts 'Example' (79.75% vs 55.5%) is suggestive, at the aggregate level, that off-the-shelf LLMs overgeneralize in this task. The BERTScore result (0.7173) is a reasonable rough indicator that generated descriptions are somewhat similar but not close. The paper is clearly written and discusses threats to validity candidly.

Soft spots, in order of severity. First, RQ2 never compares individual predictions. The 'identical' claim in the abstract comes from comparing marginal proportions: 55.5% vs 79.75% Example. That does not show alignment. A classifier that always outputs 'Example' would produce a majority that matches the human majority while agreeing on zero items. The paper itself concedes in Section V that full metrics like accuracy and F1 were not implemented. This is a load-bearing flaw for RQ2, though it is fixable: compute per-item agreement between LLM and human labels for the 400 snippets and report a confusion matrix or Cohen's Kappa. Second, the manual labels are treated as ground truth, and while the 30-sample double-coding with Kappa 0.80 is a reasonable starting point, the remaining 370 are labeled by one author. For the RQ2 comparison, that is secondary to the missing per-item analysis, but it would be worth rechecking a sample. Third, the '0.9 indicates some irrelevance' threshold is asserted without validation; BERTScore is not a calibrated relevance measure. Minor: no artifacts are released, so the specific samples and annotations are not verifiable.

Bottom line: the RQ1 taxonomy and the aggregate LLM behavior are worth knowing, but the paper's main alignment claim is not established. It deserves a serious referee because the underlying idea is sound and the flaw is addressable, but it needs a major revision, not acceptance as is.

Referee Report

3 major / 6 minor

Summary. This paper investigates what kinds of descriptions developers write for code snippets in npm README files and whether an LLM (Llama 3 via Ollama) can classify and generate such descriptions. Using 400 randomly sampled snippets from the NPM Code Snippets dataset, the authors manually label descriptions into Instruction, Example, and Unclear (with subtypes); apply a structured prompt to have the LLM classify the same snippets; and compare LLM-generated descriptions with the originals using BERTScore. The reported results are: developers predominantly write example-based descriptions (55.5%), the LLM classifies 79.75% of descriptions as Example, and the average BERTScore between generated and original descriptions is 0.7173. The paper argues that LLMs can assist documentation but tend to overgeneralize toward the Example category.

Significance. The paper addresses a relatively unexplored problem: the specific role and classification of code snippet descriptions in README files, rather than README structure as a whole. If the measurements were robust, the paper would provide a useful descriptive baseline for developer documentation practices and for LLM behavior on this task. Strengths include the reuse of a large existing dataset, a structured prompt design, explicit category definitions, a manual coding step with a reliability check, and a concrete research-question structure. However, as discussed below, the main empirical claims for RQ2 and RQ3 are not fully supported by the presented analysis, so the current contribution is mostly a preliminary report.

major comments (3)

[Section IV-B, Tables III and IV] The claim that Llama's classifications 'align' with human labels is not established because the paper only compares aggregate category proportions (manual: 55.5% Example; LLM: 79.75% Example) and never presents a per-item comparison of the LLM's label with the manual label for the same snippet. A classifier that always outputs 'Example' would produce a majority-Example aggregate distribution while agreeing with the human labels on zero of the 222 non-Example items, so marginal alignment cannot support the stated conclusion. Section V itself concedes that 'full evaluation metrics (e.g., accuracy, F1-score) and human judgment were not implemented.' Please report a confusion matrix and per-class precision, recall, and Cohen's Kappa (or a comparable agreement statistic) over the 400 samples; without this, the RQ2 result is not a measurement of classification accuracy.
[Section IV-A, coding procedure] The ground-truth labels used to evaluate the LLM in RQ2 and to summarize developer behavior in RQ1 are produced by a single author for 370 of the 400 samples, after a reliability check on only 30 items (Cohen's Kappa = 0.80). Kappa on 30 items, with two authors reaching consensus through discussion, does not provide strong evidence that the remaining single-author labels are reliable, especially for the difficult subtype distinctions (e.g., 'Usage Example' vs. 'Code Example' vs. 'Feature Explanation'). Please either double-code an additional random sample (e.g., 30–50 items) and report per-category reliability, or explicitly bound the effect of annotator noise on the RQ1 and RQ2 conclusions in the threats-to-validity section.
[Section IV-C, Fig. 3] The interpretation of the BERTScore results is under-specified. The paper states that 'scores below 0.9 indicate some irrelevance' but provides no calibration, baseline, or human evaluation linking a particular BERTScore value to 'relevance.' The average score of 0.7173 is called 'relatively high' without any comparison (e.g., similarity between independently written human descriptions, or a random baseline). Moreover, the paper does not report which BERTScore variant was used (precision, recall, or F1) or which model was used for embedding. These omissions make the RQ3 conclusion unfalsifiable; please provide additional context or remove the threshold-based claim unless it is empirically justified.

minor comments (6)

[Abstract and Section IV-B] The abstract says the LLM's 79.75% 'Example' classification is 'identical to our manual finding,' but the manual finding was 55.5% Example. The intended meaning seems to be that the majority category is the same; please rephrase to avoid the inaccurate 'identical.'
[Section IV-C, reference [25]] BERTScore is cited via reference [25], which is Malik et al., 'From Tarzan to Tolkien: Controlling the language proficiency level of LLMs for content generation.' BERTScore was introduced by Zhang et al., 'BERTScore: Evaluating Text Generation with BERT' (ICLR 2020). Please cite the correct original paper and specify the BERTScore configuration.
[Section IV-A, Kappa interpretation] The paper reports 'Kappa score at 80%,' which it calls 'almost perfect agreement' citing Viera and Garrett [24]. In that reference, 0.61–0.80 is typically labeled 'substantial' and 0.81–0.99 'almost perfect.' Please correct the label or the reported score.
[Section IV-A, sample size] The sample-size justification uses a survey calculator for a proportion with 95% confidence and 5% margin of error, which yields 385; the paper rounds to 400. This is reasonable, but the paper does not report the resulting confidence intervals for the 55.5% or 44.0% proportions in Table III. Including confidence intervals would strengthen the RQ1 claim.
[Section VI] The statement 'the LLM successfully recognized all instances without ambiguity' is misleading: the LLM classified 0 of 400 descriptions as 'Unclear,' which may reflect overconfidence rather than successful recognition. Section V correctly identifies 'overconfidence bias,' so Section VI should be rephrased to avoid contradiction.
[General] The paper does not include a link to the data, prompts, or analysis scripts. Providing these artifacts would improve reproducibility and allow other researchers to verify the aggregate and per-item results.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the study compares independent manual labels, LLM classifications, and BERTScore similarities; Section V's admitted lack of per-item metrics is an evidentiary limitation, not a constructional equivalence.

full rationale

The paper's chain is empirical rather than derivational. The authors define three description categories (Table II), manually label 400 snippets (Table III), prompt Llama with the same category definitions and examples (Fig. 2), collect LLM classifications (Table IV), and compare original versus generated descriptions with BERTScore (Fig. 3). No quantity is fitted to the target it later 'predicts': the manual 55.5% Example proportion is not used to produce the LLM's 79.75% Example proportion, and no equation or construction forces one table to equal another. The dataset is taken from prior work [18], which is an external artifact rather than a self-citation carrying the paper's conclusion, and the Kappa-based coding procedure cites [23] only as methodological precedent. The paper itself concedes in Section V that 'full evaluation metrics (e.g., accuracy, F1-score) and human judgment were not implemented,' relying instead on 'indirect quantitative signals (such as category distribution alignment and BERT-based similarity scoring).' That admission correctly identifies that RQ2's aggregate-proportion comparison does not establish per-item agreement between Llama and human labels, but it is a threat to validity and inferential strength, not circularity. There is no step where an output is defined in terms of the input, no renamed known result, and no load-bearing uniqueness claim imported from the authors' prior work. Accordingly, the appropriate finding is no significant circularity, score 0.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper's claims rest on a sampled dataset, author-produced manual labels, a single LLM prompt, and an unvalidated similarity threshold. No new entities or fitted constants are introduced; the main burden is the implicit validity of the ground-truth labels and the BERTScore threshold.

free parameters (1)

Similarity relevance threshold = 0.9
A cutoff below which BERTScore values are interpreted as 'some irrelevance'; chosen post hoc from examples in Section IV-C and repeated in Section VI, with no derivation or validation.

assumptions (4)

domain assumption The random 400-snippet sample from the NPM Code Snippets dataset is representative of README code snippet descriptions in the wild.
Section III-A computes sample size with a survey calculator but gives no evidence that the underlying dataset or the random draw covers the population of npm READMEs.
domain assumption The authors' manual labels are accurate ground truth for code snippet description categories.
Section IV-A double-codes only the first 30 samples, then has a single author label the remaining 370; no external developers validate the categories.
domain assumption BERTScore is a valid measure of description relevance and the 0.9 threshold separates relevant from irrelevant outputs.
Section IV-C and Section V use BERTScore and assert the threshold without a baseline or human rating.
domain assumption Providing category definitions and example outputs in the prompt does not materially bias the LLM's classifications.
The prompt in Fig. 2 explicitly shows the expected output format and categories; the paper discusses prompt influence in Section V but performs no ablation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Uncovering Intention through LLM-Driven Code Snippet Description Generation." pith.science (2026). https://pith.science/paper/EEWIA467

@misc{pith2026250615453,
  author       = {Pith},
  title        = {Pith review of: Uncovering Intention through LLM-Driven Code Snippet Description Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EEWIA467}},
  note         = {Machine review of arXiv:2506.15453}
}

read the original abstract

Documenting code snippets is essential to pinpoint key areas where both developers and users should pay attention. Examples include usage examples and other Application Programming Interfaces (APIs), which are especially important for third-party libraries. With the rise of Large Language Models (LLMs), the key goal is to investigate the kinds of description developers commonly use and evaluate how well an LLM, in this case Llama, can support description generation. We use NPM Code Snippets, consisting of 185,412 packages with 1,024,579 code snippets. From there, we use 400 code snippets (and their descriptions) as samples. First, our manual classification found that the majority of original descriptions (55.5%) highlight example-based usage. This finding emphasizes the importance of clear documentation, as some descriptions lacked sufficient detail to convey intent. Second, the LLM correctly identified the majority of original descriptions as "Example" (79.75%), which is identical to our manual finding, showing a propensity for generalization. Third, compared to the originals, the produced description had an average similarity score of 0.7173, suggesting relevance but room for improvement. Scores below 0.9 indicate some irrelevance. Our results show that depending on the task of the code snippet, the intention of the document may differ from being instructions for usage, installations, or descriptive learning examples for any user of a library.

Figures

Figures reproduced from arXiv: 2506.15453 by the authors.

**Figure 2.** Example of prompts input to LLM for README file description creation. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Similarity values between the original and generated comments of [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 22 canonical work pages

[1]

Software documentation: The practition- ers’ perspective,

E. Aghajani, C. Nagy, M. Linares-Vasquez, L. Moreno, G. Bavota, M. Lanza, and D. C. Shepherd, “Software documentation: The practition- ers’ perspective,” in Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering , 2020, pp. 590–601

work page 2020
[2]

Augmenting API documentation with insights from Stack Overflow,

C. Treude and M. P. Robillard, “Augmenting API documentation with insights from Stack Overflow,” in 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE) , 2016, pp. 392–403

work page 2016
[3]

Project-specific code summarization with in-context learning,

S. Yun, S. Lin, X. Gu, and B. Shen, “Project-specific code summarization with in-context learning,” Journal of Systems and Software , vol. 216, p. 112149, 2024

work page 2024
[4]

Let’s discover more API relations: A large language model-based AI chain for unsupervised API relation inference,

Q. Huang, Y . Sun, Z. Xing, Y . Cao, J. Chen, X. Xu, H. Jin, and J. Lu, “Let’s discover more API relations: A large language model-based AI chain for unsupervised API relation inference,” ACM Transactions on Software Engineering and Methodology , vol. 33, no. 8, pp. 1–34, 2024

work page 2024
[5]

Software testing with large language models: Survey, landscape, and vision,

J. Wang, Y . Huang, C. Chen, Z. Liu, S. Wang, and Q. Wang, “Software testing with large language models: Survey, landscape, and vision,”IEEE Transactions on Software Engineering , 2024

2024
[6]

Evaluating diverse large language models for automatic and general bug reproduction,

S. Kang, J. Yoon, N. Askarbekkyzy, and S. Yoo, “Evaluating diverse large language models for automatic and general bug reproduction,” IEEE Transactions on Software Engineering , 2024

work page 2024
[7]

Cat- egorizing the content of GitHub readme files,

G. A. A. Prana, C. Treude, F. Thung, T. Atapattu, and D. Lo, “Cat- egorizing the content of GitHub readme files,” Empirical Software Engineering, vol. 24, pp. 1296–1327, 2019

work page 2019
[8]

How readme files are structured in open source Java projects,

Y . Liu, E. Noei, and K. Lyons, “How readme files are structured in open source Java projects,” Information and Software Technology , vol. 148, p. 106924, 2022

work page 2022

Show all 25 references

[9]

Language models are few-shot learners,

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language models are few-shot learners,” Advances in neural information processing systems , vol. 33, pp. 1877–1901, 2020

1901
[10]

Chatgpt for good? on opportunities and challenges of large language models for education,

E. Kasneci, K. Seßler, S. Kuchemann, M. Bannert, D. Dementieva, F. Fischer, U. Gasser, G. Groh, S. Gunnemann, E. Hullermeier et al., “Chatgpt for good? on opportunities and challenges of large language models for education,” Learning and individual differences , vol. 103, p. 1...

2023
[11]

An innovative approach for treating chronic vaginitis based on AI-driven drug repurposing,

H. Daungsupawong and V . Wiwanitkit, “An innovative approach for treating chronic vaginitis based on AI-driven drug repurposing,” Khaz- anah Informatika: Jurnal Ilmu Komputer dan Informatika , vol. 10, no. 1, pp. 30–35, 2024

2024
[12]

Autogenics: Automated gener- ation of context-aware inline comments for code snippets on program- ming Q&A sites using LLM,

S. D. Bappon, S. Mondal, and B. Roy, “Autogenics: Automated gener- ation of context-aware inline comments for code snippets on program- ming Q&A sites using LLM,” in 2024 IEEE International Conference on Source Code Analysis and Manipulation (SCAM) . IEEE, 2024, pp. 24–35

2024
[13]

A survey on large language models for code generation,

J. Jiang, F. Wang, J. Shen, S. Kim, and S. Kim, “A survey on large language models for code generation,” 2024. [Online]. Available: https://arxiv.org/abs/2406.00515

2024 arXiv
[14]

Large language models for code analysis: Do LLMs really do their job?

C. Fang, N. Miao, S. Srivastav, J. Liu, R. Zhang, R. Fang, R. Tsang, N. Nazari, H. Wang, H. Homayoun et al., “Large language models for code analysis: Do LLMs really do their job?” in 33rd USENIX Security Symposium (USENIX Security 24) , 2024, pp. 829–846

2024
[15]

An empirical study of readme contents for JavaScript packages,

S. Ikeda, A. Ihara, R. G. Kula, and K. Matsumoto, “An empirical study of readme contents for JavaScript packages,” IEICE TRANSACTIONS on Information and Systems , vol. 102, no. 2, pp. 280–288, 2019

2019
[16]

Understanding the archived projects on github,

X. Xia, S. Zhao, X. Zhang, Z. Lou, W. Wang, and F. Bi, “Understanding the archived projects on github,” in 2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) . IEEE, 2023, pp. 13–24

2023
[17]

Quantifying synergy between software projects using readme files only,

R. El Baff, S. Santhanam, and T. Hecking, “Quantifying synergy between software projects using readme files only,” in33rd International Conference on Software Engineering and Knowledge Engineering, SEKE 2021, vol. 33. KSI Research Inc. and Knowledge Systems Institute Graduate ...

2021
[18]

Ncq: Code reuse support for node.js developers,

B. Reid, M. d’Amorim, M. Wagner, and C. Treude, “Ncq: Code reuse support for node.js developers,” IEEE Transactions on Software Engineering, vol. 49, no. 5, pp. 3205–3225, 2023

2023
[19]

Generating diverse code explanations using the GPT-3 large language model,

S. MacNeil, A. Tran, D. Mogil, S. Bernstein, E. Ross, and Z. Huang, “Generating diverse code explanations using the GPT-3 large language model,” in Proceedings of the 2022 ACM Conference on International Computing Education Research-Volume 2 , 2022, pp. 37–39

2022
[20]

Bias as- sessment and mitigation in LLM-based code generation,

D. Huang, Q. Bu, J. Zhang, X. Xie, J. Chen, and H. Cui, “Bias as- sessment and mitigation in LLM-based code generation,” arXiv preprint arXiv:2309.14345, 2023

2023 arXiv
[21]

Larch: Large language model-based automatic readme creation with heuristics,

Y . Koreeda, T. Morishita, O. Imaichi, and Y . Sogawa, “Larch: Large language model-based automatic readme creation with heuristics,” in Proceedings of the 32nd ACM International Conference on Information and Knowledge Management , 2023, pp. 5066–5070

2023
[22]

Prompt programming for large language models: Beyond the few-shot paradigm,

L. Reynolds and K. McDonell, “Prompt programming for large language models: Beyond the few-shot paradigm,” in Extended abstracts of the 2021 CHI conference on human factors in computing systems , 2021, pp. 1–7

2021
[23]

How are project-specific forums utilized? a study of participation, content, and sentiment in the eclipse ecosystem,

Y . S. Nugroho, S. Islam, K. Nakasai, I. Rehman, H. Hata, R. G. Kula, M. Nagappan, and K. Matsumoto, “How are project-specific forums utilized? a study of participation, content, and sentiment in the eclipse ecosystem,” Empirical Software Engineering , vol. 26, pp. 1–33, 2021

2021
[24]

Understanding interobserver agreement: the kappa statistic,

A. J. Viera, J. M. Garrett et al., “Understanding interobserver agreement: the kappa statistic,” Fam med, vol. 37, no. 5, pp. 360–363, 2005

2005
[25]

From Tarzan to Tolkien: Controlling the language proficiency level of LLMs for content generation,

A. Malik, S. Mayhew, C. Piech, and K. Bicknell, “From Tarzan to Tolkien: Controlling the language proficiency level of LLMs for content generation,” in Findings of the Association for Computational Linguistics ACL 2024, 2024, pp. 15 670–15 693

2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Software documentation: The practition- ers’ perspective,

E. Aghajani, C. Nagy, M. Linares-Vasquez, L. Moreno, G. Bavota, M. Lanza, and D. C. Shepherd, “Software documentation: The practition- ers’ perspective,” in Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering , 2020, pp. 590–601

work page 2020

[2] [2]

Augmenting API documentation with insights from Stack Overflow,

C. Treude and M. P. Robillard, “Augmenting API documentation with insights from Stack Overflow,” in 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE) , 2016, pp. 392–403

work page 2016

[3] [3]

Project-specific code summarization with in-context learning,

S. Yun, S. Lin, X. Gu, and B. Shen, “Project-specific code summarization with in-context learning,” Journal of Systems and Software , vol. 216, p. 112149, 2024

work page 2024

[4] [4]

Let’s discover more API relations: A large language model-based AI chain for unsupervised API relation inference,

Q. Huang, Y . Sun, Z. Xing, Y . Cao, J. Chen, X. Xu, H. Jin, and J. Lu, “Let’s discover more API relations: A large language model-based AI chain for unsupervised API relation inference,” ACM Transactions on Software Engineering and Methodology , vol. 33, no. 8, pp. 1–34, 2024

work page 2024

[5] [5]

Software testing with large language models: Survey, landscape, and vision,

J. Wang, Y . Huang, C. Chen, Z. Liu, S. Wang, and Q. Wang, “Software testing with large language models: Survey, landscape, and vision,”IEEE Transactions on Software Engineering , 2024

2024

[6] [6]

Evaluating diverse large language models for automatic and general bug reproduction,

S. Kang, J. Yoon, N. Askarbekkyzy, and S. Yoo, “Evaluating diverse large language models for automatic and general bug reproduction,” IEEE Transactions on Software Engineering , 2024

work page 2024

[7] [7]

Cat- egorizing the content of GitHub readme files,

G. A. A. Prana, C. Treude, F. Thung, T. Atapattu, and D. Lo, “Cat- egorizing the content of GitHub readme files,” Empirical Software Engineering, vol. 24, pp. 1296–1327, 2019

work page 2019

[8] [8]

How readme files are structured in open source Java projects,

Y . Liu, E. Noei, and K. Lyons, “How readme files are structured in open source Java projects,” Information and Software Technology , vol. 148, p. 106924, 2022

work page 2022