Pith. sign in

REVIEW 3 major objections 6 minor 57 references

How Well Do LLMs Generate Taxonomies in the SE Domain? A Multi-perspective Evaluation Framework

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Automated taxonomy generation can match human quality in SE—when you pay for the slower method.

desk verdict A solid empirical comparison of TnT-LLM and CLIMB on SE taxonomies, with a trustworthy RQ1/RQ2/RQ4 core and an RQ3 reliability analysis that rests on unvalidated LLM annotators. read the letter →

arxiv 2608.01592 v1 pith:JLS5L5VN submitted 2026-08-03 cs.SE

Sota Nakashima , Yuta Ishimoto , Masanari Kondo , Tao Xiao , Yasutaka Kamei This is my paper · ORCID
classification cs.SE
keywords taxonomygenerationlargelanguagemodelssoftwareengineeringartifactsempiricalevaluationLLM-as-a-judgeinter-annotatoragreementTnT-LLMCLIMB
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to determine whether automated taxonomy generation—building a hierarchy of categories from a collection of artifacts—can succeed on software engineering materials, which mix natural language and code and often demand inference beyond surface text. To answer this, it builds a multi-perspective evaluation framework that scores taxonomies on quality, alignment with human-defined taxonomies, reliability under independent annotation, and efficiency, then applies it to two state-of-the-art methods (TnT-LLM and CLIMB) across eight human-defined taxonomies from seven SE studies. The central claim is a trade-off: TnT-LLM produces taxonomies of quality comparable to human-built ones, while CLIMB is 15–40× faster and 8–49× cheaper but loses quality when latent technical concepts must be inferred, especially on orthogonality. A sympathetic reader would take this as evidence that automated taxonomy generation is practical in SE, with method choice depending on artifact type, taxonomy depth, and budget.

What carries the argument

The load-bearing machinery is the multi-perspective evaluation framework itself, together with the two competing methods. TnT-LLM is an end-to-end LLM pipeline that summarizes each input instance, then iteratively generates, updates, and reviews categories to build a hierarchy; CLIMB first embeds and clusters instances into coherent groups, then uses LLM agents to name leaf categories and synthesize higher levels. The framework scores each generated taxonomy on four axes: quality (12 criteria in clarity, hierarchical coherence, orthogonality, completeness, rated by LLM judges validated against human raters), alignment with human-defined taxonomies (Heading Soft Recall, Catalogue Edit Distance Similarity, and Nodes Ratio), reliability (Krippendorff's alpha and label utilization across a panel of LLM annotators), and efficiency (API cost and wall-clock runtime). The metrics are what carry the argument: they convert the vague idea of 'good taxonomy' into four measurable quantities that can disagree, which is exactly how the trade-off between the two methods becomes visible.

What would settle it

Take any one of the eight datasets, recruit two human annotators to assign leaf categories from a generated taxonomy to a sample of the artifacts, and compare their agreement with the paper's LLM-panel agreement. If human-human alpha is substantially lower than the reported LLM-panel alpha (for example, below 0.4 where the paper reports 0.5–0.6), the reliability claim fails; if comparable, it is confirmed. A cheaper falsifier is to re-run RQ1 without the structural constraints that force generated taxonomies to match human layer counts, and check whether TnT-LLM's quality parity persists.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that the two fully automated, seed-free methods transfer to SE with opposite strengths. TnT-LLM, which summarizes each artifact and iteratively generates, updates, and reviews categories, achieves quality scores within +0.018 of human-defined taxonomies on average and recovers more human-like categories (HSR 0.874–0.898), but it over-generates on deep structures—in the Quantum Classical Issue dataset it produced on average 307 nodes versus 92 human nodes—and costs 8–49× more than CLIMB. CLIMB, which embeds and clusters artifacts and then has LLM agents name categories, is structurally closer to human taxonomies (Nodes ratios near 1, higher CEDS in several datasets) but loses quality on orthogonality criteria when latent technical concepts must be inferred. Reliability measured as LLM annotator agreement ranges from 0.489 to 0.647 across generators, with label utilization 0.859–0.981, and agreement correlates with non-overlap and structural alignment. The paper concludes that both methods are usable in practice if researchers first probe cost and taxonomy complexity on a subset of their data.

Load-bearing premise

The load-bearing premise is that a panel of LLM annotators agrees with human annotators closely enough that the measured reliability numbers (Krippendorff's alpha 0.489–0.647, label utilization 0.859–0.981) tell us how well people could apply these taxonomies—and the paper never validates LLM annotation against human annotation on the same taxonomies.

Editorial extensions

If this is right

  • SE researchers can use TnT-LLM to obtain taxonomies whose quality matches human-built ones, at the price of higher API cost and runtime and with a tendency to over-generate on deep hierarchies.
  • CLIMB is the right choice when cost and runtime dominate and the artifacts are topic-oriented, where surface-level similarity aligns with category membership; on such datasets it approaches human quality.
  • Taxonomy reliability depends on category clarity, non-overlap, and structural alignment (CEDS ρ=0.461), so annotator agreement can serve as a feedback signal to identify ambiguous or overlapping categories.
  • For multi-layer taxonomies, annotator agreement decreases at deeper layers, so human post-processing effort should concentrate on lower-level subcategories.
  • Before adopting either method at scale, researchers should run a small subset of the target data to estimate taxonomy complexity and cost, as the paper recommends.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the structural constraints (matching human layer count and leaf count) were removed, TnT-LLM's quality advantage might shrink, since part of its parity may come from being told the target size; this is testable by re-running RQ1 with free structure.
  • Replacing the LLM annotator panel with human annotators on a subsample is the obvious validity check for RQ3; until that comparison exists, the claim that generated taxonomies are 'broadly usable' rests on an unvalidated proxy.
  • The significant correlation between CEDS and agreement suggests a practical repair loop: adjust a generated taxonomy until its tree-edit distance to a human reference drops, and annotation reliability should rise accordingly.
  • The cost estimates assume no prompt caching and treat all requests as cache misses, so real deployment costs with caching could be lower than reported, narrowing the gap between TnT-LLM and CLIMB in practice.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This experience paper proposes a multi-perspective evaluation framework for automated taxonomy generation on software engineering (SE) artifacts, covering taxonomy quality (RQ1), alignment with human-defined taxonomies (RQ2), reliability under independent annotation (RQ3), and cost/runtime efficiency (RQ4). The authors curate seven SE papers with eight human-defined taxonomies, run two automated methods (TnT-LLM and CLIMB) with five LLMs, and validate their LLM-as-judge quality scoring against two human evaluators in a preliminary study, including a self-bias check. The main finding is a trade-off: TnT-LLM produces taxonomies whose judged quality is comparable to human-defined ones but at substantially higher cost and runtime and with a tendency toward overly complex structures, whereas CLIMB is 15--40x faster and 8--49x cheaper but scores lower on quality when technical inference beyond surface-level similarity is required. The paper also reports that inter-annotator agreement is moderately high and label utilization is high for generated taxonomies, with agreement correlated with category clarity and structural alignment. The study is framed as a first step toward systematic understanding of automated taxonomy generation in SE, with actionable guidance for method selection and human post-processing.

Significance. If the results hold, the paper makes a useful contribution: it is, to my knowledge, the first multi-dataset evaluation of seed-free LLM-based taxonomy generation on technically complex SE artifacts, and it provides a reusable evaluation framework and a curated dataset of eight human-defined taxonomies with public artifacts. The authors' care in validating the RQ1 LLM-as-judge setup against human evaluators on all datasets, checking for self-bias, and reporting a variability analysis for generation and evaluation are concrete methodological strengths. The paper's practical recommendation—to run a small pilot before committing to automated taxonomy generation—is sensible and actionable. However, the significance is conditional: the RQ3 reliability results, which support the 'broadly usable' claim, rest entirely on LLM annotators with no human-annotation baseline, and the structural constraints used for fair method comparison are not always respected. These two issues affect load-bearing parts of the central practical-usability conclusion, so the manuscript needs revision before the findings can be accepted at face value.

major comments (3)
  1. [Section 9.1, Tables 9–10, Section 12] The RQ3 reliability results are computed from a panel of LLM annotators, and the rows labeled 'Human' in Tables 9 and 10 are LLM annotations of the human-defined taxonomies, not human annotations. The paper validates the LLM-as-judge setup for RQ1 quality scoring (Section 6) but provides no analogous validation for the RQ3 annotation task. Consequently, the claims 'generated taxonomies are broadly usable' (RQ3 answer) and the Table 11 correlations describe inter-LLM agreement. Because LLM annotators may share systematic biases from common pretraining and instruction-following conventions, their agreement and label utilization may differ from human annotators' agreement. The paper's own Section 12 Construct Validity concedes that LLM judgments 'may not perfectly reflect how human experts would assess taxonomy quality (RQ1) or how annotators would apply the same taxonomies in practice (RQ3).' To support the practical-usability conclusion, the authors should add a human-annotation baseline on a subset of taxonomies, or at minimum reword the RQ3 claims to refer to LLM-based annotation reliability rather than general usability.
  2. [Section 5.2 and Table 7] The experimental setup states that both methods are instructed to match the structural properties of the human-defined taxonomy (number of layers, top-level categories, and leaf categories) to enable a fair comparison. However, Table 7 reports Nodes Ratios for TnT-LLM as high as 3.341 (Quantum Classical Issue) and 1.294 (Fairness API Issue Topic), which indicate that the structural constraints are not reliably followed. Since the abstract's central trade-off includes 'TnT-LLM tends to generate overly complex taxonomies,' the paper should clarify whether these constraints are enforced (e.g., by post-processing or re-prompting) or are only prompt-level instructions, and discuss how unenforced constraints affect the interpretation of over-generation as a property of the method rather than as an instruction-following artifact. This also bears on the fairness of the RQ1 quality comparison and the RQ3 agreement comparison between methods, because CLIMB's closer adherence to the target size may penalize or benefit it relative to TnT-LLM.
  3. [Section 9.2, Table 11] The associations reported in Table 11 (e.g., OR-NonOv with rho=0.313 and CEDS with rho=0.461) are correlations between LLM-annotator agreement and LLM-evaluated quality/alignment. Since both sides of each correlation are produced by LLMs, common biases in how LLMs perceive category clarity and apply labels could inflate these associations independently of any true relationship with human annotation behavior. The self-bias check described in Section 12 only covers RQ1 evaluation, not the RQ3 annotation task. A human-annotation baseline would allow the authors to test whether these associations hold for human annotators; without it, the guidance to 'leverage annotation reliability as feedback for refining ambiguous categories' (Section 11.3) is built on inter-LLM agreement only.
minor comments (6)
  1. [Section 4, CEDS and Nodes Ratio definitions] Please clarify the scale and normalization of Catalogue Edit Distance Similarity (CEDS); the reported values (e.g., 52.001 and 59.313 in Table 7) suggest a transformation of a normalized tree-edit distance, but the paper does not state the range or what value indicates perfect similarity. Similarly, Nodes Ratio is defined as a ratio, but the paper should state explicitly that values far from 1 indicate size mismatches and how ties to the instructed structure constraints are handled.
  2. [Section 5.2, implementation details] The implementation description mentions K-means clustering for CLIMB but does not state the random seed or number of restarts; specifying these details, or noting that default settings were used, would improve reproducibility of the clustering step.
  3. [Section 6.2, cross-dataset validation] The cross-dataset validation of the evaluator LLMs uses only taxonomies generated by GPT-5.2. Since RQ1 also reports results for other generator LLMs and for CLIMB, the validation coverage is incomplete; if feasible, extending the human-evaluator comparison to at least one CLIMB taxonomy and one weaker generator per dataset would strengthen the claim that the LLM-as-judge validation generalizes across methods.
  4. [Section 9.1, Tables 9 and 10] The column header 'Human' in Tables 9 and 10 is misleading because the annotations are performed by LLMs on human-defined taxonomies. Rename these columns to something like 'LLM-annotated Human-defined' or 'Human taxonomy (LLM annotations)' to avoid implying that human annotators produced the agreement values.
  5. [Section 11.3] The text contains a garbled icon sequence 'hand-p⌢int-right' before several implication paragraphs; this appears to be a rendering artifact and should be removed or replaced with proper markup.
  6. [Section 5.1] The study selection process is called a systematic literature review, but it searches only two digital libraries and five venues; consider describing it as a targeted or structured search to avoid overclaiming generality, and acknowledge in the external-validity discussion the possibility of missing relevant taxonomies published in other venues or libraries.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the evaluation is self-contained against external human-defined taxonomies and human-evaluator agreement, with no fitted parameter renamed as a prediction.

full rationale

The paper is an empirical measurement study, not a derivation, and its central claims do not reduce to their inputs. RQ1 quality is scored by evaluator LLMs whose scores are calibrated against two human evaluators on one dataset (Section 6.1, Tables 5-6) and cross-validated on the remaining seven datasets (Section 6.2, mean pairwise |Δ| < 0.15); the comparison baseline is the externally constructed human-defined taxonomy. RQ2 alignment (HSR, CEDS, Nodes Ratio) is computed against human-defined taxonomies from seven curated SE papers (Section 5.1), i.e., an external benchmark, and although the methods are prompted with target structural properties for fair comparison, Table 7 shows the generated node counts vary substantially (e.g., Nodes Ratio 0.756-3.341), so the alignment scores are not forced by construction. RQ4 cost and runtime are direct measurements of API usage and wall-clock time (Section 10.1). The only notable weakness is construct validity: RQ3 reliability is measured with an LLM annotator panel rather than human annotators, and the paper explicitly concedes in Section 12 that LLM judgments 'may not perfectly reflect how human experts would assess taxonomy quality (RQ1) or how annotators would apply the same taxonomies in practice (RQ3).' This is a validity threat, not a circular reduction: inter-LLM agreement is compared across independently sampled LLM annotators and against LLM annotations of the human-defined taxonomies, but no parameter is fitted from the outcome being claimed, and no claim is justified solely by a self-citation. The self-citation [30] is contextual and non-load-bearing. Therefore, no step in the paper's chain is equivalent to its own input by construction, and the score is 0.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The study does not introduce new entities or fitted parameters. Its central claims rest on assumptions about LLM-based evaluation and annotation fidelity, dataset representativeness, and the effectiveness of structural matching, which are stated and partially validated.

assumptions (5)
  • domain assumption LLM evaluators' quality scores approximate human expert quality judgments
    RQ1 relies on LLM-as-a-judge; validated with two human evaluators on all datasets (mean pairwise |Δ| < 0.15, Section 6.2), but the human sample is small.
  • domain assumption LLM annotators are valid proxies for human annotators in measuring taxonomy usability
    RQ3 uses a panel of LLM annotators without human-annotation validation (Section 9.1), which is load-bearing for reliability claims.
  • domain assumption The seven selected papers (eight taxonomies) are representative of SE taxonomy construction
    Selection based on systematic review with inclusion/exclusion criteria (Section 5.1); external validity is limited as acknowledged in Section 12.
  • domain assumption Instructing methods to match human taxonomy structure (layers, top categories, leaf categories) yields a fair comparison
    Section 5.2; observed deviations (e.g., Nodes Ratio 3.341 for TnT-LLM on Quantum Classical Issue) suggest imperfect control, potentially confounding comparisons.
  • standard math Krippendorff's alpha and Gwet's AC2 are appropriate agreement statistics
    Standard measures, but the small numbers of annotators (three LLMs for agreement) affect the reliability of the estimates.

how reviews work

0 comments
Cite this review

Pith. "Pith review of How Well Do LLMs Generate Taxonomies in the SE Domain? A Multi-perspective Evaluation Framework." pith.science (2026). https://pith.science/paper/JLS5L5VN

@misc{pith2026260801592,
  author       = {Pith},
  title        = {Pith review of: How Well Do LLMs Generate Taxonomies in the SE Domain? A Multi-perspective Evaluation Framework},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JLS5L5VN}},
  note         = {Machine review of arXiv:2608.01592}
}
abstract

Taxonomies provide a shared conceptual framework for organizing heterogeneous observations in software engineering (SE) research. Manually constructing such taxonomies is labor-intensive and requires annotators with expertise in the SE domain. While advances in Large Language Models (LLMs) have led to the emergence of automated taxonomy generation methods outside the SE domain, their applicability to technically complex SE artifacts remains unclear. In this experience paper, we present the first comprehensive empirical evaluation of how state-of-the-art automated methods perform on SE artifacts through a multi-perspective evaluation framework, including taxonomy quality, alignment with taxonomies defined by human experts, reliability under independent annotation, and efficiency. To support this evaluation, we systematically collect seven SE papers with publicly available artifacts and human-defined taxonomies, and conduct experiments using two automated methods (TnT-LLM and CLIMB) with five state-of-the-art LLMs. Our evaluation reveals a clear trade-off: TnT-LLM constructs high-quality taxonomies comparable to human-defined ones but incurs substantially higher cost and runtime and tends to generate overly complex taxonomies, whereas CLIMB is 15--40$\times$ faster and 8--49$\times$ cheaper but tends to score lower on quality when technical inference beyond surface-level similarity is required. These findings suggest that TnT-LLM and CLIMB can be used in practical situations in the SE domain, while researchers should first assess the complexity of the generated taxonomies and their cost using a subset of the target data to decide whether to use automated methods or human experts. Our work represents a first step toward a systematic understanding of automated taxonomy generation in SE, offering actionable insights for future research and practice.

Figures

Figures reproduced from arXiv: 2608.01592 by the authors.

Figure 1
Figure 1. Taxonomy generation and annotation in SE [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Study identification and selection process [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Example contents included in our prompts [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: RQ1 - Average quality score aggregated by dataset [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: RQ1 - Average quality score aggregated by generator LLM [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Representative branches of taxonomies for [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 21 canonical work pages

  1. [1]

    Waleed Abdeen, Michael Unterkalmsteiner, Alexandros Chirtoglou, Christoph Paul Schimanski, Heja Goli, and Krzysztof Wnuk. 2025. Taxonomic Trace Links: Rethinking Traceability and its Benefits. doi:10.48550/arXiv.2504.20507

    work page Pith review arXiv doi:10.48550/arxiv.2504.20507 2025

  2. [2]

    Toufique Ahmed and Premkumar Devanbu. 2022. Few-Shot Training LLMs for Project-Specific Code-Summarization. InProc. of the 37th IEEE/ACM Interna- tional Conference on Automated Software Engineering. Association for Computing Machinery, New York, NY, USA, 1–5. doi:10.1145/3551349.3559555

    arXiv 2022

  3. [3]

    Toufique Ahmed, Premkumar Devanbu, Christoph Treude, and Michael Pradel

  4. [4]

    R Artusi, P Verderio, and E Marubini. 2002. Bravais-Pearson and Spearman Correlation Coefficients: Meaning, Test of Hypothesis and Confidence Interval. The International journal of biological markers17, 2 (2002), 148–151. doi:10.1177/ 172460080201700213

    work page 2002

  5. [5]

    Simone Balloccu, Patrícia Schmidtová, Mateusz Lango, and Ondrej Dusek. 2024. Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed- Source LLMs. InProc. of the 18th Conference of the European Chapter of the Asso- ciation for Computational Linguistics. Association for Computational Linguistics, St. Julian’s, Malta, 67–93. doi:10.186...

    doi:10.18653/v1/2024.eacl-long.5 2024

  6. [6]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language Models Are Few-Shot Learners.Advances in Neural Information Processing Systems33 (2020), 1877–1901

    work page 2020

  7. [7]

    Giuseppe Crupi, Rosalia Tufano, Alejandro Velasco, Antonio Mastropaolo, Denys Poshyvanyk, and Gabriele Bavota. 2025. On the Effectiveness of LLM-as-a- Judge for Code Generation and Summarization.IEEE Transactions on Software Engineering51, 8 (2025), 2329–2345. doi:10.1109/TSE.2025.3586082

    arXiv 2025

  8. [8]

    Ajoy Das, Gias Uddin, Shaiful Chowdhury, Mostafijur Rahman Akhond, and Hadi Hemmati. 2026. Applications and Challenges of Fairness APIs in Machine Learning Software.ACM Transactions on Software Engineering and Methodology 35, 6 (2026), 1–47. doi:10.1145/3765735

    work page doi:10.1145/3765735 2026

Show all 57 references
  1. [9]

    Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, and Ge Li. 2024. Generalization or Memorization: Data Contamination and Trustwor- thy Evaluation for Large Language Models. InFindings of the Association for Computational Linguistics. Association for Computati...

    2024 doi

  2. [10]

    Amir Mohammad Ebrahimi, Gustavo A Oliva, and Ahmed E Hassan. 2023. Self- Admitted Technical Debt in Ethereum Smart Contracts: A Large-Scale Exploratory Study.IEEE Transactions on Software Engineering49, 9 (2023), 4304–4323. doi:10. 1109/TSE.2023.3289808

    2023

  3. [11]

    Fu Fan, Yanjie Jiang, Tianyi Chen, Hengshun Zhang, Yuxia Zhang, Nan Niu, and Hui Liu. 2025. An Empirical Study on Common Sense-Violating Bugs in Mobile Apps.ACM Transactions on Software Engineering and Methodology34, 6 (2025), 1–26. doi:10.1145/3709356

    2025 doi

  4. [12]

    Pasi Fränti and Radu Mariescu-Istodor. 2023. Soft Precision and Recall.Pattern Recognition Letters167 (2023), 115–121. doi:10.1016/j.patrec.2023.02.005

    2023 doi

  5. [13]

    Jinlan Fu, See Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2024. Gptscore: Evalu- ate as you desire. InProc. of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). Association f...

    2024 doi

  6. [14]

    Michael Gunn, Dohyun Park, and Nidhish Kamath. 2024. Creating a Fine-Grained Entity Type Taxonomy Using LLMs. doi:10.48550/arXiv.2402.12557

    2024 doi arXiv

  7. [15]

    Kilem Li Gwet. 2008. Computing Inter-Rater Reliability and Its Variance in the Presence of High Agreement.Brit. J. Math. Statist. Psych.61, 1 (2008), 29–48. doi:10.1348/000711006X126600

    2008 doi

  8. [16]

    Rajarshi Haldar and Julia Hockenmaier. 2025. Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks. InFindings of the Association for Computational Linguistics. Association for Computational Linguistics, Suzhou, China, 24986– 25004. doi:10.18653/v1/2025.findings-emnlp.1361

    2025 doi

  9. [17]

    Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large Language Models for Software Engineering: A Systematic Literature Review.ACM Transactions on Software Engineering and Methodology33, 8 (2024), 1–79. d...

    2024 doi

  10. [18]

    Jiaxin Huang, Yiqing Xie, Yu Meng, Yunyi Zhang, and Jiawei Han. 2020. Corel: Seed-Guided Topical Taxonomy Construction by Concept Learning and Relation Transferring. InProc. of the 26th ACM SIGKDD International Conference on Knowl- edge Discovery & Data Mining. Association for...

    2020

  11. [19]

    Priyanka Kargupta, Nan Zhang, Yunyi Zhang, Rui Zhang, Prasenjit Mitra, and Jiawei Han. 2025. TaxoAdapt: Aligning LLM-Based Multidimensional Taxonomy Construction to Evolving Research Corpora. InProc. of the 63rd Annual Meeting of the Association for Computational Linguistics. ...

    2025 doi

  12. [20]

    Yutaro Kashiwa, Ryoma Nishikawa, Yasutaka Kamei, Masanari Kondo, Emad Shihab, Ryosuke Sato, and Naoyasu Ubayashi. 2022. An Empirical Study on Self-Admitted Technical Debt in Modern Code Review.Information and Software Technology146 (2022), 106855. doi:10.1016/j.infsof.2022.106855

    2022

  13. [21]

    2019.Content Analysis: An Introduction to Its Methodology

    Klaus Krippendorff. 2019.Content Analysis: An Introduction to Its Methodology. SAGE Publications, Inc. doi:10.4135/9781071878781

    2019 doi

  14. [22]

    Dongha Lee, Jiaming Shen, SeongKu Kang, Susik Yoon, Jiawei Han, and Hwanjo Yu. 2022. Taxocom: Topic Taxonomy Completion with Hierarchical Discovery of Novel Topic Clusters. InProc. of the ACM Web Conference. Association for Computing Machinery, New York, NY, USA, 2819–2829. do...

    2022 doi

  15. [23]

    Dawei Li, Renliang Sun, Yue Huang, Ming Zhong, Bohan Jiang, Jiawei Han, Xiangliang Zhang, Wei Wang, and Huan Liu. 2025. Preference Leakage: A Con- tamination Problem in LLM-as-a-Judge. doi:10.48550/arXiv.2502.01534

    2025 doi

  16. [24]

    Nan Li, Bo Kang, and Tijl De Bie. 2025. Building Data-Driven Occupation Tax- onomies: A Bottom-Up Multi-Stage Approach via Semantic Clustering and Multi- Agent Collaboration. InProc. of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track. As...

    2025 doi

  17. [25]

    Yucheng Li, Frank Guerin, and Chenghua Lin. 2024. Latesteval: Addressing data contamination in language model evaluation through dynamic and time-sensitive test construction.Proceedings of the AAAI Conference on Artificial Intelligence38, 17 (2024), 18600–18607. doi:10.1609/aa...

    2024 doi

  18. [26]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. In Proc. of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, S...

    2023

  19. [27]

    Henry B Mann and Donald R Whitney. 1947. On a test of whether one of two random variables is stochastically larger than the other.The annals of mathematical statistics(1947), 50–60

    1947

  20. [28]

    Oleksandr Marchenko and Danylo Dvoichenkov. 2024. TaxoRankConstruct: A Novel Rank-Based Iterative Approach to Taxonomy Construction with Large Language Models. InProc. of the Information Technology and Implementation. 11–27

    2024

  21. [31]

    Sota Nakashima, Yuta Ishimoto, Masanari Kondo, Tao Xiao, and Yasutaka Kamei

  22. [32]

    How Far Have LLMs Come Toward Automated SATD Taxonomy Con- struction?. InProc. of the 32nd Asia-Pacific Software Engineering Conference. IEEE, Piscataway, NJ, USA, 832–836. doi:10.1109/APSEC66846.2025.00087

    2025

  23. [33]

    OpenAI. 2023. tiktoken. https://github.com/openai/tiktoken

    2023

  24. [34]

    OpenRouter. [n.d.]. OpenRouter. https://openrouter.ai/. Accessed: 2025-10-11

    2025

  25. [35]

    Olivier Nourry, Yutaro Kashiwa, Weiyi Shang, Honglin Shu, and Yasutaka Kamei

  26. [36]

    ACM Transactions on Software Engineering and Methodology34, 2 (2025), 1–30

    My fuzzers won’t build: An empirical study of fuzzing build failures. ACM Transactions on Software Engineering and Methodology34, 2 (2025), 1–30. doi:10.1145/3688842

    2025 doi

  27. [37]

    Chirag Shah, Ryen White, Reid Andersen, Georg Buscher, Scott Counts, Sarkar Das, Ali Montazer, Sathish Manivannan, Jennifer Neville, Nagu Rangan, et al

  28. [38]

    Jingbo Shang, Xinyang Zhang, Liyuan Liu, Sha Li, and Jiawei Han. 2020. Nettaxo: Automated Topic Taxonomy Construction from Text-Rich Network. InProc. of the ACM Web Conference. Association for Computing Machinery, New York, NY, USA, 1908–1919. doi:10.1145/3366423.3380259

    2020

  29. [39]

    Arjun Panickssery, Samuel Bowman, and Shi Feng. 2024. LLM Evaluators Recog- nize and Favor Their Own Generations.Advances in Neural Information Processing Systems37 (2024), 68772–68802. doi:10.52202/079017-2197

    2024 doi

  30. [41]

    Rosalia Tufano, Ozren Dabić, Antonio Mastropaolo, Matteo Ciniselli, and Gabriele Bavota. 2024. Code Review Automation: Strengths and Weaknesses of the State of the Art.IEEE Transactions on Software Engineering50, 2 (2024), 338–353. doi:10.1109/TSE.2023.3348172

    2024

  31. [42]

    doi:10.1145/3732294

    Using Large Language Models to Generate, Validate, and Apply User Intent Taxonomies.ACM Transactions on the Web19, 3 (2025), 1–29. doi:10.1145/3732294

    2025 doi

  32. [43]

    Mengting Wan, Tara Safavi, Sujay Kumar Jauhar, Yujin Kim, Scott Counts, Jennifer Neville, Siddharth Suri, Chirag Shah, Ryen W White, Longqi Yang, et al. 2024. TnT-LLM: Text Mining at Scale with Large Language Models. InProc. of the 30th SIGKDD Conference on Knowledge Discovery...

    2024 doi

  33. [44]

    Yijia Shao, Yucheng Jiang, Theodore Kanell, Peter Xu, Omar Khattab, and Monica Lam. 2024. Assisting in Writing Wikipedia-like Articles from Scratch with Large Language Models. InProc. of the 2024 Conference of the North American Chapter of the Association for Computational Lin...

    2024 doi

  34. [45]

    Jiaming Shen, Zeqiu Wu, Dongming Lei, Chao Zhang, Xiang Ren, Michelle T Vanni, Brian M Sadler, and Jiawei Han. 2018. Hiexpan: Task-Guided Taxonomy Construction by Hierarchical Tree Expansion. InProc. of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data...

    2018 doi

  35. [46]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-Thought Prompting Elicits Rea- soning in Large Language Models.Advances in Neural Information Processing Systems35 (2022), 24824–24837

    2022

  36. [47]

    Muhammad Usman, Ricardo Britto, Jürgen Börstler, and Emilia Mendes. 2017. Taxonomies in Software Engineering: A Systematic Mapping Study and a Revised Taxonomy Development Method.Information and Software Technology85 (2017), 43–59. doi:10.1016/j.infsof.2017.01.006

    2017 doi

  37. [48]

    Jake Zappin, Trevor Stalnaker, Oscar Chaparro, and Denys Poshyvanyk. 2025. When Quantum Meets Classical: Characterizing Hybrid Quantum-Classical Issues Discussed in Developer Forums. In2025 IEEE/ACM 47th International Conference on Software Engineering. IEEE, Piscataway, NJ, U...

    2025

  38. [49]

    Bo Wang, Chong Chen, Junjie Chen, Bowen Xu, Chen Ye, Youfang Lin, Guoliang Dong, and Jun Sun. 2025. A Comprehensive Study of OOP-Related Bugs in C++ Compilers.IEEE Transactions on Software Engineering51, 6 (2025), 1762–1782. doi:10.1109/TSE.2025.3566490

    2025

  39. [51]

    He Zhang, Muhammad Ali Babar, and Paolo Tell. 2011. Identifying Relevant Studies in Software Engineering.Information and Software Technology53, 6 (2011), 625–637. doi:10.1016/j.infsof.2010.12.010

    2011 doi

  40. [52]

    Jiongchi Yu, Weipeng Jiang, Xiaoyu Zhang, Qiang Hu, Xiaofei Xie, and Chao Shen

  41. [53]

    doi:10.48550/arXiv.2510.04997

    AutoEmpirical: LLM-Based Automated Research for Empirical Software Fault Analysis. doi:10.48550/arXiv.2510.04997

    doi

  42. [54]

    Kun Zhu, Xiaocheng Feng, Xiachong Feng, Yingsheng Wu, and Bing Qin. 2023. Hierarchical Catalogue Generation for Literature Review: A Benchmark. InFind- ings of the Association for Computational Linguistics: EMNLP 2023. Association for Computational Linguistics, Singapore, 6790...

    2023 doi

  43. [55]

    Qingkai Zeng, Yuyang Bai, Zhaoxuan Tan, Shangbin Feng, Zhenwen Liang, Zhihan Zhang, and Meng Jiang. 2024. Chain-of-Layer: Iteratively Prompting Large Language Models for Taxonomy Induction from Limited Examples. In Proc. of the 33rd ACM International Conference on Information ...

    2024

  44. [57]

    Chao Zhang, Fangbo Tao, Xiusi Chen, Jiaming Shen, Meng Jiang, Brian Sadler, Michelle Vanni, and Jiawei Han. 2018. Taxogen: Unsupervised Topic Taxonomy Construction by Adaptive Term Embedding and Clustering. InProc. of the 24th ACM SIGKDD International Conference on Knowledge D...

    2018

  45. [59]

    Haiqi Zhang, Zhengyuan Zhu, Zeyu Zhang, and Chengkai Li. 2025. LLMTaxo: Leveraging Large Language Models for Constructing Taxonomy of Factual Claims from Social Media. InFindings of the Association for Computational Linguistics. Association for Computational Linguistics, Vienn...

    2025

  46. [60]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judg- ing LLM-as-a-Judge with MT-Bench and Chatbot Arena.Advances in Neural Information Processing Systems36 (2023), 46595–46623

    2023

  47. [2025]

    Can LLMs Replace Manual Annotation of Software Engineering Artifacts?. InProc. of the 22nd International Conference on Mining Software Repositories. IEEE, Piscataway, NJ, USA, 526–538. doi:10.1109/MSR66628.2025.00086

    2025

  48. [2026]

    https://doi.org/10.5281/zenodo.21388470

    Replication Package. https://doi.org/10.5281/zenodo.21388470

    doi

  49. [3102]

    doi:10.1145/3627673.3679608

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.