Pith. sign in

REVIEW 4 major objections 5 minor 55 references

CMER: A Context-Aware Approach for Mining Ethical Concern-related App Reviews

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CMER claims that combining natural language inference with finance-specific hypotheses and a zero-shot Llama-style LLM recovers privacy and security reviews that keyword-based sampling misses, demonstrated by 2,178 confirmed reviews from…

desk verdict Useful dataset and clearly described pipeline, but the paper's central claim about keyword-free extraction is unverified and contradicted by its own examples. read the letter →

arxiv 2507.09049 v2 pith:OVF7QLA5 submitted 2025-07-11 cs.SE

Aakash Sorathiya , Gouri Ginde This is my paper
classification cs.SE
keywords appreviewsprivacyandsecuritynaturallanguageinferencelargemodelszero-shotclassificationdomain-specifichypothesesmobileinvestmentappsrequirementsengineering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that ethical concerns in app reviews, especially privacy and security, are too domain-dependent for keyword- or regex-based sampling, and that a two-stage pipeline can recover them. Stage one uses natural language inference with 17 finance-specific hypotheses to filter a large corpus down to plausible privacy and security reviews; stage two uses a decoder-only LLM, Llama-3.1-8B-Instruct, with zero-shot prompting to classify the survivors. On a labeled test set, domain-specific hypotheses raise NLI F1 from 0.56 to 0.65, and the zero-shot LLM reaches 0.85, beating a fine-tuned T5 baseline at 0.80. Applied to 382,432 unlabeled reviews, the pipeline produced 2,178 privacy and security reviews that a previous keyword-based study missed, all confirmed by manual inspection. If valid, this gives requirements engineers a way to surface ethical concerns without building labeled datasets for every app domain.

What carries the argument

The carrying mechanism is the two-stage filtering chain. Stage one is NLI with domain-specific hypotheses: for each review, an NLI model computes entailment scores against 17 hypotheses such as "The app collects more financial data than necessary"; a review becomes a candidate when it satisfies NE(i,0.85)>=1 or NE(i,0.75)>=3 or NE(i,0.7)>=5, where NE(i,t) counts hypotheses with entailment score above t. This reduces 382,432 reviews to 14,678 candidates. Stage two is zero-shot classification by Llama-3.1-8B-Instruct with a role-based prompt asking for a yes/no privacy/security label; five runs at temperature 0 with majority vote narrow the candidates to 3,160, and manual inspection confirms 2,178. The two stages trade off recall and precision so that human effort is spent only on a small, concentrated set.

What would settle it

Re-run the RQ1 comparison with the 31 generic hypotheses from [9] but the paper's new threshold rule (NE(i,0.85)>=1 or NE(i,0.75)>=3 or NE(i,0.7)>=5); if F1 rises to about 0.65 with generic hypotheses, the claimed benefit of domain-specific context awareness is not supported.

Watch

Extended reading notes

Core claim

The paper's central claim is that ethical concerns in app reviews are domain-dependent, so generic privacy language misses them, and that a two-stage pipeline can recover them without labeled training data. In the first stage, natural language inference scores each review against 17 finance-specific hypotheses derived from a privacy and security risk taxonomy for banking apps; a review is flagged as a potential privacy/security review if enough hypotheses entail it under a specified threshold rule. In the second stage, Llama-3.1-8B-Instruct classifies the flagged reviews as privacy/security related or not using a zero-shot role-based prompt with temperature 0 and majority voting over five runs. The paper reports that the domain-specific hypotheses improve NLI F1 from 0.56 to 0.65 over generic privacy hypotheses, that the zero-shot LLM reaches F1 0.85 and beats a fine-tuned T5 baseline at 0.80, and that on 382,432 unlabeled investment-app reviews the full pipeline produced 3,160 positive classifications of which 2,178 were confirmed by manual inspection as privacy/security reviews missed by a keyword-based study.

Load-bearing premise

The load-bearing premise is that the F1 improvement in RQ1 comes from the domain-specific hypotheses themselves, rather than from the new threshold rule that was introduced together with them; the comparison changes both at once.

Editorial extensions

If this is right

  • Keyword- and regex-based sampling of app reviews systematically misses a class of privacy/security complaints that mention no predefined term, so previous estimates of privacy concern volume in finance apps are undercounts.
  • Zero-shot instruction-tuned LLMs can replace fine-tuned supervised classifiers for this review-classification task, removing the need for labeled datasets when a new domain or concern is targeted.
  • Domain-specific NLI hypotheses improve candidate filtering over generic privacy hypotheses, supporting the paper's premise that ethical concerns are expressed differently across domains.
  • The released 2,178-review dataset gives requirements engineers concrete evidence for privacy/security requirements in investment apps, and the pipeline is reusable for other domains and ethical concerns such as safety, transparency, and accountability by writing new hypotheses.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I infer that the NLI stage acts mainly as a recall-oriented prefilter whose value is economic: because privacy/security reviews are a tiny fraction of feedback, shrinking the corpus before LLM inference is what makes the approach practical at scale; the paper asserts this but does not quantify the cost saved.
  • A direct extension would ablate the two components: feed the LLM a random sample of the full 382K reviews instead of the NLI-filtered candidates, and compare precision, recall, and compute. The paper leaves this comparison unmeasured.
  • I infer that the threshold rule, not just the hypotheses, carries part of the RQ1 gain; an experiment holding hypotheses fixed and varying only the thresholds would separate the two effects and is an immediate next step for a reader wanting to reuse the method.
  • Because no fine-tuning is required, the same CMER design should transfer to other domains and channels, such as health and fitness reviews, ride-hailing reviews, and Reddit posts, with new hypotheses; the open dataset makes this testable without labeling from scratch.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes CMER, a two-stage pipeline that uses Natural Language Inference (NLI) with finance-domain-specific hypotheses to filter potentially privacy/security-related app reviews, then uses a zero-shot decoder-only LLM (Llama-3.1-8B-Instruct) to classify the remaining reviews. The authors evaluate four NLI models and compare domain-specific hypotheses with generic privacy hypotheses on a labeled dataset from Ebrahimi et al., compare three LLM classifiers against a fine-tuned T5 baseline, and apply the full pipeline to 382,432 unlabeled investment-app reviews, claiming to extract 2,178 additional PSRs that escaped the previous keyword-based method. The paper also reports a manually validated dataset and states that the code and data are open-sourced.

Significance. If the results hold, the hybrid NLI+LLM approach would offer a practical, label-free way to mine ethical-concern reviews at scale, and the released dataset of 2,178 PSRs would be a reusable resource for requirements engineering. The paper has several strengths: RQ1 and RQ2 are evaluated against an externally labeled dataset; the RQ3 manual validation involves multiple annotators with a reported Cohen's kappa of 0.78; the zero-shot LLM setup is described in reasonable detail; and the authors commit to releasing code and data for reproducibility. However, the current evidence does not yet support the central claims as stated, because the RQ1 comparison is confounded, the T5 baseline protocol is under-specified, and the RQ3 claim that the extracted reviews contain no predefined keywords is unverified and appears contradicted by the paper's own examples.

major comments (4)
  1. [Section IV.1; Table IV] The comparison between generic and domain-specific hypotheses is confounded with the choice of heuristics. The generic baseline uses the heuristics from Harkous et al., while the domain-specific condition uses the newly introduced rule NE(i,0.85)>=1 or NE(i,0.75)>=3 or NE(i,0.7)>=5 (Section IV.1). Because the F1 improvement from 0.56 to 0.65 could be driven entirely by the threshold rule rather than by the finance-specific hypotheses, the claimed advantage of domain-specific context awareness is not established. Please report both hypothesis sets under identical heuristics, and also report a cross-condition comparison (generic hypotheses with the new heuristics, and domain-specific hypotheses with the Harkous heuristics).
  2. [Section IV.2; Table V] The T5 baseline is reported as fine-tuned 'reproducing the one used in Harkous et al. [9]', but the paper provides no train/test split, no training-set description, no hyperparameters, and no evaluation protocol. If T5 was fine-tuned on the same 1,805 reviews used for evaluation, the zero-shot LLM comparison is invalid. Please specify the exact data partition used for fine-tuning and testing, and ensure that all models are evaluated on the same held-out set.
  3. [Section V RQ3; Section VI; Table VI] The paper's central claim that CMER extracted 2,178 PSRs 'that did not contain any predefined keywords' is not supported by any verification. No keyword list is given, no keyword-removal step is described in Section IV.3, and no containment check on the 2,178 reviews is reported. Moreover, the examples in Table VI contain terms such as 'hacked', 'password', 'security', 'bank account', and 'credit checks', which are likely to be in the keyword list of Ebrahimi et al. [1]; the paper itself notes that 'social security' was such a keyword. Please provide the exact keyword/regex list, verify that none of the 2,178 reviews contain any of those terms, and if the check fails, revise the 'additional/overlooked' claim accordingly.
  4. [Section IV.1, Algorithm 1 lines 8-27] The best NLI model is selected using the same 3,519 labeled reviews that are later used to report the F1 scores in Tables III and IV. Since there is no held-out validation set, the reported F1 values are selection-biased estimates, and the comparison between generic and domain-specific hypotheses is evaluated on the same data used for model selection. Please use a separate validation set (e.g., cross-validation) for model selection and for reporting final performance.
minor comments (5)
  1. [Section III; Table I] The paper reports 385,951 1-2 star reviews in Table I but uses 382,432 in RQ3; please state explicitly that the 382,432 unlabeled reviews are the 385,951 1-2 star reviews minus the 3,519 labeled reviews.
  2. [Section IV.3, Manual inspection setup] The annotators are the first author and three graduate students from the authors' lab; please provide their background and state whether they were blind to the study's hypotheses and to the keyword-based method, to reduce confirmation bias.
  3. [Section IV.1, footnote 3] The new heuristics were selected based on 'experiments performed on a different dataset' in another study, but that dataset is not described; readers cannot assess the risk of overfitting to that dataset. Please provide details or a reference when available.
  4. [Table VI] Dates in Table VI are inconsistent in format (e.g., 07/04/2013 appears alongside 14/12/2017); please use a single date format.
  5. [Section V RQ3] From the reported numbers, the LLM's precision on the unlabeled set is approximately 2,178/3,160 = 0.69; please report this figure and discuss the false positives rejected by manual inspection.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the RQ3 'keyword-free' claim is unsupported but not a definitional reduction.

full rationale

CMER's central derivation chain is not circular. RQ1 and RQ2 are evaluated against the externally labeled 3,519-review dataset from Ebrahimi et al. [1]; the NLI model and hypotheses are scored on those labels, and the LLM is evaluated in zero-shot mode against the same external labels, so no predicted quantity is defined as a fit and no parameter is renamed as a result. The domain-specific hypotheses are sourced from the Chen et al. taxonomy, and the RQ3 manual annotation also uses that taxonomy, which creates a shared construct; however, the annotators are human and the 2,178-output is not entailed by the hypotheses by construction, so this is a construct-validity consideration rather than a definitional reduction. Footnote 3 states that the threshold heuristics were selected on a different dataset in a broader study, so the reported F1 is not a resubstitution on the same evaluation set, though the broader study is not yet citable and that limits verification. The RQ3 claim that the 2,178 PSRs 'did not contain any predefined keywords' is not substantiated: no keyword list is given, no containment check is reported, and several Table VI examples contain terms like 'hacked' and 'security.' This is an unsupported empirical assertion, not a circular one, because 'additional' is asserted as a fact about the previous study's output rather than defined into the extraction procedure. No load-bearing self-citation or imported-uniqueness step appears in the paper. Score 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central evaluation rests on hand-crafted hypotheses, tuned thresholds, and assumed label quality. No new physical or formal entities are introduced.

free parameters (3)
  • NLI threshold heuristics = NE(i,0.85) >= 1 OR NE(i,0.75) >= 3 OR NE(i,0.7) >= 5
    The thresholds for labeling a review 'maybe-psr' were selected based on experiments on a different dataset and the best-performing set chosen (footnote 3). This is a hand-tuned decision rule, not derived from theory.
  • 17 domain-specific hypotheses = Table II
    The hypotheses are manually written based on Chen et al.'s taxonomy; their exact wording directly determines NLI entailment scores and is an unverified modeling choice.
  • LLM prompt template and temperature = Role-based prompt (Fig. 2), temperature=0
    The prompt design is manually engineered following [30]; output labels depend on prompt wording, which is not systematically varied in the paper.
assumptions (4)
  • domain assumption The labeled dataset from Ebrahimi et al. provides accurate ground-truth PSR/non-PSR labels.
    All precision/recall/F1 computations for RQ1 and RQ2 rely on these labels; the paper does not re-audit label quality.
  • domain assumption The taxonomy of Chen et al. fully covers privacy/security risks relevant to investment app reviews.
    The 17 hypotheses are derived from this taxonomy; if the taxonomy omits relevant risks, the pipeline will miss corresponding reviews.
  • domain assumption Annotators' manual judgments are a valid gold standard for PSR classification (kappa=0.78).
    The 2,178-item dataset is defined by these judgments; annotator agreement is substantial but not perfect.
  • ad hoc to paper T5 baseline was trained and evaluated under a fair protocol, despite no train/test split being reported.
    The RQ2 comparison assumes the fine-tuned T5's metrics are not inflated by evaluating on its training data; no split details are given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CMER: A Context-Aware Approach for Mining Ethical Concern-related App Reviews." pith.science (2026). https://pith.science/paper/OVF7QLA5

@misc{pith2026250709049,
  author       = {Pith},
  title        = {Pith review of: CMER: A Context-Aware Approach for Mining Ethical Concern-related App Reviews},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OVF7QLA5}},
  note         = {Machine review of arXiv:2507.09049}
}
read the original abstract

With the increasing proliferation of mobile applications in our daily lives, the concerns surrounding ethics have surged significantly. Users communicate their feedback in app reviews, frequently emphasizing ethical concerns, such as privacy and security. Incorporating these reviews has proved to be useful for many areas of software engineering (e.g., requirement engineering, testing, etc.). However, app reviews related to ethical concerns generally use domain-specific language and are typically overshadowed by more generic categories of user feedback, such as app reliability and usability. Thus, making automated extraction a challenging and time-consuming effort. This study proposes CMER (A \underline{C}ontext-Aware Approach for \underline{M}ining \underline{E}thical Concern-related App \underline{R}eviews), a novel approach that combines Natural Language Inference (NLI) and a decoder-only (LLaMA-like) Large Language Model (LLM) to extract ethical concern-related app reviews at scale. In CMER, NLI provides domain-specific context awareness by using domain-specific hypotheses, and the Llama-like LLM eliminates the need for labeled data in the classification task. We evaluated the validity of CMER by mining privacy and security-related reviews (PSRs) from the dataset of more than 382K app reviews of mobile investment apps. First, we evaluated four NLI models and compared the results of domain-specific hypotheses with generic hypotheses. Next, we evaluated three LLMs for the classification task. Finally, we combined the best NLI and LLM models (CMER) and extracted 2,178 additional PSRs overlooked by the previous study using a keyword-based approach, thus demonstrating the effectiveness of CMER. These reviews can be further refined into actionable requirement artifacts.

Figures

Figures reproduced from arXiv: 2507.09049 by the authors.

Figure 1
Figure 1. Overview of our methodology for LLM inference (RQ2) and extracting concern-related reviews using CMER (RQ3). [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Role-based prompt designed following the guidelines from [30]. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

55 extracted references · 13 canonical work pages

  1. [9]

    Hark: A deep learning system for navigating privacy feedback at scale,

    H. Harkous, S. T. Peddinti, R. Khandelwal, A. Srivastava, and N. Taft, “Hark: A deep learning system for navigating privacy feedback at scale,” in 2022 IEEE Symposium on Security and Privacy (SP) . IEEE, 2022, pp. 2469–2486

    2022

  2. [1]

    Unsupervised summarization of privacy concerns in mobile application reviews,

    F. Ebrahimi and A. Mahmoud, “Unsupervised summarization of privacy concerns in mobile application reviews,” in Proc. of the 37th IEEE/ACM International Conference on Automated Software Engineering, 2022, pp. 1–12

    2022

  3. [2]

    Development of a consumer financial goals ontology for use with fintech applications for improving financial capability,

    L. Bunnell et al., “Development of a consumer financial goals ontology for use with fintech applications for improving financial capability,” Expert systems with applications , vol. 165, p. 113843, 2021

    work page 2021

  4. [3]

    An empirical assessment of security risks of global android banking apps,

    S. Chen et al. , “An empirical assessment of security risks of global android banking apps,” in Proceedings of the ACM/IEEE 42nd Interna- tional Conference on Software Engineering , 2020, pp. 1310–1322

    work page 2020

  5. [4]

    Interpretable app review classification with transformers,

    M. Biswas, P. R. Anish et al. , “Interpretable app review classification with transformers,” in 2024 IEEE 32nd International Requirements Engineering Conference Workshops (REW) . IEEE, 2024, pp. 26–34

    work page 2024

  6. [5]

    Investigating user perceptions of mobile app privacy: An analysis of user-submitted app reviews,

    A. R. Besmer et al. , “Investigating user perceptions of mobile app privacy: An analysis of user-submitted app reviews,” International Journal of Information Security and Privacy (IJISP) , vol. 14, no. 4, pp. 74–91, 2020

    work page 2020

  7. [6]

    Analyzing user perspectives on mobile app privacy at scale,

    P. Nema, P. Anthonysamy, N. Taft, and S. T. Peddinti, “Analyzing user perspectives on mobile app privacy at scale,” in Proc. of the 44th International Conference on Software Engineering , 2022, pp. 112–124

    2022

  8. [7]

    Fairness concerns in app reviews: A study on ai-based mobile apps,

    A. Rezaei Nasab et al., “Fairness concerns in app reviews: A study on ai-based mobile apps,” ACM Transactions on Software Engineering and Methodology, vol. 34, no. 2, pp. 1–30, 2025

    work page 2025

Show all 55 references
  1. [8]

    Accessibility feedback in mobile application reviews: A dataset of reviews and accessibility guidelines,

    J. E. Reyes Arias et al. , “Accessibility feedback in mobile application reviews: A dataset of reviews and accessibility guidelines,” in CHI Con- ference on Human Factors in Computing Systems Extended Abstracts , 2022, pp. 1–7

    2022

  2. [10]

    Lessons from the use of natural language inference (nli) in requirements engineering tasks,

    M. Fazelnia, V . Koscinski, S. Herzog, and M. Mirakhorli, “Lessons from the use of natural language inference (nli) in requirements engineering tasks,” in 2024 IEEE 32nd International Requirements Engineering Conference (RE). IEEE, 2024, pp. 103–115

    2024

  3. [11]

    An extended model of natural logic,

    B. MacCartney et al. , “An extended model of natural logic,” in Proc. of the eight international conference on computational semantics , 2009, pp. 140–156

    2009

  4. [12]

    Transformers: State-of-the-art natural language process- ing,

    T. Wolf et al., “Transformers: State-of-the-art natural language process- ing,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations . Online: Association for Computational Linguistics, Oct. 2020, pp. 38–45

    2020

  5. [13]

    A broad-coverage challenge corpus for sentence un- derstanding through inference,

    A. Williams et al., “A broad-coverage challenge corpus for sentence un- derstanding through inference,” in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) . A...

    2018

  6. [14]

    Adversarial NLI: A new benchmark for natural language understanding,

    Y . Nie et al., “Adversarial NLI: A new benchmark for natural language understanding,” in Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics . Association for Computational Linguistics, Jul. 2020, pp. 4885–4901

    2020

  7. [15]

    A large annotated corpus for learning natural language inference,

    S. R. Bowman et al. , “A large annotated corpus for learning natural language inference,” in Proceedings of the 2015 Conference on Em- pirical Methods in Natural Language Processing . Association for Computational Linguistics, Sep. 2015, pp. 632–642

    2015

  8. [16]

    SQuAD: 100,000+ questions for machine compre- hension of text,

    P. Rajpurkar et al., “SQuAD: 100,000+ questions for machine compre- hension of text,” in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing . Association for Computa- tional Linguistics, Nov. 2016, pp. 2383–2392

    2016

  9. [17]

    The fever2. 0 shared task,

    J. Thorne et al. , “The fever2. 0 shared task,” in Proc. of the second workshop on Fact Extraction and VERification , 2019, pp. 1–6

    2019

  10. [18]

    Roberta: A robustly optimized bert pretraining approach,

    Y . Liu et al., “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692 , 2019

    1907 arXiv

  11. [19]

    Sentence-bert: Sentence embeddings using siamese bert-networks,

    N. Reimers et al., “Sentence-bert: Sentence embeddings using siamese bert-networks,” in Proc. of the 2019 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, 11 2019

    2019

  12. [20]

    Deberta: Decoding-enhanced bert with disentangled attention,

    P. He et al. , “Deberta: Decoding-enhanced bert with disentangled attention,” in 2021 International Conference on Learning Representations , May 2021, under review. [On- line]. Available: https://www.microsoft.com/en-us/research/publication/ deberta-decoding-enhanced-bert-with...

    2021

  13. [21]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    C. Raffel et al. , “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of Machine Learning Research, vol. 21, no. 140, pp. 1–67, 2020. [Online]. Available: http://jmlr.org/papers/v21/20-074.html

    2020

  14. [22]

    A taxonomy of privacy,

    D. J. Solove, “A taxonomy of privacy,” U. Pa. l. Rev., vol. 154, p. 477, 2005

    2005

  15. [23]

    Privacy-enhancing technologies,

    Y . Wang, “Privacy-enhancing technologies,” in Handbook of research on social and organizational liabilities in information security . IGI Global, 2009, pp. 203–227

    2009

  16. [24]

    A review of current trends, techniques, and challenges in large language models (llms),

    R. Patil and V . Gudivada, “A review of current trends, techniques, and challenges in large language models (llms),” Applied Sciences, vol. 14, no. 5, p. 2074, 2024

    2024

  17. [25]

    The llama 3 herd of models,

    A. Dubey et al. , “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  18. [26]

    A comprehensive capability analysis of gpt-3 and gpt-3.5 series models,

    J. Ye et al., “A comprehensive capability analysis of gpt-3 and gpt-3.5 series models,” arXiv preprint arXiv:2303.10420 , 2023

    2023 arXiv

  19. [27]

    An empirical study of the non-determinism of chatgpt in code generation,

    S. Ouyang et al., “An empirical study of the non-determinism of chatgpt in code generation,” ACM Trans. Softw. Eng. Methodol. , vol. 34, no. 2, Jan. 2025

    2025

  20. [28]

    Gpt-4o mini: Advancing cost-efficient intelli- gence,

    OpenAI, “Gpt-4o mini: Advancing cost-efficient intelli- gence,” Jul 2024. [Online]. Available: https://openai.com/index/ gpt-4o-mini-advancing-cost-efficient-intelligence

    2024

  21. [29]

    Training language models to follow instructions with human feedback,

    L. Ouyang et al., “Training language models to follow instructions with human feedback,” Advances in neural information processing systems , vol. 35, pp. 27 730–27 744, 2022

    2022

  22. [30]

    Unleashing the potential of prompt engineering for large language models,

    B. Chen et al., “Unleashing the potential of prompt engineering for large language models,” Patterns, 2025

    2025

  23. [31]

    Llama: Open and efficient foundation language models,

    H. Touvron et al. , “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971 , 2023

    2023 arXiv

  24. [32]

    Revisiting sentiment analysis for software engineering in the era of large language models,

    T. Zhang et al., “Revisiting sentiment analysis for software engineering in the era of large language models,” ACM Trans. Softw. Eng. Methodol., vol. 34, no. 3, Feb. 2025

    2025

  25. [33]

    Short text, large effect: Measuring the impact of user reviews on android app security & privacy,

    D. C. Nguyen, E. Derr, M. Backes, and S. Bugiel, “Short text, large effect: Measuring the impact of user reviews on android app security & privacy,” in 2019 IEEE symposium on Security and Privacy (SP) . IEEE, 2019, pp. 555–569

    2019

  26. [34]

    Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit

    J. Cohen, “Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit.” Psychological bulletin , vol. 70, no. 4, p. 213, 1968

    1968

  27. [35]

    Understanding interobserver agreement: the kappa statistic,

    A. J. Viera et al. , “Understanding interobserver agreement: the kappa statistic,” Fam med, vol. 37, no. 5, pp. 360–363, 2005

    2005

  28. [36]

    Open llms are necessary for current private adaptations and outperform their closed alternatives,

    V . Hanke, T. Blanchard, F. Boenisch, I. Olatunji, M. Backes, and A. Dziedzic, “Open llms are necessary for current private adaptations and outperform their closed alternatives,” Advances in Neural Informa- tion Processing Systems , vol. 37, pp. 1220–1250, 2024

    2024

  29. [37]

    An empirical study on user reviews targeting mobile apps’ security & privacy,

    D. Mukherjee, A. Ahmadi, M. V . Pour, and J. Reardon, “An empirical study on user reviews targeting mobile apps’ security & privacy,” arXiv preprint arXiv:2010.06371, 2020

    2010 arXiv

  30. [38]

    Exploring the capability of chatgpt to reproduce human labels for social computing tasks,

    Y . Zhu et al., “Exploring the capability of chatgpt to reproduce human labels for social computing tasks,” in International Conference on Advances in Social Networks Analysis and Mining . Springer, 2024, pp. 13–22

    2024

  31. [39]

    Want to reduce labeling cost? GPT-3 can help,

    S. Wang et al. , “Want to reduce labeling cost? GPT-3 can help,” in Findings of the Association for Computational Linguistics: EMNLP

  32. [40]

    The parrot dilemma: Human-labeled vs. LLM- augmented data in classification tasks,

    A. G. Møller et al. , “The parrot dilemma: Human-labeled vs. LLM- augmented data in classification tasks,” in Proceedings of the 18th Con- ference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers) . Association for Computational ...

    2024

  33. [41]

    Narratives: the unforeseen influencer of privacy concerns,

    Z. S. Li et al. , “Narratives: the unforeseen influencer of privacy concerns,” in 2022 IEEE 30th International Requirements Engineering Conference (RE). IEEE, 2022, pp. 127–139

    2022

  34. [42]

    Evaluating semantic accuracy of data-to-text generation with natural language inference,

    O. Du ˇsek et al., “Evaluating semantic accuracy of data-to-text generation with natural language inference,” in Proceedings of the 13th Interna- tional Conference on Natural Language Generation . Association for Computational Linguistics, Dec. 2020, pp. 131–137

    2020

  35. [43]

    Finding the needle in a haystack: On the automatic identification of accessibility user reviews,

    E. A. AlOmar et al., “Finding the needle in a haystack: On the automatic identification of accessibility user reviews,” in Proc. of the 2021 CHI conference on human factors in computing systems , 2021, pp. 1–15

    2021

  36. [44]

    Gpt-4 technical report,

    J. Achiam et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  37. [45]

    Scaling laws for neural language models,

    J. Kaplan et al. , “Scaling laws for neural language models,” arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  38. [46]

    Deep reinforcement learning from human pref- erences,

    P. F. Christiano et al., “Deep reinforcement learning from human pref- erences,” Advances in neural information processing systems , vol. 30, 2017

    2017

  39. [47]

    Large language models for software engineering: A systematic literature review,

    X. Hou et al. , “Large language models for software engineering: A systematic literature review,”ACM Trans. Softw. Eng. Methodol., vol. 33, no. 8, Dec. 2024

    2024

  40. [48]

    Advancing requirements engineering through generative ai: Assessing the role of llms,

    C. Arora, J. Grundy, and M. Abdelrazek, “Advancing requirements engineering through generative ai: Assessing the role of llms,” in Generative AI for Effective Software Development . Springer, 2024, pp. 129–148

    2024

  41. [49]

    Exploring the efficacy of chatgpt in generating requirements: An experimental study,

    L. Bencheikh and N. H ¨oglund, “Exploring the efficacy of chatgpt in generating requirements: An experimental study,” 2023

    2023

  42. [50]

    Investigating chatgpt’s potential to assist in requirements elicitation processes,

    K. Ronanki, C. Berger, and J. Horkoff, “Investigating chatgpt’s potential to assist in requirements elicitation processes,” in 2023 49th Euromi- cro Conference on Software Engineering and Advanced Applications (SEAA). IEEE, 2023, pp. 354–361

    2023

  43. [51]

    Chatgpt prompt patterns for improving code quality, refactoring, requirements elicitation, and software design,

    J. White, S. Hays, Q. Fu, J. Spencer-Smith, and D. C. Schmidt, “Chatgpt prompt patterns for improving code quality, refactoring, requirements elicitation, and software design,” in Generative ai for effective software development. Springer, 2024, pp. 71–108

    2024

  44. [52]

    Autonomous agents in software development: A vi- sion paper,

    Z. Rasheed et al., “Autonomous agents in software development: A vi- sion paper,” in International Conference on Agile Software Development. Springer Nature Switzerland Cham, 2024, pp. 15–23

    2024

  45. [53]

    Chatgpt: A study on its utility for ubiquitous software engineering tasks,

    G. Sridhara et al., “Chatgpt: A study on its utility for ubiquitous software engineering tasks,” arXiv preprint arXiv:2305.16837 , 2023

    2023 arXiv

  46. [54]

    Using chatgpt in software requirements engineering: A comprehensive review,

    N. Marques, R. R. Silva, and J. Bernardino, “Using chatgpt in software requirements engineering: A comprehensive review,” vol. 16, no. 6, p. 180, 2024

    2024

  47. [2021]

    2021, pp

    Association for Computational Linguistics, Nov. 2021, pp. 4195– 4205

    2021

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.