Pith. sign in

REVIEW 3 major objections 4 minor 27 references

Decoding User Concerns in AI Health Chatbots: An Exploration of Security and Privacy in App Reviews

T0 review · 3 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Gemini GenAI can automatically flag security and privacy concerns in health chatbot reviews with accuracy close to human labeling.

desk verdict A small exploratory study whose qualitative findings are fine, but whose central accuracy claim for Gemini collapses against the majority-class baseline. read the letter →

arxiv 2502.00067 v1 pith:CXCYUBRK submitted 2025-01-31 cs.CR cs.ETcs.HC

Muhammad Hassan , Abdullah Ghani , Muhammad Fareed Zaffar , Masooda Bashir This is my paper · ORCID
classification cs.CRcs.ETcs.HC
keywords AIhealthchatbotssecurityandprivacyappreviewanalysisGeminiGenBARTmanuallabelinguserconcernstransparency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether an automated system can find security- and privacy-related complaints in user reviews of AI health chatbots as reliably as a human analyst. The authors compare three labeling routes on reviews of eight Android health chatbot apps: manual coding of 1,100 reviews, a BART transformer model run in zero-shot mode, and Gemini GenAI prompted to assign concern labels. They report that Gemini matches manual labeling at 89% accuracy on the three-class task, while BART reaches only 28% accuracy yet identifies all seven manually flagged SPR reviews. They call the work exploratory and use the SPR reviews to extract three user concern themes: intrusive data collection, data misuse or sharing, and weak transparency and consent. If the accuracy claim holds, app-store review mining with a generative model could become a scalable early-warning tool for privacy and security failures in health chatbots.

What carries the argument

The object that carries the comparison is a three-label concern schema — 'Privacy-Security Concern,' 'Other Concern,' 'No Concern' — applied to 5,469 reviews collected from Google Play for eight free AI health chatbot apps. The automated contenders are BART-large-MNLI run through a zero-shot classification pipeline and Gemini GenAI 1.5 Flash prompted to return structured sentiment and concern labels with confidence scores; the reference standard is 1,100 reviews manually labeled by two reviewers, with Cohen's Kappa 0.66 for SPR labels. The evaluation works by taking each model's highest-confidence verdict and checking it against the manual gold set using accuracy, precision, recall, and macro-F1.

What would settle it

Construct a second gold set with hundreds of verified security-privacy reviews and run Gemini and BART on it; if Gemini's accuracy drops well below 89% or BART no longer recalls all positives, the paper's comparability claim would be contradicted.

Watch

Extended reading notes

Core claim

The central claim is that Gemini GenAI's SPR classification is comparable to manual labeling: on a three-class task it reaches 89% accuracy, while BART reaches 28% accuracy and performs worse than random chance for that task. At the same time, BART correctly labels all seven SPR reviews in the 1,100-review gold set, but with very low precision, while Gemini detects only four of seven. The authors read these results as showing that a generative model can flag security and privacy concerns at scale, but that neither automated method is reliable enough to replace human judgment without refinement such as better prompts or hybrid human-AI review. Qualitatively, the paper identifies three SPR concern themes in user reviews: data collection and surveillance, data misuse and third-party sharing, and inadequate transparency and consent mechanisms.

Load-bearing premise

The manually labeled gold set — 1,100 reviews containing only 7 SPR positives, with Cohen's Kappa 0.66 for SPR labels — is reliable and representative enough to judge the automated methods.

Editorial extensions

If this is right

  • Review mining with Gemini GenAI could scale privacy and security auditing of health chatbots beyond what manual coding allows.
  • BART is not usable as a standalone classifier for this task, but its perfect recall of the labeled SPR reviews makes it a candidate first-pass filter in a two-stage pipeline.
  • Because SPR reviews appear even with positive sentiment, any automated screening should ignore star ratings and scan all reviews.
  • The three qualitative themes — data collection and surveillance, data misuse and sharing, and transparency and consent — give chatbot developers a concrete checklist for user-facing privacy practices.
  • A hybrid human-AI workflow, in which Gemini flags candidate reviews and humans verify them, could combine scalability with the contextual accuracy of manual analysis.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported 89% accuracy rests on a gold set with only seven SPR positives, so a larger balanced gold set is the immediate next test; if Gemini's precision drops there, the comparability claim weakens.
  • The authors do not test whether adding an explicit definition of security-privacy concern to the Gemini prompt trades recall for precision, and that tradeoff is worth measuring.
  • App-store monitoring for privacy complaints should scan all reviews rather than only negative ones, since at least one SPR review in this dataset carried positive sentiment.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper collects 5,469 user reviews of eight AI health chatbot apps from the Google Play Store, manually labels a random subset of 1,100 reviews for sentiment and security-privacy-related (SPR) concern, and evaluates two automated zero-shot classifiers, BART-large-MNLI and Gemini GenAI 1.5 Flash, against this gold set. It reports that Gemini achieves 96% sentiment accuracy and 89% concern accuracy, while BART achieves 81% sentiment accuracy and 28% concern accuracy, and then conducts a thematic analysis of the seven manually identified SPR reviews, highlighting themes of data collection, data misuse, and transparency/consent deficiencies. The paper concludes that Gemini's SPR classification is 'comparable to manual labeling' and that automated methods can support scalable analysis of user privacy and security concerns.

Significance. If the central quantitative claim were substantiated, the paper would offer a practical result: a generative AI model that can flag security and privacy concerns in health chatbot app reviews at scale. The qualitative themes are plausible and reflect real user concerns, and the comparison of two automated methods against human annotation is a reasonable exploratory design. The authors are transparent about the exploratory nature of the study and the small dataset. However, the headline result is currently fragile because it rests on a 7-positive gold set, an accuracy metric dominated by the majority class, and the absence of SPR-class precision, recall, and confidence intervals for Gemini. As a pilot study, the paper has value, but the RQ1 conclusion needs to be re-analyzed and substantially tempered before it can be accepted.

major comments (3)
  1. [II.B, IV, V, Abstract] The claim that Gemini's SPR classification is 'comparable to manual labeling' is not supported by the reported metrics. The gold set contains 949 'No Concern', 144 'Other Concern', and 7 'Privacy-Security Concern' reviews (Section II.A.2), so a trivial always-'No Concern' classifier already achieves 86.3% accuracy. Gemini's reported 89% accuracy (Section II.B) is only 2.7 points above this baseline, and Gemini detects only 4 of the 7 SPR reviews, i.e., 57% recall. The paper reports no SPR-class precision, recall, or F1 for Gemini, nor any confidence intervals. The accuracy number mostly reflects the majority class, not demonstrated skill at the rare SPR category, and the abstract and conclusion state the comparability claim more strongly than the data support.
  2. [II.A.2 and IV.B] The ground truth used for the quantitative comparison is too fragile to anchor the RQ1 conclusion. Cohen's Kappa of 0.66 for SPR labels is computed on a gold set with only 7 positives; a shift of one or two differently resolved disagreements could change the gold labels and materially alter the evaluation. With 7 positives, the 4/7 recall estimate carries a very wide confidence interval. Section IV.B acknowledges the small dataset and potential subjective bias, but Section V still draws a firm comparative conclusion. Please report per-class confusion matrices, class-specific metrics with uncertainty bounds, and a majority-class baseline, and reframe the RQ1 conclusion as a preliminary agreement analysis.
  3. [Abstract, II.B, IV] The phrase 'comparable to manual labeling' is conceptually confusing because the manual labels are the ground truth against which Gemini's accuracy is computed. Accuracy relative to the gold set measures agreement with a particular human-annotated set, not equivalence to human labeling in general. The paper should either compare Gemini to a separate human labeling process (for example, human-human agreement versus human-model agreement), or reformulate the claim as 'Gemini agrees with the manually produced gold labels on 89% of reviews.' This distinction matters because the majority-class baseline is 86.3%, so the observed agreement is not evidence of human-level performance.
minor comments (4)
  1. [II.B] The text refers to 'Figure 4' and 'Figure 5' for sentiment and concern classification, but the figure captions are labelled 'Fig. 1' through 'Fig. 5' in an inconsistent order; please renumber the figures and correct the cross-references.
  2. [II.B] The sentence that BART 'performs worse than random chance for a three-class classification task' is ambiguous: random chance is 33% only for uniform random labeling, whereas the relevant baseline for this imbalanced dataset is the 86.3% majority-class classifier. Please define the baseline explicitly.
  3. [III.A] The review quotes are not always clearly tied to the themes: several quote blocks appear empty or misformatted (for example, after 'one user remarked:' the text breaks directly to the next quote), making it difficult to verify the qualitative coding.
  4. [Appendix A] The prompt in the appendix is presented as a raw, malformed JSON fragment; please include a clean, complete version of the prompt so that the Gemini experiment is reproducible.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the study is an empirical benchmark of automated SPR classifiers against an independently constructed manual ground-truth set, with no fitted parameters, self-citation chain, or definitional reduction.

full rationale

The paper's derivation chain is an empirical evaluation, not a mathematical derivation. Manual labeling (1,100 reviews, Sec. II.A.2) is constructed independently of the automated methods and is then used as the ground truth against which BART and Gemini are benchmarked (Sec. II.B). No model parameter is fitted to the ground-truth labels, and the reported accuracies, precisions, recalls, and F1 scores are computed from held-out comparisons, so no 'prediction' reduces to a fitted input by construction. The phrase 'Gemini's performance in SPR classification is comparable to manual labeling' is a comparison claim, not a self-definition: the manual labels were produced before and without reference to Gemini's outputs, so the target claim is not built into the measurement. The paper contains no load-bearing self-citations: the cited prior work (e.g., [24] for Gemini, [27] for BART) is external tooling or background, and no uniqueness theorem or prior author-derived ansatz is invoked to force the results. The strongest legitimate concerns about this paper are about statistical reliability and generalizability: the gold set has only 7 SPR positives, the 89% Gemini accuracy is only 2.7 points above an always-'No Concern' majority baseline, and SPR-class recall is 4/7 with no reported precision or F1. Those are validity and evidentiary-weight issues, not circularity. Because no step in the paper's argument is equivalent to its own input by definition or by self-citation, the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The study introduces no new theoretical entities or fitted parameters. Its empirical claims rest entirely on the manual gold standard, whose reliability for such a rare class is the main unvalidated assumption.

assumptions (3)
  • domain assumption Manual labels of 1,100 reviews provide a valid ground truth for SPR classification.
    Paper assumes the two-reviewer labels (with Kappa 0.66 for SPR) are accurate enough to evaluate automated methods, despite only 7 positive SPR reviews.
  • domain assumption The sampled 1,100 reviews are representative of the full 5,469-review dataset.
    The paper samples 1,100 reviews randomly and uses them as the evaluation set, implicitly assuming the SPR prevalence and content are representative.
  • domain assumption App reviews reflect genuine user security and privacy concerns.
    The qualitative analysis interprets user statements as concerns, while acknowledging in Section IV that some concerns may stem from misunderstandings.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Decoding User Concerns in AI Health Chatbots: An Exploration of Security and Privacy in App Reviews." pith.science (2026). https://pith.science/paper/CXCYUBRK

@misc{pith2026250200067,
  author       = {Pith},
  title        = {Pith review of: Decoding User Concerns in AI Health Chatbots: An Exploration of Security and Privacy in App Reviews},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CXCYUBRK}},
  note         = {Machine review of arXiv:2502.00067}
}
read the original abstract

AI powered health chatbot applications are increasingly utilized for personalized healthcare services, yet they pose significant challenges related to user data security and privacy. This study evaluates the effectiveness of automated methods, specifically BART and Gemini GenAI, in identifying security privacy related (SPR) concerns within these applications' user reviews, benchmarking their performance against manual qualitative analysis. Our results indicate that while Gemini's performance in SPR classification is comparable to manual labeling, both automated methods have limitations, including the misclassification of unrelated issues. Qualitative analysis revealed critical user concerns, such as data collection practices, data misuse, and insufficient transparency and consent mechanisms. This research enhances the understanding of the relationship between user trust, privacy, and emerging mobile AI health chatbot technologies, offering actionable insights for improving security and privacy practices in AI driven health chatbots. Although exploratory, our findings highlight the necessity for rigorous audits and transparent communication strategies, providing valuable guidance for app developers and vendors in addressing user security and privacy concerns.

Figures

Figures reproduced from arXiv: 2502.00067 by the authors.

Figure 1
Figure 1. Sentiment Classification [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Concern Classification [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 4
Figure 4. Sentiment Distribution 10 11 12 13 14 15 16 17 "concern_labels_confidence_tuple": [["label", "concern_explanation": "string" } You are an assistant specialized in analyzing app **Sentiment Analysis**: Classify the sentiment using one or more of the '→ following labels: 'Positive', 'Negative', Provide a confidence score (0-100) for each Include a brief explanation supporting your **Concern Identification**: Identify … view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Concern Distribution [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 24 canonical work pages

  1. [2]

    Health chatbots for fighting covid-19: a scoping review

    Manal Almalki and Fahad Azeez. Health chatbots for fighting covid-19: a scoping review. Acta Informatica Medica, 28(4):241, 2020

    work page 2020

  2. [3]

    6 privacy principles of the gdpr

    Robert Bateman. 6 privacy principles of the gdpr

    work page

  3. [4]

    Using thematic analysis in psychology

    Virginia Braun and Victoria Clarke. Using thematic analysis in psychology. Qualitative research in psychology, 3(2):77–101, 2006

    2006

  4. [5]

    A machine learning approach with human-ai collaboration for automated classification of patient safety event reports: Algo- rithm development and validation study

    Hongbo Chen, Eldan Cohen, Dulaney Wilson, Myrtede Alfred, et al. A machine learning approach with human-ai collaboration for automated classification of patient safety event reports: Algo- rithm development and validation study. JMIR Human Factors, 11(1):e53378, 2024

    work page 2024

  5. [6]

    Exploring the far side of mobile health: Information security and privacy of mobile health apps on ios and android

    Tobias Dehling, Fangjian Gao, Stephan Schneider, and Ali Sun- yaev. Exploring the far side of mobile health: Information security and privacy of mobile health apps on ios and android. JMIR mHealth and uHealth, 3(1):e8, Jan 2015

    work page 2015

  6. [7]

    Artificial Intelligence for Chatbots in Mental Health: Opportunities and Challenges, pages 115–128

    Kerstin Denecke and Mowafa Househ. Artificial Intelligence for Chatbots in Mental Health: Opportunities and Challenges, pages 115–128. 08 2021

    work page 2021

  7. [8]

    Analysing app reviews for software engineering: a system- atic literature review

    Jacek Dkabrowski, Emmanuel Letier, Anna Perini, and Angelo Susi. Analysing app reviews for software engineering: a system- atic literature review. Empirical Software Engineering, 27(2):43, 2022

    work page 2022

  8. [9]

    Centers for Disease Control and Prevention, Sep 2024

    work page 2024

Show all 27 references
  1. [10]

    Medical mayhem: How poor data labeling is sabotaging ai in healthcar

    Amina Khalpey, Kirtana Roopan, Jessa Deckwa, and Zain Khalpey. Medical mayhem: How poor data labeling is sabotaging ai in healthcar. Khalpey AI Lab

  2. [11]

    Roles, users, bene- fits and limitations of chatbots in healthcare: A rapid review (preprint)

    Moustafa Laymouna, Yuanchao Ma, David Lessard, Tibor Schus- ter, Kim Engler, and Bertrand Lebouche´. Roles, users, bene- fits and limitations of chatbots in healthcare: A rapid review (preprint). JMIR. Journal of medical internet research/Journal of medical internet research, ...

    2024

  3. [12]

    What’s up with privacy? user preferences and privacy concerns in intelligent personal assistants

    Lydia Manikonda, Aditya Deotale, and Subbarao Kambhampati. What’s up with privacy? user preferences and privacy concerns in intelligent personal assistants. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’18, page 229–235, New York, NY, USA, 20...

    2018

  4. [13]

    Hipaa penalties: Understanding the consequences of non-compliance, Sep 2024

    Scott Mattila. Hipaa penalties: Understanding the consequences of non-compliance, Sep 2024

    2024

  5. [14]

    Patient perspectives on the use of artificial intelligence in health care: A scoping review

    Sally Moy, Mona Irannejad, Stephanie Jeanneret Manning, Mehrdad Farahani, Yomna Ahmed, Ellis Gao, Radhika Prabhune, Suzan Lorenz, Raza Mirza, and Christopher Klinger. Patient perspectives on the use of artificial intelligence in health care: A scoping review. Journal of Patien...

    2024

  6. [15]

    Defining” broken”: User experiences and remediation tactics when {Ad-Blocking} or {Tracking-Protection} tools break a {Website’s} user experience

    Alexandra Nisenoff, Arthur Borem, Madison Pickering, Grant Nakanishi, Maya Thumpasery, and Blase Ur. Defining” broken”: User experiences and remediation tactics when {Ad-Blocking} or {Tracking-Protection} tools break a {Website’s} user experience. In 32nd USENIX Security Sympo...

    2023

  7. [16]

    Security and privacy analysis of mobile health applications: Th1e9 alarming state of practice

    Achilleas Papageorgiou, Michael Strigkos, Eugenia Politou18, Efthimios Alepis, Agusti Solanas, and Constantinos Patsakis. Security and privacy analysis of mobile health applications: Th1e9 alarming state of practice. IEEE Access, 6:9390–9403, 2018. 20

    2018

  8. [17]

    Data privacy concerns in health and wellnes2s1 apps: Balancing innovation and security, 2022

    psico smart.com. Data privacy concerns in health and wellnes2s1 apps: Balancing innovation and security, 2022

    2022

  9. [18]

    From user insights to ac- tionable metrics: A user-focused evaluation of privacy-preserving browser extensions

    Ritik Roongta and Rachel Greenstadt. From user insights to ac- tionable metrics: A user-focused evaluation of privacy-preserving browser extensions. In Proceedings of the 19th ACM Asia Conference on Computer and Communications Security, pages 1753–1769, 2024

    2024

  10. [19]

    Investigating gdpr fines in the light of data flows

    Marlene Saemann, Daniel Theis, Tobias Urban, and Martin Degeling. Investigating gdpr fines in the light of data flows. Proceedings on Privacy Enhancing Technologies, 2022

    2022

  11. [20]

    google play scraper, 2024

    Google Play Scraper. google play scraper, 2024

    2024

  12. [21]

    Mobile os market share — statista, Oct 2023

    Ahmed Sherif. Mobile os market share — statista, Oct 2023

    2023

  13. [22]

    Understanding privacy and security postures of healthcare chatbots, 2022

    Aishwarya Surani and Sanchari Das. Understanding privacy and security postures of healthcare chatbots, 2022

    2022

  14. [23]

    Extracting useful software development information from mobile application reviews: A survey of intelligent mining techniques and tools

    Mohammadali Tavakoli, Liping Zhao, Atefeh Heydari, and Goran Nenadic´. Extracting useful software development information from mobile application reviews: A survey of intelligent mining techniques and tools. Expert Systems with Applications, 113:186– 199, 2018

    2018

  15. [24]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  16. [25]

    Tzelios, C

    C. Tzelios, C. Contreras, B. Istenes, A. Astupillo, L. Lecca, K. Ramos, L. Ramos, K. Roca, J. T. Galea, M. Tovar, C. D. Mitnick, and J. Peinado. Using digital chatbots to close gaps in healthcare access during the covid-19 pandemic. Public Health Action, 12(4):180–185, Dec 2022

    2022

  17. [26]

    What is gdpr, the eu’s new data protection law?, 2024

    Ben Wolford. What is gdpr, the eu’s new data protection law?, 2024

    2024

  18. [27]

    Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach

    Wenpeng Yin, Jamaal Hay, and Dan Roth. Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach. arXiv preprint arXiv:1909.00161, 2019

    1909 arXiv

  19. [28]

    concern_labels_confidence_tuple

    Kuan Zhang, Kan Yang, Xiaohui Liang, Zhou Su, Xuemin Shen, and Henry H Luo. Security and privacy for mobile healthcare networks: from a quality of protection perspective. IEEE Wireless Communications, 22(4):104–112, 2015. APPENDIX 1 2 3 4 5 6 7 8 9 Fig. 1: Sentiment Classifica...

    2015

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.