REVIEW 5 major objections 5 minor 1 cited by
Prioritizing App Reviews for Developer Responses on Google Play
T0 review · 5 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read XGBoost, a gradient-boosted tree model, is the most effective classifier for deciding which Google Play app reviews need a developer response and which need one within three days.
desk verdict The headline F1 for Approach #1 is unreliable because a feature (F34) correlates perfectly with the label, making the 'prioritization' result circular until that is resolved. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a supervised binary-classification setup with two labels: 'respond or not' and 'respond within three days or not.' Each review is represented by a feature vector of textual and semantic features—length, readability, complexity, part-of-speech counts, sentiment, neutrality, polarity, star rating, helpfulness votes, informativeness, purpose category, LDA topic, and counts of emotion and commitment words. Pearson correlation selects features and removes collinear ones. XGBoost, an implementation of gradient-boosted decision trees, is then trained on the selected features, and its superiority over Decision Tree, Random Forest, and SVM is the paper's principal empirical result.
What would settle it
Ask app developers to independently rate a held-out set of reviews for whether each needs a reply and whether it is urgent; if the model's labels match these human judgments no better than chance, the central claim would fail. A second check is to vary the three-day cutoff and see whether the F1-scores change drastically, which would show the urgency result is an artifact of the chosen threshold.
Extended reading notes
Core claim
The paper's central claim is that the question 'does this review need a developer response?' can be answered automatically with an XGBoost classifier at F1 = 0.77, and the follow-up question 'does it need a response within three days?' at F1 = 0.87. The labels are defined by developer behavior: a review is important if a developer actually responded, and urgent if the response came within the dataset's average turnaround of about 3.76 days. The most influential features include review score, length, readability, polarity, topic, and whether the review is a feature request or problem report. The authors report that the best model generalizes across ten app categories in a case study, with mean F1-scores of 0.77 and 0.87.
Load-bearing premise
The method assumes that a review truly 'needs a response' whenever a developer actually responded, and that 'urgent' means the developer replied within three days.
Editorial extensions
If this is right
- Developers can use the trained model to cut the number of reviews they read and to route urgent complaints to quicker replies.
- The feature correlations give app teams a shortlist of signals—star rating, review length, readability, sentiment, and review purpose—that predict whether users expect a reply.
- Because the best model keeps its F1-scores in held-out case studies across app categories, the same pipeline can be applied to new apps without per-app retraining.
- The same labeling scheme and feature set can be carried over to other app stores, a direction the paper explicitly lists for future work.
Reading between the lines
- The labels are inherited from developer behavior, so the classifier will systematically reproduce developers' existing blind spots: if a kind of review never gets answered, the model will learn to deprioritize it rather than discover that it matters.
- The three-day threshold is derived from the dataset's own average response time, so the 'urgent' definition will shift if the average response time in another store or app category differs; teams would need to relabel for their own norms.
- Because star rating and review length are among the strongest features, the model may be capturing a mix of genuine urgency and developer convenience; a team that wants to prioritize underserved users would need to inspect and possibly reweight these features.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a machine-learning pipeline to prioritize Google Play user reviews for developer responses. Reviews are labeled according to two criteria: whether a developer actually responded (Approach #1) and whether a developer responded within three days (Approach #2). The authors extract textual and semantic features, including LDA topic categories, and train Decision Tree, Random Forest, SVM, and XGBoost classifiers. They report that XGBoost achieves the best F1-scores of 0.77 (Approach #1) and 0.87 (Approach #2), claim the model can help developers reduce the volume of reviews they read, and provide a public repository. The paper also includes a case-study evaluation on 10 and 8 applications respectively, and a threats-to-validity section.
Significance. If the reported results are correct, the paper would offer a practical tool for app developers to triage user feedback, a task of genuine interest to the software engineering community. The study also contributes a feature analysis of textual and semantic review properties in relation to developer response behavior. However, the significance is conditional on the validity of the evaluation: the labels are defined by observed developer behavior rather than by any independent measure of review importance, and several load-bearing evaluation details are missing or internally inconsistent. The strengths are that the paper makes its dataset and code publicly available, uses a large corpus of reviews, and compares four standard classifiers; these are useful assets that can support follow-up work.
major comments (5)
- [Section 2.8.1, Table 3] The reported correlation of F34 ('Other Review Categories') with the Approach #1 output is 1.00, and 0.83 with the Approach #2 output. A feature that correlates perfectly with the label should make the classification task trivially perfect if used, yet XGBoost is reported to achieve only F1=0.77. The paper never explains how F34 is constructed or why it is excluded/included. This is an internal inconsistency at the core of the evaluation: either F34 leaks the response label (e.g., through LDA trained on response-inclusive text) and the model scores are uninterpretable, or the table is misreported. The authors must clarify the definition of F34, the feature selection procedure, and whether this feature was used in training.
- [Section 2.5 and Section 2.8.1] The labels for Approach #1 are defined by whether a developer actually responded, and for Approach #2 by response time. This makes the model a predictor of historical developer behavior rather than of review importance, as claimed in the abstract and introduction. This is a construct-validity concern: if developers miss important reviews or respond for reasons unrelated to content (e.g., popularity, randomness), the model learns a proxy that may diverge from the stated goal. The paper should either reframe the contribution as 'predicting developer response behavior' or provide a validation that the proxy is meaningful, for instance by comparing the model's top-ranked reviews against manually annotated importance labels.
- [Section 2.8.1 versus Section 2.2] Section 2.2 states that 431,512 reviews were collected, but Section 2.8.1 says 'a dataset of 123,130 labeled reviews has been analyzed' with no explanation of how, why, or from which subset this number arises. If the 123,130 subset is not response-stratified, the reported accuracy and F1 scores may not generalize to the full collected dataset. The authors must explain the data filtering process (e.g., language filtering, removal of updated reviews, or balanced sampling) and report the label distribution before and after filtering.
- [Section 2.8.2] The urgent/non-urgent cutoff of three days is chosen because the average response time is 3.76 days, making the cutoff data-dependent. This threshold selection can inflate classifier performance if the response-time distribution has a mass near the boundary; no sensitivity analysis is provided for alternative cutoffs (e.g., 1 day, 7 days). Since the central claim of Approach #2 depends on this cutoff, the authors should evaluate robustness across a range of thresholds and report the distribution of response times.
- [Section 3, Tables 4 and 5] The evaluation lacks baselines and variance estimates. The paper does not compare against a majority-class baseline, a simple heuristic (e.g., respond to all low-star reviews), or previously published prioritization/classification methods. Additionally, only Approach #2 is described as using 5-fold cross-validation; the train/test procedure for Approach #1 is not stated. Without standard deviations or confidence intervals across folds (or repeated runs), the reported superiority of XGBoost over SVM and other models cannot be assessed statistically. The authors should add a majority baseline, report per-fold results, and conduct significance testing (e.g., paired t-test or Wilcoxon test) for the F1 differences.
minor comments (5)
- [Abstract and Section 4] The abstract says 'the XGBoost model is the most effective...' without stating F1 values; Section 4 and the conclusion state F1=0.77 for Approach #1, but Table 4 reports XGBoost F1=0.78. Section 3.1 also says Accuracy=0.78 while Table 4 reports 0.77. These numbers should be reconciled.
- [Section 3.1.1] The phrase 'the mean of F1-Score for theses 10 applications' contains a typo ('theses' should be 'these'), and the case-study section does not report the variance or range of F1 across applications, which would be more informative than a single mean.
- [Section 2.4.2] The LDA description says 'found 10 to be optimal' but does not state the metric or procedure used to select the number of topics. Also, the table in Table 2 names features 14-18 as 'Purpose of the review' and 25-34 as 'Review inclinations', but the mapping between these named features and the F17-F34 abbreviations in Table 3 is not fully specified (e.g., which topic features are 'F30-F34'). A concise feature dictionary would improve reproducibility.
- [Section 2.6] The paper states that highly correlated features are removed, but it does not specify the correlation threshold used for feature-feature redundancy (only mentions correlations 'over 0.9' in two specific cases). This threshold should be stated explicitly for each approach.
- [Section 3.3, Threats to Validity] The threats-to-validity section is superficial: it asserts that using 'well-known and accurate tools' mitigates internal validity, and that correlation with the output demonstrates relevance, but it does not address the label-construct mismatch, the unexplained dataset reduction, or the lack of cross-validation in Approach #1. The authors should expand this section to engage with the concrete threats raised above.
Circularity Check
No significant circularity: the prediction target is developer response behavior, and features are derived independently from review text and metadata.
full rationale
The paper defines its labels directly from observable developer behavior: whether a response exists (Approach #1) and whether the response occurred within three days (Approach #2). The features used to predict these labels are textual and semantic properties of the reviews, such as readability, sentiment, length, and LDA topic categories, which are computed from the review text alone and do not incorporate the response label. No model parameter is fitted to the label and then renamed as a prediction; the classifiers are standard supervised models trained on held-out or cross-validated data. The suspicious F34 correlation value of 1 in Table 3 is inconsistent with the reported F1-score of 0.77, indicating a likely typo or misrepresentation rather than a deliberate circular use of the label as a feature. The paper's construct validity argument, which uses feature-label correlation as evidence of relevance, is methodologically weak but does not constitute a derivation-level circularity because the correlation is computed over the same training data and is used only for feature selection, not to construct the target. There are no load-bearing self-citations, no imported uniqueness theorems, and no ansatz smuggled in via citation. The central claim that XGBoost predicts developer response behavior is therefore based on a legitimate, if imperfect, supervised learning setup, and the paper's derivation chain is self-contained with respect to circularity.
Assumptions & free parameters
free parameters (4)
- Urgency threshold =
3 days
- LDA topic count =
10
- Feature correlation thresholds =
0.1 (output), 0.9 (redundancy)
- Model hyperparameters =
not reported
assumptions (4)
- ad hoc to paper Developer response behavior (responded vs not, responded within 3 days) is a valid proxy for review importance.
- ad hoc to paper The 3-day cutoff separates urgent from non-urgent reviews.
- domain assumption Merging crawled data with dataset [7] yields a compatible, representative corpus.
- standard math Pearson correlation coefficients are valid for assessing feature importance for a binary outcome.
Cite this review
Pith. "Pith review of Prioritizing App Reviews for Developer Responses on Google Play." pith.science (2026). https://pith.science/paper/MRZUHNDK
@misc{pith2026250201520,
author = {Pith},
title = {Pith review of: Prioritizing App Reviews for Developer Responses on Google Play},
year = {2026},
howpublished = {\url{https://pith.science/paper/MRZUHNDK}},
note = {Machine review of arXiv:2502.01520}
}
read the original abstract
The number of applications in Google Play has increased dramatically in recent years. On Google Play, users can write detailed reviews and rate apps, with these ratings significantly influencing app success and download numbers. Reviews often include notable information like feature requests, which are valuable for software maintenance. Users can update their reviews and ratings anytime. Studies indicate that apps with ratings below three stars are typically avoided by potential users. Since 2013, Google Play has allowed developers to respond to user reviews, helping resolve issues and potentially boosting overall ratings and download rates. However, responding to reviews is time-consuming, and only 13% to 18% of developers engage in this practice. To address this challenge, we propose a method to prioritize reviews based on response priority. We collected and preprocessed review data, extracted both textual and semantic features, and assessed their impact on the importance of responses. We labelled reviews as requiring a response or not and trained four different machine learning models to prioritize them. We evaluated the models performance using metrics such as F1-Score, Accuracy, Precision, and Recall. Our findings indicate that the XGBoost model is the most effective for prioritizing reviews needing a response.
Figures
Forward citations
Cited by 1 Pith paper
-
Identification and Study of Irregular Radio Sources with SKA Continuum Surveys
A solicited SKA science chapter reviewing how bent-tail and winged radio galaxies will be identified and studied with SKA continuum surveys; no new data or derivations are presented.
Reference graph
Works this paper leans on
-
[1]
Y . Abedini and A. Heydarnoori. Can GitHub issues help in app review classifications? ACM Transactions on Software Engineering and Methodology, July 2024
work page 2024
-
[2]
N. Ailon and M. Mohri. An efficient reduction of ranking to classification. arXiv preprint arXiv:0710.2889, 2007
arXiv 2007
-
[3]
G. Capobianco, A. D. Lucia, R. Oliveto, A. Panichella, and S. Panichella. Improving IR-based traceability recovery via noun-based indexing of software artifacts. Journal of Soft- ware: Evolution and Process, 25(7):743–762, 2013
work page 2013
-
[4]
S. E. Crawford and E. Ostrom. A grammar of institutions. American political science review, 89(3):582–600, 1995
work page 1995
-
[5]
M.-Y . Day and Y .-D. Lin. Deep learning for sentiment anal- ysis on Google Play consumer review. InProceedings of the IEEE International Conference on Information Reuse and Integration (IRI), pages 382–388, 2017
work page 2017
-
[6]
C. Gao, J. Zeng, D. Lo, C.-Y . Lin, M. R. Lyu, and I. King. Infar: Insight extraction from app reviews. In Proceedings of the 26th ACM Joint Meeting on European Software Engi- neering Conference and Symposium on the Foundations of Software Engineering, pages 904–907, 2018
work page 2018
-
[7]
C. Gao, J. Zeng, X. Xia, D. Lo, M. R. Lyu, and I. King. Au- tomating app review response generation. In Proceedings of the 34th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 163–175, 2019
work page 2019
-
[8]
Y . Goto and R. Tsuchihashi. What content of user reviews is considered helpful? In Proceedings of the 12th Inter- national Congress on Advanced Applied Informatics (IIAI- AAI), pages 490–494. IEEE, 2022
work page 2022
Show all 27 references
-
[9]
Greenheld, B
G. Greenheld, B. T. R. Savarimuthu, and S. A. Licorish. Au- tomating developers’ responses to app reviews. In Proceed- ings of the 25th Australasian Software Engineering Confer- ence (ASWEC), pages 66–70, 2018
2018
-
[10]
Hassan, C
S. Hassan, C. Tantithamthavorn, C.-P. Bezemer, and A. E. Hassan. Studying the dialogue between users and developers of free apps in the Google Play store. Empirical Software Engineering, 23:1275–1312, 2018
2018
-
[11]
B. Ho, K. L. Nguyen, M. Dhulipala, V . K. Pallipuram, et al. ChatReview: A ChatGPT-enabled natural language process- ing framework to study domain-specific user reviews. Ma- chine Learning with Applications, 15:100522, 2024
2024
-
[12]
C.-F. Hsu, E. Khabiri, and J. Caverlee. Ranking comments on the social web. In Proceedings of the International Con- ference on Computational Science and Engineering , vol- ume 4, pages 90–97. IEEE, 2009
2009
-
[13]
Johann, C
T. Johann, C. Stanik, A. M. Alizadeh B., and W. Maalej. SAFE: A simple approach for feature extraction from app descriptions and app reviews. In Proceedings of the 25th IEEE International Requirements Engineering Conference (RE), pages 21–30, 2017
2017
-
[14]
V . M. A. d. Lima, J. R. Barbosa, and R. M. Marcacini. MApp-IDEA: Monitoring app for issue detection and priori- tization. In Proceedings of the XXXVII Brazilian Symposium on Software Engineering, pages 180–185, 2023
2023
-
[15]
B. Lin, F. Zampetti, G. Bavota, M. Di Penta, M. Lanza, and R. Oliveto. Sentiment analysis for software engineering: How far can we go? In Proceedings of the 40th Interna- tional Conference on Software Engineering, pages 94–104, 2018
2018
-
[16]
McIlroy, N
S. McIlroy, N. Ali, H. Khalid, and A. E. Hassan. Analyzing and automatically labelling the types of user issues that are raised in mobile app reviews. Empirical Software Engineer- ing, 21:1067–1106, 2016
2016
-
[17]
McIlroy, W
S. McIlroy, W. Shang, N. Ali, and A. E. Hassan. Is it worth responding to reviews? Studying the top free apps in Google Play. IEEE Software, 34(3):64–71, 2017
2017
-
[18]
W. T. Nakamura, E. C. C. de Oliveira, E. HT de Oliveira, and T. Conte. Ux-mapper: A user experience method to ana- lyze app store reviews. In Proceedings of the XXII Brazilian Symposium on Human Factors in Computing Systems, pages 1–11, 2023
2023
-
[19]
Pagano and W
D. Pagano and W. Maalej. User feedback in the appstore: An empirical study. In Proceedings of the 21st IEEE Inter- national Requirements Engineering Conference (RE), pages 125–134, 2013
2013
-
[20]
J. W. Pennebaker, M. E. Francis, and R. J. Booth. Linguistic inquiry and word count: Liwc 2001. Mahway: Lawrence Erlbaum Associates, 71(2001):2001, 2001
2001
-
[21]
M. V . Phong, T. T. Nguyen, H. V . Pham, and T. T. Nguyen. Mining user opinions in mobile app reviews: A keyword- based approach. In Proceedings of the 30th IEEE/ACM In- ternational Conference on Automated Software Engineering (ASE), pages 749–759, 2015
2015
-
[22]
L. L. Scientific. Sentiment-based recommendation for on- line shopping. Journal of Theoretical and Applied Informa- tion Technology, 102(9), 2024
2024
-
[23]
Tantithamthavorn, S
C. Tantithamthavorn, S. McIntosh, A. E. Hassan, and K. Matsumoto. Comments on researcher bias: the use of machine learning in software defect prediction.IEEE Trans- actions on Software Engineering, 42(11):1092–1094, 2016
2016
-
[24]
P. M. Vu, T. T. Nguyen, and T. T. Nguyen. On building an automated responding system for app reviews: What are the characteristics of reviews and their responses? arXiv preprint arXiv:1908.10816, 2019
1908 arXiv
-
[25]
P. M. Vu, T. T. Nguyen, and T. T. Nguyen. Why do app reviews get responded: A preliminary study of the relation- ship between reviews and responses in mobile apps. In Pro- ceedings of the ACM Southeast Conference, pages 237–240, 2019
2019
-
[26]
P. M. Vu, H. V . Pham, T. T. Nguyen, and T. T. Nguyen. Tool support for analyzing mobile app reviews. InProceedings of the 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 789–794, 2015
2015
-
[27]
H. Wu, W. Deng, X. Niu, and C. Nie. Identifying key fea- tures from app user reviews. In Proceedings of the 43rd IEEE/ACM International Conference on Software Engineer- ing (ICSE), pages 922–932. IEEE, 2021
2021
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.