REVIEW 4 major objections 5 minor 40 references
Towards Extracting Software Requirements from App Reviews using Seq2seq Framework
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The paper claims that extracting software requirements from app reviews can be reformulated as a sequence-to-sequence NER task, and that a lightweight BiLSTM-LSTM model with self-attention and CRF outperforms transformer-based baselines…
desk verdict A plausible seq2seq NER application to app-review requirements, but the headline 0.96 F1 is not interpretable until the matching rule and baseline protocols are pinned down. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Seq2seq NER framework itself: an encoder-decoder architecture that takes a sequence of GloVe word vectors, encodes the review with a BiLSTM, applies self-attention to emphasize requirement-bearing spans, decodes with a single-layer LSTM, and finishes with a CRF that constrains tag transitions so the final sequence of BIO tags is legal. The CRF is the element that prevents nonsensical outputs such as two consecutive B tags for one requirement, and the self-attention component is what allows the model to focus on the requirement-like parts of a noisy sentence. The whole pipeline is what carries the claim that requirements extraction can be treated as tag-sequence generation rather than per-token classification.
What would settle it
Run RE-BERT and T-FREX on the same train/test app and category splits described in Section V-C, applying the same tokenization, lemmatization, and requirement-level F1 scoring; if the reproduced baseline scores differ materially from 0.48/0.12 on Dataset 1 and 0.88/0.44 on Dataset 2, the claim that Seq2seq outperforms existing state-of-the-art no longer has a valid basis.
Extended reading notes
Core claim
The central claim is that the sequence-to-sequence generation formulation is a better fit for app-review requirements extraction than the token-classification formulation used by prior work. Concretely, the authors show that a BiLSTM encoder reading each review in both directions, followed by self-attention, an LSTM decoder, and a CRF tagger, can predict BIO tags over the review tokens. On the crowdsourced T-FREX dataset the model attains a mean F1 of 0.96, exceeding the RE-BERT baseline at 0.88 and far exceeding T-FREX at 0.44, with consistent per-category scores between 0.94 and 0.98. On the manually labeled RE-BERT dataset of eight apps, the same framework reaches 0.47, essentially tied with RE-BERT's 0.48 and well above T-FREX's 0.12. The authors attribute the large dataset gap to annotation style and dataset size, and they present the runtime advantage (15 vs 50 minutes per iteration) as a practical benefit.
Load-bearing premise
The comparison's reliability rests entirely on the baseline F1 values in Tables III and IV having been measured under the exact same cross-domain splits, preprocessing steps, and requirement-level evaluation metric as the Seq2seq system; the paper reports those baseline numbers without showing how they were produced.
Editorial extensions
If this is right
- If the reported numbers hold, a lightweight recurrent model can replace transformer-based encoders for this task, cutting training time from about 50 minutes to 15 minutes per iteration on the large dataset.
- The cross-domain training strategy means the model can be applied to an entirely new app category without requiring labeled reviews from that category.
- Requirement extraction becomes part of the standard NER toolchain for app-review mining, so downstream tasks like requirement clustering and sentiment analysis can consume the extracted spans directly.
- The reformulation as a generation task, rather than classification, gives other review-based RE tasks a template for using encoder-decoder architectures.
Reading between the lines
- What the paper leaves untested is the individual contribution of the CRF layer; an ablation removing CRF (which the authors list as future work) would show whether the gain comes from the tag-transition constraints or from the attention-plus-seq2seq backend.
- The near-perfect scores on the crowdsourced dataset may partly reflect the way Dataset 2 was created: requirements were transferred into reviews from an external crowdsourced list, so the tag patterns may be simpler and more repetitive than in naturally written feature requests; testing on a fresh manually annotated sample from the same 10 categories would separate this effect.
- The same encoder-decoder tagging scheme should transfer to other noisy software texts, such as issue comments or forum posts, where requirement-like spans appear in informal language; if it fails there, the method is tuned to review-specific phrasing.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a sequence-to-sequence (Seq2seq) framework for extracting software requirements from mobile app reviews, reformulating the task as BIO-tagging/NER. The architecture is a BiLSTM encoder, an LSTM decoder, self-attention, GloVe embeddings, and a CRF layer. The model is evaluated on two datasets: a manually annotated set of 1,000 reviews (Dataset 1) and a crowdsourced set of 23,816 reviews (Dataset 2), using a leave-one-domain-out cross-validation strategy. The reported mean F1 is 0.47 on Dataset 1 and 0.96 on Dataset 2, compared with RE-BERT and T-FREX. The paper characterizes the results as preliminary and includes a research plan for further validation.
Significance. If the reported F1 of 0.96 on Dataset 2 is credible and comparable to the baseline numbers, the framework would be a substantially faster and simpler alternative to transformer-based methods for requirements extraction from app reviews, with the additional strength of publicly released code and evaluation on two independently curated datasets. The use of a cross-domain protocol and averaging over 15 runs are also positive aspects. However, the evaluation-matching rule is underspecified and the provenance of the baseline scores is unclear, so the central comparative claim is not yet established.
major comments (4)
- [V-D, Tables III and IV] The requirement-level evaluation is not precisely defined. Section V-D states that predicted B/I tokens are unified into spans and then compared with ground truth, but it never states whether a match requires exact span equality, partial overlap, or token-level overlap. This is load-bearing because Table II shows that 30,501 of 34,059 gold requirements in Dataset 2 are single-word; a lenient overlap rule can yield high F1 even when multi-word boundaries are wrong. The authors should specify the matching rule and report exact-span F1, ideally alongside token-level F1, for both datasets.
- [V-D, Tables III and IV] The baseline scores for RE-BERT and T-FREX are presented without any statement of how they were produced. The paper does not say whether these numbers were re-run under the same cross-domain folds, the same preprocessing, the same tag vocabulary, or the same matching rule used for the proposed model. T-FREX, for example, uses B-feature/I-feature tags, and its original evaluation protocol may differ in other ways. If the numbers are copied from the original papers, the comparison is not valid; if they were re-run, the configuration and code should be described. Without this information, the claim that Seq2seq outperformed state-of-the-art methods is unsupported.
- [IV] The proposed architecture is under-specified. There are no equations for the BiLSTM encoder states, the self-attention scores, the LSTM decoder update, or the CRF scoring function. The description says the decoder generates tag representations but also states that the decoder is not directly used to generate BIO tags; it is unclear whether the CRF is applied to decoder outputs, to encoder outputs, or as a global re-ranker, and what loss function is optimized. This makes the framework non-reproducible from the paper alone despite the availability of code.
- [V-C and VI, Tables III and IV] The paper states that 15 runs were performed for each app/category, but Tables III and IV report only mean F1 values with no standard deviations, ranges, or significance tests. The reported difference between Seq2seq (0.96) and RE-BERT (0.88) on Dataset 2 could fall within run-to-run variation, so the authors should report variance across the 15 runs and test the statistical significance of the differences.
minor comments (5)
- [I] The phrase 'condition random fields' should be 'conditional random fields'.
- [Tables I and III] The app name is given as 'Photo editor' in Table I but as 'Phone editor' in Table III; the naming should be consistent.
- [References] References [12] and [20] both cite the same GloVe paper; they should be consolidated into a single reference.
- [VI] The runtime comparison of 50 minutes per iteration for RE-BERT versus 15 minutes for Seq2seq is anecdotal; the authors should specify what 'iteration' includes and whether the comparison was made on the same hardware and folds.
- [VIII] The research plan itself acknowledges the lack of an ablation study and sensitivity analysis; these should be performed before attributing the results to specific components such as the CRF layer or self-attention.
Circularity Check
No significant circularity: the central F1 claims rest on held-out cross-domain evaluation with externally sourced hyperparameters, not on fitted inputs or self-citation chains.
full rationale
The paper's derivation chain is: reformulate requirements extraction as BIO-tag NER; build a standard BiLSTM encoder/LSTM decoder with self-attention, GloVe embeddings, and a CRF layer; train on app/category splits from two externally curated datasets; and measure held-out F1. Nothing in this chain defines the target result in terms of the model's own outputs: the ground-truth tags are created from the datasets' published annotations, the train/test splits are disjoint by app/category (Section V-C), the hyperparameters are standard values from prior work [21] rather than fitted to the test folds, and the architecture choices are justified from general NLP literature [13]-[17], [21]. The paper's self-citations ([28], [39]) appear only in future-work discussion about ethical concerns and do not support the quantitative claims. The limitations section explicitly acknowledges dataset-construction and hyperparameter threats (Section VII). The main weakness is not circularity: Tables III and IV report RE-BERT and T-FREX F1 values without demonstrating that these baselines were re-run under identical preprocessing, tokenization, and match criteria, and Section V-D does not define whether a 'requirement-level' match requires exact span equality. That is an evaluation-comparability threat, not a circular derivation. Accordingly, no circular step is exhibited, and the score is 1 (minor non-load-bearing self-citation only, with no reduction of the claimed result to its inputs).
Assumptions & free parameters
free parameters (1)
- training hyperparameters =
lr=0.001, embedding_dim=300, batch_size=32
assumptions (5)
- domain assumption Dataset 1 auto-generated BIO tags, created by matching lemmatized feature phrases to lemmatized sentence tokens, faithfully reproduce the RE-BERT ground truth.
- domain assumption Dataset 2's crowdsourced ground truth from T-FREX is complete and correct.
- domain assumption Cross-domain training transfers: models trained on some app categories can tag requirements in unseen categories.
- domain assumption GloVe embeddings contain sufficient semantic and syntactic information for this domain.
- domain assumption Standard hyperparameters from prior seq2seq NER work [21] are appropriate for this task.
Cite this review
Pith. "Pith review of Towards Extracting Software Requirements from App Reviews using Seq2seq Framework." pith.science (2026). https://pith.science/paper/IHDMKH6F
@misc{pith2026250709039,
author = {Pith},
title = {Pith review of: Towards Extracting Software Requirements from App Reviews using Seq2seq Framework},
year = {2026},
howpublished = {\url{https://pith.science/paper/IHDMKH6F}},
note = {Machine review of arXiv:2507.09039}
}
read the original abstract
Mobile app reviews are a large-scale data source for software improvements. A key task in this context is effectively extracting requirements from app reviews to analyze the users' needs and support the software's evolution. Recent studies show that existing methods fail at this task since app reviews usually contain informal language, grammatical and spelling errors, and a large amount of irrelevant information that might not have direct practical value for developers. To address this, we propose a novel reformulation of requirements extraction as a Named Entity Recognition (NER) task based on the sequence-to-sequence (Seq2seq) generation approach. With this aim, we propose a Seq2seq framework, incorporating a BiLSTM encoder and an LSTM decoder, enhanced with a self-attention mechanism, GloVe embeddings, and a CRF model. We evaluated our framework on two datasets: a manually annotated set of 1,000 reviews (Dataset 1) and a crowdsourced set of 23,816 reviews (Dataset 2). The quantitative evaluation of our framework showed that it outperformed existing state-of-the-art methods with an F1 score of 0.96 on Dataset 2, and achieved comparable performance on Dataset 1 with an F1 score of 0.47.
Figures
Reference graph
Works this paper leans on
-
[1]
App store effects on software engineering practices,
A. A. Al-Subaihin, F. Sarro, S. Black, L. Capra, and M. Harman, “App store effects on software engineering practices,” IEEE Transactions on Software Engineering, vol. 47, no. 2, pp. 300–319, 2019
work page 2019
-
[2]
How do users like this feature? a fine grained sentiment analysis of app reviews,
E. Guzm ´an and W. Maalej, “How do users like this feature? a fine grained sentiment analysis of app reviews,” in 2014 IEEE 22nd inter- national requirements engineering conference (RE) . Ieee, 2014, pp. 153–162
work page 2014
-
[3]
Toward data-driven requirements engineering,
W. Maalej, M. Nayebi, T. Johann, and G. Ruhe, “Toward data-driven requirements engineering,” IEEE software , vol. 33, no. 1, pp. 48–54, 2015
work page 2015
-
[4]
App store analysis for software engineering,
W. J. Martin, “App store analysis for software engineering,” Ph.D. dissertation, UCL (University College London), 2017
work page 2017
-
[5]
User feedback in the appstore: An empir- ical study,
D. Pagano and W. Maalej, “User feedback in the appstore: An empir- ical study,” in 2013 21st IEEE international requirements engineering conference (RE). IEEE, 2013, pp. 125–134
work page 2013
-
[6]
M. Tavakoli, L. Zhao, A. Heydari, and G. Nenadi ´c, “Extracting useful software development information from mobile application reviews: A survey of intelligent mining techniques and tools,” Expert Systems with Applications, vol. 113, pp. 186–199, 2018
work page 2018
-
[7]
Bug report, feature request, or simply praise? on automatically classifying app reviews,
W. Maalej and H. Nabil, “Bug report, feature request, or simply praise? on automatically classifying app reviews,” in 2015 IEEE 23rd international requirements engineering conference (RE) . IEEE, 2015, pp. 116–125
work page 2015
-
[8]
Re-bert: automatic extraction of software requirements from app reviews using bert language model,
A. F. de Ara ´ujo and R. M. Marcacini, “Re-bert: automatic extraction of software requirements from app reviews using bert language model,” in Proceedings of the 36th annual ACM symposium on applied computing , 2021, pp. 1321–1327
work page 2021
Show all 40 references
-
[9]
Safe: A simple approach for feature extraction from app descriptions and app reviews,
T. Johann, C. Stanik, W. Maalej et al. , “Safe: A simple approach for feature extraction from app descriptions and app reviews,” in 2017 IEEE 25th international requirements engineering conference (RE) . IEEE, 2017, pp. 21–30
2017
-
[10]
An unsupervised aspect extraction strategy for monitoring real-time reviews stream,
M. Dragoni, M. Federici, and A. Rexha, “An unsupervised aspect extraction strategy for monitoring real-time reviews stream,”Information processing & management , vol. 56, no. 3, pp. 1103–1118, 2019
2019
-
[11]
T- frex: A transformer-based feature extraction method from mobile app reviews,
Q. Motger, A. Miaschi, F. Dell’Orletta, X. Franch, and J. Marco, “T- frex: A transformer-based feature extraction method from mobile app reviews,” in 2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) . IEEE, 2024, pp. 227–238
2024
-
[12]
GloVe: Global vectors for word representation,
J. Pennington, R. Socher, and C. Manning, “GloVe: Global vectors for word representation,” in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) , A. Moschitti, B. Pang, and W. Daelemans, Eds. Doha, Qatar: Association for Computation...
2014
-
[13]
Bidirectional recurrent neural net- works,
M. Schuster and K. K. Paliwal, “Bidirectional recurrent neural net- works,” IEEE transactions on Signal Processing , vol. 45, no. 11, pp. 2673–2681, 1997
1997
-
[14]
Attention is all you need,
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017
2017
-
[15]
Long short-term memory,
S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997
1997
-
[16]
An introduction to conditional random fields,
C. Sutton, A. McCallum et al. , “An introduction to conditional random fields,” F oundations and Trends® in Machine Learning , vol. 4, no. 4, pp. 267–373, 2012
2012
-
[17]
Neural architectures for named entity recognition,
G. Lample, “Neural architectures for named entity recognition,” arXiv preprint arXiv:1603.01360, 2016
2016 arXiv
-
[18]
Neural machine translation by jointly learning to align and translate,
D. Bahdanau, “Neural machine translation by jointly learning to align and translate,” arXiv preprint arXiv:1409.0473 , 2014
2014 arXiv
-
[19]
Tackling structured knowledge extraction from polymer nanocomposite literature as an ner/re task with seq2seq,
B. Hu, A. Lin, and L. C. Brinson, “Tackling structured knowledge extraction from polymer nanocomposite literature as an ner/re task with seq2seq,” Integrating Materials and Manufacturing Innovation , vol. 13, no. 3, pp. 656–668, 2024
2024
-
[20]
Glove: Global vectors for word representation,
J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors for word representation,” in Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) , 2014, pp. 1532–1543
2014
-
[21]
Fine grained named entity recognition via seq2seq framework,
H. Zhu, C. He, Y . Fang, and W. Xiao, “Fine grained named entity recognition via seq2seq framework,” IEEE Access , vol. 8, pp. 53 953– 53 961, 2020
2020
-
[22]
Decoding domain-specific ner: A performance evaluation of chatgpt, bi-lstm, and bert,
K. A. Rafique, M. Pansuriya, F. Wawrzik, and C. Grimm, “Decoding domain-specific ner: A performance evaluation of chatgpt, bi-lstm, and bert,” in 2024 11th International Conference on Machine Intelligence Theory and Applications (MiTA) . IEEE, 2024, pp. 1–8
2024
-
[23]
Advancing requirements engineering through generative ai: Assessing the role of llms,
C. Arora, J. Grundy, and M. Abdelrazek, “Advancing requirements engineering through generative ai: Assessing the role of llms,” in Generative AI for Effective Software Development . Springer, 2024, pp. 129–148
2024
-
[24]
Investigating chatgpt’s potential to assist in requirements elicitation processes,
K. Ronanki, C. Berger, and J. Horkoff, “Investigating chatgpt’s potential to assist in requirements elicitation processes,” in 2023 49th Euromi- cro Conference on Software Engineering and Advanced Applications (SEAA). IEEE, 2023, pp. 354–361
2023
-
[25]
Using chatgpt in software requirements engineering: A comprehensive review,
N. Marques, R. R. Silva, and J. Bernardino, “Using chatgpt in software requirements engineering: A comprehensive review,” vol. 16, no. 6, p. 180, 2024
2024
-
[26]
Comparison of pipeline, sequence-to- sequence, and gpt models for end-to-end relation extraction: experiments with the rare disease use-case,
S. Gupta, X. Ai, and R. Kavuluru, “Comparison of pipeline, sequence-to- sequence, and gpt models for end-to-end relation extraction: experiments with the rare disease use-case,” arXiv preprint arXiv:2311.13729 , 2023
2023 arXiv
-
[27]
Named entity recognition and relation extraction: State-of-the-art,
Z. Nasar, S. W. Jaffry, and M. K. Malik, “Named entity recognition and relation extraction: State-of-the-art,” ACM Computing Surveys (CSUR) , vol. 54, no. 1, pp. 1–39, 2021
2021
-
[28]
Towards extracting ethical concerns- related software requirements from app reviews,
A. Sorathiya and G. Ginde, “Towards extracting ethical concerns- related software requirements from app reviews,” in Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, 2024, pp. 2251–2255
2024
-
[29]
Nltk: the natural language toolkit,
S. Bird, “Nltk: the natural language toolkit,” in Proceedings of the COLING/ACL 2006 interactive presentation sessions , 2006, pp. 69–72
2006
-
[30]
C. C. Aggarwal and C. C. Aggarwal, Machine learning for text: An introduction. Springer, 2018
2018
-
[31]
Abstractive text summarization using sequence-to-sequence rnns and beyond,
R. Nallapati, B. Zhou, C. Gulcehre, B. Xiang et al. , “Abstractive text summarization using sequence-to-sequence rnns and beyond,” arXiv preprint arXiv:1602.06023, 2016
2016 arXiv
-
[32]
A neural conversational model,
O. Vinyals and Q. Le, “A neural conversational model,” arXiv preprint arXiv:1506.05869, 2015
2015 arXiv
-
[33]
Bidirectional lstm-crf for named entity recognition
R. Panchendrarajan and A. Amaresan, “Bidirectional lstm-crf for named entity recognition.” 32nd Pacific Asia Conference on Language, Information and Computation, 2018
2018
-
[34]
Mobile feature-oriented knowl- edge base generation using knowledge graphs,
Q. Motger, X. Franch, and J. Marco, “Mobile feature-oriented knowl- edge base generation using knowledge graphs,” in European Conference on Advances in Databases and Information Systems . Springer, 2023, pp. 269–279
2023
-
[35]
Stanza: A Python natural language processing toolkit for many human languages,
P. Qi, Y . Zhang, Y . Zhang, J. Bolton, and C. D. Manning, “Stanza: A Python natural language processing toolkit for many human languages,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations , 2020. [Online]. Avail...
2020
-
[36]
Confidence estimation for attention-based sequence-to- sequence models for speech recognition,
Q. Li, D. Qiu, Y . Zhang, B. Li, Y . He, P. C. Woodland, L. Cao, and T. Strohman, “Confidence estimation for attention-based sequence-to- sequence models for speech recognition,” in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICAS...
2021
-
[37]
Why people hate your app: Making sense of user feedback in a mobile app store,
B. Fu, J. Lin, L. Li, C. Faloutsos, J. Hong, and N. Sadeh, “Why people hate your app: Making sense of user feedback in a mobile app store,” in Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining , 2013, pp. 1276–1284
2013
-
[38]
How should i improve the ui of my app? a study of user reviews of popular apps in the google play,
Q. Chen, C. Chen, S. Hassan, Z. Xing, X. Xia, and A. E. Hassan, “How should i improve the ui of my app? a study of user reviews of popular apps in the google play,” ACM Transactions on Software Engineering and Methodology (TOSEM) , vol. 30, no. 3, pp. 1–38, 2021
2021
-
[39]
Ethical software requirements from user re- views: A systematic literature review,
A. Sorathiya and G. Ginde, “Ethical software requirements from user re- views: A systematic literature review,”arXiv preprint arXiv:2410.01833, 2024
2024 arXiv
-
[40]
Uncovering patterns in users’ ethical concerns about software,
¨O. Karac ¸am, T. P. Humbert, and E. Guzm ´an, “Uncovering patterns in users’ ethical concerns about software,” in2024 IEEE 32nd International Requirements Engineering Conference (RE) . IEEE, 2024, pp. 466–474
2024
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.