Pith. sign in

REVIEW 3 major objections 4 minor 33 references

Toward Inclusive Low-Code Development: Detecting Accessibility Issues in User Reviews

T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A hybrid transformer-plus-keyword model detects accessibility complaints in low-code app reviews at 78% accuracy and 78% F1, beating BERT-class baselines.

desk verdict The 78% F1 is real under the paper's own labels, but the labels are so broad that the model likely measures ordinary UX complaints, not disability-specific accessibility. read the letter →

arxiv 2504.19085 v2 pith:MIJ3P2LA submitted 2025-04-27 cs.SE

Mohammadali Mohammadkhani , Sara Zahedi Movahed , Hourieh Khalajzadeh , Mojtaba Shahin , Khuong Tran Hoang This is my paper · ORCID
classification cs.SE
keywords accessibilitylow-codedevelopmentuserreviewshybridclassifiersentenceembeddingskeyword-basedclassificationnaturallanguageprocessingsoftwareengineering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to answer whether accessibility-related complaints in low-code application reviews can be detected automatically, and argues that a hybrid classifier—two sentence-embedding transformers combined with a keyword-based fallback—reaches 78% accuracy and 78% F1 on a held-out balanced test set. The motivation is that low-code platforms rely on graphical interfaces and can exclude users with visual impairments, and developers currently have no scalable way to surface such complaints from unstructured user reviews. The main contribution is the demonstration that this detection is feasible at a usable level, together with the public release of a 4,762-review dataset.

What carries the argument

The central machinery is the hybrid decision pipeline: two frozen sentence-embedding transformers produce a concatenated vector (1,152 dimensions), a five-layer feed-forward network maps it to two classes, and a confidence threshold at 80% routes low-confidence predictions to a keyword-based fallback. The keyword system carries two lexicons—one marking accessibility-related language, one marking developer-side issues—and is invoked only when the neural network is unsure, which lets the model keep linguistic flexibility while gaining explicit term detection. The 80% threshold is what makes the fusion work: high-confidence neural predictions stand, and ambiguous cases are decided by transparent keyword rules.

What would settle it

Take a random sample of the test reviews and relabel them under a stricter rule requiring explicit evidence of visual impairment, assistive technology use, or standard accessibility vocabulary (color, contrast, screen reader, magnification), then recompute the model's F1 on the stricter labels; a large drop would show that the 78% figure tracks general UX complaints rather than accessibility barriers.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a hybrid classifier can identify accessibility-related complaints in low-code application reviews at a level that beats fine-tuned transformer baselines: 78.07% accuracy, 78.52% F1, and 82.70% recall on a balanced held-out test set of 716 reviews. The model concatenates embeddings from two sentence-embedding models, all-MiniLM-L6-v2 and LaBSE, into a 1,152-dimensional vector, classifies with a five-layer fully connected network, and, when the network's confidence is below 80%, defers to a keyword-based system built from accessibility-related and developer-side keyword lists. The authors interpret the result as evidence that the two mechanisms complement each other: the keyword fallback raises recall by catching explicit accessibility terms the transformers miss, while the transformers handle implicit phrasing. They also report that removing the keyword component drops F1 to 75.89%, and that the hybrid's precision (74.73%) is lower than BERT's, meaning false positives are the main remaining cost.

Load-bearing premise

The result rests on the annotation rule that a review reporting any bug or inconvenience in the UI, navigation, customization, or usability counts as accessibility-related; if those labels capture general dissatisfaction rather than barriers faced by users with disabilities, the model is detecting a broader construct than accessibility.

Editorial extensions

If this is right

  • Low-code platform teams could deploy the model as a review triage filter, routing accessibility-related complaints to accessibility-focused developers.
  • Because the keyword fallback acts only on low-confidence predictions, ambiguous reviews can be surfaced for human review rather than silently accepted.
  • Adding the keyword component raises recall from 77.82% to 82.70%, so teams that prefer to miss few accessibility complaints would accept more false positives in exchange.
  • The released 4,762-review dataset gives later work a shared benchmark for accessibility detection in low-code app reviews.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same 80%-confidence architecture could serve as an active-learning loop: send low-confidence reviews to human annotators and add their labels to the training set, which would shrink the keyword system's workload over time.
  • Because the annotation rule labels any UI, navigation, customization, or usability complaint as accessibility-related, the model likely measures general usability barriers as much as disability-specific ones; a stricter visual-impairment definition would be a natural next test, and performance would probably fall.
  • Given that prior work cited in the paper found only 1.2% of mobile reviews mention accessibility, a deployment on naturally distributed reviews would face a severe class imbalance the balanced test set does not reflect; the 78% F1 should be read as performance on balanced data rather than on live review streams.
  • One testable extension is cross-lingual transfer: since LaBSE is language-agnostic, the same hybrid could be evaluated on non-English low-code reviews without retraining, which the paper does not do.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper addresses automatic detection of accessibility-related issues in user reviews of low-code platforms. The authors compile a dataset of 4,762 reviews from TrustRadius, G2, Gartner, and an existing accessibility-review dataset; annotate them as accessibility-related or not; build a hybrid classifier that concatenates sentence embeddings from all-MiniLM-L6-v2 and LaBSE, passes them through a five-layer neural network, and optionally applies a keyword-based override when the network's confidence is below 80%. The model is evaluated on a held-out balanced test set of 716 reviews, reporting 78.07% accuracy and 78.52% F1, and is compared against fine-tuned BERT, RoBERTa, DistilBERT, and a no-keyword ablation. The paper also releases the dataset and code on Zenodo.

Significance. If the central claim holds, the contribution is useful: a public dataset of low-code app reviews, a reproducible crawler, and a classifier that could help developers triage accessibility feedback in a domain where accessibility research is still limited (the authors cite their own systematic review). The paper is transparent about data collection, releases code and data, and discusses validity threats. However, the significance is currently conditional on whether the annotation definition in Section 2.1.2 actually captures disability-relevant accessibility rather than general UX dissatisfaction. The evaluation also lacks statistical confidence measures, and the deployment setting is likely imbalanced, so the magnitude of the claimed practical contribution is not yet established.

major comments (3)
  1. [Section 2.1.2, Rule 2] The positive-class definition is too broad to support the abstract's claim of detecting 'accessibility-related issues.' Rule 2 labels as accessibility-related any review that 'reports a bug or inconvenience that affects users and renders the app or some of its features inaccessible,' and it explicitly includes 'problems with the user interface, navigation, customization, usability, and similar aspects.' Nothing in this rule requires the reported difficulty to be tied to a user's disability, impairment, or need for assistive technology; an ordinary complaint such as 'settings are hard to find' qualifies. Under this rule 52.8% of the corpus is positive, which is implausibly high for disability-specific accessibility issues and is inconsistent with the paper's own related-work observation that only about 1.2% of Google Play reviews mention accessibility. The 0.3% annotator disagreement does not mitigate this concern because both annotators applied the same broad rule. The paper's threats-to-validity section mentions 'potential subjectivity in labeling' but does not re-examine whether Rule 2 measures the construct promised in the title. I therefore regard the central empirical claim as not yet established: an independent audit of test-set positives by annotators with accessibility expertise, or a redefinition of the positive class, is needed to verify that the model detects disability-relevant accessibility barriers rather than general usability dissatisfaction.
  2. [Table 1] The claimed advantage of the hybrid model over its no-keyword ablation is not statistically supported. In Table 1 the hybrid model achieves F1 78.52% versus 75.89% for Hybrid (No Keywords), a difference of 2.63 percentage points, with no confidence intervals, multiple-seed runs, or significance tests reported. The same is true for the comparison against fine-tuned RoBERTa (F1 74.17%). Because the keyword-override threshold and the keyword lists are themselves fitted on validation data, the improvement could be due to overfitting of these free parameters. Please report repeated runs with standard deviations, bootstrap confidence intervals, or paired significance tests, and describe how the 80% confidence threshold and the keyword sets were selected on validation data. Without this, the claim that the keyword component 'improves metrics' is not yet convincing.
  3. [Section 2.1.4 and Section 4.2] The evaluation setup may not reflect the intended deployment distribution. The test set is balanced (347/716 positive, 48.5%), but the related-work discussion itself notes that accessibility mentions are rare in app reviews (about 1.2% in Google Play); if the real-world prior is similarly low, precision and recall at the reported operating point are not informative about deployed performance, since the false-positive rate would dominate. Additionally, the random 90/10 split is done at the review level without grouping by low-code platform or data source, so reviews from the same platform can appear in both training and test sets; the model may then exploit platform-specific vocabulary rather than general accessibility language. I ask the authors to stratify the split by platform or otherwise assess within-platform versus cross-platform generalization, and to report metrics under a more realistic class prior or at least discuss the expected precision-recall tradeoff.
minor comments (4)
  1. [Section 2.1.1, Step 3] The text says 'G2 and Gertner employ a strict...'; 'Gertner' should be 'Gartner.'
  2. [Section 4.1] The survey credited to 'Ballantyne et al.' with reference [24] is actually authored by Camila Silva, Marcelo Medeiros Eler, and Gordon Fraser; please correct the in-text attribution and the reference entry.
  3. [Section 2.2.4] The paper does not report the final keyword lists or the number of keywords in each set, which limits reproducibility and makes it hard to assess how much of the model's behavior is driven by the keyword override; consider including the lists in an appendix or supplement.
  4. [Section 3.1] The fine-tuning details for the baseline models (BERT, RoBERTa, DistilBERT) are not reported; specifying their epochs, learning rates, and batch sizes would make the comparison fairer and more reproducible.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the model's reported F1 is an empirical held-out evaluation, not a construction equivalent to its inputs.

full rationale

This paper reports a supervised-learning evaluation rather than a mathematical derivation. The target class is defined by the annotation rules in Section 2.1.2, the training/validation split is described in Section 2.1.4, and the final Table 1 metrics are computed on a held-out test set of 716 reviews that the models did not access during training. The hybrid model combines fixed sentence-transformers with a neural classifier and a keyword-based decision aid; the keywords and confidence threshold are part of the model configuration, and the paper reports both the full hybrid model and the Hybrid (No Keywords) ablation on the same held-out data. Nothing in the paper sets the claimed 78% accuracy/F1 equal to the labeling rule or to a parameter fitted on the test labels. The only self-citation, the authors' systematic literature review [16], is used as background motivation for why low-code accessibility matters and does not participate in the detection method or its evaluation. Concerns that Rule 2 may operationalize accessibility as general usability/UX dissatisfaction are construct-validity questions, not circularity, and the paper itself acknowledges labeling subjectivity as a validity threat.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central empirical claim rests on the annotation operationalization (Rule 2), the source-selection assumptions, the balanced-split assumption, and the fixed-embedding assumption. No new theoretical entities are introduced. The model's free parameters (threshold, keyword lists, hyperparameters) are tuned on training and validation, which is standard practice but not reported with sensitivity analysis.

free parameters (4)
  • Confidence threshold for keyword override = 80%
    In Section 2.2.4, if model confidence is 80 percent or below, the keyword system overrides the prediction. The threshold is a hand-set hyperparameter; its effect on performance is not ablated across values.
  • Keyword sets for accessibility and developer-side cues = Two curated lists
    Section 2.2.4: sets are extracted with YAKE from training reviews, then manually tailored. The exact lists are not published in the paper, only referenced via the repository, so their reproducibility depends on the Zenodo artifacts.
  • Neural network hyperparameters = epochs=3, lr=0.005, batch=32
    Section 2.2.3: chosen because they gave the best validation results. No search details or sensitivity analysis are reported.
  • Minimum review length threshold = 5 words
    Section 2.1.3: reviews shorter than five words are removed before annotation; this changes the distribution of the test set as well as training.
assumptions (4)
  • domain assumption Rule 2 equates UI, navigation, customization, and usability complaints with accessibility issues.
    Section 2.1.2 defines label 1. If this operationalization is wrong, the model detects general UX dissatisfaction rather than disability-related accessibility barriers, which would change the meaning of the central claim.
  • domain assumption The collected review sources (TrustRadius, G2, Gartner) plus a prior accessibility dataset are representative of low-code user feedback.
    Section 2.1.1 builds the dataset from these sources; the top-10 platform list is taken from a PCMag article [19] and G2/Gartner reviews are sampled at 20 percent. Biases in these sources directly shape both training and evaluation.
  • domain assumption A balanced train and test split approximates the operating condition of the classifier.
    Section 2.1.4 yields a 53/47 split. Prior work [12] reports about 1.2 percent accessibility mentions in general app reviews; if low-code reviews follow a similar base rate, the reported 78 percent F1 will not transfer to real-world deployment without recalibration.
  • domain assumption Pre-trained embeddings (all-MiniLM-L6-v2, LaBSE) capture the semantics of short, phrase-like reviews.
    Section 2.2.2 uses these embeddings as fixed feature extractors; TrustRadius reviews are bullet points, not full sentences, which may sit outside the distribution these models were trained on.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Toward Inclusive Low-Code Development: Detecting Accessibility Issues in User Reviews." pith.science (2026). https://pith.science/paper/MIJ3P2LA

@misc{pith2026250419085,
  author       = {Pith},
  title        = {Pith review of: Toward Inclusive Low-Code Development: Detecting Accessibility Issues in User Reviews},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MIJ3P2LA}},
  note         = {Machine review of arXiv:2504.19085}
}
read the original abstract

Low-code applications are gaining popularity across various fields, enabling non-developers to participate in the software development process. However, due to the strong reliance on graphical user interfaces, they may unintentionally exclude users with visual impairments, such as color blindness and low vision. This paper investigates the accessibility issues users report when using low-code applications. We construct a comprehensive dataset of low-code application reviews, consisting of accessibility-related reviews and non-accessibility-related reviews. We then design and implement a complex model to identify whether a review contains an accessibility-related issue, combining two state-of-the-art Transformers-based models and a traditional keyword-based system. Our proposed hybrid model achieves an accuracy and F1-score of 78% in detecting accessibility-related issues.

Figures

Figures reproduced from arXiv: 2504.19085 by the authors.

Figure 1
Figure 1. Three sample app reviews in our dataset. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The overall pipeline of our proposed method. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 26 canonical work pages

  1. [1]

    Toward Inclusive Low-Code Development: Analyzing Accessibility Issues in User Reviews

    2024. Toward Inclusive Low-Code Development: Analyzing Accessibility Issues in User Reviews. Zenodo. doi:10.5281/zenodo.14060778

    work page doi:10.5281/zenodo.14060778 2024

  2. [2]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  3. [3]

    Wajdi Aljedaani, Furqan Rustam, Stephanie Ludi, Ali Ouni, and Mohamed Wiem Mkaouer. 2021. Learning sentiment analysis for accessibility user reviews. In 2021 36th IEEE/ACM International conference on automated software engineering workshops (ASEW). IEEE, 239–246

    work page 2021

  4. [4]

    Eman Abdullah AlOmar, Wajdi Aljedaani, Murtaza Tamjeed, Mohamed Wiem Mkaouer, and Yasmine N El-Glaly. 2021. Finding the needle in a haystack: On the automatic identification of accessibility user reviews. In Proceedings of the 2021 CHI conference on human factors in computing systems . 1–15

    work page 2021

  5. [5]

    Abdulaziz Alshayban, Iftekhar Ahmed, and Sam Malek. 2020. Accessibility issues in android apps: state of affairs, sentiments, and ways forward. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering . 1323–1334

    work page 2020

  6. [6]

    Asm Iftekhar Anam and Mohammed Yeasin. 2013. Accessibility in smartphone ap- plications: what do we learn from reviews?. InProceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility . 1–2

    work page 2013

  7. [7]

    Tingting Bi, Xin Xia, David Lo, and Aldeida Aleti. 2021. A first look at accessibility issues in popular github projects. In2021 IEEE International Conference on Software Maintenance and Evolution (ICSME) . IEEE, 390–401

    work page 2021

  8. [8]

    Ben Caldwell, Michael Cooper, Loretta Guarino Reid, Gregg Vanderheiden, Wendy Chisholm, John Slatin, and Jason White. 2008. Web content accessibility guide- lines (WCAG) 2.0. WWW Consortium (W3C) 290, 1-34 (2008), 5–12

    work page 2008

Show all 33 references
  1. [9]

    Taís Borges Ferreira Marcelo da Costa Ferreira Daniel Sannomia. 2019. Evaluation of the Use of Accessibility Tools in the Information Technology Industry: A Case Study. InProceedings of the 2019 IEEE Symposium on Visual Languages and Human- Centric Computing (VL/HCC). IEEE. do...

    2019

  2. [10]

    Jacob Devlin. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)

    2018 arXiv

  3. [11]

    Mohamed El-Attar. 2023. Evaluating the accessibility of a PoN-enabled misuse case notation by the red–green colorblind community. Software and Systems Modeling 22, 1 (2023), 247–272

    2023

  4. [12]

    Marcelo Medeiros Eler, Leandro Orlandin, and Alberto Dumont Alves Oliveira

  5. [13]

    Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2020. Language-agnostic BERT sentence embedding. arXiv preprint arXiv:2007.01852 (2020)

    2020 arXiv

  6. [14]

    International Organization for Standardization. 2008. Ergonomics of human- system interaction: Part 171: Guidance on software accessibility; ISO 9241-171: 2008 (E). ISO

    2008

  7. [15]

    Computer as a Tool

    Alenka Kavcic. 2005. Software accessibility: Recommendations and guidelines. In EUROCON 2005-The International Conference on" Computer as a Tool" , Vol. 2. IEEE, 1024–1027

    2005

  8. [16]

    Hourieh Khalajzadeh and John Grundy. 2025. Accessibility of low-code ap- proaches: A systematic literature review. Information and Software Technology 177 (2025), 107570. doi:10.1016/j.infsof.2024.107570

    2025

  9. [17]

    Yinhan Liu. 2019. Roberta: A robustly optimized bert pretraining approach.arXiv preprint arXiv:1907.11692 364 (2019)

    2019 arXiv

  10. [18]

    Ellen Francine Barbosa Maria Lydia Fioravanti, Renata Pontin de Mattos Fortes

  11. [19]

    Rob Marvin. 2018. The Best Low-Code Development Platforms. PCMag Australia (2018)

    2018

  12. [20]

    In Proceedings of the 2019 IEEE Frontiers in Education Conference (FIE)

    Accessibility in Mobile Applications for Elderly Users: A Systematic Map- ping. In Proceedings of the 2019 IEEE Frontiers in Education Conference (FIE) . IEEE. doi:10.1109/FIE.2019.8659017

    2019

  13. [21]

    Shashiwadana Nirmani, Mojtaba Shahin, Hourieh Khalajzadeh, and Xiao Liu

  14. [22]

    Forough Mehralian, Navid Salehnamadi, Syed Fatiul Huq, and Sam Malek. 2022. Too much accessibility is harmful! automated detection and analysis of overly accessible elements in mobile apps. In Proceedings of the 37th IEEE/ACM Interna- tional Conference on Automated Software En...

    2022

  15. [23]

    V Sanh. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 (2019)

    2019 arXiv

  16. [24]

    Camila Silva, Marcelo Medeiros Eler, and Gordon Fraser. 2018. A survey on the tool support for the automatic evaluation of mobile accessibility. In Proceedings of the 8th International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting...

    2018

  17. [25]

    Helen Petrie, Andreas Savva, and Christopher Power. 2015. Towards a unified definition of web accessibility. In Proceedings of the 12th International Web for All Conference. 1–13

    2015

  18. [26]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023)

    2023 arXiv

  19. [27]

    Paul Vincent, Kimihiko Iijima, Mark Driver, Jason Wong, and Yefim Natis. 2019. Magic quadrant for enterprise low-code application platforms. Gartner report 120 (2019)

    2019

  20. [28]

    Murtaza Tamjeed. 2020. Accessibility in user reviews for mobile apps: An automated detection approach. Rochester Institute of Technology

    2020

  21. [29]

    Yaxin Zhao, Lina Gong, Wenhua Yang, and Yu Zhou. 2024. How accessibility affects other quality attributes of software? A case study of GitHub. Science of Computer Programming 231 (2024), 103027

    2024

  22. [31]

    Yawen Wang, Junjie Wang, Hongyu Zhang, Xuran Ming, Lin Shi, and Qing Wang

  23. [2019]

    InProceedings of the 18th Brazilian symposium on human factors in computing systems

    Do Android app users care about accessibility? an analysis of user reviews on the Google play store. InProceedings of the 18th Brazilian symposium on human factors in computing systems . 1–11

  24. [2022]

    InProceedings of the 44th international conference on software engineering

    Where is your app frustrating users?. InProceedings of the 44th international conference on software engineering . 2427–2439

  25. [2024]

    arXiv preprint arXiv:2407.09872 (2024)

    A Systematic Literature Review on Task Recommendation Systems for Crowdsourced Software Engineering. arXiv preprint arXiv:2407.09872 (2024)

    2024 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.