Pith. sign in

REVIEW 4 major objections 6 minor 58 references

LaQual: An Automated Framework for LLM App Quality Evaluation

T0 review · 4 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LaQual auto-scores LLM apps, matching human judgments

desk verdict Useful framework, public code, but the headline consistency result collapses once the clustering is taken into account, and the thresholds table doesn't match the data. read the letter →

arxiv 2508.18636 v2 pith:5X4BGIPO submitted 2025-08-26 cs.SE cs.AI

Yan Wang , Xinyi Hou , Junjun Si , Yanjie Zhao , Weiguo Lin , Haoyu Wang This is my paper
classification cs.SEcs.AI
keywords LLMappstoresautomatedqualityevaluationLLM-as-a-judgescenario-adaptivemetricsstaticindicatorfilteringuserengagementrecommendationsystemsSpearmancorrelation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LaQual is an automated quality-evaluation framework for apps in LLM app stores, where rankings currently rely on static popularity metrics that can be gamed. The paper claims that a three-stage pipeline—LLM-based labeling and hierarchical classification, time-weighted static filtering, and LLM-generated scenario-specific metrics and tasks—can score app quality without human labor. The key evidence is that LaQual's automated scores correlate with expert human ratings (Spearman's rho around 0.60–0.62 in legal and travel scenarios) and that its static filter cuts candidate pools by roughly two-thirds to four-fifths. A user study reports faster comparison and higher trust than the store's native ranking. If correct, the framework gives store managers, developers, and users a scalable, objective alternative to popularity-based ranking.

What carries the argument

The load-bearing mechanism is the dynamic scenario-adaptive evaluation module: an LLM takes the label produced in stage one (e.g., 'travel itinerary planning') and generates three scenario-specific metrics with five-level scoring criteria and concrete test tasks, then acts as judge on the app's responses with temperature zero. This is what lets one framework span diverse app categories without hand-built benchmarks. The preceding static stage (time-weighted quarterly averages of views, interactions, favorites, copies, plus plugin/knowledge-base/model counts) is what makes the process affordable, trimming the candidate pool before the LLM evaluation. Together they embody the paper's answer to

What would settle it

Re-run the human-agreement analysis after averaging each app's three metric scores into one app-level score and then computing Spearman correlation across the six apps; if the correlation is no longer significant, the claimed consistency rests on counting the same app and human panel multiple times.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that LLM app quality can be assessed automatically by first classifying each app into a functional taxonomy, then using cheap static indicators to remove low-quality candidates, and finally letting an LLM generate evaluation metrics, scoring rubrics, and test questions that fit the app's specific scenario. The generated metrics are not fixed; they adapt to each app's label, and the evaluator assigns 1–5 scores for content quality plus a separate response-efficiency score, merged with 0.8/0.2 weights. The result is a per-app quality score and a structured evaluation report. LaQual's automated scores show moderate-to-strong rank agreement with

Load-bearing premise

The load-bearing premise is that the 18 scored items (six apps times three metrics) can be treated as independent evidence; the three metrics for one app come from the same app and the same human raters, so averaging per app leaves six independent observations and the reported statistical significance may not be real.

Editorial extensions

If this is right

  • App stores could run LaQual at onboarding and at regular review intervals, publishing quality sub-scores alongside popularity counts to counter ranking manipulation.
  • Developers would receive scenario-specific weakness reports (e.g., legal citation accuracy) that point to concrete fixes before release.
  • Users could compare apps with structured reports instead of star ratings and fragmented reviews, cutting cognitive load in discovery.
  • The pipeline is designed to be reusable across new categories because evaluation metrics are generated per label rather than curated per domain.
  • Because the final score combines content quality (weight 0.8) and response efficiency (0.2), stores can tune the weights to their own priorities.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the same label-to-metric generation were run on a third scenario (say, coding help), the two-scenario Spearman result would be a testable generalization claim; the paper does not report such a cross-check.
  • Adding retrieval-based fact-checking to the content-quality stage would directly address the paper's own acknowledged blind spot (e.g., recommending a restaurant that has closed), without changing the rest of the pipeline.
  • The time-weighted engagement filter could serve as a lightweight, continuous quality-monitoring signal for stores that do not want to pay LLM evaluation costs on every update.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper presents LaQual, a three-stage framework for automated quality evaluation of LLM applications in app stores. Stage 1 uses an LLM-based annotator to generate labels and maps apps to a three-level taxonomy; Stage 2 computes time-weighted user-engagement and functional-capability indicators and applies category-specific thresholds to filter candidates; Stage 3 dynamically generates scenario-specific metrics, scoring criteria, and tasks, then scores app responses with an LLM-as-judge (content quality) and a response-efficiency lookup table, combined as Score = 0.8*S_CQ + 0.2*S_RP. The authors evaluate on AppBuilder in two scenarios (legal consulting and travel planning), reporting Spearman correlations between LaQual scores and average human ratings (rho=0.62, p=0.006; rho=0.60, p=0.009), candidate-pool reductions of 66.7% and 81.3%, and a 12-participant user study in which LaQual beats the baseline on decision confidence, comparison efficiency, and report value. The paper positions LaQual as a scalable, user-centric alternative to static store ranking.

Significance. LLM app stores are a new and fast-growing distribution paradigm, and a general, scenario-adaptive evaluation framework is a timely contribution. The paper's design has several strengths: the evaluation metrics and tasks are generated per scenario, the LLM judge is run with deterministic settings (temperature=0), the code is publicly available, and the multi-stakeholder implications section shows awareness of deployment concerns. If the reported results were valid, LaQual would be a practically useful tool for app-store quality screening. However, the headline quantitative claim—high consistency with human judgment—rests on a correlation analysis over 18 non-independent items per scenario (six apps x three metrics), so the effective sample size for the main comparison is six. In addition, the static filtering thresholds in Table 3 are directly contradicted by the shortlisted-app indicator values in Table 5. These issues, combined with multiple empirically tuned free parameters on the same dataset, mean the current evidence does not support the paper's central claims.

major comments (4)
  1. [§4.1.3, Fig. 7] The Spearman correlations are computed over 18 evaluation items per scenario ('six apps, three metrics each'). The three metrics per app are scored by the same human panel for the same app and are likely correlated; the actual independent units are the six apps. Re-computing with app-level (averaged) scores gives n=6 per scenario, for which a Spearman rho of about 0.6 has a two-sided p-value around 0.24, not the reported 0.006/0.009. The abstract's 'high consistency' claim and the Answer to RQ1 therefore rely on pseudoreplication. Please supply a cluster-robust or app-level analysis, report confidence intervals, and temper the claim accordingly, or collect additional apps for the human-rating study.
  2. [Table 3 vs. Table 5, §4.1.2] Table 3 sets admission thresholds of ≥20 for Views, Interactions, Favorites, and Copies for all categories. Table 5, which summarizes the shortlisted apps that passed the static filter, lists Favorites values between 0 and 4 and Copies between 0 and 1 for every retained app. This is a direct contradiction: either the thresholds are not applied as stated, or the numbers in Table 5 are not the quantities compared to the thresholds. If the latter, the table needs a clear label and the weighted values should be shown. Without this, the static filtering results and RQ2's pool-reduction claims are not reproducible.
  3. [§3.1.1, §3.2.2, §3.2.3, §3.3.3] The framework has several free parameters: text2vec label-similarity threshold (0.7), decay base beta (0.99), static admission thresholds (Table 3), composite weights alpha/beta (0.8/0.2), and response-efficiency cutoffs (Table 4). All are described as 'empirically determined' from the same AppBuilder dataset used for the main evaluation; no held-out data, cross-validation, or sensitivity analysis is provided. This is especially problematic for RQ2 because the pool-reduction percentages are the direct output of the threshold mechanism. Please add robustness checks (e.g., varying thresholds, beta, and weights) or validate on a separate platform/period.
  4. [§4.1.1, §4.1.3] Human-evaluation details are under-reported: the same 28 raters evaluated the six shortlisted apps with the same metrics, but no inter-rater reliability statistic (e.g., ICC or Fleiss' kappa) is given. If the human gold standard is an average of highly dispersed ratings, the Spearman comparison is not meaningful. In addition, the paper does not report how many raters scored each app or whether the panel is identical across the two scenarios. Please report rating distributions, agreement metrics, and rater-level variance.
minor comments (6)
  1. [§3.2.2, Eqs. (2)-(3)] The term 'standardization' is misleading; no standardization is performed, only multiplication of a quarterly mean by a decay factor. Also clarify that beta=0.99 gives a 1% per-quarter discount, not '99% of the previous period' in the running text.
  2. [Table 5] The column header 'La1 1' appears to be a typo for 'La1', and the note for 'Ta12' is ambiguous. Please clean up the table header and footnotes.
  3. [Abstract and §4.1.3] The abstract says 'high consistency' while the body says 'moderately strong' correlation. Use one consistent, calibrated descriptor, especially because the correlation values are around 0.6.
  4. [§4.3.3, Table 8] Report test statistics, degrees of freedom, p-values, and effect sizes for the paired t-tests. The statement 'p<0.001' in Fig. 8 lacks supporting statistical detail, and with n=12 the analysis is sensitive to outliers.
  5. [§7.4] Reference [10] is cited as 'Belz and Reiter', but the reference list entry [10] is Dong et al., 'A survey of natural language generation'. The citation is mismatched and should be corrected.
  6. [Fig. 8 caption] The caption contains corrupted Unicode sequences ('/uni00000027/...'), likely an extraction artifact. Please fix the source file.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the main human-judgment comparison is an external benchmark, and fitted parameters (thresholds, decay base, weights, text2vec cutoff) are empirical settings, not outputs predicted from themselves.

full rationale

Walked the claimed derivation chain. LaQual's stages are: (1) LLM-based labeling and hierarchical classification; (2) static indicator filtering with time-weighted engagement and functional thresholds; (3) dynamic LLM-generated metrics/tasks and scoring. The headline claim, high consistency with human judgments, is tested in RQ1 by correlating LaQual content scores with average ratings from a panel of 28 human raters, including domain experts (Section 4.1.3). That human benchmark is external to LaQual's own scoring pipeline, so the central claim does not reduce to an internal fit. Parameters such as the decay base (β=0.99), the text2vec similarity cutoff (0.7), the admission thresholds (Table 3), and composite weights (α=0.8, β=0.2) are explicitly described as empirically determined settings (e.g., 'the threshold of 0.7 is empirically determined', 'based on empirical analysis', 'determined through multiple rounds of empirical validation'), not as predicted outcomes; their validation is a calibration/robustness concern rather than circularity. The taxonomy design cites Su et al. [41], an empirical study by overlapping authors, but this is a background input to classification, not a load-bearing uniqueness theorem, and the classification stage is not the source of the paper's main claim. The 66.7%–81.3% pool reduction is a direct arithmetic consequence of applying the chosen thresholds and is presented as such, with manual review of excluded apps as separate evidence. A separate internal inconsistency exists between Table 3 thresholds (e.g., Favorites ≥20) and Table 5 shortlisted apps (Favorites 0–4), and the statistical comparison treats 18 items as independent despite only six apps per scenario; these are correctness/validity threats, not circular reductions. No quoted equation or construction shows an output being defined as its own input, so no circular step is identified.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The framework is largely defined by manually set constants (thresholds, decay, weights, similarity cutoff) and assumes LLM-as-a-judge reliability plus platform-indicator validity. The external anchor is a small human-rating study whose statistical basis is questionable.

free parameters (5)
  • text2vec label similarity threshold = 0.7
    Used to accept or reject LLM-generated labels; selected empirically by the authors based on label quality and iteration efficiency (Section 3.1.1).
  • time decay base beta = 0.99
    Discounts historical engagement per quarter; set manually, affects all static engagement scores (Section 3.2.2).
  • static indicator admission thresholds = Views/Interactions/Favorites/Copies >=20; Plugins >=0/2/0; Knowledge Bases >=1/0/0; Built-in Models >=1/1/1
    Determined through multiple rounds of empirical validation on AppBuilder data; Table 3. Contradicted by Table 5.
  • composite weights alpha and beta = 0.8/0.2
    Weights for content vs response performance in the final score, chosen by empirical analysis (Section 3.3.3).
  • response efficiency thresholds = 25/20/15/10 tokens per second
    Lookup table mapping throughput to 5-1 scores; thresholds are arbitrary (Section 3.3.2, Table 4).
assumptions (4)
  • domain assumption An LLM can reliably grade open-ended response quality using generated criteria (LLM-as-a-judge validity).
    Used throughout Stage 3; the authors acknowledge in Section 6 that LLMs carry intrinsic bias and cannot fact-check, so the grading reliability is assumed rather than demonstrated.
  • domain assumption Static platform indicators (views, interactions, favorites, copies, plugins, knowledge bases, built-in models) reflect app quality.
    Section 3.2.1 selects these as proxies for popularity and capability with no validation that they correlate with user-perceived quality.
  • domain assumption The three-category, ten-subcategory, 62-tag taxonomy covers the LLM app ecosystem.
    Section 3.1.2 derives the taxonomy from the authors' own earlier GPT Store analysis, not from an external standard.
  • domain assumption Human ratings from 28 self-selected raters are a valid ground truth.
    Section 4.1.3 uses averaged human scores as the benchmark; no inter-rater reliability is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LaQual: An Automated Framework for LLM App Quality Evaluation." pith.science (2026). https://pith.science/paper/5X4BGIPO

@misc{pith2026250818636,
  author       = {Pith},
  title        = {Pith review of: LaQual: An Automated Framework for LLM App Quality Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5X4BGIPO}},
  note         = {Machine review of arXiv:2508.18636}
}
read the original abstract

Representing a new paradigm in software distribution, LLM app stores are rapidly emerging, offering users diverse choices for content generation, coding assistance, education, and more. However, current ranking and recommendation mechanisms in LLM app stores predominantly rely on static metrics, such as user interactions and favorites, making it challenging for users to efficiently identify high-quality apps. At the same time, current academic research focuses on specific vertical fields and lacks a general, automated evaluation framework applicable to the diverse LLM app ecosystem. To address the above challenges, we present LaQual, an automated framework for LLM app quality evaluation. LaQual integrates three key stages: (1) LLM app labeling and hierarchical classification for precise scenario mapping; (2) static indicator evaluation using time-weighted user engagement and functional capability indicators to filter low-quality apps; and (3) dynamic scenario-adapted evaluation, where an LLM generates scenario-specific evaluation metrics, scoring criteria, and tasks for comprehensive quality evaluation. Experiments on a mainstream LLM app store demonstrate the effectiveness of LaQual. Its automated scores show high consistency with human judgments. Through effective screening, LaQual can reduce the candidate LLM app pool by 66.7% to 81.3%. User studies further validate its significant outperformance over baseline systems, particularly in comparison efficiency (mean 5.45 vs. 3.30) and value of explanatory information (4.75 vs. 2.25). These results demonstrate that LaQual provides a scalable, objective, and user-centric solution for high-quality discovery and recommendation of LLM apps in real-world scenarios.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

58 extracted references · 37 canonical work pages

  1. [1]

    Xuechunzi Bai, Angelina Wang, Ilia Sucholutsky, and Thomas L Griffiths. 2024. Measuring implicit bias in explicitly unbiased large language models. arXiv preprint arXiv:2402.04105 (2024)

    arXiv 2024

  2. [2]

    baidu. 2024. Baidu app center. https://console.bce.baidu.com/ai_apaas/appCenter

    work page 2024

  3. [3]

    Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. 65–72

    2005

  4. [4]

    ByteDance. 2024. Cici. https://www.ciciai.com/chat/

    work page 2024

  5. [5]

    ByteDance. 2024. Coze. https://www.coze.com/

    work page 2024

  6. [6]

    Yong Cao, Li Zhou, Seolhwa Lee, Laura Cabello, Min Chen, and Daniel Hershcovich. 2023. Assessing cross-cultural alignment between ChatGPT and human societies: An empirical study. arXiv preprint arXiv:2303.17466 (2023)

    arXiv 2023

  7. [7]

    Hoi, Shaohua Li, and Xiaokui Xiao

    Ning Chen, Steven C.H. Hoi, Shaohua Li, and Xiaokui Xiao. 2016. Mobile App Tagging. InProceedings of the Ninth ACM International Conference on Web Search and Data Mining (San Francisco, California, USA) (WSDM ’16). Association for Computing Machinery, New York, NY, USA, 63–72. https://doi.org/10.1145/2835776.2835812

    arXiv 2016

  8. [8]

    Cheng-Han Chiang and Hung-yi Lee. 2023. Can large language models be an alternative to human evaluations? arXiv preprint arXiv:2305.01937 (2023)

    arXiv 2023

Show all 58 references
  1. [9]

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. 2024. Chatbot arena: An open platform for evaluating llms by human preference. In Forty-first International Conf...

    2024

  2. [10]

    Chenhe Dong, Yinghui Li, Haifan Gong, Miaoxin Chen, Junxin Li, Ying Shen, and Min Yang. 2022. A survey of natural language generation. Comput. Surveys 55, 8 (2022), 1–38

    2022

  3. [11]

    GPTStore.AI. 2024. GPTStore.AI. https://gptstore.ai/gpts

    2024

  4. [12]

    Christian Grévisse. 2024. LLM-based automatic short answer grading in undergraduate medical education. BMC Medical Education 24, 1 (2024), 1060

    2024

  5. [13]

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. 2024. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594 (2024)

    2024 arXiv

  6. [14]

    Neel Guha, Julian Nyarko, Daniel Ho, Christopher Ré, Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, et al. 2023. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. Advances in ...

    2023

  7. [15]

    Yue Guo, Zian Xu, and Yi Yang. 2023. Is chatgpt a financial expert? evaluating language models on financial natural language processing. arXiv preprint arXiv:2310.12664 (2023)

    2023 arXiv

  8. [16]

    Xinyi Hou, Yanjie Zhao, and Haoyu Wang. 2025. On the (In)Security of LLM App Stores . In 2025 IEEE Symposium on Security and Privacy (SP) . IEEE Computer Society, Los Alamitos, CA, USA, 317–335. https://doi.org/10.1109/SP61157. 2025.00117

    2025

  9. [17]

    Xinyi Hou, Yanjie Zhao, Shenao Wang, and Haoyu Wang. 2024. GPTZoo: A Large-scale Dataset of GPTs for the Research Community. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (Sacramento, CA, USA) (ASE ’24). Association for Computin...

    2024

  10. [18]

    GPTs Hunter. 2024. GPTs Hunter. https://www.gptshunter.com/. , Vol. 1, No. 1, Article . Publication date: September 2025. LaQual: A Novel Framework for Automated Evaluation of LLM App Quality 23

    2024

  11. [19]

    Argyro Kavadella, Marco Antonio Dias Da Silva, Eleftherios G Kaklamanos, Vasileios Stamatopoulos, Kostis Gian- nakopoulos, et al. 2024. Evaluation of ChatGPT’s real-life implementation in undergraduate dental education: mixed methods study. JMIR Medical Education 10, 1 (2024), e51344

    2024

  12. [20]

    Jack Krolik, Herprit Mahal, Feroz Ahmad, Gaurav Trivedi, and Bahador Saket. 2024. Towards leveraging large language models for automated medical q&a evaluation. arXiv preprint arXiv:2409.01941 (2024)

    2024 arXiv

  13. [21]

    Haitao Li, Junjie Chen, Jingli Yang, Qingyao Ai, Wei Jia, Youfeng Liu, Kai Lin, Yueyue Wu, Guozhi Yuan, Yiran Hu, et al. 2024. Legalagentbench: Evaluating llm agents in legal domain. arXiv preprint arXiv:2412.17259 (2024)

    2024 arXiv

  14. [22]

    Haitao Li, You Chen, Qingyao Ai, Yueyue Wu, Ruizhe Zhang, and Yiqun Liu. 2024. Lexeval: A comprehensive chinese legal benchmark for evaluating large language models. Advances in Neural Information Processing Systems 37 (2024), 25061–25094

    2024

  15. [23]

    Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74–81

    2004

  16. [24]

    M. H. Maqbool, Umar Farooq, Adib Mosharrof, A. B. Siddique, and Hassan Foroosh. 2023. MobileRec: A Large Scale Dataset for Mobile Apps Recommendation. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (Taipei, Ta...

    2023

  17. [25]

    Zabir Al Nazi and Wei Peng. 2024. Large language models in healthcare and medical domain: A review. In Informatics, Vol. 11. MDPI, 57

    2024

  18. [26]

    PR Newswire. 2024. FlowGPT, The Open Ecosystem AI Platform, Raises $10M. https://www.prnewswire.com/news- releases/flowgpt-the-open-ecosystem-ai-platform-raises-10m-302070574.html

    2024

  19. [27]

    OpenAI. 2023. Introducing GPTs. https://openai.com/blog/introducing-gpts

    2023

  20. [28]

    OpenAI. 2024. GPT Store Search Change Step Back. Available online. https://community.openai.com/t/gptstore- search-change-step-back/600782, Accessed: 2025-08-12

    2024

  21. [29]

    OpenAI. 2024. Has anyone else noticed changes in the GPTs search ranking algorithm lately? Available on- line. https://community.openai.com/t/has-anyone-else-noticed-changes-in-the-gpts-search-ranking-algorithm- lately/1258050, Accessed: 2025-08-12

    2024

  22. [30]

    OpenAI. 2024. Ranking of top 500 GPTs by # conversations. Available online. https://community.openai.com/t/ranking- of-top-500-gpts-by-conversations/578607, Accessed: 2025-08-12

    2024

  23. [31]

    OpenAIapts. 2023. ChatGPT. https://chatgpt.com/gpts

    2023

  24. [32]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems 35...

    2022

  25. [33]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics . 311–318

    2002

  26. [34]

    Quora. 2024. Poe. https://poe.com

    2024

  27. [35]

    Jaromir Savelka. 2023. Unlocking practical applications in legal domain: Evaluation of gpt for zero-shot semantic annotation of legal texts. In Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law . 447–451

    2023

  28. [36]

    Patrick Schober, Christa Boer, and Lothar A Schwarte. 2018. Correlation coefficients: appropriate use and interpretation. Anesthesia & analgesia 126, 5 (2018), 1763–1768

    2018

  29. [37]

    Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. BLEURT: Learning robust metrics for text generation. arXiv preprint arXiv:2004.04696 (2020)

    2020 arXiv

  30. [38]

    Shreya Shankar, JD Zamfirescu-Pereira, Björn Hartmann, Aditya Parameswaran, and Ian Arawjo. 2024. Who validates the validators? aligning llm-assisted evaluation of llm outputs with human preferences. In Proceedings of the 37th Annual ACM Symposium on User Interface Software an...

    2024

  31. [39]

    SimilarWeb. 2025. Website Analysis: huggingface.co. Available online. https://pro.similarweb.com/, Accessed: 2025-08-12

    2025

  32. [40]

    SimilarWeb. 2025. Website Analysis: poe.com. Available online. https://pro.similarweb.com/, Accessed: 2025-08-12

    2025

  33. [41]

    Dongxun Su, Yanjie Zhao, Xinyi Hou, Shenao Wang, and Haoyu Wang. 2024. Gpt store mining and analysis. arXiv preprint arXiv:2405.10210 (2024)

    2024 arXiv

  34. [42]

    Dongxun Su, Yanjie Zhao, Xinyi Hou, Shenao Wang, and Haoyu Wang. 2025. GPT Store Mining and Analysis

    2025

  35. [43]

    Tencent. 2024. Tencent agent shop. https://yuanqi.tencent.com/agent-shop

    2024

  36. [44]

    Jiaqi Wang, Huan Zhao, Zhenyuan Yang, Peng Shu, Junhao Chen, Haobo Sun, Ruixi Liang, Shixin Li, Pengcheng Shi, Longjun Ma, et al. 2024. Legal evalutions and challenges of large language models. arXiv preprint arXiv:2411.10137 (2024)

    2024 arXiv

  37. [45]

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al . 2024. Mmlu-pro: A more robust and challenging multi-task language understanding , Vol. 1, No. 1, Article . Publication date: September 2025...

    2024

  38. [46]

    Qiuhong Wei, Zhengxiong Yao, Ying Cui, Bo Wei, Zhezhen Jin, and Ximing Xu. 2024. Evaluation of ChatGPT-generated medical responses: a systematic review and meta-analysis. Journal of biomedical informatics 151 (2024), 104620

    2024

  39. [47]

    Yuhao Wu, Evin Jaff, Ke Yang, Ning Zhang, and Umar Iqbal. 2025. An In-Depth Investigation of Data Collection in LLM App Ecosystems (WWW ’25). Association for Computing Machinery

    2025

  40. [48]

    Changrong Xiao, Sean Xin Xu, Kunpeng Zhang, Yufang Wang, and Lei Xia. 2023. Evaluating reading comprehension exercises generated by LLMs: A showcase of ChatGPT in education applications. In Proceedings of the 18th workshop on innovative use of NLP for building educational appl...

    2023

  41. [49]

    Qianqian Xie, Weiguang Han, Zhengyu Chen, Ruoyu Xiang, Xiao Zhang, Yueru He, Mengxi Xiao, Dong Li, Yongfu Dai, Duanyu Feng, et al. 2024. Finben: A holistic financial benchmark for large language models. Advances in Neural Information Processing Systems 37 (2024), 95716–95743

    2024

  42. [50]

    Yinglin Xie, Xinyi Hou, Yanjie Zhao, Kai Chen, and Haoyu Wang. 2025. LLM App Squatting and Cloning. InProceedings of the 33rd ACM International Conference on the Foundations of Software Engineering . 64–74

    2025

  43. [51]

    Chuan Yan, Bowei Guan, Yazhi Li, Mark Huasong Meng, Liuhuo Wan, and Guangdong Bai. 2025. Understanding and Detecting File Knowledge Leakage in GPT App Ecosystem. In Proceedings of the ACM on Web Conference 2025 (Sydney NSW, Australia) (WWW ’25). Association for Computing Machi...

    2025

  44. [52]

    Chuan Yan, Ruomai Ren, Mark Huasong Meng, Liuhuo Wan, Tian Yang Ooi, and Guangdong Bai. 2024. Exploring ChatGPT App Ecosystem: Distribution, Deployment and Security. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (Sacramento, CA,...

    2024

  45. [53]

    Jian Yang, Jiaxi Yang, Ke Jin, Yibo Miao, Lei Zhang, Liqun Yang, Zeyu Cui, Yichang Zhang, Binyuan Hui, and Junyang Lin. 2024. Evaluating and aligning codellms on human preference. arXiv preprint arXiv:2412.05210 (2024)

    2024 arXiv

  46. [54]

    Jing Yao, Xiaoyuan Yi, and Xing Xie. 2024. CLAVE: An Adaptive Framework for Evaluating Values of LLM Generated Responses. In Advances in Neural Information Processing Systems , A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37. C...

    2024

  47. [55]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 (2019)

    2019 arXiv

  48. [56]

    Zejun Zhang, Li Zhang, Xin Yuan, Anlan Zhang, Mengwei Xu, and Feng Qian. 2024. A First Look at GPT Apps: Landscape and Vulnerability. arXiv:2402.15105 [cs.CR] https://arxiv.org/abs/2402.15105

    2024 arXiv

  49. [57]

    Yanjie Zhao, Xinyi Hou, Shenao Wang, and Haoyu Wang. 2025. LLM App Store Analysis: A Vision and Roadmap. ACM Trans. Softw. Eng. Methodol. 34, 5, Article 125 (May 2025), 25 pages. https://doi.org/10.1145/3708530

    2025 doi

  50. [58]

    Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Lei Shen, Zihan Wang, Andi Wang, Yang Li, et al. 2023. Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x. In Proceedings of the 29th ACM SIGKDD Conference on Knowle...

    2023

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.