REVIEW 3 major objections 7 minor 1 cited by
Tracking GPTs Third Party Service: Automation, Analysis, and Insights
T0 review · 3 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read GPTs-ThirdSpy is an automated framework that extracts the third-party services and privacy policies a GPT declares, directly from the GPT Store's privacy settings panel.
desk verdict A useful small empirical tool paper whose headline numbers don't add up; worth a revision, not a desk reject. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is a coordinate-click extraction loop: GPTs-ThirdSpy constructs a chatgpt.com URL from a GPT's unique 9-character GizmoID, uses cliclick to click a precomputed absolute screen coordinate for the dropdown button, then clicks the precomputed coordinate of the Privacy settings entry, and finally reads the pop-up window that lists each declared third-party domain and privacy policy link. The presence of that privacy-policy button is itself the detection signal: since OpenAI requires a privacy policy for GPTs with Actions, the click sequence both identifies action-based GPTs and extracts their declared external-service metadata. The framework also categorizes GPTs by metadata into prompt-based, knowledge-based, and action-based types, with only action-based GPTs interacting with third-party services.
What would settle it
Re-run GPTs-ThirdSpy on a different macOS machine, with a different screen resolution or browser zoom, or after an OpenAI UI update; if the privacy panel fails to open or extracted domain lists change for unchanged GPTs, the claimed real-time precise retrieval is environment-specific. Separately, recompute the headline result from the paper's Figure 2 counts: 409 + 79 + 8 + 2 + 2 = 500, which means only 91 GPTs have third-party domains, so if the correct number is 109 GPTs rather than 109 domains, the reported demonstration is internally inconsistent.
Extended reading notes
Core claim
On its own terms, the paper's central claim is that OpenAI's anti-automation defenses on GPT interaction pages can be bypassed by simulating user input at precomputed screen coordinates, using AppleScript and the macOS command-line tool cliclick, rather than by DOM-based automation such as Selenium or Puppeteer. Because OpenAI requires any GPT that uses Actions to display a Privacy settings button, the workflow clicks that button's precomputed coordinate and parses the pop-up listing each third-party domain with its corresponding privacy policy link. The paper reports that among the 500 most popular GPTs, 409 use no third-party domains, 79 use one, and 12 use two or more, yielding 91 GPTs with a total of 109 external domains; the abstract and introduction state the result as 109 GPTs, which conflicts with the body's figures. For those 109 domains, 92 privacy policy links were accessible, while the rest were broken links, placeholders, homepages, timeouts, or server errors. The paper frames the tool and the resulting privacy-setting dataset as its contribution: a real-time and more reliable foundation for privacy compliance audits, security evaluations, and future research on the GPT app ecosystem.
Load-bearing premise
The pipeline assumes the GPT Store page layout is fixed enough that precomputed absolute screen coordinates for the dropdown button and the Privacy settings button always line up with the click target on a particular macOS screen; if OpenAI changes the layout, the window size differs, or the buttons move, the clicks miss and no data is extracted.
Editorial extensions
If this is right
- Researchers can obtain current third-party service metadata for GPTs directly from the GPT Store without waiting for third-party trackers to update their records.
- Privacy compliance audits become feasible at scale because the extracted data exposes which declared privacy policy links are broken, placeholder URLs, homepages, timeouts, or server errors.
- The three-way categorization separates action-based GPTs, the only class that calls external APIs, from prompt- and knowledge-based GPTs, giving analysts a structured way to scope privacy and security risk.
- Repeated runs of GPTs-ThirdSpy can track when a GPT adds, removes, or swaps an external service, supporting real-time monitoring of changes in data-sharing behavior.
Reading between the lines
- The coordinate-loop design ties every extraction run to one measured screen layout on a particular macOS setup; the paper gives no detail on how the coordinates were chosen or recalibrated, so 'real-time' is best read as 'as fast as the GUI allows on a fixed configuration' rather than as a durable API-like guarantee.
- Because a GPT without a declared Actions section shows no Privacy settings button, the tool only sees third-party integrations that developers disclose, so the resulting dataset measures declared integrations rather than actual network activity.
- The same pattern of precomputed coordinates, simulated clicks, and pop-up parsing could be re-parameterized for other web platforms that hide privacy or security metadata behind GUI interactions, but each new target would need its own coordinate calibration.
- Before relying on the headline counts, a corrected dataset should reconcile the abstract's 109 GPTs with the body's 91 GPTs using 109 domains; the paper leaves that numerical gap unresolved.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces GPTs-ThirdSpy, an automation framework that uses AppleScript and cliclick to simulate mouse clicks on the ChatGPT/GPT Store interface, extract the third-party domain names and privacy-policy links shown in each GPT's privacy settings, and organize them into a dataset. The authors categorize GPTs into prompt-based, knowledge-based, and action-based types, then demonstrate the tool on the 500 most popular GPTs from GPTsHunter. They report that 91 GPTs use at least one third-party domain (109 domains in total), 409 use none, and they analyze the accessibility of the associated privacy-policy links (92 accessible, with the remainder broken, homepage-only, timing out, or returning server errors). The paper claims that the tool provides real-time, precise retrieval of GPTs' interactions with external services and discusses applications in privacy compliance, service monitoring, and penetration testing.
Significance. If the extraction accuracy is confirmed, the paper would provide a useful tool and dataset for researchers studying third-party integrations in the GPT ecosystem, addressing a real timeliness problem in existing third-party GPT directories. The paper ships a source-code availability statement, and the headline measurement (91 of 500 popular GPTs with at least one third-party domain, across 109 domains) is a concrete, potentially reusable artifact. The paper is empirical rather than theoretical, so its value rests on the reliability, reproducibility, and internal consistency of the measurement; these are currently not fully established. The arithmetic in Figure 2 is internally consistent, which is a positive sign, but the accompanying text and tables contain unresolved count conflicts.
major comments (3)
- [Abstract and §4.1] The headline number is inconsistent across the paper. The Abstract and Introduction state that the tool 'detects 109 GPTs relying on third-party services' within the 500-GPT sample, but Figure 2 reports 79 GPTs with one domain, 8 with two, 2 with three, and 2 with four domains, which sums to 91 GPTs. Section 4.2 correctly refers to '109 domains associated with third-party services.' The paper cannot simultaneously claim 109 GPTs and 109 domains; the counts and all derived statements (including the abstract and conclusion) need to be reconciled and re-verified.
- [§3 and §4.1] The central claim of 'real-time and precise retrieval' is not supported by any end-to-end accuracy measurement. The workflow relies on precomputed absolute screen coordinates for the GPT dropdown button and the 'Privacy settings' button (Section 3, Figure 1), yet the paper reports no precision/recall against manual inspection, no comparison with an independent source for the 500 GPTs, and no failure rate for the automated runs. A missed click due to layout shift, CAPTCHA, or delayed rendering would be recorded as 'no third-party services,' inflating the 409 count in Figure 2. To support the claimed reliability, the authors should add a validation experiment with ground truth, or at least compare the 91 detected GPTs against the number of GPTs whose metadata (Section 2) indicates that they have actions.
- [§4.2, Table 2] The privacy-policy counts do not add up. Table 2 reports 92 accessible, 9 broken, 5 homepage-only, 2 timeout, and 2 server-error links, which sums to 110, while the text states that there are 109 domains associated with third-party services. The prose also says '9 links are invalid' and then separately lists 5 homepage-only, 2 timeout, and 2 server-error cases, implying 18 non-accessible links; this should be 17 if the total is 109. The unit of analysis (GPTs versus domains) must be clarified and the table and text must be made consistent.
minor comments (7)
- [Abstract] The phrase 'third-party integrate to GPTs' is ungrammatical; it should be 'third-party integration into GPTs.'
- [§1] There are several language issues: 'In response this trend' is missing 'to'; 'OpenAI introduce' should be 'OpenAI introduced'; and 'from multiple perspective' should be 'from multiple perspectives.'
- [§1] The future-work sentence contains a duplicated phrase: 'supporting privacy compliance audits, supporting privacy compliance audits.'
- [§2] The text says 'enhance functionally' but should say 'enhance functionality.' In addition, the set-based notation for GPT metadata is hard to parse; a table or a few concrete examples would improve readability.
- [§3] The sentence 'the next step it to retrieve' should be 'the next step is to retrieve.' Also, Figure 1's text 'Privacy setting returns True' is unclear; the text should explain what condition is being tested and how the boolean result is obtained.
- [§4.2] The phrase 'meaningful full and enforceable privacy data' should be 'meaningful, full, and enforceable privacy data.'
- [General] The paper uses 'GPTs' for both singular and plural throughout. Standardizing on 'GPT' for the singular form and 'GPTs' for the plural would improve clarity.
Circularity Check
No circularity: the tool's extraction is an empirical measurement specified independently of its inputs.
full rationale
GPTs-ThirdSpy is an empirical tool-and-measurement paper rather than a derivation. The claimed result is that the tool retrieves third-party domain names and privacy-policy links from GPT Store pages in real time. Its workflow is fully specified in Section 3: construct a URL from a GizmoID, use AppleScript/cliclick to click a dropdown and a 'Privacy settings' button, then parse the displayed domain and policy data. None of these steps defines its output in terms of its input; the output is observed from the live UI, not computed from a fitted parameter. The GPT-type definitions in Equations (1)-(3) are nominal taxonomies, and the statement that only Action-based GPTs interact with third-party services follows from the definitions, but the later detection of third-party services is not derived from those equations — it is scraped from the store's privacy-settings dialog. Using the presence of the 'Privacy settings' button as a proxy for Actions may produce false negatives and is a validity concern, not a circularity concern. The paper's self-citations (e.g., [23], [25]) document prior context and prior use of AppleScript for ChatGPT automation, but the present tool's behavior is specified independently in Section 3, its code is released on GitHub, and the central measurement does not reduce to any cited result. The abstract/body count discrepancy (109 GPTs vs. 91 GPTs with 109 domains) is an internal-consistency error in data reporting, not circular reasoning. Because the central claim is a measurement of external store data with no fitted-input/prediction structure, the circularity score is 0.
Assumptions & free parameters
free parameters (1)
- cliclick screen coordinates for dropdown and privacy buttons =
e.g., (330,153) and (400,290)
assumptions (3)
- domain assumption OpenAI requires GPTs with Actions to provide a privacy policy, and a 'Privacy settings' button appears in the GPT dropdown.
- domain assumption The 500 most popular GPTs from GPTsHunter serve as a representative sample of the GPT Store.
- ad hoc to paper macOS AppleScript and cliclick can simulate clicks without triggering CAPTCHA on the GPT Store.
Cite this review
Pith. "Pith review of Tracking GPTs Third Party Service: Automation, Analysis, and Insights." pith.science (2026). https://pith.science/paper/N4Y7U6SU
@misc{pith2026250617315,
author = {Pith},
title = {Pith review of: Tracking GPTs Third Party Service: Automation, Analysis, and Insights},
year = {2026},
howpublished = {\url{https://pith.science/paper/N4Y7U6SU}},
note = {Machine review of arXiv:2506.17315}
}
read the original abstract
ChatGPT has quickly advanced from simple natural language processing to tackling more sophisticated and specialized tasks. Drawing inspiration from the success of mobile app ecosystems, OpenAI allows developers to create applications that interact with third-party services, known as GPTs. GPTs can choose to leverage third-party services to integrate with specialized APIs for domain-specific applications. However, the way these disclose privacy setting information limits accessibility and analysis, making it challenging to systematically evaluate the data privacy implications of third-party integrate to GPTs. In order to support academic research on the integration of third-party services in GPTs, we introduce GPTs-ThirdSpy, an automated framework designed to extract privacy settings of GPTs. GPTs-ThirdSpy provides academic researchers with real-time, reliable metadata on third-party services used by GPTs, enabling in-depth analysis of their integration, compliance, and potential security risks. By systematically collecting and structuring this data, GPTs-ThirdSpy facilitates large-scale research on the transparency and regulatory challenges associated with the GPT app ecosystem.
Figures
Forward citations
Cited by 1 Pith paper
-
Security study based on the Chatgptplugin system: ldentifying Security Vulnerabilities
A measurement study of the ChatGPT plugin store found widespread manifest exposure, weak API authentication, and metadata inconsistencies, with some vulnerabilities reduced after reporting to OpenAI.
Reference graph
Works this paper leans on
-
[25]
Chuan Yan, Ruomai Ren, Mark Huasong Meng, Liuhuo Wan, Tian Yang Ooi, and Guangdong Bai. 2024. Exploring chatgpt app ecosystem: Distribution, deployment and security. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. 1370–1382
work page 2024
-
[1]
Tracking GPTs Third Party Service: Automation, Analysis, and Insights (GPTs-ThirdSpy Source Code)
2025. Tracking GPTs Third Party Service: Automation, Analysis, and Insights (GPTs-ThirdSpy Source Code). https://github.com/UQ-Trust-Lab/GPTs-ThirdSpy
work page 2025
-
[2]
Apple. 2024. Introduction to AppleScript Language Guide . https: //developer.apple.com/library/archive/documentation/AppleScript/Conceptual/ AppleScriptLangGuide/introduction/ASLR_intro.html
work page 2024
-
[3]
b12. 2024. b12 Official Website. https://www.b12.io/
work page 2024
-
[4]
Carsten Blüm. 2024. cliclick: macOS CLI tool for emulating mouse and keyboard events. https://github.com/BlueM/cliclick
work page 2024
-
[5]
Google. 2024. Puppeteer website. https://pptr.dev/
work page 2024
-
[6]
gptbuilder. 2024. gptbuilder Official Website. https://gptbuilder.tools/
work page 2024
-
[7]
GPTsApp.io. 2024. AI PDF Drive: Chat, Create, Organize . https://gptsapp.io/gpts/ ai-pdf-ai/xf37cpcez
work page 2024
Show all 29 references
-
[8]
Xinyi Hou, Yanjie Zhao, and Haoyu Wang. 2024. On the (in) security of llm app stores. arXiv preprint arXiv:2407.08422 (2024)
2024 arXiv
-
[9]
Jason Huggins. 2024. Selenium website. https://www.selenium.dev/
2024
-
[10]
Umar Iqbal, Tadayoshi Kohno, and Franziska Roesner. 2024. Llm platform security: applying a systematic evaluation framework to openai’s chatgpt plugins. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society , Vol. 7. 611– 623
2024
-
[11]
jobright. 2024. jobright Official Website. https://jobright.ai/
2024
-
[12]
Ningke Li, Yuekang Li, Yi Liu, Ling Shi, Kailong Wang, and Haoyu Wang. 2024. Drowzee: Metamorphic testing for fact-conflicting hallucination detection in large language models. Proceedings of the ACM on Programming Languages 8, OOPSLA2 (2024), 1843–1872
2024
-
[13]
myaidrive. 2024. AI PDF Drive: Chat, Create, Organize . https://chatgpt.com/g/g- V2KIUZSj0-ai-pdf-drive-chat-create-organize Tracking GPTs Third Party Service: Automation, Analysis, and Insights Conference’17, July 2017, Washington, DC, USA
2024
-
[14]
OpenAI. 2023. OpenAI official website. https://openai.com/
2023
-
[15]
OpenAI. 2024. ChatGPT: Verify that you are human . https://community.openai. com/t/verify-that-you-are-human-stop-it/857988
2024
-
[16]
OpenAI. 2024. Introducing the GPT Store . https://openai.com/index/introducing- the-gpt-store/
2024
-
[17]
Liuhuo Wan, Kailong Wang, Kulani Mahadewa, Haoyu Wang, and Guangdong Bai. 2024. Don’t Bite Off More than You Can Chew: Investigating Excessive Permission Requests in Trigger-Action Integrations. In Proceedings of the ACM Web Conference 2024. 3106–3116
2024
-
[18]
Liuhuo Wan, Kailong Wang, Haoyu Wang, and Guangdong Bai. 2024. Is it safe to share your files? an empirical security analysis of google workspace. In Proceedings of the ACM Web Conference 2024 . 1892–1901
2024
-
[19]
Zihan Wang, Zhongkui Ma, Xinguo Feng, Ruoxi Sun, Hu Wang, Minhui Xue, and Guangdong Bai. 2024. Corelocker: Neuron-level usage control. In 2024 IEEE Symposium on Security and Privacy (SP) . IEEE, 2497–2514
2024
-
[20]
WebPilot. 2024. WebPilot Official Website. https://www.webpilot.ai/post-gpts/
2024
-
[21]
Fuman Xie, Yanjun Zhang, Chuan Yan, Suwan Li, Lei Bu, Kai Chen, Zi Huang, and Guangdong Bai. 2022. Scrutinizing privacy policy compliance of virtual personal assistant apps. In Proceedings of the 37th IEEE/ACM international conference on automated software engineering. 1–13
2022
-
[22]
Yinglin Xie, Xinyi Hou, Yanjie Zhao, Kai Chen, and Haoyu Wang. 2024. LLM App Squatting and Cloning. arXiv preprint arXiv:2411.07518 (2024)
2024 arXiv
-
[23]
Chuan Yan, Bowei Guan, Yazhi Li, Mark Huasong Meng, Liuhuo Wan, and Guang- dong Bai. 2025. Understanding and Detecting File Knowledge Leakage in GPT App Ecosystem. In THE WEB CONFERENCE 2025
2025
-
[24]
Chuan Yan, Mark Huasong Meng, Fuman Xie, and Guangdong Bai. 2024. Investi- gating Documented Privacy Changes in Android OS. Proceedings of the ACM on Software Engineering 1, FSE (2024), 2701–2724
2024
-
[26]
Chuan Yan, Fuman Xie, Mark Huasong Meng, Yanjun Zhang, and Guangdong Bai
-
[27]
Zhiqiang Yuan, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, Xin Peng, and Yiling Lou. 2024. Evaluating and improving chatgpt for unit test generation. Proceedings of the ACM on Software Engineering 1, FSE (2024), 1703–1726
2024
-
[28]
Xiaoyu Zhang, Juan Zhai, Shiqing Ma, Qingshuang Bao, Weipeng Jiang, Chao Shen, and Yang Liu. 2025. Unveiling Provider Bias in Large Language Models for Code Generation. arXiv preprint arXiv:2501.07849 (2025)
2025
-
[2024]
Proceedings on Privacy Enhancing Technologies (2024)
On the quality of privacy policy documents of virtual personal assistant applications. Proceedings on Privacy Enhancing Technologies (2024)
2024
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.