Pith. sign in

REVIEW 4 major objections 5 minor 51 references

ASSURE: Metamorphic Testing for AI-powered Browser Extensions

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Automated metamorphic testing surfaces 531 bugs in AI-powered browser extensions, including 202 security vulnerabilities such as prompt injection, at 6.4 times the throughput of manual testing.

desk verdict A well-built, honest test harness for a real problem, but the 531-issue count is an output of its own validators, not an independently verified number. read the letter →

arxiv 2507.05307 v1 pith:AVDFKTHW submitted 2025-07-07 cs.SE cs.AI

Xuanqi Gao , Juan Zhai , Shiqing Ma , Siyi Xie , Chao Shen This is my paper · ORCID
classification cs.SEcs.AI
keywords metamorphictestingAI-poweredbrowserextensionspromptinjectiontestoracleproblemLLMreliabilityautomatedsecurityvulnerabilitiesbehavioralconsistency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that AI-powered browser extensions—summarizers, translators, and writing assistants—share common, detectable failure modes that ordinary testing misses, and that a metamorphic testing framework can surface them automatically. It presents ASSURE, which generates families of related web pages, runs the extension under test on them, and checks relations between outputs rather than exact expected answers. On six popular extensions, ASSURE reported 531 distinct issues, including 202 security vulnerabilities and 102 metamorphic-relation violations. If correct, this makes systematic, automated testing of LLM-based browser extensions practical enough for continuous integration.

What carries the argument

The load-bearing machinery is the metamorphic relation combined with a configurable validation pipeline. Metamorphic testing replaces the test oracle with relations between related inputs and outputs; ASSURE generates two families of relations, semantic equivalence relations and security boundary relations, from parameterized HTML templates. The validation pipeline then checks behavioral consistency and security invariants, flagging cases where the extension processes hidden content, follows injected instructions, or responds differently to structurally distinct but semantically identical inputs.

What would settle it

Manually audit a random sample of the 531 reported issues against the extensions' documented behavior and visible design intentions; if a substantial fraction of the content-alignment and metamorphic-relation violations turn out to be intentional processing of hidden metadata or acceptable variation, the claim that these are genuine defects collapses.

Watch

Extended reading notes

Core claim

The central claim is that LLM-powered browser extensions exhibit systematic, testable failure modes at the intersection of web content and AI processing: they incorporate hidden text into summaries, behave inconsistently across semantically equivalent pages, follow embedded instructions, degrade superlinearly with content size, and misalign summaries with the visual importance of page elements. ASSURE's contribution is an oracle-free validation pipeline that checks metamorphic relations and five validation dimensions—metamorphic, consistency, performance, security, and content alignment—instead of requiring exact output matches. Over six extensions, the framework identified 531 distinct issues, with security vulnerabilities detected in an average of 12.4 minutes per extension and an overall throughput of 5.1 test cases per minute.

Load-bearing premise

The framework's bug reports are only as valid as its normative assumptions about what extensions should do—namely ignore hidden content, produce identical outputs for semantically equivalent inputs, and never follow embedded instructions—even though the paper itself concedes that some findings are false positives because extensions intentionally process hidden metadata.

Editorial extensions

If this is right

  • Developers can integrate ASSURE into development pipelines and continuous integration, catching prompt-injection and hidden-text vulnerabilities within roughly 12 minutes per extension on average.
  • Content summarization extensions are shown to be especially vulnerable to hidden-text manipulation, supporting a visible-only processing principle for extension inputs.
  • Translation extensions suffer DOM-structure-dependent inconsistency, suggesting that semantic equivalence checks should be built into translation tools.
  • Performance testing reveals superlinear scaling in translation extensions, motivating chunked and progressive loading strategies for long content.
  • The modular design of test generators and validators allows the framework to be extended to new extension categories and new bug patterns as they emerge.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The oracle-free validation approach could transfer to other LLM-integrated environments, such as coding assistants or browser agents, where exact output correctness is similarly hard to specify.
  • The hidden-text processing findings imply a class of invisible prompt-injection attacks using CSS-hidden or off-screen elements, which malicious sites could exploit and which ASSURE could help detect.
  • ASSURE could be used for differential testing across similar extensions, comparing how different products handle the same metamorphic input families to reveal relative robustness.
  • The reported 531 issues likely include a non-negligible fraction of false positives, so a manual precision audit on a sample of the reported issues would strengthen the central claim.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes ASSURE, a modular metamorphic testing framework for LLM-powered browser extensions. The framework consists of a test case generation engine, an automated execution framework, and a configurable validation pipeline with five validators (metamorphic relations, consistency, performance, security, and content alignment). The evaluation applies ASSURE to six popular extensions across summarization, translation, and writing assistance, reporting 531 distinct issues (including 202 security vulnerabilities and 102 metamorphic relation violations), a 6.4x testing throughput improvement over manual testing, and detection of critical security issues within an average of 12.4 minutes. The architecture, open-source artifacts, and use of relation-based validation instead of exact output matching are strengths, but the central effectiveness numbers are produced by ASSURE's own validators that encode the paper's normative assumptions, and the paper acknowledges but does not quantify or independently verify the resulting false positives.

Significance. If the effectiveness claims withstand scrutiny, ASSURE would address a real and under-served problem: testing browser extensions whose behavior is non-deterministic and context-dependent because of LLM integration. The modular design is sensible, and the decision to validate via metamorphic relations, consistency, performance, security, and content alignment is a reasonable way to avoid the oracle problem. The public availability of implementation, configurations, and datasets is a concrete plus for reproducibility. However, the reported 531-issue total, the per-category breakdowns, and the comparison with manual testing are not yet anchored to an independent ground truth. The paper's own threat-to-validity section concedes false-positive rates of 8.3% (metamorphic) and 12.7% (content alignment) and acknowledges that extensions may intentionally process hidden metadata, yet no corrected counts or manual triage are provided. The throughput and scalability measurements (Tables 3 and 4) stand more independently, but the central effectiveness claim needs additional validation before the results can be taken as established.

major comments (4)
  1. [Section 4.3, Table 2, Section 6] The central claim that ASSURE identified 531 distinct issues rests entirely on ASSURE's own validators, which encode the paper's normative principles of visible-only processing and output equivalence across structurally different but semantically equivalent pages. Section 6 explicitly concedes false-positive rates of 8.3% for metamorphic relations and 12.7% for content alignment, and notes that some findings stem from the extensions' intentional processing of hidden metadata. The paper does not report how many of the 531 findings were manually triaged, nor does it provide precision values per validator or a corrected issue total. Because every validator flags deviations from ASSURE's own definitions of correct behavior, the bug count is circular without an independent oracle. Please provide a manual triage of all reported issues (or a statistically representative sample, with confidence intervals) and report per-validator precision and a corrected total.
  2. [Section 3.4, Table 2] The consistency and metamorphic validators depend on acceptance thresholds and semantic-similarity metrics that are described only qualitatively (for example, 'adaptive thresholds' and 'progressive thresholds'), with no concrete values, no calibration procedure, and no sensitivity analysis. The issue counts in Table 2, such as 55 consistency issues for Merlin and 31 metamorphic issues for Merlin, are therefore uninterpretable: with sufficiently permissive thresholds, ordinary LLM output variance would be counted as bugs. The paper should report the exact threshold settings used for each validator and extension category, how these thresholds were selected, and how the reported counts change over a range of threshold values.
  3. [Section 4.4] The comparison with manual testing is described only as a 'structured process' with 'three domain experts' following familiarization, exploration, targeted testing, and documentation. The paper does not report the concrete manual test procedure, the number of test cases manually executed, the raw issue counts per expert, the deduplication methodology, or inter-rater agreement. Because the reported 6.4x throughput improvement and the 92% vs. 65% detection-rate comparison both depend on this baseline, the manual protocol must be specified in enough detail to be replicated, and the raw per-expert data should be included.
  4. [Section 3.4, Section 4.3] The security validator classifies 'following embedded instructions' as a vulnerability, but the paper provides no demonstration that any of the 202 reported security issues are actually exploitable (for example, by showing a concrete malicious page that leads to data exfiltration, account compromise, or harmful side effects). The paper itself acknowledges that hidden-metadata processing may be intentional, and some 'prompt injection' behaviors may be benign instruction-following rather than security vulnerabilities. Please provide an exploitability analysis or independent confirmation (e.g., by extension vendors or a security review) for a sample of the security findings, and report the precision of the security validator.
minor comments (5)
  1. [Section 4.1.3] The claim that the generated templates are representative of real webpages is based on 'structural similarity' with 50 popular websites, but no similarity scores or feature distributions are reported; please include the measured values for DOM nesting depth, visible/hidden element ratios, and interactive element distribution.
  2. [Figure 1] In Case 1, the 'Buggy Response' shown looks like a plausible summary rather than a clearly incorrect one because it is not obvious which sentence in the page is hidden and how the visible/hidden boundary is marked; please annotate the figure to make the reported violation explicit.
  3. [Section 4.2] The 'Results' paragraph for RQ1 presents no quantitative findings, despite the research question asking for types of bugs; please include counts or frequencies per category or explicitly reframe RQ1 as a qualitative analysis.
  4. [Throughout] The framework name is inconsistently capitalized as 'Assure' in most of the text but 'ASSURE' in the title and abstract; please unify the spelling.
  5. [Section 7.3] In the related-work text, the framework is referred to as 'PromptBench [45]', but reference [45] is titled 'PromptRobust'; please correct the mismatch between the cited title and the in-text name.

Circularity Check

1 steps flagged · score 5.0 of 10

The 531-issue headline is substantially an output of ASSURE's own validators: a content-alignment 'issue' is by definition a content-alignment flag, and Section 6 concedes that some flags are false positives caused by intentional hidden-metadata processing.

  1. self definitional [Section 3.4 (Configurable Validation Pipeline); Section 4.3 (RQ2, Table 2); Section 6 (Threat to Validity)]
    "flagging cases where extension outputs reflect information from elements that users cannot see. This mechanism ensures that extension behavior aligns with user perception of page content. ... Content Alignment 45 51 – – 15 18 ... We observed false positives primarily in metamorphic relations (8.3% false positive rate) and content alignment (12.7% false positive rate), often stemming from the inherent variability in AI outputs or extensions’ intentional processing of hidden metadata."

    Content alignment 'issues' in Table 2 are, by construction, the count of cases in which ASSURE's content alignment validator fired; the validator fires whenever output reflects hidden elements. The paper reports these counts as discovered bugs ('content alignment problems (45 and 51 issues)') and then, in Section 5, recommends 'visible-only processing' on the basis of those detections. The Section 6 concession that some flags are false positives 'stemming from ... extensions' intentional processing of hidden metadata' shows that the visible-only rule is not the extensions' actual contract.

full rationale

The derivation chain that supports the headline '531 distinct issues' passes through the Configurable Validation Pipeline (Section 3.4). Each reported issue is a case where one of ASSURE's five validators fired. For at least the Content Alignment and Security validators, the firing condition is the validator's own normative rule: content alignment flags output derived from invisible elements; security flags following embedded instructions. The paper then presents these flags as discovered defects and, in Section 5, recommends 'visible-only processing' based on the same rule. Section 6 concedes that some flags are false positives caused by extensions' intentional processing of hidden metadata, so the visible-only rule is not an established contract. This is a self-definitional loop: the oracle defines the bug class, and the bug count is the oracle's output. The framework's architecture, throughput measurements, and the concrete case studies in Table 1 (e.g., QuillBot suggesting 'Climate change is a hoax') are independent content, and the manual-testing comparison is an attempted external anchor, which prevents a score of 8+. However, because the aggregate 531 count and the per-category counts are not independently triaged, the central effectiveness claim is partially circular. No load-bearing self-citation chain or imported uniqueness theorem is present.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The framework's reported issue counts depend on several hand-set thresholds and normative oracle choices, which are not independently benchmarked. No new physical or mathematical entities are introduced.

free parameters (5)
  • Consistency acceptance thresholds
    Section 3.4 states the consistency validator 'defines acceptance thresholds for different extension categories'; issue counts depend on these hand-chosen thresholds.
  • Performance progressive thresholds
    Section 3.4 states the performance validator 'implements progressive thresholds that adapt to content complexity'; these thresholds determine which scaling behaviors are flagged.
  • Security manipulation signature catalog
    Section 3.4 says the security validator 'maintains a catalog of manipulation signatures that evolves'; the catalog determines which prompt injections and hidden-text manipulations are detected.
  • Content alignment extraction thresholds
    Section 3.4 says the content alignment validator 'employs information extraction techniques to identify content origins'; the thresholds for flagging hidden-content leakage are not specified.
  • Test case relation mix per extension = 800 semantic, 200 security
    Section 4.3 chooses 800 semantic equivalence relations and 200 security boundary relations per extension; this distribution directly shapes the number of issues found per category.
assumptions (5)
  • domain assumption Semantic equivalence metamorphic relations preserve meaning across presentation changes.
    Section 3.2 defines visibility, proportionality, and format-preservation relations as invariant; if extensions legitimately treat different DOM structures as different inputs, the metamorphic violations (102 issues) are not bugs.
  • domain assumption Extensions should process only content visible to the user.
    Section 3.4 content alignment validator and Section 5 'visible-only processing' recommendation assert this principle rather than deriving it from extension specifications.
  • domain assumption The manual testing protocol with three domain experts provides a fair, comparable baseline.
    Section 4.4 uses three experts, 60 minutes per extension, and a 'structured process'; no details on inter-rater agreement or expertise, so the throughput comparison could be biased.
  • domain assumption Semantic similarity metrics correctly measure output stability for consistency validation.
    Section 3.4 consistency validator relies on semantic similarity with acceptance thresholds; no validation that these metrics align with human judgments of inconsistency.
  • domain assumption Synthetic webpage templates from 50 popular websites are representative of real pages for these extensions.
    Section 4.1.3 describes the template validation process, but representativeness is assessed via structural features only, not via extension behavior on real pages.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ASSURE: Metamorphic Testing for AI-powered Browser Extensions." pith.science (2026). https://pith.science/paper/AVDFKTHW

@misc{pith2026250705307,
  author       = {Pith},
  title        = {Pith review of: ASSURE: Metamorphic Testing for AI-powered Browser Extensions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AVDFKTHW}},
  note         = {Machine review of arXiv:2507.05307}
}
read the original abstract

The integration of Large Language Models (LLMs) into browser extensions has revolutionized web browsing, enabling sophisticated functionalities like content summarization, intelligent translation, and context-aware writing assistance. However, these AI-powered extensions introduce unprecedented challenges in testing and reliability assurance. Traditional browser extension testing approaches fail to address the non-deterministic behavior, context-sensitivity, and complex web environment integration inherent to LLM-powered extensions. Similarly, existing LLM testing methodologies operate in isolation from browser-specific contexts, creating a critical gap in effective evaluation frameworks. To bridge this gap, we present ASSURE, a modular automated testing framework specifically designed for AI-powered browser extensions. ASSURE comprises three principal components: (1) a modular test case generation engine that supports plugin-based extension of testing scenarios, (2) an automated execution framework that orchestrates the complex interactions between web content, extension processing, and AI model behavior, and (3) a configurable validation pipeline that systematically evaluates behavioral consistency and security invariants rather than relying on exact output matching. Our evaluation across six widely-used AI browser extensions demonstrates ASSURE's effectiveness, identifying 531 distinct issues spanning security vulnerabilities, metamorphic relation violations, and content alignment problems. ASSURE achieves 6.4x improved testing throughput compared to manual approaches, detecting critical security vulnerabilities within 12.4 minutes on average. This efficiency makes ASSURE practical for integration into development pipelines, offering a comprehensive solution to the unique challenges of testing AI-powered browser extensions.

Figures

Figures reproduced from arXiv: 2507.05307 by the authors.

Figure 1
Figure 1. Examples of AI browser extensions’ bug. by incorporating sophisticated natural language processing capabil￾ities directly into the web browsing experience. While conventional browser extensions typically operate through predetermined rules and static content manipulation, AI-powered extensions introduce dynamic, context-aware processing that adapts to user interactions and webpage content in real-time [38]. These ex… view at source ↗
Figure 2
Figure 2. Overview of Assure. static content and predetermined user interactions, our engine must generate test cases that can effectively evaluate both the browser extension functionality and its AI-powered features. We design this engine to systematically create test cases that explore the complex interaction space between web content, browser extension behavior, and AI processing. The architecture employs two complementary… view at source ↗
Figure 3
Figure 3. Comparison of testing efficiency between manual testing [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Breakdown of Assure testing time. count, with testing time increasing proportionally to the number of test cases. This predictable scaling is important for planning testing resources and schedules. Across all extensions, we observed an average throughput of 5.1 test ca…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 21 canonical work pages

  1. [1]

    [n. d.]. Anonymized Repository - Anonymous GitHub. https://anonymous.4open.science/r/ASSURE-5D33/README.md

    work page

  2. [2]

    Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. 2024. Hallucination of Multimodal Large Language Models: A Survey. https://doi.org/10.48550/arXiv.2404.18930 arXiv:2404.18930 [cs]

    arXiv doi:10.48550/arxiv.2404.18930 2024

  3. [3]

    Sruthi Bandhakavi, Nandit Tiku, Wyatt Pittman, Samuel T. King, P. Madhusu- dan, and Marianne Winslett. 2011. Vetting Browser Extensions for Secu- rity Vulnerabilities with VEX. Commun. ACM 54, 9 (Sept. 2011), 91–99. https://doi.org/10.1145/1995376.1995398

    work page arXiv 2011

  4. [4]

    Marcel Böhme, Van-Thuan Pham, and Abhik Roychoudhury. 2016. Coverage- Based Greybox Fuzzing as Markov Chain. InProceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security (CCS ’16) . Association for Computing Machinery, New York, NY, USA, 1032–1043. https://doi.org/10.1145/ 2976749.2978428

    arXiv 2016

  5. [5]

    Robertson, and Engin Kirda

    Ahmet Salih Buyukkayhan, Kaan Onarlioglu, William K. Robertson, and Engin Kirda. 2016. CrossFire: An Analysis of Firefox Extension-Reuse Vulnerabilities.. In NDSS

    work page 2016

  6. [6]

    Stefano Calzavara, Michele Bugliesi, Silvia Crafa, and Enrico Steffinlongo. 2015. Fine-Grained Detection of Privilege Escalation Attacks on Browser Extensions. In Programming Languages and Systems, Jan Vitek (Ed.). Springer, Berlin, Heidelberg, 510–534. https://doi.org/10.1007/978-3-662-46669-8_21

    work page doi:10.1007/978-3-662-46669-8_21 2015

  7. [7]

    Nicholas Carlini, Adrienne Porter Felt, and David Wagner. 2012. An Evaluation of the Google Chrome Extension Security Architecture. In Proceedings of the 21st USENIX Conference on Security Symposium (Security’12) . USENIX Association, USA, 7

    work page 2012

  8. [8]

    Nazanin Bayati Chaleshtari, Fabrizio Pastore, Arda Goknil, and Lionel C. Briand

    work page

Show all 51 references
  1. [9]

    Quan Chen and Alexandros Kapravelos. 2018. Mystique: Uncovering Informa- tion Leakage from Browser Extensions. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security . ACM, Toronto Canada, 1687–1700. https://doi.org/10.1145/3243734.3243823

    2018

  2. [10]

    T. Y. Chen, S. C. Cheung, and S. M. Yiu. 2020. Metamorphic Testing: A New Approach for Generating Next Test Cases. https://doi.org/10.48550/arXiv.2002. 12543 arXiv:2002.12543 [cs]

    2020 doi arXiv

  3. [11]

    Tsong Yueh Chen, Pak-Lok Poon, and Xiaoyuan Xie. 2016. METRIC: METamor- phic Relation Identification Based on the Category-choice Framework. Journal of Systems and Software 116 (June 2016), 177–190. https://doi.org/10.1016/j.jss. 2015.07.037

    2016 doi

  4. [12]

    Gelei Deng, Yi Liu, Yuekang Li, Kailong Wang, Ying Zhang, Zefeng Li, Haoyu Wang, Tianwei Zhang, and Yang Liu. 2024. MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots. InProceedings 2024 Network and Distributed System Security Symposium. https://doi.o...

    2024

  5. [13]

    Yao Deng, Guannan Lou, Xi Zheng, Tianyi Zhang, Miryung Kim, Huai Liu, Chen Wang, and Tsong Yueh Chen. 2021. BMT: Behavior Driven Development-based Metamorphic Testing for Autonomous Driving Models. In 2021 IEEE/ACM 6th International Workshop on Metamorphic Testing (MET). 32–36...

    2021

  6. [14]

    Yao Deng, Xi Zheng, Tianyi Zhang, Huai Liu, Guannan Lou, Miryung Kim, and Tsong Yueh Chen. 2023. A Declarative Metamorphic Testing Framework for Autonomous Driving. IEEE Transactions on Software Engineering 49, 4 (April 2023), 1964–1982. https://doi.org/10.1109/TSE.2022.3206427

    2023

  7. [15]

    Yanjie Gao, Zhengxian Li, Haoxiang Lin, Hongyu Zhang, Ming Wu, and Mao Yang

  8. [16]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu

  9. [17]

    Nav Jagpal, Eric Dingle, Jean-Philippe Gravel, Panayiotis Mavrommatis, Niels Provos, Moheeb Abu Rajab, and Kurt Thomas. 2015. Trends and Lessons from Three Years Fighting Malicious Extensions. In 24th USENIX Security Symposium (USENIX Security 15). 579–593

    2015

  10. [18]

    Alexandros Kapravelos, Chris Grier, Neha Chachra, Christopher Kruegel, Gio- vanni Vigna, and Vern Paxson. 2014. Hulk: Eliciting Malicious Behavior in Browser Extensions. In 23rd USENIX Security Symposium (USENIX Security 14) . 641–654

    2014

  11. [19]

    Mosh Levy, Alon Jacoby, and Yoav Goldberg. 2024. Same Task, More Tokens: The Impact of Input Length on the Reasoning Performance of Large Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , Lun-W...

    2024

  12. [20]

    Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, Jie Huang, Fanpu Meng, and Yangqiu Song. 2023. Multi-Step Jailbreaking Privacy Attacks on ChatGPT. https: //doi.org/10.48550/arXiv.2304.05197 arXiv:2304.05197 [cs]

    2023 doi arXiv

  13. [21]

    Manning, Christopher Ré, Diana Acosta-Navas, Drew A

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michi- hiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Nav...

    2023 doi arXiv

  14. [22]

    Zeyi Liao, Lingbo Mo, Chejian Xu, Mintong Kang, Jiawei Zhang, Chaowei Xiao, Yuan Tian, Bo Li, and Huan Sun. 2024. EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakage. https://doi.org/10.48550/arXiv. 2409.11295 arXiv:2409.11295 [cs]

    2024 doi arXiv

  15. [23]

    Jiawei Liu, Yuheng Huang, Zhijie Wang, Lei Ma, Chunrong Fang, Mingzheng Gu, Xufan Zhang, and Zhenyu Chen. 2023. Generation-Based Differential Fuzzing for Deep Learning Libraries. ACM Trans. Softw. Eng. Methodol. 33, 2 (Dec. 2023), 50:1–50:28. https://doi.org/10.1145/3628159

    2023 doi

  16. [24]

    Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2024. AutoDAN: Gen- erating Stealthy Jailbreak Prompts on Aligned Large Language Models. https://doi.org/10.48550/arXiv.2310.04451 arXiv:2310.04451 [cs]

    2024 doi arXiv

  17. [25]

    Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Zihao Wang, Xiaofeng Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, and Yang Liu. 2024. Prompt Injection Attack against LLM-integrated Applications. https://doi.org/10.48550/ arXiv.2306.05499 arXiv:2306.05499 [cs]

    2024 doi arXiv

  18. [26]

    Nikolaos Pantelaios, Nick Nikiforakis, and Alexandros Kapravelos. 2020. You’ve Changed: Detecting Malicious Browser Extensions through Their Update Deltas. In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communi- cations Security. ACM, Virtual Event USA, 477–4...

    2020

  19. [27]

    Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. https: //doi.org/10.48550/arXiv.2005.04118 arXiv:2005.04118 [cs]

    2020 doi arXiv

  20. [28]

    Parejo, Javier Troya, and Antonio Ruiz-Cortés

    Sergio Segura, José A. Parejo, Javier Troya, and Antonio Ruiz-Cortés. 2018. Meta- morphic Testing of RESTful Web APIs.IEEE Transactions on Software Engineering 44, 11 (Nov. 2018), 1083–1099. https://doi.org/10.1109/TSE.2017.2764464

    2018

  21. [29]

    Hossain Shahriar, Komminist Weldemariam, Mohammad Zulkernine, and Thibaud Lutellier. 2014. Effective Detection of Vulnerable and Malicious Browser Extensions. Computers & Security 47 (Nov. 2014), 66–84. https://doi.org/10. 1016/j.cose.2014.06.005

    2014

  22. [30]

    Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali...

    2023 doi arXiv

  23. [31]

    Oleksii Starov and Nick Nikiforakis. 2017. Extended Tracking Powers: Measuring the Privacy Diffusion Enabled by Browser Extensions. In Proceedings of the 26th International Conference on World Wide Web . International World Wide Web Conferences Steering Committee, Perth Austra...

    2017

  24. [32]

    Chang-Ai Sun, Hepeng Dai, Ning Geng, Huai Liu, Tsong Yueh Chen, Peng Wu, Yan Cai, and Jinqiu Wang. 2023. An Interleaving Guided Metamorphic Testing Approach for Concurrent Programs. ACM Trans. Softw. Eng. Methodol. 33, 1 (Nov. 2023), 8:1–8:21. https://doi.org/10.1145/3607182

    2023 doi

  25. [33]

    Chang-Ai Sun, An Fu, Pak-Lok Poon, Xiaoyuan Xie, Huai Liu, and Tsong Yueh Chen. 2021. METRIC++: A Metamorphic Relation Identification Technique Based on Input Plus Output Domains. IEEE Transactions on Software Engineering 47, 9 (Sept. 2021), 1764–1785. https://doi.org/10.1109/...

    2021

  26. [34]

    Yuchi Tian, Kexin Pei, Suman Jana, and Baishakhi Ray. 2018. DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars. https://doi.org/10. Assure: Metamorphic Testing for AI-powered Browser Extensions Conference’17, July 2017, Washington, DC, USA 48550/arXiv.17...

    2018 arXiv

  27. [35]

    Dung Vo. 2024. IntellectNavigator: Enhancing Search Tools with LLMs-Powered Query Instruction. In Companion Proceedings of the ACM Web Conference 2024 (WWW ’24). Association for Computing Machinery, New York, NY, USA, 1880–

    2024

  28. [36]

    Jindong Wang, Xixu Hu, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang, Haojun Huang, Wei Ye, Xiubo Geng, Binxin Jiao, Yue Zhang, and Xing Xie. 2023. On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective. https://doi.org/10.48550/arXiv.23...

    2023 doi arXiv

  29. [37]

    Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, Wei Ye, Shikun Zhang, and Yue Zhang. 2024. PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization. https://doi.org/10.48550/a...

    2024 doi arXiv

  30. [38]

    Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, Rui Zheng, Xiaoran Fan, Xiao Wang, Limao Xiong, Yuhao Zhou, Weiran Wang, Changhao Jiang, Yicheng Zou, Xiangyang Liu, Zhangyue Yin, Shihan Dou, Rongxiang Weng, W...

    2023 doi arXiv

  31. [39]

    Yezi Xu, Zhi Quan Zhou, Xiaoxia Zhang, Jing Wang, and Mingyue Jiang. 2022. Metamorphic Testing of Named Entity Recognition Systems: A Case Study. IET Software 16, 4 (Aug. 2022), 386–404. https://doi.org/10.1049/sfw2.12058

    2022 doi

  32. [40]

    Linyi Yang, Shuibai Zhang, Libo Qin, Yafu Li, Yidong Wang, Hanmeng Liu, Jindong Wang, Xing Xie, and Yue Zhang. 2023. GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspective. https://doi.org/10.48550/arXiv.2211.08073 arXiv...

    2023 doi arXiv

  33. [41]

    Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Zhibo Sun, and Yue Zhang

  34. [42]

    Lifan Yuan, Yangyi Chen, Ganqu Cui, Hongcheng Gao, Fangyuan Zou, Xingyi Cheng, Heng Ji, Zhiyuan Liu, and Maosong Sun. 2023. Revisiting Out-of- Distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluations. In Proceedings of the 37th International Conference on Neur...

    2023

  35. [43]

    Yifan Zhang, Dave Towey, and Matthew Pike. 2023. Automated Metamorphic- Relation Generation with ChatGPT: An Experience Report. In 2023 IEEE 47th Annual Computers, Software, and Applications Conference (COMPSAC). 1780–1785. https://doi.org/10.1109/COMPSAC57700.2023.00275

    2023

  36. [44]

    Zhi Quan Zhou, Shaowen Xiang, and Tsong Yueh Chen. 2016. Metamorphic Testing for Software Quality Assessment: A Study of Search Engines. IEEE Transactions on Software Engineering 42, 3 (March 2016), 264–284. https://doi. org/10.1109/TSE.2015.2478001

    2016

  37. [45]

    Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Yue Zhang, Neil Gong, and Xing Xie. 2024. PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts. In Proceedings of the 1st ACM Workshop on ...

    2024

  38. [46]

    Xiaogang Zhu, Sheng Wen, Seyit Camtepe, and Yang Xiang. 2022. Fuzzing: A Survey for Roadmap. ACM Comput. Surv. 54, 11s (Sept. 2022), 230:1–230:36. https://doi.org/10.1145/3512345

    2022 doi

  39. [1883]

    https://doi.org/10.1145/3589335.3665843

  40. [2022]

    In Proceedings of the 44th International Conference on Software Engineering (ICSE ’22)

    Refty: Refinement Types for Valid Deep Learning Models. In Proceedings of the 44th International Conference on Software Engineering (ICSE ’22) . Association for Computing Machinery, New York, NY, USA, 1843–1855. https://doi.org/10. 1145/3510003.3510077

  41. [2023]

    https://doi.org/10.48550/ arXiv.2208.09505 arXiv:2208.09505 [cs]

    Metamorphic Testing for Web System Security. https://doi.org/10.48550/ arXiv.2208.09505 arXiv:2208.09505 [cs]

    doi arXiv

  42. [2024]

    High-Confidence Computing 4, 2 (June 2024), 100211

    A Survey on Large Language Model (LLM) Security and Privacy: The Good, The Bad, and The Ugly. High-Confidence Computing 4, 2 (June 2024), 100211. https://doi.org/10.1016/j.hcc.2024.100211

    2024

  43. [2025]

    ACM Transactions on Information Systems 43, 2 (March 2025), 1–55

    A Survey on Hallucination in Large Language Models: Principles, Taxon- omy, Challenges, and Open Questions. ACM Transactions on Information Systems 43, 2 (March 2025), 1–55. https://doi.org/10.1145/3703155 arXiv:2311.05232 [cs]

    2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.