{"id":"627d5ed7-5434-44e0-86aa-b2267469c53b","arxiv_id":"2501.13351","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"A multimodal AI detector, DPGuard, combined with a unified 21-category taxonomy, claims state-of-the-art detection of deceptive UI patterns and finds them in 47% of popular websites and 24% of mobile screenshots.","lead":"The authors built DPGuard, a system that uses a commercial multimodal AI model to automatically spot deceptive patterns (dark patterns) in mobile app and website screenshots. They also propose a unified 21-category taxonomy and report that roughly half of popular websites and a quarter of mobile screenshots contain at least one deceptive pattern.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 6's SOTA claim is clouded: unsupported baseline categories are scored as zero in the micro/macro averages, so the reported F1 gap conflates taxonomy coverage with detection capability.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing concern: the baseline comparison in Table 6 is not like-for-like because unsupported categories are scored as zero and the baselines are not adapted to the unified taxonomy. The arithmetic of the macro averages confirms this interpretation, so the concern is concrete rather than hypothetical. I do not recommend moving the verdict away from CONDITIONAL because the dataset construction and prompt-mutation method are substantive contributions, and even a restricted recomputation suggests DPGuard may retain an advantage on macro F1; however, the headline 0.73/0.44 numbers need to be re-established under a fair comparison. The wild prevalence figures add a second, related calibration risk because the promised manual validation is not reported. These are addressable with additional analyses, so the original conditional acceptance remains appropriate.","tokens_in":17969,"tokens_out":11237,"duration_ms":96441,"concrete_test":"Recompute Table 6's micro and macro F1 for DPGuard restricted to the category support set of each baseline (e.g., the 13 mobile categories UIGuard supports and the 7 mobile/website categories AidUI supports), using the same instance weights as the original table, and compare those numbers with the corresponding baseline averages over the same categories. If DPGuard no longer leads on both micro and macro in this intersection-set comparison, Takeaway 2 should be weakened to a taxonomy-coverage claim rather than a detection-quality claim. A secondary check is to release the manual-review precision/recall for DPGuard on the 12,301 wild screenshots that Appendix B.1 says was performed; if wild precision is materially below the test-set precision, the 23.61% and 47.27% prevalence numbers need recalibration.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The claim that DPGuard outperforms SOTA (Takeaway 2) depends entirely on Table 6. The table marks many categories as unsupported ('-') for UIGuard and AidUI, yet the macro averages are computed over the full taxonomy with those categories scored as F1=0. This is not speculation: UIGuard's mobile macro 0.2851 is exactly the sum of its 13 non-'-' F1 values (6.2712) divided by 22, and AidUI's 0.0878 is exactly its 1.9323 divided by 22; the same pattern holds for the website rows. Consequently, UIGuard and AidUI are penalized for every instance in categories their taxonomies cannot express (e.g., Forced Enrollment, Intermediate Currency, Hidden Costs), so DPGuard's advantage reflects taxonomy breadth as much as detection skill. The paper neither restricts the comparison to the common support set nor adapts/retrains the baselines to the unified taxonomy. A quick restricted macro recomputation shows the gap over UIGuard-supported mobile classes shrinks from about 0.153 to about 0.060, so the qualitative direction may survive, but the specific numbers in Takeaway 2 (0.73 micro / 0.44 macro) are not established as like-for-like. Appendix B.1 says wild data were manually reviewed to assess DPGuard's real-world performance, but no such validation results are reported; the Section 5.2 prevalence numbers therefore inherit the same calibration uncertainty.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a unified taxonomy of deceptive patterns with 21 categories, a new dataset of 6,725 UI images containing 10,421 DP instances, and DPGuard, a hybrid detection framework that combines a fine-tuned binary classifier (ResNet101) with a commercial multimodal LLM (GPT-4o) whose prompts are refined through a mutation-based prompt engineering process. The authors report that DPGuard outperforms existing SOTA detectors UIGuard and AidUI on both mobile and website datasets, and they present a large-scale empirical study of 12,301 wild UI images claiming that 23.61% of mobile screenshots and 47.27% of website screenshots contain at least one deceptive pattern. The paper also includes four case studies linking the expanded taxonomy categories to security and privacy implications.","tokens_in":18303,"tokens_out":9096,"duration_ms":72261,"significance":"If the reported results are valid, the paper would provide a substantial community resource: a large cross-platform deceptive-pattern dataset, a unified taxonomy that explicitly incorporates privacy and security concerns, and a working automatic detector that could scale beyond the small manually curated corpora used by prior work. The prompt-mutation approach to zero-shot MLLM adaptation is an interesting engineering contribution, and the code is released. The wild prevalence figures, if independently validated, would be an important quantification of deceptive design in popular apps and websites. However, the significance is contingent on resolving several evaluation and validation concerns, most notably the construction of the SOTA comparison in Table 6 and the lack of independent ground-truth validation for the in-the-wild measurements.","major_comments":[{"comment":"The overall DPGuard evaluation in Table 6 does not specify which data split is used. The 'Instances' column sums to the full dataset counts (after removing the Sneak into Basket and Tricked Questions categories), rather than to a held-out test set. Since the binary classifier was fine-tuned on a 6:2:2 split and the prompt mutation process selected prompts using batches from the training portion (Appendix A), an evaluation on the full dataset would allow training and prompt-selection data to be present in the evaluation set, potentially inflating DPGuard's reported F1 scores. The paper must state the exact evaluation split and report results on the held-out test set (e.g., the 20% reserved for the MLLM evaluation) separately from any in-distribution numbers.","section":"Section 5.1, Table 6"},{"comment":"The macro F1 averages for UIGuard and AidUI are computed over the full 22-class taxonomy with unsupported categories ('-') treated as zero. This is verifiable: UIGuard's mobile macro F1 of 0.2851 equals 6.2712/22 (the sum of its 13 supported F1 values divided by 22), and AidUI's 0.0878 equals 1.9323/22; the website rows follow the same pattern. Consequently, the reported SOTA advantage conflates taxonomy coverage with detection capability. The comparison should be restricted to the common support set, or the baselines should be adapted/retrained to the unified taxonomy. A restricted recomputation over the UIGuard-supported mobile classes reduces the macro gap from about 0.153 to about 0.060, so the specific Takeaway 2 numbers are not established as like-for-like SOTA improvements.","section":"Table 6, Takeaway 2"},{"comment":"The wild prevalence figures (23.61% of mobile screenshots and 47.27% of website screenshots) are DPGuard's own predictions on the collected wild data. Appendix B.1 states that the authors 'randomly sampled some data for manual review to assess the actual performance of our model in the wild,' but no such validation results are reported anywhere in the paper. Without independent ground-truth labels on a random sample of the wild images, these prevalence rates are model outputs, not measurements. The paper should report the manual review outcome (e.g., precision and recall on the wild sample) and present calibrated prevalence estimates, or clearly label the figures as model-predicted rates.","section":"Section 5.2 and Appendix B.1"},{"comment":"The taxonomy definitions, the dataset labels, and the evaluation labels are all produced by the same authors, and no annotation protocol or inter-annotator agreement is reported. Because DPGuard is optimized and evaluated on these self-produced labels, the reported F1 scores may partly reflect the authors' interpretation of the taxonomy rather than an objective ground truth. The paper should provide annotation guidelines, inter-annotator agreement statistics, and ideally an external validation set (e.g., from prior taxonomies or independent reviewers) to support both the SOTA performance claim and the prevalence measurements.","section":"Section 3 (Dataset creation) and Appendix A"}],"minor_comments":[{"comment":"The front-matter abstract reports prevalence as '23.61% of mobile screenshots and 47.27% of website screenshots,' while the paper's abstract section reports '25.7% of mobile apps and 49.0% of websites.' These are different metrics (image-level vs. app/domain-level), and the headline claim should use one consistent quantity or explicitly present both.","section":"Abstract (front matter vs. paper text)"},{"comment":"The conclusion states that the taxonomy was 'refining it with 24 subcategories,' but Table 2 lists 21 DP categories. Please reconcile this number.","section":"Section 7"},{"comment":"The text says prompt mutation terminates if the best prompt has not been updated for three rounds, but the pseudocode in Algorithm 1 loops until t<T without a break for that condition. Make the pseudocode consistent with the stated termination rule.","section":"Algorithm 1 and Section 4.3"},{"comment":"The text contains a typo: 'Fiugre 9(c)' should be 'Figure 9(c)'.","section":"Section 6.4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be the camera-ready of a WWW 2025 paper (with ACM DOI and copyright notice). If this is being considered as a journal submission, the editorial office should clarify the overlap with the prior conference version and any originality requirements. The main technical obstacles to acceptance are the unspecified evaluation split in Table 6 and the treatment of unsupported categories as zero in the macro averages; both are fixable in revision. The wild-prevalence claim also needs the promised manual validation to be reported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful dataset-and-taxonomy paper with a plausible detector, but the headline SOTA claim is built on a comparison that is not like-for-like. The stress-test note is right: Table 6 scores UIGuard and AidUI as zero on categories their taxonomies don't express, so the macro F1 gap partly reflects taxonomy breadth, not detection skill. Reading the table, the micro F1 advantage is smaller, and the macro advantage shrinks when you restrict to categories the baselines support. The direction probably survives, but the specific numbers in Takeaway 2 aren't trustworthy as stated.\n\nWhat's genuinely good: the merged dataset (6,725 images, 10,421 instances) is the largest I've seen for this problem, and the taxonomy refinements are thoughtful. Removing Bait-and-Switch because it overlaps other categories, and reintroducing Forced Enrollment with a clear constraint (no skip option), are sensible moves. Applying PromptBreeder-style mutation to an MLLM for DP detection is a reasonable adaptation, and the binary-classifier-then-MLLM pipeline is practical.\n\nSoft spots beyond Table 6: the wild prevalence numbers (23.61% mobile, 47.27% website) are the model's own predictions, and the paper says Appendix B.1 includes manual review but never reports those results. Without a validated sample, these are model outputs, not measurements. The dataset isn't released, and there are no error bars anywhere, so we can't tell how stable the performance numbers are.\n\nOverall: this is a solid engineering contribution with real resource value. A serious referee should ask for a like-for-like comparison, a manual validation of a wild sample, and artifact release. That's addressable in revision.","headline":"Useful dataset and taxonomy, but the SOTA claim rests on an unfair comparison and the wild prevalence numbers need manual validation.","tokens_in":18869,"tokens_out":1841,"would_cite":true,"duration_ms":25405,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DPGuard, a binary classifier plus multimodal LLM with mutation-evolved prompts, outperforms prior deceptive-pattern tools and finds patterns in 23.61% of mobile and 47.27% of website screenshots.","keywords":["deceptive patterns","dark patterns","taxonomy","DPGuard","multimodal large language model","UI screenshot detection","privacy","empirical study"],"falsifier":"Recompute Table 6 using only the categories that UIGuard and AidUI support, so all three tools are scored on the identical subset of instances; if DPGuard's micro and macro F1 no longer exceed the baselines on that common subset, the 'outperforms state-of-the-art' claim reduces to a taxonomy-coverage artifact and the detection advantage disappears.","tokens_in":17756,"feed_emoji":"🕵️","tokens_out":10386,"duration_ms":80837,"temperature":0.7,"pith_summary":"This paper tries to establish that deceptive patterns in mobile apps and websites can be mapped onto one unified taxonomy and detected automatically by a two-stage tool called DPGuard. The tool first uses a fine-tuned image classifier to decide whether a screenshot is deceptive, then hands only suspicious screenshots to a multimodal large language model whose classification prompt has been improved through mutation. Evaluated on a newly assembled dataset of 6,725 images containing 10,421 labeled pattern instances, DPGuard reports higher micro and macro F1 scores than the two previous state-of-the-art detectors, UIGuard and AidUI. Applying DPGuard to 2,000 popular apps and websites, the authors report that 23.61% of mobile screenshots and 47.27% of website screenshots contain at least one deceptive pattern instance. If these numbers hold, regulators and platform reviewers would have a practical way to audit interface manipulation at scale.","feed_headline":"Nearly half of popular website screenshots contain deceptive patterns","feed_subtitle":"A new detector, DPGuard, claims to beat prior tools and flags these patterns on a quarter of mobile screenshots, too.","key_machinery":"The central mechanism is DPGuard's two-stage inference pipeline: a fine-tuned ResNet101 binary classifier decides whether a screenshot is deceptive, and only positively classified images are passed to GPT-4o, a multimodal LLM that names the specific deceptive-pattern category. The prompt used by the LLM is produced by prompt mutation, an adaptation of PromptBreeder in which the model paraphrases, adds, or deletes actions; a quality checker keeps mutated prompts whose cosine similarity to the initial prompt exceeds a threshold, and a prompt queue retains the best-performing prompts across mutation rounds using binary cross-entropy loss on a balanced batch. The unified 21-category taxonomy is the annotation scheme that ties the dataset, the system prompt, and the evaluation together, defining what counts as a deceptive pattern instance.","core_discovery":"The paper's central claim is that a two-stage pipeline—a low-cost binary classifier that filters out non-deceptive screenshots, followed by a commercial multimodal large language model guided by a mutation-evolved prompt—is sufficient to detect deceptive patterns at the state of the art. On the authors' cross-platform dataset, DPGuard achieves micro/macro F1 of 0.73/0.44 on mobile screenshots and 0.50/0.34 on website screenshots, exceeding UIGuard and AidUI on both. The paper also claims that its refined 21-category taxonomy, which reintroduces Forced Enrollment and expands the scope of five existing categories, is necessary because previous taxonomies missed security- and privacy-relevant designs such as undisclosed subscription fees, fake scarcity, and privacy terms buried in hyperlinks. Under this taxonomy, the empirical study of 2,000 popular services finds that 23.61% of mobile screenshots and 47.27% of website screenshots contain at least one deceptive pattern, with websites averaging more instances per screen than mobile apps.","pith_inferences":["The same two-stage design could be repurposed for other visual-manipulation audits, such as cookie-consent banners, in-app purchase flows, or political advertising, by swapping in a domain-specific taxonomy in the system prompt.","The website prevalence figure of 47.27% is probably an underestimate: the crawler visits a limited number of pages per domain and cannot trigger dynamic consent modals, checkout stages, or post-trial states where many hidden costs actually appear.","The reported F1 numbers are tied to one commercial model snapshot; if future GPT-4o updates change behavior, the exact scores will drift, so the durability of the framework should be assessed by re-running the mutation loop on the same data rather than by fixing confidence to the current metric values.","An open-weights model fine-tuned on the same dataset could test how much of the gain comes from the prompt-mutation method versus the underlying commercial model, and would also make the detector usable on interfaces where screenshots cannot be sent to external APIs."],"forward_implications":["Because DPGuard only queries the LLM for screenshots the binary classifier flags, bulk auditing of app stores or the web becomes far cheaper than running an LLM on every image.","Since the taxonomy lives in the system prompt, updating the detector to a revised or extended taxonomy only requires re-running the prompt-mutation loop, which is the paper's intended answer to concept drift in deceptive designs.","The reported prevalence numbers imply that deceptive patterns are not rare edge cases: nearly half of website screenshots and roughly a quarter of mobile screenshots from popular services carry at least one instance, so any platform-level mitigation would affect a large fraction of user interactions.","The four scope expansions give security practitioners concrete new pattern types to test for, such as post-trial subscription fees, fake scarcity countdowns, privacy terms hidden in hyperlinks, and plan-comparison barriers.","Removing Bait-and-Switch while adding Forced Enrollment changes what a detector will flag; suites of screenshots from prior studies may need re-annotation before results from different taxonomies can be compared."],"supporting_citations":[{"why":"UIGuard supplies the base taxonomy the paper refines, contributes mobile dataset images, and is the primary state-of-the-art baseline for detection comparison.","marker":"[8]"},{"why":"AidUI is the other state-of-the-art detection baseline and contributes dataset images; its taxonomy is merged into the unified framework.","marker":"[23]"},{"why":"This mobile dark-pattern study provides one of the founding taxonomies that the unified taxonomy builds on and contributes mobile screenshots to the dataset.","marker":"[11]"},{"why":"Mathur et al.'s shopping-website taxonomy informs the category and scope refinement and supplies the scale of deceptive patterns on the web.","marker":"[24]"},{"why":"Brignull's original deceptive-pattern catalog defines Bait-and-Switch, which the paper removes based on this definition, and supplies early DP instances.","marker":"[7]"},{"why":"PromptBreeder is the origin of the mutation-based prompt engineering approach that DPGuard adapts for multimodal UI detection.","marker":"[13]"},{"why":"AndroZoo is the source of the 1,000 Android APKs used in the empirical prevalence study.","marker":"[1]"},{"why":"The Majestic Million list selects the 1,000 popular websites crawled for the empirical prevalence study.","marker":"[18]"},{"why":"WebUI provides the crawling template that the paper modifies to collect website screenshots.","marker":"[39]"}],"fun_headline_variants":["Nearly half of popular websites hide deceptive UI tricks","DPGuard spots deceptive patterns on 47% of websites","Deceptive patterns found on half of popular websites","New tool uncovers dark patterns in 47% of websites","Unified taxonomy reveals web dark patterns dominate"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that scoring UIGuard and AidUI on a dataset annotated with a different taxonomy—with many categories marked unsupported for those tools—still yields micro and macro F1 averages comparable to DPGuard's, so the reported state-of-the-art advantage could be an artifact of taxonomy coverage rather than detection skill.","fun_headline_variants_meta":{"raw":{"variants":["Nearly half of popular websites hide deceptive UI tricks","DPGuard spots deceptive patterns on 47% of websites","Deceptive patterns found on half of popular websites","New tool uncovers dark patterns in 47% of websites","Unified taxonomy reveals web dark patterns dominate"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000137,"raw_usage":{"total_tokens":1172,"prompt_tokens":986,"completion_tokens":186,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":602,"completion_tokens_details":{"reasoning_tokens":111}},"tokens_in":602,"tokens_out":186,"duration_ms":8111,"temperature":1.0,"reasoning_tokens":111,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T16:13:49.450454+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute Table 6 using only the categories that UIGuard and AidUI support, so all three tools are scored on the identical subset of instances; if DPGuard's micro and macro F1 no longer exceed the baselines on that common subset, the 'outperforms state-of-the-art' claim reduces to a taxonomy-coverage artifact and the detection advantage disappears.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"UIGuard supplies the base taxonomy the paper refines, contributes mobile dataset images, and is the primary state-of-the-art baseline for detection comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"AidUI is the other state-of-the-art detection baseline and contributes dataset images; its taxonomy is merged into the unified framework."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"This mobile dark-pattern study provides one of the founding taxonomies that the unified taxonomy builds on and contributes mobile screenshots to the dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Brignull's original deceptive-pattern catalog defines Bait-and-Switch, which the paper removes based on this definition, and supplies early DP instances."},{"cited_title":"Bissyandé, Jacques Klein, and Yves Le Traon","cited_arxiv_id":null,"evidence_quote":"AndroZoo is the source of the 1,000 Android APKs used in the empirical prevalence study."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Majestic Million list selects the 1,000 popular websites crawled for the empirical prevalence study."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"WebUI provides the crawling template that the paper modifies to collect website screenshots."}],"review_version":1}