{"id":"e697c38c-5972-42ae-b1fb-ae5b4eda5110","arxiv_id":"2505.06836","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"PhishXplain uses a local LLM to generate real-time phishing warnings with annotated screenshots and contextual explanations, and a user study found these warnings improved later phishing detection.","lead":"This paper describes PhishXplain, a browser tool that uses a small AI model running on the user's computer to replace generic phishing warnings with explanations that point out the specific suspicious parts of a flagged website. In a 150-person study, people who saw these detailed warnings were better at spotting phishing sites later, especially users with little security experience.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Transfer test in §5.2 is near-transfer: every unguided phishing site reuses cue types from the warning phase, so Group A's advantage supports practice on familiar cues, not the generalizable 'even without any warning' claim.","rationale":"The reader's weakest assumption identifies the same load-bearing point. The central claim is not about immediate compliance; it is about durable, independent detection. The unguided phase in §5.2 is the only support for that claim, and it is deliberately aligned with the warning phase's cue types. The paper says the design allows assessing generalization 'to new, unseen phishing websites,' but new websites are not the same as new cues: all cue categories in the test set had already been highlighted in the warning phase. This is a validity concern about the scope of the inference, not a dispute with the broader anti-phishing consensus. The operational contributions (94% warning generation, roughly five-second latency, local processing, released extension) are credible and checkable, and the immediate-difference results may be real. The issue is only whether they support the generalization claim. A held-out-cue transfer study would settle it. Since the concern is addressable and the reader already assigned CONDITIONAL, no verdict change is needed.","tokens_in":21935,"tokens_out":5080,"duration_ms":52060,"concrete_test":"Re-run the unguided transfer assessment with four phishing sites whose cue categories are disjoint from the warning phase. For example, if the warning phase covers suspicious URLs, urgency, IDN homographs, third-party hosting, and sensitive-data requests, use test cues such as fake security badges, pre-filled credentials, mismatched branding, expired TLS certificate, or QR-code login. Keep the same protocol, sample-size calculation, and proficiency stratification, and pre-register a comparison for Group A on novel cues. If Group A's advantage disappears or drops below significance, the headline claim is near-transfer; if it persists, the mental-model interpretation survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline claim is that explainable warnings improve independent phishing detection, including 'even without any warning.' The only evidence is the unguided assessment in §5.2, but that assessment is constructed as near-transfer. The text states: 'The phishing websites were carefully selected from PhishTank and were chosen to contain malicious features similar to those highlighted in the warning phase (for Group A).' Table 2 confirms complete cue overlap: Phishing-1 (IDN homograph, third-party hosting) reuses Warning-3 and Warning-2 cues; Phishing-2 (suspicious URL, urgency) reuses Warning-1; Phishing-3 (suspicious URL, unrealistic claim, sensitive-info request) reuses Warning-1, Warning-3, and Warning-2; Phishing-4 (suspicious URL, third-party hosting, sensitive-info request) reuses Warning-1 and Warning-2. No test site contains a cue category that was absent from the warning phase. Group A participants therefore had the opportunity to practice exactly the feature categories they were later asked to recognize. The benign-site results strengthen this reading: Group A did not significantly outperform Group B on benign sites (p = 0.055) and did not significantly differ in false cues on benign sites (p = 0.129), consistent with cue-specific learning rather than a generalizable suspiciousness calibration. This does not undermine PXP's practical value as an immediate teaching aid for common cues, but it does undercut the stronger claim that users acquire a transferable mental model that operates 'even without any warning.'","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents PhishXplain (PXP), a browser-extension framework that augments existing anti-phishing tools by replacing generic warnings with contextual explanations produced by a local 4-bit quantized LLaMA 3.2:3B model. PXP parses the rendered DOM, uses a two-prompt structured architecture constrained by a 26-feature lookup table, and generates annotated screenshots with up to four highlighted suspicious features. The authors evaluate latency, memory, and reliability on 100 live PhishTank sites, report a one-month longitudinal deployment over 7,091 GSB-flagged URLs with a 94% warning-generation rate and a 96% manually assessed correctness rate on a 500-site sample, and run a between-subjects user study (n=150) comparing PXP warnings with generic Safe Browsing warnings. The study reports that PXP participants were more accurate on unguided phishing websites (H1, p<0.01), reported higher confidence and helpfulness, and improved cue articulation, especially for low-proficiency users. The paper argues that contextual warnings support learning and transfer to unassisted detection.","tokens_in":22217,"tokens_out":6206,"duration_ms":61529,"significance":"The paper's direction is valuable: replacing blocklist warnings with concrete, user-visible cues is a plausible way to improve both immediate compliance and phishing cue awareness, and the local-LLM design with constrained output is a practical engineering contribution. The H1 phishing-only difference (p<0.01), the significant proficiency-by-warning interaction, and the release of the extension and codebook are strengths. However, the headline correctness figure is self-referential, the transfer test reuses the same cue categories, and one reported significant result (H2) does not meet the paper's own Bonferroni threshold. The practical contribution remains defensible, but the stronger generalizability claims in the abstract and conclusion are not yet supported.","major_comments":[{"comment":"The 96% correctness figure in the abstract and Section 4.3 rests on a circular design: the same two author-coders who constructed the 26-feature lookup table and template text (Section 3.3) manually reviewed the 500-site longitudinal sample, and the LLM is constrained by the prompt architecture to emit only features from that table. No inter-rater reliability is reported for the longitudinal sample, and no independent ground-truth source is used. As written, the 1,316/1,368 agreement measures consistency with the authors' own feature taxonomy rather than external correctness. Please report independent coding, inter-rater statistics, and disagreement resolution, or re-scope the correctness claim accordingly.","section":"§4.2 and §4.3"},{"comment":"The transfer evidence for the abstract's claim that users achieve higher accuracy \"even without any warning\" is near-transfer. The text states that the four unguided phishing websites \"were chosen to contain malicious features similar to those highlighted in the warning phase (for Group A),\" and Table 2 confirms complete cue overlap: Phishing-1 reuses IDN homograph and third-party hosting from Warning-2/Warning-3, Phishing-2 reuses suspicious URL and urgency from Warning-1, and the other test sites similarly reuse warning-phase cue categories. No unguided site introduces a cue category absent from the warning phase, so Group A's advantage is consistent with practiced feature recognition rather than a generalizable mental model. The paper should either add test sites with novel cue categories or restrict the claim to immediate transfer on previously encountered cue types.","section":"§5.2 and Table 2"},{"comment":"In RQ2 the authors report H2 as significant with p<0.015, but the Bonferroni-corrected significance threshold stated earlier in Section 5.3 for performance-related hypotheses (H1–H4) is α=0.0125. Since 0.015 > 0.0125, the reported p-value does not meet the stated threshold. Exact p-values and a corrected conclusion for H2 are needed, because this result currently appears in the abstract's confidence and trust claims.","section":"§5.3, RQ2/H2"}],"minor_comments":[{"comment":"The reliability formula is typeset ambiguously as \"Reliability = 10× CFR+(1−FMR)+AA+CSA /4\" without parentheses; please display the intended expression, presumably 10 × (CFR + (1−FMR) + AA + CSA)/4.","section":"§4.2"},{"comment":"Table 5 mislabels hypotheses relative to the text in Section 5.1: it pairs H3 with \"Accuracy×Proficiency interaction,\" H4 with \"Recognition of phishing cues,\" and H6 with \"Perceived warning helpfulness,\" whereas the text defines H3 as perceived helpfulness, H4 as the low-proficiency interaction, H6 as cue recognition, and H5 as the helpfulness-by-proficiency interaction. Please correct the table to match the hypothesis definitions.","section":"§5.3, Table 5"},{"comment":"The sentence \"90.67% of Group A participants correctly identified Phishing-1, compared to only 52% for Phishing-3\" appears to compare different websites; the intended contrast is likely Group B's 52% on Phishing-1, so the wording should be corrected.","section":"§5.3, RQ1"},{"comment":"Exact p-values and effect sizes should be reported for the Mann-Whitney and ANOVA results instead of inequalities such as p<0.01 and p<0.005, which makes it difficult to verify results against the stated Bonferroni thresholds.","section":"§5.3"},{"comment":"Section 3.4 references \"Figure 7\" twice for apparently different illustrations; the second reference should point to a distinct appendix figure or be renumbered.","section":"§3.4"},{"comment":"The limitations section acknowledges the absence of a long-term field study and the benign-site false-positive issue, but it should also address the circularity of the reliability assessment and the near-transfer design of the unguided test set, since these directly bound the headline claims.","section":"§6"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the manuscript is salvageable as a systems-and-user-study paper if the authors re-scope the generalizability claims, add independent reliability coding or an inter-rater reliability report, and fix the H2 statistical threshold error. There is no indication of fabrication, but the self-referential evaluation and the near-transfer test design currently prevent the abstract's strongest claims from being supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The system is real and the core effect is probably real, but the paper oversells the generalization. PhishXplain is a solid piece of engineering: a local quantized Llama 3.2 3B, a two-prompt lookup-table architecture, annotated screenshots, and a month-long deployment on 7,091 live PhishTank sites with 94% warning coverage, ~5s median latency, and 2.7GB memory on consumer hardware. Those operational numbers are credible and worth having. The user study is also a step up from the usual warning-paper: 150 participants, between-subjects, HAIS-Q-based proficiency stratification, and coded open-ended responses with kappa 0.72. The main finding, that explainable warnings improve phishing detection accuracy in the immediate task, is plausible and statistically supported for the phishing sites.\n\nThe soft spots are real but not fatal. The biggest is the transfer claim. Section 5.2 says the unguided phishing sites were chosen to contain malicious features similar to those in the warning phase, and Table 2 confirms complete cue overlap: every test phishing site reuses cue types from the warning set. So Group A's advantage on those sites is better described as practice on familiar cues, not evidence of a generalizable mental model that works \"even without any warning.\" The benign-site results reinforce this: no significant group difference on benign sites (p=0.055) and no significant difference in false cues (p=0.129), which is consistent with cue-specific learning rather than a broad suspiciousness calibration. The abstract's \"even without any warning\" claim is overstated.\n\nSecond, the 96% correctness figure comes from a manual review by the same two authors who built the lookup table, and no inter-rater reliability is reported for that 500-site sample. Since the LLM is constrained to output from that exact taxonomy, the evaluation is circular to a degree. Independent coders or a released coding of the sample would help. Third, there is a stats inconsistency: they set a Bonferroni threshold of alpha=0.0125 for H1-H4, then report H2 at p<0.015, which would not clear that bar. Exact p-values and effect sizes are needed. Minor: the composite reliability score is an arbitrary equal-weight average, and the four-feature cap is a tuning choice.\n\nI would take this paper seriously, and it deserves peer review. It is not a desk reject. The near-transfer issue is addressable with a held-out test set that includes novel cue categories, and the stats are fixable. The system is a useful template for usable-security researchers and browser vendors, and the user-study result is worth replicating. I'd say yes to reviewing it, with a strong push to temper the generalization claim and release the coded data.","headline":"PXP is a genuinely useful system and the user study shows a real effect, but the 'even without any warning' headline claim overreaches because the transfer test is near-transfer on the same cue types.","tokens_in":22781,"tokens_out":2563,"would_cite":true,"duration_ms":27628,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that replacing generic anti-phishing warnings with short, specific explanations of why a site is suspicious trains users to recognize phishing on their own—with the effect largest for novices and the warning engine…","keywords":["phishing warnings","explainable security","contextual warnings","large language models","user study","phishing detection","mental models","browser security"],"falsifier":"Run the same two-group study but assemble the unassisted phishing set from cues absent from the warning phase—for example QR-code payment lures or brand impersonation through lookalike HTTPS subdomains with no obvious URL anomaly. If Group A no longer outperforms Group B on those sites, the claimed transfer is cue-specific practice, not a generalizable detection skill.","tokens_in":21706,"feed_emoji":"🎣","tokens_out":5746,"duration_ms":54637,"temperature":0.7,"pith_summary":"PhishXplain claims that swapping a browser's generic 'this site is deceptive' warning for a short explanation of exactly which visible cues make the site suspicious trains users to recognize phishing on their own. In a controlled study, participants who saw these explainable warnings correctly classified 6.44 of 8 unassisted websites on average, versus 5.16 for those who saw generic warnings, with the gap driven by phishing sites and by the least security-savvy users. The same warning generator ran for a month on 7,091 live phishing sites, producing warnings for 94% and correctly describing features in about 96% of sampled indicators. If the transfer effect is real, explainable warnings turn a moment of blocking into a micro-lesson, and do it locally on consumer hardware without sending browsing data to a third party.","feed_headline":"Explained phishing warnings lift user detection from 5.2 to 6.4 of 8","feed_subtitle":"One month of live data: 94% of flagged sites get warnings, and novices gain most.","key_machinery":"PhishXplain's engine is a two-step prompt pipeline around a fixed lookup table of 26 user-visible phishing cues. A small local language model first reads the fully rendered page source, with HTML elements wrapped in unique delimiters, and returns up to four matching cues as a JSON object; a second prompt then fills pre-written template sentences with concrete artifacts (typos, URLs, demands) from the flagged code. A separate renderer re-draws the page with colored bounding boxes on those elements and captures a screenshot, so the warning shows both text and visual proof. The lookup table is what keeps a small model grounded and reproducible: the model only maps to known features and fills in blanks, rather than generating open-ended explanations.","core_discovery":"The paper's central claim is that contextual, feature-specific warnings—not just a generic block page—cause users to build or activate phishing cues they can apply later, without any warning. The evidence is a between-subjects study in which Group A saw PhishXplain warnings that named, explained, and visually highlighted up to four suspicious elements, while Group B saw a standard generic block-page warning; both groups then judged eight unassisted websites. Group A averaged 6.44/8 correct versus 5.16 for Group B (p < 0.01 for phishing sites), and low-proficiency users jumped from 5.0 to 6.8 correct, nearly matching high-proficiency users. Group A also named more correct cues in unassisted phishing sites (7.84 versus 5.45). The paper does not claim the tool improves detection itself; it claims the explanation transforms user responses, and that this can be done in about 5 seconds with 2.7GB of memory on a mid-range consumer machine.","pith_inferences":["A likely mechanism is cue rehearsal rather than durable mental-model formation: the four unassisted phishing sites shared cue types with the warning phase, so a delayed re-test with novel cues is needed to tell the two apart.","Adding explicit 'looks legitimate because...' signals for benign sites would be a natural extension; the paper's own limitations note that explaining only malicious cues leaves benign judgments unchanged.","The same two-step lookup-table scaffolding could be reused for other warnings (malware, credential theft, tech-support scams), with the table swapped for the target domain's cue inventory.","If future phishing campaigns shift to cues outside the 26-entry table, coverage is testable: the system's 94% generation rate and 96% correctness are only as current as the table."],"forward_implications":["If the result holds, anti-phishing tools can convert every block into a brief teaching moment without a separate training program.","Low-proficiency users gain the most, so explainable warnings could narrow the security-skill gap that generic warnings leave untouched.","The 94% warning-generation rate and 96% feature-correctness indicate most blocklisted phishing pages contain visible, explainable cues; only backend-only evasions are unexplainable.","Because false-positive rates on benign sites did not improve, the training effect is specific to recognizing malicious cues rather than general caution.","The local, 5-second, roughly 2.7GB deployment makes the approach practical as a privacy-preserving browser extension rather than a cloud API."],"supporting_citations":[{"why":"Shows users often ignore generic browser security warnings, the gap PhishXplain targets.","marker":"[7]"},{"why":"Prior work on explanatory warning dialogs that PhishXplain builds on.","marker":"[15]"},{"why":"Shows language models can identify phishing features from website source, motivating the LLM-based feature extraction.","marker":"[22]"},{"why":"Documents how evasive phishing sites stay undetected by blocklists, motivating user-side education.","marker":"[44]"},{"why":"Supplies the standardized security-awareness questionnaire used to measure cybersecurity proficiency.","marker":"[47]"},{"why":"Identifies the HTML tag categories frequently weaponized in phishing, justifying the parser's tag set.","marker":"[54]"},{"why":"Provides the proficiency-tier scoring and study design basis for measuring phishing knowledge.","marker":"[57]"},{"why":"Shows users already detect low-effort phishing well, justifying the use of realistic adversarial phishing pages in the study.","marker":"[65]"}],"fun_headline_variants":["Explained phishing alerts teach cues that stick after warning is gone","Contextual warnings turn novices into phishing spotters","Why this site is risky: Real-time explanations outdo generic alerts","PhishXplain: Warning with reasons boosts user detection by 25%","Explain, don't just block: Users learn to detect phishing alone"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The strongest result depends on the unassisted phishing sites sharing the same cue types the explainable warnings had just practiced; if those shared cues are removed, the learning advantage may vanish.","fun_headline_variants_meta":{"raw":{"variants":["Explained phishing alerts teach cues that stick after warning is gone","Contextual warnings turn novices into phishing spotters","Why this site is risky: Real-time explanations outdo generic alerts","PhishXplain: Warning with reasons boosts user detection by 25%","Explain, don't just block: Users learn to detect phishing alone"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000769,"raw_usage":{"total_tokens":3467,"prompt_tokens":1065,"completion_tokens":2402,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":681,"completion_tokens_details":{"reasoning_tokens":2312}},"tokens_in":681,"tokens_out":2402,"duration_ms":17183,"temperature":1.0,"reasoning_tokens":2312,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:32:04.869202+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same two-group study but assemble the unassisted phishing set from cues absent from the warning phase—for example QR-code payment lures or brand impersonation through lookalike HTTPS subdomains with no obvious URL anomaly. If Group A no longer outperforms Group B on those sites, the claimed transfer is cue-specific practice, not a generalizable detection skill.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the proficiency-tier scoring and study design basis for measuring phishing knowledge."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows users often ignore generic browser security warnings, the gap PhishXplain targets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior work on explanatory warning dialogs that PhishXplain builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents how evasive phishing sites stay undetected by blocklists, motivating user-side education."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the standardized security-awareness questionnaire used to measure cybersecurity proficiency."}],"review_version":1}