{"id":"150778ad-cf0e-4949-9ae4-44f178338862","arxiv_id":"2504.16449","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A survey that classifies malicious URL detection research by data modality and lists datasets and code, but it overclaims to provide a 'first unified benchmark' without actually running one.","lead":"This paper surveys methods for detecting malicious URLs, organizing them by the type of data they use: the URL itself, webpage HTML, JavaScript, visual appearance, and combinations. It also lists public datasets and open-source code repositories, and outlines design principles for product-level detectors.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'first unified benchmark' claim is unsupported: the paper curates links but provides no evaluation protocol, and Table 3 contains concrete misattributions (e.g., [182], [228]).","rationale":"The reader's verdict of CONDITIONAL is appropriate, but the weakest assumption should be strengthened. The reader focuses on link stability and taxonomy consistency; these are real, and the concrete misattributions in Table 3 show the list is not trustworthy. However, the more load-bearing gap is conceptual: the paper calls a curated list a 'unified benchmark.' A benchmark requires a shared evaluation apparatus, not just pointers to resources. Even if every link resolved and every paper were correctly categorized, the central claim would still fail because the paper provides no way to reproduce a comparison. This is why the recommendation is to either soften the wording or add the missing harness. The conditionality of the verdict is unchanged: the paper can be made acceptable by revising the claim and verifying the resources.","tokens_in":42738,"tokens_out":3616,"duration_ms":33385,"concrete_test":"Access the companion GitHub repository and determine whether it contains an executable benchmark harness — e.g., scripts that download a designated dataset, split it into standard train/test sets, compute accuracy/F1, and run at least one baseline model. If the repository contains only links and READMEs, the 'first unified benchmark' claim is unsupported. In the same pass, resolve every URL in Table 3 and compare each repository's title and description with the cited reference; document mismatches such as [182] (Haplophysh vs. expose) and [228] (Microsoft/CNTK). The claim should be retracted or revised based on the outcome.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim (Section 8) is that it 'establish[es] the first unified benchmark for reproducible evaluation.' A unified benchmark requires (i) a fixed set of datasets with agreed train/test splits, (ii) common evaluation metrics computed by a shared harness, and (iii) baseline results reproducible by running the provided code. This paper provides none of these: there are no evaluation scripts, no split definitions, no metric computations, and no baseline results. The companion GitHub repository is described only as 'ongoing curating datasets and open-source implementations,' not as a benchmarking harness. A curated list of links, even if perfect, does not constitute a reproducible benchmark. The fragility is not hypothetical: Table 3 entry [182] points to https://github.com/MjafarMashhadi/Haplophysh, yet reference [182] is Saxe and Berlin's 'expose' paper; entry [228] points to Microsoft/CNTK, a general deep-learning framework, not a malicious-URL detector. Both appear to be misattributed, indicating the curation has not been validated. Additionally, Section 7.1.1 cites a nonexistent reference [404]. These errors undermine confidence that the resource list can serve as a basis for standardized baselines. The claim should be softened to 'a curated resource collection' unless the authors actually provide the benchmark harness and verify all links.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This survey reviews the field of malicious URL detection, organizing methods by data modality (URL, HTML, JavaScript, visual, and hybrid) and covering traditional machine learning, deep learning, Transformer/GNN, and LLM-based approaches. It also includes a short section on Arabic-language detection studies, tables of published algorithms with code repositories, tables of public datasets, definitions of evaluation metrics, design principles for deployed systems, and a discussion of challenges and future directions. The authors claim that the paper establishes 'the first unified benchmark for reproducible evaluation' by curating datasets and open-source implementations, and they maintain a GitHub repository of resources.","tokens_in":42981,"tokens_out":3824,"duration_ms":38615,"significance":"If its resource collection were accurate and its claims appropriately bounded, the survey would be a useful entry point for researchers: the modality-based taxonomy is a reasonable organizing principle, the coverage of Transformer/GNN/LLM-based detectors goes beyond several earlier surveys, and Tables 3 and 4 could help practitioners locate datasets and code. The paper does not present machine-checked proofs, experimental results, or a runnable evaluation harness; its main contribution is curation and classification of literature. However, the central 'unified benchmark' claim is substantially overreaching relative to the content, and the curation contains concrete errors (misattributed repository links, a nonexistent reference number, and a citation mismatch). These issues are fixable in revision, so the paper has potential as a survey after correction and recalibration of its claims.","major_comments":[{"comment":"The conclusion asserts that the paper 'establish[es] the first unified benchmark for reproducible evaluation', but a reproducible benchmark requires fixed datasets with agreed train/test splits, a shared evaluation harness computing common metrics, and baseline results obtained by running that harness. The manuscript provides none of these: Section 5.1 lists repository URLs, Section 5.2 lists dataset sources, and the GitHub repository is described as 'ongoing curating datasets and open-source implementations' rather than as a benchmark harness. The claim should be softened to a curated resource collection unless the authors supply the actual harness, split definitions, metric implementations, and baseline results.","section":"Section 8, Tables 3-4, companion repository"},{"comment":"Table 3 contains misattributed code-repository entries that undermine the curation. Entry [182] links to https://github.com/MjafarMashhadi/Haplophysh, but reference [182] is Saxe and Berlin's 'expose' paper, which is not the Haplophysh project. Entry [228] links to https://github.com/Microsoft/CNTK, a general deep-learning framework, while reference [228] is Wei et al.'s URL phishing detector with a convolutional neural network. Because the stated purpose of Table 3 is to provide usable baseline implementations, these misattributions need to be corrected and every entry should be verified before the resource can support benchmarking claims.","section":"Table 3"},{"comment":"Section 7.1.1 says that cloaking 'aims to evade the detection of phishing detection systems [404]', but the reference list contains no entry [404]; the subsequent discussion of server-side and client-side cloaking cites [243]-[249]. The in-text citation numbering appears corrupt. The authors must align the in-text citations with the final bibliography before resubmission.","section":"Section 7.1.1"},{"comment":"Section 2.2 attributes TfidfVectorizer to reference [14], which is Wang's 2022 comparison study 'Malicious url detection an evaluation of feature extraction and machine learning algorithm' rather than the documentation or original source of TfidfVectorizer. This citation mismatch, together with the errors in Table 3 and Section 7.1.1, indicates that the bibliography has not been systematically verified. Please re-check all reference-citation pairs.","section":"Section 2.2"}],"minor_comments":[{"comment":"The Result column reports accuracy values obtained on different datasets and with different experimental protocols; without a dataset/column or a cautionary note, the numbers suggest direct comparability that does not exist.","section":"Table 2"},{"comment":"The sentence 'The specific creation time, creator, and link information are shown in Table 3' should refer to Table 4, since Table 4 lists the datasets.","section":"Section 5.2"},{"comment":"Several works reviewed in the Arabic section concern Arabic spam detection and fake news in social media rather than malicious URL detection (e.g., [213], [214], [216]); consider retitling the section or explicitly framing these as adjacent tasks that inform URL-centric detection.","section":"Section 4"},{"comment":"Figures 2 and 3 appear without in-text references, even though Figure 3 presents the proposed taxonomy; add appropriate callouts.","section":"Figures 2 and 3"},{"comment":"The phrase 'We rigorous coverage of Transformer, GNNs and LLMs' is an ungrammatical fragment, and the conclusion's claim of 'three high-impact directions' is not clearly aligned with the broader set of directions in Section 7.2.","section":"Section 8"},{"comment":"The sentence 'ROW enhances the robustness...' should be 'AROW enhances the robustness...', and the stacked-matrix notation in Equation (13) should be reconciled with the separate gate equations (14)-(16).","section":"Section 3.1.2, PA/CW/AROW paragraph"}],"recommendation":"major_revision","confidential_remarks":"The paper's central contribution is presented as a 'first unified benchmark', which is likely to be the target of scrutiny by readers expecting a runnable evaluation infrastructure. Combined with the concrete citation and repository errors, the manuscript needs a verification pass before it can serve as a reliable resource; I do not see this as a fundamental scope problem, provided the authors revise the benchmark claim and validate the tables."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the genuinely useful part is the modality-based taxonomy plus the consolidated dataset and code tables; the paper does bring in recent Transformer/GNN/LLM work that earlier surveys miss. Second, the headline claim—'first unified benchmark for reproducible evaluation'—is not supported by anything in the paper, and the curation has concrete errors. I'd treat it as a useful starting point, not a benchmark.\n\nWhat it does well: organizing the literature by data modality (URL, HTML, JavaScript, visual, hybrid) is a reasonable and readable frame, and the coverage of LLM-based detection is a real gap-filler. Tables 2 and 4 give a compact picture of performance numbers and available datasets. For a newcomer, this is a convenient map.\n\nWhere it goes soft: no evaluation protocol exists. A benchmark needs fixed splits, a shared harness, metrics computed by the same code, and baseline results. This paper provides none of those; the GitHub repo is described as 'ongoing curating,' not as a harness. So the concluding claim overstates what was delivered.\n\nThe stress-test note is right about Table 3. Entry [182] points to github.com/MjafarMashhadi/Haplophysh, but reference [182] is Saxe and Berlin's expose paper. Entry [228] points to Microsoft/CNTK, but [228] is Wei et al.'s URL phishing detector. Those are not subtle distinctions. Section 7.1.1 also cites a reference [404] that does not exist. None of this is load-bearing for the survey's descriptive sections, but it matters because the resource list is the one concrete deliverable.\n\nThe Arabic section sits oddly inside the survey: it is not integrated into the modality taxonomy and the prose quality drops noticeably. It could stay with a clear rationale, or it could be cut.\n\nBottom line: for someone entering the area, this is a useful map with good recent coverage. Cite it if you need a pointer to the LLM/Transformer literature, but don't cite it as a benchmark, and verify any links before relying on them. I would send it to peer review, with a firm request to soften the benchmark claim, fix the citation and link errors, and make the Arabic section cohere.","headline":"Useful modality-organized survey with recent LLM coverage, but the 'unified benchmark' claim is unsupported and the resource tables contain concrete misattributions.","tokens_in":43499,"tokens_out":2169,"would_cite":false,"duration_ms":22223,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A modality-based taxonomy reorganizes the malicious URL detection literature, and a curated registry of datasets and open-source code is offered as the field's first unified benchmark for reproducible evaluation.","keywords":["malicious URL detection","multimodality","phishing detection","datasets","open-source implementations","large language models","graph neural networks","benchmarking"],"falsifier":"Pick a random sample of the repository URLs in Table 3 and the dataset access links in Table 4 and follow them: if a substantial fraction resolve to dead pages, relocated projects, or repositories containing code that does not match the cited algorithm, the 'unified benchmark for reproducible evaluation' claim is not met. A second check would take two papers the taxonomy assigns to different modalities and show that the same detection system could be described equally well in both categories, indicating the taxonomy's boundaries are not well-defined.","tokens_in":42540,"feed_emoji":"🛡️","tokens_out":4050,"duration_ms":35890,"temperature":0.7,"pith_summary":"This survey argues that malicious URL detection research should be organized by the data modality each method consumes—URL text, HTML structure, JavaScript behavior, visual appearance, or several at once—rather than by the machine-learning algorithm family used. It claims that prior reviews miss this distinction, ignore Transformer, graph-neural-network, and large-language-model defenses, and leave researchers without centralized lists of datasets and open-source code. To fix that, it assembles a modality-based taxonomy, a curated table of public datasets from 2016 to 2024, a table of open-source implementations from 2013 to 2025, and a live GitHub repository, calling the result the first unified benchmark for reproducible evaluation. A sympathetic reader would care because reproducible baselines are what would let the field know whether new detection methods actually beat old ones.","feed_headline":"Malicious URL detection reorganized by data modality","feed_subtitle":"New survey curates datasets and open-source code so detection methods can finally be compared on shared baselines.","key_machinery":"The carrying mechanism is a hierarchical, modality-based taxonomy defined in Section 3: URL-based, HTML-based, JavaScript-based, visual-based, and hybrid-modality detection, each with its own technical principles and representative algorithms. This taxonomy is what turns a scattered literature into a structured map. Its companion machinery is the curated registry—Table 3 listing open-source implementations with links and Table 4 listing public datasets with sources, sizes, and access methods—together with the maintained GitHub repository that keeps the registry current. The taxonomy does the analytical work; the registry does the benchmarking work.","core_discovery":"On the paper's own terms, the central discovery is that the entire malicious-URL detection literature can be re-read through a modality lens that classifies each method by its primary data source: lexical URL features, HTML document features, JavaScript execution features, visual page features, and hybrid combinations. Within each category it surveys techniques from blacklists and heuristics through classical machine learning up to Transformers, GNNs, and LLM-based prompting and fine-tuning. It further claims to establish the first unified benchmark for reproducible evaluation by compiling 15 open-source code repositories and 14 public datasets, and it distills design principles for real-world deployment from systems like Monarch. The conclusion is that this consolidated resource framework enables meaningful cross-method performance comparisons and tracking of genuine methodological advancement.","pith_inferences":["Editorial: the claim of being the 'first unified benchmark' depends on the registry staying alive; a survey can seed a benchmark, but only verified, archived copies of datasets and code would make the benchmark durable.","Editorial: the taxonomy's five modalities could be extended to a sixth—network-traffic and user-behavior features—which the paper mentions but deliberately leaves out, suggesting a natural expansion for a follow-up survey.","Editorial: a testable extension would be to run a single classifier from each modality category on a common dataset collection and report cross-modal accuracy, which would turn the curated registry into an actual leaderboard rather than a list of links."],"forward_implications":["A researcher choosing a baseline for a new malicious-URL method can now start from the curated repository list in Table 3 instead of hunting through papers for code links.","The modality taxonomy predicts which information channels—URL, HTML, JavaScript, visual—are already saturated and which combinations remain underexplored, guiding new multimodal fusion work.","LLM- and Transformer-based defenses, which earlier reviews omitted, are folded into the same modality map, so their relationship to classical methods becomes comparable.","The design principles in Section 6 give product teams a concrete checklist (accuracy, speed, scalability, adaptability, flexibility) for turning detection research into a service.","If the curated registry stays current through the GitHub repository, cross-paper performance comparisons become reproducible rather than anecdotal."],"supporting_citations":[{"why":"Establishes the algorithm-centric survey landscape that the paper distinguishes itself from.","marker":"[1]"},{"why":"A prior review that omits Transformer, GNN, and LLM approaches, motivating the paper's coverage claim.","marker":"[5]"},{"why":"Prophiler, the two-stage URL classification system that anchors discussion of lightweight URL features and content-based analysis.","marker":"[35]"},{"why":"Early multimodal malicious-web-content detection using URL, HTML, and JavaScript features; basis for the hybrid modality category.","marker":"[40]"},{"why":"PhishDef, a URL-only lexical system, supplies the ISCX-URL-2016 dataset usage mentioned in the data section.","marker":"[53]"},{"why":"URLNet, an open-source deep learning URL representation model, is a load-bearing entry in the code-repository table.","marker":"[62]"},{"why":"TransURL, a multi-layer transformer method, demonstrates the paper's coverage of Transformer-based defenses.","marker":"[74]"},{"why":"VisualPhishNet, a visual similarity method, supplies a code repository and anchors the visual modality category.","marker":"[147]"},{"why":"LLM-based one-shot URL classification, representing the large-language-model approaches included in the survey.","marker":"[166]"},{"why":"Monarch, the large-scale real-time URL filtering service, underpins the design-principles and service-framework discussion.","marker":"[180]"}],"fun_headline_variants":["Malicious URL detection remapped by data modality","New survey groups URL defenses by input channel","Modality-based taxonomy for URL threat research","Benchmarking URL defenses: curated datasets and code","How URL detectors use text, visuals, and scripts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's central benchmark claim assumes that the curated dataset and code-repository links are correct, complete, and stable, and that each cited work can be assigned unambiguously to exactly one modality; it verifies neither the links nor the assignment rules.","fun_headline_variants_meta":{"raw":{"variants":["Malicious URL detection remapped by data modality","New survey groups URL defenses by input channel","Modality-based taxonomy for URL threat research","Benchmarking URL defenses: curated datasets and code","How URL detectors use text, visuals, and scripts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000211,"raw_usage":{"total_tokens":1441,"prompt_tokens":1000,"completion_tokens":441,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":616,"completion_tokens_details":{"reasoning_tokens":370}},"tokens_in":616,"tokens_out":441,"duration_ms":4585,"temperature":1.0,"reasoning_tokens":370,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:02:53.455759+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Pick a random sample of the repository URLs in Table 3 and the dataset access links in Table 4 and follow them: if a substantial fraction resolve to dead pages, relocated projects, or repositories containing code that does not match the cited algorithm, the 'unified benchmark for reproducible evaluation' claim is not met. A second check would take two papers the taxonomy assigns to different modalities and show that the same detection system could be described equally well in both categories, indicating the taxonomy's boundaries are not well-defined.","supporting_citations":[{"cited_title":"A hybrid transformer ensemble approach for phishing website detection, in: 2023 International Conference on Self Sustainable Artificial Intelligence Systems (ICSSAS), pp","cited_arxiv_id":null,"evidence_quote":"TransURL, a multi-layer transformer method, demonstrates the paper's coverage of Transformer-based defenses."}],"review_version":1}