{"id":"08c2dc72-4da4-438c-9680-f3f720cd7ead","arxiv_id":"2505.21406","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A weighted behavior DFA built from 10 malicious JavaScript sequences produces match percentages that label traces as benign, partially malicious, or malicious, but no accuracy metrics validate those labels.","lead":"These authors build a deterministic finite automaton from 10 malicious JavaScript execution traces, attach expert-chosen weights to individual behaviors, and score new traces by how closely they match known attack paths. A generalist might care because this is a transparent, rule-based alternative to black-box machine learning classifiers, though the evaluation does not yet show it detects real threats.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The match percentage in §3 is not a valid similarity measure: its denominator is the total weight of the nearest final path from q0, so identical prefixes can get different scores and longer overlaps can score lower; the partial-similarity claim rests on an internally inconsistent metric.","rationale":"I read the paper's central claim as: the behavior DFA, together with weighted match percentages, successfully identifies both exact and partial similarities to known malicious JavaScript behaviors, enabling classification into benign/partially/fully malicious. For that claim to hold, the match percentage must at least rank sequences by their degree of overlap with known attack patterns. The formula in §3 fails this in principle: the denominator depends on the nearest final state's total path weight, not on the observed sequence, so equivalent observed prefixes can receive different scores and longer overlaps can receive lower scores. This is not merely a missing evaluation; it is a structural defect in the proposed metric. The case study in §4 (sequence 1058, labeled benign, scored 18.75%) illustrates the downstream consequence, but the problem is identifiable purely from the definitions. The reader's weakest assumption also targeted the match percentage, but framed as lack of validation; my concern is that the metric is internally inconsistent. I agree with the REJECT verdict: even with held-out data, the reported match percentages would not support the 'partial similarity' claim unless the formula is redefined. The concrete test above would settle whether the metric is salvageable by e.g. using residual distance or a different normalization.","tokens_in":9996,"tokens_out":11616,"duration_ms":124779,"concrete_test":"Implement the described DFA construction and BFS on two malicious sequences S1=[7,5] and S2=[7,1,5,1,5,1] with behavior weights 7→3, 5→3, 1→2. Run sequence X=[7] and sequence Y=[7,1]. If X is scored 50% and Y is scored 33.3% as predicted, then the match percentage does not monotonically reflect overlap length, invalidating it as a partial-similarity measure. An independent re-derivation of Eq. (1) from the §3 definitions should also check whether the denominator is ever computed from the current state; if the paper intended residual distance, the formula and examples need correction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 defines Match Percentage = Total Matched Behavior Weight / Total Weight to Nearest Final State × 100. The denominator is the weight of the path from q0 to the final state selected by BFS from the sequence's last state, not a property of the observed sequence alone. In the paper's own example (sequence [7,5]), the denominator is the full path [7,5,1,7,5,1] (weight 16) even though BFS from q2 to q6 has residual weight 10. Because the denominator includes the matched prefix plus an attack-specific residual, two sequences with identical observed prefixes can receive different percentages if their nearest final states have different total path weights; conversely, a longer matched prefix can yield a lower percentage when the nearest final state's total path is longer. For instance, in a DFA built from two attacks sharing prefix [7] (weight 3), with attack A = [7,5] (total 6) and attack B = [7,1,5,1,5,1] (total 15), the sequence [7] is scored 3/6 = 50% (nearest final A), while [7,1] is scored 5/15 ≈ 33% (nearest final B). The longer overlap receives a lower score, contradicting the claim that higher match percentages indicate greater similarity to known malicious patterns. The system's classification of partially malicious sequences is therefore based on a metric that does not measure what the central claim asserts.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a weighted DFA ('behavior DFA') for JavaScript malware detection. Behavior sequences from a proprietary dataset are encoded as transitions, with expert-assigned weights; a match percentage is computed as the ratio of matched behavior weight to the total weight of the path from the initial state to the nearest final state. Sequences are classified as benign, partially malicious, or malicious based on exact matches and match percentages. The system is evaluated on 1,058 sequences, reporting 10 malign, 288 partially malign, and 760 benign classifications, with a case study for one sequence (ID 1058).","tokens_in":10276,"tokens_out":2743,"duration_ms":29319,"significance":"The motivation is reasonable: an interpretable, deterministic alternative to black-box ML classifiers could be useful in security operations. The paper's stated strengths are transparency, expert-driven weighting, and partial-match analysis for novel threat detection. However, the current manuscript does not provide a valid evaluation of detection performance: the reported '10 malign' detections are the same 10 sequences used to build the DFA, no precision/recall or baseline comparisons are given, and the match-percentage metric is internally inconsistent as a similarity measure. If the metric and evaluation were repaired, the idea of weighted automata for behavioral sequence classification could be a modest contribution, but as presented the central claims are unsupported.","major_comments":[{"comment":"The evaluation is circular with respect to the central detection claim. The paper states that the behavior DFA was built from 10 malicious sequences and then reports 'classifying 10 as malign' as 'consistent with expectations'. Since the DFA's accepting paths are constructed directly from those 10 sequences, exact-match detection of those sequences is guaranteed by construction and carries no evidential weight for detection ability. No held-out malicious sequences, independent attack families, or cross-validation are used, so the claim that the system 'detects' known malicious patterns is not tested.","section":"§4 (Results and Evaluation)"},{"comment":"The match-percentage formula is not a valid similarity measure for the observed sequence. The denominator, 'Total Weight to Nearest Final State', is the total weight of the path from q0 to the nearest final state, which includes the matched prefix plus the remaining attack-specific residual. Consequently, two sequences with identical observed prefixes can receive different percentages if their nearest final states have different total path weights, and a longer matched prefix can receive a lower percentage than a shorter one. For example, in a DFA with attacks [7,5] (total 6) and [7,1,5,1,5,1] (total 15), the prefix [7] scores 50% while [7,1] scores 33%, contradicting the claim that higher percentages indicate greater similarity to known malicious patterns. The partial-similarity classification therefore rests on a metric that does not measure what the central claim asserts.","section":"§3 (Match Percentage formula) and §4 (case study)"},{"comment":"No accuracy metrics, label-based evaluation, or baseline comparison are provided. The evaluation reports only counts of classifications (10 malign, 288 partially malign, 760 benign) and match-percentage distributions; there is no precision, recall, F1-score, false-positive rate, or comparison against existing detectors such as Xue et al. [26] or ML-based approaches. The case study explicitly notes that sequence ID 1058 is labeled benign in the dataset but is scored at 18.75% and classified as partially malign, so the score does not track ground-truth labels. Without label-based metrics, the abstract's claim that the system 'detect[s] and classify[ies] threats effectively' is unsupported.","section":"§4 (Evaluation criteria and case study)"},{"comment":"The decision boundary between 'benign' and 'partially malicious' is not defined. Section 3 states that a sequence is 'benign if there is no strong similarity', but no threshold for 'strong similarity' is given, nor is a cutoff match percentage specified. Section 4 classifies 288 sequences as partially malign with percentages as low as 18.75%, and 760 as benign, but the rule that separates these groups is absent, making the classification non-reproducible and untestable.","section":"§3 (Classification decision rule)"}],"minor_comments":[{"comment":"The text attributes the 'register automata and pushdown systems' work to 'Lemberger et al. [24]', but reference [24] is authored by Touili; the citation/name mismatch should be corrected.","section":"§2 Related Work"},{"comment":"The weight assignments for behaviors are described only by example (weights 2, 3, 1, 4, 5), but the full mapping of behavior identifiers to weights is not provided. Without that table, the worked example for the sequence [7,5] and the later case-study percentages cannot be independently verified.","section":"§3 Methodology"},{"comment":"The phrase 'to known threats to known threats' appears as a duplicated fragment and should be corrected.","section":"§4 Results and Evaluation"},{"comment":"There are several typographical errors, including 'subquences' (should be 'subsequences') and 'it's capabilities' (should be 'its capabilities').","section":"§5 Conclusion"},{"comment":"The claim that the system achieves 'high precision' is not supported by any precision measure in the paper; this wording should be removed or replaced with a precise, computed metric.","section":"§4 Results and Evaluation"}],"recommendation":"reject","confidential_remarks":"The manuscript is not ready for publication in its current form. The most serious issue is that the evaluation is circular (the 10 detected malicious sequences are the exact sequences used to build the automaton), which no amount of local revision can fix within the paper's current scope. The additional problem of an internally inconsistent match-percentage metric would require redefining the core similarity measure and re-running the entire evaluation, essentially a new study. The proprietary, unavailable dataset further limits reproducibility. For the editor: I concur with the reader's high-confidence reject recommendation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: this paper has a legitimate incremental idea—weighted automata over JavaScript behavior sequences—but the reported evaluation is close to vacuous, and the partial-match score it centers on is not a coherent similarity measure. I'd send it to a referee only because the idea is worth straightening out, not because the current claims hold.\n\nWhat's new: Xue et al. build DFAs that abstract malicious JavaScript behavior, but they don't assign expert weights to behaviors or use a match percentage to rank partial overlaps. That's a real, if modest, extension. The system is transparent and incrementally updatable, which is a genuine advantage over black-box ML. The authors also state plainly that the DFA was built from 10 malicious sequences, so there's no attempt to hide the circularity at that level.\n\nSoft spots: First, the evaluation recognizes the 10 sequences used to construct the automaton. Reporting 10 malign detections is structural, not evidence. There are no held-out malicious sequences, no precision/recall, no false-positive rate, no baseline comparison. Second, the match percentage formula is internally inconsistent as a similarity measure. The denominator is the total weight of the path from q0 to the nearest final state, not a property of the observed sequence. Two sequences with the same observed prefix get different scores if their nearest final states differ, and a longer overlap can score lower than a shorter one. The paper's own example works numerically, but the measure doesn't do what the central claim needs it to do. Third, the case study sequence 1058 is labeled benign in the dataset but gets an 18.75% match and is classified partially malign. The authors mention this without treating it as a false positive, which undercuts the 'effective detection' language.\n\nWho is this for? Researchers working on interpretable behavior-based malware detection might find the weighted automaton idea a useful starting point. Nobody should cite it as a working detection method.\n\nRecommendation: I'd let it into peer review with a 'major revision' expectation—the method is salvageable, but it needs a held-out evaluation, proper detection metrics, and a redefinition of match percentage that actually reflects similarity. Right now it's a prototype, not a result.","headline":"A clear but unevaluated prototype: the weighted DFA idea is new, the match-percentage metric has a load-bearing flaw, and the detection claim rests on recognizing the training set.","tokens_in":10825,"tokens_out":2660,"would_cite":false,"duration_ms":28031,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68Q45","68M25"],"pacs":[],"model":"deepseek-v4-flash","headline":"A weighted deterministic finite automaton over JavaScript execution behaviors classifies scripts as benign, partially malicious, or malicious by the degree to which their action sequences match known attack paths.","keywords":["JavaScript malware detection","deterministic finite automaton","behavior-based classification","partial match detection","behavior weighting","dynamic analysis","web security"],"falsifier":"Compute precision and recall for the three-way classification by comparing the behavior-DFA predictions against the dataset's labels; if sequences scored 18.75% or 30% are no more likely to be labeled malicious than sequences scored 0%, the match percentage is not a valid maliciousness proxy and the central claim fails.","tokens_in":9803,"feed_emoji":"🛡️","tokens_out":7486,"duration_ms":77320,"temperature":0.7,"pith_summary":"The paper proposes a JavaScript malware detector that models known malicious execution traces as a weighted deterministic finite automaton, the behavior DFA. New scripts are run through the automaton: reaching a final state marks a full match, while ending elsewhere triggers a breadth-first search to the nearest final state and computes a match percentage from the ratio of matched behavior weight to the weight of the shortest path to that state. The claim is that this percentage separates benign scripts from partially malicious ones and lets the system flag unknown variants that share only part of an attack's behavior. A sympathetic reader would care because the method is transparent and does not depend on large training sets, unlike black-box classifiers, and it can be extended by simply adding new attack sequences.","feed_headline":"Weighted automaton flags partial matches to JavaScript malware","feed_subtitle":"A behavior DFA scores how closely script actions match known attack patterns, flagging partial matches.","key_machinery":"The central object is the behavior DFA, a deterministic finite automaton $(Q, \\Sigma, \\delta, q_0, F)$ whose alphabet is a set of JavaScript runtime behaviors such as \"Add Event Handler\", \"Set Callback\", \"Find DOM Element(s)\", and \"Send Data\", and whose transitions carry expert-assigned weights. A path from the initial state to a final state encodes a known malicious pattern. For a new sequence that does not end in a final state, breadth-first search finds the nearest final state; the match percentage is the total weight of transitions the sequence actually followed up to its last state divided by the total weight of the shortest path from $q_0$ to that nearest final state, times 100. This ratio is the mechanism that turns automaton membership into graded risk.","core_discovery":"The paper's central claim is that a DFA whose transitions are labeled by weighted JavaScript behaviors, constructed from known malicious sequences, can classify unseen execution traces into benign, partially malicious, and fully malicious categories. Exact matches reach a final state; partial matches end in a non-final state and receive a match percentage equal to the total weight of matched behaviors divided by the total weight of the shortest path from the initial state to the nearest final state, multiplied by 100. On a proprietary dataset of 1,058 sequences, the system classified 10 as malign, 288 as partially malign, and 760 as benign, with partial-match percentages clustered at 18.75%, 30%, 37.5%, and 50%. The authors take the spread of partial scores as evidence that the system measures diverse threat levels from partially matching behaviors and therefore adapts to emerging attacks while keeping decisions transparent.","pith_inferences":["The paper does not report precision or recall for the partial buckets, so an immediate test would be checking whether the 288 partially malicious sequences are enriched in malicious labels compared with the 760 benign ones; the case study of sequence ID 1058, dataset-labeled benign but scored 18.75% partially malign, shows the score can disagree with ground truth.","Because the score depends on expert-assigned behavior weights, a sensitivity analysis varying those weights would reveal whether the observed bucket boundaries are stable or artifacts of the chosen weights.","The same automaton could be turned into an alarm system by choosing a match-percentage threshold, but the paper does not specify how such a threshold should be selected or calibrated.","A natural extension is to combine the weighted DFA with subsequence mining so that a short malicious fragment inside a long benign trace, rather than only a prefix, drives the score."],"forward_implications":["A script that shares only the opening actions of a known attack, such as adding an event handler, can be scored at 18.75%, giving an analyst a concrete and explainable reason to monitor it.","Because the automaton is built from known malicious sequences rather than trained, adding a newly discovered attack pattern is a matter of extending the DFA, not retraining a model.","The weighting of behaviors means high-risk actions like dynamic code injection or data exfiltration dominate the score, so a sequence containing a high-risk step can be flagged even if its other actions are common in benign scripts.","Partial-match percentages give operators a graded list of suspicious scripts, allowing them to prioritize follow-up analysis before a full attack pattern is reached.","The authors point to subpattern detection as the next step, which would let the same DFA find malicious action sequences embedded inside larger benign-looking workflows."],"supporting_citations":[{"why":"Supplies the DFA-based attack-behavior modeling for malicious JavaScript that the behavior DFA extends with weights.","marker":"[26]"},{"why":"Provides the partial-behavior-overlap rationale for generalizing from known attacks to novel variants.","marker":"[15]"},{"why":"Defines richer automata models for malware specification whose complexity motivates the simpler weighted DFA.","marker":"[24]"},{"why":"Establishes dynamic execution and behavior extraction as the detection basis for drive-by-download attacks.","marker":"[3]"},{"why":"Represents the static-analysis baseline that struggles with obfuscated scripts, motivating behavior-based detection.","marker":"[4]"},{"why":"Prior FSA-based anomaly detection over system call traces that the paper adapts to weighted JavaScript behaviors.","marker":"[20]"}],"fun_headline_variants":["Behavior DFA scores script actions to catch partial malware","Weighted DFA rates partial malware matches transparently","JavaScript malware caught by weighted behavior automaton","Partial match scores reveal hidden JavaScript malware"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the match percentage the system computes, the share of matched behavior weight along the way to the nearest known attack pattern, actually tracks how malicious a script is; the paper never validates this against the dataset's benign and malicious labels for the partial buckets.","fun_headline_variants_meta":{"raw":{"variants":["Behavior DFA scores script actions to catch partial malware","Weighted DFA rates partial malware matches transparently","JavaScript malware caught by weighted behavior automaton","Partial match scores reveal hidden JavaScript malware"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000192,"raw_usage":{"total_tokens":1306,"prompt_tokens":864,"completion_tokens":442,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":480,"completion_tokens_details":{"reasoning_tokens":384}},"tokens_in":480,"tokens_out":442,"duration_ms":4983,"temperature":1.0,"reasoning_tokens":384,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:28:42.303723+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute precision and recall for the three-way classification by comparing the behavior-DFA predictions against the dataset's labels; if sequences scored 18.75% or 30% are no more likely to be labeled malicious than sequences scored 0%, the match percentage is not a valid maliciousness proxy and the central claim fails.","supporting_citations":[{"cited_title":"2013 International Conference on Security and Cryptography (SECRYPT) pp","cited_arxiv_id":null,"evidence_quote":"Provides the partial-behavior-overlap rationale for generalizing from known attacks to novel variants."},{"cited_title":"In: Proceedings of the 20th USENIX Conference on Security","cited_arxiv_id":null,"evidence_quote":"Represents the static-analysis baseline that struggles with obfuscated scripts, motivating behavior-based detection."},{"cited_title":"In: Proceedings 2001 IEEE Sym- posium on Security and Privacy","cited_arxiv_id":null,"evidence_quote":"Prior FSA-based anomaly detection over system call traces that the paper adapts to weighted JavaScript behaviors."}],"review_version":1}