{"id":"0e137040-8553-4441-a03e-b12269bb51e6","arxiv_id":"2501.14008","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"WAFBOOSTER combines a shadow model, an RNN payload generator, and automatic signature extraction to harden web application firewalls, but its headline rejection-rate improvement is measured on the same payloads used to build the signatures.","lead":"WAFBOOSTER is a proposed system that automatically finds mutated attack payloads that slip past web application firewalls, then writes new blocking rules for them. The paper claims it raises the rejection rate of mutated attacks from 21% to 96%, but that improvement is measured on the exact payloads used to create the new rules.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table V evaluates signatures on the same bypassing payloads used to generate them, so the reported 21% to 96% TRR improvement is an in-sample artifact rather than evidence of generalized WAF hardening.","rationale":"The reader's verdict of REJECT is well founded, but the formal weakest_assumption, namely the stable-unique-substring assumption in Section III-E, is not the most load-bearing problem. The more fundamental issue is that Table V scores signatures on the exact payloads from which they were generated, so the headline 21% to 96% TRR and 0% FRR are expected even if every component works as described. This is an evaluation-protocol flaw independent of the signature producer's correctness. I therefore agree with the overall rejection but identify a different central weakness, hence partial agreement. A held-out re-evaluation is the one check that could convert the claim from an in-sample demonstration into evidence of generalized WAF hardening. No independent code, artifacts, or machine-checked proofs are provided, and the small case studies in Code 1 are suggestive but cannot substitute for a controlled evaluation.","tokens_in":23197,"tokens_out":4278,"duration_ms":38470,"concrete_test":"Re-run the Section IV-F experiment with a strict held-out split: generate signatures on one half of the bypassing payloads for each (WAF, attack) pair, then measure TRR and FRR on the held-out half, or stronger, on a fresh batch of mutations generated after the signatures are fixed. If TRR on unseen payloads remains near 96% with FRR=0, the claim survives; if it drops toward the \"Before\" values, the reported improvement is an artifact of in-sample evaluation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section IV-F and Table V is not supported because the evaluation is circular. The signature producer (Algorithm 2, Section III-E) consumes the bypassing malicious payloads, i.e., those that evaded both the shadow model and the real WAF, and builds regex signatures by clustering substrings of exactly those payloads. Table V then measures TRR \"after\" by applying the generated signatures to \"bypassing adversarial payloads and benign payloads from the original dataset\" (Section IV-F). That is, the test set is the same set from which the signatures were derived. A regex constructed to match a payload's own important tokens will, by construction, match that payload; the observed jump from roughly 16-34% to 90-100% TRR therefore does not demonstrate that unseen mutated payloads are blocked. The FRR=0 result is also uninformative because no new benign traffic or mutated benign traffic is tested, and the signatures are selected for high specificity on the training set. The \"basic assumption\" in Section III-E, that a stable unique substring survives mutation, is a real prerequisite, but it is not the decisive flaw: even a perfect signature extractor would produce the same inflated numbers under this evaluation protocol. The shadow-model generalization results in Tables III and IV further show that generated payload sets are highly WAF-specific, making cross-WAF generalization claims weaker.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"WAFBOOSTER is a learning-based framework that aims to automatically strengthen Web Application Firewalls (WAFs) against mutated malicious payloads. The system trains a CNN shadow model to imitate a target WAF, uses an RNN-based generator plus a payload corrector to produce mutated payloads that bypass the shadow model, and finally runs a signature producer that scores substrings, clusters them by edit distance, and emits simplified regular-expression signatures to update the WAF's rules. The paper evaluates the pipeline on eight real-world WAFs against SQL injection, XSS, and command injection, and claims that the generated signatures raise the true rejection rate (TRR) of mutated payloads from 21% to 96% with no false rejections, and that WAFBOOSTER's payload generation achieves a false acceptance rate three times better than state-of-the-art mutation/generation tools.","tokens_in":23485,"tokens_out":5214,"duration_ms":46407,"significance":"If the reported results were valid, WAFBOOSTER would be a practically valuable contribution: it would automate the discovery of WAF bypasses and the generation of new signatures without per-attack expert knowledge, which is a meaningful step beyond manual rule maintenance and prior mutation/generation tools that target only specific attack types or ML-based WAFs. The idea of combining a shadow model, RNN-based payload generation, and substring-based signature synthesis is coherent and worth exploring. However, the current evidence does not support the headline numbers: the signature evaluation is in-sample, the shadow model is validated only at the aggregate level, and the state-of-the-art comparison is not a fair apples-to-apples benchmark. With a proper held-out evaluation and clarified experimental protocol, the underlying approach could still be of interest to the security testing community.","major_comments":[{"comment":"The main evaluation of signature effectiveness is circular. In Section III-E, the signature producer (Algorithm 2) consumes the 'bypassing malicious payloads against WAF' and builds regular expressions by extracting common substrings from exactly those payloads. Section IV-F then measures TRR 'after' by applying the generated signatures to 'bypassing adversarial payloads and benign payloads from original dataset.' A regex constructed from a payload's own tokens will match that payload by construction, so the observed increase from roughly 16-34% to 90-100% TRR does not demonstrate that unseen mutated payloads are blocked. The FRR=0 result is likewise uninformative because the benign set is the same set used to select for specificity. Please re-run the evaluation on a disjoint set of mutated payloads (e.g., payloads generated with held-out seeds or from a separate mutation campaign) and report TRR/FRR on those held-out samples.","section":"Section IV-F, Table V"},{"comment":"The scoring functions DS, RS, HS, and TS in Algorithm 2 are defined in terms of f, described as 'the discrimination result of WAF.' The paper does not specify whether f is the real WAF or the shadow model. If f is the real WAF, the signature producer needs four black-box queries per token per payload, which is costly and contradicts the authors' earlier rationale for building a shadow model. If f is the shadow model, the identified 'important' tokens are only important for the shadow model, and the paper provides no evidence that these tokens transfer to the real WAF's decision boundary. This ambiguity directly affects the central claim that the generated signatures repair real WAF security gaps, so it must be resolved and justified.","section":"Section III-E, Algorithm 2"},{"comment":"The shadow model is validated only through aggregate TRR and TAR values in Table I. Because the payload generator is optimized to produce inputs that bypass the shadow model's per-payload predictions, aggregate agreement is insufficient: two classifiers with identical TRR/TAR can disagree on the specific adversarial payloads being generated. The paper should report per-sample agreement between the shadow model and the real WAF, especially on generated bypassing payloads, or otherwise demonstrate that a payload that evades the shadow model also evades the real WAF. Without this, the transfer of the generated adversarial payloads from shadow model to real WAF is an unverified assumption.","section":"Section III-B and IV-B"},{"comment":"The comparison with WAF-A-MoLE is not a fair state-of-the-art comparison. The authors removed WAF-A-MoLE's mutation-prioritization step and state that the adapted version 'may not fully utilize WAF-A-MoLE's optimization techniques.' The subsequent claim that WAFBOOSTER's FAR is 'more than 3×' that of WAF-A-MoLE/SQLMap is therefore not a valid measure of superiority. Additionally, the Abstract says 'false acceptance rate 3X lower than state-of-the-art methods,' while Section IV-D reports a FAR of 84% for WAFBOOSTER versus 24% and 22% for the baselines, which is higher, not lower. These contradictory statements must be reconciled, and the SOTA comparison should either run WAF-A-MoLE as intended or clearly explain and justify the adaptation and its effect on the comparison.","section":"Section IV-D"}],"minor_comments":[{"comment":"The descriptions of cross-site scripting and command injection are somewhat run-on; splitting them into shorter sentences would improve readability.","section":"Section II-A"},{"comment":"The Smith-Waterman algorithm is credited to reference [44], which is an efficient GPU implementation paper; please cite the original Smith-Waterman paper or a standard algorithm reference instead.","section":"Section III-E"},{"comment":"The phrase 'challenge collapsar attack' appears to be a typo or undefined technical term; please clarify what protection is being disabled.","section":"Section IV-A"},{"comment":"The sentence 'the shadow models are individually limited on specific attacks, cannot not fully reflect the behavior of the target W AF' contains a double negative; it should read 'cannot fully reflect.'","section":"Section VI"},{"comment":"The header layout of Table I is confusing because the 'W AF' and 'Shadow model' columns are not clearly separated; realigning the columns would help readers compare TRR and TAR across models.","section":"Table I"},{"comment":"Figure 3 is hard to parse because the three attack types appear to be represented by nearly identical-looking payload strings; adding labeled panels or distinct axes for each attack type would make the figure comprehensible.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The core idea is interesting and the system design is coherent, but the main evaluation is currently circular: signatures are tested on the same payloads used to generate them. The authors should be asked to provide a proper held-out evaluation, clarify whether the scoring functions use the real WAF or the shadow model, and present a fair comparison with WAF-A-MoLE. If these points are addressed with new experiments, the paper could become a useful contribution to WAF testing and hardening."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, quick take on WAFBOOSTER. The headline number—TRR up from 21% to 96%—is not supported by the evaluation as it stands. Table V measures signatures on the same bypassing payloads that were used to build them: Section III-E feeds bypassing payloads into the signature producer, and Section IV-F then tests on \"bypassing adversarial payloads\" from the original dataset. A regex fitted to a payload's own tokens will match that payload by construction, so the post-signature rejection rate is largely predetermined. This is the paper's central claim, so the flaw is load-bearing.\n\nThat said, the paper has real merits. The end-to-end pipeline—CNN shadow model, GRU payload generator, edit-distance corrector, and Smith-Waterman-based regex extraction—is new as a WAF-hardening framework, and the goal of automatically generating signature rules from discovered bypasses is practically valuable. The authors evaluate on eight real WAFs across three attack types, and the concrete bypass cases (e.g., `ca$1t /etc/pa$5sswd` and `/*! union*/%0Aselect`) are plausible real evasions. The shadow-model generalization experiments (Tables III–IV) are a good idea, even if the results are mixed.\n\nThe soft spots beyond the circularity: the shadow model is only validated with aggregate TRR/TAR, not per-sample agreement with the target WAF, so the generator might be optimizing against a model that disagrees with the WAF on individual payloads. The SOTA comparison is also tilted: WAF-A-MoLE is adapted by removing its prioritization step, which is its core mechanism, and the authors concede it may not be used to full effect. The FRR=0 claim is uninformative because no fresh benign traffic or mutated benign payloads are tested. Minor presentation issues: Figures 3 and 4 appear garbled, and Table I is hard to parse.\n\nThe \"basic assumption\" of a stable unique substring is a real prerequisite, but it isn't the decisive problem—even a perfect signature extractor would produce the same inflated numbers under this evaluation protocol. The fix is straightforward: hold out a set of mutated payloads not used for signature generation (ideally produced by a different seed set), report TRR on that, and test FRR on mutated benign traffic.\n\nWho is this for? Researchers working on WAF testing and automated rule generation. The pipeline idea is worth citing as related work, but the quantitative claims need a redo. I'd send it to peer review—it's a legitimate system paper with a serious methodological gap that reviewers should catch and push the authors to fix.","headline":"WAFBOOSTER's end-to-end pipeline is novel and worth knowing, but the headline 21% to 96% TRR claim is an in-sample artifact because signatures are evaluated on the same bypassing payloads used to build them.","tokens_in":24019,"tokens_out":3638,"would_cite":false,"duration_ms":30165,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"WAFBOOSTER claims an automatic pipeline that finds mutated payloads bypassing WAFs and generates signatures that lift true rejection from 21% to 96% with zero false rejections.","keywords":["web application firewall","SQL injection","cross-site scripting","command injection","mutated payloads","signature generation","shadow model","true rejection rate"],"falsifier":"Take the set of payloads that bypass a WAF under one mutation family, remove every substring that also appears in a large benign corpus, and check whether any common substring remains across the group; if for some family nothing remains, the basic assumption behind the signature producer fails and the reported combination of high true rejection with zero false rejections cannot generalize to that family.","tokens_in":23011,"feed_emoji":"🛡️","tokens_out":14294,"duration_ms":114485,"temperature":0.7,"pith_summary":"WAFBOOSTER tries to prove that web application firewalls can be hardened automatically against payload mutations that slip past their rules. The method trains a shadow model to imitate the target WAF, generates mutated malicious payloads that evade both the shadow model and the real WAF, filters those mutations so they stay executable, and then mines signatures from the surviving bypassing payloads. The headline result is that adding these signatures raises the true rejection rate of mutated SQL injection, cross-site scripting, and command injection payloads from 21% to 96% on eight WAFs, with no benign traffic falsely rejected. If the result holds, WAF rule maintenance shifts from hand-written expert rules to a learning loop that can keep pace with new evasions.","feed_headline":"WAF patch generation lifts mutated attack blocking from 21% to 96%","feed_subtitle":"Generated rules lift mutated-attack blocking to 96% while keeping false rejections at zero.","key_machinery":"The load-bearing mechanism is the signature producer built on four token scoring functions. For each token in a payload, it compares the WAF decision before and after deleting the token, replacing it with a fixed placeholder, removing everything after it, and removing everything before it; tokens whose scores indicate a large effect on the WAF's decision are treated as important. Important tokens are clustered by edit distance, and within each cluster a common subsequence is extracted with a sequence-alignment dynamic program using a match reward of 1 and a gap penalty of 0.8. The output is a simplified regular expression, defined in the paper as a regular expression whose only wildcard is `\\S*`, meaning any run of non-whitespace characters, so a signature such as `\\S*select\\S*from\\S*` matches the attack skeleton that survives across mutations.","core_discovery":"The paper's central claim is that the vulnerable gap in a WAF's rule set can be found and patched automatically, without per-attack expert rule writing. WAFBOOSTER trains a CNN shadow model to imitate a black-box WAF, feeds it mutated payloads produced by a sequence generator built from gated recurrent units, repairs or discards payloads whose tokens are too far from known attack keywords, and then learns signatures from the payloads that bypass both the shadow model and the real WAF. The signature producer scores each token by how much the WAF decision changes under deletion, replacement, head truncation, and tail truncation, clusters the important tokens by edit distance, and emits a simplified regular expression. In the evaluation, adding these signatures to eight WAFs raises the true rejection of mutated payloads from 21% to 96%, with false rejections at zero.","pith_inferences":["Inference: if the pipeline is run repeatedly after each signature update, an adversary can learn which substring the signature depends on and mutate that substring too, so the 96% figure is best read as a snapshot of one hardening cycle rather than a permanent security level.","Inference: the same generate-correct-signature loop should transfer to other structured attack families, such as LDAP, XML/XPath, or NoSQL injection, wherever a shadow model can imitate the detector and a validity check can enforce the target language; the architecture has no attack-specific component except the training data.","Inference: the zero-false-rejection result is measured on the evaluation's benign set, so a deployment should re-estimate false rejections on live benign traffic before treating the generated signatures as safe to enable in production."],"forward_implications":["If the reported measurements are right, WAFBOOSTER's signatures lift true rejection of mutated SQL injection, cross-site scripting, and command injection payloads to at least 89% (average 96%) across eight real WAFs, while false rejections stay at zero.","WAF hardening becomes an automated loop: generate mutated payloads, keep only valid ones, test them against the shadow model and the real WAF, and derive new rules from the bypassing set, with no per-attack mutation operators designed by hand.","A shadow model trained for one WAF can be reused for another WAF with only small changes in true rejection rate, which lowers the cost of applying the method to many WAFs.","The generated bypassing payloads are harder for WAFs to catch than those from the two comparison baselines; in the reported SQL-injection comparison the false-acceptance rate reaches 84%, versus 24% and 22% for the mutation-based and penetration-testing baselines."],"supporting_citations":[{"why":"Supplies the mutation operators and the mutation-based baseline against which the generated payloads are compared in Section IV-D.","marker":"[8]"},{"why":"Supplies the gated recurrent unit used by the payload generator to produce mutated payload sequences.","marker":"[40]"},{"why":"Supplies the edit-distance metric used by the payload corrector and by the signature clustering step.","marker":"[43]"},{"why":"Supplies the sequence-alignment dynamic program used by the signature producer to find common subsequences.","marker":"[44]"},{"why":"Supplies the benign HTTP request corpus used to measure true acceptance and false rejection rates.","marker":"[53]"},{"why":"Supplies the penetration-testing baseline whose generated payload sets are compared in Section IV-D.","marker":"[56]"}],"fun_headline_variants":["Auto-generated rules lift mutated attack blocking to 96%","WAFBOOSTER finds and fixes WAF blind spots, blocking 96% of mutated payloads","From 21% to 96%: automatic WAF patching stops mutated attacks","AI-generated signatures close WAF bypasses, zero false rejections"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The signature-generation step rests on the paper's basic assumption that a single payload substring remains unchanged through the mutation process and is unique enough to use as a signature without matching benign traffic; if a mutation family leaves no such stable, unique substring, the generated signatures will either miss the evasions or reject legitimate requests.","fun_headline_variants_meta":{"raw":{"variants":["Auto-generated rules lift mutated attack blocking to 96%","WAFBOOSTER finds and fixes WAF blind spots, blocking 96% of mutated payloads","From 21% to 96%: automatic WAF patching stops mutated attacks","AI-generated signatures close WAF bypasses, zero false rejections"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000627,"raw_usage":{"total_tokens":2906,"prompt_tokens":955,"completion_tokens":1951,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":571,"completion_tokens_details":{"reasoning_tokens":1866}},"tokens_in":571,"tokens_out":1951,"duration_ms":12046,"temperature":1.0,"reasoning_tokens":1866,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:33:53.038310+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the set of payloads that bypass a WAF under one mutation family, remove every substring that also appears in a large benign corpus, and check whether any common substring remains across the group; if for some family nothing remains, the basic assumption behind the signature producer fails and the reported combination of high true rejection with zero false rejections cannot generalize to that family.","supporting_citations":[{"cited_title":"Waf-a-mole: evading web application firewalls through adversarial machine learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the mutation operators and the mutation-based baseline against which the generated payloads are compared in Section IV-D."},{"cited_title":"Fast string correction with levenshtein automata,","cited_arxiv_id":null,"evidence_quote":"Supplies the edit-distance metric used by the payload corrector and by the signature clustering step."},{"cited_title":"An efficient implementation of smith waterman al- gorithm on gpu using cuda, for massively parallel scanning of sequence databases,","cited_arxiv_id":null,"evidence_quote":"Supplies the sequence-alignment dynamic program used by the signature producer to find common subsequences."},{"cited_title":"Http data set csic 2010,","cited_arxiv_id":null,"evidence_quote":"Supplies the benign HTTP request corpus used to measure true acceptance and false rejection rates."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the penetration-testing baseline whose generated payload sets are compared in Section IV-D."}],"review_version":1}