REVIEW 4 major objections 6 minor 44 references
ChatHTTPFuzz: Large Language Model-Assisted IoT HTTP Fuzzing
T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read A large language model can guide HTTP fuzzing of IoT devices, reporting 103 vulnerabilities, 68 previously undisclosed, 23 with CVEs.
desk verdict Solid LLM-fuzzing system with real CVE finds, but the superiority claim over baselines is undercut by unequal oracles and sloppy reporting. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The seed template is the central object: an HTTP packet skeleton with variable fields marked, typed, and paired with a mutation space of LLM-inferred legal values. Three mechanisms carry the argument: LLM-guided variable annotation identifies mutable fields with zero false negatives on the paper's 603-parameter corpus; the LLM-guided seed template enrichment algorithm analyzes backend code to generate new packets and expand field value sets; and the dual-factor gain scheduling algorithm combines Thompson sampling with an exploration-balance factor and a mutation-potential factor to choose templates. The load-bearing capability is the LLM's translation of code logic into valid HTTP values, which the paper measures at 98.58% field recognition.
What would settle it
Run the proposed fuzzer and the three comparison fuzzers from the paper on the same devices with the same session setup, time budget, seed traffic, and criteria for counting a vulnerability. If the comparison fuzzers collectively find at least as many unique vulnerabilities, or if the reported CVEs cannot be reproduced, the headline outperformance claim is not supported.
Extended reading notes
Core claim
ChatHTTPFuzz is an LLM-guided black-box fuzzer for IoT HTTP services. It treats an LLM as a protocol-aware seed generator: given captured HTTP requests, it annotates every variable field with type and encoding, builds seed templates that preserve fixed protocol elements, then uses decompiled service code to infer new routes, branch-triggering values, and format constraints such as sscanf patterns and Base64 encoding. Mutations are applied only to labeled variable fields, with type-aware operators and context-specific values. A dual-factor gain Thompson sampling scheduler scores templates by historical reward, invocation frequency, and mutation-space size. The paper's claim is that this combination finds 103 vulnerabilities across 14 devices, 68 of them undisclosed and 23 CVE-assigned, and outperforms the three tested black-box fuzzers.
Load-bearing premise
The claim that ChatHTTPFuzz beats the comparison fuzzers depends on those fuzzers being tested on the same devices, with the same time budget, seeds, and rules for counting a bug; the paper publishes a full device-by-device count for only one of the three.
Editorial extensions
If this is right
- On the 14 devices tested, ChatHTTPFuzz reports 103 vulnerabilities, of which 68 are previously undisclosed and 23 have been assigned CVE identifiers.
- The seed-enrichment step is the main driver: removing it drops discovered vulnerabilities from 57 to 24 across four devices, about a 2.38x reduction.
- The scheduling algorithm does not add many vulnerabilities but halves the time needed to reach the same number of detections over 24 hours.
- LLM annotation of HTTP fields can be automated with near-zero false negatives on standard and custom parameter formats, removing the need to hand-write protocol templates.
- Protocol-aware mutation avoids corrupting fixed HTTP elements, so more generated packets remain parseable by the target service.
Reading between the lines
- If the advantage is concentrated in the seed-enrichment step, attaching LLM-guided code-aware seed generation to existing template fuzzers could reproduce much of the gain; the paper's augmented-baseline experiment points in that direction.
- The same field-annotation and code-branch prompts should transfer to other structured protocols with similar backend routing, such as RTSP, SNMP, or Matter, so the method is not inherently HTTP-specific.
- The concentration of critical CVEs in a few device families suggests the fuzzer surfaces systematic classes of command-injection bugs in a vendor's codebase, not only isolated flaws.
- A cost question the paper leaves open is how much of the result depends on the specific LLM and on manual reverse-engineering of routing tables; substituting a cheaper local model or automating that extraction would be a testable variation.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes ChatHTTPFuzz, an LLM-guided black-box fuzzer for HTTP services in IoT devices. It uses GPT-4o to annotate variable fields in HTTP packets into seed templates, to generate and enrich seed templates from statically analyzed backend code, and to schedule templates via a dual-factor Thompson sampling algorithm. The authors evaluate the system on real IoT devices and report 103 discovered vulnerabilities, 68 previously undisclosed, and 23 assigned CVEs, claiming that ChatHTTPFuzz finds more vulnerabilities than Boofuzz, Snipuzz, and Mutiny.
Significance. If the empirical claims are substantiated, ChatHTTPFuzz would be a meaningful advance in automated IoT HTTP fuzzing: it demonstrates a concrete way to use LLM protocol understanding and code-guided seed generation to reduce invalid mutations and expand seed coverage. The approach is not circular: the scheduling formulas are defined a priori, and vulnerability detection is based on device behavior and CVE confirmation rather than on the LLM's own output. The categorized protocol-parsing study in Table 3 and the use of embedded template IDs for command-injection verification are concrete contributions. However, the current reporting does not yet allow independent verification of the headline superiority claim, so the paper's significance depends on fixes to the experimental presentation.
major comments (4)
- [§6.3, Tables 7 and 8] The comparative claim that ChatHTTPFuzz 'finds more vulnerabilities than SNIPUZZ, BOOFUZZ, and MUTINY' is not supported by the reported data. Table 7 contains columns only for Boofuzz, Boofuzz-LSTE, and ChatHTTPFuzz, yet the text states 'As shown in Table 7' when comparing Mutiny; Table 8 lists only 11 named CVEs with binary checkmarks and no per-tool per-device totals. No table or text reports total vulnerabilities found by Snipuzz and Mutiny per device, nor the time budget, seed set, number of runs, and detection criteria used for each tool. Without a unified per-device comparison under identical conditions, the superiority claim is unverifiable.
- [§5, Anomaly Monitoring; §6.3] The comparison conflates the fuzzer with its detection and seed-generation advantages. ChatHTTPFuzz detects command injection by embedding a unique seed-template ID into each command and correlating executed commands on the device, and flags buffer overflow via 'Connection reset'; the paper does not state that Boofuzz, Snipuzz, or Mutiny were given the same detection harness, the LLM-generated seed templates, or the mutation dictionaries. The discussion in §6.3 credits Boofuzz's failure to find the DNS-320L command injection to Boofuzz's lack of mutation methods, but that capability is supplied to ChatHTTPFuzz by LSTE and the custom oracle. The experiment should either provide the same oracle and seeds to all tools or be framed explicitly as an end-to-end comparison, not a fuzzer-quality comparison.
- [§6.4 and Table 9] The headline vulnerability counts are internally inconsistent. The abstract reports 103 vulnerabilities, 68 unique, and 23 CVEs; Section 1 reports 103 vulnerabilities, 59 unique, and 23 CVEs; Section 6.4 states 92 vulnerabilities, 59 undisclosed, and 22 CVEs; Table 9 totals 103, 68, and 23. Device lists also disagree: Table 6 lists 15 devices including Netgear WNAP320, while Table 9 lists 14 devices, replaces Cisco RV110W with RV100W, and omits WNAP320. These inconsistencies must be reconciled before the empirical claims can be assessed.
- [§3.1–3.2, Tables 3–5] The LLM capability evaluations rely on small self-constructed samples (140 packets in Table 3, 60 generated packets across three devices in Table 4, and 30 code samples in Table 5) with no confidence intervals, no description of how the samples were selected, and no independent test-set construction. If these numbers are intended to motivate the design, the paper should say so explicitly; if they are formal evaluations, the sampling procedure and variance should be reported.
minor comments (6)
- [Abstract] The abstract contains the typo 'resent significant security risks'; it should read 'present significant security risks.'
- [§6.1] The text uses the spelling 'Snipfuzz' while the rest of the paper uses 'Snipuzz'; please make the tool name consistent throughout.
- [Algorithm 2 and Equation (3)] The top-template fraction 'Top0.1N' and the Beta prior parameters are introduced without justification or sensitivity analysis; at minimum, report the values used in the experiments and test whether the qualitative results depend on them.
- [§6.4] The text states that CVE-2024-7828 was rated high severity, but Table 10 lists it as 9.8/Critical; please reconcile this inconsistency.
- [Table 9] The Tenda AC15 version is listed as 'V15.03.05.19 multi', which is not a version identifier; please clarify what 'multi' refers to.
- [§6.1 and §8] The paper says '15 different IoT device models' in Section 6.2 and Table 6, while the abstract and conclusion say 14; please align the device count and the listed devices.
Circularity Check
No significant circularity: the LLM capability evaluations, seed scheduling formulas, and vulnerability counts are derived from independent experiments, external artifacts, and device behavior rather than from the claims they support.
full rationale
The paper's derivation chain is not circular. The claimed LLM parsing and generation capabilities are established by independent experiments in Section 3: 140 manually constructed parameter packets reviewed by experts (Table 3) and gdb-verified field assignment on three devices (Table 4), with no fitted parameters. The LSTE and STSA designs are a priori heuristics: E_factor and M_factor are defined from invocation counts and field-set sizes (Eqs. 1-3), and the scheduling score is not tuned to reproduce any target vulnerability count. Vulnerability detection itself is externally grounded in device behavior and artifacts: CVE identifiers assigned by vendors, 'Connection reset' observations for buffer overflows, and command-template correlation for command injection (Section 5). The main weakness is comparison fairness, because ChatHTTPFuzz's custom anomaly-monitoring oracle and LSTE-enriched seed templates are not described as being given to Boofuzz, Snipuzz, or Mutiny, and Table 8 lacks complete per-tool per-device vulnerability totals. That is a threat to the comparative evidence, not a circular reduction, since the claimed superiority is not built by construction from the method's own outputs. No load-bearing self-citation chain is present; the cited prior work is external baselines or general references. Therefore, no step in the paper reduces a prediction to its inputs, and the core results retain independent content.
Assumptions & free parameters
free parameters (2)
- Top template selection fraction =
0.1N
- Beta prior hyperparameters =
unstated (assumed 1,1)
assumptions (4)
- domain assumption GPT-4o reliably parses HTTP fields and firmware code from prompts, as measured by 140 and 60 handcrafted examples.
- domain assumption Response-content changes indicate newly triggered code branches.
- domain assumption Static analysis via IDA Python correctly maps routes to handler code.
- standard math Standard probability and bandit mathematics.
Cite this review
Pith. "Pith review of ChatHTTPFuzz: Large Language Model-Assisted IoT HTTP Fuzzing." pith.science (2026). https://pith.science/paper/SGTZLMWY
@misc{pith2026241111929,
author = {Pith},
title = {Pith review of: ChatHTTPFuzz: Large Language Model-Assisted IoT HTTP Fuzzing},
year = {2026},
howpublished = {\url{https://pith.science/paper/SGTZLMWY}},
note = {Machine review of arXiv:2411.11929}
}
read the original abstract
Internet of Things (IoT) devices offer convenience through web interfaces, web VPNs, and other web-based services, all relying on the HTTP protocol. However, these externally exposed HTTP services resent significant security risks. Although fuzzing has shown some effectiveness in identifying vulnerabilities in IoT HTTP services, most state-of-the-art tools still rely on random mutation trategies, leading to difficulties in accurately understanding the HTTP protocol's structure and generating many invalid test cases. Furthermore, These fuzzers rely on a limited set of initial seeds for testing. While this approach initiates testing, the limited number and diversity of seeds hinder comprehensive coverage of complex scenarios in IoT HTTP services. In this paper, we investigate and find that large language models (LLMs) excel in parsing HTTP protocol data and analyzing code logic. Based on these findings, we propose a novel LLM-guided IoT HTTP fuzzing method, ChatHTTPFuzz, which automatically parses protocol fields and analyzes service code logic to generate protocol-compliant test cases. Specifically, we use LLMs to label fields in HTTP protocol data, creating seed templates. Second, The LLM analyzes service code to guide the generation of additional packets aligned with the code logic, enriching the seed templates and their field values. Finally, we design an enhanced Thompson sampling algorithm based on the exploration balance factor and mutation potential factor to schedule seed templates. We evaluate ChatHTTPFuzz on 14 different real-world IoT devices. It finds more vulnerabilities than SNIPUZZ, BOOFUZZ, and MUTINY. ChatHTTPFuzz has discovered 103 vulnerabilities, of which 68 are unique, and 23 have been assigned CVEs.
Reference graph
Works this paper leans on
-
[1]
Available at: https://github.com/qemu/qemu
Bellard F (2024) qemu/qemu. Available at: https://github.com/qemu/qemu
work page 2024
-
[2]
In: Proceedings of the 34th International Conference on Neural Information Processing Systems
Brown TB, Mann B, Ryder N, et al (2020) Language models are few-shot learners. In: Proceedings of the 34th International Conference on Neural Information Processing Systems . Curran Associates Inc., Red Hook, NY, USA, NIPS '20, pp 1877--1901
work page 2020
-
[3]
doi:10.48550/arXiv.2310.14735, 2310.14735 [cs]
Chen B, Zhang Z, Langrené N, et al (2024) Unleashing the potential of prompt engineering in large language models: a comprehensive review. doi:10.48550/arXiv.2310.14735, 2310.14735 [cs]
-
[4]
In: Proceedings 2018 Network and Distributed System Security Symposium
Chen J, Diao W, Zhao Q, et al (2018) IoTFuzzer : Discovering memory corruptions in IoT through app-based fuzzing. In: Proceedings 2018 Network and Distributed System Security Symposium. Internet Society, doi:10.14722/ndss.2018.23159
-
[5]
https://github.com/Cisco-Talos/mutiny-fuzzer
Cisco (2017) Cisco-talos/mutiny-fuzzer. https://github.com/Cisco-Talos/mutiny-fuzzer
work page 2017
-
[6]
Computers & Security 120:102817
Cui L, Cui J, Hao Z, et al (2022) An empirical study of vulnerability discovery methods over the past ten years. Computers & Security 120:102817. doi:10.1016/j.cose.2022.102817
arXiv 2022
-
[7]
In: Proceedings of the IEEE / ACM 46th International Conference on Software Engineering
Du X, Liu M, Wang K, et al (2024) Evaluating Large Language Models in Class - Level Code Generation . In: Proceedings of the IEEE / ACM 46th International Conference on Software Engineering . Association for Computing Machinery, New York, NY, USA, ICSE '24, pp 1--13
work page 2024
-
[8]
The Internet Of Things pp 9--12
Ezechina MA, Okwara KK, Ugboaja CAU (2015) The Internet of Things ( Iot ): A Scalable Approach to Connecting Everything . The Internet Of Things pp 9--12
work page 2015
Show all 44 references
-
[9]
In: Proceedings of the 45th International Conference on Software Engineering
Fan Z, Gao X, Mirchev M, et al (2023) Automated Repair of Programs from Large Language Models . In: Proceedings of the 45th International Conference on Software Engineering . IEEE Press, Melbourne, Victoria, Australia, ICSE '23, pp 1469--1481
2023
-
[10]
Feng X, Sun R, Zhu X, et al (2021) Snipuzz: Black-box fuzzing of IoT firmware via message snippet inference
2021
-
[11]
Available at: https://sourceware.org/gdb/
gdb (Last updated 2024) GDB : The GNU project debugger. Available at: https://sourceware.org/gdb/
2024
-
[12]
https://hex-rays.com//products/ida/support/idapython_docs/
Gergely Erdelyi EC (2004) Idapython. https://hex-rays.com//products/ida/support/idapython_docs/
2004
-
[13]
Available at: https://www.w3.org/Protocols/rfc2616/rfc2616.html
Hypertext Transfer Protocol (1995) Hypertext transfer protocol -- HTTP /1.1. Available at: https://www.w3.org/Protocols/rfc2616/rfc2616.html
1995
-
[14]
In: Proceedings of the 44th International Conference on Software Engineering
Jain N, Vaidyanath S, Iyer A, et al (2022) Jigsaw: large language models meet program synthesis. In: Proceedings of the 44th International Conference on Software Engineering . Association for Computing Machinery, New York, NY, USA, ICSE '22, pp 1219--1231
2022
-
[15]
https://github.com/jtpereyda/boofuzz
Jtpereyda (Last updated 2024) jtpereyda/boofuzz: A fork and successor of the sulley fuzzing framework. https://github.com/jtpereyda/boofuzz
2024
-
[16]
Kallus B, Anantharaman P, Locasto M, et al (2024) The HTTP garden: Discovering parsing vulnerabilities in HTTP /1.1 implementations by differential fuzzing of request streams
2024
-
[17]
Proceedings of the 11th ACM Workshop on Artificial Intelligence and Security pp 37--47
Karamcheti S, Mann G, Rosenberg D (2018) Adaptive Grey - Box Fuzz - Testing with Thompson Sampling . Proceedings of the 11th ACM Workshop on Artificial Intelligence and Security pp 37--47
2018
-
[18]
In: 2016 IEEE Annual India Conference ( INDICON ), pp 1--6
Kumar K, Bose J, Tripathi S (2016) A unified web interface for the internet of things. In: 2016 IEEE Annual India Conference ( INDICON ), pp 1--6
2016
-
[19]
IEEE Transactions on Reliability 67(3):1199--1218
Liang H, Pei X, Jia X, et al (2018) Fuzzing: State of the art. IEEE Transactions on Reliability 67(3):1199--1218. doi:10.1109/TR.2018.2834476, conference Name: IEEE Transactions on Reliability
2018
-
[20]
Lindsey O'Donnell (2020) More Than Half of IoT Devices Vulnerable to Severe Attacks Threatpost
2020
-
[21]
https://www.statista.com/statistics/1183457/iot-connected-devices-worldwide/
Lionel Sujay V (2024) Number of internet of things ( IoT ) connections worldwide from 2022 to 2023, with forecasts from 2024 to 2033. https://www.statista.com/statistics/1183457/iot-connected-devices-worldwide/
2024
-
[22]
In: 2024 IEEE Symposium on Security and Privacy (SP)
Liu H, Gan S, Zhang C, et al (2024) Labrador: Response Guided Directed Fuzzing for Black -box IoT Devices . In: 2024 IEEE Symposium on Security and Privacy (SP). IEEE Computer Society, pp 1920--1938
2024
-
[23]
In: USENIX Security Symposium, pp 4783--4800
Ma X, Luo L, Zeng Q (2024) From one thousand pages of specification to unveiling hidden bugs: Large language model assisted fuzzing of matter IoT devices. In: USENIX Security Symposium, pp 4783--4800
2024
-
[24]
Available at: https://github.com/wireghoul/doona, original-date: 2012-05-01T04:10:01Z
Marcussen E (2024) wireghoul/doona. Available at: https://github.com/wireghoul/doona, original-date: 2012-05-01T04:10:01Z
2024
-
[25]
In: Proceedings 2024 Network and Distributed System Security Symposium
Meng R, Mirchev M, Böhme M, et al (2024) Large language model guided protocol fuzzing. In: Proceedings 2024 Network and Distributed System Security Symposium. Internet Society
2024
-
[26]
Minaee S, Mikolov T, Nikzad N, et al (2024) Large Language Models : A Survey
2024
-
[27]
url https://github.com/nccgroup/TriforceAFL
NCC -Group (2017) Triforce. url https://github.com/nccgroup/TriforceAFL
2017
-
[28]
Sensors (Basel, Switzerland) 23(13):6067
Noman HA, Abu-Sharkh OMF (2023) Code Injection Attacks in Wireless - Based Internet of Things ( IoT ): A Comprehensive Review and Practical Implementations . Sensors (Basel, Switzerland) 23(13):6067. ://www.ncbi.nlm.nih.gov/pmc/articles/PMC10346793/
2023
-
[29]
ACM Transactions on misc Engineering and Methodology 5
Offutt J, Lee A, Rothermel G, et al (1996) An experimental determination of sufficient mutant operators. ACM Transactions on misc Engineering and Methodology 5. doi:10.1145/227607.227610
1996
-
[30]
https://platform.openai.com/docs/models/gpt-4o, accessed: 28-Oct-2024
OpenAI (2024) Gpt-4 turbo: Enhanced capabilities and use cases. https://platform.openai.com/docs/models/gpt-4o, accessed: 28-Oct-2024
2024
-
[31]
https://www.peach.tech/
peach (2011) Peach fuzzing platform. https://www.peach.tech/
2011
-
[32]
Internet of Things 23:100888
Pourrahmani H, Yavarinasab A, Monazzah AMH, et al (2023) A review of the security vulnerabilities and countermeasures in the internet of things solutions: A bright future for the blockchain. Internet of Things 23:100888
2023
-
[33]
2011 3rd International Conference on Electronics Computer Technology pp 297--301
Ramya CM, Shanmugaraj M, Prabakaran R (2011) Study on ZigBee technology. 2011 3rd International Conference on Electronics Computer Technology pp 297--301. doi:10.1109/ICECTECH.2011.5942102, conference Name: 2011 3rd International Conference on Electronics Computer Technology (...
2011
-
[34]
Renze M, Guven E (2024) The Effect of Sampling Temperature on Problem Solving in Large Language Models
2024
-
[35]
IEEE Internet of Things Journal 10(13):11224--11239
Siwakoti YR, Bhurtel M, Rawat DB, et al (2023) Advances in IoT Security : Vulnerabilities , Enabled Criminal Services , Attacks , and Countermeasures . IEEE Internet of Things Journal 10(13):11224--11239. doi:10.1109/JIOT.2023.3252594, ://ieeexplore.ieee.org/document/10059147/...
2023
- [36]
-
[37]
Tsai CH, Tsai SC, Huang SK (2021) REST API Fuzzing by Coverage Level Guided Blackbox Testing . In: 2021 IEEE 21st International Conference on Software Quality , Reliability and Security ( QRS ), pp 291--300, doi:10.1109/QRS54544.2021.00040, ://ieeexplore.ieee.org/abstract/docu...
2021
- [38]
-
[39]
In: Symposium on Security and Privacy (SP)
Wang J, Yu L, Luo X (2024) LLMIF : Augmented large language model for fuzzing IoT devices. In: Symposium on Security and Privacy (SP). IEEE Computer Society, pp 881--896, doi:10.1109/SP54263.2024.00211
2024
-
[40]
In: Proceedings of the IEEE / ACM 46th International Conference on Software Engineering
Xia CS, Paltenghi M, Le Tian J, et al (2024) Fuzz4All : Universal Fuzzing with Large Language Models . In: Proceedings of the IEEE / ACM 46th International Conference on Software Engineering . Association for Computing Machinery, New York, NY, USA, ICSE '24, pp 1--13
2024
-
[41]
In: 2023 IEEE International Conference on Consumer Electronics ( ICCE ), pp 1--7, doi:10.1109/ICCE56470.2023.10043520, ISSN : 2158-4001
Zegeye W, Jemal A, Kornegay K (2023) Connected smart home over matter protocol. In: 2023 IEEE International Conference on Consumer Electronics ( ICCE ), pp 1--7, doi:10.1109/ICCE56470.2023.10043520, ISSN : 2158-4001
2023
-
[42]
In: USENIX Security Symposium, pp 1099--1114
Zheng Y, Davanian A, Yin H, et al (2019) FIRM - AFL : High-throughput greybox fuzzing of IoT firmware via augmented process emulation. In: USENIX Security Symposium, pp 1099--1114
2019
-
[43]
, " * write output.state after.block = add.period write newline
ENTRY address archive author booktitle chapter doi edition editor eid eprint howpublished institution journal key keywords month note number organization pages publisher school series title type url volume year archivePrefix primaryClass adsurl adsnote version label extra.labe...
-
[44]
write newline
" write newline "" before.all 'output.state := FUNCTION add.period duplicate empty 'skip "." * add.blank if FUNCTION if.digit duplicate "0" = swap duplicate "1" = swap duplicate "2" = swap duplicate "3" = swap duplicate "4" = swap duplicate "5" = swap duplicate "6" = swap dupl...
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.