REVIEW 3 major objections 5 minor 33 references
STGen: A Novel Lightweight IoT Testbed for Generating Sensor Traffic for the Experimentation of IoT Protocol and its Application in Hybrid Network
T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read A new testbed emulates thousands of IoT sensors as lightweight processes, booting 500 nodes in 1.6 seconds and cutting memory by up to 89% compared with Docker/VM-based testbeds.
desk verdict A real, working-looking IoT traffic generator, but the headline comparison against Gotham/GothX is not supported by the data presented. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is representing each sensor as an independent lightweight process communicating with a physical STGen Core over a custom application-layer protocol on top of UDP, in place of the virtual machines and Docker containers used by Gotham and GothX. BSON (Binary JSON) serialization shrinks payloads to about 76% of equivalent JSON size, reducing network traversal time. The architecture is modular and decoupled: sensors, core, and clients run independently, with dual archiving to MongoDB and local logs, and network traffic streamed through Logstash into Elasticsearch for Kibana visualization. This design is what carries the paper's scalability and rapid-setup claims.
What would settle it
Run STGen and Gotham (or GothX) on the same machine with an identical node count, traffic workload, and duration, using the respective testbeds' own documented setup procedures, and measure wall-clock initialization time and peak memory. If the observed memory reduction and startup-time reduction fall substantially below 84% and 99.9%, respectively, the paper's central performance claim is refuted.
Extended reading notes
Core claim
The central discovery, stated on the paper's own terms, is that a hybrid testbed which replaces VMs and containers with lightweight per-sensor processes retains scalability while slashing resource use. Concretely, the paper reports that STGen boots 500 sensor nodes plus one client and one core in 1.645 seconds, compared with approximately 26 minutes for GothX to establish a topology with four VM nodes and 498 Docker nodes—a 99.9% reduction in setup time. Memory footprint is claimed to be 84–89% lower than Gotham across node counts from 20 to 120, and 2.56 GB versus 20.4 GB for GothX at 450 nodes (roughly 87.5% lower). On a 36 GB machine, the authors demonstrate booting 6,000 emulated sensors in 21.981 seconds with an average sensor-lookup latency of 0.01965 seconds, before hitting system resource limits.
Load-bearing premise
The comparison assumes that the Gotham and GothX performance figures were measured under comparable, correctly configured conditions; the paper provides no baseline installation, configuration, or measurement protocol, and contrasts 500 lightweight processes against 498 Docker nodes plus four VMs, so the 99.9% startup and 89% memory reductions could be inflated by architecture mismatch.
Editorial extensions
If this is right
- Researchers can prototype and validate IoT protocols on a single commodity machine instead of a large physical sensor deployment, shrinking cost and iteration time.
- CI/CD pipelines for IoT protocol development can spin up hundreds of virtual sensors in seconds, making automated regression testing practical.
- The 84–89% memory reduction means a fixed server can run several times more concurrent simulations or larger node counts than with Docker/VM-based testbeds.
- Because the core exposes a REST API and OpenAPI spec, the testbed can be scripted and integrated into automated experiment workflows.
- The combination of traffic generation with ELK-based real-time analytics lets researchers observe protocol behavior (latency, loss, throughput) while the simulation is still running.
Reading between the lines
- The 99.9% setup-time reduction likely conflates STGen's flat process spawn with GothX's network topology provisioning; a fairer comparison would measure STGen under a comparable topology (e.g., distributed clusters or network namespaces) rather than a single-host flat mesh.
- If memory scales at roughly 6.4 MB per sensor (38.3 GB for 6,000 nodes), a 128 GB server could plausibly host tens of thousands of emulated nodes, with the next bottlenecks being UDP port ranges and OS process limits rather than RAM.
- Because STGen's custom UDP protocol is simple and BSON-encoded, real constrained hardware that speaks the same protocol could replace emulated sensors without changing the core or client, enabling a clean emulation-to-physical migration path the paper does not explicitly test.
- A straightforward testable extension is to reproduce the authors' measurements on identical hardware with Gotham/GothX configured by their own documented procedures; the 84–99% improvements either hold or shrink to architecture-dependent numbers.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces STGen, a Python-based IoT sensor traffic generator and testbed that emulates wireless sensor network (WSN) nodes on commodity hardware. The system comprises emulated sensor nodes, a physical 'STGen core' middleware acting as a sink node, client applications, BSON serialization over a custom UDP-based protocol, MongoDB archival, and an ELK-stack integration for real-time traffic analysis. The authors report startup times, memory footprints, and latency measurements for node counts up to 6,000 and claim that STGen outperforms the Gotham and GothX testbeds by 84-89% in memory usage and 99.9% in setup time. They also describe CLI, web UI, and REST API interfaces and a modular architecture intended to support IoT protocol experimentation and big-data analytics.
Significance. If the empirical claims are substantiated, STGen would be a valuable low-cost tool for IoT protocol experimentation, particularly for researchers who need to emulate thousands of sensor nodes without physical hardware. The paper's strengths include a modular, extensible design; availability of open-source repositories; integration with the ELK stack for real-time visualization; and support for distributed deployment. However, the headline quantitative comparisons to Gotham and GothX are not yet verifiable, and the paper's own reported memory footprint conflicts with the 'lightweight' characterization. The central contribution is a prototype plus preliminary benchmarks, and the comparison claims require substantial additional evidence before they can be accepted.
major comments (3)
- [Section 10, Figure 11] The comparison with Gotham/GothX is not controlled or reproducible. The paper provides no hardware specifications, software versions/commits, configuration parameters, workload definitions, or measurement methodology for the Gotham/GothX baselines. Figure 11 plots memory usage for STGen and Gotham only, yet the text quotes a 20.4 GB GothX memory number and a 26-minute setup time for GothX that do not appear in any figure. Without a same-hardware, same-workload, same-node-count measurement protocol, the claimed 84-89% memory reduction and 99.9% startup-time reduction are unsupported.
- [Section 9, Section 8 (Hardware Configuration)] The paper characterizes STGen as 'lightweight' but reports that a 6,000-node simulation consumes 38.3 GB of memory, including 6 GB of swap, on a machine with only 36 GB of physical RAM. The paper also states that a 7,000-node test failed due to limited memory. These numbers undermine the 'lightweight' label and the claim that STGen can scale to thousands of nodes on commodity hardware, because the system relies on swap memory and the authors admit the benchmark was run in CLI mode to avoid the web application's memory overhead. The central scalability claim is thus presented with insufficient context about the trade-offs involved.
- [Section 9, Figure 7 and Table 5] The reported performance numbers (e.g., 21.981 s for 6,000 nodes, 0.01965 s average retrieval latency) are presented as single-point measurements. Although the text says each experiment was repeated ten times, no error bars, standard deviations, or per-run values are reported, making it impossible to assess run-to-run variability. This is a load-bearing issue because the comparison with Gotham/GothX in Section 10 depends on the reliability of these measured values.
minor comments (5)
- [Section 10, title and text] The section heading 'Comparision with Gotham & GothX' contains a typo; it should read 'Comparison'. Additionally, the sentence describing GothX's setup time compares different node counts (498 Docker nodes plus 4 VMs versus 500 STGen sensor nodes), which is also a conceptual mismatch, not just a presentation issue.
- [Section 6.1, BSON Serialization] The claim that 'BSON achieves approximately 76% of the size of an equivalent JSON payload' is reported without an experimental context or reference. Please specify the payload structure, sensor type, and measurement setup, or remove the claim.
- [Section 9, Figure 8] The CPU utilization plot for 3,000 sensor processes shows a peak of 0.6% at startup and 0% afterward, but the text does not explain how the aggregation was performed (per-process versus total). Clarify the metric definition to avoid misinterpretation.
- [Appendix A, installation steps] Several shell commands in the appendix are broken across lines with '↩' markers, which will not copy-paste correctly. Please format them as single-line commands or with explicit continuation prompts.
- [Table 1] The table entry for 'Sensor Node Initialization Time' lists 'Very High' for Gotham and 'Low' for STGen, but the actual measured values are not given in the table. Since the paper's main quantitative claim is about this metric, the table should refer to the measured results in Section 10.
Circularity Check
No circular derivation: STGen's performance claims are direct measurements of the authors' own implementation, and the Gotham/GothX comparison is an external-baseline fairness issue rather than a circularity.
full rationale
STGen does not derive its headline results from a fitted parameter or from a prior result by the same authors. Section 9 reports direct measurements: 'Time taken to boot STGen platform consisting of six thousand sensors: 21.981 seconds' and 'Memory consumption by the STGen platform with the 6K nodes (including swap memory): 38.3 GB,' with the experimental hardware and repeated trials described in Section 8 (AMD Ryzen 3600X, 36 GB RAM, ten runs). Equations (1) and (2) define simulation semantics (execution time equals simulation period; adjusted interval is a percentage scaling), not predicted outputs derived from inputs. The comparison in Section 10 benchmarks against Gotham and GothX, external systems cited to Poisson et al. 2024 and Saez-de Camara et al. 2023. The 84-89% memory and 99.9% startup-time reductions therefore rest on whether the baseline figures were obtained under comparable conditions, which is an experimental-design and verification concern, not a circular one. The only self-references are GitHub repository links for the STGen tool itself (footnotes 1-4), and the authors' own testbed is the object under measurement; no central claim is justified solely by a self-citation. Several passages explicitly acknowledge limitations (7k-node startup failed due to limited resources; CLI benchmarks exclude WebUI overhead), which further shows the measurements are not post hoc fitted to a predetermined conclusion. Accordingly there is no step in the paper's derivation chain where a prediction reduces by construction to its input or to a self-citation.
Assumptions & free parameters
assumptions (4)
- domain assumption The custom UDP-based application-layer protocol is a valid stand-in for real IoT protocol workloads.
- domain assumption Gotham and GothX memory and startup figures were measured under comparable, correctly configured conditions.
- domain assumption Emulated sensor processes produce traffic representative of real physical WSN nodes.
- ad hoc to paper The authors' definition of lightweight includes using 38.3 GB of memory, including swap, for 6,000 nodes on a machine with 36 GB of RAM.
Cite this review
Pith. "Pith review of STGen: A Novel Lightweight IoT Testbed for Generating Sensor Traffic for the Experimentation of IoT Protocol and its Application in Hybrid Network." pith.science (2026). https://pith.science/paper/INKMU4YK
@misc{pith2026250417725,
author = {Pith},
title = {Pith review of: STGen: A Novel Lightweight IoT Testbed for Generating Sensor Traffic for the Experimentation of IoT Protocol and its Application in Hybrid Network},
year = {2026},
howpublished = {\url{https://pith.science/paper/INKMU4YK}},
note = {Machine review of arXiv:2504.17725}
}
read the original abstract
A Wireless Sensor Network (WSN) is a network that does not rely on a fixed infrastructure and consists of numerous sensors, such as temperature, humidity, GPS, and cameras, equipped with onboard processors that manage and monitor the environment in a specific area. As a result, building a real sensor network testbed for verifying, validating, or experimenting with a newly designed protocol presents considerable challenges in adapting a laboratory scenario due to the significant financial and logistical barriers, such as the need for specialized hardware and large-scale deployments. Additionally, WSN suffers from severe constraints such as restricted power supply, short communication range, limited bandwidth availability, and restricted memory storage. Addressing these challenges, this work presents a flexible testbed solution named STGen that enables researchers to experiment with IoT protocols in a hybrid environment that emulates WSN implementations with the physical Internet through a dedicated physical server named STGen core, which receives sensor traffic and processes it for further actions. The STGen testbed is lightweight in memory usage and easy to deploy. Most importantly, STGen supports large-scale distributed systems, facilitates experimentation with IoT protocols, and enables integration with back-end services for big data analytics and statistical insights. The key feature of STGen is the integration of real-world IoT protocols and their applications with WSN. Its modular and lightweight design makes STGen efficient and enables it to outperform other popular testbeds, such as Gotham and GothX, reducing memory usage by 89\%. While GothX takes approximately 26 minutes to establish a large topology with four VM nodes and 498 Docker nodes, STGen requires only 1.645 seconds to initialize the platform with 500 sensor nodes.
Figures
Figures from the paper (9 more)
Reference graph
Works this paper leans on
-
[4]
https://doi.org/10.3390/telecom4030026 , Vol
MDPI, 597––610. https://doi.org/10.3390/telecom4030026 , Vol. 1, No. 1, Article . Publication date: April
-
[5]
IEEE Internet of Things Journal 3, 1 (2015), 124–133
Testing protocols for the internet of things on the EuWIn platform. IEEE Internet of Things Journal 3, 1 (2015), 124–133. https://doi.org/10.1109/jiot.2015.2462030 Miroslav Bures, Bestoun S Ahmed, Vaclav Rechtberger, Matej Klima, Michal Trnka, Miroslav Jaros, Xavier Bellekens, Dani Almog, and Pavel Herout
-
[8]
tc - Linux traffic control utility. Retrieved 2025-03-22 from https://manpages.ubuntu.com/manpages/focal/en/ man8/tc.8.html Accessed: March 22,
work page 2025
-
[10]
Retrieved 2025-03-23 from https://datatracker.ietf.org/doc/ html/rfc8085 RFC Editor
RFC 8085: UDP Usage Guidelines. Retrieved 2025-03-23 from https://datatracker.ietf.org/doc/ html/rfc8085 RFC Editor. Elastic. n.d.. Create visualizations in Kibana. https://www.elastic.co/guide/en/kibana/6.8/createvis.html Accessed: 2025-03-20. Robert Falkenberg, Mojtaba Masoudinejad, Markus Buschhoff, Aswin Karthik Ramachandran Venkatapathy, Birte Friese...
work page 2025
-
[11]
In 2017 Federated Conference on Computer Science and Information Systems (FedCSIS)
PhyNetLab: An IoT-based warehouse testbed. In 2017 Federated Conference on Computer Science and Information Systems (FedCSIS) . IEEE, 1051–1055. https://doi.org/10.15439/2017f267 Syed Ghazanfar, Faisal Hussain, Atiq Ur Rehman, Ubaid U Fayyaz, Farrukh Shahzad, and Ghalib A Shah
-
[12]
In 2020 International Conference on Emerging Trends in Smart Technologies (ICETST)
Iot-flock: An open-source framework for iot traffic generation. In 2020 International Conference on Emerging Trends in Smart Technologies (ICETST) . IEEE, 1–6. https://doi.org/10.1109/icetst49965.2020.9080732 HashiCorp. n.d.. Get Started with AWS Using Terraform. Retrieved 2025-02-23 from https://developer.hashicorp.com/terraform/tutorials/aws- get-starte...
-
[13]
ACM Transactions on Internet of Things 1, 3 (2020), 1–25
Makesense: An iot testbed for social research of indoor activities. ACM Transactions on Internet of Things 1, 3 (2020), 1–25. https://doi.org/10.1145/3381914 Jean Baptiste Minani, Fatima Sabir, Naouel Moha, and Yann-Gaël Guéhéneuc
-
[14]
IEEE Transactions on Software Engineering (2024), 785–815
A systematic review of IoT systems testing: Objectives, approaches, tools, and challenges. IEEE Transactions on Software Engineering (2024), 785–815. https://doi.org/10.1109/TSE.2024.3363611 Jaewon Moon, Seungwoo Kum, and Sangwon Lee
arXiv 2024
Show all 33 references
-
[17]
In IEEE INFOCOM 2019-IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS)
OpenTestBed: Poor man’s IoT testbed. In IEEE INFOCOM 2019-IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS). IEEE, 467–471. https://doi.org/10.1109/INFCOMW.2019.8845269 Thi Thu Trang Ngo, David Sarramia, Myoung-Ah Kang, and François Pinet
2019
-
[20]
Scientific Programming 2022, 1 (2022), 1—-14
Challenges in integration of heterogeneous internet of things. Scientific Programming 2022, 1 (2022), 1—-14. https://doi.org/10.1155/2022/8626882 Jianli Pan, Raj Jain, Subharthi Paul, Tam Vu, Abusayeed Saifullah, and Mo Sha
2022 doi
-
[21]
IEEE internet of things journal 2, 6 (2015), 527–537
An internet of things framework for smart energy in buildings: designs, prototype, and experiments. IEEE internet of things journal 2, 6 (2015), 527–537. https://doi.org/10.1109/jiot.2015.2413397 Georgios Z Papadopoulos, Julien Beaudaux, Antoine Gallais, Thomas Noel, and Guill...
2015
-
[23]
https://doi.org/10.1016/j.comnet.2017.03.012 Manuel Poisson, Rodrigo Carnier, and Kensuke Fukuda
Thorough IoT testbed characterization: From proof-of-concept to repeatable experimentations.Computer Networks 119 (2017), 86–101. https://doi.org/10.1016/j.comnet.2017.03.012 Manuel Poisson, Rodrigo Carnier, and Kensuke Fukuda
2017 doi
-
[24]
In Proceedings of the 17th Cyber Security Experimentation and Test Workshop
GothX: a generator of customizable, legitimate and malicious IoT network traffic. In Proceedings of the 17th Cyber Security Experimentation and Test Workshop . 65–73. https://doi.org/10.1145/3675741.36757 Xabier Sáez-de Cámara, Jose Luis Flores, Cristóbal Arellano, Aitor Urbie...
-
[25]
IEEE Transactions on Dependable and Secure Computing 21, 1 (2023), 186–203
Gotham testbed: a reproducible IoT testbed for security experiments and dataset generation. IEEE Transactions on Dependable and Secure Computing 21, 1 (2023), 186–203. https://doi.org/10.1109/tdsc.2023.3247166 Sain Saginbekov and Chingiz Shakenov
2023
-
[28]
In 2020 IEEE 25th Pacific Rim International Symposium on Dependable Computing (PRDC)
Generative deep learning for Internet of Things network traffic generation. In 2020 IEEE 25th Pacific Rim International Symposium on Dependable Computing (PRDC) . IEEE, 70–79. https://10.1109/PRDC50213.2020.00018 Z. Shelby, M. Koster, C. Bormann, and P. van der Stok
2020
-
[29]
Retrieved 2025-03-23 from https://datatracker.ietf.org/doc/rfc9176/ RFC Editor
RFC 9176: Constrained RESTful Environments (CoRE) Resource Directory. Retrieved 2025-03-23 from https://datatracker.ietf.org/doc/rfc9176/ RFC Editor. Souhila Silmi, Zouina Doukha, Rebiha Kemcha, and Samira Moussaoui
2025
-
[30]
arXiv preprint arXiv:2009.03640 (2020)
Wireless sensor networks simulators and testbeds. arXiv preprint arXiv:2009.03640 (2020). Sulayman K Sowe, Takashi Kimata, Mianxiong Dong, and Koji Zettsu
2020 arXiv
-
[31]
In 2014 IEEE 38th international computer software and applications conference workshops
Managing heterogeneous sensor data on a big data platform: IoT services for data-intensive science. In 2014 IEEE 38th international computer software and applications conference workshops . IEEE, 295–300. https://doi.org/10.1109/compsacw.2014.52 Sotirios Tsakalidis, George Tso...
2014 doi
-
[33]
com / subratanath123 / STGen -A - Novel - Lightweight - ↩→ Sensor - Traffic - Generator - Backend - Launcher • Step 6: Change the directory to the downloaded codebase
$ git clone https :// github . com / subratanath123 / STGen -A - Novel - Lightweight - ↩→ Sensor - Traffic - Generator - Backend - Launcher • Step 6: Change the directory to the downloaded codebase. $ cd STGen -A - Novel - Lightweight - Sensor - Traffic - Generator - Web - UI ...
2025
-
[241]
https://doi.org/10.1007/s42979-022-01628-6 Muhammad Noaman, Muhammad Sohail Khan, Muhammad Faisal Abrar, Sikandar Ali, Atif Alvi, and Muhammad Asif Saleem
-
[1511]
https://doi.org/10.3390/s24051511 Ibrahim S Alsukayti
-
[2013]
In 2013 IEEE 9th International Conference on Wireless and Mobile Computing, Networking and Communications (WiMob)
Adding value to WSN simulation using the IoT-LAB experimental platform. In 2013 IEEE 9th International Conference on Wireless and Mobile Computing, Networking and Communications (WiMob). IEEE, 485–490. https://doi.org/10.1109/WiMOB.2013.6673403 Georgios Z Papadopoulos, Antoine...
2013
-
[2014]
Computer Networks 61 (2014), 217–238
SmartSantander: IoT experimentation over a smart city testbed. Computer Networks 61 (2014), 217–238. https://doi.org/10.1016/j.bjp.2013.12.020 Mustafizur R Shahid, Gregory Blanc, Houda Jmila, Zonghua Zhang, and Hervé Debar
2014 doi
-
[2015]
In 2015 IEEE 2nd World Forum on Internet of Things (WF-IoT)
FIT IoT-LAB: A large scale open experimental IoT testbed. In 2015 IEEE 2nd World Forum on Internet of Things (WF-IoT) . IEEE, 459–464. Reham Almutairi, Giacomo Bergami, and Graham Morgan
2015
-
[2016]
arXiv preprint arXiv:1602.01567 (2016)
Testing wireless sensor networks with hybrid simulators. arXiv preprint arXiv:1602.01567 (2016). https://doi.org/10.48550/arXiv.1602.01567 Luis Sanchez, Luis Muñoz, Jose Antonio Galache, Pablo Sotres, Juan R Santana, Veronica Gutierrez, Rajiv Ramdhany, Alex Gluhak, Srdjan Krco...
-
[2017]
IEEE Internet of Things Journal 5, 3 (2017), 1637–1647
Internet of things (iot): Research, simulators, and testbeds. IEEE Internet of Things Journal 5, 3 (2017), 1637–1647. https://doi.org/10.1109/jiot.2017.2786639 Ubuntu Developers
2017
-
[2019]
5 status prediction
A heterogeneous IoT data analysis framework with collaboration of edge-cloud computing: Focusing on indoor PM10 and PM2. 5 status prediction. Sensors 19, 14 (2019),
2019
-
[2020]
Wireless Communications and Mobile Computing 2020, 1 (2020), 8849433
A multidimensional internet of things testbed system: Development and evaluation. Wireless Communications and Mobile Computing 2020, 1 (2020), 8849433. https://doi.org/10.1155/2020/8849433 Chiara Buratti, Andrea Stajkic, Gordana Gardasevic, Sebastiano Milardo, M Danilo Abrigna...
2020 doi
-
[2021]
In 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST)
Patriot: Iot automated interoperability and integration testing framework. In 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE, 454–459. https://doi.org/10.1109/icst49551.2021.00059 Maxim Chernyshev, Zubair Baig, Oladayo Bello, and Sheral...
2021
-
[2022]
Retrieved 2025-03-23 from https://datatracker.ietf.org/doc/html/rfc9293 RFC Editor
RFC 9293: Transmission Control Protocol (TCP). Retrieved 2025-03-23 from https://datatracker.ietf.org/doc/html/rfc9293 RFC Editor. L. Eggert, G. Fairhurst, and G. Shepherd
2025
-
[2023]
SN Computer Science 4, 3 (2023),
A new approach based on ELK stack for the analysis and visualisation of geo-referenced sensor data. SN Computer Science 4, 3 (2023),
2023
-
[2024]
Sensors 24, 5 (2024),
Advancements and challenges in IoT simulators: A comprehensive review. Sensors 24, 5 (2024),
2024
-
[3038]
https://doi.org/10.3390/s19143038 Jonathan Munoz, Fabian Rincon, Tengfei Chang, Xavier Vilajosana, Brecht Vermeulen, Thijs Walcarius, Wim Van de Meerssche, and Thomas Watteyne
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.