{"id":"ba0f232d-4c3a-4847-b4fa-326107444592","arxiv_id":"2505.06307","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A two-stage chain-of-thought prompt built for IoT vulnerability advice raises LLM answer scores in an LLM-judged comparison, but the evaluation lacks human validation, external baselines, and released artifacts.","lead":"This paper presents an IoT security assistant that first asks a large language model to analyze the vulnerability and the user's role, then generates advice using that analysis. It reports higher quality scores than plain LLM answers, but the scoring was done by an LLM judge rather than human experts or external benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table I's claimed gains rest entirely on GPT-4o judging outputs it may have generated itself; no human validation or ground truth controls for the judge's stylistic preference for ICoT's structured, verbose responses.","rationale":"The reader's weakest_assumption correctly identifies the load-bearing point: the entire numerical case for ICoT depends on GPT-4o as judge, and for the GPT-4o conditions the judge is assessing its own family's outputs. I agree that this is the single most fragile link. If the judge favors structured, verbose, role-labeled text, which ICoT explicitly generates, all positive deltas in Table I could be artifacts. The paper offers no human evaluation, no known-answer accuracy checks against CVE records, and no statistical significance testing, so there is no independent evidence that ICoT improves the correctness of security advice. I also note the paper's own limitations section concedes redundancy and hallucination risks, which further undercuts the reliability claim. The method itself is plausible and the exposition is clear, but the central empirical claim is not supported as stated. Since the reader already reached REJECT on these grounds, my stress-test does not change the verdict; it confirms it. I would not manufacture a separate objection, and I see no internal inconsistency in the ICoT construction itself. The concern is purely about the validity of the evaluation evidence.","tokens_in":14338,"tokens_out":3329,"duration_ms":33901,"concrete_test":"Sample 100 vulnerability-query pairs from VARiOT, generate ICoT and LLM-only answers with the authors' exact prompts, then have three independent IoT security experts blindly score anonymized answers on the five defined metrics without knowing which condition produced them. Compute per-metric mean deltas and confidence intervals; if the expert-rated improvements are not significantly above zero, or are materially smaller than GPT-4o's Table I scores, the reported gains are an LLM-judge artifact. A secondary control is to feed the same judge the ICoT outputs stripped of JSON formatting and role headers; a large score drop would confirm stylistic bias.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim is that ICoT 'significantly improves the understanding of IoT security issues' and delivers 'higher accuracy and reliability.' The only quantitative support is Table I, produced by a fixed GPT-4o judge under the template in Fig. 3. For the GPT-4o and GPT-4o-mini rows, the judge is the same model family that generated the answers; for all rows, the judge is a closed API model with unknown biases. ICoT outputs are explicitly structured, role-aware, and verbose, including a JSON analysis stage; an LLM judge asked to score Reliability, Relevance, Detail, Technicality, and Friendliness is likely to reward those surface features regardless of factual correctness. The deltas in Table I (e.g., +0.99 Relevance for DeepSeek-V3 General User) are therefore uninterpretable as evidence of improved accuracy. No sample size, variance, statistical test, human rating, or comparison against known vulnerability ground truth is reported. The paper's own Discussion admits that ICoT can produce redundant information and that LLMs hallucinate, making the 'higher accuracy' claim especially unsupported: accuracy cannot be established by the same kind of model that may be fabricating the advice. Consequently, the central comparative claim collapses if the judge's preference is not controlled.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ICoT, a two-stage chain-of-thought prompting method for an LLM-based IoT security assistant. In the first stage, the LLM analyzes the user role and the vulnerability description (type, location, impact) and emits a structured JSON summary; in the second stage, it uses that summary to generate a personalized security recommendation. The method is tested on GPT-4o, GPT-4o-mini, DeepSeek-V3, and DeepSeek-R1 using a fixed GPT-4o evaluator that scores outputs on five subjective quality metrics. The abstract claims that ICoT 'significantly improves the understanding of IoT security issues' and achieves 'higher accuracy and reliability' compared with LLM-only prompting.","tokens_in":14499,"tokens_out":3299,"duration_ms":33516,"significance":"If the empirical claim were supported, the paper would offer a practical prompt-only way to make LLM IoT security advice more tailored and useful, with no fine-tuning. The strength of this contribution, however, is entirely dependent on the evaluation. The manuscript provides no human validation, no statistical analysis, and no comparison against the closest prior system (ChatIoT), even though that system is cited. The evaluation design has a self-judging flaw: GPT-4o scores answers that, in the GPT-4o and GPT-4o-mini conditions, the same model family generated. For these reasons, the results in Table I do not establish the claimed accuracy and reliability gains, and the central contribution is not currently supported.","major_comments":[{"comment":"The evaluation uses GPT-4o as a fixed judge to score answers, but for the GPT-4o and GPT-4o-mini conditions the judged answers were generated by the same model family. The judge is a closed API model with no human validation or external ground truth. The template asks for subjective scores on qualities such as Detail and Friendliness, which are likely to reward the longer, structured, role-aware ICoT outputs regardless of their factual correctness. The paper's own Section V.D admits that LLMs hallucinate and can produce redundant information, so the abstract's claim of 'higher accuracy and reliability' is not supported by this design. The numeric gains in Table I therefore cannot be read as evidence of improved security understanding.","section":"V.B and Fig. 3"},{"comment":"The metric definitions are inconsistent: Section V.B defines Accuracy, Relevance, Detail, Technicality, and Friendliness, while Fig. 3 and Table I report Reliability instead of Accuracy. Table I also contains an invalid numeric entry, '3.3.66', in the Developer/DeepSeek-R1/Detail row. The text states only that scores are in [0,5], with no sample size, variance, number of repeated runs, or significance tests. Deltas as small as +0.05 (e.g., GPT-4o General User Reliability) are presented as improvements, and the abstract uses the word 'significantly' without any statistical support. The evaluation protocol is under-specified and does not allow the reader to assess whether the observed differences are real or noise.","section":"V.C and Table I"},{"comment":"The paper cites ChatIoT [3] as an existing LLM-based IoT security assistant that uses retrieval-augmented generation, but it never compares ICoT against ChatIoT or any other LLM-based baseline. The only baseline is a bare LLM with no IoT-specific prompting. This cannot support the contributions section's claim that ICoT 'outperforming traditional methods' or the Discussion's claim of more accurate and in-depth analysis than traditional vulnerability detection tools. A direct comparison with the closest prior assistant, or at least a clear discussion of why such a comparison is not needed, is required to position the contribution.","section":"Introduction and II.B"}],"minor_comments":[{"comment":"The entry '3.3.66' in the Developer/DeepSeek-R1/Detail row should be corrected to a valid score in [0,5].","section":"Table I"},{"comment":"The dataset description does not state how many vulnerability descriptions or user queries were used, nor how they were selected from V ARIoT. Provide these counts and the selection criteria so the reader can gauge the evaluation's coverage.","section":"V.A"},{"comment":"The claim that judging both outputs at the same time 'minimizes the impact of any inherent randomness' is not supported; report the number of evaluation runs and the observed variance.","section":"V.B"},{"comment":"ChatIoT [3] is cited in the introduction but not discussed in Section II.B; add it to the related-work discussion and explicitly explain how ICoT differs from and improves upon it.","section":"II.B"},{"comment":"Figure 2 is difficult to read: the prompt text is truncated and the structure is not self-explanatory. A cleaner diagram with the full prompt template would help.","section":"Fig. 2"},{"comment":"The notation '[V][T][Sin]' and '[V][Sg][C][T]' is introduced only informally; define each component explicitly (e.g., what T contains, how Sin differs from C) so the two-stage process is reproducible.","section":"IV, Eq. (2) and (4)"}],"recommendation":"reject","confidential_remarks":"The manuscript's central claim rests on an evaluation with a self-judging GPT-4o evaluator, inconsistent metric definitions, and no comparison to the cited ChatIoT assistant. These are not local presentation issues; they require new experiments (human evaluation, external ground truth, proper baselines) to be addressed. The paper is not ready for publication in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read the ICoT paper. The core idea is simple and sensible: split the prompt into two stages, first having the LLM analyze the user's role and the vulnerability into structured JSON, then feeding that analysis back to generate the final response. That is a clean, no-fine-tuning way to personalize IoT security advice, and the writing makes the design easy to follow. Credit where due: the method is clearly presented, the two-stage decomposition is a legitimate extension of CoT into the IoT security domain, and the authors do not oversell the need for fine-tuning.\n\nThe soft spot is the evaluation, and it is load-bearing. Table I is the only quantitative support for \"significantly improves accuracy and reliability.\" The judge is a fixed GPT-4o model scoring answers that, for the GPT-4o rows, the same model family generated. ICoT outputs are deliberately structured, verbose, and role-aware—exactly the surface features an LLM judge would reward when asked to rate detail and technicality. Without human ratings, a baseline against the closest prior system (ChatIoT), or ground truth, the deltas are uninterpretable as accuracy gains. There is also no sample size, no variance, no statistical test. The metric names shift between text, Figure 3, and Table I, and the table contains a typo like \"3.3.66.\" These are fixable, but they erode confidence.\n\nThe paper's own Discussion admits ICoT can produce redundant information and that LLMs hallucinate. That makes the \"higher accuracy\" claim especially shaky, since accuracy cannot be established by the same kind of model that might be fabricating the advice. The stress-test note is accurate on this point.\n\nSo, the contribution is a real but modest extension, and the reported evidence cannot carry the central claim as stated. That said, the idea is not dead—it just needs a proper evaluation. The authors should run a human study or at least compare against a retrieval-augmented baseline like ChatIoT, report per-item counts and inter-annotator agreement, and fix the metric naming.\n\nWho is this for? Anyone building LLM-based security assistants would get a useful design pointer from the two-stage prompt, and a workshop or short paper could be a home for a revised version. As it stands, I would not cite the empirical result, but I would send it to peer review rather than desk reject it—the flaws are real but fixable, and the method is understandable enough that a referee can give concrete guidance.\n\nRecommendation: engage with the paper if it crosses your desk, but with a clear request for a real evaluation before acceptance.","headline":"The two-stage role-and-vulnerability prompt is a sensible, clearly presented idea, but the only evidence for the central claim is an unvalidated LLM judge scoring the same model's outputs, so the headline assertion of higher accuracy and reliability is not supported.","tokens_in":15121,"tokens_out":1891,"would_cite":false,"duration_ms":17812,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes ICoT, a two-stage chain-of-thought prompt that makes LLMs analyze a user's role and a vulnerability's type, location, and impact before generating personalized IoT security advice, and reports that this prompt…","keywords":["Internet of Things security","large language model","chain-of-thought prompting","ICoT","vulnerability analysis","personalized security advice","GPT-4o","DeepSeek"],"falsifier":"Have independent human IoT security experts blind-rate the same ICoT and LLM-only answers for factual correctness and actionability, or run a swap test in which the labels 'ICoT' and 'LLM only' are exchanged before GPT-4o scores them; if human experts find no consistent advantage, or if GPT-4o's scores track the labels instead of the content, the claim that ICoT improves understanding of IoT security issues is falsified.","tokens_in":14065,"feed_emoji":"🛡️","tokens_out":7149,"duration_ms":56206,"temperature":0.7,"pith_summary":"The paper tries to establish that a carefully structured prompt can make off-the-shelf large language models produce better, more personalized security advice for Internet of Things devices, with no fine-tuning. The proposed method, ICoT, is a two-stage chain-of-thought scheme: the model first analyzes the person asking (their knowledge, goals, and constraints) and the vulnerability (its type, location, and impact), and then uses that analysis to write the final answer. Across GPT-4o, GPT-4o-mini, DeepSeek-V3, and DeepSeek-R1, and for three user roles, ICoT responses score above plain LLM responses on reliability, relevance, detail, technicality, and friendliness, as judged by GPT-4o. If the claim holds, it gives a parameter-free way to upgrade LLM-based security assistants in heterogeneous IoT environments. The weight-bearing part of the evaluation is the judge, since the fixed evaluator is one of the models under test.","feed_headline":"Split-prompt ICoT lifts LLM security advice across four models","feed_subtitle":"Role and vulnerability analysis before answering raises relevance, detail, and friendliness scores","key_machinery":"The central object is the ICoT prompt, a two-round chain-of-thought template. Round one asks the LLM to output a JSON object describing the user's knowledge, goals, and requirements alongside the vulnerability's type, location, and impact ($S_g = f_\\theta(P_{in}^{(1)})$). Round two injects that JSON as context and instructs the model to answer using it ($R = f_\\theta(P_{in}^{(2)})$). The work it does is to force the model to separate who is asking from what is broken before recommending a fix, which is what the authors argue produces context-aware and personalized security advice without retraining or access to model parameters.","core_discovery":"The central discovery is that decomposing a security query into a role-and-vulnerability analysis, serialized as JSON, and feeding that analysis back into the model produces responses the authors judge more reliable, relevant, detailed, technically precise, and user-friendly than the same model answering directly. The pipeline is formalized as two inferences, $S_g = f_\\theta(P_{in}^{(1)})$ for the intermediate analysis and $R = f_\\theta(P_{in}^{(2)})$ for the final response, where the first prompt is $[V][T][S_{in}]$ and the second is $[V][S_g][C][T]$. The reported results show positive deltas in nearly every model-role-metric cell of Table I, with the largest gains often on technicality and relevance, and the paper acknowledges that redundant output and hallucination can blunt the gains in some cases.","pith_inferences":["A cheap falsification-style check follows directly from the setup: relabel the two answer sets before asking GPT-4o to score them; if scores follow the labels rather than the content, part of the reported gain is judge self-preference for structured, role-aware text rather than improved security understanding.","The role-and-vulnerability JSON profile could be carried across a multi-turn conversation, turning a one-shot assistant into a persistent advisor that remembers the user's context; the paper does not explore this.","Pairing ICoT with retrieval of up-to-date vulnerability databases would directly address the paper's stated weakness that descriptions can be stale or incomplete, since the prompt structure is agnostic to where the vulnerability text comes from.","Sending user roles, goals, and infrastructure context to a commercial LLM API raises a privacy cost the paper mentions but does not quantify; a local or privacy-preserving inference variant is a natural next step."],"forward_implications":["Because ICoT operates only through prompts, any existing LLM-based assistant can adopt it by changing the prompt, with no retraining and no access to model parameters.","The two-stage design is user-extensible: users can supply their own role templates and choose which vulnerability characteristics to analyze, so the same mechanism covers roles beyond the three tested.","The authors argue the method carries over to other security domains, naming industrial control systems, smart grids, and smart-home security as direct targets.","For non-expert users, the measured gains in friendliness and relevance translate into advice that is easier to act on, which is the practical payoff the paper emphasizes.","The paper's own limitations imply the gains are not uniform: verbosity, hallucinated details, and inconsistent reasoning on poorly specified vulnerabilities can reduce clarity and may offset the improvements in some cases."],"supporting_citations":[{"why":"The direct prior LLM-based IoT security assistant that ICoT positions itself against; the paper also cites it when arguing the paired-judge setup is fair.","marker":"[3]"},{"why":"Supplies DeepSeek-V3 as one of the four base models on which ICoT is evaluated.","marker":"[28]"},{"why":"Introduces chain-of-thought prompting, the reasoning technique ICoT adapts to IoT security.","marker":"[38]"},{"why":"Identifies GPT-4o as a base model and as the fixed evaluator that produces the Table I scores.","marker":"[41]"},{"why":"Supplies DeepSeek-R1 as one of the four base models on which ICoT is evaluated.","marker":"[42]"},{"why":"Provides the VARIoT Vulnerabilities and Exploits datasets that supply the vulnerability descriptions used in the prompts.","marker":"[43]"},{"why":"Underpins the paper's own limitation that chain-of-thought reasoning does not always reflect the model's true assessment, used to temper the reliability claims.","marker":"[44]"}],"fun_headline_variants":["ICoT split-prompt lifts IoT security advice in all four models","Role-vulnerability JSON step makes LLM security answers more reliable","Chain-of-thought decomposition boosts LLM IoT security across models","Two-step ICoT prompt improves LLM security responses for IoT","LLM security assistant benefits from split role and vulnerability analysis"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison's validity rests on the assumption that GPT-4o, acting as the fixed judge, scores ICoT and LLM-only answers impartially and on their factual merits; if the judge merely rewards the structured, role-aware, verbose style that ICoT deliberately produces, the reported gains are an artifact rather than a genuine improvement in understanding.","fun_headline_variants_meta":{"raw":{"variants":["ICoT split-prompt lifts IoT security advice in all four models","Role-vulnerability JSON step makes LLM security answers more reliable","Chain-of-thought decomposition boosts LLM IoT security across models","Two-step ICoT prompt improves LLM security responses for IoT","LLM security assistant benefits from split role and vulnerability analysis"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000214,"raw_usage":{"total_tokens":1433,"prompt_tokens":961,"completion_tokens":472,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":384}},"tokens_in":577,"tokens_out":472,"duration_ms":4803,"temperature":1.0,"reasoning_tokens":384,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:14:57.873292+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent human IoT security experts blind-rate the same ICoT and LLM-only answers for factual correctness and actionability, or run a swap test in which the labels 'ICoT' and 'LLM only' are exchanged before GPT-4o scores them; if human experts find no consistent advantage, or if GPT-4o's scores track the labels instead of the content, the claim that ICoT improves understanding of IoT security issues is falsified.","supporting_citations":[{"cited_title":"Automatic actionable information processing and trust man- agement towards safer internet of things,","cited_arxiv_id":null,"evidence_quote":"Provides the VARIoT Vulnerabilities and Exploits datasets that supply the vulnerability descriptions used in the prompts."},{"cited_title":"Reasoning models don’t always say what they think","cited_arxiv_id":null,"evidence_quote":"Underpins the paper's own limitation that chain-of-thought reasoning does not always reflect the model's true assessment, used to temper the reliability claims."}],"review_version":1}