{"id":"34d0e325-1b02-4460-bc12-08664ae5396e","arxiv_id":"2607.26390","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A taxonomy of 32 security and privacy issues in LLM-native IDEs derived from 446 verified Reddit posts, showing most reported problems are system-level design flaws rather than LLM failures.","lead":"This paper mined Reddit to catalog 446 developer-reported security and privacy problems in AI-powered IDEs like Cursor and Copilot, and found most reports blame the IDE's system design, not the language model. It is a useful map of real-world complaints and workarounds for anyone building or choosing these tools.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 7/10 system-level attribution rests on an unvalidated, post-hoc classification; the paper never defines or audits the system-vs-LLM labeling, so the headline conclusion may be an artifact of category construction.","rationale":"The reader's CONDITIONAL verdict is appropriate and I do not see grounds to move it. The paper has real strengths: a transparent collection pipeline, manual validation of all LLM-positive posts with high inter-rater agreement (Cohen's kappa = 0.967), a recall-oriented filter audit with zero false negatives in the sampled negative set, preservation of Reddit identifiers, and an honest threats-to-validity section that explicitly disclaims confirmed vulnerabilities. My concern is more specific than the reader's weakest assumption. The reader emphasized that Reddit anecdotes may misattribute failures; I focus on the fact that the system/LLM split—the paper's headline quantity—has no described methodology at all. It is not merely a reliability question about the raw data, but a missing validation step in the analysis pipeline. The taxonomy coding is carefully reported, but the second-order coding (system vs LLM) is not. Because the categories themselves are system-flavored, the 7/10 result could be an artifact of the category-construction choices rather than an independent finding. The concrete test I propose—an independent re-coding with a pre-registered codebook and agreement measurement—would settle whether the split is reproducible. Until then, the strongest claim should be read as a hypothesis derived from developer discussions, not a measured vulnerability attribution. This does not change the reader's verdict: the paper remains a useful qualitative taxonomy and mitigation catalog, but its central quantitative claim is weaker than presented.","tokens_in":20475,"tokens_out":2270,"duration_ms":25166,"concrete_test":"Download the Zenodo replication package and locate the per-post system/LLM/both labels behind Figure 11. Have two independent annotators, blind to the authors' labels, reclassify all 446 posts using a pre-registered codebook that defines 'system-level' and 'LLM-level' with concrete decision rules (e.g., 'would the reported failure occur if the model were perfect but the access-control layer unchanged?'). Compute Cohen's kappa between the original and independent labels and compare the resulting 7/10 split. If kappa < 0.6 or the system-level percentage shifts by more than 10 percentage points, the headline attribution is not robust and should be reported as an unvalidated coding judgment, not a quantitative result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claim—\"7 out of 10 security and privacy issues were system-level issues\" (Section 5)—depends on a binary (actually ternary) attribution of each of the 446 posts to \"system\" vs \"LLM\" vs \"both.\" Nowhere in the manuscript is the procedure for this attribution described: no codebook, no decision rule, no inter-rater agreement, and no worked examples. The claims points to a replication package, but the method is absent from the paper itself. This matters because the attribution is not a neutral measurement: the taxonomy categories are themselves defined in system-centric terms. C1 (\"Unauthorized File Operations,\" 43.1% of security issues) is framed as the LIDE executing actions without enforcing access controls; C6 (\"Lack of Transparency,\" 45.9% of privacy issues) is inherently a system-opacity property. If the coders labeled posts using these category definitions and then mapped categories to system/LLM, the 7/10 split is partly tautological—a restatement of how the categories were built, not an empirical discovery about where defects originate. The authors' own validity section concedes that Reddit posts lack configuration detail and root-cause evidence, yet the split requires exactly that root-cause inference (e.g., deciding whether unauthorized file access is due to the LLM's plan or the system's failure to enforce constraints). Without a validated attribution protocol, a reader cannot tell whether the headline conclusion would survive re-coding by independent annotators. This is the weakest load-bearing point because the abstract's central assertion rests entirely on it.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper investigates security and privacy issues in LLM-native IDEs (LIDEs) by mining Reddit. The authors collect a large corpus (reported inconsistently as 1.1M posts from 29 subreddits or 15.4M posts/comments from 46 subreddits), apply an LLM-based recall-oriented filter, manually validate all positive candidates, and identify 446 posts and 6,280 comments about LIDE security/privacy issues. They qualitatively code the posts into a 10-category taxonomy containing 32 low-level issues, extract 13 mitigation strategies from the comments, and derive six design recommendations. The headline finding is that '7 out of 10' reported issues are system-level (integration logic, permissions, transparency) rather than LLM-level, with about 1 in 10 attributed to both.","tokens_in":20831,"tokens_out":9217,"duration_ms":101360,"significance":"If the 70/20/10 system-vs-LLM split were supported, it would shift the security research agenda for AI coding tools from model alignment toward the IDE integration layer: file access, permission enforcement, autonomous action, and transparency. The paper has notable strengths: all LLM-positive posts were manually verified with near-perfect agreement (Cohen's kappa = 0.967), the recall-oriented filter was audited at 1.00 recall on 200 negative samples, and the authors share a replication package with data, prompts, codebooks, and scripts. The taxonomy of reported issues and the catalog of mitigation strategies are useful community resources independent of the split. However, the central quantitative claim is not backed by a described, audited attribution method, which currently limits the paper's contribution to the taxonomy and mitigation findings.","major_comments":[{"comment":"The central claim '7 out of 10 security and privacy issues were system-level issues' is not reproducible from the manuscript. Section 5 refers only to 'the replication package'; no codebook, decision rule, coder agreement, or worked examples for the ternary system/LLM/both attribution are given in the paper, and raw counts are not reported. The taxonomy itself is defined in system-centric terms (e.g., C1 'Unauthorized File Operations', C6 'Lack of Transparency'), so if the attribution was made by mapping taxonomy categories to system/LLM, the 70/20 split is partly a restatement of category construction rather than an empirical finding about root cause. The paper should describe the attribution protocol, report inter-rater reliability for this label, give per-category agreement, and address multi-label posts (Table 3) explicitly.","section":"§5, Fig. 11"},{"comment":"The construct-validity paragraph concedes that Reddit posts 'lack configuration detail or root-cause evidence,' yet the 7/10 split requires exactly that root-cause inference (e.g., deciding whether unauthorized file access is due to the LLM's plan or the system's failure to enforce constraints). The abstract's 'most issues in LIDEs stem from system-level design choices' therefore overstates what the data support. At minimum, the claim should be restricted to 'developer-reported issues' or to 'issue discussions on Reddit,' and the split should be presented as coder judgment. The authors' reproduction attempts (Fig. 1) are a promising triangulation but are not described in enough detail to validate the attribution.","section":"§7"}],"minor_comments":[{"comment":"The text says '17.8% of security vulnerabilities to unsafe generation (O3)', but Section 3.2 and Figure 6 report C3 as 18.2%. Please reconcile or explain the different denominator.","section":"§5, 'Verification-First Code Recommendations'"},{"comment":"The abstract states 1.1M posts from 29 subreddits; Section 2.1 says 15.4M posts/comments from 46 subreddits, and Section 2.2 says the final dataset came from 29 subreddits. Clarify what the 1.1M counts and why the corpus size differs.","section":"Abstract vs §2.1/§2.2"},{"comment":"The text says Cursor accounts for '130 of the 376 IDE-referenced posts', but Section 3.1 says 383 posts explicitly named a LIDE, and the Figure 4 counts sum to 383. Correct the 376 figure.","section":"§3.4"},{"comment":"The assertion that EXTOOL is 'responsible for 5 out of 10' identified issues is causal and is not supported by the described feature-support matrix (Table 5) or Figure 12. Reframe as a hypothesis or provide an explicit association/attribution analysis.","section":"§5, 'Promises and Perils of Coding Agents'"},{"comment":"The audit of 200 LLM-negative posts found 0 missed relevant posts; report a confidence interval (e.g., binomial 95% CI) so the reader can calibrate the residual false-negative risk.","section":"§2.2"},{"comment":"The 'General Security Concerns' (15.5%) and 'General Privacy Concerns' (26.8%) categories are large residual buckets. Consider breaking them down or discussing what they contain, as they limit the precision of the taxonomy.","section":"§3.2/§3.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid qualitative study, and the taxonomy and mitigation catalog could be useful community resources. The main issue is fixable: the authors should either document the system/LLM attribution protocol and present the split as coder judgment with appropriate caveats, or demote the 7/10 split from the abstract and headline. I would not reject on the current evidence, but the central claim needs to be earned before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth knowing about: this is the first systematic taxonomy of security/privacy issues in LLM-native IDEs built from unfiltered Reddit discussion. The pipeline is genuinely careful. The LLM-based filter was audited for recall (1.00 on 200 negatives), all 3,801 candidates were manually screened with Cohen's kappa 0.967, and the final 446 posts plus 6K comments were open-coded iteratively with consensus. The resulting 10-category taxonomy (32 issues) and the 13 mitigation strategies are useful design inputs, and the authors are honest in their threats-to-validity section that these are developer-reported issues, not confirmed vulnerabilities.\n\nThe soft spot is the one that matters. The abstract and Section 5 claim that 7-out-of-10 issues are system-level rather than LLM-level, with Figure 11 quantifying that per IDE. But the manuscript never describes the procedure for that attribution: no codebook, no decision rule, no inter-rater agreement, no worked examples. The text simply says 'see replication package.' That would be okay for a coding detail, except the categories themselves are defined in system-centric terms—C1 is 'Unauthorized File Operations,' C6 is 'Lack of Transparency.' If posts were labeled using those definitions and then mapped onto system-vs-LLM, the 7/10 split is partly a restatement of how the categories were built, not an empirical finding about where defects originate. The authors' own validity section concedes Reddit posts lack configuration detail and root-cause evidence, but that is exactly the evidence the attribution requires. So the headline conclusion is load-bearing and under-evidenced.\n\nMinor notes: the tool frequency shares (Cursor at 130 of 383 named posts) are discussion-frequency artifacts, which the paper acknowledges; and I could not independently verify the Zenodo package or quoted posts, though the Reddit IDs help.\n\nWho gets value: LIDE designers, and SE/HCI researchers studying developer trust and AI-assisted tools. The taxonomy and mitigation list are worth a serious look. The system-vs-LLM split needs to be either substantiated with a documented coding schema and reliability numbers, or softened to a qualitative observation. I'd send it to peer review, but flag that the central claim needs to be fixed before acceptance.","headline":"A careful taxonomy of developer-reported security/privacy complaints in AI IDEs, but the headline '7 in 10 issues are system-level' is a coding judgment the paper never documents.","tokens_in":21321,"tokens_out":2080,"would_cite":true,"duration_ms":24222,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Most security and privacy problems in AI-native IDEs come from the tools' system design, not the language models themselves.","keywords":["LLM-native IDE","AI coding assistants","security and privacy","developer-reported issues","taxonomy","Reddit analysis","autonomous agents","secrets leakage"],"falsifier":"A controlled experiment that configures several LIDEs with identical projects containing secrets in .env files, explicit ignore rules, and approval gates, then instruments the tool to log every file read, command executed, and network transmission. If, across thousands of runs, the majority of security violations trace to the model's output rather than to the integration layer's enforcement logic, the paper's 7/10 system-level attribution would be overturned.","tokens_in":20402,"feed_emoji":"🔐","tokens_out":5587,"duration_ms":54913,"temperature":0.7,"pith_summary":"This paper claims that the security and privacy problems developers encounter in LLM-native IDEs (LIDEs) — AI assistants embedded in coding tools like Cursor, Copilot, and Codex — mostly originate in the integration layer rather than in the language models themselves. Analyzing 446 Reddit posts and 6,000 comments from 29 subreddits, the authors build a taxonomy of 32 reported issues grouped into 10 categories, from unauthorized file operations and destructive autonomous actions to opaque data flows and cross-session context leakage. They report that 7 out of 10 issues are system-level, arising from how LIDEs access files, enforce permissions, and execute autonomous actions, while only 2 out of 10 trace to the LLM's own reasoning. Developers respond with external safeguards — sandboxing, manual code review, version control, and sensitive-file protection — a pattern the paper reads as widespread mistrust of built-in LIDE controls. The stakes: if the paper is right, making AI coding tools safer means redesigning the IDE as a security boundary, not just improving the model.","feed_headline":"Seven in ten AI coding-tool security issues are design flaws","feed_subtitle":"Analysis of 446 Reddit-reported incidents shows the integration layer, not the LLM, is the main risk.","key_machinery":"The central instrument is the taxonomy itself: 32 low-level issue codes grouped into 10 high-level categories, built through open and axial coding grounded in ISO/IEC 27001 (the CIA triad) and ISO/IEC 29100 (privacy principles). This taxonomy converts unstructured Reddit anecdotes into structured claims about where LIDEs fail. The accompanying system-level versus LLM-level attribution is the lens that carries the argument: for each issue, the authors judged whether the root cause lay in integration logic (file access, permission enforcement, autonomous action, data flow) or in model reasoning (unsafe generation, hallucination). A recall-oriented LLM filter plus full manual verification of al","core_discovery":"The central discovery, on the paper's own terms, is that LIDE security and privacy failures are primarily a system-engineering problem. The authors analyzed 1.1 million Reddit posts, manually verified 446 incident reports, and coded them into a taxonomy of 32 issues under 10 categories (five security, five privacy). The load-bearing result: roughly 7 of 10 issues are attributed to system-level design choices — an IDE reading a .env file despite ignore rules, bypassing approval gates, or deleting files autonomously — while about 2 in 10 trace to the LLM itself. The paper also documents 13 mitigation strategies developers use because they do not trust built-in safeguards, and concludes that LI","pith_inferences":["Because the 7/10 split is inferred from Reddit anecdotes rather than instrumented reproductions, the ratio most plausibly describes the distribution of what developers notice and report, not the true prevalence of vulnerabilities; a controlled benchmark could test whether the same split holds under systematic probing.","If system-level issues dominate, then security research on AI coding tools should shift emphasis from prompt-injection and model-alignment failures toward the permission model of the IDE's agent harness — the layer that decides which files can be read and which commands can run.","A natural testable extension is a security benchmark for LIDEs: run a standard set of adversarial development tasks (secrets in .env files, explicit 'do not modify' instructions, malicious MCP servers) across tools and measure how often the integration layer violates the stated constraints; the paper's taxonomy provides the category labels for such a benchmark.","The paper's reliance on public Reddit reports means enterprise and closed-source deployments are underrepresented; if LIDEs are adopted in regulated settings, the same taxonomy could be validated against incident reports from those environments to see whether system-level issues still dominate."],"forward_implications":["LIDE vendors should prioritize permission enforcement, file-access boundaries, and approval gates over model-level safety tuning, because most reported failures live in the integration layer.","Designing LIDEs as software systems first implies native verification layers that validate generated code and tool calls before they act, rather than relying on developers to catch mistakes after the fact.","The documented reliance on external mitigations (sandboxing, manual review, version control, sensitive-file protection) indicates that built-in safeguards are not currently meeting user expectations; closing that gap is a concrete design target.","The finding that external tool execution (e.g., MCP servers and plugins) is implicated in a disproportionate share of issues points to the need for centralized vetting and compliance processes for coding agents and extensions, analogous to app-store review."],"fun_headline_variants":["AI coding tools' privacy flaws are design bugs, not AI","Most AI IDE security risks come from design, not models","LLM IDEs fail on system design, not the LLM itself","Reddit analysis: AI dev tools' flaws are mostly architectural","Study: 7 in 10 AI code editor issues are design-level"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The central claim rests on treating informal Reddit posts as reliable enough to attribute each issue to either system-level design or LLM-level behavior, even though the posts often lack configuration details and confirmed root-cause evidence.","fun_headline_variants_meta":{"raw":{"variants":["AI coding tools' privacy flaws are design bugs, not AI","Most AI IDE security risks come from design, not models","LLM IDEs fail on system design, not the LLM itself","Reddit analysis: AI dev tools' flaws are mostly architectural","Study: 7 in 10 AI code editor issues are design-level"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0002,"raw_usage":{"total_tokens":1228,"prompt_tokens":780,"completion_tokens":448,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":359}},"tokens_in":524,"tokens_out":448,"duration_ms":5167,"temperature":1.0,"reasoning_tokens":359,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T01:39:46.409957+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled experiment that configures several LIDEs with identical projects containing secrets in .env files, explicit ignore rules, and approval gates, then instruments the tool to log every file read, command executed, and network transmission. If, across thousands of runs, the majority of security violations trace to the model's output rather than to the integration layer's enforcement logic, the paper's 7/10 system-level attribution would be overturned.","supporting_citations":[],"review_version":2}