{"id":"36c49bc7-d95b-4a04-8d91-80e65ee6e700","arxiv_id":"2608.10622","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A study of 12,110 .cursorrules files on GitHub shows these AI prompt files appear mostly in small personal projects, focus on code quality, and rarely mention security.","lead":"This paper analyzes over 12,000 .cursorrules files from GitHub to see how developers use these AI-prompt configuration files. It finds the files mostly appear in small, low-activity projects and focus on code style rather than security.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Survivorship bias: the dataset only contains files present in April/October 2025 snapshots, so .cursorrules files created and deleted before April 2025 are unobserved, which undermines the emergence timeline and the 'toy project' interpretation.","rationale":"The paper is a valuable first look at .cursorrules, with a large quantitative dataset and transparent qualitative coding. The reader's concern about keyword-based partitioning of GitHub's 1,000-result cap is legitimate but secondary: without the actual keyword list we cannot quantify the coverage loss, and the direction of bias is speculative. The survivorship issue is more fundamental because it follows from the data source itself: GitHub Code Search only sees current state, and the paper's own deprecation timeline guarantees that a non-random subset of files was removed before the first snapshot. This directly threatens the historical emergence claim (RQ1.1) and the 'toy project' claim, because the files that remain are exactly the ones that were not migrated away by active projects. The paper should either restrict its claims to files present in 2025 or add a historical robustness check. Since the finding is not internally inconsistent and can be repaired with a clearly specified analysis, a conditional acceptance remains appropriate; the condition should require either the GH Archive cross-check or a pre/post deprecation split, and softening of causal language about 'toy projects'.","tokens_in":15013,"tokens_out":8719,"duration_ms":82337,"concrete_test":"Query the GitHub Archive (GH Archive) on Google BigQuery for all PushEvents from January 2023 through September 2025 whose commit payload includes a file named '.cursorrules'. For each such event, record the repository ID and commit timestamp. Compare the repository metadata (stars, contributors, commit counts) of these events against the paper's 11,427 repositories. If the GH Archive reveals substantial numbers of .cursorrules commits in high-activity, multi-contributor repositories whose files were later deleted before the April 2025 snapshot, then the paper's emergence timeline and 'toy project' conclusion are survivorship artifacts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The dataset is built from GitHub Code Search at two snapshots (April and October 2025), which returns only files present on the default branch at query time. Consequently, any .cursorrules file created and deleted before April 2025 is unobservable, and files deleted between snapshots are only captured if they survived to April. The paper uses creation timestamps from this survivor set to claim that '.cursorrules files emerged rapidly from mid-2024' (RQ1.1) and that adoption is concentrated in low-activity, single-maintainer 'toy projects' (Sections 4.1.3 and 5). The February 2025 deprecation announcement and the observed decline in new file creation (Figure 3) create a plausible mechanism for exactly this bias: active, professional projects may have adopted .cursorrules early, then migrated to .cursor/rules or AGENTS.md, removing the file, while abandoned or toy projects left it in place. The Threats to Validity section acknowledges API incompleteness but never notes that every finding is conditional on survival to April 2025. The qualitative comparison to .mdc files is also based only on files that survived, and the 14 .mdc files are insufficient to establish 'continuity of themes' for the broader population.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a mixed-methods empirical study of .cursorrules files in public GitHub repositories. The authors collect 12,110 such files from 11,427 repositories using keyword-partitioned GitHub Code Search across two snapshots (April and October 2025), analyze file and repository metadata to address RQ1-RQ2, and perform thematic analysis on 65 files (51 .cursorrules and 14 .mdc) to address RQ3-RQ4. The main claims are that .cursorrules adoption rose rapidly from mid-2024, is concentrated in small, low-activity, single-maintainer repositories, that file content is dominated by code quality and engineering practices, and that security receives little attention. The paper also reports thematic continuity with the newer .mdc format.","tokens_in":15279,"tokens_out":6621,"duration_ms":59402,"significance":"If the findings hold, this is one of the first large-scale empirical characterizations of Cursor's legacy prompt configuration files, and the codebook plus the anonymous replication package are reusable assets for future research on prompt-file engineering. The quantitative dataset is substantial, and the qualitative analysis follows a systematic thematic-analysis procedure with inter-coder reliability reporting. The main caveats are that the sampling strategy and the survivorship of files are not fully accounted for, which may bias the prevalence and temporal claims, and that the 'toy projects' interpretation goes beyond what the repository metrics can establish.","major_comments":[{"comment":"The keyword-partitioned GitHub Code Search strategy is not described with enough detail to assess representativeness. The paper never lists the keywords used, nor does it report how many queries hit the 1,000-result cap or whether saturation was reached. Because GitHub Code Search returns at most 1,000 results per query, any .cursorrules file whose content matches none of the chosen keywords is invisible to the search, so the 12,110-file corpus is not necessarily a random or complete sample of public .cursorrules files. The abstract's quantitative generalizations (e.g., 51.7% web languages, 68.5% single-maintainer repositories) are conditional on this sampling frame. The authors should publish the full query list and provide a coverage analysis, for example by comparing keyword-based retrieval against a sample from an independent source or by measuring the number of queries that hit the result cap.","section":"§3.1, Data Collection"},{"comment":"The dataset is built from two snapshots (April and October 2025) of files present on the default branch at query time. Any .cursorrules file created and deleted before April 2025 is unobservable, and files deleted between snapshots are only represented if they survived to April. The February 2025 deprecation announced in §2 provides a concrete mechanism for such deletion: active projects migrating to .cursor/rules or AGENTS.md would remove .cursorrules, while abandoned projects would leave them in place. Consequently, the emergence timeline in Figure 3 and RQ1.1 and the characterization of host repositories as small, low-activity, single-maintainer (Sections 4.1.1 and 5) may reflect survivor bias rather than the true population. This threat is not mentioned in §5, which only notes API incompleteness and snapshot uncertainty. The authors should quantify the extent of deletion where possible (e.g., by tracking repositories that had a .cursorrules file at the April snapshot but not in October) or substantially soften the causal and prevalence claims.","section":"§3.1 and §5, Threats to Validity"},{"comment":"The inference from repository size, commit counts, and contributor counts to \"toy projects rather than professional development\" is not supported. Small repositories with few commits and a single maintainer are common for legitimate personal projects, libraries, and early-stage startups; low activity does not imply non-professional or educational use. This is a value-laden interpretation of a structural proxy. The data support only the descriptive statement \"mostly small, low-activity, single-maintainer repositories.\" The abstract and §5 should be reworded to avoid the \"toy projects\" characterization, or the authors should provide additional evidence (e.g., repository descriptions, README content, or package publication status) to justify it.","section":"§4.1.1 (RQ1.3) and §5"},{"comment":"There is an internal inconsistency about how repository counts were obtained. The text first states that counts such as commits, releases, pull requests, and contributors were capped at 100 and recorded as 101 when exceeding the threshold, but then states that the same attributes were retrieved via the GitHub GraphQL API, which exposes totalCount fields and reduces the number of API calls. If GraphQL totalCount was used, the counts are exact and the 101-cap does not apply. This matters because RQ1.3's activity tiers (e.g., 62.5% with 50 or fewer commits, 68.5% single-maintainer) and the commit-interval analysis in RQ2.2 depend on these counts. Please clarify which mechanism was actually used, and if the 100-cap was applied to some attributes, state which ones and re-analyze the affected distributions with exact counts.","section":"§3.1, Metadata Collection"},{"comment":"The 14 .mdc files used for the cross-type comparison are not clearly sampled. The text says \"we randomly selected 65 prompt files – 51 .cursorrules and 14 newer .mdc files,\" but it does not state whether the .mdc files were drawn from the same random procedure or from a separate convenience sample, nor does it describe the sampling frame for .mdc files. Since the qualitative claim of \"continuity of themes and topics\" between .cursorrules and .mdc (abstract, RQ3) rests on comparing 51 and 14 files, the authors should specify the .mdc selection process and acknowledge that 14 files cannot establish generalizable continuity. The existing caveat that the qualitative sample is not representative is helpful, but it should be connected explicitly to the .mdc comparison.","section":"§3.3, Qualitative Analysis"}],"minor_comments":[{"comment":"The paper reports a \"65-code codebook,\" but §3.3.1 states that after cross-type coding of .mdc files two new codes were added (Rule File Description, Scopes & Nested Structures; Prompt/Rule Example/Template). If these are included in the final codebook, the count should be 67; if not, the abstract's \"65-code\" claim should be clarified.","section":"Abstract and §3.3.1"},{"comment":"The dataset is described as covering \"15 April 2024 to 30 September 2025,\" yet §4.1.1 reports an outlier file created on May 29, 2023. Please state explicitly whether the outlier is excluded from all analyses and whether Figure 3's time axis is truncated accordingly.","section":"§3.1 and §4.1.1"},{"comment":"The legend label \"t category\" appears truncated; it should read \"Δt category\" to match the text.","section":"Figure 8"},{"comment":"The prose reports the Security theme as 4.2% in Table 5, then states that Security Awareness, Mechanisms & Practices combined with Access Control reaches roughly 4.4%. Please reconcile these percentages and state explicitly whether Access Control is included in the Security theme in Table 5.","section":"§4.2.1 and Table 5"},{"comment":"The term \"toy projects\" should be removed or replaced with descriptive language throughout the paper, including the abstract and conclusion, for the reasons given in Major Comment 3.","section":"§5 and Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a software engineering venue and the dataset is a useful contribution, but the sampling and survivorship issues need to be addressed before publication. The 'toy projects' wording should be softened, and the metadata-collection inconsistency about GraphQL versus capped counts should be resolved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a read. This is the first large-scale look at .cursorrules files, and it fills a real gap: prior work stopped at AGENTS.md and related markdown formats. The dataset (12,110 files) is substantial, and the mixed-methods design is honest and careful. The qualitative part follows a legitimate inductive coding process with inter-coder reliability reported (Alpha 0.674 then 0.897), and the authors explicitly flag the small sample and the subjectivity of thematic analysis. The related-work placement is accurate, including the one self-citation, which supports a methodological limitation and is not load-bearing. Credit where due: this is a solid descriptive study that future work on prompt configuration files will likely build on.\n\nNow the soft spots, in proportion. The biggest one is survivorship bias, and the stress-test note lands. The dataset is built from two GitHub Code Search snapshots (April and October 2025), so files created and deleted before April 2025 are simply invisible. The paper uses creation timestamps from this survivor set to claim that adoption 'emerged rapidly from mid-2024' and that repositories are 'toy projects.' Given Cursor deprecated .cursorrules in February 2025, professional projects plausibly migrated to .cursor/rules or AGENTS.md, deleting the old file, while abandoned or toy projects left it in place. That mechanism fits the observed decline in new file creation and would bias both the timeline and the repo-profile interpretation. The Threats to Validity section acknowledges API incompleteness and the snapshot merging, but never says that every finding is conditional on survival to April 2025. That needs to be stated, and the 'toy projects' phrasing in the discussion and abstract is an interpretive leap, not a finding. The search strategy's 1,000-result cap is acknowledged but the coverage loss is never quantified, which matters for the language and activity distributions. The qualitative continuity claim between .cursorrules and .mdc files rests on 14 .mdc files; that is thin, and 'continuity of themes' should be presented as preliminary. These are not fatal flaws; the central quantitative claims about repository sizes and file evolution are still useful, but the temporal emergence curve and the 'toy project' label should be softened.\n\nWho is this for? Researchers studying AI-assisted development workflows, prompt engineering, and repository mining. Tool designers might also skim it for adoption signals. It deserves a serious referee; I would send it out with a request for revisions focused on the survivorship-bias caveat and the interpretive framing, not on redoing the analysis. I would cite it in future work on prompt files, with the caveat in mind.","headline":"A useful first empirical baseline for .cursorrules files, but the temporal and 'toy project' claims should be softened given survivorship bias and an unquantified search cap.","tokens_in":15790,"tokens_out":1110,"would_cite":true,"duration_ms":12810,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Cursor rule files cluster in small hobby repos, study finds","keywords":["Cursor","cursorrules","prompt engineering","prompt configuration files","mining software repositories","GitHub","thematic analysis","security smells"],"falsifier":"Compute a complete census of public .cursorrules files from a full GitHub archive rather than keyword-partitioned search queries; if most files turn out to live in large, multi-contributor, actively committed repositories, the paper's toy-project claim collapses. The security claim could be tested separately by measuring whether code generated with security-themed prompt files has a lower vulnerability density than code generated without them.","tokens_in":14840,"feed_emoji":"🤖","tokens_out":8268,"duration_ms":69211,"temperature":0.7,"pith_summary":"This paper tries to establish where Cursor's project-level prompt configuration files actually live and what they say. The authors collected 12,110 .cursorrules files from 11,427 GitHub repositories and found that the format spread rapidly from mid-2024, then settled into small, low-activity, single-maintainer repositories that look more like toy projects than production codebases. A thematic analysis of 65 sampled files shows that developer instructions focus on code quality, engineering practices, project structure, and maintainability, while security content is rare and potential security smells appear in about 1.5% of coded instances. The study also argues that the themes in the legacy .cursorrules format carry over to the newer .mdc files, so the findings are not just about an outdated format.","feed_headline":"Cursor rule files cluster in small hobby repos, study finds","feed_subtitle":"A study of 12,110 files finds prompt rules rarely reach big projects and seldom mention security.","key_machinery":"The central object is the .cursorrules file: a free-form, metadata-free text file that Cursor injects into the model context so that its instructions persist across all sessions in a project. The argument is carried by a mixed-methods pipeline: a keyword-partitioned GitHub Code Search query that assembled 12,110 unique files, GitHub API metadata on repository activity and commit history, and an inductively built 65-code codebook applied to a random sample of 65 prompt files. The quantitative metadata grounds the claims about where the files appear and how they evolve, while the codebook's ten themes, anchored by codes such as Languages, Stacks & Usage Guidance and Security Awareness, Mechanisms & Practices, grounds the claims about content priorities.","core_discovery":"The central discovery is that the free-form .cursorrules file functioned as a first-wave prompt-file format whose real-world footprint is concentrated outside professional development. Quantitatively, 68.5% of hosting repositories have a single contributor, 62.5% have at most fifty commits, and 75.0% are under 11 MB; 67.3% of the files were never changed after creation, and when they are changed, over half of the modifications touch fewer than ten lines. Qualitatively, the largest theme is Code Quality and Engineering Practices (30.4% of coded instances), followed by Project Structure and Configuration (18.3%) and Maintainability and Evolution (14.4%), while Security accounts for only about 4.4% when access-control content is included. The authors read this as evidence that prompt files in the wild are mostly informal, style-oriented guidance, and that security is not yet a systematic part of how developers configure AI coding assistants.","pith_inferences":["As an inference beyond the paper's claims, if most .cursorrules files are boilerplate or template-like guidance, the content priorities may reflect what prompt templates recommend rather than what individual developers need; clustering near-duplicate files would test this directly.","Because the GitHub Code Search pipeline is keyword-partitioned and API-capped, it may undercount files in large repositories; a complete archive census could overturn the toy-project characterization if those files prove numerous.","The low security content suggests a concrete experiment: generating the same feature with and without a security-focused prompt file and comparing vulnerability density would show whether prompt files can meaningfully add security assurance.","The same mixed-methods design applied to AGENTS.md, CLAUDE.md, or copilot-instructions.md would reveal whether the toy-project concentration and security neglect persist as prompt-file formats mature."],"forward_implications":["Prompt-file adoption studies that sample only mature, popular open-source projects will systematically miss the bulk of the phenomenon, because most .cursorrules files sit in small single-maintainer repositories.","Project-level prompt files in the wild are unlikely to act as security guardrails for AI-generated code, since security-related guidance accounts for only about 4.4% of coded content.","The topical continuity between .cursorrules and .mdc files means the qualitative findings likely transfer to current Cursor rule formats and to related agent rule files such as AGENTS.md.","The format's rapid rise and decline after Cursor deprecated it in February 2025 shows that prompt-file ecosystems track tooling decisions closely.","Repository setup is a key adoption moment: 40.7% of .cursorrules files appear within 24 hours of repository creation, often within minutes."],"supporting_citations":[{"why":"Documents the .cursorrules format and its February 2025 deprecation, defining the artifact under study.","marker":"Cursor AI, 2025"},{"why":"Supplies the thematic-analysis procedure used to build the codebook from sampled prompt files.","marker":"Braun and Clarke, 2006"},{"why":"Closest prior analysis of project-level prompt configuration files in open source, providing the comparison the paper extends to Cursor.","marker":"Mohsenimofidi et al., 2025"},{"why":"Studies AGENTS.md and related context files and reports that security rarely receives attention, a result the paper corroborates.","marker":"Chatlatanagulchai et al., 2025"},{"why":"Provides the inductive open-coding strategy behind the 65-code codebook.","marker":"Saldaña, 2021"},{"why":"Releases the collected dataset and full codebook, making the quantitative and qualitative analyses reproducible.","marker":"Anonymous, 2026"}],"fun_headline_variants":["Cursor rules live in hobby repos, rarely mention security","Study: .cursorrules files cluster in small, low-activity repos","Prompt files: mostly hobby projects, security gets little attention","Cursor rules: used in toy repos, rarely updated, security sparse","First-wave prompt files found in small repos, missing security focus"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that keyword-partitioned GitHub Code Search gives a representative sample of all public .cursorrules files; if files in large, active, or security-conscious repositories are systematically missed, the toy-project and low-security conclusions would be biased.","fun_headline_variants_meta":{"raw":{"variants":["Cursor rules live in hobby repos, rarely mention security","Study: .cursorrules files cluster in small, low-activity repos","Prompt files: mostly hobby projects, security gets little attention","Cursor rules: used in toy repos, rarely updated, security sparse","First-wave prompt files found in small repos, missing security focus"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000261,"raw_usage":{"total_tokens":1610,"prompt_tokens":982,"completion_tokens":628,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":542}},"tokens_in":598,"tokens_out":628,"duration_ms":6022,"temperature":1.0,"reasoning_tokens":542,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:39:35.754818+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute a complete census of public .cursorrules files from a full GitHub archive rather than keyword-partitioned search queries; if most files turn out to live in large, multi-contributor, actively committed repositories, the paper's toy-project claim collapses. The security claim could be tested separately by measuring whether code generated with security-themed prompt files has a lower vulnerability density than code generated without them.","supporting_citations":[],"review_version":1}