{"id":"59ad715a-f2ac-4508-8546-c22024b733a2","arxiv_id":"2501.12521","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":7,"one_line_summary":"An automated linting and repair tool finds 3.46% of developer prompts biased, 10.75% injection-vulnerable, and improves a fraction of suboptimal prompts.","lead":"This paper introduces PromptDoctor, a tool that scans prompts embedded in open-source software for bias, prompt-injection risk, and performance issues, then rewrites them. It reports that 3.46% of sampled developer prompts show bias, 10.75% are vulnerable to injection, and that PromptDoctor repairs a portion of each.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Aggregate bias-prevalence and de-biasing claims rely on race/sexuality detectors with F1=0.46 and 0.13 versus a 0.59 baseline; the headline percentages are unsupported until detector precision at the actual scanning threshold is validated.","rationale":"The reader's named weakest assumption is that LLM-generated prompt-hole values are representative of real runtime values; that is a genuine validity threat, but it is not the most load-bearing one. The more immediate problem is that the bias measurement instrument itself is unreliable for two of the three bias categories: the race detector is below the benchmark baseline and the sexuality detector is near chance. Since the aggregate 3.46% bias prevalence is a central quantitative claim, and since the 68.29% de-biasing rate is evaluated by the same flawed detectors, the paper's headline results cannot be relied upon until detector precision at the actual scanning operating point is established. This overlaps with concerns the reader raised in their rationale (self-referential evaluation and underperforming detectors) but not with their stated weakest assumption, so my agreement is partial. I credit the paper for a strong gender-bias detector, a replication package, and gold-dataset checks for grounded optimization tasks, which is why the paper remains conditionally acceptable rather than rejected. The verdict should stay CONDITIONAL: the tool and method are plausible, but the quantitative prevalence and repair claims for bias need validation or recalibration before they are presented as established empirical findings.","tokens_in":23619,"tokens_out":3351,"duration_ms":38002,"concrete_test":"Run a human-annotation study on a stratified random sample of about 200 prompts from the 2,173-prompt analysis set, oversampling prompts flagged as biased. Two independent annotators label each prompt with the paper's bias taxonomy. Compute precision and recall for each of the three detectors at the exact operating threshold used in §4.1.2, then re-estimate the aggregate 3.46% bias prevalence using only categories whose detectors show precision significantly above chance, or using human labels directly. If the re-estimated aggregate or the 68.29% repair rate moves materially outside the reported values, the headline quantitative claims require revision before the paper is accepted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline finding that 3.46% of Dev Prompts are prone to generating biased responses aggregates three bias detectors described in §3.2.1 and benchmarked in §4.1.1. The gender detector is credible (F1=0.93), but the race and sexuality detectors score F1=0.46 and F1=0.13, respectively, against the same RoBERTa baseline's 0.59. An F1 of 0.13 is near chance; using that detector to report a 0.09% sexuality-bias prevalence, and then to validate that 50% of such prompts were 'de-biased', is not an empirically grounded measurement. The problem is compounded because remediation success in §3.2.2 is judged by the same detector that flags the issue: the loop terminates when the detector says the rewrite is clean, so any systematic detector error propagates directly into the 68.29% repair rate. The paper's own statement that these results 'give credence to the accuracy of these prompts' contradicts the reported numbers. No confidence intervals, precision-recall operating points, or per-category error calibration are provided for the 3.46% aggregate or the 68.29% repair rate. If the race and sexuality detectors cannot reliably separate biased from unbiased prompts at the threshold used in §4.1.2, the aggregate bias-prevalence claim and the corresponding repair claim are not supported by the reported evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents PromptDoctor, an LLM-based tool for detecting and repairing three classes of problems in Developer Prompts (prompts embedded in source code): bias, injection vulnerability, and sub-optimal performance. The tool uses LLM-based detectors and generation-evaluation loops to rewrite prompts. The authors evaluate on a stratified sample of 2,173 Dev Prompts from PromptSet, reporting 3.46% bias-proneness, 10.75% injection vulnerability, and (in the introduction) 36% QA under-performance, with repair rates of 68.29% (de-biasing), 41.81% (hardening), and 37.1% (QA optimization using GPT-4o). They also provide a VSCode extension and a replication package.","tokens_in":23926,"tokens_out":11679,"duration_ms":95054,"significance":"If the empirical findings hold, this is a substantial contribution to an emerging area of software engineering: it would be the first large-scale study of these issues in embedded prompts and would demonstrate that a fine-tuning-free rewriting approach can mitigate a meaningful fraction of them. The paper's strengths include the focus on a novel artifact, the scale of the dataset, and the release of code and data. However, the central quantitative claims are currently supported only in part: the bias prevalence and repair numbers rest on detectors whose performance is near chance for two of three categories, and the vulnerability results rely on an undisclosed attack suite without validation. The QA optimization results lack external benchmarking. These gaps are load-bearing and must be addressed before the numbers can be accepted.","major_comments":[{"comment":"The aggregate bias prevalence (3.46%) and de-biasing success (68.29%) are not supported by the detector validation reported in §4.1.1. The gender detector is credible (F1=0.93), but the race and sexuality detectors achieve F1=0.46 and 0.13, respectively, both below the 0.59 fine-tuned baseline. Using these near-chance detectors to report the 0.75% race-bias and 0.09% sexuality-bias prevalence, and to accept rewrites as 'de-biased' when the same detector passes them (§3.2.2), propagates detector error directly into the headline repair rate. Moreover, the category percentages in §4.1.2 do not sum to the reported 3.46% aggregate (3.03% gender + 0.75% race + 0.09% sexuality = 3.87%, or more if sexuality-proneness is counted), so the aggregate needs to be reconciled. The authors should either restrict the prevalence and repair claims to the gender category, or provide precision-recall operating points at the scanning threshold for the weak detectors and recalibrate the aggregate accordingly.","section":"§4.1.1, §4.1.2, Finding 1"},{"comment":"The vulnerability prevalence (10.75%) and hardening rate (41.81%) are measured against an undisclosed 42-attack suite, with no false-positive or false-negative analysis for the detection procedure. Because hardening success is defined as all 42 attacks failing on a rewrite, the reported rates are entirely conditional on the composition of this suite. Without releasing the suite, showing its coverage of known attack families, and validating the detection against a labeled set of vulnerable and benign prompts, the headline vulnerability numbers are not empirically grounded. The footnote that 'Double blind policy forbids from being specific' does not satisfy the need for a reproducible attack corpus.","section":"§3.3.1, §4.2, Finding 2"},{"comment":"The patching process that fills prompt holes with LLM-generated mock values is not validated against real user inputs. Since the bias and vulnerability analyses are executed exclusively on patched prompts (Sections 3.2 and 3.3), the prevalence estimates (3.46% and 10.75%) are conditional on a synthetic value distribution. If real interpolated values differ in length, topic, or adversarial intent, both the prevalence and the repair rates could change materially. The authors should either validate patching against a sample of real usage data or explicitly rephrase the claims as 'with our generated values' and add this as a limitation with a concrete sensitivity analysis.","section":"§3.1.3"},{"comment":"The introduction claims that '36% of Question-Answering Dev Prompts quantitatively under-performed when tested against real-world benchmarks,' but RQ3's evaluation (§4.3.1) uses synthetic test data and an LLM-as-judge scorer derived from the original prompt, with no real-world benchmark for QA. No definition or measurement of the '36%' figure is given anywhere in the paper, and the 37.1% improvement rate for GPT-4o is also based solely on the synthetic set. The paper should either remove the unsupported 36% claim, or add an external QA benchmark evaluation, and should discuss the risk of self-confirmation in using the same synthetic distribution for training and evaluation.","section":"Introduction, §4.3.1, Figure 16a"}],"minor_comments":[{"comment":"The sentence about sexuality bias says responses are 'likely to generate Race-Biased responses'; this appears to be a typo for Sexuality-Biased responses, and the sentence should be repaired.","section":"§4.1.2"},{"comment":"There are several grammatical slips, such as 'added at interpolated before being sent' (§3.1.1) and 'values shaped by by users' (§3.3.1); a proofreading pass is needed.","section":"§3.1.1, §3.3.1"},{"comment":"Labels such as 'Figure 1212.a' and 'Figure 1616.a' should be corrected to 'Figure 12a' and 'Figure 16a' respectively.","section":"§4.2.1, §4.3.1"},{"comment":"The model name appears inconsistently as 'LLama3.1' and 'Llama3.1'; please standardize the spelling.","section":"§4.3.1"},{"comment":"Section 7 states that human validation was performed, but no details or results are reported; either add a description of the human validation protocol and findings, or remove the claim.","section":"§7"}],"recommendation":"major_revision","confidential_remarks":"The paper's topic is well within the journal's scope, and the artifact (code, data, extension) is a strength. However, the 'empirically-grounded' label is premature: the bias detectors for race and sexuality are near chance, the attack suite is undisclosed, and the QA optimization is not externally validated. The editor may wish to require a revised version that either narrows the claims to the well-validated parts or adds the missing validation. The note in §3.3.1 that double-blind policy forbids disclosing the attack suite should be resolved in a way that makes the work reproducible, e.g., via an anonymized appendix or a data-sharing agreement."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the bias prevalence and repair claims are not yet supported by the reported evidence. The race and sexuality bias detectors score F1=0.46 and 0.13 against a 0.59 baseline, and the paper says that \"gives credence to the accuracy of these prompts.\" That is a plain internal contradiction. An F1 of 0.13 is near chance, and both prevalence and de-biasing success for those categories are built on it.\n\nWhat is genuinely new: Dev Prompts—prompts embedded in code with runtime-filled holes—are a real and underexamined artifact. This is the first large-scale measurement of bias, injection vulnerability, and sub-optimality in them, and the paper ships a tool, a VSCode extension, and a replication package. The gender-bias detector is carefully benchmarked (F1=0.93 vs 0.81), and the grounded-task optimization (translation, summarization, error correction) uses standard metrics and external gold datasets, which is credible.\n\nThe soft spots are concentrated in the headline numbers. The aggregate 3.46% bias prevalence mixes the credible gender detector with the two near-chance ones. The remediation loop terminates when the same detector says the rewrite is clean, so its errors propagate straight into the 68.29% repair rate. The injection results rest on an undisclosed 42-attack set with no false-positive/false-negative validation; the 41.81% hardening rate is relative to that unvalidated set. The 36% sub-optimality figure appears in the abstract and intro, but I could not find its derivation in the experimental section. Also, all patched hole values are LLM-generated, one value per hole, with no comparison against real user-supplied inputs, which could move prevalence either way. No confidence intervals are reported anywhere.\n\nThis is not a desk-reject. The Dev Prompt framing is useful, the dataset and tool are real, and the gender-bias plus grounded-task parts hold up. But \"empirically-grounded\" is too strong for the bias and injection claims as they stand. A serious referee should demand re-analysis using only validated detectors, disclosure of the attack set, and a derivation of the 36% figure. If those are fixed, this is a solid ICSE/FSE-style contribution. As is, I would not rely on the headline percentages.","headline":"The paper's bias and de-biasing headline numbers are not supported by its own detector F1 scores (race 0.46, sexuality 0.13 vs baseline 0.59), but the Dev Prompt artifact, dataset, and gender-bias results are real contributions worth refereeing.","tokens_in":24479,"tokens_out":4614,"would_cite":true,"duration_ms":44448,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Developer Prompts embedded in open-source code carry measurable bias, injection, and sub-optimality risks, and an LLM-based rewrite pipeline fixes a substantial share of each without fine-tuning.","keywords":["developer prompts","prompt linting","prompt repair","bias detection","prompt injection","prompt optimization","large language models","software engineering"],"falsifier":"Repatch the paper's 2,173 Dev Prompts using real runtime values captured from instrumented deployments instead of LLM-generated mock values, then rerun the identical bias, injection, and optimization pipelines; if the prevalence figures (3.46%, 10.75%, 36%) or the repair rates (68.29%, 41.81%, 37.1%) move materially, the empirical claims are an artifact of the patching step.","tokens_in":23397,"feed_emoji":"🛡️","tokens_out":9580,"duration_ms":81696,"temperature":0.7,"pith_summary":"PromptDoctor is a tool for inspecting and repairing Developer Prompts—the natural-language prompts embedded in source code and filled with runtime values. The paper claims that on a representative sample of 2,173 prompts drawn from 40,573 open-source Dev Prompts, 3.46% are biased or bias-prone, 10.75% are vulnerable to prompt-injection attacks, and 36% of question-answering prompts under-perform against synthetic benchmarks. It further claims that an LLM-driven rewrite loop can de-bias 68.29% of biased prompts, harden 41.81% of vulnerable ones, and improve 37.1% of sub-optimal ones, without any model fine-tuning. If right, the paper turns prompt quality into a checkable, repairable software property rather than an art form, and shows these failure modes are common enough in real code to warrant dedicated tooling.","feed_headline":"One in ten developer prompts is open to injection attacks","feed_subtitle":"A 2,173-prompt audit of open-source code finds real bias and injection risks, and LLM rewrites fix a large share.","key_machinery":"The central mechanism is the generate-and-evaluate loop applied to a canonicalized Developer Prompt. Each prompt is first canonicalized so every runtime interpolation becomes a marked hole of the form '{placeholder}', and each hole is then 'patched' with an LLM-generated value, produced sequentially so later values are consistent with earlier ones. Bias handling uses a hand-crafted multi-shot LLM prompt that classifies a patched prompt as explicitly biased, bias-prone, or clean, and remediation rewrites the prompt at most ten times, keeping only rewrites the classifier passes. Injection handling injects each of 42 known attacks into each hole and checks whether the model emits the attack's expected target string; hardening rewrites the prompt until no attack succeeds. For optimization, the patching process is extended into a synthetic dataset, seed prompts are drawn from 26 prompt-engineering principles, and a self-improving search keeps the highest-scoring $n$ candidate prompts at each step, scoring them with BLEU for translation, GLEU for grammar correction, embedding cosine similarity for summarization, and an LLM-as-judge for question answering.","core_discovery":"Developer Prompts are a distinct software artifact: natural-language instructions with programmatic holes whose values are interpolated at runtime. The paper establishes empirically that these artifacts fail in measurable, tool-detectable ways. Analyzing a stratified random sample of 2,173 Dev Prompts from the cleaned 40,573-prompt corpus, it reports that 3.46% are explicitly biased or bias-prone along gender, race, or sexuality dimensions; that 10.75% contain at least one prompt hole that yields to at least one of 42 tested injection attacks; and that 36% of question-answering prompts under-perform against synthetic test data. It then shows that a generation-evaluation loop, capped at ten iterations, repairs a substantial fraction of these flaws: 68.29% of biased prompts are rewritten to pass the bias checks, 41.81% of vulnerable prompts are hardened so all 42 attacks fail, and 37.1% of sub-optimal QA prompts improve on held-out data, with translation, summarization, and grammar-correction prompts improving consistently.","pith_inferences":["The reported numbers depend on the patch values that fill prompt holes; re-running the same pipelines with real runtime values captured from deployed software would test whether the 3.46% and 10.75% figures hold in production.","The same generate-and-evaluate machinery could be pointed at other prompt faults the paper does not measure, such as leaking the prompt's hidden instructions or drifting off-task on long inputs.","Because an LLM judge both detects bias and scores question-answering rewrites, the repair rates could partly reflect the judge's own blind spots; an adversarial audit of the judge against known biased outputs would clarify what the rates mean.","Adoption of the VS Code extension is untested; a user study of whether developers accept and keep the rewrites would show whether the repair rates translate into shipped behavior."],"forward_implications":["Prompt quality becomes a lintable software property: developers can catch bias, injection, and sub-optimality in the IDE before the prompt ships.","Teams using API-only LLMs they cannot fine-tune still gain a repair path for these three failure modes.","The prevalence figures—3.46% biased, 10.75% injection-vulnerable, 36% of QA prompts sub-optimal—serve as a measuring stick for future prompt-safety tools and studies.","The per-category repair rates (82.81% for gender bias but 12.50% for race bias) show where automatic rewriting is already sufficient and where it needs stronger methods."],"supporting_citations":[{"why":"Supplies the 61,448 Dev Prompts mined from open-source projects that form the corpus behind the cleaned 40,573-prompt set and the 2,173-prompt analysis sample.","marker":"[5]"},{"why":"The LLM report for the model that carries out bias detection, injection testing, and rewrite generation in the main experiments.","marker":"[16]"},{"why":"Shows that LLMs make gender and race assumptions even from neutral prompts, motivating the paper's bias-proneness detection and remediation design.","marker":"[23]"},{"why":"Provides the early categorization of prompt-injection attacks that shapes the 42-attack test set used to measure Dev Prompt vulnerability.","marker":"[26]"},{"why":"Supplies the gender-bias benchmark against which the multi-shot bias detector scores F1 0.93 versus the benchmark's 0.81.","marker":"[67]"},{"why":"Supplies the benchmark used to validate the race- and sexuality-bias detection prompts.","marker":"[68]"},{"why":"Provides the optimization approach whose four published Dev Prompts are re-optimized as the comparison baseline.","marker":"[55]"},{"why":"Source of the 26 prompt-engineering principles from which seed candidate prompts for optimization are generated.","marker":"[61]"},{"why":"Provides the LLM-as-a-judge technique used to score question-answering outputs during prompt optimization.","marker":"[66]"}],"fun_headline_variants":["Nearly one in ten developer prompts vulnerable to injection","Audit of 2,173 developer prompts finds bias, injection holes, and suboptimal QA","New tool auto-repairs developer prompts: 68% debiased, 42% hardened","Developer prompts: new software artifact with measurable bias and injection risks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The prevalence and repair figures assume that the LLM-generated mock values used to patch prompt holes behave like the values real users supply at runtime, but the paper does not validate this against deployed traffic or a corpus of actual interpolations.","fun_headline_variants_meta":{"raw":{"variants":["Nearly one in ten developer prompts vulnerable to injection","Audit of 2,173 developer prompts finds bias, injection holes, and suboptimal QA","New tool auto-repairs developer prompts: 68% debiased, 42% hardened","Developer prompts: new software artifact with measurable bias and injection risks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001226,"raw_usage":{"total_tokens":5116,"prompt_tokens":1095,"completion_tokens":4021,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":711,"completion_tokens_details":{"reasoning_tokens":3938}},"tokens_in":711,"tokens_out":4021,"duration_ms":29454,"temperature":1.0,"reasoning_tokens":3938,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:06:18.541757+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repatch the paper's 2,173 Dev Prompts using real runtime values captured from instrumented deployments instead of LLM-generated mock values, then rerun the identical bias, injection, and optimization pipelines; if the prevalence figures (3.46%, 10.75%, 36%) or the repair rates (68.29%, 41.81%, 37.1%) move materially, the empirical claims are an artifact of the patching step.","supporting_citations":[{"cited_title":"PromptSet: A Programmer's Prompting Dataset","cited_arxiv_id":"2402.16932","evidence_quote":"Supplies the 61,448 Dev Prompts mined from open-source projects that form the corpus behind the cleaned 40,573-prompt set and the 2,173-prompt analysis sample."},{"cited_title":"Marked personas: Using natural language prompts to measure stereotypes in language models,","cited_arxiv_id":null,"evidence_quote":"Shows that LLMs make gender and race assumptions even from neutral prompts, motivating the paper's bias-proneness detection and remediation design."},{"cited_title":"“call me sexist, but","cited_arxiv_id":null,"evidence_quote":"Supplies the gender-bias benchmark against which the multi-shot bias detector scores F1 0.93 versus the benchmark's 0.81."},{"cited_title":"Xhate-999: Analyzing and detecting abusive language across domains and languages,","cited_arxiv_id":null,"evidence_quote":"Supplies the benchmark used to validate the race- and sexuality-bias detection prompts."},{"cited_title":"Prompt engineering,","cited_arxiv_id":null,"evidence_quote":"Source of the 26 prompt-engineering principles from which seed candidate prompts for optimization are generated."},{"cited_title":"Judging llm-as-a-judge with mt-bench and chatbot arena,","cited_arxiv_id":null,"evidence_quote":"Provides the LLM-as-a-judge technique used to score question-answering outputs during prompt optimization."}],"review_version":1}