{"id":"6c49f963-433d-4fd6-a981-d80962808bd2","arxiv_id":"2505.12655","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Embedding iteratively refined HTML policy text can make web-retrieval LLMs refuse to disclose page content, with measured defense success rising from 2.5% to 88.6%.","lead":"This paper proposes a defense that embeds hidden instructions in web pages to stop AI chatbots from quoting or summarizing the content. The method reportedly raises the success rate of blocking such retrieval from about 2.5% to nearly 89% across several chatbots.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DSR on fictitious single-source pages may not measure prevention of real-world disclosure; high DSR could coexist with the LLM answering from other indexed sources or parametric memory.","rationale":"The reader's conditional verdict is appropriate: the mechanism is plausible and the iterative policy construction is a real engineering contribution, but the headline number is only meaningful if the evaluation approximates the deployment scenario. I agree with the reader's weakest assumption and sharpen it: the synthetic pages are not merely fake but informationally unique, so the DSR conflates 'the model obeys the policy template' with 'the user did not receive the information.' Real content is redundant across sources, so the defense is not a global barrier. This does not make the paper unworthy of publication; it means the central claim should be treated as a proof-of-concept pending an evaluation that measures information leakage on real, multiply-sourced pages. Since the reader already assigned CONDITIONAL with essentially this gap, my read leaves the verdict unchanged. I also note the paper self-reports the fictitious-page limitation in Sec. 7 and the Gemini retrieval limitation in Table 1, which supports a cautious interpretation.","tokens_in":14073,"tokens_out":6377,"duration_ms":71795,"concrete_test":"Apply the Iteration-3 hidden-policy defense to 20 real, currently indexed web pages whose key facts also appear on other domains (e.g., major news stories covered by multiple outlets). Query a retrieval-enabled LLM with 50 held-out paraphrased questions not used in policy optimization. Separately score (a) the paper's DSR (whether the model emits the prescribed refusal/redirection text) and (b) an information-leakage score: can a blinded annotator extract the target facts from the response? If (b) is high while (a) is high, the defense blocks reproduction of one page's wording but not unauthorized retrieval of the underlying content, and the headline 2.5% to 88.6% claim does not transfer.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Eq. (1) defines the defense objective as similarity between the LLM response r and the target page's visible content φ(w), and the experiments measure DSR as \"the percentage of cases in which an LLM follows exactly the above defense goal\" (Sec. 5.2). On the ten fictitious websites (Sec. 5.1), the content exists nowhere else: it appears in no other search result and not in the model's pretraining data. Therefore, a refusal to quote the page is equivalent to withholding the information, and the reported 70–100% DSR (Tables 1–2) looks like successful prevention. On real, indexed content, the same facts are typically available from multiple pages and from the model's parametric memory. A hidden policy on one page can make the model decline to copy that page's text, but it cannot prevent the model from answering the user from other retrieved documents or from internal knowledge. Thus, the central claim that the defense prevents unauthorized real-time retrieval is not established by the current metric: DSR measures template compliance, not information disclosure. The paper's own Limitation (Sec. 7) flags fictitious pages as a possible source of discrepancy, but the more specific confound is content uniqueness/redundancy rather than realism per se. The fixed ten-query protocol also leaves open overfitting to the query templates used during iterative optimization.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a defense against real-time retrieval by LLMs: the defender embeds LLM-generated policy text inside a webpage's HTML so that a retrieval-augmented LLM declines to disclose the page's content. The authors formulate the defense as a min-max optimization (Eq. 2), approximate the inner maximization with adversarial user queries, and approximate the outer minimization by iteratively refining HTML policy cues with a proxy LLM. They evaluate three defense goals (refusal to answer, partial masking, redirection) on ten fictitious websites and two real homepages against GPT-4o, GPT-4o mini, Gemini, and ERNIE 4.5 Turbo, reporting defense success rates rising from near 0% to 70–100% for single queries and improved follow-up robustness for their Iteration-3 policy. They also compare against robots.txt and study the effects of policy position and visibility.","tokens_in":14261,"tokens_out":4201,"duration_ms":47853,"significance":"The problem is timely and the proposed direction—using the semantic understanding of the LLM itself as a provider-independent protection mechanism—is a useful complement to robots.txt and post-hoc enforcement. The paper makes a concrete practical contribution in showing that carefully worded HTML policy prompts can steer current retrieval LLMs, and the three-goal formulation (refusal, masking, redirection) plus the sensitivity analysis on policy position and visibility are valuable. The authors also commit to releasing datasets and scripts. However, the evidence as presented does not establish the paper's central claim that the defense prevents unauthorized real-time retrieval of real, indexed web content: the experiments measure instruction-following on mostly fictitious pages whose content is unique to the protected page, and the evaluation protocol is underspecified.","major_comments":[{"comment":"The DSR metric conflates instruction-following with actual prevention of information disclosure. On the ten fictitious websites, the content appears nowhere else on the web and is absent from pretraining data, so a refusal to quote the page is equivalent to withholding the information. On real, indexed content, the same facts are typically available from multiple pages and from the model's parametric memory; a hidden policy on one page cannot prevent the LLM from answering from other retrieved documents or internal knowledge. The paper should evaluate on real content with redundant coverage and measure information leakage (e.g., whether the answer contains facts present only on the protected page) rather than only whether the prescribed refusal string or template was followed.","section":"Sec. 5.1–5.2"},{"comment":"The min-max problem in Eq. (2) is not actually solved. The practical method is a heuristic loop: generate adversarial queries, collect attacker responses, and ask a proxy LLM to refine the injected HTML policy. There is no convergence guarantee, no coverage guarantee over the space of user queries, and no justification that the proxy LLM's iterative feedback approximates the outer minimization of Eq. (2). Because the queries used during optimization are the same fixed templates used in evaluation, the reported success may reflect overfitting to those templates. The paper should either provide a principled analysis of the optimization or explicitly reframe the contribution as a heuristic prompt-optimization method, and it should test robustness on unseen query phrasings and adversarial paraphrases.","section":"Sec. 3.2–3.3"},{"comment":"The headline improvement \"2.5% to 88.6%\" cannot be traced to the reported tables. Averaging the Baseline cells in Table 1 does not yield 2.5%, and the 88.6% value is not labeled as a specific condition or model. Moreover, Gemini is evaluated only on two real homepages, with all fictitious-site and Heroku cells omitted, yet the abstract claims the method works \"on different LLMs.\" The DSR evaluation protocol is also unspecified: no details are given on whether DSR was scored automatically or manually, whether scoring was blinded, how partial credit was handled, or whether multiple trials were run. The paper should report per-condition tables with confidence intervals or significance tests and a complete, reproducible evaluation protocol for DSR and FDSR.","section":"Sec. 5.1, Table 1, Abstract"},{"comment":"The generalization claim is weakened by unreported failures in the appendix. In Table 4, ERNIE 4.5 Turbo under the Iteration-3 policy achieves 0% FDSR for both Partial Protect and Redirection on GitHub, yet the appendix text states that Iteration 3 \"led to near-perfect FDSRs in almost all settings.\" This inconsistency should be explained, and the claim of consistent robustness across models and goals should be tempered accordingly.","section":"Table 4 / Figure 4"}],"minor_comments":[{"comment":"The model labels \"GPT-4∗\" and \"GPT-o∗\" in Table 3 are undefined; the exact model versions tested should be specified in the caption or text.","section":"Table 3"},{"comment":"The y-axis in Figure 4 extends to 150%, which is misleading for percentage bars that cannot exceed 100%; the axis should be capped at 100%.","section":"Figure 4"},{"comment":"The statement that \"Gemini and ERNIE show limitations in following more fine-grained instructions\" is only partially supported because Table 2 contains '-' entries for Gemini on Partial Masking and Redirection; the text should discuss only the cells that were actually measured.","section":"Sec. 5.3"},{"comment":"The appendix shows visible and invisible HTML prompt examples, but it never specifies exactly which prompts were used in which table, and it does not clarify whether invisible prompts (display:none) are still present in the raw HTML fetched by the LLM or whether they affect search-engine indexing; this should be stated explicitly.","section":"Appendix A.2"},{"comment":"There are several presentation issues: \"reinforcedinequality\" in Sec. 1, inconsistent hyphenation such as \"Heroku-hosted webpages,\" and reference formatting inconsistencies (e.g., incomplete author lists and missing access dates). These should be cleaned up.","section":"Sec. 1, Sec. 5.3, References"},{"comment":"The table note says Gemini is omitted because it does not provide sufficient evidence under synthetic settings, but the caption could state this more prominently so that readers do not infer Gemini was tested in all multi-round conditions.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The core idea is plausible and the prompt-optimization heuristics are interesting, but the experimental evaluation as it stands does not substantiate the central claim of preventing retrieval on real indexed content. The DSR metric's confound with content uniqueness, the unexplained discrepancy in Table 4, and the unspecified evaluation protocol are all fixable but require substantial additional experiments rather than copy-editing. I recommend major revision and would want to see a real-content evaluation with independent query sets and an information-leakage metric before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper to know about: it turns prompt injection into a defense for web IP protection. A proxy LLM iteratively refines hidden HTML policy text that instructs a retrieval-enabled LLM to refuse, mask, or redirect rather than reproduce page content. That defensive framing is genuinely new relative to the injection literature, and the iterative loop is a reasonable engineering trick. The paper also shares detailed prompt examples and shows consistent gains across GPT-4o, GPT-4o mini, and ERNIE, with a useful sensitivity analysis on policy placement and visibility.\n\nThe soft spots are mostly empirical. The headline 2.5% to 88.6% DSR comes from ten fictitious websites plus two real homepages. The fictitious content exists nowhere else, so a refusal to quote the page does withhold the information. On real, indexed content, the same facts typically appear on many pages and in the model's parametric memory. A hidden policy on one page can stop the model from copying that page, but it cannot stop the model from answering from other retrieved documents or internal knowledge. So DSR as defined—the percentage of cases where the LLM follows the defense goal—measures template compliance, not prevention of information disclosure. That is a real confound, and the paper's own limitation section flags the fictitious-page issue but not this redundancy problem specifically.\n\nAlso, the min-max formulation in Eq. (2) is not actually solved; the iteration is a heuristic. That is fine if framed as engineering, but the abstract's claim of effectively addressing an intractable black-box optimization problem overstates it. No code or data are included yet, the DSR evaluation protocol is underspecified (who judges \"follows exactly\"?), and Gemini results only cover two real pages with some cells missing, which the paper does disclose. These are fixable with a clear protocol and tests on real pages where the answer is available elsewhere.\n\nIf I read this for my own work, I would take the threat model and policy design as the contribution, not the quantitative claim. The paper is worth a serious referee: the idea is timely, the related work is honest, and the limitations are stated. I would send it to review with a request for code, a sharper evaluation metric that checks whether the sensitive information is actually absent from the response, and experiments on pages whose content also appears elsewhere.","headline":"A clever defensive reframing of prompt injection with real engineering value, but the headline DSR overclaims because it is measured on fictitious unique-content pages and does not show prevention of disclosure on real indexed content.","tokens_in":14844,"tokens_out":1588,"would_cite":false,"duration_ms":16949,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A defense that embeds iteratively optimized policy cues in HTML can prevent retrieval-enabled language models from disclosing web content, raising defense success from 2.5% to 88.6% across tested models.","keywords":["real-time LLM retrieval","web intellectual property protection","adversarial HTML policy","black-box min-max optimization","defense success rate","robots.txt alternatives","retrieval-augmented generation","prompt-injection defense"],"falsifier":"Run the same optimized policies on a sample of real, indexed web pages under fresh, human-written queries that were not used during optimization, and measure how often the LLM discloses substantive content anyway. If the defense success rate drops to the level of a plain privacy notice or robots.txt on those pages, the reported 2.5%-to-88.6% improvement is an artifact of the synthetic pages and query templates rather than a general property of the embedded HTML policy.","tokens_in":13817,"feed_emoji":"🛡️","tokens_out":6793,"duration_ms":68387,"temperature":0.7,"pith_summary":"This paper argues that content creators can stop retrieval-enabled large language models from summarizing or copying their webpages by embedding specially optimized policy text directly in the page's HTML. The policy text is invisible to human readers but is read by the LLM when it fetches the page, and it instructs the model to refuse, to reveal only a permitted subset, or to redirect the user to the owner's chosen URL. The authors frame the task as a min-max game between an adversarial user and the defender, and solve it by iteratively querying the target LLM and refining the embedded policy with a proxy language model. Across tested models this raises the defense success rate from roughly 2.5% to 88.6%, and it survives follow-up attempts to bypass the policy better than a manually written notice or robots.txt. If true, this gives individual publishers a proactive, provider-independent way to protect web content from AI extraction.","feed_headline":"Hidden HTML policy text lifts LLM web-defense success to 88.6%","feed_subtitle":"Publishers can stop retrieval-enabled chatbots from redistributing page content, without breaking search indexing.","key_machinery":"The central object is a hidden HTML policy string z, produced by the iteration loop, that is appended to the page source w to give w⊕z while preserving the visible rendering, that is, φ(w)=φ(w⊕z). It works because LLMs read raw HTML and weigh early tokens more heavily; policies are placed at the top of the file and phrased as direct instructions with strict, repeated constraints. The optimization machinery is the min-max objective in Eq. (2): an inner maximization finds an adversarial user query q that would extract content, and an outer minimization finds a policy z that defeats that query, solved in practice by black-box feedback between the attacker LLM and the proxy policy generator.","core_discovery":"The central claim is that the gap between the HTML source of a page and its rendered appearance—many different HTML documents can render identically—can be used as a channel for a semantic defense. The defender appends a hidden policy z to raw HTML so that the rendered page is unchanged, while retrieval-enabled LLMs, which consume the raw HTML, are steered by the policy. The paper's disclosed mechanism is iterative: simulate the worst-case user query, ask the retrieval LLM, evaluate the response against the defense objective, and feed the failure back to a proxy LLM that generates a stronger policy. Two strategic patterns emerge from this process: instruction-guided response templates (the policy tells the LLM exactly what it may and may not say) and proactive bypass prevention (repeated strict constraints that anticipate attempts to ignore the policy). The paper reports that this turns near-zero baseline compliance into 70 to 100 percent defense success on refusal, masking, and redirection goals, and that the iterated policy beats robots.txt on both regular and reasoning models.","pith_inferences":["Because the defense steers the LLM through instruction-following rather than any API-specific hook, the same policy text could plausibly influence AI agents and voice assistants that fetch the page, not just chat LLMs.","The method is dual-use: the same min-max loop that produces protective policies could produce deceptive web content that silently misleads future retrieval models, so publishers and platform designers may need ways to authenticate page-owner policies.","A publisher could automate the loop, regenerating the policy whenever content changes, since the policy generator is an offline LLM and the only requirement is HTML controlled by the publisher."],"forward_implications":["Websites can publish content normally and still block LLM redistribution: search engines and human readers see an unaltered page.","Publishers get three dials of control: full refusal, release only an allowed summary, or steer users to a specified URL.","The defense does not depend on AI companies choosing to honor robots.txt, so it remains effective even against providers that ignore crawler directives.","With the bypass-prevention iteration, the defense keeps most tested models from disclosing content even when users explicitly tell the LLM to ignore site policies.","Because the raw HTML is modified rather than the visible page, the method can be layered on top of existing legal or configuration-based measures."],"supporting_citations":[{"why":"Defines the robots.txt protocol, the configuration-based baseline that the proposed semantic defense is compared against.","marker":"Koster (1996)"},{"why":"Sets up retrieval-augmented generation, the architecture that makes real-time web retrieval by LLMs possible.","marker":"Lewis et al. (2020)"},{"why":"Establishes indirect prompt injection through retrieved content, the vulnerability mechanism the defense exploits.","marker":"Greshake et al. (2023)"},{"why":"Shows that invisible HTML elements can steer LLM and agent behavior, the closest prior technique that this defense extends.","marker":"Liao et al. (2025)"},{"why":"Explains position bias in LLMs, used to account for why top-of-page policy placement achieves the highest defense success.","marker":"Wang et al. (2025)"},{"why":"Documents that AI providers do not consistently honor robots.txt, motivating a defense that does not rely on crawler compliance.","marker":"Community (2024b)"}],"fun_headline_variants":["Hidden HTML text steers LLMs away from web content, 88.6% success rate","Web publishers can hide policy text to block LLM retrieval, 88.6% effective","LLM web scraping blocked 88.6% of the time via hidden HTML policy text","LLM defense success jumps from 2.5% to 88.6% via hidden HTML text","Hidden HTML policy beats robots.txt for blocking LLM web reuse, 88.6% success"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the way retrieval-enabled LLMs followed embedded policies on ten fictitious pages, when asked a handful of fixed query templates, is representative of how they will behave on real indexed sites with the full variety of human questions.","fun_headline_variants_meta":{"raw":{"variants":["Hidden HTML text steers LLMs away from web content, 88.6% success rate","Web publishers can hide policy text to block LLM retrieval, 88.6% effective","LLM web scraping blocked 88.6% of the time via hidden HTML policy text","LLM defense success jumps from 2.5% to 88.6% via hidden HTML text","Hidden HTML policy beats robots.txt for blocking LLM web reuse, 88.6% success"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001536,"raw_usage":{"total_tokens":6134,"prompt_tokens":921,"completion_tokens":5213,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":5092}},"tokens_in":537,"tokens_out":5213,"duration_ms":37362,"temperature":1.0,"reasoning_tokens":5092,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:29:05.734914+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same optimized policies on a sample of real, indexed web pages under fresh, human-written queries that were not used during optimization, and measure how often the LLM discloses substantive content anyway. If the defense success rate drops to the level of a plain privacy notice or robots.txt on those pages, the reported 2.5%-to-88.6% improvement is an artifact of the synthetic pages and query templates rather than a general property of the embedded HTML policy.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the robots.txt protocol, the configuration-based baseline that the proposed semantic defense is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Explains position bias in LLMs, used to account for why top-of-page policy placement achieves the highest defense success."}],"review_version":1}