{"id":"1aab22aa-5075-46e6-b31b-ec8540124eba","arxiv_id":"2508.19980","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"PasswordEval shows frontier models frequently leak passwords or confidential information, jailbreaks worsen failures, and reasoning traces leak secrets even when final answers do not.","lead":"This paper introduces PasswordEval, a benchmark that tests whether language models reveal confidential information only when the correct password is supplied. Across many open and frontier models, most fail at this task, and their internal reasoning traces often contain the very secrets they are supposed to protect.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Token-cap truncation may confound the reasoning comparisons and the trace-leak result; rerun with a larger max_tokens cap to check.","rationale":"The reader's weakest assumption was external validity: the password-in-prompt, exact-string setup may not represent real confidential-information handling. I agree that is a limitation, but the paper explicitly frames the benchmark as a forward-looking native authentication setting and recommends external verification in the Discussion, so it does not by itself threaten the paper's internal validity. The token-cap issue is more load-bearing because it affects the measurements behind the reasoning comparison and the trace-leak finding, not just the scope of the conclusions. The released code and dataset and the simple, verifiable string-matching protocol are real strengths, and the non-reasoning frontier-model results, such as GPT-4o's 83.4% compliant accuracy, are less exposed to this artifact. However, the paper does not report how often reasoning responses hit the 400-token cap, and all reasoning-controlled comparisons in Figures 2 and 4 depend on that cap. A fixed-cap artifact would make reasoning look unhelpful and would make trace-only leakage look more severe than it is. This warrants the same CONDITIONAL verdict the reader reached, with an added concrete reproducibility check. I do not see a reason to reject the paper or to accept it without revision.","tokens_in":16136,"tokens_out":8413,"duration_ms":82720,"concrete_test":"Rerun the reasoning-enabled configurations (Qwen-3 4B/8B/14B with thinking enabled, Gemini-2.5-Flash, and o4-mini if the API permits) on the single-turn and multi-turn PasswordEval sets with max_tokens=400 versus max_tokens=2000, or with a stop condition that allows the model to finish its final answer. Record finish_reason and response lengths in both settings. Compare Compliant Correctness, Non-Compliant Correctness, and trace-versus-output leak rates. If the 400-token runs show material truncation (finish_reason='length') and lower compliant accuracy, and the 2000-token runs raise compliant accuracy or shrink the output/trace leak gap, the reasoning and trace-leak conclusions must be revised. If the metrics are unchanged, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing issue is internal to the experimental protocol. In Section 4 (Experimental Setting), the authors cap generations at 100 new tokens, or 400 new tokens for reasoning models. For reasoning-enabled models, this cap is counted across the entire generation, including the chain-of-thought. If a model produces more than 400 tokens before its final answer, which is common for Qwen-3 and Gemini thinking modes, the generation stops before a complete final answer is emitted. The paper then computes output metrics on the truncated text: Compliant Correctness is scored as a failure because the confidential string is absent, Non-Compliant Correctness can be scored as a success because neither the password nor the confidential string appears in the empty suffix, and Figure 4's 'output only' leak rates become artificially low while the trace still contains the secret. This directly threatens the headline claims that reasoning does not generally improve rule-following (Figure 2) and that reasoning traces leak even when outputs do not (Figure 4). The manuscript does not report finish_reason, token counts, or the fraction of truncated responses. Because the same cap applies to o4-mini and o3, their reported Compliant Correctness in Table 1 may also be understated. This is not merely a scope caveat; it is a possible artifact in the measurements supporting two central conclusions.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PasswordEval, a benchmark in which a model is given a password and a confidential string in its system prompt and must reveal that string only when the user supplies the correct password. The authors evaluate a range of open and closed models under direct requests, template-based jailbreaks, GCG and PAIR adversarial attacks, and multi-turn multi-password variants. They report that frontier models often fail on compliant requests, that template jailbreaks degrade non-compliant correctness, that enabling reasoning does not consistently improve rule-following, and that reasoning traces leak the password or confidential string even when final outputs do not.","tokens_in":16366,"tokens_out":3456,"duration_ms":33664,"significance":"If confirmed, the results are practically important: PasswordEval offers a simple, verifiable access-control stress test, the code and data are publicly released, and the finding that reasoning traces leak secrets is directly relevant to deployment decisions about exposing chain-of-thought. The core measurements are direct and the evaluation criteria are transparent. However, the central comparative claims currently rest on measurements that may be affected by generation truncation and that lack reported statistical precision, so the headline conclusions need additional support before the paper is accepted.","major_comments":[{"comment":"The 100/400-token generation cap is a potential confound for every reasoning-model result. For reasoning models the cap is counted over the chain-of-thought plus the final answer; Qwen-3 and Gemini-2.5 thinking modes frequently exceed 400 tokens, so a truncated generation can be scored as a Non-Compliant success (neither string present), a Compliant failure (confidential string absent), or an artificially low 'Output Only' leak in Figure 4. The manuscript never reports finish_reason, token counts, or the fraction of truncated responses. Please rerun with a substantially larger cap or filter to complete responses and report the truncation rate; otherwise the claims in Section 4.2 that reasoning does not improve rule-following and in Section 4.4 that traces leak when outputs do not are not established.","section":"Section 4, Experimental Setting; Tables 1-2; Figures 2 and 4"},{"comment":"No confidence intervals, standard errors, or per-condition sample sizes are reported. With 500 scenarios, a difference of a few percentage points (e.g., 82.4% vs 83.0% for o4-mini vs GPT-4o-mini in Table 1) is within binomial sampling noise, so the conclusion that reasoning does not improve performance is not supported at the displayed precision. Please report N per condition and interval estimates, or raw counts, for each metric and each model.","section":"Tables 1-2 and Figures 2-3"},{"comment":"The evaluation operationalizes leakage as exact string containment for the generated password and confidential string, and the 500 scenarios are generated by GPT-4o without any reported human validation of correctness or diversity. This makes all leak rates lower bounds, since paraphrased leaks are missed, and makes external validity depend on an unstated assumption that the generated scenarios are representative and correctly formatted. At minimum, the paper should report validation statistics (e.g., the fraction of generated examples that were rejected or manually corrected) and discuss how paraphrased leakage would affect the headline conclusions.","section":"Section 3.1"}],"minor_comments":[{"comment":"The text refers to 'Gemini-Flash-2.5' while Tables 1 and 2 use 'Gemini-2.5-Flash'; please standardize the model name throughout.","section":"Figure 4 caption and Section 4.4"},{"comment":"The metrics are defined as CompliantAcc and NonCompliantAcc, but Table 1 and the main text use 'Compliant Correctness' and 'Non-Compliant Correctness'; please align the metric names for consistency.","section":"Section 3.1 and Table 1"},{"comment":"The abstract's claim that models are 'not well-suited to handling confidential information' is broader than the evaluated setting, which is password-gated, exact-string matching in text prompts. A scope sentence in the abstract or conclusion would prevent overgeneralization.","section":"Abstract and Section 5"}],"recommendation":"major_revision","confidential_remarks":"The self-citation pattern is not a concern here: the central claims rest on fresh measurements on a new benchmark, and the cited prior works are directly relevant. The main risk is the token-cap truncation issue; if a rerun with a larger cap confirms the current conclusions, this would become a strong paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things up front. First, this paper gives us PasswordEval, a clean password-gated access benchmark with 500 GPT-4o-generated scenarios, public code and data, and exact-string metrics across many model families. Second, the genuinely new result is that reasoning traces leak passwords and confidential information even when the final output is clean. That finding is reproducible and directly relevant to anyone deciding whether to expose chain-of-thought. The benchmark itself is also reusable and already quantifies a specific safety gap.\n\nThe paper does several things well. The benchmark design is simple and the evaluation criteria are transparent. The multi-turn variant with ordered passwords is a nice stress test, and the comparison of direct requests, template jailbreaks, GCG, and PAIR is thorough. The credit for the trace-leak result is earned: Greenblatt et al. studied password-locked models but did not analyze trace leakage across jailbreaks and multi-turn settings. The measurements look direct and the code/data release makes the work easy to build on.\n\nNow the soft spots, in proportion. The stress-test concern about token caps is real and I agree it is the load-bearing issue. Generations are capped at 100 tokens, or 400 for reasoning models, and the chain-of-thought counts against that cap. The paper never reports finish_reason or the fraction of truncated responses. If Qwen-3 or Gemini thinking modes run past 400 tokens, their output is cut before the final answer, so Compliant Correctness gets understated and trace-leak rates can also be understated. That threatens the two headline claims: reasoning does not help rule-following (Figure 2) and traces leak even when outputs do not (Figure 4). The authors should rerun with a higher cap or report truncation statistics before making those claims strongly. This is an addressable experimental fix, not a fatal flaw. The other weaknesses are minor by comparison: no confidence intervals or per-condition sample sizes, no human validation of the generated scenarios, and exact-string metrics will miss paraphrased leaks. The scope caveat about password-in-prompt setups is acknowledged in the discussion, which explicitly recommends tool-based authentication, so I would not call the central claim overbroad.\n\nWho is this for? People working on agent safety, access control, and reasoning-trace transparency. It deserves a serious referee, and I would want the truncation analysis to be requested in revision. The benchmark is worth citing either way; the trace-leak result is important and mostly solid.","headline":"A useful new benchmark with a genuinely novel trace-leak finding, but the token-cap truncation issue needs to be addressed before the reasoning claims can be trusted.","tokens_in":16870,"tokens_out":1604,"would_cite":true,"duration_ms":17206,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Even simple password gates defeat frontier language models, and their reasoning traces leak the secret.","keywords":["contextual robustness","rule following","password-gated access control","information leakage","reasoning traces","jailbreaking","language model safety","benchmark"],"falsifier":"A single counterexample would refute the blanket conclusion: any frontier model that scores at least 98% on both CompliantAcc and NonCompliantAcc on the full PasswordEval benchmark, including the template jailbreak, and leaks neither the password nor the secret in its reasoning traces on the non-compliant set.","tokens_in":15914,"feed_emoji":"🔐","tokens_out":10457,"duration_ms":79471,"temperature":0.7,"pith_summary":"This paper introduces PasswordEval, a benchmark in which a language model is given a password and a piece of confidential information and must reveal the information only to a user who supplies the correct password. Across open and closed frontier models, the paper finds that this seemingly simple rule is routinely violated: models leak the password or the secret, over-refuse even when the correct password is given, and become substantially more vulnerable when requests are run through simple jailbreak templates. The paper's central claim is that current models are not well-suited to handling confidential information. It further finds that enabling reasoning does not reliably improve rule following, and that reasoning traces frequently contain the very password or secret the model is supposed to protect, which the paper argues is a security risk if such traces are exposed to users or released publicly.","feed_headline":"LLMs fail a simple password gate and leak secrets in reasoning","feed_subtitle":"New 500-scenario benchmark shows even correctly passworded requests get refused, and thinking traces reveal the secret.","key_machinery":"The central object is PasswordEval, a benchmark of 500 password-gated scenarios generated by GPT-4o, plus a multi-turn variant (MultiPasswordEval) that requires $d$ ordered passwords for $d$ turns with $d \\in [2,10]$. Each scenario embeds a password $p$ and a confidential string $c$ in the system prompt; the model is judged by exact string containment: $\\mathrm{CompliantAcc} = \\mathbf{1}[p \\in u] \\cdot \\mathbf{1}[c \\subset T]$ and $\\mathrm{NonCompliantAcc} = \\mathbf{1}[p \\notin u] \\cdot \\mathbf{1}[c \\not\\subset T \\wedge p \\not\\subset T]$, together with leak metrics $\\mathrm{ConfInfoLeak}$ and $\\mathrm{PasswordLeak}$. This string-match protocol is what makes the evaluation verifiable: the rule-following criteria are exact and unambiguous. The benchmark also scales difficulty by adding adversarial user pressure (a fixed jailbreak template, gradient-based GCG suffixes, and LLM-optimized PAIR prompts) and by increasing the number of required passwords over conversation turns. The mechanism that carries the argument is the controlled variation of task difficulty and the separate measurement of output leakage versus reasoning-trace leakage.","core_discovery":"PasswordEval defines correct behavior by two verifiable conditions: when the password $p$ is in the user prompt, the model's output must contain the confidential information $c$; when $p$ is absent, the output must contain neither $p$ nor $c$. On 500 GPT-4o-generated scenarios, no evaluated frontier model satisfies both conditions at high rates: while many models score near 100% at withholding information when the password is absent (Non-Compliant Correctness), their accuracy at revealing the secret when the correct password is given (Compliant Correctness) is often below 90% and sometimes below 40%. The paper finds that template-based jailbreaks degrade performance substantially, that adaptive attacks (GCG, which optimizes adversarial suffixes, and PAIR, which iteratively edits prompts via an attacker LLM) transfer across models, and that multi-password, multi-turn verification becomes harder as the number of required passwords grows. The paper further finds that reasoning traces—the thinking text produced by models such as Qwen-3 and Gemini-2.5-Flash—frequently contain the password or the secret even when the final output is correct, with leak rates above 90% for some models under the jailbreak template. The paper concludes that reasoning, as currently trained, does not make models safer for confidential-information handling, and that releasing reasoning traces in such applications is unsafe without further supervision such as process reward modeling.","pith_inferences":["Because PasswordEval scores only exact string containment, paraphrased leaks of the secret or password are counted as correct refusals; measuring semantic leakage with an LLM judge would likely raise the measured leak rates and strengthen the paper's caution about exposing reasoning traces.","The benchmark's password-in-the-prompt setup may not transfer to deployments that use hashed passwords, hardware tokens, or external verification, where errors would be prevented before the model sees the secret; the conclusion that models cannot handle confidential information is best read as applying to native text-based authentication.","A natural follow-up is to test whether reasoning-trace supervision that rewards traces never mentioning the secret reduces output leakage without hurting helpfulness; the paper's data suggest the leakage is a training-signal problem rather than a fixed property of reasoning.","The multi-turn results imply that agentic systems accumulating secrets over long conversations face compounding risk; extending PasswordEval to hundreds of turns or interleaved tool calls could quantify that risk."],"forward_implications":["Reasoning traces from models like Qwen-3 and Gemini-2.5-Flash should not be exposed to users or released in safety-critical applications, because they can leak the password or the secret even when the final answer is correct.","Access control for confidential information should not be implemented purely through textual instructions; the paper argues for external authentication tools or API-level gates.","Current post-training and reasoning-training recipes do not reliably produce contextual robustness, so new training signals, such as process reward modeling that supervises reasoning traces, are needed.","Simple template-based jailbreaks are a strong threat model: for several open models they degrade refusal performance more than gradient-based or LLM-optimized attacks.","Scaling the number of required passwords up to ten monotonically degrades rule-following performance, so composed tasks that look straightforward can become unsafe in agentic settings."],"supporting_citations":[{"why":"Provides the password-locked model setup that PasswordEval adapts for its access-control scenarios.","marker":"Greenblatt et al., 2024"},{"why":"Supplies the fixed jailbreak template used to stress-test whether models obey the password rule under adversarial pressure.","marker":"Andriushchenko et al., 2024"},{"why":"Supplies GCG, the gradient-based adversarial suffix attack whose outputs are transferred to all evaluated models.","marker":"Zou et al., 2023"},{"why":"Supplies PAIR, the LLM-driven adaptive jailbreak used alongside GCG in the adversarial evaluation.","marker":"Chao et al., 2023"},{"why":"Provides prior evidence that models are brittle at following system-prompt instructions, which PasswordEval's failures build on.","marker":"Mu et al., 2025"},{"why":"Shows that scaling reasoning can hurt instruction following, consistent with the paper's finding that reasoning does not help rule following.","marker":"Fu et al., 2025"},{"why":"Argues that releasing reasoning traces is a security concern, the position the trace-leakage experiments support.","marker":"Savani et al., 2025"},{"why":"Provides process reward modeling, the approach the paper suggests for supervising reasoning traces so they stop leaking secrets.","marker":"Lightman et al., 2023"},{"why":"Shows models can learn to use tools, the kind of external verification mechanism the paper recommends instead of textual password rules.","marker":"Schick et al., 2023"}],"fun_headline_variants":["LLMs leak secrets in reasoning traces, even with password","PasswordEval: Frontier models fail simple confidentiality test","Reasoning doesn't help LLMs keep secrets, study finds","LLM reasoning traces reveal passwords and secrets at 90%","Model thinking leaks confidential info despite correct password"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results assume that putting a plaintext password in the system prompt and checking exact string containment in the output captures how confidential information is actually guarded; real deployments with hashed passwords or external authentication might not fail the same way.","fun_headline_variants_meta":{"raw":{"variants":["LLMs leak secrets in reasoning traces, even with password","PasswordEval: Frontier models fail simple confidentiality test","Reasoning doesn't help LLMs keep secrets, study finds","LLM reasoning traces reveal passwords and secrets at 90%","Model thinking leaks confidential info despite correct password"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000163,"raw_usage":{"total_tokens":1288,"prompt_tokens":1037,"completion_tokens":251,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":653,"completion_tokens_details":{"reasoning_tokens":172}},"tokens_in":653,"tokens_out":251,"duration_ms":3285,"temperature":1.0,"reasoning_tokens":172,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:48:56.791252+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A single counterexample would refute the blanket conclusion: any frontier model that scores at least 98% on both CompliantAcc and NonCompliantAcc on the full PasswordEval benchmark, including the template jailbreak, and leaks neither the password nor the secret in its reasoning traces on the non-compliant set.","supporting_citations":[],"review_version":1}