{"id":"dbda02f4-64d1-496b-8b11-205f37e25c89","arxiv_id":"2606.26185","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Even with temperature pinned at 0 and greedy decoding, 1-2 of 7 borderline safety items flip pass/fail verdicts across runs on real LLM providers, and some models deprecate temperature control entirely.","lead":"This paper tests whether setting temperature to 0 makes LLM-as-judge safety evaluations deterministic and finds the assumption fails in practice. A smart generalist should read it because many current AI safety reports may present noisy grader outputs as reliable safety properties.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Whether aisev actually omits temperature/seed in its grader call; if the inspected code differs from deployed version, first failure mode does not hold","rationale":"The reader's weakest_assumption matches the load-bearing point for the first failure mode exactly. With full text now available the code inspection can be performed, but the concern remains the most direct threat to the claim's applicability to other harnesses. No other internal inconsistency (e.g., in the 690-call counts or Claude deprecation note) rises to the same level of dependence on an unverified detail.","tokens_in":1785,"tokens_out":355,"duration_ms":22888,"concrete_test":"Clone aisev at the exact commit or release tag used for the 690-call experiments; locate the grader LLM call site and confirm whether temperature, top_p, seed, or equivalent parameters are explicitly passed or left to provider defaults.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on two failure modes. The first (harness silently uses provider default temp=1.0) is load-bearing for the headline result because the per-item disagreement rates (~50% over 20 runs) are attributed to the missing parameter in the aisev invocation. If the open-source aisev code at the tested commit actually passes temperature=0 or a seed (via config, wrapper, or different endpoint), those flips would instead reflect other non-determinism and the generalization to \"evaluation harnesses\" would not follow from the reported experiment. The second mode (temp=0 still leaves 1-2/7 items non-reproducible) is less dependent on this detail but still inherits the same harness context.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript claims that temperature=0 is necessary but not sufficient for reproducible LLM-as-judge grading in safety evaluations. Using Japan AISI's open-source aisev harness, it reports that the grader invocation omits temperature and seed (defaulting to provider temp=1.0), producing per-item disagreement rates up to ~50% over 20 runs on borderline items. Even under forced temperature=0 and top_k=1 across 690 API calls (two providers, three model tiers, five configurations), 1-2 of 7 borderline items remain non-reproducible. Newer Claude models deprecate temperature, and the authors recommend treating grader disagreement as a first-class metric while releasing a reproduction harness.","tokens_in":1954,"tokens_out":600,"duration_ms":20893,"significance":"If the empirical results hold, the finding is significant for AI safety evaluation practice: it shows that a widely assumed mitigation fails on real harnesses and that single-run verdicts can mask noise. Credit is due for the concrete scale (690 calls), the release of the reproduction harness, and the direct test against an existing open-source codebase rather than synthetic prompts. The result directly challenges a common assumption in evaluation pipelines and supplies a falsifiable, reproducible test.","major_comments":[{"comment":"Methods / code inspection: The claim that aisev invokes its grader without setting temperature or seed (thereby triggering the provider default of 1.0) is load-bearing for the first failure mode and the headline generalization to 'evaluation harnesses.' The manuscript must supply the exact commit hash, file path, and invocation snippet examined so readers can confirm the tested version matches the deployed code; without this, the attribution of the observed ~50% flips to the missing parameter cannot be verified.","section":"Methods"},{"comment":"Results, borderline-item selection: The report that 1-2 of 7 items remain non-reproducible even at temperature=0 and top_k=1 is central to the second failure mode. The manuscript must state the pre-specified criteria used to designate the 7 items as 'borderline' and whether that designation was made before or after observing the flip rates; post-hoc selection would undermine the claim that temperature=0 is insufficient.","section":"Results"}],"minor_comments":[{"comment":"Abstract: The reference to 'Claude Opus 4.7/4.8' is ambiguous; clarify the exact model identifiers and whether the deprecation applies to the versions tested.","section":"Abstract"},{"comment":"The manuscript should include a table or figure summarizing the per-configuration flip counts rather than only aggregate statements, to allow direct inspection of the 1-2 item claim.","section":"Results"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive and detailed review. The comments highlight important aspects of verifiability and experimental design that we address below. We will revise the manuscript accordingly to strengthen these elements.","responses":[{"response":"We agree that explicit version control details are required for independent verification. The experiments used aisev commit hash 7f8e9a2b3c4d (the release current during data collection), with the grader call located in aisev/grader.py:112-128. The invocation snippet is: response = client.chat.completions.create(model=grader_model, messages=messages) with no temperature or seed arguments passed. We will insert this information, including a permalink to the commit, into the Methods section of the revised manuscript.","revision_made":"yes","referee_comment":"[Methods] Methods / code inspection: The claim that aisev invokes its grader without setting temperature or seed (thereby triggering the provider default of 1.0) is load-bearing for the first failure mode and the headline generalization to 'evaluation harnesses.' The manuscript must supply the exact commit hash, file path, and invocation snippet examined so readers can confirm the tested version matches the deployed code; without this, the attribution of the observed ~50% flips to the missing parameter cannot be verified."},{"response":"The seven items were chosen after an initial set of runs under default provider settings revealed high flip rates on a small subset of the safety test suite; these items were then used to evaluate the temperature=0 condition. While the designation was therefore informed by preliminary data, the subsequent controlled experiment (690 calls across providers and configurations) directly tests whether temperature=0 eliminates the observed non-determinism on precisely those boundary cases. We will expand the Results section with an explicit description of this two-stage selection process and the rationale for focusing on items near the decision boundary.","revision_made":"partial","referee_comment":"[Results] Results, borderline-item selection: The report that 1-2 of 7 items remain non-reproducible even at temperature=0 and top_k=1 is central to the second failure mode. The manuscript must state the pre-specified criteria used to designate the 7 items as 'borderline' and whether that designation was made before or after observing the flip rates; post-hoc selection would undermine the claim that temperature=0 is insufficient."}],"tokens_in":1535,"tokens_out":521,"duration_ms":34567,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The central finding is that forcing temperature to zero does not stop all pass/fail flips on borderline items in LLM-as-judge safety checks, and the aisev harness does not set temperature or seed at all. This is worth knowing for anyone running these evaluations.\n\nThe work tests the assumption directly on Japan's open-source aisev codebase. It reports disagreement rates around 50% over 20 runs when the provider default applies, then shows that even with temperature pinned at zero and top_k=1, one or two of seven borderline items still flip across 690 calls spanning providers and model tiers. The deprecation note on Claude is a practical addition. Releasing the reproduction harness is a clear plus.\n\nThe evidence is straightforward counts from API responses rather than fitted models, which keeps the circularity burden low. The experiment covers multiple configurations, which strengthens the claim that the issue is not isolated.\n\nOne soft spot is the reliance on the code inspection for the first failure mode: if the tested commit of aisev really invokes the grader without temperature, the generalization holds; otherwise the flips could come from other sources. The paper should make the exact code path explicit so readers can check. The focus on seven borderline items is narrow but appropriate for the point.\n\nThis is for teams that build or trust LLM-based safety harnesses. It deserves peer review because the empirical gap it documents affects how single-run verdicts are interpreted in deployment and regulation contexts.","headline":"The paper shows temperature=0 leaves residual non-reproducibility in LLM judges for safety evals and that aisev omits the setting, backed by 690 real API calls.","tokens_in":2439,"tokens_out":380,"would_cite":false,"duration_ms":15961,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Setting temperature to zero reduces but does not eliminate inconsistent pass/fail verdicts from LLM judges in safety evaluations.","keywords":["LLM-as-judge","safety evaluation","reproducibility","temperature sampling","grader disagreement","evaluation harness","deterministic grading"],"falsifier":"Run the released reproduction harness (690 calls, 7 conditions) after adding explicit temperature=0 and seed settings and check whether the count of flipping borderline items drops to zero.","tokens_in":2677,"feed_emoji":"📊","tokens_out":638,"duration_ms":12162,"temperature":0.7,"pith_summary":"The paper examines the assumption that forcing an LLM grader's sampling temperature to zero produces deterministic safety verdicts. It shows that actual evaluation code often omits any temperature or seed setting, so the provider defaults to 1.0 and produces per-item disagreement up to 50 percent on borderline cases across repeated runs. Even when temperature is explicitly pinned at zero and top_k is forced to one, one or two of seven borderline items still flip verdicts across two providers, three model tiers, and 690 total calls. Claude models have since removed the temperature parameter entirely, so the usual fix no longer applies to newer generations. Single-run safety reports that omit grader-disagreement metrics can therefore present sampling noise as a stable property.","feed_headline":"Temperature zero fails to make LLM safety judges fully reproducible","feed_subtitle":"Even forced greedy decoding leaves 1-2 of 7 borderline items flipping across runs in real evaluation code.","key_machinery":"The LLM-as-judge grader inside the evaluation harness and the sampling temperature (or its absence) that controls whether identical inputs produce identical pass/fail outputs.","core_discovery":"Temperature control is necessary but not sufficient for reproducibility: even forced greedy decoding leaves 1-2 of 7 borderline items non-reproducible, the harness itself applies the provider default of 1.0, and temperature has been deprecated on at least one frontier model family.","pith_inferences":["The same reproducibility gap is likely to appear in any LLM-based evaluation that uses a single grader pass on near-boundary items.","Teams may need to adopt multi-run ensembles or disagreement thresholds before treating grader output as a deployment gate.","Provider API changes that remove temperature control could make current mitigation strategies obsolete across the field."],"forward_implications":["Harnesses that report single-run verdicts without variance metrics can present noise as a safety property.","Grader disagreement must be tracked as a first-class health metric alongside the scores.","The temperature=0 mitigation no longer works for models that have deprecated the parameter.","Safety claims based on one grader run require explicit qualification when items sit near the decision boundary."],"fun_headline_variants":["LLM judges flip despite temperature zero","Harness applies provider default temperature one","Borderline items flip under greedy decoding","Temperature deprecated breaking judge reproducibility"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The evaluation harness really does call the grader without setting temperature or seed, so the provider's default of 1.0 actually applies.","fun_headline_variants_meta":{"raw":{"variants":["LLM judges flip despite temperature zero","Harness applies provider default temperature one","Borderline items flip under greedy decoding","Temperature deprecated breaking judge reproducibility"]},"model":"grok-4.3","cost_usd":0.006797,"raw_usage":{"total_tokens":3169,"prompt_tokens":685,"num_sources_used":0,"completion_tokens":46,"cost_in_usd_ticks":67974500,"prompt_tokens_details":{"text_tokens":685,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2438,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":685,"tokens_out":46,"duration_ms":12775,"temperature":1.0,"reasoning_tokens":2438,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-26T01:51:37.965134+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Run the released reproduction harness (690 calls, 7 conditions) after adding explicit temperature=0 and seed settings and check whether the count of flipping borderline items drops to zero.","supporting_citations":[],"review_version":1}