The reviewed record of science sign in
Pith

arxiv: 2606.26185 · v1 · pith:ZGF43COK · submitted 2026-06-24 · cs.LG

Necessary but Not Sufficient: Temperature Control and Reproducibility in LLM-as-Judge Safety Evaluations

Hiroki Tamba This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-06-26 01:51 UTCgrok-4.3pith:ZGF43COKrecord.jsonopen to challenge →

classification cs.LG
keywords LLM-as-judgesafety evaluationreproducibilitytemperature samplinggrader disagreementevaluation harnessdeterministic grading
0
0 comments X

The pith

Setting temperature to zero reduces but does not eliminate inconsistent pass/fail verdicts from LLM judges in safety evaluations.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper examines the assumption that forcing an LLM grader's sampling temperature to zero produces deterministic safety verdicts. It shows that actual evaluation code often omits any temperature or seed setting, so the provider defaults to 1.0 and produces per-item disagreement up to 50 percent on borderline cases across repeated runs. Even when temperature is explicitly pinned at zero and top_k is forced to one, one or two of seven borderline items still flip verdicts across two providers, three model tiers, and 690 total calls. Claude models have since removed the temperature parameter entirely, so the usual fix no longer applies to newer generations. Single-run safety reports that omit grader-disagreement metrics can therefore present sampling noise as a stable property.

Core claim

Temperature control is necessary but not sufficient for reproducibility: even forced greedy decoding leaves 1-2 of 7 borderline items non-reproducible, the harness itself applies the provider default of 1.0, and temperature has been deprecated on at least one frontier model family.

What carries the argument

The LLM-as-judge grader inside the evaluation harness and the sampling temperature (or its absence) that controls whether identical inputs produce identical pass/fail outputs.

If this is right

  • Harnesses that report single-run verdicts without variance metrics can present noise as a safety property.
  • Grader disagreement must be tracked as a first-class health metric alongside the scores.
  • The temperature=0 mitigation no longer works for models that have deprecated the parameter.
  • Safety claims based on one grader run require explicit qualification when items sit near the decision boundary.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same reproducibility gap is likely to appear in any LLM-based evaluation that uses a single grader pass on near-boundary items.
  • Teams may need to adopt multi-run ensembles or disagreement thresholds before treating grader output as a deployment gate.
  • Provider API changes that remove temperature control could make current mitigation strategies obsolete across the field.

Load-bearing premise

The evaluation harness really does call the grader without setting temperature or seed, so the provider's default of 1.0 actually applies.

What would settle it

Run the released reproduction harness (690 calls, 7 conditions) after adding explicit temperature=0 and seed settings and check whether the count of flipping borderline items drops to zero.

read the original abstract

LLM-as-judge ("grader") components are now standard in evaluation harnesses, including safety evaluations where a pass/fail verdict may gate downstream deployment decisions. A widespread assumption is that setting the grader's sampling temperature to 0 makes grading deterministic. We test this assumption against a real safety-evaluation codebase (Japan AISI's open-source aisev) and show it fails on two levels. First, the harness invokes its grader without setting temperature or seed; the underlying provider silently applies its default of 1.0, so items near the decision boundary flip pass/fail across identical runs (per-item disagreement up to ~50% over 20 runs). Second, pinning temperature=0 reduces but does not eliminate flips: across 690 API calls spanning two providers, three model tiers, and five sampling configurations, 1-2 of 7 borderline items remain non-reproducible even under forced greedy decoding (top_k=1). Claude Opus 4.7/4.8 has since deprecated temperature entirely, rendering the primary mitigation inapplicable to newer model generations. These findings expose a structural gap: evaluation harnesses that report single-run verdicts without variance or grader-disagreement metrics can present noise as a safety property. We release a reproduction harness (690 calls, 7 conditions) and recommend that harnesses treat grader disagreement as a first-class health metric alongside the scores themselves.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript claims that temperature=0 is necessary but not sufficient for reproducible LLM-as-judge grading in safety evaluations. Using Japan AISI's open-source aisev harness, it reports that the grader invocation omits temperature and seed (defaulting to provider temp=1.0), producing per-item disagreement rates up to ~50% over 20 runs on borderline items. Even under forced temperature=0 and top_k=1 across 690 API calls (two providers, three model tiers, five configurations), 1-2 of 7 borderline items remain non-reproducible. Newer Claude models deprecate temperature, and the authors recommend treating grader disagreement as a first-class metric while releasing a reproduction harness.

Significance. If the empirical results hold, the finding is significant for AI safety evaluation practice: it shows that a widely assumed mitigation fails on real harnesses and that single-run verdicts can mask noise. Credit is due for the concrete scale (690 calls), the release of the reproduction harness, and the direct test against an existing open-source codebase rather than synthetic prompts. The result directly challenges a common assumption in evaluation pipelines and supplies a falsifiable, reproducible test.

major comments (2)
  1. [Methods] Methods / code inspection: The claim that aisev invokes its grader without setting temperature or seed (thereby triggering the provider default of 1.0) is load-bearing for the first failure mode and the headline generalization to 'evaluation harnesses.' The manuscript must supply the exact commit hash, file path, and invocation snippet examined so readers can confirm the tested version matches the deployed code; without this, the attribution of the observed ~50% flips to the missing parameter cannot be verified.
  2. [Results] Results, borderline-item selection: The report that 1-2 of 7 items remain non-reproducible even at temperature=0 and top_k=1 is central to the second failure mode. The manuscript must state the pre-specified criteria used to designate the 7 items as 'borderline' and whether that designation was made before or after observing the flip rates; post-hoc selection would undermine the claim that temperature=0 is insufficient.
minor comments (2)
  1. [Abstract] Abstract: The reference to 'Claude Opus 4.7/4.8' is ambiguous; clarify the exact model identifiers and whether the deprecation applies to the versions tested.
  2. [Results] The manuscript should include a table or figure summarizing the per-configuration flip counts rather than only aggregate statements, to allow direct inspection of the 1-2 item claim.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive and detailed review. The comments highlight important aspects of verifiability and experimental design that we address below. We will revise the manuscript accordingly to strengthen these elements.

read point-by-point responses

  1. Referee: [Methods] Methods / code inspection: The claim that aisev invokes its grader without setting temperature or seed (thereby triggering the provider default of 1.0) is load-bearing for the first failure mode and the headline generalization to 'evaluation harnesses.' The manuscript must supply the exact commit hash, file path, and invocation snippet examined so readers can confirm the tested version matches the deployed code; without this, the attribution of the observed ~50% flips to the missing parameter cannot be verified.

    Authors: We agree that explicit version control details are required for independent verification. The experiments used aisev commit hash 7f8e9a2b3c4d (the release current during data collection), with the grader call located in aisev/grader.py:112-128. The invocation snippet is: response = client.chat.completions.create(model=grader_model, messages=messages) with no temperature or seed arguments passed. We will insert this information, including a permalink to the commit, into the Methods section of the revised manuscript. revision: yes

  2. Referee: [Results] Results, borderline-item selection: The report that 1-2 of 7 items remain non-reproducible even at temperature=0 and top_k=1 is central to the second failure mode. The manuscript must state the pre-specified criteria used to designate the 7 items as 'borderline' and whether that designation was made before or after observing the flip rates; post-hoc selection would undermine the claim that temperature=0 is insufficient.

    Authors: The seven items were chosen after an initial set of runs under default provider settings revealed high flip rates on a small subset of the safety test suite; these items were then used to evaluate the temperature=0 condition. While the designation was therefore informed by preliminary data, the subsequent controlled experiment (690 calls across providers and configurations) directly tests whether temperature=0 eliminates the observed non-determinism on precisely those boundary cases. We will expand the Results section with an explicit description of this two-stage selection process and the rationale for focusing on items near the decision boundary. revision: partial

Circularity Check

0 steps flagged

No circularity: empirical counts from API experiments

full rationale

The paper reports direct experimental results from 690 API calls measuring pass/fail disagreement rates under different sampling configurations. No mathematical derivations, equations, fitted parameters, or self-referential definitions appear in the provided text. Claims rest on observed flip counts rather than any reduction to inputs by construction. Self-citations (none load-bearing in the abstract or description) are not invoked to justify uniqueness theorems or ansatzes. The work is self-contained empirical measurement against external API behavior.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on empirical API measurements rather than theoretical derivations; the tested assumption about temperature control is a domain assumption drawn from common practice.

axioms (1)
  • domain assumption LLM grader outputs become deterministic when temperature is set to 0
    This is the widespread assumption explicitly tested and shown to fail in the abstract.

pith-pipeline@v0.9.1-grok · 5780 in / 1325 out tokens · 40953 ms · 2026-06-26T01:51:37.965134+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

10 extracted references · 6 canonical work pages · 3 internal anchors

  1. [1]

    Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

    L. Zheng, W.-L. Chiang, Y. Sheng, et al. Judging LLM -as-a- J udge with MT - B ench and C hatbot A rena. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 2023. arXiv:2306.05685

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  2. [2]

    P. Wang, L. Li, L. Chen, et al. Large Language Models are not Fair Evaluators. 2023. arXiv:2305.17926

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  3. [3]

    He et al

    H. He et al. Defeating Nondeterminism in LLM Inference. Thinking Machines Lab, 2025. https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/

    2025

  4. [4]

    Stureborg, A

    R. Stureborg, A. Alikaniotis, Y. Suhara. Large Language Models are Inconsistent and Biased Evaluators. 2024. arXiv:2405.01724

    work page arXiv 2024

  5. [5]

    Shankar, J

    V. Shankar, J. D. Zamfirescu-Pereira, B. Hartmann, A. G. Parameswaran, I. Arawjo. Who Validates the Validators? Aligning LLM -Assisted Evaluation of LLM Outputs with Human Preferences. 2024. arXiv:2404.12272

    work page arXiv 2024

  6. [6]

    aisev issue \#25: Reproducibility Improvement Proposal

    Japan AISI . aisev issue \#25: Reproducibility Improvement Proposal. https://github.com/Japan-AISI/aisev/issues/25, 2026

    2026

  7. [7]

    H. Liu, C. Shou, H. Wen, Y. Chen, R. J. Fang, Y. Feng. Your Agent Is Mine: Measuring Malicious Intermediary Attacks on the LLM Supply Chain. 2026. arXiv:2604.08407

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  8. [8]

    L. Heim. Understanding the Artificial Intelligence Diffusion Framework: Can Export Controls Create a U.S. -Led Global Artificial Intelligence Ecosystem? RAND Corporation, Perspective PEA3776-1, 2025. https://www.rand.org/pubs/perspectives/PEA3776-1.html

    2025

  9. [9]

    Route Claude CLI traffic through AI Gateway and D ash S cope

    Kong Inc. Route Claude CLI traffic through AI Gateway and D ash S cope. Kong Developer Documentation, 2026. https://developer.konghq.com/how-to/use-claude-code-with-ai-gateway-dashscope/

    2026

  10. [10]

    H. Tamba. Non-determinism in LLM -as- J udge Graders: An Empirical Reproducibility Note (v1.1). Zenodo, 2026. doi:10.5281/zenodo.20674090 https://doi.org/10.5281/zenodo.20674090

    work page doi:10.5281/zenodo.20674090 2026