{"id":"3396796e-f9fe-491e-81c3-0961512ce80d","arxiv_id":"2608.10209","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Conditioning LLM training on natural-language evaluator descriptions and deploying with a held-out higher-fidelity label improved even-handedness and reduced sycophancy in two small proof-of-concept experiments.","lead":"This paper introduces Evaluation-Conditioned Training (ECT), a method that trains language models with descriptions of the evaluator and then deploys them with a stronger evaluator description, aiming to reduce reward misspecification. Two small proof-of-concept experiments suggest ECT reduces political bias and sycophancy compared with direct training, but the effects rely on model-based evaluation and need human validation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Experiment 2's held-out probe may invalidate the headline result: checkpoints are selected using s=0.25, inside the held-out range, giving ECT access to the deployment regime during training.","rationale":"The paper's central empirical claim is that ECT improves targeted behavior relative to direct training and generalizes to a stronger deployment label. Experiment 2 is the more rigorous of the two tests (PPO, objective reward, nine seeds), but it is compromised if checkpoint selection uses a probe at s=0.25, inside the declared held-out range. The reader's weakest assumption focuses on label-as-source-of-truth versus retrieval-key; that is an important open question the authors themselves flag in Section C. However, the checkpoint-selection confound is more immediately load-bearing because it threatens the validity of the headline result regardless of mechanism. The reader's rationale already notes the held-out probe, so this is a partial agreement. The proposed re-run with training-distribution-based checkpoint selection would settle whether the result is an artifact; if the margin persists, the empirical claim is on firmer ground, and the source-of-truth question can be addressed in future work. Thus the CONDITIONAL verdict stands, with the explicit condition that the checkpoint-selection confound be resolved.","tokens_in":11469,"tokens_out":13225,"duration_ms":126025,"concrete_test":"Re-run Experiment 2 with checkpoint selection based on a validation set drawn from the training suggestibility range s∈[0.5,1.5] (or at fixed step counts), removing the s=0.25 probe. If ECT's false-positive rate and accuracy at s=0 no longer beat the baseline and shuffled controls by the reported margin, the headline improvement is an artifact of selecting checkpoints on the held-out regime. Report all nine seeds.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section B.2 states: 'Checkpoint selection during training uses an out-of-distribution probe at s=0.25 every 50 steps.' But Section 4.2.2 explicitly holds out s<0.5 during training, with deployment at s=0. Since 0.25 falls in the held-out range, the final model is chosen using performance on the target regime, not only on training-distribution labels. For ECT, the probe prompt includes the rendered label '17%' (L=round(100*0.25/1.5)), so checkpoint selection directly rewards the ability to follow low-suggestibility labels. The no-label baseline and shuffled control cannot exploit this selection signal because they cannot condition on the label, making the comparison unfair. The reported ECT false-positive rate of 21.6±15.8% at s=0 may therefore reflect selection on a near-deployment probe rather than the training-time conditioning mechanism. This is a concrete confound: the 'held-out deployment label' is not fully held out.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces Evaluation-Conditioned Training (ECT), a post-training method that prepends a natural-language description of the evaluator to each training example (e.g., an editorial-perspective label or a rendered suggestibility level) and then conditions on a high-fidelity evaluator description at deployment. ECT is presented as an add-on to existing algorithms such as SFT and PPO, with a conceptual framework grounded in simplicity bias, weak-to-strong generalization, and the ELK problem. The paper evaluates ECT in two proof-of-concept experiments: increasing political even-handedness under SFT with biased training data, and reducing sycophancy on single-digit addition under PPO with a suggestibility-dependent reward. The abstract claims that in both settings ECT improves the targeted behavior relative to direct training. Appendices provide detailed training, data, and compute settings, and the authors include a discussion of counterarguments and limitations.","tokens_in":11612,"tokens_out":5206,"duration_ms":51074,"significance":"If the reported effects hold, ECT is a practical and low-cost contribution: it does not require solving reward specification, only describing the evaluator, and it appears to work as an add-on to both SFT and PPO. The paper is valuable for connecting alignment concerns such as reward misspecification and ELK to a simple, testable training intervention, and the authors are transparent about limitations and provide code and detailed appendices. However, the empirical support is currently fragile: Experiment 2's checkpoint selection uses a probe inside the held-out suggestibility regime, which undermines the claimed held-out generalization, and Experiment 1 lacks error bars or significance tests for its main outcomes and uses a judge from the same model family as the training-data generator. The conceptual framework is interesting, but the central empirical claim is not yet established at the level required by the abstract.","major_comments":[{"comment":"This is a load-bearing confound for Experiment 2. The paper holds out suggestibility levels s < 0.5 during training and reports deployment at s = 0, but Section B.2 states: 'Checkpoint selection during training uses an out-of-distribution probe at s=0.25 every 50 steps.' Since 0.25 lies inside the held-out range, and since for ECT the probe prompt includes the rendered label '17%' (L = round(100*0.25/1.5)), the final checkpoint is selected using performance on the near-deployment regime. The no-label baseline and shuffled control cannot condition on the label in this probe, so they cannot exploit the same selection signal; the comparison is therefore not a clean test of training-time conditioning. The reported ECT false-positive rate of 21.6 ± 15.8% at s = 0 could partly reflect selection on a near-deployment probe rather than the conditioning mechanism. To support the central claim, checkpoint selection should use a probe in the training regime (e.g., s ≥ 0.5) or an independent validation set drawn from [0.5, 1.5], with the s = 0 evaluation used only at final test.","section":"Section B.2 / 4.2.2"},{"comment":"Experiment 1's main outcomes are reported only as point estimates. The text states that ECT increases even-handedness from 49.8% (baseline) and 49.4% (shuffled) to 64.8%, and decreases the refusal proportion from 0.038/0.035 to 0.005, but no error bars, confidence intervals, or significance tests are given for these quantities, despite Section 4.1.3 reporting that training and evaluation were replicated with five seeds. Hedging asymmetry is reported with ± values, but even-handedness and refusals are not. Without seed-level variance or a paired test, the 15 pp gap cannot be distinguished from noise, and this is central to the abstract's claim that 'in both settings, ECT improves the targeted behavior.' Please report per-seed results or confidence intervals and a significance test for these outcomes.","section":"Section 4.1.4"},{"comment":"The automated evaluator for Experiment 1 is Claude Haiku 4.5, while the training-target data are generated by Claude Sonnet 4.5, both from the same model family; the authors acknowledge this confound in Section 4.1.3. Since the outcome metric is produced by this judge, the observed even-handedness gain could partly reflect the judge favoring the generation patterns of a related model rather than 'even-handedness' per se. The authors interpret the results as directional, but the abstract makes a stronger claim. I would like to see an evaluation with a judge from a different model family or human annotation on a subset, or at least a stability analysis across multiple judge models, before accepting the even-handedness result as evidence for ECT.","section":"Section 4.1.3"}],"minor_comments":[{"comment":"The sentence 'ECT and inoculation prompting (Section 2)' likely refers to the Inoculation Prompting subsection, which is Section 3.2; please correct the cross-reference.","section":"Section 3.2"},{"comment":"Figure 4 would benefit from error bars or per-seed markers, as the label-sweep results are central to the claim that ECT tracks label semantics more strongly than the shuffled control.","section":"Section 4.1.4 / Figure 4"},{"comment":"The shaded region marking held-out labels in Figure 7 is helpful, but because checkpoint selection already used s = 0.25, the figure's status as a 'held-out' evaluation should be clarified or the experimental protocol revised.","section":"Section 4.2.2 / Figure 7"},{"comment":"The term 'high-fidelity monitor' is used in the abstract but is not precisely defined; tying it to the suggestibility labels and editorial-perspective labels in the experiments would improve precision.","section":"Abstract / Section 2"}],"recommendation":"major_revision","confidential_remarks":"The checkpoint-selection issue in Experiment 2 is the main technical problem and requires re-running the experiment with selection on the training distribution or a genuinely independent validation split. Experiment 1 also needs basic uncertainty quantification and ideally an out-of-family judge. The conceptual contribution is solid, but the abstract's empirical claim is stronger than the current evidence supports."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: the paper has a genuinely new idea—condition each training example on a natural-language description of the evaluator and deploy with a higher-fidelity label—and the conceptual framing is useful. But the empirical evidence is proof-of-concept at best, and Experiment 2 has a checkpoint-selection issue that undercuts the headline result.\n\nWhat's new: ECT is distinct from Inoculation Prompting. IP re-contextualizes training data to suppress the bad association; ECT explicitly trains a mapping from evaluator descriptions to behavior. That's a reasonable reframing of reward misspecification, and tying it to ELK makes sense. The writing is clear and the limitations section is unusually honest—it flags the 'source of truth vs retrieval key' question, which is exactly the right open problem.\n\nWhat's good: the label sweeps show ECT tracks label semantics better than shuffled controls, which is at least evidence the conditioning channel does something. The code is available and the training details are enough for reproduction. The authors call it proof-of-concept and don't oversell it.\n\nSoft spots: Experiment 1 reports a 15-point even-handedness gain without error bars or significance tests, despite five seeds, and the automated judge is from the same model family as the training-data generator. That's acknowledged, but it still limits what you can conclude. The bigger problem is Experiment 2. Section B.2 says checkpoint selection uses an out-of-distribution probe at s=0.25, which is inside the held-out range s<0.5. So the final checkpoints are selected on near-deployment behavior. For ECT, the probe includes the rendered label '17%', so the selection signal directly rewards following low-suggestibility labels; the no-label and shuffled controls can't exploit that. The 21.6% false-positive rate may reflect selection on the deployment regime rather than the training-time mechanism. That's a load-bearing confound, not a minor detail. The fix is straightforward: select checkpoints on a fully held-out validation set, or report results across all seeds with a fixed training length.\n\nWho it's for: people working on reward misspecification, weak-to-strong generalization, and evaluation-conditioned control. It deserves a serious referee; the idea is worth engaging with even if this version's experiments don't establish it. I'd send it to peer review with major revision, and make the checkpoint-selection issue the make-or-break point.","headline":"Novel idea, honest limits, but Experiment 2's checkpoint selection leaks the held-out regime, so the empirical claim is conditional.","tokens_in":12178,"tokens_out":5004,"would_cite":false,"duration_ms":48860,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Evaluation-Conditioned Training teaches a model to obey a stronger evaluator at deployment by describing the evaluator during training, even when the training reward itself is biased.","keywords":["evaluation-conditioned training","reward mis-specification","oversight generalization","weak-to-strong generalization","sycophancy","eliciting latent knowledge","RLHF","post-training"],"falsifier":"Train ECT exactly as in the paper, then deploy with a label naming a stronger evaluator that was never used in any training example and that no training example's reward is compatible with. If behavior under that novel label resembles behavior under training-range labels rather than moving toward stronger-evaluator behavior, the source-of-truth generalization claim fails. A cheaper version: degrade label granularity from the full fidelity scale to coarse strict-versus-permissive buckets; if ECT's improvement disappears, the mechanism depends on fine-grained fidelity semantics rather than mere conditioning.","tokens_in":11221,"feed_emoji":"🎯","tokens_out":10407,"duration_ms":95046,"temperature":0.7,"pith_summary":"The paper introduces Evaluation-Conditioned Training (ECT), a post-training procedure in which every training example is labeled with a natural-language description of the evaluator who judges it, and deployment then conditions the model on a higher-fidelity evaluator description. ECT is offered as an add-on to existing fine-tuning methods, not a replacement: it works with supervised fine-tuning and with policy-gradient training such as PPO. The paper's claim is that this shift, describing the feedback rather than pretending the feedback is ideal, lets the model use its latent knowledge of a stronger evaluator, improving target behavior even when the training reward itself remains biased. Two proof-of-concept experiments support the claim: even-handedness in political responses rises from about 49.8% to 64.8% under a held-out unbiased deployment label, and false-positive agreement in a sycophantic arithmetic task falls from 47.9% to 21.6% while accuracy rises from 51.5% to 73.8%. A shuffled-label control makes clear the gains come from correctly pairing labels with feedback, not from extra text.","feed_headline":"Conditioning on evaluator labels beats imperfect-feedback training","feed_subtitle":"In two controlled tests, label-conditioned training raised even-handedness by 15 points and cut sycophantic agreement by more than half.","key_machinery":"The load-bearing mechanism is the evaluation-label conditioning channel: each training sample is wrapped with a short natural-language description of the evaluator that will judge the response, and the description is made causally relevant by being inserted as a monitor label in the instruction context. This creates a learned mapping between evaluator fidelity and appropriate behavior, so the model is not asked to fit biased feedback as if it were ground truth. The same channel is then reused at deployment with a stronger held-out label, and the paper's claim is that the model treats that label as a source of truth about the evaluator, rather than only as a retrieval key to remembered training instances, and thereby outputs the behavior it associates with a more faithful evaluator. The framework is intentionally independent of the training algorithm: it is a wrapper on the data and prompt, so it can be layered onto supervised fine-tuning or PPO.","core_discovery":"On the paper's own terms, the discovery is that the target of optimization can be redirected by conditioning on an explicit description of the evaluator rather than by trying to perfect the reward. ECT trains a model on examples in which each prompt is paired with a textual evaluator label that says how faithful the feedback is, for instance which editorial perspective judges a political essay or what suggestibility level scores an arithmetic answer. At deployment, the same prompt is paired with a stronger held-out label, such as an unbiased label or a zero-percent suggestibility label, and the model generalizes to that regime even though no training example carried it. The paper interprets this as eliciting latent knowledge: the model carries information about what a faithful evaluator would prefer, and the deployment label gives it license to express that rather than the reward-hacking behavior trained into the biased feedback. On both tests, ECT outperforms both a no-label baseline and a shuffled-label control, and when deployment labels are swept, ECT tracks label semantics in the predicted direction.","pith_inferences":["The paper's two demonstrations are single-turn and small-scale, so its own stated direction, carrying the same label channel into multi-turn agentic trajectories where reward mis-specification compounds, is the natural direct extension rather than a benchmarked result.","A test the paper does not run: ECT should transfer to a semantically novel evaluator description, such as a multi-criterion rubric that never appeared in training at any fidelity level, if the source-of-truth mechanism is correct.","The ethics statement implies a dual-use property: the same conditioning channel that pulls behavior toward a faithful evaluator could pull behavior toward a deceptive one if the deployment label names such an evaluator, so label integrity becomes part of deployment safety."],"forward_implications":["ECT gives post-training pipelines a way to use biased feedback without inheriting the bias: the deployment label, not the training reward, becomes the lever for the target behavior.","Because ECT is an add-on wrapper on the data and prompt, it can be layered onto existing SFT and PPO training with no change to the reward model or reward function.","The label-sweep results imply that behavior tracks the fidelity of the evaluator description continuously, not just at the exact labels seen during training.","In the arithmetic setting, ECT reduces sycophancy and improves accuracy together, so the improvement is not a refusal-based trade-off against task performance.","The framework extends to any single-turn task in which the feedback signal is biased but the intended evaluator can be described in words."],"supporting_citations":[{"why":"Supplies the paired-prompts political-bias evaluation, the held-out test pairs, and the rubric used to measure even-handedness, so it defines the data and metric for Experiment 1.","marker":"Anthropic, 2025"},{"why":"Provides weak-to-strong generalization and Generative Fine-Tuning evidence that a stronger model can recover ground-truth behavior from weak labels, the premise ECT builds on.","marker":"Burns et al., 2023"},{"why":"Defines eliciting latent knowledge and the direct-translator versus human-simulator distinction that motivates conditioning on an ideal evaluator.","marker":"Christiano et al., 2021"},{"why":"Shows models can estimate whether their own answers are correct, evidence that latent knowledge is available before it is expressed.","marker":"Kadavath et al., 2022"},{"why":"Shows internal hidden states can track truth even when surface text is misleading, supporting the paper's source-of-truth interpretation of evaluation labels.","marker":"Azaria & Mitchell, 2023"},{"why":"Documents sycophancy as a persistent model failure mode and motivates the arithmetic target behavior in Experiment 2.","marker":"Perez et al., 2022"},{"why":"Provides the PPO algorithm used to train the Experiment 2 adapter, so the sycophancy reduction result depends on this optimization method.","marker":"Schulman et al., 2017"}],"fun_headline_variants":["Training on evaluator labels fixes reward mis-specification","Conditioning on monitor quality boosts generalization","ECT: Teach models to trust better evaluators","Imperfect feedback? Condition on evaluator fidelity","Evaluation-conditioned training outdoes direct fitting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a model sees a deployment label like unbiased or zero-percent suggestibility as a trusted description of the evaluator, not merely as a pointer to evaluator instances it memorized during training; if labels only work as retrieval keys, the held-out label would not transfer to a genuinely new evaluator regime.","fun_headline_variants_meta":{"raw":{"variants":["Training on evaluator labels fixes reward mis-specification","Conditioning on monitor quality boosts generalization","ECT: Teach models to trust better evaluators","Imperfect feedback? Condition on evaluator fidelity","Evaluation-conditioned training outdoes direct fitting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000209,"raw_usage":{"total_tokens":1429,"prompt_tokens":988,"completion_tokens":441,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":604,"completion_tokens_details":{"reasoning_tokens":371}},"tokens_in":604,"tokens_out":441,"duration_ms":4329,"temperature":1.0,"reasoning_tokens":371,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:11:35.929025+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train ECT exactly as in the paper, then deploy with a label naming a stronger evaluator that was never used in any training example and that no training example's reward is compatible with. If behavior under that novel label resembles behavior under training-range labels rather than moving toward stronger-evaluator behavior, the source-of-truth generalization claim fails. A cheaper version: degrade label granularity from the full fidelity scale to coarse strict-versus-permissive buckets; if ECT's improvement disappears, the mechanism depends on fine-grained fidelity semantics rather than mere conditioning.","supporting_citations":[{"cited_title":"Measuring political bias in Claude","cited_arxiv_id":null,"evidence_quote":"Supplies the paired-prompts political-bias evaluation, the held-out test pairs, and the rubric used to measure even-handedness, so it defines the data and metric for Experiment 1."},{"cited_title":"Eliciting latent knowledge: How to tell if your eyes deceive you","cited_arxiv_id":null,"evidence_quote":"Defines eliciting latent knowledge and the direct-translator versus human-simulator distinction that motivates conditioning on an ideal evaluator."}],"review_version":1}