{"id":"481b0aed-25b8-43bb-aa28-24cdf5db74b3","arxiv_id":"2506.16507","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Crome trains reward models on LLM-generated causal and neutral augmentations, improving RewardBench accuracy by up to 5.4% and robustness to spurious transformations.","lead":"The paper introduces Crome, a training method that makes reward models for AI assistants more robust by adding counterfactual examples along quality attributes and tie-labeled examples under irrelevant questions. It reports higher accuracy on RewardBench and better best-of-N selection on safety and reasoning tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim depends on unverified oracle counterfactual fidelity: if LLM edits move off-target causal attributes or neutral ties preserve imperfectly, the augmented labels teach incorrect sensitivity/invariance and the reported gains may not transfer.","rationale":"The reader's weakest assumption is the fidelity of LLM-generated counterfactuals, and I agree that this is the correct load-bearing point. The empirical headline—up to 5.4% average and 13.2%/7.2% category gains on RewardBench—can only be attributed to causal disentanglement if the augmented labels are faithful to the intended attribute interventions. The paper's own Appendix A limitation admits the premise is imperfect, and the LLM-based verification (Appendix H.3/K.6) is not shown to be independent of the generator, so 'Pass' verdicts may be self-confirming. I also note an internal inconsistency: the formal theorem (Appendix I, Assumption 1.2) assumes spurious attributes are non-descendants of causal attributes, while the main text's causal graph and Appendix G explicitly model causal edits dragging spurious attributes (SP2); this makes the theoretical support inapplicable to the mechanism IQN is designed to address. That inconsistency does not by itself falsify the empirical claim, but it removes the formal backstop and makes the oracle-fidelity dependence more acute. The Gemma-3 oracle experiment (Table 4) shows some robustness to oracle choice but does not measure attribute isolation, so it does not resolve the concern. A retraining experiment on independently verified augmentations would settle whether the reported robustness is a genuine causal-invariance effect or an artifact of noisy labels and increased data volume. Since this is an unresolved empirical question, I would keep the reader's CONDITIONAL verdict: the claim is plausible and well-evidenced in its current form, but it should be conditioned on release of the augmentation pipeline/data and on an independent fidelity check.","tokens_in":44715,"tokens_out":7592,"duration_ms":77154,"concrete_test":"Randomly sample 200 causal and 200 neutral augmented pairs from the Daug used in the Gemma-2-9B PairPM run (before filtering). Have an independent judge—human annotators or a different frontier LLM that did not generate or verify the data—score each causal pair for whether exactly the targeted causal attribute changed with no other causal attribute change, and each neutral pair for whether causal content is preserved under the irrelevant query. Retrain the same Crome recipe on the subset that passes independent verification and compare average RewardBench and reWordBench accuracy to Table 2/Figure 5.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central assertion is that training on LLM-generated causal augmentations and irrelevant-query neutrals yields an RM sensitive to true quality drivers and invariant to unknown spurious attributes. The load-bearing premise is that the oracle LLM (Gemini 2.0 Flash) produces counterfactuals that change exactly one causal attribute and neutrals that preserve causal content. The paper itself concedes this premise is fragile: Appendix A states 'ensuring perfect attribute isolation in causal augmentations or complete causal content preservation in neutral pairs is challenging,' and Section 3.3/Remark 1 calls the counterfactuals 'approximations.' The verification step (Appendix H.3, K.6) is also LLM-based and is not reported to use an independent judge; if the generator and verifier are the same model, systematic off-target edits can pass verification. If causal augmentations co-move other causal attributes, the preference label 'upgraded > original' teaches sensitivity to a mixture of attributes, not the targeted one. If neutral pairs accidentally leave causal differences relevant to the (irrelevant) query, the tie label teaches the model to suppress genuine quality signals. Under either failure, the improved RewardBench/reWordBench numbers could reflect extra training data or generic regularization rather than causal disentanglement, and the claimed robustness to unforeseen spurious cues would not transfer. The theory (Appendix I, Assumption 1.2) assumes spurious variables are non-descendants of causal variables, which contradicts the paper's own Figure 2/Appendix G discussion that causal edits can drag spurious attributes (SP2); this makes the formal argument inapplicable to the very mechanism IQN is designed to handle, leaving the empirical claim as the sole support.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Crome, a data augmentation framework for training reward models against reward hacking. Crome prompts an oracle LLM (Gemini 2.0 Flash) to (i) identify five causal quality attributes for a query-answer pair; (ii) generate counterfactual upgraded/degraded answers intended to isolate one causal attribute (causal augmentations); and (iii) build tie-labeled neutral pairs by pairing answers with irrelevant queries (IQN) or by causally aligning answers (CAN). The RM is trained on original preferences plus filtered augmentations using a composite preference-plus-tie loss (Eq. 2). Empirical sections report RewardBench gains up to 5.4%, robustness gains on a reimplemented reWordBench, and Best-of-N improvements on RewardBench, WildGuardTest, and GSM8K. A theoretical section (Appendix I) analyzes an idealized boolean quadratic setting and claims that L1-constrained regression on causal augmentations recovers true causal coefficients with error depending weakly on the spurious dimension.","tokens_in":44938,"tokens_out":7064,"duration_ms":72934,"significance":"Conditional on the empirical claims and on the availability of reproducible artifacts, Crome would be a practically useful, spurious-factor-agnostic augmentation recipe for reward modeling: it requires no prior specification of spurious attributes, gives consistent gains across three base models and two RM formulations, includes multiple-run variance for the 9B model, and the paper is unusually honest about its limitations. The ablations comparing IQN, CAN, PARA, and causal-only variants and the oracle-LLM robustness check with Gemma-3-27B-IT are definite strengths. However, the central claim that augmentations isolate causal attributes and preserve causal content in neutrals rests on oracle counterfactual fidelity that is not independently verified, and the printed training loss in Eq. (2) appears to have the wrong sign for tie pairs. The robustness results additionally rely on a reimplemented benchmark, and the Best-of-N tables lack error bars. The current evidence therefore does not yet justify acceptance, although the identified issues appear fixable within the scope of the paper.","major_comments":[{"comment":"As printed, the neutral tie term has the wrong sign. The second term in Eq. (2) is -lambda * sum [ -1/2 (log sigma(Delta12) + log sigma(-Delta12)) ] = +lambda/2 * sum (log sigma(Delta12) + log sigma(-Delta12)). Since log sigma(Delta) <= 0, this term is negative and is minimized by making the tie pair's scores as different as possible, which is the opposite of the stated invariance objective. Appendix I.1 instead defines Ltie = -E[-1/2(log sigma + log sigma(-))] = 1/2 E(log sigma + log sigma(-)) and adds lambda * Ltie as a penalty, so the definition in Eq. (2) is internally inconsistent with the definition used in the theoretical section. Please fix the sign and confirm that all reported results were obtained with the intended tie loss, or correct the presentation and rerun affected experiments.","section":"§4.2, Eq. (2) and Appendix H.4 Eq. (3)"},{"comment":"The load-bearing premise is that LLM-generated counterfactuals modify exactly one causal attribute and that neutral rewrites preserve causal content. The paper itself states in Appendix A that \"ensuring perfect attribute isolation in causal augmentations or complete causal content preservation in neutral pairs is challenging.\" The verification step in Appendix K.6 is an LLM-based evaluator, but the paper does not report whether the verifier is the same model as the generator, and it provides no human-annotated error rates or per-attribute off-target statistics. Without such an audit, off-target edits or imperfect neutrals could teach the reward model a mixture of attributes instead of the targeted one, and the observed gains could be attributable to extra training data or generic regularization. Please add an independent (human or differently-modeled) audit of a random sample of causal and neutral pairs, and report off-target rates per attribute and per neutral strategy.","section":"§3.3, Remark 1, Appendix A, Appendix K.6"},{"comment":"The theoretical analysis assumes that all spurious variables are non-descendants of all causal variables (Assumption 1.2), so a causal intervention leaves Delta S_j = 0 for all spurious features. This is what produces the 4/k column-incoherence bound in Theorem 2. Yet Appendix G and Figure 19 explicitly model a subset SP2(A) of spurious attributes that co-move when causal attributes change, and the paper motivates IQN precisely as a way to provide invariance to SP2. The theorem therefore does not cover the mechanism that the paper identifies as central. Please extend the analysis to allow SP2 to be a descendant of C, or state explicitly that Theorem 2 applies only to the idealized case without SP2 and that the empirical SP2-related gains are supported only by experiments.","section":"Appendix I.2, Assumption 1.2 vs §3.2, Figure 19, Appendix G"},{"comment":"The reWordBench results are computed on the authors' independently reproduced version of the benchmark, not on the official release. The reproduction uses different generation models (e.g., gpt-4o for paraphrasing and back-translation) than those described for the original reWordBench, and the paper reports no per-transformation agreement analysis and does not release the reproduced data. Because the robustness claims in Figures 5, 6, and 9 hinge on this benchmark, please release the reimplementation and code, and if feasible compare a subset against the official reWordBench to establish comparability.","section":"Appendix D, reWordBench reproduction; Figures 5-6"},{"comment":"The Best-of-N results are reported without error bars or statistical significance tests, even though the Gemma-2-9B setting admits multiple training runs as done for RewardBench in Table 5. The ASR reductions in Figure 7 range from about 0.3 to 2.7 percentage points, and the GSM8K gaps in Figure 8 are shown as point curves only. Without repeated Best-of-N runs or bootstrap confidence intervals, the claim of \"consistent gains across increasing N\" in the abstract and Section 6.2 is not fully supported. Please report means and variances over at least three seeds for the BoN evaluations, or provide intervals on the win-rate tables.","section":"Table 3, Figures 7-8, Appendix C.1"}],"minor_comments":[{"comment":"The abstract states average accuracy gains \"up to 5.4%\" and Safety/Reasoning gains of 13.2%/7.2%, while the introduction says \"up to 4.5%\" with substantial Safety and Reasoning gains. These should be aligned with the numbers in Table 2.","section":"Abstract and §1"},{"comment":"The loss hyperparameter lambda is fixed at 1 and the filtering threshold tau is fixed at 0.2, with no sensitivity analysis. Since the tie loss is the mechanism for spurious invariance, please report at least a small sweep over lambda and tau, or state why these values are robust.","section":"§4.2 and Appendix H.3"},{"comment":"The qualitative example contains stray newlines, misrendered JSON, and irregular indentation (e.g., the causal elements block and the chosen-response edits block). Please format this appendix as clean code blocks so that the example is readable.","section":"Appendix J"},{"comment":"Typo: \"mdoels\" should be \"models\" in the sentence describing base transformer architectures.","section":"Appendix E.5"},{"comment":"The caption states \"dim(C(A)) << dim(SP(A)) for all A\" but does not define the dimension of an attribute set; please clarify whether this refers to the number of attributes or to a latent dimension of the attribute representation.","section":"Figure 2 caption"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has a strong, well-scoped empirical core and the authors are transparent about the main limitations. The most urgent issues are the apparent sign error in the tie loss in Eq. (2) and the lack of independent verification of oracle counterfactual fidelity; both are fixable but are load-bearing for the central claim. I would also ask the editor to insist on release of the reimplemented reWordBench data and code, since the robustness claims currently rest on an unreleased, independently reconstructed benchmark."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: useful synthesis, not a new paradigm. The combination of attribute-isolating counterfactuals and irrelevant-query tie labels is new, and the empirical story is mostly convincing. I'd send it to review, but I'd push hard for reproducibility and for fixing the theory/mechanism mismatch.\n\nThe good parts: Crome trains reward models on LLM-generated upgrades/degradations along single causal rubrics, plus neutral pairs with irrelevant queries, without knowing spurious features. The evaluation spans three base models, two RM formulations, RewardBench, reWordBench, and best-of-N on safety and reasoning. The main RewardBench results have three runs with small variance, and gains concentrate in Safety and Reasoning, where reward hacking is most worrying. The ablation separating causal-only from causal-plus-IQN shows the neutral term does real work. I find the empirical claim believable.\n\nSoft spots are real but mostly not fatal. The load-bearing premise is that Gemini 2.0 Flash edits exactly one causal attribute and preserves causal content in neutrals. The paper concedes this is hard, and verification is done by the same kind of model, so systematic off-target edits could pass. That means the improved numbers could partly reflect more data or generic regularization rather than clean disentanglement. This should be tested with an independent judge and ideally human checks.\n\nMore annoying is the theory. Theorem 2 assumes spurious variables are non-descendants of causal variables, but Figure 2 and Appendix G explicitly describe spurious attributes (SP2) that move when causal attributes are intervened on. The formal analysis is therefore not applicable to the very mechanism IQN is designed to handle. The theorem may still be a useful idealized warm-up, but as written it overclaims.\n\nReproducibility is the biggest practical issue: no code or data, and reWordBench was reimplemented. Best-of-N win rates lack error bars. The abstract's 5.4% versus intro's 4.5% mismatch is minor but sloppy.\n\nThis paper is for RLHF practitioners who want a concrete augmentation recipe. A serious referee can handle it. I'd ask for code/data, error bars on BoN, and a rewrite of the theory section that honestly scopes its assumptions. Send to review, conditional on those.","headline":"A genuinely useful augmentation recipe for reward-model robustness, but the causal theory oversells its scope and the reproducibility is incomplete.","tokens_in":45632,"tokens_out":2229,"would_cite":true,"duration_ms":22819,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training a reward model on LLM-generated causal counterfactuals and tie-labeled neutral pairs makes it track genuine quality drivers and ignore unknown spurious attributes, improving RewardBench accuracy by up to 5.4% over baselines.","keywords":["reward hacking","reward model robustness","counterfactual data augmentation","spurious correlations","causal attributes","RLHF","irrelevant query neutrals","RewardBench"],"falsifier":"Evaluate a Crome-trained model on a test set where a genuinely novel cosmetic feature (for example, a fixed decorative prefix appended to every chosen answer) has been injected to correlate with preference. If the preference-accuracy drop is as large for Crome as for a standard reward model, the claimed invariance to unknown spurious attributes fails. A complementary check is to randomly flip a fraction of the causal augmentation labels during training: if the RewardBench gains survive the scrambling, the causal-sensitivity mechanism is not what carries the result.","tokens_in":44449,"feed_emoji":"🎯","tokens_out":13332,"duration_ms":106596,"temperature":0.7,"pith_summary":"Reward models used to align large language models with human feedback tend to reward superficial cues such as length and formatting, because standard preference training cannot separate those cues from genuine quality. Crome adds two kinds of synthetic training pairs: causal augmentations, where an oracle LLM rewrites an answer to upgrade or degrade a single quality attribute such as factuality, and neutral augmentations, where answer pairs are shown under an irrelevant query with a tie label. The paper claims this teaches a reward model sensitivity to true quality drivers and invariance to unknown spurious attributes, improving average RewardBench accuracy by up to 5.4% over baselines, with the largest gains in safety (up to 13.2%) and reasoning (up to 7.2%). If correct, Crome is a practical recipe for reducing reward hacking without knowing in advance which spurious features will be exploited.","feed_headline":"Counterfactual training cuts reward hacking, up to 5.4% on RewardBench","feed_subtitle":"Causal and neutral LLM-generated pairs teach reward models to value real quality signals over style and formatting.","key_machinery":"The load-bearing device is a causal graph in which the true reward $R^*$ depends only on the query $Q$ and causal attributes $C(A)$, while spurious attributes $SP(A)$ are unknown, high-dimensional, and merely correlated with preferences in the training data. Training combines a preference loss on original and causally augmented pairs (one attribute upgraded or degraded at a time, teaching sensitivity to each quality driver) with a neutral tie loss that forces the score difference toward zero on pairs shown under an irrelevant query, where the original causal signal is null and only spurious differences remain. This 'irrelevant query neutral' construction is what delivers invariance without ever naming or intervening on spurious factors; the composite objective is the preference loss plus $\\lambda$ times the tie loss, with $\\lambda = 1$ in the reported experiments.","core_discovery":"The paper's central claim is that a reward model trained on preference pairs plus LLM-written counterfactual pairs that differ along exactly one causal quality attribute, plus tie-labeled pairs of the same answers under an irrelevant query, learns to make its score depend on the stable causal attributes $C(A)$ and to be independent of the unknown, unstable spurious attributes $SP(A)$. The empirical statement is that Crome 'significantly outperforms standard baselines on RewardBench, improving average accuracy by up to 5.4% and achieving gains of up to 13.2% and 7.2% in specific categories,' with consistently smaller accuracy drops under meaning-preserving transformations on reWordBench and better Best-of-N selection across RewardBench, WildGuardTest, and GSM8K. A supporting theoretical result says that $\\ell^1$-constrained regression on causally augmented examples recovers the true reward coefficients with error depending mainly on the number of causal attributes $k$ and the number of augmentations $m$, with only a logarithmic dependence on the spurious dimension $\\ell$.","pith_inferences":["The irrelevant-query neutral is the most transferable idea in the paper: it turns unknown spurious attributes into provably irrelevant ones by swapping the conditioning query, a trick that could apply beyond reward models to any classifier or embedding that must ignore style.","If the mechanism claim is right, Crome's advantage should widen as test-time spurious variation increases; an explicit stress test that scales the number of injected cosmetic cues and measures the accuracy-drop gap over baselines would sharpen this prediction.","The paper's own future-work sketch, using causal rubrics to generate synthetic data for base-model training, is the natural frontier: the same upgrade, degrade, and tie machinery would teach the quality attributes themselves rather than only a reward model's sensitivity to them.","The theory ties recovery error to the causal dimension $k$, which suggests a testable knob: shrinking the rubric from five attributes to fewer should measurably degrade robustness, matching the predicted trade-off."],"forward_implications":["Models trained with Crome keep their edge under out-of-distribution transformations: aggregate reWordBench accuracy rises by up to 9.1% over baselines, and the accuracy drop from RewardBench to reWordBench is the smallest among compared methods.","Best-of-N selection with Crome beats selection with either baseline at every N tested, and on WildGuardTest it lowers the attack-success rate by up to about 2.7% at N=32 without increasing refusals of benign prompts.","Because no spurious attribute is ever specified or intervened upon, the recipe covers a far wider class of spurious cues than methods that target known biases like length or formatting.","The theoretical result implies the learned reward coefficients concentrate on causal drivers, with recovery error only logarithmic in the number of unknown spurious attributes, so the method does not need to know in advance what to be invariant to.","Gains reproduce across base models (Gemma-2-2B, Gemma-2-9B-IT, Qwen2.5-7B), both pairwise-preference and Bradley-Terry formulations, and with a weaker oracle LLM, indicating that the augmentation design rather than oracle strength drives the improvement."],"supporting_citations":[{"why":"Supplies the main baseline (RRM, robust reward model training via non-contextual pairs) that Crome's gains are measured against, along with the training and filtering setup Crome inherits.","marker":"Liu et al. (2024)"},{"why":"Provides reWordBench, the meaning-preserving-transformation benchmark that measures spurious robustness, plus the Best-of-N and LLM-as-judge methodology Crome adopts.","marker":"Wu et al. (2025)"},{"why":"Provides RewardBench, the principal evaluation benchmark on which the headline 5.4% average gain and the safety and reasoning gains are reported.","marker":"Lambert et al. (2024)"},{"why":"Supplies UltraFeedback, the preference dataset from which all original pairs and Crome's augmentations are generated.","marker":"Cui et al. (2023)"},{"why":"The pairwise preference model underlying the reward-model training objective in Equation (1).","marker":"Bradley and Terry (1952)"},{"why":"Define and formalize reward hacking and reward gaming, the failure mode Crome is designed to mitigate.","marker":"Gao et al. (2023); Skalse et al. (2022)"},{"why":"Supplies the cone-condition machinery used in the proof of the recovery bound in Theorem 2.","marker":"Negahban et al. (2009)"},{"why":"The lineage for counterfactual data augmentation that Crome's LLM-rewrite approach extends to reward modeling.","marker":"Kaushik et al. (2019)"}],"fun_headline_variants":["Causal rubrics teach reward models to ignore style, boost accuracy 5.4%","Reward hacking tamed: LLM-crafted causal pairs boost RewardBench 5.4%","Crome: causal augmentations make reward models robust, up to +5.4%","Stop reward hacking: causal rubrics improve reward modeling by 5.4%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the oracle LLM's rewrites being faithful counterfactuals, changing exactly the intended quality attribute while leaving all other attributes, especially causal content, intact, and the paper's own limitations section concedes that perfect attribute isolation in causal augmentations and complete causal content preservation in neutral pairs are not guaranteed.","fun_headline_variants_meta":{"raw":{"variants":["Causal rubrics teach reward models to ignore style, boost accuracy 5.4%","Reward hacking tamed: LLM-crafted causal pairs boost RewardBench 5.4%","Crome: causal augmentations make reward models robust, up to +5.4%","Stop reward hacking: causal rubrics improve reward modeling by 5.4%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000529,"raw_usage":{"total_tokens":2611,"prompt_tokens":1069,"completion_tokens":1542,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":685,"completion_tokens_details":{"reasoning_tokens":1447}},"tokens_in":685,"tokens_out":1542,"duration_ms":11509,"temperature":1.0,"reasoning_tokens":1447,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:24:38.995608+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate a Crome-trained model on a test set where a genuinely novel cosmetic feature (for example, a fixed decorative prefix appended to every chosen answer) has been injected to correlate with preference. If the preference-accuracy drop is as large for Crome as for a standard reward model, the claimed invariance to unknown spurious attributes fails. A complementary check is to randomly flip a fraction of the causal augmentation labels during training: if the RewardBench gains survive the scrambling, the causal-sensitivity mechanism is not what carries the result.","supporting_citations":[{"cited_title":"non-contextuals","cited_arxiv_id":null,"evidence_quote":"Supplies the main baseline (RRM, robust reward model training via non-contextual pairs) that Crome's gains are measured against, along with the training and filtering setup Crome inherits."},{"cited_title":"We useGemma-2-9B-IT as the base model and sample N responses for each prompt in this set","cited_arxiv_id":null,"evidence_quote":"Provides RewardBench, the principal evaluation benchmark on which the headline 5.4% average gain and the safety and reasoning gains are reported."}],"review_version":2}