{"id":"986b4ac7-e60a-472b-8b98-717ad2307ebf","arxiv_id":"2505.12843","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"FiMi-RM reduces length bias in RLHF reward models by fitting a nonlinear length-reward curve and training the reward model to be uncorrelated with that fitted curve.","lead":"A new three-stage method, FiMi-RM, trains a reward model, fits a curve that predicts reward scores from response length alone, then retrains the reward model to ignore that curve. This reduces the known bias toward longer answers, with small reported gains in length-controlled win rate for DPO and Best-of-N, but the paper releases no code.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Debiasing loss Eq. (9) enforces zero correlation with the fitted curve, not with length itself; a nonlinear fitted curve can be uncorrelated with length while length bias remains.","rationale":"The reader's weakest assumption identifies essentially the same gap: the fitting model sees only length, and zero Pearson correlation with one fitted curve does not imply independence from length. My reading sharpens this into a concrete objective mismatch at Eq. (9): the debiasing loss optimizes rho(r, \\hat r_detach), not rho(r, len), and because \\hat r is an arbitrary nonlinear function of length, the two targets can diverge sharply. This is a structural limitation of the method, not merely a missing evaluation. The paper's own Limitations section acknowledges that some length preference may be genuine, which means the fitted marginal relation can include true signal as well as bias. The empirical results are consistent with partial debiasing, but they do not verify the central mechanistic claim that length and reward are decoupled. This does not contradict the reader's CONDITIONAL verdict; it reinforces the condition under which the method would be acceptable, namely that the surrogate correlation objective actually removes length dependence in practice. A direct test comparing the proposed loss against a loss that penalizes rho(r, len) directly would settle whether the concern lands. No independent verification of the code or data is available, so the concern cannot be dismissed from the paper alone.","tokens_in":12239,"tokens_out":5107,"duration_ms":59118,"concrete_test":"Run a controlled experiment on the released or reproduced 7B checkpoint: (1) compute Pearson rho(r, len) and a nonparametric dependence measure (e.g., HSIC or binned conditional-mean deviation) on the held-out set for Vanilla RM, ODIN, and FiMi-RM; (2) replace Eq. (9) with L_direct = |rho(r, len)| + L_BT (or an HSIC penalty) and retrain; (3) compare rho(r, len), LC-WR, and output length. If FiMi-RM's rho(r, len) is not near zero, or if the direct-penalty variant achieves materially lower rho(r,len) with similar or better LC-WR, then optimizing Eq. (9) is not sufficient to support the 'decoupling length from reward' claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that FiMi-RM decouples reward from response length. The load-bearing step is Eq. (9): L'_pearson = |rho(r, \\hat r_detach)|, where \\hat r = modelf(len(y)). Minimizing this only forces r to be uncorrelated with one scalar function of length, not with length itself. Since \\hat r is a nonlinear map, zero Pearson correlation with \\hat r does not imply zero (or even reduced) correlation with len. Example: if \\hat r(len)=sin(len) and r(len)=len over a batch spanning a period, rho(r,\\hat r) approx 0 while rho(r,len) approx 1. In this paper's setting the fitted curve is monotonic with saturation, so the constraint is not vacuous, but it is still a single linear constraint in function space; r can retain linear, quadratic, or other length dependence that is orthogonal to \\hat r. The paper never reports rho(r,len) after debiasing, only binned scatter plots, so the claimed decoupling is not directly verified. A second facet of the same assumption: \\hat r is trained to approximate the marginal relation E[r|len] of a warm-up RM trained on human preferences. That marginal relation mixes genuine length preference with content preferences correlated with length; the Limitations section concedes that some length-preference may be genuine. Removing projection onto \\hat r can therefore remove true signal, not only bias. For the central claim to hold, the fitted curve must span all undesired length dependence and only that, which is neither guaranteed by the architecture nor tested.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FiMi-RM, a three-stage method to mitigate length bias in RLHF reward models. Stage 1 trains a standard Bradley-Terry reward model, deliberately retaining length bias. Stage 2 fits a lightweight model f(len(y)) — a sinusoidal length encoding followed by a two-layer ResNet and linear head — to the reward model's output using a combination of Pearson correlation and MSE losses, producing a predicted bias score. Stage 3 debiases the reward model by training with L'_pearson = |rho(r, hat_r_detach)| plus the original BT loss, alternating between fitting-model and reward-model updates. The authors evaluate on Anthropic HH data with Qwen2.5-1.5B and Qwen2.5-7B, reporting accuracy on C-longer/R-longer subsets, length-reward scatter plots, length-controlled Alpaca-Eval win rates under BoN and DPO, and length distributions of selected responses. They also report a fitted three-phase length-reward pattern: strongly linear for short responses, sublinear for medium lengths, and noise-like for long lengths.","tokens_in":12535,"tokens_out":2539,"duration_ms":29911,"significance":"If the central claim holds, the paper offers a practical, lightweight alternative to linear length-penalty methods and to ODIN's two-head architecture, with the potential advantage of capturing nonlinear length-reward relations. The use of length-controlled Alpaca-Eval as an external evaluation is a genuine strength, as it provides a check that is not defined by the method's own fitted curve. The balanced accuracy on C-longer and R-longer subsets is also a useful diagnostic. However, the significance is tempered by two unresolved issues: the debiasing objective in Eq. (9) only enforces zero Pearson correlation with one fitted scalar function of length, not independence from length, and the fitted curve is learned from the same reward model that is later debiased, so the reported three-phase pattern is not independently validated. The absence of confidence intervals and of an ablation against a linear version of the same fitting framework further limits the strength of the empirical claims.","major_comments":[{"comment":"The debiasing loss L'_pearson = |rho(r, hat_r_detach)| only forces the reward model output to be uncorrelated with the specific fitted scalar hat_r = f(len(y)). Zero Pearson correlation with one nonlinear function of length does not imply zero or even reduced correlation with length itself; for example, a reward that is linear in length can be nearly uncorrelated with a saturating or sinusoidal fitted curve over some ranges. The paper never reports rho(r, len) after debiasing, and Figure 2 shows only binned scatter plots. Since the central claim is that FiMi-RM decouples reward from length, the authors should directly report the post-debiasing Pearson correlation between r and len, or an equivalent independence test, to verify the mechanism.","section":"§3.3, Eq. (9)"},{"comment":"The fitted curve is trained on the output of the same warm-up reward model that is later debiased, and it approximates the marginal relation E[r | len]. That marginal relation mixes genuine length preference with content preferences that correlate with length; the Limitations section itself concedes that some length preference may be genuine. Consequently, removing the projection onto hat_r can remove true signal rather than only bias. The downstream LC-WR evaluation is external and partially addresses this, but the paper should provide additional evidence that the fitted curve corresponds to bias rather than to legitimate preference, for example by evaluating on a dataset with explicit length annotations or by comparing against a content-controlled baseline.","section":"§3.2, Fig. 5; Limitations"},{"comment":"The reported LC-WR and WR numbers are point estimates without confidence intervals, significance tests, or multiple seeds. Several differences are small (e.g., DPO 1.5B LC-WR 73.84 vs. 73.22; BoN 1.5B LC-WR 74.83 vs. 73.72; DPO 7B LC-WR 70.19 vs. 68.17), so it is unclear whether the improvements are statistically reliable. Additionally, the paper's key claim that nonlinear fitting is superior to linear debiasing is not directly supported: there is no ablation that replaces the ResNet fitting model with a linear function of length while keeping all other components unchanged. Such an ablation is necessary to establish that the improved LC-WR comes from the nonlinear structure rather than from the extra training signal or the decorrelation objective alone.","section":"Tables 2 and 3; §4.2"},{"comment":"The alternating training schedule in Eq. (10)-(11) is a central design choice but is not analyzed. The period a = 8 is fixed, and the paper does not report sensitivity to this hyperparameter, nor whether convergence of the reward model or the fitting model is affected by the alternation. At minimum, the authors should show that the results are stable for reasonable values of a, because the decorrelation effect depends on the fitting model remaining aligned with the current reward model during the debiasing stage.","section":"§3.3, Eq. (8)-(11)"}],"minor_comments":[{"comment":"There are several grammatical and typographical errors, including 'these approaches either mitigate bias without characterizing the bias form' (missing comma and subject-verb agreement), 'A part of existing approaches alleviate length bias' (number agreement), and 'does not characterizing the bias form' in §2.","section":"Abstract and §1"},{"comment":"The caption states that the average curve is 'more parallel to the x-axis' for the proposed method, but parallelism is not quantified. Reporting the slope or correlation of the binned averages would make the visual claim precise.","section":"Figure 2"},{"comment":"The table would benefit from explicit counts or percentages for the C-longer and R-longer subsets in the caption or text; the text mentions 58% and 40% but the table itself does not show these numbers, making the accuracy comparison harder to interpret.","section":"Table 1"},{"comment":"The DPO objective is written with beta in the logarithm, but the authors do not state the value of beta used in experiments or whether it differs between the two model sizes; this is needed for reproducibility.","section":"§4.1, Eq. (13)"},{"comment":"The step numbers in Figure 5 are described in the text but the training step at which the final debiased reward model is evaluated is not specified; the paper should clarify how the fitted curve shown at step 500 relates to the curve used in the debiasing stage.","section":"Figure 5"},{"comment":"The conclusion and limitations sections are somewhat brief and do not discuss the lack of a direct test of the decorrelation mechanism, which is the main theoretical assumption of the method; a short paragraph acknowledging this and pointing to future work would improve the paper.","section":"§5 and Limitations"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a relevant problem and the external LC-WR evaluation is a good choice, but the central mechanism — that minimizing correlation with a self-fitted length-only curve removes length bias — is not directly verified. The missing linear ablation and the lack of confidence intervals further weaken the empirical support. I believe the claims are defensible after additional experiments, so I recommend major revision rather than rejection. I would also encourage the authors to make clear that the three-phase pattern is a property of the fitted model, not an independently established property of the reward model's bias."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a serious referee. The most useful thing here is the clean extension of ODIN: instead of assuming a linear length-reward relation, fit a lightweight ResNet to the warm-up reward model's marginal length-reward curve, then train the reward model to be uncorrelated with the fitted curve. The three-stage framing is sensible, and the downstream evals (Alpaca-Eval LC-WR, BoN, DPO relabeling) are the right kind of evidence. Credit where it is earned: the balanced C-longer/R-longer accuracy in Table 1 is a direct check that the debiasing does what it claims, and the three-phase fitted curve is a useful descriptive observation. The Limitations section is honest, conceding that some length preference may be genuine.\n\nThe main soft spot is exactly the one in the stress-test note. Equation (9) minimizes |rho(r, hat_r_detach)| where hat_r is the fitted nonlinear function of length. Zero correlation with that scalar function does not imply zero correlation with length itself. The fitted curve appears monotone saturating, so the constraint is not vacuous, but it is still one linear functional constraint; residual length dependence orthogonal to the fitted curve is untouched. The paper never reports rho(r, len) after debiasing, only binned scatter plots. That is the missing direct test. Adding rho(r, len), and ideally a residual plot of reward versus length after conditioning on the fitted curve, would substantially strengthen the claim.\n\nThe secondary issues are real but addressable. There are no confidence intervals or significance tests on Tables 2 and 3; differences of 1-3 LC-WR points could be noise. There is no ablation against a linear fitting model with the same two-stage procedure, so the nonlinearity motivation is not directly supported. No code, data splits, or seeds are released, so the numbers cannot be independently checked. The circularity worry is milder than it first looks: the fit is deliberately describing the warm-up reward model, not a ground-truth bias label, and the downstream LC-WR evaluation is external. Still, the paper would be stronger if it showed the debiased reward model's residual length dependence rather than treating the fitted curve as the sole target.\n\nBottom line: this is a straightforward, useful contribution to a known failure mode, not a framework change. With direct residual-length reporting, a linear ablation, and significance checks, it would be a solid paper. I would send it to a serious referee as is, with those asks explicit.","headline":"A useful incremental extension of ODIN that fits a nonlinear length-reward curve; the main gap is that the debiasing loss targets correlation with the fitted curve, not with length itself, and the paper never reports the direct residual correlation.","tokens_in":13108,"tokens_out":2048,"would_cite":true,"duration_ms":21683,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Length bias in reward models is a learnable non-linear curve, and removing its correlation with the reward score improves alignment outcomes.","keywords":["length bias","reward hacking","reward model","RLHF","debiasing","non-linear fitting","Direct Preference Optimization","Best-of-N"],"falsifier":"Compare the residual reward after subtracting the fitted curve against length on a held-out set using Spearman rank correlation or an estimate of mutual information; if substantial length dependence remains, zero Pearson correlation with one fitted curve has not removed length bias.","tokens_in":11958,"feed_emoji":"📏","tokens_out":9498,"duration_ms":85755,"temperature":0.7,"pith_summary":"This paper proposes FiMi-RM, a three-stage procedure that treats length bias in reward models as a learnable non-linear function rather than a constant or linear penalty. The authors train a standard reward model, fit a lightweight model that maps only response length to the reward score, and then retrain the reward model while penalizing the Pearson correlation between its score and that fitted curve. When the debiased reward model is used in Direct Preference Optimization and Best-of-N, they report higher length-controlled win rate and reduced verbosity while keeping preference-modeling ability, at two model scales. They also characterize the fitted bias as strongly linear for short responses, decelerating for medium-length responses, and statistically indistinguishable from noise for very long responses.","feed_headline":"One fitted curve exposes reward models' length bias","feed_subtitle":"Debiasing against that curve lifts length-controlled win rate in DPO and Best-of-N while cutting verbosity.","key_machinery":"The load-bearing object is the fitting model $f(\\mathrm{len}(y)) = W_{\\mathrm{reg}}\\cdot\\mathrm{ResNet}(\\mathrm{LE}(\\mathrm{len}(y))) + b_{\\mathrm{reg}}$, where $\\mathrm{LE}$ is a sinusoidal length encoding modeled on positional encoding, and the ResNet has two residual layers. The model is trained only on length-reward pairs from the warm-up reward model, with a loss that maximizes Pearson correlation and minimizes mean squared error between the fitted scalar and the detached reward. That fitted scalar defines what the paper counts as the length-bias component; the debiasing stage then trains the reward model to have zero Pearson correlation with this component while preserving Bradley-Terry preference accuracy.","core_discovery":"The paper's central claim is that length bias in a reward model can be isolated by a lightweight fitting model $f$ that takes only the response length $\\mathrm{len}(y)$ as input, and that removing the correlation with $f$'s prediction debiases the reward without destroying preference learning. Concretely, after a warm-up reward model is trained with the Bradley-Terry loss $\\mathcal{L}_{BT}$, the fitting model is trained with $\\mathcal{L}_{\\mathrm{fit}} = -|\\rho(r_{\\mathrm{detach}}, \\hat{r})| + \\mathcal{L}_{\\mathrm{mse}}$, where $\\rho$ is the Pearson correlation coefficient. Then the reward model is retrained with $\\mathcal{L}_{\\mathrm{debiased}} = |\\rho(r, \\hat{r}_{\\mathrm{detach}})| + \\mathcal{L}_{BT}$. The authors report that this produces near-balanced accuracy between preference pairs where the chosen response is longer and pairs where the rejected response is longer, flattens the length-reward scatter, and raises length-controlled win rate under Direct Preference Optimization and Best-of-N at two model scales. The fitted curve reveals three phases: strong linearity below about 100 tokens, a decelerating upward trend from 100 to 200 tokens, and behavior indistinguishable from noise beyond 200 tokens.","pith_inferences":["Inference: the fitted three-phase curve suggests a cheap piecewise-linear penalty, steep below 100 tokens, shallow from 100 to 200, and flat beyond 200, could approximate the full fitting model in deployment.","Inference: because zero Pearson correlation with one fitted curve does not imply statistical independence from length, residual higher-order length dependence may remain; a mutual-information version of the penalty would test whether this mechanism is complete.","Inference: the method may remove genuine length preference when users explicitly ask for detailed answers; conditioning the fitting model on the prompt or on user instructions would turn the bias component into a context-dependent quantity.","Inference: the same fit-then-decorrelate recipe could be applied to other reward-hacking signals such as formatting artifacts, by replacing the length encoder with an encoder of those surface features."],"forward_implications":["Debiasing a reward model with a learned non-linear bias curve yields higher length-controlled win rate than both the vanilla reward model and a linear-assumption baseline under Direct Preference Optimization and Best-of-N.","The three-phase bias curve implies that a single linear length penalty is miscalibrated, under-penalizing short responses and over-penalizing long ones.","Downstream policies trained with the debiased reward model produce shorter and more balanced outputs while retaining or improving win rate, reducing verbosity-driven reward hacking.","Because the method works at two model scales with only mild overall accuracy loss on preference pairs, the fitted-curve correction can be added without reworking the rest of the alignment pipeline."],"supporting_citations":[{"why":"Supplies the Bradley-Terry pairwise preference model used in the warm-up and debiasing losses.","marker":"[3]"},{"why":"Supplies the ResNet architecture used inside the length-bias fitting model.","marker":"[17]"},{"why":"Defines the Pearson correlation coefficient used in both the fitting loss and the debiasing penalty.","marker":"[30]"},{"why":"Documents length bias in RLHF and the linear length-penalty baseline this work extends.","marker":"[41]"},{"why":"Provides the dual-headed baseline that assumes length and quality scoring can be separated; the paper compares against it.","marker":"[5]"},{"why":"Supplies the Direct Preference Optimization objective used as a downstream alignment evaluation.","marker":"[32]"},{"why":"Supplies the Best-of-N selection method used as a downstream alignment evaluation.","marker":"[16]"},{"why":"Supplies the length-controlled evaluation metric used for win-rate comparisons.","marker":"[10]"},{"why":"Supplies the positional-encoding idea adapted into the sinusoidal length encoding of the fitting model.","marker":"[46]"},{"why":"Supplies the human preference dataset used to train and evaluate the reward models.","marker":"[2]"}],"fun_headline_variants":["Fit a curve to kill length bias in RLHF","Curve fitting strips reward models of length bias","Debias reward models by fitting the length-reward curve","One fitted curve uncovers and fixes length bias","Length bias in RLHF? Fit it out"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the fitting model's output, computed from length alone, captures exactly the length-bias component of the reward model, so that forcing the reward model to be uncorrelated with that single fitted curve removes the bias without touching genuine quality signals.","fun_headline_variants_meta":{"raw":{"variants":["Fit a curve to kill length bias in RLHF","Curve fitting strips reward models of length bias","Debias reward models by fitting the length-reward curve","One fitted curve uncovers and fixes length bias","Length bias in RLHF? Fit it out"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000191,"raw_usage":{"total_tokens":1407,"prompt_tokens":1070,"completion_tokens":337,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":686,"completion_tokens_details":{"reasoning_tokens":263}},"tokens_in":686,"tokens_out":337,"duration_ms":3836,"temperature":1.0,"reasoning_tokens":263,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:25:35.240781+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare the residual reward after subtracting the fitted curve against length on a held-out set using Spearman rank correlation or an estimate of mutual information; if substantial length dependence remains, zero Pearson correlation with one fitted curve has not removed length bias.","supporting_citations":[{"cited_title":"Deep residual learning for im- age recognition","cited_arxiv_id":null,"evidence_quote":"Supplies the ResNet architecture used inside the length-bias fitting model."},{"cited_title":"Notes on regression and inheritance in the case of two parents.Proceedings of the Royal Society of London, 58:240–242, 1895","cited_arxiv_id":null,"evidence_quote":"Defines the Pearson correlation coefficient used in both the fitting loss and the debiasing penalty."},{"cited_title":"ODIN: Disentangled reward mitigates hacking in RLHF","cited_arxiv_id":null,"evidence_quote":"Provides the dual-headed baseline that assumes length and quality scoring can be separated; the paper compares against it."},{"cited_title":"Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022","cited_arxiv_id":null,"evidence_quote":"Supplies the human preference dataset used to train and evaluate the reward models."}],"review_version":1}