{"id":"0cc36411-eb10-4a4b-bfe2-eaea14d67436","arxiv_id":"2412.16475","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Under four structural conditions linking proxy and true preference policies, the true policy is a low-dimensional adapter of the proxy policy, reducing the number of true preference samples needed.","lead":"This paper proves that if a cheap proxy reward and an expensive true reward induce policies that share the same grouping of inputs and a low-dimensional response structure, then the true policy can be learned with far fewer true labels by first training on proxy data. It gives a theoretical sample-complexity bound and a two-stage adapter parameterization for preference learning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 5's sample-complexity bound is derived by an incorrect algebraic inversion of Eqs. (142)-(143); solving for n gives O(D/ε^2 log(D/ε)+ε^{-2}log(1/ω)), not the exponential-in-D expression in Eq. (144), so the claimed superexponential improvement is unsupported.","rationale":"I identify one concern as load-bearing: the proof of the sample-complexity theorem contains a concrete algebraic error. The paper's central claim is not merely that a factorization exists (Theorem 3) but that it yields a provable reduction in true-preference sample complexity from D′ to D. That reduction is quantified by Theorems 5 and 6, and Theorem 5's proof is where the D-dependence is established. The error at Eqs. (142)-(144) means the exponential-in-D bound is not derived from the paper's own concentration inequality; the correct inversion gives a much milder O(D/ε² log(D/ε) + ε^{-2} log(1/ω)) dependence. The same defect propagates to Theorem 6. Even setting aside the structural and Stage-1 issues, the quantitative consequence claimed in the abstract and introduction would not follow as written. This is therefore the single most load-bearing point. The reader's verdict REJECT is consistent with this concern; the reader's stated weakest assumption (exact Stage 1 recovery) is a different gap, though the reader's rationale also cites Eqs. (143)-(144). I would not change the reader's verdict, so I mark the verdict as unchanged. The proposed test is purely algebraic and can be run in a few lines.","tokens_in":23928,"tokens_out":16389,"duration_ms":151565,"concrete_test":"Re-derive n(ε,ω) from Eq. (142) exactly: substitute κ=ε/48 and δ=ε/(48Lφ‖Θ‖p L̄π) into the covering-number bound, set the covering bound times exp(-nε²/(8C²)) equal to ω, and take logarithms to solve for n. Compare the resulting expression with Eq. (144). If the correct expression is n = (8C²/ε²)( D ln(96Lφ‖Θ‖p√D/ε) + D ln(96Lφ‖Θ‖p L̄π√D/ε) - ln ω ), then the exponential factor in Eq. (144) is an algebraic artifact. Repeat the same inversion for Theorem 6. This check settles whether the D-versus-D′ sample-complexity improvement is exponential in dimension or merely linear-in-dimension with log factors.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Appendix C.1, after bounding the covering number, the paper writes (Eq. 143) ω = exp(-nε²/(8C²)) (96Lφ‖Θ‖p√D/ε)^D (96Lφ‖Θ‖p L̄π√D/ε)^D. Taking logarithms and solving for n gives n = (8C²/ε²)( D ln(96Lφ‖Θ‖p√D/ε) + D ln(96Lφ‖Θ‖p L̄π√D/ε) - ln ω ), i.e. n = O(D/ε² log(D/ε) + ε^{-2} log(1/ω)). The factor (96Lφ‖Θ‖p L̄π√D/ε)^D cannot be pulled out of the logarithm; it enters only through the log of the covering number. The expression in Eq. (144), which reports n = Ω(D/ε² (96Lφ‖Θ‖p L̄π√D/ε)^D log(...) - log ω), is therefore not a consequence of the preceding algebra. The same error appears in the proof of Theorem 6 (Eqs. 154-155). This is load-bearing because the central claim that proxy pre-training reduces the true-preference sample complexity from a D′-dimensional dependence to a D-dimensional dependence is carried by Theorems 5 and 6. After correction, the paper's own concentration bound yields at best a factor D versus D′ times logarithmic terms, not the stated exponential-in-D sample complexity. The 'consequently' in the paper's strongest claim is thus not justified as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies when abundant proxy preference data can provably reduce the number of true preference samples needed to learn a target policy. It introduces four sufficient conditions (shared level sets, image inclusion, finite-dimensional encoding, and Lipschitz similarity) and proves a factorization theorem: under these conditions the true policy can be written as the proxy decoder and encoder composed with a low-dimensional Lipschitz adapter. The authors then propose a two-stage procedure (learn the proxy components with proxy data, learn the adapter with true data) and state sample-complexity bounds, Theorems 5 and 6, claiming that the true-preference sample complexity scales with the small intrinsic dimension D rather than the large embedding dimension D′, yielding a 'superexponential' improvement.","tokens_in":24289,"tokens_out":6848,"duration_ms":64494,"significance":"If the claimed results were correct, this would be a constructive contribution to RLHF theory: it gives explicit sufficient conditions under which proxy data can be safely reused, and a concrete parametrisation for LLMs that realizes the improvement. The paper is clearly organized, the problem is well motivated, and the factorization idea (Conditions 1-4 leading to a shared decoder/encoder with a low-dimensional adapter) is attractive. However, the central sample-complexity statements are not supported by the proofs: the algebra in Theorems 5 and 6 is wrong, and the claimed superexponential improvement disappears after correction. The paper also does not analyze the sample complexity of the first stage, so the total two-stage guarantee is incomplete. These issues are load-bearing rather than cosmetic.","major_comments":[{"comment":"The inversion of the concentration bound is incorrect. Eq. (143) is ω = exp(-nε²/(8C²)) A^D B^D with A = 96Lφ‖Θ‖p√D/ε and B = 96Lφ‖Θ‖p L̄π√D/ε. Taking logarithms and solving for n gives n = (8C²/ε²)(D log A + D log B - log ω), i.e. n = O(D/ε² log(D/ε) + ε^{-2} log(1/ω)). The factor B^D cannot be pulled out of the logarithm; it contributes only D log B. Therefore Eq. (144), which states n = Ω(D/ε² B^D log A - log ω), is not a consequence of the preceding equations. The same algebraic error appears in the proof of Theorem 6 in Eqs. (154)-(155). Because Theorems 5 and 6 are the statements that carry the central sample-complexity claim, the paper's advertised superexponential improvement over D′ is unsupported as written. After correction, the comparison would be between roughly O(D log D/ε²) and O(D′ log D′/ε²), a much weaker statement than the one made in Section 3.","section":"Section 5, Theorem 5, Appendix C.1, Eqs. (142)-(144)"},{"comment":"The analysis assumes 'given that we have identified the true φ~, Θ~ and τ~° from the proxy dataset in the first stage.' The paper provides no bound on the number of proxy samples needed to identify these components, nor an analysis of how approximation error in Stage 1 propagates into the Stage 2 bound. This is an extra, unquantified condition beyond Conditions 1-4. Since the central claim is about the total sample complexity of learning π† with proxy data, the stated improvement is not established for the full two-stage procedure. The paper should either prove a Stage 1 sample-complexity bound or explicitly frame the results as conditional on exact identification.","section":"Section 5, first paragraph"},{"comment":"The Stage 2 objective is not well defined as written. The adapter ¯π†θ is defined as a map ΔD → ΔD, but Eq. (13) evaluates ¯π†θ(yw,i | xi) and ¯π†θ(yl,i | xi), treating it as a policy over completions. The DPO ratio should be based on the full model π†θ = φ~θ ∘ Θ~ ¯π†θ ∘ τ~°θ, i.e. β log π†θ(yw,i|xi)/πref(yw,i|xi) minus the corresponding term for yl,i. As written, the loss cannot be computed from the stated model, so the algorithm cannot be implemented without further clarification.","section":"Section 4.3, Eq. (13)"}],"minor_comments":[{"comment":"The proof claims PY = ⋃_{d=1}^∞ Δd, but this union contains only distributions with finite support. Since PY contains distributions with infinite support, the equality is false. The conclusion that the topological dimension is infinite may still be true, but the given proof does not establish it.","section":"Appendix A, Proposition 1"},{"comment":"The constants E(p,D) and E′(p,D′) are used in the covering-number bounds but are never defined, and the assertion E(p,D) ≤ 1 for simplices is stated without proof. The reader cannot verify the constants in Theorems 5 and 6.","section":"Appendix C.1"},{"comment":"The no-proxy baseline class is defined with Lipschitz constant Lφ‖Θ‖pL̄π, but these constants come from the proxy decomposition. The paper does not explain why a learner without proxy data would know or satisfy this constant, so the comparison is not fully grounded.","section":"Theorem 6"},{"comment":"The implicit reward is written as r(x,y) = β log π(yw | x)/πref(yw | x), but the argument on the left is y, not yw. This appears to be a typo.","section":"Section 2, Eq. (7)"},{"comment":"The statement 'to the best of our knowledge, we are the first to consider this setting' is difficult to verify and could be softened, especially since the related-work section does not discuss the possibility of prior theoretical treatments of proxy-data sample-complexity gains in preference learning.","section":"Section 7"}],"recommendation":"major_revision","confidential_remarks":"The algebraic error in Theorems 5 and 6 is serious because it invalidates the headline superexponential claim. However, the underlying idea is not destroyed: after correction there is still a dimension reduction from D′ to D in the sample-complexity expression. I recommend major revision rather than rejection, provided the authors fix the theorem statements, remove or recalibrate the superexponential language, and either bound or clearly condition on the Stage 1 error. The manuscript currently has no empirical evaluation; the authors state this is ongoing work, but the paper's framing implies more than the proofs deliver."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: the paper has a real idea — a set of sufficient conditions under which the true policy is a low-dimensional adapter of the proxy policy — but the main quantitative claim, the superexponential sample-complexity improvement, is not established by the proof. The algebraic step in Theorem 5's derivation is simply wrong: taking logs of Eq. (143) and solving for n gives n = O(D/ε² log(D/ε) + ε⁻² log(1/ω)), not the stated Ω(D/ε² (…)^D log(…)). The factor involving L̄π enters inside the logarithm, not as an exponent. The same error repeats in Theorem 6. This matters because the whole 'provably improved sample complexity' narrative rests on those theorems.\n\nWhat is genuinely new: the factorization result (Theorem 3) — under Conditions 1–4, π† = φ~ ∘ Θ~ π̄† ∘ τ~° — is derived, not assumed, and the parameterization that reuses the proxy's encoder/decoder is a plausible way to attack reward hacking with a small expert set. The paper is honest about the strength of the conditions and frames them as guidance for data collection. The writing is clear and the related work is handled fairly.\n\nThe soft spots, in order of severity. First, the algebraic inversion above; the stated exponential-in-D dependence is unsupported. Second, the Stage 2 loss in Eq. (13) is not a valid DPO loss for the composed policy: it compares β log(π̄†(y)/π_ref(y)), but the implicit reward of the actual policy uses β log(π†(y)/π_ref(y)) = β log(φ~(Θ~π̄†(τ~(x)))(y)/π_ref(y)). So even the learning procedure is not clearly justified. Third, the analysis assumes Stage 1 recovers φ~, Θ~, τ~° exactly, with no bound on proxy sample size or error propagation. Minor: the conditions are strong, but the paper says they are sufficient, not necessary, so that's acceptable; circularity is only partial.\n\nWho is this for: people working on theory of RLHF and reward hacking. The structural result is worth a serious referee. My recommendation: send it to review, but the authors need to fix the algebra, fix the Stage 2 loss, and either analyze Stage 1 error or weaken the claims. As written, the central quantitative conclusion should not be taken at face value.","headline":"Genuinely interesting factorization theorem, but the main sample-complexity claim rests on a wrong algebraic inversion and an invalid Stage 2 loss.","tokens_in":24829,"tokens_out":2329,"would_cite":false,"duration_ms":19397,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Proxy data can slash the sample cost of true preference learning.","keywords":["preference learning","reward hacking","proxy rewards","sample complexity","direct preference optimization","low-dimensional adapter","large language models","reward over-optimization"],"falsifier":"One concrete falsifier is a real or synthetic preference dataset containing two prompts that the proxy maps to the same response distribution but the true policy maps to different distributions; such level-set disagreement violates Condition 1, and if it is widespread the identity $\\pi^{\\dagger} = \\tilde{\\phi} \\circ \\tilde{\\Theta} \\bar{\\pi}^{\\dagger} \\circ \\tilde{\\tau}^{\\circ}$ cannot hold.","tokens_in":2153,"feed_emoji":"📉","tokens_out":6476,"duration_ms":93981,"temperature":0.7,"pith_summary":"The paper asks when cheap, abundant proxy preference data—say, ratings from a student doctor—can provably reduce how many expensive true/expert preference samples are needed to learn a ground-truth policy. It identifies four sufficient conditions on the proxy relative to the true policy: shared level sets, inclusion of the true policy's outputs in the proxy's outputs, a low-dimensional encoding of those outputs, and Lipschitz similarity between the two policies. Under these conditions the true policy equals the proxy's decoder and encoder wrapped around a low-dimensional adapter map on a $D$-simplex (Theorem 3). Consequently the number of true preference samples needed scales with the small intrinsic dimension $D$ rather than the high embedding dimension $D'$ (Theorems 5 and 6). If right, this turns reward hacking from a pure risk into a design question: collect proxy feedback that satisfies the conditions, reuse its learned structure, and fine-tune only a small adapter on expert data.","feed_headline":"Proxy data can slash the sample cost of true preference learning","feed_subtitle":"Under four structural conditions, true-policy learning needs samples scaled by dimension D, not D′.","key_machinery":"The machinery is a factored parametrization of policies: any policy $\\pi$ is written as $\\phi \\circ \\Theta \\circ \\tau$, where $\\tau$ maps prompts into a $D$-simplex $\\Delta^D$, $\\Theta$ is a linear map from the simplex to a convex polytope $V$, and $\\phi$ is a bi-Lipschitz decoder from $V$ to response distributions. The key identity (Theorem 3) states that if Conditions 1–4 hold, every decomposition of the proxy policy as $\\tilde{\\pi} = \\tilde{\\phi} \\circ \\tilde{\\Theta} \\tilde{\\tau}^{\\circ}$ can be reused to express the true policy as $\\pi^{\\dagger} = \\tilde{\\phi} \\circ \\tilde{\\Theta} \\bar{\\pi}^{\\dagger} \\circ \\tilde{\\tau}^{\\circ}$, with $\\bar{\\pi}^{\\dagger} : \\Delta^D \\to \\Delta^D$ Lipschitz. The adapter $\\bar{\\pi}^{\\dagger}$ is the only component that needs to be learned from true expert preference data, and its low dimension $D$ is what makes the sample-complexity gain possible.","core_discovery":"The central claim is that proxy feedback can improve sample complexity exactly when the proxy policy and the true policy are the same map up to a low-dimensional interchange: they must partition prompts identically (Condition 1), the true policy's responses must be expressible by the proxy's response decoder (Condition 2), that response space must admit a bi-Lipschitz coordinate chart of dimension $D$ (Condition 3), and the re-assignment from proxy responses to true responses must be Lipschitz (Condition 4). Theorem 3 shows that under these conditions the true policy decomposes as $\\pi^{\\dagger} = \\tilde{\\phi} \\circ \\tilde{\\Theta} \\bar{\\pi}^{\\dagger} \\circ \\tilde{\\tau}^{\\circ}$, where $\\tilde{\\phi}$, $\\tilde{\\Theta}$, $\\tilde{\\tau}^{\\circ}$ are the proxy policy's decoder, linear map, and encoder, and $\\bar{\\pi}^{\\dagger}$ is a Lipschitz adapter from $\\Delta^D$ to $\\Delta^D$. This decomposition is what carries the sample-complexity gain: the adapter is a map between $D$-simplices, so the covering-number bound, and hence the number of true preference labels needed for generalization, depends on $D$ rather than on the dimension $D'$ of the prompt embedding. The paper's proof adapts a standard generalization-error route via covering numbers, giving a bound for the adapter hypothesis class (Theorem 5) and a comparison bound for the same class learned without proxy structure (Theorem 6).","pith_inferences":["The paper leaves the Stage 1 guarantee open: its sample-complexity theorem assumes the true $\\tilde{\\phi}$, $\\tilde{\\Theta}$, $\\tilde{\\tau}^{\\circ}$ have already been recovered. A natural extension would bound the proxy-sample size needed for Stage 1 and show how Stage-1 estimation error inflates the Stage-2 $D$-scaling; until then, the practical regime is one where proxy data are so abundant that","The sufficient conditions suggest a testable screening procedure before training: collect paired proxy/expert ratings on a probe set and estimate how often the proxy's level-set partition disagrees with the expert's; high disagreement would predict that the $D$-scaling guarantee will not hold.","If the $D$-scaling result holds, it also gives a diagnostic for reward hacking: the norm $\\|\\bar{\\pi}^{\\dagger}(p) - p\\|$ measures how far the proxy policy is from the true policy at each latent point, and large jumps would flag regions where proxy data should be distrusted.","Combining the adapter view with active learning is a natural next step: choose which expert preferences to query by maximizing the reduction in the adapter's Lipschitz uncertainty, which could reduce the constant in front of the $D$-scaling bound."],"forward_implications":["Under the two-stage scheme, the number of true preference samples needed for $\\epsilon$-generalization with probability $1-\\omega$ scales like $\\Omega\\!\\left(\\frac{D}{\\epsilon^2}\\left(\\frac{C\\sqrt{D}}{\\epsilon}\\right)^D \\log(C/\\epsilon) - \\log\\omega\\right)$, with the intrinsic dimension $D$ in the exponent (Theorem 5).","Learning the same true policy from scratch, without the proxy structure, requires a bound with the embedding dimension $D'$ in the exponent, where $D' \\gg D$ (Theorem 6).","The decomposition gives a concrete LLM parametrization: train the encoder–linear–decoder triplet on abundant proxy preferences, then fine-tune only a low-dimensional adapter on the small expert dataset while keeping the proxy components frozen.","The four conditions function as data-collection guidance: recruit proxy raters who group prompts the same way experts do, can in principle express every expert response, and stay within a Lipschitz distance of expert judgment.","Because the adapter lives on $\\Delta^D$, the architecture can be realized by adapting existing encoder–decoder language models without retraining the pretrained components on expert data."],"supporting_citations":[{"why":"Supplies the logistic preference model that turns reward differences into pairwise comparison probabilities, which the DPO formulation builds on.","marker":"Bradley & Terry (1952)"},{"why":"Establishes direct preference optimization and the implicit reward form that lets policies be optimized directly from preferences, motivating the factored parametrization.","marker":"Rafailov et al. (2023)"},{"why":"Supplies the group-symmetry PAC proof technique that the paper adapts for its generalization bounds and covering-number arguments.","marker":"Elesedy (2022)"},{"why":"Provides the standard generalization-error versus covering-number bound used in the convergence-rate analysis.","marker":"Mohri et al. (2012, Exercise 3.31)"},{"why":"Defines and characterizes reward gaming, the phenomenon the paper aims to address constructively.","marker":"Skalse et al. (2022)"},{"why":"Documents open problems in RLHF and reward hacking, motivating the question of when proxy data can provably help.","marker":"Casper et al. (2023)"}],"fun_headline_variants":["Proxy data can cut sample need for true preferences when four conditions hold","Proxy feedback shrinks preference learning sample complexity under stated conditions","Proxy feedback can drive sample complexity down from D' to D","Four conditions let proxy data lower true policy sample complexity","Proxies improve sample complexity under four precise conditions"],"cache_read_input_tokens":26880,"weakest_assumption_plain":"The sample-complexity guarantee assumes the first stage has already identified the true proxy encoder, linear map, and decoder exactly; the paper gives no bound on how much proxy data that identification needs or on how Stage-1 estimation error inflates the Stage-2 guarantee.","fun_headline_variants_meta":{"raw":{"variants":["Proxy data can cut sample need for true preferences when four conditions hold","Proxy feedback shrinks preference learning sample complexity under stated conditions","Proxy feedback can drive sample complexity down from D' to D","Four conditions let proxy data lower true policy sample complexity","Proxies improve sample complexity under four precise conditions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001668,"raw_usage":{"total_tokens":6662,"prompt_tokens":1033,"completion_tokens":5629,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":649,"completion_tokens_details":{"reasoning_tokens":5547}},"tokens_in":649,"tokens_out":5629,"duration_ms":35360,"temperature":1.0,"reasoning_tokens":5547,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:34:17.825719+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"One concrete falsifier is a real or synthetic preference dataset containing two prompts that the proxy maps to the same response distribution but the true policy maps to different distributions; such level-set disagreement violates Condition 1, and if it is widespread the identity $\\pi^{\\dagger} = \\tilde{\\phi} \\circ \\tilde{\\Theta} \\bar{\\pi}^{\\dagger} \\circ \\tilde{\\tau}^{\\circ}$ cannot hold.","supporting_citations":[{"cited_title":"Group symmetry in pac learning","cited_arxiv_id":null,"evidence_quote":"Supplies the group-symmetry PAC proof technique that the paper adapts for its generalization bounds and covering-number arguments."},{"cited_title":"Open problems and fundamental limitations of reinforcement learning from human feedback","cited_arxiv_id":null,"evidence_quote":"Documents open problems in RLHF and reward hacking, motivating the question of when proxy data can provably help."}],"review_version":1}