{"id":"c5d99f39-fb87-4d04-85ad-f6f92f8739c0","arxiv_id":"2509.05605","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"ICON2 uses representation-space steering to generate preference data from the model itself, improving alignment benchmarks and cutting cost.","lead":"A new method, ICON2, builds AI preference-training data by reading the model's own internal direction vectors and using them to create good and bad answer pairs directly. The authors report higher alignment scores on AlpacaEval and Arena-Hard with up to 48% lower compute than conventional sampling-based data construction.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central premise that PCA directions from contrastive 'helpful/harmless/honest' prompts encode human preference is unvalidated; the reported gains could arise from steering style rather than alignment.","rationale":"The reader correctly identifies the weakest assumption as the validity of the extracted direction vectors. My stress-test confirms this: every downstream component—instruction filtering (Eq. 2), steering for chosen/rejected generation (Eq. 4), and the DPO training on those pairs—depends on the direction being aligned with human preferences. Without that, the method may simply make the model more consistent with its own internal biases, and the benchmark numbers could be inflated by the judge's stylistic preferences. The paper provides no human evaluation of the preference pairs, and its robustness analyses (Appendix G) only address stability of the vectors across feature datasets, not their semantic content. The comparison with direct prompting (Appendix I) shows that ICON2 avoids the trivial 'reward hacking' of system-prompt-based pairs, but it does not validate the labels. Thus, the central claim is plausible but not fully established. I recommend keeping the reader's CONDITIONAL verdict: the method is coherent and shows consistent gains, but the preference direction must be validated against humans or shown to be necessary via a random-direction control before the claim can be accepted as demonstrated.","tokens_in":21593,"tokens_out":5653,"duration_ms":55003,"concrete_test":"Two-part check: (1) Take 100–200 instructions from D_filt and have at least three independent human raters choose which of the ICON2-generated responses they prefer (or mark a tie). Compute agreement between the synthetic labels and human choices. If agreement is not significantly above chance (e.g., accuracy > 60% with reasonable CI), the direction vectors do not encode human preference. (2) Run the full ICON2 pipeline with a control direction: replace u_c in Eq. (4) with a random unit vector (or with vectors from unrelated contrastive prompts such as 'You are a banana' vs. 'You are a table'). Compare AlpacaEval 2.0 LC win rates. If the random-direction baseline matches ICON2's gains, the improvement is not due to the semantic content of the extracted direction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The entire ICON2 pipeline rests on the claim that the direction vectors extracted in Eq. (1) and applied in Eq. (4) encode human preferences. The vectors come from the first principal component of hidden-state differences between simple positive and negative system prompts (e.g., 'You are a helpful assistant' vs. 'You are a helpless assistant', Appendix E). There is no validation—human or otherwise—that these directions correspond to what people actually prefer. The sensitivity analysis in Appendix G only shows that the vectors are stable across D_feat, not that they are semantically meaningful. Appendix I demonstrates that direct prompting causes reward hacking, but that comparison merely shows ICON2 pairs are less trivially separable; it does not establish that the labels match human preferences. If the direction instead captures a stylistic property (politeness, hedging, formatting, or length), then the generated 'chosen' and 'rejected' responses are mislabeled, and DPO will amplify the model's own stylistic biases. The reported benchmark improvements could then reflect overfitting to GPT-4's judge preferences rather than genuine alignment. This is the load-bearing assumption: if the direction is not preference-aligned, the method's contribution reduces to a style-shifting trick.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ICON2, a framework that constructs preference data for DPO without external annotations or multi-response sampling. It extracts layer-wise direction vectors as the first PCA component of contrastive hidden-state differences between positive and negative system prompts for criteria C = {honesty, harmlessness, helpfulness, general}; filters self-synthesized instructions by a mean-pooled dot-product consistency score; and generates chosen/rejected response pairs by adding/subtracting the direction vector during decoding. The resulting pairs are used for offline DPO. Experiments on Llama3-8B and Qwen2-7B report improvements over SFT and several baselines on AlpacaEval 2.0, Arena-Hard, and MT-Bench, with reduced GPU cost. Appendices provide pseudocode, leakage analysis, sensitivity analysis, a direct-prompting comparison, and a no-DPO hyperparameter selection scheme.","tokens_in":21868,"tokens_out":6895,"duration_ms":63434,"significance":"If the extracted directions genuinely encode human preferences, the method is a novel and efficient route to tailored preference data: it avoids multi-response sampling, is model-specific, and ships detailed pseudocode, a leakage analysis (Appendix D), a stability analysis (Appendix G), an explicit comparison with direct prompting (Appendix I), and a reward-based hyperparameter selection procedure (Appendix J). These are real strengths. However, the paper's central scientific claim—that the PCA directions represent human preferences rather than stylistic correlates—is not yet supported by any human or independent validation. Combined with single-run headline numbers, the current evidence is suggestive rather than conclusive. The contribution would be significantly strengthened by a human pairwise-preference study on generated pairs and multi-seed evaluation.","major_comments":[{"comment":"The load-bearing premise that the first principal component of contrastive hidden-state differences encodes human preference is not validated. Appendix G shows only that the vectors are stable across D_feat size and source (Table 9) and that downstream results are stable (Table 11); it does not show semantic alignment with human judgment. Because Eq. (4) uses these directions to label chosen and rejected responses, all DPO gains could in principle reflect steering of stylistic properties (length, politeness, hedging) rather than genuine alignment. The authors' own Ethical Considerations acknowledge this risk. I request a human-annotation study on a sample of generated pairs, or an independent pre-registered preference probe, reporting agreement rates, together with a comparison of the extracted directions against established behavioral preference directions.","section":"Section 3.1, Eq. (1); Appendix G"},{"comment":"All headline results are single runs without error bars, confidence intervals, or significance tests. For example, the 23.22 versus 18.38 LC gap between ICON2(General+3H) and Self-Refine in Table 1 is presented as a strong improvement, but no variance or paired comparison is reported. AlpacaEval 2.0 and Arena-Hard win rates are stochastic, and MT-Bench scores are known to be noisy; I request at least three seeds or bootstrap confidence intervals over judge samples to support the comparative claims. Without this, the reported gains cannot be distinguished from run-to-run noise.","section":"Tables 1 and 2; Section 4.2"},{"comment":"The preference labels are produced in a self-referential loop: the same SFT model provides the contrastive directions, the steered chosen and rejected responses, and the DPO training signal, with no external human or independent reward-model verification. While self-rewarding pipelines also use model-generated labels, they at least rely on an LLM judge scoring responses; here the label is an unvalidated geometric direction. Table 12 shows that IC pairs do not saturate reward accuracy, but that only indicates the pairs are harder to classify, not that the labels reflect human preferences. I request an external evaluation of the generated pairs—for example, a human pairwise-preference sample or a held-out reward model—to show that the chosen/rejected distinction corresponds to actual preference rather than to the model's internal bias.","section":"Section 3.3, Eq. (4); Appendix I"}],"minor_comments":[{"comment":"The abstract states an average win-rate improvement of 13.89% on AlpacaEval 2.0, but Table 1 gives (17.63 + 10.05) / 2 = 13.84 for length-controlled win rates, and the abstract does not specify whether the number refers to LC or raw WR. Please align the text with the table and define the metric explicitly.","section":"Abstract; Table 1"},{"comment":"The consistency score is an unnormalized mean-pooled dot product between hidden states and direction vectors. Although the max over criteria is unaffected by the shared hidden-state norm, the absolute score used for thresholding and ranking can be dominated by instruction length or layer-wise norm differences; please normalize the vectors or justify that the filtering is invariant to these scale effects.","section":"Section 3.2, Eq. (2)"},{"comment":"The experiments in Table 3 use 20k instructions for each instruction-construction method, while the main experiments use 100k filtered instructions. Please clarify whether the 20k subset is drawn from the same 1M raw pool and whether the same consistency threshold is applied, so the comparison is apples-to-apples.","section":"Table 3"},{"comment":"The row labeled 'Manual Collection' uses UltraFeedback, whose preference annotations come from GPT-4 rather than human annotators; the label is misleading and should be changed to something like 'UltraFeedback (GPT-4 labels)' to avoid implying human annotation.","section":"Table 1; Section 4.1"},{"comment":"The no-DPO hyperparameter selection procedure is described for γ values but the paper does not state whether the same γ is used for all four criteria in the main experiments; since Eq. (4) defines γ_{c*}, please clarify whether per-criterion coefficients were tuned or a single global coefficient was used.","section":"Section J; Table 14"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this paper has a genuinely new idea, and it mostly executes it well. Instead of sampling multiple responses and judging them, ICON2 extracts direction vectors from contrastive system prompts, uses them to filter self-synthesized instructions, and then steers token representations up or down those directions to produce chosen/rejected pairs in exactly two generation passes. That combination is not in the cited literature. The gains over SFT are large, and the win over Self-Refine is real but more moderate. The cost table is also the right kind of evidence: 149 dollars versus 287 for Self-Rewarding is a meaningful efficiency claim.\n\nWhat the paper does well: the method is described carefully, the ablations are useful, and the appendix actually answers some obvious objections. The leakage analysis is standard but fine. The sensitivity analysis on the feature dataset shows the directions are stable. Most importantly, Appendix I compares against direct prompting with opposing system prompts: those models collapse to near-SFT performance and reward accuracy saturates at 0.99, while ICON2 does not saturate. That is real evidence that the steering is not just doing something trivial. The hyperparameter selection via reward scores is clever and they do check downstream robustness on six benchmarks.\n\nWhere the soft spots are: the load-bearing assumption is still only indirectly supported. The PCA directions come from \"helpful\" versus \"helpless\" prompts and the like, and there is no human validation that those directions correspond to what people prefer. Appendix G shows the vectors are stable across datasets, not that they are semantically preference-aligned. Appendix I shows the pairs are harder to separate, which rules out the crudest reward hacking, but it does not prove the labels match human preferences. If the direction captures style rather than substance, DPO will amplify style, and GPT-4 judges may reward that. The MT-Bench improvements and the fact that the method beats stronger baselines across three benchmarks make me think the effect is not purely stylistic, but the gap between \"plausible\" and \"established\" is exactly the missing human or judge-label validation. Also: the main results in Tables 1 and 2 are single runs with no error bars or significance tests; no code or data is released; and the introduction contains an unrelated inserted passage about paper rejection that must be removed before publication. None of this is fatal, but it is real.\n\nThe citation pattern looks appropriate; the related work is current and the representation-engineering lineage is credited. This is a paper for anyone working on self-synthetic preference data or representation-based alignment. It deserves a serious referee. I would send it to review with the request that the authors add multiple seeds, ideally a small human preference check or at least a held-out judge that was not used in development, and release the generated preference data.\n\nRecommendation: engage with it; it is a solid conditional accept after those additions.","headline":"Genuinely novel method for self-synthetic preference data via representation steering, with solid ablations and a real efficiency win, but the core premise that PCA directions encode human preference is only indirectly validated and the reporting lacks error bars and code release.","tokens_in":22395,"tokens_out":1805,"would_cite":true,"duration_ms":19468,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that human preference criteria are linearly encoded in a language model's hidden states, so a single steering direction per criterion lets the model generate its own chosen and rejected response pairs, and training on…","keywords":["preference data construction","representation engineering","inherent control","self-synthetic data","contrastive system prompts","instruction filtering","direct preference optimization","alignment"],"falsifier":"Take a model held out from the paper's procedure, extract the helpfulness direction from 1024 unrelated instructions using the paper's contrastive methodology, and generate 200 response pairs by positive and negative steering. If human annotators do not prefer the positive-steered responses over the negative-steered ones at a rate reliably above chance (say, at least 80%, comparable to the paper's implied separation), then the extracted direction is not encoding a human preference. A second, sharper test: fine-tune a model to be deliberately unhelpful; if the extracted 'helpfulness' direction from that model does not flip or disappear, the vector is capturing a superficial prompt artifact rather than the intended preference.","tokens_in":21407,"feed_emoji":"🧭","tokens_out":5094,"duration_ms":43004,"temperature":0.7,"pith_summary":"The paper proposes ICON$^{2}$, a framework that builds preference datasets without human annotation, external judges, or multiple sampled responses. It extracts layer-wise direction vectors from contrastive system prompts representing criteria like helpfulness, honesty, and harmlessness, then uses these vectors both to filter self-synthesized instructions and to steer token representations during decoding, directly producing preferred and dispreferred response pairs. On AlpacaEval 2.0, Llama3-8B reaches a length-controlled win rate of 23.22% (up from 5.59% after SFT) and Qwen2-7B reaches 20.00% (up from 9.95%), with Arena-Hard gains of 13.7 and 13.2 points. The claimed efficiency payoff is up to 48.1% lower compute cost than Self-Rewarding, because only two generation passes per instruction are needed instead of sampling and ranking many candidates.","feed_headline":"Models can write their own training pairs via latent steering","feed_subtitle":"Directing token vectors yields chosen/rejected responses in two passes, beating SFT by 17.63 LC points on Llama3-8B.","key_machinery":"The central mechanism is the layer-wise preference direction vector $u^l_c$, defined as the first principal component of the contrastive representations $h^{l,+}_{i,c} - h^{l,-}_{i,c}$ obtained from positive and negative system prompts for criterion $c$. This vector carries the argument in two places: it computes an inherent-consistency score $\\text{consistency}_{i,c} = \\text{meanpool}(\\{h^l_i^\\top u^l_c\\}_{l=1}^N)$ that filters self-synthesized instructions, and it steers token representations during decoding through $\\hat{z}^l_k = z^l_k + \\gamma_{c^*} u^l_{c^*}$, with positive $\\gamma$ producing the chosen response and negative $\\gamma$ producing the rejected response. The paper names the overall method ICON$^{2}$ and shows that middle layers (10--20) give the best steering results.","core_discovery":"The central claim is that sophisticated human preferences can be captured as linear directions in the representation space of an SFT model, and that adding or subtracting these directions during decoding creates response pairs with clear alignment differences. Formally, for each criterion $c$, the paper computes the contrastive vector $v^l_{i,c} = h^{l,+}_{i,c} - h^{l,-}_{i,c}$ between hidden states under positive and negative system prompts, aggregates over a small feature set ($D_{\\text{feat}}$ of 1024 Alpaca instructions), and takes the first principal component $u^l_c$ per layer as the direction vector. These directions then serve two purposes: instruction filtering by mean-pooled dot-product consistency (max over criteria), and token-level steering via $\\hat{z}^l_k = z^l_k + \\gamma_{c^*} u^l_{c^*}$ during decoding at layers 10--20. The paper reports that training with direct preference optimization (DPO) on the resulting pairs substantially beats sampling-based baselines on length-controlled win rate and raw win rate on AlpacaEval 2.0 and Arena-Hard, and improves MT-Bench scores, including second-turn gains above 1.05 points.","pith_inferences":["A direct test of the method's premise would be cross-model transfer: extract direction vectors from one model family (say, a Qwen model) and steer a different family (say, Llama) to see whether the direction encodes a general human preference or is idiosyncratic to the model that produced it; the paper only measures same-model steering.","Because the steering strength $\\gamma$ is a continuous parameter, the method can generate graded preference pairs with controlled margins, which could be useful for preference-optimization variants that need fine-grained reward differences rather than binary chosen/rejected labels.","The paper's own Limitations section notes that online DPO settings and multi-turn dialogue generalization are unverified; whether the extracted direction vectors remain stable when the model weights change during online training is an open empirical question.","The consistency filter selects each instruction based on its maximum alignment with a single criterion; a natural extension would be to explicitly balance the filtered set across honesty, harmlessness, helpfulness, and general quality to avoid over-specialization in one direction."],"forward_implications":["Preference dataset construction becomes a two-pass generation problem: one positive-steered pass and one negative-steered pass per instruction, eliminating the need to sample, score, and rank five or more responses.","The reported cost reduction (up to 48.1% versus Self-Rewarding on Llama3-8B, with total cost \\$149.1 versus \\$287.2) implies that large-scale preference data can be synthesized on commodity hardware within hours rather than days.","Instruction synthesis from pre-query templates without seed instructions, combined with inherent-consistency filtering, yields tailored instructions that outperform manual collections and prior synthetic sets when evaluated under the same DPO pipeline.","Hyperparameters for steering (layer range and $\\gamma$ values) can be selected from 100 reward-scored responses in under one GPU hour, without running DPO, making the approach easy to port to new base models.","MT-Bench second-turn improvements of more than 1.05 points suggest that even single-turn preference pairs improve multi-turn dialogue quality, likely by making the model's early responses more aligned."],"supporting_citations":[{"why":"Supplies the contrastive-vector and PCA methodology that the paper's Eq. (1) and direction-vector extraction follow.","marker":"Zou et al., 2023"},{"why":"Provides the representation-engineering approach to aligning LLMs with human preferences, which motivates the criteria set and the inherent-consistency monitoring.","marker":"Liu et al., 2023"},{"why":"Introduces pre-query templates for instruction synthesis without seed instructions, which the paper adapts to generate the raw instruction set.","marker":"Xu et al., 2024"},{"why":"Provides the UltraFeedback manual preference dataset used as the 'Manual Collection' baseline and as the feature set for direction extraction.","marker":"Cui et al., 2023"},{"why":"Defines the Self-Rewarding baseline whose compute cost ICON2 is directly compared against and reduces by up to 48.1%.","marker":"Yuan et al., 2024"},{"why":"Supplies the self-boosting/self-refine baseline that samples multiple responses, which the paper contrasts with its single-pair generation approach.","marker":"Dong et al., 2024"},{"why":"Provides the ArmoRM-Llama3-8B reward model used to rank sampled responses in the Sampling-Ranking baseline.","marker":"Wang et al., 2024a"},{"why":"Supports the superposition hypothesis that motivates adding or subtracting direction vectors to steer model behavior during decoding.","marker":"Templeton et al., 2024"}],"fun_headline_variants":["LLMs self-generate preference pairs via latent steering","Internal vectors write chosen/rejected responses for training","Self-synthetic DPO data cuts alignment compute by 48%","Steering hidden states yields better self-taught alignment","Icon2: latent directions craft preference data on the fly"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the first principal component of the difference between hidden states under 'helpful', 'honest', and 'harmless' prompts and their negations actually points along a direction that human judges would consider better-to-worse; if that direction is not aligned with human preferences, ICON2 merely makes the model more consistent with its own internal biases.","fun_headline_variants_meta":{"raw":{"variants":["LLMs self-generate preference pairs via latent steering","Internal vectors write chosen/rejected responses for training","Self-synthetic DPO data cuts alignment compute by 48%","Steering hidden states yields better self-taught alignment","Icon2: latent directions craft preference data on the fly"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000362,"raw_usage":{"total_tokens":1982,"prompt_tokens":1005,"completion_tokens":977,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":621,"completion_tokens_details":{"reasoning_tokens":898}},"tokens_in":621,"tokens_out":977,"duration_ms":8780,"temperature":1.0,"reasoning_tokens":898,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:22:04.376514+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a model held out from the paper's procedure, extract the helpfulness direction from 1024 unrelated instructions using the paper's contrastive methodology, and generate 200 response pairs by positive and negative steering. If human annotators do not prefer the positive-steered responses over the negative-steered ones at a rate reliably above chance (say, at least 80%, comparable to the paper's implied separation), then the extracted direction is not encoding a human preference. A second, sharper test: fine-tune a model to be deliberately unhelpful; if the extracted 'helpfulness' direction from that model does not flip or disappear, the vector is capturing a superficial prompt artifact rather than the intended preference.","supporting_citations":[{"cited_title":"Daniel Freeman, Theodore R","cited_arxiv_id":null,"evidence_quote":"Supports the superposition hypothesis that motivates adding or subtracting direction vectors to steer model behavior during decoding."}],"review_version":2}