{"id":"6d98ddec-8dac-4df0-901f-767d8975e2d5","arxiv_id":"2509.04794","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A systematic comparison of ICL, LoRA fine-tuning, and activation steering for Big Five personality control, with new contrastive data and evaluation metrics, but with key claims contradicted by the reported experiments.","lead":"This paper compares three ways to give large language models a chosen Big Five personality: prompting, fine-tuning, and activation steering, and measures how each affects reasoning and bias. It introduces a contrastive training dataset, an openness-purification trick, and a stability metric, but the headline trade-offs are undercut by missing results, circular validation, and single-run statistics.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central trade-off claim is not supported by the reported data: LLaMA-3 has no mechanistic-steering row, and on Gemma-2 MS alignment is below ICL and PEFT on every trait, so the abstract's competitive MS statement rests on absent or contradictory evidence.","rationale":"I read the paper as claiming a practical, cross-model ranking of three personality-control methods. For that claim to hold, alignment must be measured validly and reported for both models and all three methods. The weakest point is not a subtle statistical issue but a direct evidentiary gap: the reported tables contradict the abstract. Gemma-2 MS is not competitive with ICL or PEFT on alignment, and LLaMA-3 MS is entirely absent. This alone defeats the central claim as stated. The reader's weakest assumption about classifier trustworthiness is related and important: all delta-alignment values come from a same-group classifier, and the only supposedly independent check in Appendix G is a table with no description of the task. I therefore partially agree with the reader. I would not accuse the authors of misconduct; the issue is that the paper's own evidence does not support its headline. The limitations appendix candidly notes single runs and partial subsets, and the checklist promises future release, but a future release cannot repair the internal inconsistency in the reported tables. The ICL-versus-PEFT comparison is plausible and could be salvaged by a revised paper that removes or re-scopes the MS claims; as written, the central trade-off claim should be rejected.","tokens_in":17814,"tokens_out":5362,"duration_ms":47368,"concrete_test":"Implement the Appendix E mechanistic-steering protocol on LLaMA-3-8B-Instruct exactly as specified, including layers 5, 10, 15, and 20 and linear strength calibration, then compute the five trait-alignment deltas under the same within-run protocol. In parallel, take a stratified sample of responses from both models across all three methods and score them with an independent Big Five assessment not derived from Jain et al., either human raters or a separately trained classifier. If LLaMA-3 MS alignment is comparable to the reported Gemma-2 MS values and the independent scoring preserves the ICL/PEFT/MS ordering on both models, the trade-off claim gains support; if the MS row cannot be produced, or the ordering changes materially, the abstract's cross-model competitive claim fails and the paper must be re-scoped to the conditions actually measured.","verdict_should_be":"REJECT","load_bearing_attack":"The single most load-bearing concern is that the paper's own Tables 1 and 2 undermine the central claim before any question about the measurement instrument. The abstract says experiments on Gemma-2-2B-IT and LLaMA-3-8B-Instruct reveal clear trade-offs and that MS provides lightweight runtime control with competitive effectiveness. But no LLaMA-3 MS results appear anywhere: Table 1 has ICL and PEFT rows for LLaMA-3 but no MS row, Appendix G's alignment table omits L3-S, and Appendix H's performance tables omit it as well. On Gemma-2, MS alignment is +0.64/+0.44/+0.50/+0.10/+0.29, while ICL is +0.91/+0.50/+0.97/+0.24/+0.81 and PEFT is +0.78/+0.97/+0.95/+0.21/+0.78. MS is below both other methods on every trait, with trait averages of about 0.394 versus 0.686 for ICL and 0.738 for PEFT. The characterisation of MS as competitive is therefore contradicted by the paper's own numbers, and the cross-model half of the trade-off claim is missing its evidence. A related but distinct problem is the alignment measure itself: every delta-alignment score comes from the Jain et al. classifier built by overlapping authors, the same classifier is used to calibrate steering strengths in Appendix E.4, and Appendix G reports the supposedly independent validation task only as a table of deltas with no task description, no independent labels, and no released data. The paper's Appendix A also concedes single runs and partial subsets, so precise rankings such as the stability hierarchy in Table 6 are statistically fragile. Because the headline conclusion is precisely the ICL/PEFT/MS trade-off, the missing MS condition and circular measurement leave the central claim unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares three methods for manipulating Big Five personality traits in Gemma-2-2B-IT and LLaMA-3-8B-Instruct: in-context learning (ICL), parameter-efficient fine-tuning via LoRA (PEFT), and activation-space mechanistic steering (MS). It introduces a contrastive high/low-trait dataset, a within-run Δ evaluation framework over MMLU, GAIA, and BBQ, a purification scheme intended to separate openness from conscientiousness, and a three-level stability metric. The headline claim is a trade-off: ICL gives strong alignment with minimal capability loss, PEFT gives the strongest alignment but degrades task performance, and MS provides lightweight, competitive runtime control. The paper also argues that these methods serve as complementary interpretability probes into how personality is represented in LLMs.","tokens_in":18149,"tokens_out":7447,"duration_ms":60377,"significance":"If the claims were supported, the paper would provide a practical decision framework for personality control and an interpretability probe spanning surface conditioning, parameter encoding, and activation-level steering. The within-run Δ design is a sensible attempt to avoid cross-method baseline artifacts, and the contrastive dataset addresses a real imbalance in existing personality-manipulation resources. The paper also reports concrete benchmark deltas that are falsifiable. However, the central findings are not currently established: the reported numbers contradict the abstract's characterization of steering, the alignment measure is circular, and all comparisons rest on single runs without uncertainty. The work is therefore of potential interest but cannot be accepted in its present form.","major_comments":[{"comment":"The headline claim that mechanistic steering is \"competitive\" is contradicted by the paper's own results. On Gemma-2, Table 1 shows MS trait alignment below both ICL and PEFT for every trait, e.g., Extraversion +0.64 vs +0.91/+0.78, Openness +0.10 vs +0.24/+0.21, and Conscientiousness +0.29 vs +0.81/+0.78. Moreover, no LLaMA-3 steering row appears in Table 1, in the alignment table in Appendix G, or in the MMLU/GAIA/BBQ delta tables in Appendix H, so the abstract's cross-model claim that \"Experiments on Gemma-2-2B-IT and LLaMA-3-8B-Instruct reveal clear trade-offs\" has no supporting evidence for MS. The claim that PEFT delivers alignment \"at the cost of degraded task performance\" is also only visible on Gemma-2: LLaMA-3 PEFT deltas are small and sometimes positive (e.g., GAIA +0.02, +0.00, +0.02, +0.04, +0.02 in Table 4). These are load-bearing contradictions between the stated conclusions and the reported data.","section":"Abstract; Table 1; Appendix G (Table 2); Appendix H (Tables 3–5)"},{"comment":"The alignment measurement is circular. Trait alignment scores are produced by the Jain et al. (2025) personality classifier; the interventions are constructed from the Jain et al. dataset (PEFT and ICL on high-trait examples, MS on high/low contrast pairs); and steering strengths are calibrated with that same classifier (Appendix E.4). Appendix F.2 states that the dataset's reliability is itself \"validated through the personality classifier Jain et al. [2025].\" The supposedly independent \"dedicated alignment task\" is never described: Appendix G reports only a table of deltas, with no task definition, prompt list, scoring rubric, or independent labels. As a result, the alignment scores cannot currently be taken as evidence about genuine trait expression, and the comparative effectiveness claims rest on the same instrument used to construct the interventions.","section":"Section 3; Appendix F.2; Appendix E.4; Appendix G"},{"comment":"All comparative results are single-run and carry no uncertainty. Appendix A explicitly concedes \"single benchmark evaluation runs and partial dataset subsets,\" yet Section 3 and Table 6 report precise rankings and stability scores (e.g., openness stability 0.0411 vs neuroticism 0.0309) without standard errors, confidence intervals, or repeated seeds. The NeurIPS checklist states that \"stability analysis reports variance across runs,\" but no multi-run variance estimates are presented anywhere, and Appendix L's stability formula is computed from deltas rather than from run-to-run variation. Without error bars, the paper cannot support rank claims among ICL, PEFT, and MS, or the trait-stability ordering.","section":"Appendix A; Section 3; Table 6"},{"comment":"The stability metric as written does not measure run-to-run stability. With a single run per condition, \"normalized_variance\" and \"normalized_range\" appear to be computed over traits or benchmarks within that run, not over repeated runs; the consistency term 1/(1+mean_abs_deltas) rewards small absolute deltas, which is better interpreted as a capability-preservation measure. The normalization factors (÷10000 for variance, ÷1000 for range) are stated in Appendix L.3 to be empirically derived and can dominate the score. The three-level stability hierarchy in Table 6 is therefore not a validated construct.","section":"Appendix L.1, Eq. (1)"},{"comment":"The text claims run-to-run variability, e.g., \"LLaMA-3 exhibits variability across runs\" in Section 4 and \"we avoid cross-run comparisons due to baseline variance across runs\" in Appendix C.3, while Appendix A says only single benchmark evaluation runs were performed. These statements are inconsistent. If multiple runs exist, they should be reported with means and errors; if they do not, the variability claims should be removed.","section":"Section 4; Appendix A; Appendix C.3"}],"minor_comments":[{"comment":"The alignment-table labels are inconsistent with the main text: Appendix G uses G2-P, G2-S, G2-F, L3-P, and L3-F with abbreviations defined as in Table 3, where P=Prompting, F=PEFT, and S=Steering, while the body uses ICL and MS; this makes cross-referencing unnecessarily difficult.","section":"Appendix G"},{"comment":"The claim that \"Independent alignment validation shows strong alignment for most traits (e.g., Gemma extraversion 1.00, neuroticism 1.00)\" does not match Table 1 or Table 2, which report +0.91 and +0.97 for Gemma-2 ICL on those traits.","section":"Appendix C.3"},{"comment":"The text asserts \"statistically significant alignment\" and \"statistically significant improvements,\" but no significance tests, p-values, or confidence intervals are reported anywhere in the paper.","section":"Appendix E.6"},{"comment":"The sentence \"Rao et al. [2023] demonstrated ... while Rao et al. [2023] revealed ...\" cites the same reference twice in a way that reads as though two different studies are intended; please disambiguate.","section":"Appendix B.2"},{"comment":"The paper acknowledges that the trait-purification confidence threshold is \"somewhat arbitrary\" but provides no sensitivity analysis; since the openness purification is presented as a contribution, its robustness should be demonstrated.","section":"Appendix A; Appendix E.4"},{"comment":"The entry \"Jen tse Huang\" appears to contain a formatting error in the author name, and the in-text citation \"tse Huang et al.\" should also be normalized.","section":"Reference list"}],"recommendation":"reject","confidential_remarks":"The alignment instrument and the training data both come from Jain et al. (2025), a prior paper with overlapping authorship; the manuscript does not establish the independence of the validation. The verdict is based on the internal contradictions and circular measurement reported above, not on authorship itself."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper has a genuinely useful new artifact and a sensible evaluation design, but its headline trade-off claim is not supported by its own tables. There is no LLaMA-3 mechanistic-steering row anywhere, and on Gemma-2 the MS alignment scores are below ICL and PEFT for every trait. The abstract's \"competitive effectiveness\" for MS is contradicted by the numbers in Table 1.\n\nWhat's actually new: the contrastive high/low trait dataset, the openness purification step, the within-run delta protocol, and the three-level stability framework. These are real and could be useful to people working on personality control. The ICL vs. PEFT comparison on Gemma-2 looks like a plausible finding: ICL gets strong alignment with small capability deltas, PEFT gets higher alignment but some negative deltas.\n\nSoft spots: First, the missing MS results for LLaMA-3. The paper claims experiments on both models, but Appendix G and H omit the L3-S row. That is a load-bearing gap because the abstract makes a cross-model claim. Second, the alignment measurement is partly circular. The classifier comes from Jain et al. (2025) with overlapping authors, and it is used both to calibrate the steering vectors (Appendix E.4) and to score alignment. The \"dedicated alignment task\" in Appendix G is just a table of deltas, with no task description or independent labels. Third, all deltas come from single runs with no error bars; Appendix A admits this, yet the stability rankings in Table 6 are presented as precise. The checklist's \"Yes\" on statistical significance does not square with the single-run statement.\n\nOn balance: the core ICL vs. PEFT result might survive more careful measurement, but the three-way trade-off story as told is not supported. I would want the authors to either add the missing MS condition or rephrase the abstract, report error bars or multiple seeds, and describe the alignment validation properly. The paper's own limitations section is honest, which counts in its favor.\n\nThis paper is for practitioners working on persona control and for researchers interested in activation steering. It deserves a serious referee, but with major revision expected. I would send it to review—the new dataset and the within-run evaluation idea are worth engaging with—but I would not accept it in its current form.","headline":"Useful dataset and evaluation idea, but the headline three-way trade-off is not backed by the paper's own numbers, and the alignment measure is partly circular.","tokens_in":18745,"tokens_out":2028,"would_cite":true,"duration_ms":17631,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Personality control in LLMs comes with three trade-offs: ICL, PEFT, and steering each buy alignment at a different price.","keywords":["Big Five personality","mechanistic steering","in-context learning","parameter-efficient fine-tuning","LLM evaluation","trait alignment","bias benchmark","contrastive dataset"],"falsifier":"Take a sample of model outputs from each method and trait condition, and have independent human raters score them for Big Five traits. If the human scores do not match the classifier's alignment deltas (e.g., classifier says +0.9 agreeableness but humans detect no change), the comparative trade-off claims lose their foundation.","tokens_in":17573,"feed_emoji":"🎭","tokens_out":1181,"duration_ms":11816,"temperature":0.7,"pith_summary":"The paper systematically compares three ways to give a large language model a Big Five personality—in-context learning (ICL), parameter-efficient fine-tuning (PEFT), and mechanistic steering (MS)—and measures what each costs in reasoning ability, agent performance, and social bias. It claims that ICL achieves strong personality alignment with minimal capability loss, PEFT delivers the strongest alignment but degrades task performance, and steering offers lightweight runtime control with competitive effectiveness. The authors treat personality manipulation as a probe into how traits are represented in a model, distinguishing surface conditioning, parameter-level encoding, and activation-level steering. A sympathetic reader would take away a practical map for choosing a method under deployment constraints.","feed_headline":"Three ways to give an LLM a personality, and what each costs","feed_subtitle":"In-context prompts, fine-tuning, and steering vectors trade alignment against capability in predictable ways.","key_machinery":"The load-bearing mechanism is the contrastive dataset—each high-trait response paired with a low-trait response generated by GPT-4.1 Mini—plus a within-run delta (Δ) evaluation protocol that measures capability, bias, and alignment changes relative to each method's own baseline. Steering vectors are computed as the mean difference between trait-positive and trait-negative activations at post-attention layer norms (layers 5, 10, 15, 20), with a purification step that combines an openness vector with an openness-versus-conscientiousness contrast vector.","core_discovery":"The central claim is that personality manipulation in LLMs is governed by a consistent three-way trade-off, and that the same Big Five traits are accessible at three different representational levels. On two instruction-tuned models, ICL produces high trait-alignment with small within-run performance deltas on MMLU and GAIA; PEFT achieves the highest alignment scores but incurs larger negative capability deltas, especially on Gemma-2; and mechanistic steering gives moderate alignment with trait-dependent deltas while requiring no parameter updates. The paper also claims that openness is uniquely difficult to manipulate, agreeableness resists ICL, and personality encoding concentrates around intermediate transformer layers (around layer 15).","pith_inferences":["If personality alignment really is a multi-level phenomenon, then hybrid methods—for example, ICL plus a light steering vector—should outperform any single method at fixed capability cost; the paper does not test this directly.","The alignment classifier's measurement of traits is the linchpin for every comparative claim, so a human-evaluation replication on a small sample of responses would be the most direct test of whether the trade-offs are real.","The large BBQ bias shifts reported for steering and PEFT on Gemma-2 suggest that personality interventions can inadvertently move bias in either direction, which raises a governance question the paper only touches on."],"forward_implications":["Practitioners can pick a personality-control method by constraint: ICL when capability preservation matters most, PEFT when alignment strength dominates, and steering when lightweight runtime control is required.","The finding that personality encoding consolidates around intermediate layers suggests that future steering interventions should target those layers rather than shallow or deep layers.","The trait-specific results imply that openness and agreeableness will need bespoke manipulation strategies, since generic conditioning works less well for them.","The stability framework provides a quantitative way to rank method-trait combinations under deployment constraints, with steering plus conscientiousness scoring highest.","The contrastive dataset construction method could be reused to build balanced training data for other behavioral attributes beyond personality."],"supporting_citations":[{"why":"Supplies the original personality manipulation dataset and the personality classifier used to score trait alignment.","marker":"Jain et al. [2025]"},{"why":"Defines the LoRA method used for the parameter-efficient fine-tuning arm.","marker":"Hu et al. [2022]"},{"why":"Provides the activation-engineering approach that the steering vector construction is based on.","marker":"Turner et al. [2023]"},{"why":"Defines the MMLU benchmark used for capability evaluation.","marker":"Hendrycks et al. [2021]"},{"why":"Defines the GAIA benchmark used for agent-style reasoning evaluation.","marker":"Mialon et al. [2023]"},{"why":"Defines the BBQ benchmark and the ambiguous subset used for bias measurement.","marker":"Parrish et al. [2022]"}],"fun_headline_variants":["LLM personality: three levers, one trade-off","ICL, PEFT, or steering? Each LLM personality fix has a price","Openness resists, agreeableness defies ICL, layers matter in LLM personality","Mechanistic steering: lightweight way to give LLMs a personality","The cost of character: PEFT nails traits but harms task performance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The alignment scores from the personality classifier are accurate measures of genuine trait expression, even though the classifier was built in prior work with overlapping authors and the training and steering data come from the same dataset.","fun_headline_variants_meta":{"raw":{"variants":["LLM personality: three levers, one trade-off","ICL, PEFT, or steering? Each LLM personality fix has a price","Openness resists, agreeableness defies ICL, layers matter in LLM personality","Mechanistic steering: lightweight way to give LLMs a personality","The cost of character: PEFT nails traits but harms task performance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000321,"raw_usage":{"total_tokens":1826,"prompt_tokens":980,"completion_tokens":846,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":596,"completion_tokens_details":{"reasoning_tokens":748}},"tokens_in":596,"tokens_out":846,"duration_ms":8079,"temperature":1.0,"reasoning_tokens":748,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:27:09.832505+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a sample of model outputs from each method and trait condition, and have independent human raters score them for Big Five traits. If the human scores do not match the classifier's alignment deltas (e.g., classifier says +0.9 agreeableness but humans detect no change), the comparative trade-off claims lose their foundation.","supporting_citations":[{"cited_title":"From Text to Emoji: How PEFT-Driven Personality Manipulation Unleashes the Emoji Potential in LLMs","cited_arxiv_id":"2409.10245","evidence_quote":"Supplies the original personality manipulation dataset and the personality classifier used to score trait alignment."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the BBQ benchmark and the ambiguous subset used for bias measurement."}],"review_version":2}