{"id":"33ca24fe-9aa0-4939-95d3-ed0fd0d46478","arxiv_id":"2601.10960","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"SWAI controls LLM style by biasing the top candidate words using z-normalized word frequencies from target-style corpora, with no training or activation access.","lead":"This paper introduces SWAI, a method that steers language models by adding a small bias to likely next-word choices using word-frequency statistics from labeled texts. The authors report large gains on readability, politeness, and toxicity control across two Llama models, without training an auxiliary model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The proof of control rests on a GPT-4o judge whose rubric explicitly rewards the same surface cues SWAI injects; without human validation the headline gains may be an artifact of evaluator circularity.","rationale":"The reader's weakest assumption—that the LLM judge shares the surface cues SWAI amplifies—is exactly the most load-bearing concern. The paper's evidence for control is a judge that explicitly uses those cues, and no human validation is offered. My stress-test confirms this concern is central: without an independent measure of perceived readability/politeness/toxicity, the measured gains are consistent with the method merely inserting tokens the judge is told to seek. I also flag an additional reporting inconsistency (identical total accuracy/F1 in Table 1 and Table 2) that makes the numerical evidence even less secure. These issues do not invalidate the method in principle; they show that the current evidence is conditional on obtaining non-circular evaluation. Hence the reader's CONDITIONAL verdict is appropriate and unchanged.","tokens_in":12542,"tokens_out":5116,"duration_ms":58426,"concrete_test":"Run a human rating study on a stratified sample of generated texts (e.g., 100 per class per dataset, across both SWAI and prompt-only baseline) with at least three native-speaker raters per item, using the same label sets (E/I/A or P/N/I) plus a fluency score. Compute human-label accuracy and F1 for each condition, and measure inter-rater agreement and human–GPT-4o agreement (Cohen's κ). If humans judge SWAI outputs as significantly more aligned with the target characteristic than baseline and agree reasonably with GPT-4o, the circularity concern is resolved; if not, the headline claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that corpus-derived token z-scores applied as logit biases produce large, multi-task control gains—is measured almost entirely by GPT-4o, whose judging prompts in Appendix D name the very cues SWAI amplifies. For WIKIPOL, the prompt explicitly lists 'please', gratitute, mitigators, direct imperatives, etc.; for OSE, it lists 'spell-out paraphrases', 'sentence splitting', 'repetition' and similar simplification signals. Since the z-score table is built from the same kind of surface statistics, the judge may label a generation POLITE/ELEMENTARY simply because the injected tokens are present, regardless of whether a human would perceive the text that way. The paper reports no human ratings and no alternative judge that avoids these cues. Consequently, the measured +47%p accuracy and 50×F1 improvements may reflect the judge rewarding the injected signal, not genuine control of perceived readability, politeness, or toxicity.\n\nA second, concrete reporting inconsistency reinforces the need for scrutiny: Table 2's open-ended generation results list the same total accuracy (84.53%) and F1 (0.845) for Llama3.1 8B as Table 1's paraphrase results, even though class-level accuracies (91.73/84.93/92.40) make a total of 84.53% arithmetically implausible. This suggests a possible copy-paste error or data quality issue, further undermining confidence in the numerical evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"SWAI proposes a training-free, inference-time steering method that operates entirely in logit space. From labeled corpora it precomputes z-normalized one-vs-rest log-odds scores for each token (Eq. 1-2); during decoding it adds a constant bias δ to the highest-scoring tokens within the model's top-K candidate set. Experiments on OSE readability, WIKIPOL politeness, and REALTOX toxicity, using Llama 3.1 8B and Llama 3.2 1B and a GPT-4o judge, report large gains over a prompt-only baseline: up to +47p accuracy on OSE and about 55x F1 improvement on REALTOX. The paper also includes class-wise analyses, an open-ended generation study, and an analysis of source-label dependence. The central claim is that statistically grounded logit-table lookup can serve as a simple, architecture-agnostic, and interpretable controller without training or internal-layer access.","tokens_in":12896,"tokens_out":5415,"duration_ms":55986,"significance":"If the reported empirical gains survive scrutiny, SWAI would be a practically valuable baseline: it is genuinely training-free, requires only a precomputed lookup table and a vector addition, and is architecture-agnostic. The statistical formulation with Dirichlet smoothing and variance normalization is standard and reasonable, and the top-K restriction is a sensible safeguard for fluency. However, the current evidence is not yet convincing. The evaluation relies on a judge whose prompt enumerates exactly the surface cues that SWAI's corpus-derived scores are likely to amplify; no human evaluation or cue-agnostic judge is reported. The paper also claims improvement over prior logit-level methods that are never compared, promises ablations that do not appear, and contains a concrete arithmetic inconsistency in Table 2. The conceptual contribution is solid enough to warrant a major revision, but the headline numbers cannot be accepted as stated.","major_comments":[{"comment":"The GPT-4o judge prompt for WIKIPOL explicitly instructs the judge to reward gratitude, 'please' as a softener, indirectness/hedges, and other surface cues; the OSE prompt rewards spell-out paraphrases, sentence splitting, and repetition. SWAI's z-scores are built from corpus labels using exactly this kind of surface statistics, so the judge may label a text POLITE/ELEMENTARY simply because the injected tokens are present. No human ratings or a cue-agnostic judge are reported, so the +47p accuracy and 50x F1 gains may reflect evaluator circularity. Please report human evaluation on a sample, or an alternate judge whose rubric does not name these cues, and show correlation with human perception.","section":"Appendix D; §4.2"},{"comment":"The abstract and §2.1 claim that SWAI 'consistently improves over prompt-based and prior logit-level baselines,' but Table 1 only compares against a prompt-only baseline. DExperts, GeDi, FUDGE, PPLM, and RSA-Control are discussed in Related Work but never evaluated. Without these comparisons, the claim of superiority over prior logit-level steering is unsupported. Add at least DExperts and a simple logit-bias baseline (e.g., uniform bias) with identical evaluation, or remove the claim.","section":"Abstract; §2.1; Table 1"},{"comment":"The Total row for Llama3.1 8B in Table 2 exactly duplicates the SWAI row in Table 1 (accuracy 84.53%, F1 0.845, precision/recall 84.53%, confidence 0.847). The class-level accuracies (91.73%, 84.93%, 92.40%) average approximately 89.7% over balanced classes, not 84.53%. This suggests a copy-paste error or inconsistent computation. Please provide per-class support, the full confusion matrix, and recompute the aggregate metrics.","section":"Table 2"},{"comment":"No error bars, significance tests, or sample sizes are reported for any table. The method has four free hyperparameters (α, K, ρ, δ) plus class-dependent max lengths for OSE; no sensitivity analysis is provided. The abstract promises 'selectivity and lookup-table ablations,' but no such ablation appears in the main text or appendices. Please add variance over random seeds/prompt variants, ablation studies for each hyperparameter, and the promised ablations.","section":"§4; §5; A.1"},{"comment":"The source-label analysis uses the same GPT-4o judge as the main evaluation, so it inherits the cue-circularity problem. In addition, OSE uses different maximum generation lengths per class (650/800/1000 tokens), which introduces length as a confound for the readability judgment. Please report whether length is balanced or controlled in the evaluation.","section":"§5.2; A.1"}],"minor_comments":[{"comment":"The caption says 'prompt-based baselines' but no baseline prompt is defined anywhere. Please provide the exact baseline prompt in the appendix.","section":"Table 1"},{"comment":"'50×F1 improvement' is based on REALTOX F1 going from 0.010 to 0.552; this is an absolute-F1 ratio. State this explicitly, since relative improvement from near zero is uninterpretable.","section":"Abstract"},{"comment":"The notation r(v) is used for the score function, but r is also the class index. Consider renaming the score to s_r(v) or z_r(v) consistently.","section":"Eq. (1)"},{"comment":"The reference to Zou et al. (2023) lists 'and 1 others'; expand to include all authors.","section":"References"},{"comment":"The OSE judge instruction says 'If corruption prevents reliable judging, choose the LOWER label' — this can bias toward ELEMENTARY for long or noisy generations. Please justify or remove.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The paper's simple logit-table method is potentially interesting and would be a useful baseline for the controllable-generation community. However, the evaluation design is currently circular: the judge prompt enumerates the lexical cues SWAI injects. Combined with the missing prior-logit baselines and the Table 2 duplicate row, I would not accept the paper without a major revision that includes human validation or a cue-agnostic judge, real baseline comparisons, the promised ablations, and corrected aggregate metrics. The authors seem capable of addressing these, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear Colleague,\n\nThe short version: SWAI is a clean, plausible method for logit-level style steering, and the paper is worth a serious look, but the evaluation as written overclaims — the abstract says it beats prior logit-level baselines, yet no such baseline appears in the experiments, and there is a clear copy-paste error in Table 2.\n\nWhat is genuinely new: the specific recipe, z-normalized one-vs-rest log-odds from labeled corpora applied as a constant bias only within the model's top-K candidate set, is not in the cited prior work. It is simple, training-free, architecture-agnostic, and interpretable. The method section is clear enough to reimplement, and restricting intervention to high-probability candidates is a sensible way to preserve fluency. If the numbers held up, this would be a useful baseline for the controllable-generation community.\n\nWhere it gets soft. First, the abstract claims improvement over 'prior logit-level baselines,' but the experiments compare only against a prompt-based baseline. DExperts, FUDGE, and GeDi are discussed in related work and should be in the tables. Second, the scoring formula is essentially the Monroe et al. (2008) log-odds z-score, which is not cited; the contribution is the application, not the statistic. Third, the main evaluation uses GPT-4o, and the judge prompts in Appendix D explicitly reward the same surface cues SWAI injects — gratitude, 'please,' hedges, direct imperatives for politeness; simplification signals for OSE. Without human ratings or a judge blind to those cues, the reported gains may partly reflect the judge detecting injected tokens rather than genuine perceived control. Fourth, the paper never says whether the score table is built from the same corpora used for evaluation. Since OSE and WIKIPOL are small, this matters. Fifth, Table 2's total rows are wrong: the class accuracies for Llama3.1 8B (91.73/84.93/92.40) cannot produce a total of 84.53, and the total row exactly matches Table 1's SWAI row, the same for Llama3.2 1B. That looks like a copy-paste.\n\nNone of this kills the central idea; all of it is fixable. The right next step is peer review with a request for revision: add real baselines, human or blind-judge evaluation, explicitly held-out score tables, and correct Table 2. I'd not cite the numbers yet, but I'd cite the method once cleaned up.\n\nBest","headline":"A clean, plausible logit-steering baseline whose abstract overclaims: no prior logit-level baselines tested, judge rubric mirrors injected cues, and Table 2 has a copy-paste error.","tokens_in":13384,"tokens_out":4470,"would_cite":false,"duration_ms":46884,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A table of token z-scores, applied as a logit bias to a model's top candidates, steers generation toward target readability, politeness, or toxicity without training or internal access.","keywords":["controllable text generation","logit steering","inference-time intervention","token-level statistics","z-normalized log-odds","readability control","politeness control","toxicity mitigation"],"falsifier":"Take the same steered generations and have human raters (or a judge model whose rubric is blind to the specific cues in the statistical table) classify readability, politeness, and toxicity; if the large accuracy/F1 gains vanish under human judgment while the original LLM judge still reports them, the claim that the method controls the intended characteristics is falsified. Alternatively, if steering with an inverted sign of the bias fails to push outputs in the opposite direction, the mechanism is not operating as claimed.","tokens_in":12437,"feed_emoji":"🎛️","tokens_out":6056,"duration_ms":59403,"temperature":0.7,"pith_summary":"This paper aims to show that effective control of language-model output characteristics does not require learned controllers, parameter updates, or access to internal activations. It introduces SWAI, which precomputes token-level z-scores from labeled corpora and applies a constant logit bias to high-scoring tokens within a model's top-K candidate set at each decoding step. Across readability, politeness, and toxicity tasks, the authors report large, consistent gains — up to +47 percentage points in accuracy and 50× F1 — over prompt-based and prior logit-level baselines. If true, this gives practitioners a transparent, training-free way to shift generated text toward a desired style or safety property while preserving fluency.","feed_headline":"Up to +47 percentage points: a token-score table steers LLM style","feed_subtitle":"No parameter updates, no hidden-layer access: corpus z-scores rerank a model's top candidates at decoding time.","key_machinery":"The central object is the statistical token score table: for each token v and target characteristic r, z_r(v) is the z-normalized log-odds of v in the characteristic corpus versus all other corpora, with a Dirichlet prior for smoothing and a variance estimate that downweights low-frequency tokens. This table is used at each decoding step by forming a candidate set from the model's top-K logits, selecting the top m tokens by z-score, and adding a fixed bias δ to their logits before softmax. The mechanism's work is to rerank only contextually plausible tokens, so the model's grammar and semantics are preserved while the distribution is shifted toward tokens statistically associated with the ta","core_discovery":"The central claim is that statistically grounded logit steering under high-probability candidates is sufficient for controllable generation. SWAI computes a z-normalized one-vs-rest log-odds score for every token from a target-characteristic corpus versus a contrastive corpus, producing a static lookup table. At decoding time, it restricts attention to a top-K candidate set (K=100), selects the top half by z-score, and adds a fixed bias of 1.5 to their logits before softmax. On OSE, WIKIPOL, and REALTOX, this simple intervention shifts outputs toward the desired reading level, politeness, or toxicity as judged by an LLM, improving on prompts. The message: target-specific statistics under hig","pith_inferences":["A natural extension is to make the bias magnitude adaptive to the local context or to the distance between source and target style; the paper's fixed δ=1.5 may over-steer when the target characteristic conflicts with the source content, as its open-ended OSE results hint.","Because the score table encodes purely lexical associations, characteristics that depend on syntax or discourse (e.g., some forms of formality or persuasion) may need n-gram or sentence-level statistics; the current token-level table is a first representation.","The method could be combined with prompt-based control, using the statistical bias to stabilize what prompts leave underspecified; the paper only compares to prompts, not to their combination.","If user-adaptive style is of interest, the z-score table can be updated per user or per domain without retraining, since it is just a lookup; this would be a cheap way to personalize style."],"forward_implications":["Any autoregressive model can be steered at decoding time by table lookup, with negligible overhead, regardless of architecture or instruction tuning.","Safety-relevant control such as toxicity mitigation becomes feasible without training a discriminator or editing weights, recovering meaningful recall while keeping high precision.","Control strength is explicitly tunable through three hyperparameters — bias magnitude, number of favored tokens, and candidate-set size — giving a granular trade-off between adherence and fluency.","Because the score table is external and static, the same table can be reused across models, and new characteristics only require labeling a small corpus.","The method's transparency makes it possible to audit which tokens the intervention favors, unlike prompt or activation steering."],"fun_headline_variants":["Logit-level steering: corpus z-scores rerank top-K candidates","No training, no hidden-layer access: token table steers LLMs","Token-score table biases top-K logits for style control","SWAI: static token stats steer reading level, politeness, toxicity","Training-free: z-normalized log-odds bias top tokens for control"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The reported gains are measured by an LLM judge whose evaluation prompt names the same surface cues SWAI amplifies (e.g., 'please', gratitude, simplification signals), so the central claim presupposes that this judge is a faithful proxy for human perception of the target characteristics rather than rewarding the injected lexical signal.","fun_headline_variants_meta":{"raw":{"variants":["Logit-level steering: corpus z-scores rerank top-K candidates","No training, no hidden-layer access: token table steers LLMs","Token-score table biases top-K logits for style control","SWAI: static token stats steer reading level, politeness, toxicity","Training-free: z-normalized log-odds bias top tokens for control"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000268,"raw_usage":{"total_tokens":1444,"prompt_tokens":726,"completion_tokens":718,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":470,"completion_tokens_details":{"reasoning_tokens":625}},"tokens_in":470,"tokens_out":718,"duration_ms":7485,"temperature":1.0,"reasoning_tokens":625,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T10:08:27.074620+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same steered generations and have human raters (or a judge model whose rubric is blind to the specific cues in the statistical table) classify readability, politeness, and toxicity; if the large accuracy/F1 gains vanish under human judgment while the original LLM judge still reports them, the claim that the method controls the intended characteristics is falsified. Alternatively, if steering with an inverted sign of the bias fails to push outputs in the opposite direction, the mechanism is not operating as claimed.","supporting_citations":[],"review_version":1}