{"id":"f072db4d-86e8-4734-a9ce-65d27304b61c","arxiv_id":"2608.06069","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A token-level, training-free steering framework that improves LLM personalized co-writing by mixing the base model's posterior with a kernel-density estimate from a small user dataset.","lead":"SteerWrite is a training-free method that steers a base LLM's next-token distribution using a small external dataset, without any gradient updates. It aims to turn a local model into a personalized co-writing assistant that reduces human editing effort.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (2) requires a context-dependent likelihood ratio p(x|D2)/p(x|D1), but the algorithm replaces it with a fixed scalar, so the theoretical grounding does not match the implemented method.","rationale":"The reader's weakest assumption points to the silent replacement of the context-dependent likelihood ratio by a global scalar, and this is the most load-bearing flaw I can identify. The abstract's central claim is empirical, but the paper's stated contribution is a 'theoretical insight' that the method 'effectively approximates' an ideal retrained model through posterior correction. If the derivation in Eq. (2) is not the algorithm that is actually run, then the theoretical foundation is invalid, regardless of the experimental numbers. This is not a matter of disagreement with a consensus; it is an internal inconsistency between the derived formula and the implementation. The evaluation concerns (test-set hyperparameter selection, possibly undertuned baselines) are real and reinforce CONDITIONAL, but they are second-order: the gains are large and consistent, and a validation-based rerun might still preserve the qualitative SOTA claim. The theory gap, by contrast, is certain and material: the implemented mixing weight is constant, while the derived weight is context-dependent. Because the paper frames the method as theoretically grounded, this flaw directly undermines the central contribution. A concrete test can settle it by measuring whether the likelihood ratio is actually constant and whether a context-dependent version changes outcomes. The appropriate verdict remains CONDITIONAL: the empirical claim is plausible but the theoretical claim needs either correction or removal, and the requested code/data releases would allow the empirical claim to be checked independently.","tokens_in":19704,"tokens_out":15562,"duration_ms":142332,"concrete_test":"On Qwen3-0.6B-Base, sample 500 test prefixes from UER and estimate the per-prefix log-ratio log[p(x|D2)/p(x|D1)] using kernel density estimates of the prefix hidden state in the supplement D2 versus a held-out general-text sample; report the distribution of these log-ratios. If the interquartile range exceeds about 2.0, the constant-ratio assumption is false. Then implement a context-dependent weight w(x) proportional to p(D2)p(x|D2)/[p(D1)p(x|D1)] using the same retrieval densities, compare against the fixed 0.6 version on UER, and check whether the fixed version is materially worse. If it is not, the reported gains do not stem from the derived posterior correction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The derivation in Section 3 (Eq. 2) expands the ideal mixed-data posterior as p(y|x) proportional to p(D1)p(x|D1)p(y|x,D1) plus p(D2)p(x|D2)p(y|x,D2). After normalization, the weight of the external term is p(D2)p(x|D2) / [p(D1)p(x|D1)], which is context-dependent through the likelihood ratio p(x|D2)/p(x|D1). The implemented SteerWrite never estimates p(x|D2) or p(x|D1); Appendix A fixes log[p(D2)/p(D1)] at 0.6 and uses this scalar as the mixing weight for all contexts, with Appendix E sweeping this value on the UER test set. Rank-based calibration and temporal momentum shape the external distribution, but they do not modulate the mixture weight by the likelihood of the current context under D1 versus D2. Consequently, the algorithm is not the posterior correction derived in Section 3; it is a fixed-weight interpolation resembling kNN-LM plus additional heuristics. The paper's claim that an ideal retrained model is 'effectively approximated' by this procedure is therefore unsupported. If p(x|D2)/p(x|D1) varies across contexts, the derived formula does not imply the implemented rule, and the central theoretical contribution collapses to post-hoc motivation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SteerWrite, a training-free framework for token-level personalized co-writing. At each decoding step, SteerWrite retrieves contexts from a small external dataset using the base model's hidden states, calibrates the retrieval weights with rank-based calibration and temporal momentum, applies frequency scaling, estimates an external next-token distribution via a KDE-style aggregation, and mixes this distribution with the base model's posterior. The authors derive a Bayesian posterior mixture in Section 3, then evaluate on four domain datasets (CodeNet, HMR, UER, Law) with Qwen3-0.6B/1.7B/4B and Llama-3.2 models, reporting large gains in Levenshtein-distance and keystroke reduction as well as similarity metrics. The paper also contributes a benchmark, an evaluation protocol, ablations, runtime measurements, and a public code release.","tokens_in":19951,"tokens_out":7407,"duration_ms":70600,"significance":"The benchmark datasets and evaluation protocol for interactive co-writing are potentially useful, and the empirical gains are large, consistent across model families, and supported by ablations showing that the two stabilization components matter. The paper also reports latency figures and a model-family generalization study, which strengthen the practical contribution. However, the claimed theoretical grounding does not match the implemented algorithm: the derivation requires a context-dependent likelihood ratio that the implementation never computes, and key hyperparameters appear to be selected on the test set. These issues affect the central 'theoretically derived' contribution and the reliability of the reported SOTA margins, though the engineering contribution remains plausible if the theoretical framing is revised.","major_comments":[{"comment":"The derivation of the mixed-data posterior contains the context-dependent likelihood ratio p(x|D2)/p(x|D1), but the implemented SteerWrite never estimates this ratio. Appendix A fixes log[p(D2)/p(D1)] = 0.6 and uses this scalar as the mixture weight for all contexts; rank calibration, temporal momentum, and frequency scaling affect the shape of the external distribution but do not modulate the mixture weight by the likelihood of the current context under D1 versus D2. The algorithm is therefore not the posterior correction derived in Section 3, and the claim that an ideal retrained model is 'effectively approximated' by the procedure is unsupported. The theory section should either be reframed as motivation for a heuristic interpolation, or the likelihood ratio must be estimated and included in the mixture.","section":"Section 3, Eqs. (2)-(3); Appendix A/E"},{"comment":"The log data ratio is swept over the UER test set (values 0.0, 0.2, 0.4, 0.6, 0.8, 1.0) and the best value 0.6 is then fixed for all four datasets, so the headline numbers in Table 1 are produced with a hyperparameter chosen on the test set. The same concern applies to the per-model choices of the similarity metric and lambda_2 described in Appendix A as 'according to the empirical behavior' without a held-out validation protocol. Please add a validation split or nested cross-validation and report how Table 1's margins change under this protocol; otherwise the SOTA claims are not statistically trustworthy.","section":"Appendix E, Table 7; Appendix A"},{"comment":"The derivation in Section 3 approximates p(y|x,D2) with a kernel density estimate over neighboring contexts, but the implementation replaces kernel weights with a rank-based template, temporal momentum, and frequency scaling. The ablation in Table 8 shows the components are important, but it does not show that the resulting distribution is a principled approximation of the derived posterior. Please clarify whether Section 3 is intended as a formal derivation or as an intuitive motivation; the current text conflates the two and overstates the theoretical grounding.","section":"Section 3, Eqs. (6)-(7); Section 4.3, Eqs. (8)-(10)"}],"minor_comments":[{"comment":"The phrase 'wherewL+1 is the special<EOS> token' contains missing spacing and formatting; it should read 'where w_{L+1} is the special <EOS> token'.","section":"Section 4.2"},{"comment":"The text sets log[p(D2)/p(D1)] = 1.0 for the visualization, while Appendix A reports 0.6 for the main experiments; please explain or reconcile this inconsistency.","section":"Figure 2"},{"comment":"Several entries lack spacing (e.g., '29.4574.9279.49' in the UER RAG row for Qwen3-1.7B-Base), making the table hard to read; please reformat all numeric entries consistently.","section":"Table 1"},{"comment":"The rank template B and the 'top fraction' (e.g., 0.1%) are never specified numerically; please provide exact values in Appendix A for reproducibility.","section":"Section 4.3"},{"comment":"The text says greedy decoding is employed, but Figure 1 and Section 4.1 describe 'sampling' from the mixture posterior; please clarify whether tokens are selected by argmax or by sampling.","section":"Section 5.2"},{"comment":"The keystroke metric is described only in a footnote; please provide the exact formula so readers can reproduce the reported reductions.","section":"Section 5.3"},{"comment":"The paper claims 'state-of-the-art' performance, but the comparison is limited to seven training-free baselines; please qualify the claim as state-of-the-art among training-free methods or include additional baselines.","section":"Abstract and Section 6"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The punchline: this is a solid empirical paper that overstates its theoretical grounding. The actual algorithm—token-level retrieval from a small datastore, rank-based calibration, temporal momentum, frequency scaling, and a fixed scalar mixture weight—works well and the gains are large and consistent. But the derivation in Section 3 presents a Bayesian posterior with a context-dependent likelihood ratio p(x|D2)/p(x|D1), and the implementation never estimates that ratio. Instead, it replaces it with a global scalar log(p(D2)/p(D1)) set to 0.6, tuned on the test set (Appendix E, Table 7). So the theory does not ground the algorithm; the method is more honestly described as a well-tuned variant of kNN-LM with stabilizers.\n\nWhat's genuinely new: the combination of these stabilizers for co-writing, the four-domain benchmark (including the new HMR and UER clinical datasets), and the evaluation protocol that quantifies editing effort. The ablations show both rank calibration and momentum are necessary, and the runtime analysis demonstrates the overhead is acceptable. The improvements over baselines are so large and uniform that they are unlikely to be mere artifacts.\n\nWhere it's soft: the theory-practice gap is the main issue. The scalar data ratio is a free parameter fit to the same evaluation, which inflates the reported numbers. The similarity metric and other hyperparameters are chosen per model based on \"empirical behavior,\" which is vague. The test/supplementary split from the same corpus is reasonable for personalization but likely flatters templated domains like medical reports. I'd also want to see kNN-LM with better-tuned hyperparameters, since the gap with SteerWrite is so wide that it might be partly due to baseline under-tuning.\n\nNone of this sinks the paper. The empirical contribution is real and the benchmark is useful. But the theoretical claims need to be reined in, and the reproducibility issues need fixing: move hyperparameter selection to a validation split, release the code and datasets with commit hashes, and clarify what is derived versus what is heuristic.\n\nWho should read it: NLP researchers interested in inference-time adaptation or small-model personalization, and anyone building co-writing systems. It deserves a serious referee, but the revision needs to be substantive, not cosmetic.\n\nRecommendation: send to peer review with major revision. The system works; the framing needs to match reality.","headline":"Strong empirical system paper whose theoretical framing overclaims; the algorithm is a tuned scalar-weighted interpolation, not the derived posterior.","tokens_in":20540,"tokens_out":3199,"would_cite":true,"duration_ms":36543,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A training-free, token-level steering method lets a small base model outperform a much larger one on specialized co-writing tasks.","keywords":["training-free adaptation","token-level steering","co-writing","personalized LLM","kernel density estimation","retrieval-augmented generation","kNN-LM","Levenshtein distance"],"falsifier":"Estimate $\\log p(x|D_2)/p(x|D_1)$ for many prefixes $x$ from the held-out test sets using a density model or an auxiliary language model trained on both corpora. If the ratio varies by more than a small margin across contexts, then the fixed scalar 0.6 in the implemented algorithm does not match the derivation; a second experiment would compare SteerWrite with a context-dependent ratio estimate and with the scalar set per-context by an oracle, and show whether the scalar version is actually recovering the theoretical mixture.","tokens_in":19442,"feed_emoji":"✍️","tokens_out":5327,"duration_ms":44413,"temperature":0.7,"pith_summary":"SteerWrite claims that a large language model can be personalized to a specialized writing style at inference time, with no gradient updates, by correcting the next-token probability using a small external dataset. The paper derives this correction as a Bayesian mixture: the ideal model retrained on the union of pretraining data and the user's data is a weighted combination of the original model's distribution and a distribution built from retrieved, similar contexts in the user's dataset. It then implements that correction with a datastore of hidden states from the base model, rank-based weighting, temporal smoothing, and frequency penalties. On four domain datasets (code, two kinds of medical reports, and law) and on three model sizes, the method reports state-of-the-art reductions in edit distance and keystrokes, and the smallest 0.6B model with steering outperforms the unsteered 4B model on all datasets. If true, this provides a practical route to co-writing assistants that can be updated instantly with new user data, without training.","feed_headline":"A 0.6B model with token steering outwrites a 4B model","feed_subtitle":"Adding a small user dataset at inference time cuts editing effort and keystrokes on medical, legal, and code writing.","key_machinery":"The central object is the mixture posterior in Eq. (2): $p(y|x,D)=p(D_1|x)p(y|x,D_1)+p(D_2|x)p(y|x,D_2)$, with $p(D_i|x)\\propto p(D_i)p(x|D_i)$. The paper approximates $p(y|x,D_1)$ with the pretrained model's own output distribution and estimates $p(y|x,D_2)$ from a datastore of hidden states taken from the base model's prefill pass over the user dataset. The carrying mechanisms are (1) rank-based calibration, which replaces raw similarity scores with a fixed sparse template so only a small top fraction of retrieved contexts contributes; (2) temporal momentum, which blends current retrieval weights with the previous step's smoothed weights to preserve sequential continuity; and (3) frequency scaling, which down-weights globally frequent tokens to prevent repetition loops. Together these turn a sparse, noisy retrieval signal into a stable token-level steering distribution that is merged with the base posterior before sampling.","core_discovery":"On its own terms, the paper's discovery is that the difference between a generic base model and a model specialized to a small user or domain dataset can be approximated as an inference-time mixture of two token-level distributions. Starting from the posterior over mixed data, the paper writes the ideal retrained model's next-token probability as $p(y|x,D)=p(D_1|x)p(y|x,D_1)+p(D_2|x)p(y|x,D_2)$ and then argues that the first component is the pretrained model itself, while the second can be estimated from retrieved similar contexts in $D_2$ using kernel density estimation. This turns personalization into a local, token-level correction problem: retrieve the nearest contexts in the user's data to the current hidden state, weight them with a calibrated sparse template, smooth the weights across time steps, penalize overly frequent tokens, and sample from the merged distribution. The empirical claim is that this correction outperforms prompt-level RAG and the standard kNN-LM interpolation across all four datasets and most metrics, and that it lets a 0.6-billion-parameter base model beat a standalone 4-billion-parameter model on all four evaluation sets.","pith_inferences":["The context-dependent likelihood ratio $p(x|D_2)/p(x|D_1)$ is not measured in the paper; the method fixes $\\log(p(D_2)/p(D_1))=0.6$. A testable extension would estimate this ratio with a lightweight density model and use it per context, which would show whether the remaining margin comes from the Bayesian correction or from the hand-tuned scalar.","Because the datastore is built from the base model's own hidden states, the same mechanism could in principle steer non-textual generation if hidden states from other modalities can be aligned; the paper names multimodality as future work but does not test it.","The keystroke and Levenshtein reductions are measured against verbatim ground truth, so gains on AWV and Qwen3 similarity suggest the method is not simply copying retrieved suffixes; a useful stress test is to see whether performance persists when the supplementary dataset contains deliberately varied phrasings of the same content.","The framework treats the base model's distribution as a faithful $p(y|x,D_1)$; if a model family's pretraining distribution is poorly calibrated, the mixture correction may need rebalancing, which the paper's scalar already implicitly does."],"forward_implications":["A user's new writing data can be injected into a deployed model in seconds: only an offline prefill pass over the supplementary dataset is needed, and updates require re-encoding the new dataset, not retraining.","Small models become competitive with much larger ones on domain-specific co-writing: the 0.6B base model with SteerWrite outperforms the standalone 4B base model on all four datasets tested.","Interactive stream-based co-writing is feasible at modest latency: reported time-to-first-token stays below 47 ms and time-per-output-token below 30 ms across tested model sizes.","Shorter, periodic suggestions are more efficient than long completions: the generation-length study shows editing-effort reduction degrades with longer windows and can go negative for baselines.","The components generalise across model families: on Llama-3.2-1B and 3B, SteerWrite keeps reductions above 50 points where RAG yields near-zero or negative keystroke reductions."],"supporting_citations":[{"why":"Supplies the kNN-LM interpolation baseline that SteerWrite extends and must beat, and the token-level retrieval datastore idea.","marker":"(Khandelwal et al. 2019)"},{"why":"Defines the retrieval-augmented generation setup that the paper contrasts with token-level steering.","marker":"(Lewis et al. 2020)"},{"why":"Provides the Context-Aware Decoding baseline and the context-drift motivation for continuous steering.","marker":"(Shi et al. 2024)"},{"why":"Supplies the Qwen3 base models used in all main experiments.","marker":"(Yang et al. 2025)"},{"why":"Defines the edit-distance metric used as the primary measure of human editing effort.","marker":"(Levenshtein et al. 1966)"},{"why":"Argues kNN-LM does not improve open-ended generation, motivating the calibration and momentum mechanisms.","marker":"(Wang et al. 2023a)"},{"why":"Supplies the kernel density estimation viewpoint behind the relaxed neighborhood weighting in Eq. (7).","marker":"(Davis, Lii, and Politis 2011)"}],"fun_headline_variants":["Token steering lets 0.6B model outwrite 4B","No fine-tuning: token-level steering personalizes LLMs","Training-free token mix beats prompt RAG for co-writing","Inference-time kernel steering slashes editing effort","Small model, big win: token steering trumps size"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the influence of the user's dataset relative to pretraining is a single constant across all contexts, so one fixed scalar (0.6 in log space) can stand in for the context-dependent likelihood ratio $p(x|D_2)/p(x|D_1)$; the paper does not measure that ratio.","fun_headline_variants_meta":{"raw":{"variants":["Token steering lets 0.6B model outwrite 4B","No fine-tuning: token-level steering personalizes LLMs","Training-free token mix beats prompt RAG for co-writing","Inference-time kernel steering slashes editing effort","Small model, big win: token steering trumps size"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000302,"raw_usage":{"total_tokens":1721,"prompt_tokens":912,"completion_tokens":809,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":528,"completion_tokens_details":{"reasoning_tokens":727}},"tokens_in":528,"tokens_out":809,"duration_ms":7877,"temperature":1.0,"reasoning_tokens":727,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:32:43.950023+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Estimate $\\log p(x|D_2)/p(x|D_1)$ for many prefixes $x$ from the held-out test sets using a density model or an auxiliary language model trained on both corpora. If the ratio varies by more than a small margin across contexts, then the fixed scalar 0.6 in the implemented algorithm does not match the derivation; a second experiment would compare SteerWrite with a context-dependent ratio estimate and with the scalar set per-context by an oracle, and show whether the scalar version is actually recovering the theoretical mixture.","supporting_citations":[],"review_version":1}