{"id":"565d150d-4d14-450d-9e1b-03de92f58eb2","arxiv_id":"2411.18247","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Contrastive activation steering on about 30 Italian prompts matches or beats instruction fine-tuning for Italian on MMLU, HellaSwag, and ARC, with markedly higher Italian language consistency.","lead":"Steering a language model's internal activations toward Italian, using under 100 example prompts, makes Llama 3, Phi 3, and Llama 2 answer Italian benchmarks about as well as, and sometimes better than, models fine-tuned on 240,000 Italian examples. The cheap inference-time method also produces more consistently Italian text, with the largest gains on older models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Appendix B regex answer extractor likely under-scores the fine-tuned ANITA baseline on HellaSwag, making the headline 'comparable or better than fine-tuning' rest on a possible scoring artifact.","rationale":"The reader's CONDITIONAL verdict is appropriate; my concern selects one of the two fragile premises the reader listed. The regex extractor issue is the most load-bearing because the headline is a comparative claim, and Table 1's largest gap favoring steering (HellaSwag) is exactly the benchmark where ANITA's output format is most likely to be penalized by the extractor. MMLU and ARC differences are within one to two points, so without fixing the scoring asymmetry the paper cannot support 'comparable or better than fine-tuning' beyond a null result. I do not see a circularity or soundness problem: the steering mechanism, data efficiency, and qualitative Italian consistency are plausible, the baselines come from external sources, and the 30-prompt direction transfer is an empirical design choice rather than an internal inconsistency. The concern is concrete and testable, and the existing CONDITIONAL verdict already reflects the need for re-evaluation; no verdict change is needed, but the evaluation fix should be the condition.","tokens_in":12126,"tokens_out":6082,"duration_ms":54978,"concrete_test":"Re-score the Llama 3 models on MMLU, HellaSwag, and ARC with a permissive answer extractor that accepts any leading answer token (e.g., regex ^\\s*\\(?([ABCD])\\)? on the first decoded token) or with lm-evaluation-harness log-likelihood scoring, for both ANITA and the ITA-steered model. If ANITA's HellaSwag score rises by more than about 3 points, or reaches/beats the steered 50.00, the headline comparative claim is not supported by the current experiments.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Italian steering matches 240K-example fine-tuning is carried by Table 1. The evaluation uses a regex extractor (Appendix B) that requires a colon or 'e’' immediately before the answer letter. A response beginning with a bare 'A', '(A)', or 'A\\n...' is not matched. Fine-tuned ANITA is precisely the model that produces such outputs: Table 2 shows ANITA answering 'A\\n(mixed Thai and Arabic text...)' and 'B \\n(Ela explicação)...'—both would fail the separator requirement. The largest Llama 3 advantage for steering is on HellaSwag (50.00 vs 42.49, +7.5 points), while MMLU and ARC are within 0.94 and 1.16 points, respectively. If ANITA's HellaSwag answers are routinely bare-letter, the reported gap is a scoring artifact and the claim reduces to 'comparable within noise'—especially since no error bars or significance tests are reported. The 30-prompt transferability is a design assumption, but the scoring asymmetry is more load-bearing because it can invalidate the headline comparison even if the steering mechanism works as intended.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript investigates contrastive activation steering as a low-resource alternative to instruction tuning for adapting English-centric LLMs to Italian. The authors extract a steering vector from the mean activation difference between English and Italian (or Italian-answer-to-English-question) Alpaca prompts, then add this direction to all attention-head outputs during generation, with a diminishing intensity. They evaluate on Llama 3 8B Instruct, Phi 3 mini Instruct, and Llama 2 7B Instruct on Italian MMLU, HellaSwag, ARC, and a language-detection metric, comparing against the fine-tuned Italian models ANITA and LLaMAntino 2. Their headline finding is that steering with 30 prompts and no training performs comparably to or better than fine-tuning on several benchmarks; for example, ITA steering on Llama 3 scores 55.95 vs 55.01 (ANITA) on MMLU, 50.00 vs 42.49 on HellaSwag, and 71.38 vs 72.54 on ARC, while also improving Italian-language consistency (0.996 vs 0.715). On Llama 2 ARC, steering ITA-full raises accuracy from 32.84 to 41.06, exceeding the fine-tuned model's 34.98.","tokens_in":12260,"tokens_out":10640,"duration_ms":92246,"significance":"The paper addresses a practical problem—language adaptation without expensive fine-tuning—and the proposed method is clearly described and easy to reproduce from the text, including the contrastive dataset construction, K=30, the steering intensity, and the evaluation protocol. The reported gain on Llama 2 ARC is striking and, if real, would be an important demonstration that activation steering can transfer to unseen reasoning benchmarks. At the same time, the central comparison against fine-tuning is weakened by two issues: a regex-based answer extractor that appears to penalize ANITA's response format, and a complete absence of uncertainty estimates. Substantial revision is therefore needed to determine whether the headline claim survives a format-agnostic evaluation.","major_comments":[{"comment":"The regex evaluation in Appendix B requires a separator (':' or 'e’') immediately before the answer letter. Table 2 contains ANITA answers marked correct that would not match this pattern: 'A\\n(mixed Thai...', 'B \\n(Ela explicação)...', 'C. Gli atomi rimangono gli stessi.', and 'D (un prato erboso...'. Since the steered models in the same table tend to produce answers like 'La risposta corretta è (B)...', which do match, the scoring is asymmetric and may under-report ANITA's accuracy on HellaSwag and other benchmarks. Because the central claim of comparability with fine-tuning rests on these numbers, please re-evaluate all models with a format-independent answer extractor (e.g., first-letter match after normalizing punctuation) or report the fraction of responses that fail the regex for each model, and update the tables accordingly.","section":"§4.2, Table 2, Appendix B"},{"comment":"All reported scores are point estimates from single deterministic runs with no error bars, confidence intervals, or significance tests. For ARC, which contains roughly 1,000 items, the difference between ITA (71.38) and ANITA (72.54) is well within one standard error of the estimate; the MMLU gap of 0.94 points is similarly not interpretable without variance information. To support the claim that steering is 'comparable to, or even better than' fine-tuning, the authors should provide bootstrap confidence intervals over the benchmark items (or an equivalent uncertainty quantification).","section":"§4.2, Tables 1 and 3"},{"comment":"The method relies on two user-set hyperparameters, K (number of contrastive prompts) and the steering intensity valmax, without any sensitivity analysis. Since the paper's main premise is that a very small number of examples (30) suffices to produce a transferable language direction, the authors should demonstrate that the reported outcomes do not sharply depend on these choices. I request a sensitivity study that varies K (e.g., 10, 30, 60) and valmax (e.g., 0.5–3.0) for at least one model/benchmark pair and reports the performance curve.","section":"§3, Eq. (3), 'Steering vector extraction'"}],"minor_comments":[{"comment":"The section heading 'Promtps' contains a typo; it should be 'Prompts'.","section":"Appendix A"},{"comment":"The two regex patterns are presented without a clear statement of how they are combined (e.g., alternatives or sequential), and without a fallback rule for responses that match neither; please specify the exact scoring function, including what is scored when no match is found.","section":"Appendix B"},{"comment":"The claim that steering uses 'less than 0.5%' of the fine-tuning data is imprecise: 30 examples is 0.0125% of 240K; giving the exact count and percentage would be more informative.","section":"Section 5 (and Abstract)"},{"comment":"Tables 7 and 8 contain only a few hand-picked generations; the abstract's claim of 'higher quality' generations is not supported by a systematic human evaluation or a quantitative metric beyond lang-detect. Please either soften the claim or provide stronger evidence.","section":"§4.2, Generation quality"},{"comment":"Please state whether the code, the extracted steering vectors, and the exact benchmark prompts will be made publicly available; this is especially relevant for a method whose value lies in being cheap and fast.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The paper appears to have originated as a workshop-style contribution (CLiC-it). For a journal, the evaluation needs to meet higher standards. The biggest risk is that the answer-extraction artifact inflates the advantage over ANITA; I recommend asking the authors to re-evaluate with a format-agnostic extractor before making a final decision, rather than rejecting on the basis of the current numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Honest take: this is a solid, well-written application paper, but the central comparative claim is on shakier ground than the abstract suggests. What's genuinely new here is showing that contrastive activation steering, with about 30 prompt pairs, can push an instruction-tuned Llama/Phi model toward Italian output and match or beat a 240K-example fine-tuned Italian model on some benchmarks. The mechanism is clearly described, the Llama 2 ARC results (32.84 to 41.06) are a real effect, and the language-consistency numbers (lang-detect) give credit to the steering approach. The paper is also honest that frontier models already handle Italian well, which is the right context.\n\nThe soft spots, in order of importance. First, the regex-based answer extractor in Appendix B requires a colon or 'e'' before the option letter. The paper's own Table 2 shows ANITA answering with bare letters like 'A\\n...' and 'B \\n...' and 'C. ...', none of which would match. That systematically under-scores the fine-tuned baseline, and the single largest steering-vs-ANITA win (HellaSwag: 50.00 vs 42.49) rides on exactly that comparison. The MMLU and ARC gaps are within a couple of points, so if HellaSwag is a scoring artifact, the claim reduces to 'comparable within noise.' This is fixable—run likelihood-based evaluation or a more permissive extractor—but it's load-bearing. Second, there are no error bars, significance tests, or repeated runs; with single runs, differences of a few points on MMLU/ARC are not meaningful. Third, the 30-prompt steering direction is a plausible design choice, but transferability to completely different benchmarks is asserted more than tested.\n\nThe self-citation of the steering method [4] is not a problem; it's the same technique, and the external benchmarks and baselines from other groups make the evaluation independent.\n\nWho's it for: people working on multilingual adaptation or efficient methods to push pretrained models into low-resource languages. It's a workshop-level contribution with a valuable observation about fine-tuned model instability (the mixed-language ANITA outputs). Deserves a serious referee and likely a revision addressing the evaluation scoring. I'd engage with it.","headline":"Useful application of steering to Italian, but the headline comparison to fine-tuning is undercut by a narrow answer-extraction regex that appears to penalize the fine-tuned baseline.","tokens_in":12901,"tokens_out":4716,"would_cite":false,"duration_ms":47573,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 30-prompt activation push matches fine-tuning for steering an LLM into Italian.","keywords":["activation steering","contrastive activation addition","Italian language adaptation","instruction tuning alternative","inference-time intervention","Italian benchmarking","language consistency","catastrophic forgetting"],"falsifier":"Re-score the fine-tuned ANITA baseline by response likelihood instead of the paper's regex extractor: if ANITA's Italian benchmark scores rise to or above the steered models' scores, the claim of comparable-or-better performance would be undermined.","tokens_in":11809,"feed_emoji":"🇮🇹","tokens_out":8151,"duration_ms":65152,"temperature":0.7,"pith_summary":"This paper claims that activation steering, adding a vector to a model's internal activations during generation, can adapt an English-instructed LLM to Italian as effectively as instruction fine-tuning, but with fewer than 100 prompts and no parameter updates. The authors test Italian steering on Llama 3 8B, Phi 3 3.8B, and Llama 2 7B, comparing against fine-tuned Italian models. They report that the steered Llama 3 matches or beats the fine-tuned ANITA on Italian MMLU, HellaSwag, and ARC, while generating more consistently Italian text. If this holds, steering is a dramatically cheaper route to language adaptation, especially when only machine-translated training data is available.","feed_headline":"30 prompts push an LLM to Italian as well as 240K fine-tuning examples","feed_subtitle":"Adding a language vector during generation beats a fine-tuned Italian model on two of three benchmarks.","key_machinery":"The machinery is the Italian steering vector, defined as the difference between averaged activations over Italian responses and averaged activations over English responses, collected from the last token of each attention head across K = 30 Alpaca-style prompts. A variant called ITA uses English questions with Italian answers, aiming to capture the language-switch direction. At generation time, the vector is added to every layer-head activation with a multiplier alpha that starts at 1.5 and linearly decays to 0 over the generated tokens, so the push is strongest early and fades as the model settles into Italian.","core_discovery":"The central claim is that a gentle push, a contrastive activation steering vector computed from 30 instruction prompts, is enough to make an English-instructed LLM answer Italian benchmarks in Italian, at a level comparable to or better than a model fine-tuned on roughly 240,000 Italian instruction examples. On Llama 3 8B, the ITA steering variant scores 55.95 on Italian MMLU versus 55.01 for ANITA; 50.00 versus 42.49 on HellaSwag; and 71.38 versus 72.54 on ARC, with Italian language detection 0.996 versus 0.715. The authors emphasize that steering pushes the model's language direction without teaching it new facts, so it preserves most of the original model's correct answers, unlike fine-tuning, which loses some. The paper does not present this as a full replacement for fine-tuning where new knowledge must be injected, but as a cheaper and often better alternative when the goal is purely language adaptation.","pith_inferences":["Because steering only reweights existing latent behavior, the method's success should depend on how much Italian the base model saw in pre-training; for languages with negligible pre-training exposure, instruction fine-tuning or additional data would remain necessary.","The extraction uses 30 fixed prompts; varying the prompt set and measuring the variance of downstream scores would test how robust the Italian direction is, a diagnostic the paper does not run.","If the steering vector captures a general language direction, the same recipe might transfer to other languages or to dialect, register, or style control, and possibly be composed with other steering vectors, though the paper only tests Italian.","The regex-based answer extraction likely penalizes ANITA more than the steered models, so a likelihood-based re-scoring could change the ranking on individual benchmarks."],"forward_implications":["Steering with under 100 contrastive prompts can substitute for instruction fine-tuning when the goal is language adaptation, cutting data requirements by more than 99 percent.","Because steering does not update weights, a steered model keeps the original model's correct answers while gaining Italian fluency, avoiding the capability loss observed in fine-tuned ANITA.","The method generalizes across model families (Llama 3, Phi 3, Llama 2), so it is a model-agnostic recipe rather than a one-off hack.","When the only available fine-tuning data is machine-translated, steering is the more effective use of that data, since it only needs a handful of prompts rather than hundreds of thousands."],"supporting_citations":[{"why":"Supplies the Alpaca instruction prompts, in English and machine-translated Italian, used to extract the contrastive activations.","marker":"[1]"},{"why":"Establishes the contrastive steering method and the diminishing intensity schedule (valmax starting at 1.5) that the paper adapts.","marker":"[4]"},{"why":"Provides the contrastive activation addition technique that the steering vector injection builds on.","marker":"[5]"},{"why":"Defines ANITA, the Llama 3-based Italian instruction-tuned model that is the paper's main fine-tuning baseline.","marker":"[9]"},{"why":"Supplies LLaMAntino 2, the fine-tuned Italian Llama 2 model used as baseline in the ARC replication.","marker":"[3]"},{"why":"Motivates the linear representation hypothesis, i.e., concepts as directions in activation space, which the method relies on.","marker":"[10]"},{"why":"Italian MMLU is one of the three benchmarks on which steering is compared with fine-tuning.","marker":"[15]"},{"why":"Italian HellaSwag is one of the three benchmarks on which steering is compared with fine-tuning.","marker":"[16]"},{"why":"Italian ARC challenge is one of the three benchmarks and the one used for the Llama 2 and overlap analyses.","marker":"[17]"},{"why":"Documents the standard response-likelihood evaluation that the paper deliberately replaces with a regex extractor, a choice that affects the reported scores.","marker":"[18]"}],"fun_headline_variants":["Steering with 30 prompts matches or beats 240K fine-tuning for Italian","A gentle push: 30 prompts adapt LLM to Italian, rivaling fine-tuning","Italian steering vector from 30 prompts beats fine-tuning on 2/3 benchmarks","Cheap steering with 30 prompts gives Italian LLM fine-tuning-level results"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a direction computed from 30 contrastive prompts isolates a general, transferable Italianness in the model's activation space, and that the base model already contains enough latent Italian knowledge for the push to make a difference.","fun_headline_variants_meta":{"raw":{"variants":["Steering with 30 prompts matches or beats 240K fine-tuning for Italian","A gentle push: 30 prompts adapt LLM to Italian, rivaling fine-tuning","Italian steering vector from 30 prompts beats fine-tuning on 2/3 benchmarks","Cheap steering with 30 prompts gives Italian LLM fine-tuning-level results"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000667,"raw_usage":{"total_tokens":3005,"prompt_tokens":870,"completion_tokens":2135,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":486,"completion_tokens_details":{"reasoning_tokens":2048}},"tokens_in":486,"tokens_out":2135,"duration_ms":13866,"temperature":1.0,"reasoning_tokens":2048,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:23:14.568216+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-score the fine-tuned ANITA baseline by response likelihood instead of the paper's regex extractor: if ANITA's Italian benchmark scores rise to or above the steered models' scores, the claim of comparable-or-better performance would be undermined.","supporting_citations":[{"cited_title":"Hendrycks, C","cited_arxiv_id":null,"evidence_quote":"Italian MMLU is one of the three benchmarks on which steering is compared with fine-tuning."},{"cited_title":"Taori, I","cited_arxiv_id":null,"evidence_quote":"Supplies the Alpaca instruction prompts, in English and machine-translated Italian, used to extract the contrastive activations."},{"cited_title":"Scalena, G","cited_arxiv_id":null,"evidence_quote":"Establishes the contrastive steering method and the diminishing intensity schedule (valmax starting at 1.5) that the paper adapts."},{"cited_title":"Basile, E","cited_arxiv_id":null,"evidence_quote":"Supplies LLaMAntino 2, the fine-tuned Italian Llama 2 model used as baseline in the ARC replication."}],"review_version":1}