REVIEW 2 major objections 5 minor 37 references
Functional and Secure Code Generation with Task Vectors
T0 review · 2 major / 5 minor · reviewed 2026-07-10 · grok-4.5
Pith's one-line read Task-vector arithmetic can steer coding LLMs so the same completion is both functional and secure, without special decoding.
desk verdict Clean empirical win: LPO task vectors plus a Secure-Anchored operator raise joint functional+secure rates on six coding models with almost no inference cost. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
SecVecCoder: the secure-anchored task-vector operator W_steered = W_base + τ_sec − θ τ_insec, where τ_sec and τ_insec are weight differences obtained by localized preference optimization on paired secure and vulnerable completions of the same instruction.
What would settle it
Re-run the same six models on the full CodeGuard+ main-CWE and unseen-CWE splits after re-selecting θ independently per model (or after expanding the validation set); if the joint sec-pass@1 gains over the base models disappear or reverse for most models, the fixed-θ claim fails.
Extended reading notes
Core claim
LPO-derived secure and insecure task vectors, combined by the secure-anchored rule W_base + τ_sec − θ τ_insec with θ = 1.5, raise the rate of completions that are simultaneously functional and secure by 2.1–36.0 percentage points over the corresponding base models on CodeGuard+, reach the highest joint score on five of six models, and retain most of that gain on CWE classes absent from the training pairs, all while leaving standard autoregressive decoding latency within roughly half a percent of the base model.
Load-bearing premise
The six validation scenarios used to pick the steering operator and the single scalar strength θ = 1.5 are assumed representative enough that the same fixed choice stays near-optimal on the held-out test scenarios.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes SecVecCoder, which constructs secure and insecure task vectors from LPO-fine-tuned LoRA adapters on paired secure/vulnerable code and applies Secure-Anchored arithmetic (W_base + τ_sec − θ τ_insec with θ = 1.5) to produce a single set of weights that generate code that is simultaneously functional and secure. On CodeGuard+, which jointly measures functionality (unit tests) and security (CodeQL), the method improves sec-pass@1 by 2.1–36.0 percentage points over the base model across six coding LLMs from three families, reaches the highest main-CWE sec-pass@1 on five of the six models, generalizes to unseen CWE types by up to 39.1 points, and keeps decoding latency within 0.6 % of the base model while using far less training compute than SafeCoder or SCoDE. Supporting analyses compare three fine-tuning sources, three steering operators, a range of θ values, temperature sensitivity, and component ablations.
Significance. If the empirical results hold, the work supplies a practical, low-overhead route to trustworthy code generation that unifies the benefits of training-time and inference-time defenses: the security signal is baked into the weights once, standard autoregressive decoding is retained, and the steering strength remains modulable. The joint functionality-and-security evaluation on CodeGuard+, the systematic operator/strength/fine-tuning ablations, the training-cost and latency measurements, and the public artifact strengthen the contribution relative to prior secure-code work that often reports security and functionality on separate benchmarks. The approach is immediately usable by practitioners who already fine-tune coding models with LoRA.
major comments (2)
- Section V-A2 and Table V: on DeepSeek-Coder-6.7B the method under-performs SVEN, SafeCoder and DeepGuard on main-CWE sec-pass@1 (68.2 % vs. 75.5 / 79.3 / 68.7). The paper notes that a higher θ might help but does not verify it. Because the abstract and contribution list claim state-of-the-art on five of six models, a short per-model θ sweep (or an explicit statement that the fixed θ = 1.5 choice is deliberately sub-optimal for this model) is needed to keep the SOTA claim precise.
- Section IV-D and V-A1: operator and the single scalar θ = 1.5 are selected on only six validation scenarios, then frozen for all held-out main-CWE and unseen-CWE tests. Figure 3 already shows that LPO Secure-Anchored remains positive across a range of θ and that θ = 1.5 maximises the mean, so the central existence claim is not falsified; nevertheless a leave-one-CWE-out or multi-seed sensitivity check would make the fixed-hyperparameter protocol more robust.
minor comments (5)
- Table I caption and surrounding text: clarify that “sign agreement” is computed only over non-zero parameters and state whether the cosine is taken after flattening all LoRA matrices.
- Figure 3: the seven discrete θ values are hard to distinguish in some panels; adding a small legend or using distinct markers would improve readability.
- Equation (12): the regularisation term that keeps non-security tokens is written with ε, but the hyper-parameter table lists (ε, β, γ) without restating the precise schedule used for the three-to-five-epoch LPO runs.
- Section II-B: a one-sentence comparison of training-corpus size and CWE coverage between the SVEN pairs used here and the larger SafeCoder/SCoDE corpora would help readers interpret the efficiency numbers in Table VI.
- Minor typographical inconsistencies appear in the arXiv header (e.g., “SecVec-Coder” vs. “SecVecCoder”) and in a few table column headers; a final pass would clean them.
Circularity Check
No significant circularity: empirical method selection and external-benchmark evaluation; results not forced by construction or self-citation.
full rationale
SecVecCoder is an empirical engineering paper. Task vectors are defined by ordinary weight differences (Eq. 1: au_sec = W_sec − W_base, au_insec = W_vul − W_base) after independent LoRA fine-tunes; the three steering operators (Eqs. 3–5) are linear combinations whose coefficients are free scalars. The paper selects the fine-tuning source (LPO), operator (Secure-Anchored) and scalar heta = 1.5 by inspecting cosine/sign-agreement diagnostics (Table I) and a six-scenario validation sweep (Figure 3, Section V-A1), then reports sec-pass@1 on held-out CodeGuard+ main-CWE and unseen-CWE splits against external baselines (SVEN, SafeCoder, CoSec, SCoDE, DeepGuard). None of these steps reduces a claimed “prediction” to its own inputs by definition: the reported percentage-point gains are measured quantities, not algebraic identities. LPO and task-vector arithmetic are cited from independent prior work (Hasan et al., Ilharco et al.); no uniqueness theorem or ansatz is imported from the present authors. Hyper-parameter choice on a small validation set is ordinary practice and does not make the test-set numbers tautological. Consequently the derivation chain contains no self-definitional loop, fitted-input-as-prediction, load-bearing self-citation, or renaming of a known result.
Assumptions & free parameters
free parameters (3)
- steering strength θ =
1.5
- LoRA rank / alpha / dropout =
16 / 32 / 0.1
- LPO (ε, β, γ) =
(0.05, 10.0, 5.4)
assumptions (3)
- domain assumption Linear task-vector arithmetic in weight space approximately composes the behaviors of the source fine-tunes.
- domain assumption CodeGuard+ unit tests plus CodeQL CWE queries jointly measure ‘trustworthy’ code.
- domain assumption Paired secure/vulnerable completions that share the same prompt isolate security-relevant token differences.
invented entities (1)
-
Secure-Anchored steering operator
Cite this review
Pith. "Pith review of Functional and Secure Code Generation with Task Vectors." pith.science (2026). https://pith.science/paper/XC7CG4HB
@misc{pith2026260707881,
author = {Pith},
title = {Pith review of: Functional and Secure Code Generation with Task Vectors},
year = {2026},
howpublished = {\url{https://pith.science/paper/XC7CG4HB}},
note = {Machine review of arXiv:2607.07881}
}
read the original abstract
Large language models (LLMs) are increasingly used for code generation, but they struggle to generate functional code free of security vulnerabilities. Prior work to improve the secure code generation abilities of such coding LLMs has largely focused on evaluating code functionality and security separately using different datasets, or focused on finding vulnerabilities post-generation. At the same time, the text-generation domain has seen significant work on alignment techniques, where models are tuned such that their outputs exhibit certain qualities (e.g., helpfulness, harmlessness). Of particular interest is task-vector arithmetic, where linear operations on LLM weights can be used to arbitrarily enhance alignment while incurring only minimal computational overhead. We develop a novel method, SecVecCoder, leveraging task vectors to produce trustworthy code that is simultaneously functional and secure without the need for post-generation adjustment. Across six coding LLMs from three families on the CodeGuard+ benchmark, SecVecCoder improves the rate of trustworthy code completions by 2.1-36.0 percentage points over the base model, with improvements on unseen CWE types reaching up to 39.1 percentage points. Since the effectiveness of the coding LLM relies only on changing the model weights, SecVecCoder requires no method-specific decoding and hence achieves a decoding latency within 0.6% of the base model's, on average.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Evaluating Large Language Models Trained on Code
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Her...
work page Pith review arXiv 2021
-
[2]
A survey on large language models for software engineering,
Q. Zhang, C. Fang, Y . Xie, Y . Zhang, S. Yu, W. Sun, Y . Yang, and Z. Chen, “A survey on large language models for software engineering,”Sci. China Inf. Sci., vol. 69, no. 4, 2026. [Online]. Available: https://doi.org/10.1007/s11432-025-4670-0
-
[3]
Asleep at the keyboard? assessing the security of github copilot’s code contributions,
H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri, “Asleep at the keyboard? assessing the security of github copilot’s code contributions,”Commun. ACM, vol. 68, no. 2, p. 96–105, Jan. 2025. [Online]. Available: https://doi.org/10.1145/3610721
-
[4]
N. Perry, M. Srivastava, D. Kumar, and D. Boneh, “Do users write more insecure code with ai assistants?” inProceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, ser. CCS ’23. New York, NY , USA: Association for Computing Machinery, 2023, p. 2785–2799. [Online]. Available: https://doi.org/10.1145/3576915.3623157
-
[5]
Lost at c: a user study on the security implications of large language model code assistants,
G. Sandoval, H. Pearce, T. Nys, R. Karri, S. Garg, and B. Dolan-Gavitt, “Lost at c: a user study on the security implications of large language model code assistants,” inProceedings of the 32nd USENIX Conference on Security Symposium, ser. SEC ’23. USA: USENIX Association, 2023
work page 2023
-
[6]
A user-centered security evaluation of copilot,
O. Asare, M. Nagappan, and N. Asokan, “A user-centered security evaluation of copilot,” inProceedings of the IEEE/ACM 46th International Conference on Software Engineering, ser. ICSE ’24. New York, NY , USA: Association for Computing Machinery, 2024. [Online]. Available: https://doi.org/10.1145/3597503.3639154
-
[7]
Large language models for code: Security hardening and adversarial testing,
J. He and M. Vechev, “Large language models for code: Security hardening and adversarial testing,” inProceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, ser. CCS ’23. New York, NY , USA: Association for Computing Machinery, 2023, p. 1865–1879. [Online]. Available: https://doi.org/10.1145/3576 915.3623175
-
[8]
Instruction tuning for secure code generation,
J. He, M. Vero, G. Krasnopolska, and M. Vechev, “Instruction tuning for secure code generation,” inProceedings of the 41st International Conference on Machine Learning, ser. ICML’24. JMLR.org, 2024
work page 2024
Show all 37 references
-
[9]
DeepGuard: Secure code generation via multi-layer semantic aggregation,
L. Huang, Z. Liu, Y . Wu, T. Yin, D. li, J. Bi, N. Mu, H. Zhang, and M. Yan, “DeepGuard: Secure code generation via multi-layer semantic aggregation,” inProceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), M. Liakata,...
2026
-
[10]
Steer your model: Secure code generation with contrastive decoding,
L. Huang, M. Yan, T. Yin, W. Sun, Z. Liu, H. Zhang, and D. Lo, “Steer your model: Secure code generation with contrastive decoding,”IEEE Transactions on Software Engineering, vol. 52, no. 3, pp. 809–834, 2026
2026
-
[11]
Vulnerability detection with code language models: How far are we?
Y . Ding, Y . Fu, O. Ibrahim, C. Sitawarin, X. Chen, B. Alomair, D. Wagner, B. Ray, and Y . Chen, “Vulnerability detection with code language models: How far are we?” inProceedings of the IEEE/ACM 47th International Conference on Software Engineering, ser. ICSE ’25. IEEE Press...
2025 doi
-
[12]
A systematic literature review on automated software vulnerability detection using machine learning,
N. Shiri Harzevili, A. Boaye Belle, J. Wang, S. Wang, Z. M. J. Jiang, and N. Nagappan, “A systematic literature review on automated software vulnerability detection using machine learning,”ACM Comput. Surv., vol. 57, no. 3, Nov. 2024. [Online]. Available: https://doi.org/10.11...
2024 doi
-
[13]
Vulrepair: a t5-based automated software vulnerability repair,
M. Fu, C. Tantithamthavorn, T. Le, V . Nguyen, and D. Phung, “Vulrepair: a t5-based automated software vulnerability repair,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC/FSE 2...
2022 doi
-
[14]
SeCuRepair: Semantics-aligned, curriculum-driven, and reasoning- enhanced vulnerability repair framework,
C. Yang, T. Zhang, J. Jiang, X. Zhou, H. Tian, M. Du, J. Shi, J. Chen, Y . Li, E. L. Ouh, L. K. Shar, and D. Lo, “SeCuRepair: Semantics-aligned, curriculum-driven, and reasoning- enhanced vulnerability repair framework,” inProceedings of the 64th Annual Meeting of the Associat...
2026
-
[15]
Examining zero-shot vulnerability repair with large language models,
H. Pearce, B. Tan, B. Ahmad, R. Karri, and B. Dolan-Gavitt, “Examining zero-shot vulnerability repair with large language models,” in2023 IEEE Symposium on Security and Privacy (SP), 2023, pp. 2339–2356
2023
-
[16]
Cosec: On-the-fly security hardening of code llms via supervised co-decoding,
D. Li, M. Yan, Y . Zhang, Z. Liu, C. Liu, X. Zhang, T. Chen, and D. Lo, “Cosec: On-the-fly security hardening of code llms via supervised co-decoding,” inProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ser. ISSTA
-
[17]
1428–1439
New York, NY , USA: Association for Computing Machinery, 2024, p. 1428–1439. [Online]. Available: https://doi.org/10.1145/3650 212.3680371
2024 doi
-
[18]
Constrained decoding for secure code generation,
Y . Fu, E. Baker, Y . Ding, and Y . Chen, “Constrained decoding for secure code generation,” 2024. [Online]. Available: https: //arxiv.org/abs/2405.00218
2024 arXiv
-
[19]
Ai alignment: A contemporary survey,
J. Ji, T. Qiu, B. Chen, J. Zhou, B. Zhang, D. Hong, H. Lou, K. Wang, Y . Duan, Z. He, L. Vierling, Z. Zhang, F. Zeng, J. Dai, X. Pan, H. Xu, A. O’Gara, K. Ng, B. Tse, J. Fu, S. Mcaleer, Y . Wang, M. Yang, Y . Liu, Y . Wang, S.-C. Zhu, Y . Guo, Y . Yang, and W. Gao, “Ai alignme...
2025 doi
-
[20]
Direct preference optimization: Your language model is secretly a reward model,
R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,” inAdvances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine...
2023
-
[21]
Training language models to follow instructions with human feedback,
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe, “Training language models to follow instructions...
2022
-
[22]
Editing models with task arithmetic,
G. Ilharco, M. T. Ribeiro, M. Wortsman, L. Schmidt, H. Hajishirzi, and A. Farhadi, “Editing models with task arithmetic,” inThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. [Online]. Available: h...
2023
-
[23]
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time,
M. Wortsman, G. Ilharco, S. Y . Gadre, R. Roelofs, R. Gontijo-Lopes, A. S. Morcos, H. Namkoong, A. Farhadi, Y . Carmon, S. Kornblith, and L. Schmidt, “Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time,” inProceedin...
2022
-
[24]
Ties- merging: Resolving interference when merging models,
P. Yadav, D. Tam, L. Choshen, C. Raffel, and M. Bansal, “Ties- merging: Resolving interference when merging models,” inAdvances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36. Curran Associates, Inc....
2023
-
[25]
Steering language models with weight arithmetic,
C. Fierro and F. Roger, “Steering language models with weight arithmetic,” inThe Fourteenth International Conference on Learning Representations, 2026. [Online]. Available: https://openreview.net/for um?id=S0D3EFWohd
2026
-
[26]
Code llama: Open foundation models for code,
B. Rozi `ere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, R. Sauvestre, T. Remez, J. Rapin, A. Kozhevnikov, I. Evtimov, J. Bitton, M. Bhatt, C. C. Ferrer, A. Grattafiori, W. Xiong, A. D ´efossez, J. Copet, F. Azhar, H. Touvron, L. Martin, N. Usunier...
2024 arXiv
-
[27]
Starcoder 2 and the stack v2: The next generation,
A. Lozhkov, R. Li, L. B. Allal, F. Cassano, J. Lamy-Poirier, N. Tazi, A. Tang, D. Pykhtar, J. Liu, Y . Wei, T. Liu, M. Tian, D. Kocetkov, A. Zucker, Y . Belkada, Z. Wang, Q. Liu, D. Abulkhanov, I. Paul, Z. Li, W.-D. Li, M. Risdal, J. Li, J. Zhu, T. Y . Zhuo, E. Zheltonozhskii,...
2024 arXiv
-
[28]
Qwen2.5-coder technical report,
B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Lu, K. Dang, Y . Fan, Y . Zhang, A. Yang, R. Men, F. Huang, B. Zheng, Y . Miao, S. Quan, Y . Feng, X. Ren, X. Ren, J. Zhou, and J. Lin, “Qwen2.5-coder technical report,” 2024. [Online]. Available: ...
2024 arXiv
-
[29]
Codegemma: Open code models based on gemma,
C. Team, H. Zhao, J. Hui, J. Howland, N. Nguyen, S. Zuo, A. Hu, C. A. Choquette-Choo, J. Shen, J. Kelley, K. Bansal, L. Vilnis, M. Wirth, P. Michel, P. Choy, P. Joshi, R. Kumar, S. Hashmi, S. Agrawal, Z. Gong, J. Fine, T. Warkentin, A. J. Hartman, B. Ni, K. Korevec, K. Schaefe...
2024 arXiv
-
[30]
CWE: Common weakness enumerations,
MITRE Corporation, “CWE: Common weakness enumerations,” 2026. [Online]. Available: https://cwe.mitre.org/
2026
-
[31]
Teaching an old LLM secure coding: Localized preference optimization on distilled preferences,
M. S. Hasan, S. Chakraborty, S. Karmaker, and N. Balasubramanian, “Teaching an old LLM secure coding: Localized preference optimization on distilled preferences,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),...
2025
-
[32]
Wizardcoder: Empowering code large language models with evol-instruct,
Z. Luo, C. Xu, P. Zhao, Q. Sun, X. Geng, W. Hu, C. Tao, J. Ma, Q. Lin, and D. Jiang, “Wizardcoder: Empowering code large language models with evol-instruct,” inInternational Conference on Learning Representations, B. Kim, Y . Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y ....
2024
-
[33]
Code alpaca: An instruction-following llama model for code generation,
S. Chaudhary, “Code alpaca: An instruction-following llama model for code generation,” https://github.com/sahil280114/codealpaca, 2023
2023
-
[34]
[Online]
GitHub, “CodeQL,” 2026. [Online]. Available: https://github.com/githu b/codeql
2026
-
[36]
Available: https://arxiv.org/abs/2108.07732
[Online]. Available: https://arxiv.org/abs/2108.07732
-
[37]
Lora: Low-rank adaptation of large language models,
E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” inThe Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. [Onlin...
2022
-
[38]
Deepseek-coder: When the large language model meets programming – the rise of code intelligence,
D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y . Wu, Y . K. Li, F. Luo, Y . Xiong, and W. Liang, “Deepseek-coder: When the large language model meets programming – the rise of code intelligence,” 2024. [Online]. Available: https://arxiv.org/abs/2401.14196
2024 arXiv
Reviewed July 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.