REVIEW 4 major objections 6 minor 26 references
Evaluating Large Language Models in Vulnerability Detection Under Variable Context Windows
T0 review · 4 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read Large language models differ sharply in whether input code length affects their Java vulnerability verdicts, with GPT-4, Mistral, and Mixtral at full context showing no measured effect.
desk verdict A useful question undercut by a load-bearing proxy tokenizer and thin statistics; the robustness claims for GPT-4, Mistral, and Mixtral should not be trusted as written. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The measurement machinery is a unified Byte Pair Encoding tokenizer with a 30,000-token vocabulary, applied externally to compute every Java file's tokenized length while each LLM receives the raw code and uses its own internal tokenization. The statistical engine is the chi-square test of independence run under two null hypotheses: no relationship between tokenized length and vulnerability-identification accuracy, and no relationship between tokenized length and response explicitness. Responses are manually categorized as correct (explicitly vulnerable), incorrect (explicitly not vulnerable), or irrelevant, and the test compares these categories across token-length groups at a 0.05 significance level, 0.80 power, and a medium effect size of 0.3.
What would settle it
Recompute the tokenized length of the 140 files with each model's native tokenizer and rerun the chi-square tests; if GPT-4 or Mistral then shows a significant length-performance relationship, the reported robustness is an artifact of the proxy tokenizer rather than a property of the models.
Extended reading notes
Core claim
The paper reports that model families have sharply different length-robustness profiles in Java vulnerability detection. At a significance level of 0.05 with power 0.8 and a medium effect size, GPT-4 accepts both null hypotheses, and Mistral and Mixtral accept both when the context window is set to 32,768 tokens, meaning no statistical relationship was detected between tokenized code length and accurate or explicit vulnerability verdicts. Most other configurations, including all Gemma variants, reject at least one hypothesis. The paper also reports that increasing parameter count alone does not guarantee robustness, that a larger context window helps only some models, and that higher-precision fp16 inference can improve length-independence for CodeGemma and Phi without resolving explicitness issues.
Load-bearing premise
The claim assumes that a single external BPE tokenizer with a 30,000-token vocabulary produces counts that faithfully track how much of each model's effective context the code occupies, even though every LLM uses its own internal tokenization.
Editorial extensions
If this is right
- For GPT-4, and for Mistral or Mixtral run at their full 32,768-token context, length-based preprocessing of Java code appears unnecessary for vulnerability detection accuracy and explicitness.
- Running Mistral or Mixtral at a restricted 2,048-token context makes them length-sensitive, so context-window allocation is a practical lever for robustness in these open models.
- Parameter count alone is not a reliable predictor of length-robustness; architecture and training choices, plus numerical precision for some families, matter more.
- For length-sensitive models such as Gemma, CodeGemma, and Phi, reducing token count while preserving code structure is a plausible route to more consistent detection, as the paper explicitly recommends.
Reading between the lines
- The unified BPE tokenizer is only a proxy for what each model actually consumes; re-measuring length with each model's native tokenizer could change which configurations accept or reject the null hypotheses.
- Because the dataset contains only vulnerable files, 'accuracy' in this study is a true-positive rate, and the length-robustness of false-positive decisions (calling safe code vulnerable) is never tested.
- The chi-square design compares length bins rather than a continuous dose-response; a regression-based analysis would indicate whether the length effect is monotonic and where a critical threshold lies.
- The fact that Mistral and Mixtral become length-robust only at the largest context window suggests that context pressure, not just architecture, drives the effect, and that intermediate context sizes could be scanned to locate the transition.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies whether the tokenized length of Java source code affects the accuracy and explicitness of ten LLM families in vulnerability detection. The authors curate 140 vulnerable Java files from Vul4J, obtain responses from 39 model/quantization/context-window configurations, measure code length with a single external BPE tokenizer, and run chi-square tests of independence between binned token counts and two response-quality categories. They report which configurations accept or reject the two null hypotheses, concluding that GPT-4 and the large-context versions of Mistral and Mixtral are robust to input length, while other models are not.
Significance. If the findings were valid, they would be practically useful for selecting and configuring LLMs for vulnerability detection, and the broad multi-model comparison is a reasonable starting point. The paper also makes a concrete falsifiable claim about length-robustness that could inform deployment decisions. However, the current evidence does not support the central claim because the length variable is a proxy that is explicitly decoupled from the tokenization each model actually uses, and because the statistical reporting is too sparse to verify the accept/reject decisions. The comparative breadth and the clear statement of hypotheses are strengths, but the methodology needs substantial rework before the conclusions can be trusted.
major comments (4)
- [Section III.E and Section IV] The independent variable in every chi-square test is the token count produced by a single externally fitted BPE tokenizer with a 30,000-vocabulary (Section III.E), which the paper states is "separate from the LLMs' detection process." This makes the measured construct a proxy, not the tokenized length that each model actually consumes. Different LLM families use different internal tokenizers, so the ranking of the 140 files by this proxy can diverge from the ranking by a model's own tokenizer. The central conclusion that GPT-4, Mistral at 32768 context, and Mixtral at 32768 context are robust to tokenized length is therefore unsupported unless the authors demonstrate that the proxy strongly correlates with each model's internal token count, or unless they re-run the analysis using each model's own tokenizer. This is the load-bearing issue for the paper's main claim.
- [Section IV and Table II] The paper reports only accept/reject decisions in Table II; it does not report chi-square statistics, p-values, degrees of freedom, effect sizes, or cell counts. With 140 files and roughly 39 separate tests at alpha=0.05, a reader cannot assess whether accepted nulls reflect genuine independence or low power, nor whether rejected nulls survive multiple-testing correction. The stated effect size (Cohen's w=0.3) and power (0.80) are not connected to the actual tests: no achieved power or minimum detectable effect is given, and the token-length binning scheme is never described, although bin boundaries determine degrees of freedom and expected counts. Please provide the full test statistics and a clear explanation of the binning procedure.
- [Section III.D and Section V] All 140 dataset files are vulnerable (Section III.A and III.D), so the "accuracy" metric is a sensitivity/true-positive rate with no false-positive component. The null hypotheses are framed as being about "vulnerability detection accuracy," but the experimental design cannot detect length effects on false-positive responses because no non-vulnerable files are presented. The robustness claims for GPT-4, Mistral, and Mixtral are therefore only about the ability to recognize known vulnerabilities in vulnerable files, not about overall detection accuracy. This limitation should be stated explicitly, or the dataset should include balanced negative samples.
- [Section III.B and Table I] Table I lists 39 configurations rather than 10 models, because several models appear with multiple quantization schemes and context windows. Treating each configuration as an independent test without accounting for the shared underlying model or the multiplicity of tests inflates the risk of false discoveries. The analysis should address this dependency structure, for example by grouping tests by model family or by applying a multiple-testing correction and reporting adjusted conclusions.
minor comments (6)
- [Title and Abstract] The title contains spacing errors ("V ariable", "T okenization"), and the abstract contains typos such as "damanges," "an d," and "t he" that should be corrected before resubmission.
- [Section III.E] The BPE tokenizer's training corpus, vocabulary construction, and any preprocessing of Java code before tokenization are not specified. Please provide these details so that the token-count measure is reproducible.
- [Table I] The reported context window for CodeLLaMA 70B is 2048, which appears inconsistent with the text stating that CodeLLaMA handles sequences up to 100,000 tokens. Please clarify which context window was used and why the table lists 2048.
- [Section II] The statement that "no studies have addressed factors influencing the quality of LLM responses" is too broad given that several cited works, including Khare et al. [8] and Dong et al. [10], examine prompt and context-related factors. Please narrow the claim to the specific relationship studied in this paper.
- [Section III.C and Section III.D] The system prompt asks the model to answer "yes or no," but the response categorization includes an "Irrelevant Response" category. Please explain how responses that do not follow the yes/no instruction were scored and whether such responses are counted as incorrect, inexplicitness, or both.
- [Section VI] The recommendation that preprocessing should reduce token count while preserving code structure is not directly supported by the experiments, which only test whether length and performance are associated; they do not test whether reducing length improves performance. Please soften this recommendation or add evidence.
Circularity Check
No circularity found: the evaluation is an empirical chi-square analysis whose inputs (response labels, token counts) are independently defined and not fitted to the outcomes.
full rationale
The paper's central claim is a statistical comparison between an externally defined token-length proxy and LLM response categories. The only independent variable is the token count produced by a BPE tokenizer with a 30,000-token vocabulary (Section III.E), and the dependent variables are manually categorized LLM responses (Section III.D). The paper explicitly states that this tokenization is 'separate from the LLMs' detection process' and is used 'solely for analysis,' so it is not a fitted parameter or a hidden input to the models being tested. No equation in the paper defines the result in terms of the hypothesis, and no conclusion is derived from a self-citation. The related-work citations by the authors are contextual and not load-bearing. The main limitations—potential mismatch between the external tokenizer and each model's internal tokenizer, low statistical power, and unreported p-values—are measurement-validity concerns, not circularity. Because the dataset (Vul4J) and ground truth are external and the analysis pipeline is transparent, the derivation is self-contained and no circular step can be exhibited.
Assumptions & free parameters
free parameters (4)
- BPE vocabulary size =
30,000 tokens
- Chi-square effect size target =
0.3 (Cohen's w)
- Token-length binning scheme =
Not disclosed
- Dataset curation threshold =
Not disclosed
assumptions (3)
- domain assumption All 140 curated Java files are vulnerable
- domain assumption External BPE token count is a valid proxy for the model's processed context length
- standard math Chi-square test assumptions hold
Cite this review
Pith. "Pith review of Evaluating Large Language Models in Vulnerability Detection Under Variable Context Windows." pith.science (2026). https://pith.science/paper/EYWKZW5B
@misc{pith2026250200064,
author = {Pith},
title = {Pith review of: Evaluating Large Language Models in Vulnerability Detection Under Variable Context Windows},
year = {2026},
howpublished = {\url{https://pith.science/paper/EYWKZW5B}},
note = {Machine review of arXiv:2502.00064}
}
read the original abstract
This study examines the impact of tokenized Java code length on the accuracy and explicitness of ten major LLMs in vulnerability detection. Using chi-square tests and known ground truth, we found inconsistencies across models: some, like GPT-4, Mistral, and Mixtral, showed robustness, while others exhibited a significant link between tokenized length and performance. We recommend future LLM development focus on minimizing the influence of input length for better vulnerability detection. Additionally, preprocessing techniques that reduce token count while preserving code structure could enhance LLM accuracy and explicitness in these tasks.
Reference graph
Works this paper leans on
-
[1]
AMAL: high- fidelity, behavior-based automated malware analysis and cl assification,
A. Mohaisen, O. Alrawi, and M. Mohaisen, “AMAL: high- fidelity, behavior-based automated malware analysis and cl assification,” Comput. Secur . , vol. 52, pp. 251–266, 2015. [Online]. Available: https://doi.org/10.1016/j.cose.2015.04.001
-
[2]
Analyzing and detecting emerging internet of things malware: A graph-based approac h,
H. Alasmary, A. Khormali, A. Anwar, J. Park, J. Choi, A. Ab usnaina, A. Awad, D. Nyang, and A. Mohaisen, “Analyzing and detecting emerging internet of things malware: A graph-based approac h,” IEEE Internet Things J. , vol. 6, no. 5, pp. 8977–8988, 2019. [Online]. Available: https://doi.org/10.1109/JIOT.2019.2925929
arXiv 2019
-
[3]
Industry-specific vulnerability assessm ent,
M. Alkinoon, H. Althebeiti, A. Alkinoon, M. Mohaisen, S. Salem, and D. Mohaisen, “Industry-specific vulnerability assessm ent,” in W eb Information Systems Engineering - WISE 2024 - 25th Internat ional Conference, Doha, Qatar , December 2-5, 2024, Proceedings, Part V , ser. Lecture Notes in Computer Science, M. Barhamgi, H. Wang , and X. Wang, Eds., vol. 1...
-
[4]
Enriching vulnerabilit y reports through automated and augmented description summarization,
H. Althebeiti and D. Mohaisen, “Enriching vulnerabilit y reports through automated and augmented description summarization,” in Information Security Applications - 24th International Conference, WI SA 2023, Jeju Island, South Korea, August 23-25, 2023, Revised Selected P apers, ser. Lecture Notes in Computer Science, H. Kim and J. M. Y oun, Eds., vol. 144...
-
[5]
C leaning the nvd: Comprehensive quality assessment, improvements, and analyses,
A. Anwar, A. Abusnaina, S. Chen, F. Li, and D. Mohaisen, “C leaning the nvd: Comprehensive quality assessment, improvements, and analyses,” IEEE Transactions on Dependable and Secure Computing , vol. 19, no. 6, pp. 4255–4269, 2022
work page 2022
-
[6]
A. Abusnaina, M. Abuhamad, H. Alasmary, A. Anwar, R. Jang , S. Salem, D. Nyang, and D. Mohaisen, “DL-FHMC: deep learning-based fine-grained hierarchical learning approa ch for robust malware classification,” IEEE Trans. Dependable Secur . Comput. , vol. 19, no. 5, pp. 3432–3447, 2022. [Online]. Available: https://doi.org/10.1109/TDSC.2021.3097296
arXiv 2022
-
[7]
Transformer-based language models for softw are vulnerability detection,
C. Thapa, S. I. Jang, M. E. Ahmed, S. Camtepe, J. Pieprzyk, and S. Nepal, “Transformer-based language models for softw are vulnerability detection,” in Annual Computer Security Applications Conference, ACSAC 2022 . ACM, 2022, pp. 481–496. [Online]. Available: https://doi.org/10.1145/3564625.3567985
arXiv 2022
-
[8]
Understanding the effectiveness of large language models in detecting security vulnerabilities,
A. Khare, S. Dutta, Z. Li, A. Solko-Breslin, R. Alur, and M . Naik, “Understanding the effectiveness of large language models in detecting security vulnerabilities,” CoRR, vol. abs/2311.16169, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2311.16169
Show all 26 references
-
[9]
Benchmarking large language models for log analysis, secu rity, and interpretation,
E. Karlsen, X. Luo, N. Zincir-Heywood, and M. I. Heywood, “Benchmarking large language models for log analysis, secu rity, and interpretation,” CoRR, vol. abs/2311.14519, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2311.14519
-
[10]
Exploring context window of large language m odels via decomposed positional vectors,
Z. Dong, J. Li, X. Men, W. X. Zhao, B. Wang, Z. Tian, W. Chen , and J.-R. Wen, “Exploring context window of large language m odels via decomposed positional vectors,” CoRR, vol. abs/2405.18009, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2405.18009
- [11]
- [12]
- [13]
-
[14]
Introducing meta llama 3: The most capable ope nly available llm to date,
Meta AI, “Introducing meta llama 3: The most capable ope nly available llm to date,” 2024, meta AI Blog. [Online]. Availa ble: https://ai.meta.com/blog/meta-llama-3/ 4
2024
- [15]
-
[16]
Codegemma - an official google release for code llms,
P . Cuenca, O. Sanseviero, V . Srivastav, P . Schmid, M. Da vaadorj, and L. B. Allal, “Codegemma - an official google release for code llms,” April 2024, accessed: 2024-05-31. [Online]. Av ailable: https://huggingface.co/blog/codegemma
2024
- [17]
- [18]
- [19]
-
[20]
Phi-2: The surprising power of small language models,
M. Abdin, J. Aneja et al. , “Phi-2: The surprising power of small language models,” Microsoft Research Blog , 2023. [Online]. Available: https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/
2023
-
[21]
Phi-3 technical report: A highly capable language model locally on your phone,
M. I. Abdin, S. A. Jacobs et al., “Phi-3 technical report: A highly capable language model locally on your phone,” CoRR, vol. abs/2404.14219,
-
[22]
Reimagining self-adaptation in the age of large language models,
R. Donakanti, P . Jain, S. Kulkarni, and K. V aidhyanatha n, “Reimagining self-adaptation in the age of large language models,” CoRR, vol. abs/2404.09866, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2404.09866
-
[23]
Incremental comprehension of garden-path sente nces by large language models: Semantic interpretation, syntacti c re-analysis, and attention,
A. Li, X. Feng, S. Narang, A. Peng, T. Cai, R. S. Shah, and S. V arma, “Incremental comprehension of garden-path sente nces by large language models: Semantic interpretation, syntacti c re-analysis, and attention,” CoRR, vol. abs/2405.16042, 2024. [Online]. Available: https://...
-
[24]
Vul4j : A dataset of reproducible java vulnerabilities geared towards the stud y of program repair techniques,
Q.-C. Bui, R. Scandariato, and N. E. D. Ferreyra, “Vul4j : A dataset of reproducible java vulnerabilities geared towards the stud y of program repair techniques,” in MSR. ACM, 2022, pp. 464–468. [Online]. Available: https://doi.org/10.1145/3524842.3528482
2022
-
[25]
A new algorithm for data compression,
P . Gage, “A new algorithm for data compression,” The C Users Journal archive , vol. 12, pp. 23–38, 1994. [Online]. Available: https://api.semanticscholar.org/CorpusID:59804030 5
1994
- [2024]
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.