Pith. sign in

REVIEW 5 major objections 4 minor 26 references

Rethinking Code Complexity Through the Lens of Large Language Models

T0 review · 5 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper tries to establish that the difficulty large language models experience when reading code is governed by hierarchical semantic composition and branching divergence, not by the control-flow statistics classical complexity metrics

desk verdict A promising model-aware complexity metric, but the headline correlations are partly fitted on the same tasks they claim to predict. read the letter →

arxiv 2602.07882 v2 pith:LBTS4EPE submitted 2026-02-08 cs.SE

Chen Xie , Xiaodong Gu , Yuling Shi , Beijun Shen This is my paper · ORCID
classification cs.SE
keywords codecomplexityLLM-perceiveddifficultysemanticcompositiontokenentropybranchingfactorprogramrepairtranslationexecutionreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Classical code complexity metrics—cyclomatic complexity, Halstead complexity, maintainability index—lose all consistent correlation with LLM performance once code length is controlled for. The paper's central claim is that what actually makes code hard for an LLM is semantic nonlinearity: how deeply semantic units are composed and how many branches diverge at each level. To capture this, the authors build LM-CC, which uses a language model's own token-level uncertainty to segment code into semantic units, arranges them into a compositional hierarchy, and scores each unit by its depth and branching factor. If this is right, code should be judged by model-perceived difficulty rather than human-oriented structural statistics, and reducing LM-CC without changing behavior becomes a direct lever for improving performance on coding tasks.

What carries the argument

The central object is LM-CC, defined as α·TotalBranch + (1−α)·TotalCompLevel, which equals a sum over semantic units v of α·b(v) + (1−α)·d(v), where d(v) is compositional depth and b(v) is branching factor. The hierarchy is built by thresholding token-level next-token entropy combined with structural delimiters to split code into semantic units, then nesting those units by indentation and boundary structure. This entropy-guided semantic compositional hierarchy is what converts the model's own predictive uncertainty into a complexity score.

What would settle it

Re-run LM-CC with a different entropy-source model on the same datasets and correlate against the same target model's pass@1; if the length-controlled partial correlations fall well below -0.9 or lose significance across tasks, the hierarchy is not capturing a model-general difficulty. A second check: construct flat and deeply nested programs of equal cyclomatic complexity and nearly equal token count; if the target model's pass@1 is indistinguishable on the two, the depth term in LM-CC is not the active driver.

Watch

Extended reading notes

Core claim

The central discovery is that LLM-perceived code difficulty is governed by semantic nonlinearity—the depth of nested semantic composition and the width of branching alternatives—rather than by the syntactic control-flow statistics that classical metrics count. After controlling for code length, LM-CC shows partial Spearman correlations between -0.92 and -0.97 with pass@1 across program repair, code translation, and code execution reasoning, whereas classical metrics mostly show non-significant or inconsistent partial correlations. Further, semantics-preserving rewrites that lower LM-CC without lowering cyclomatic complexity improve pass@1 by up to 20.9%: from 13.4% to 16.2% on program repair

Load-bearing premise

That the token-level uncertainty spikes observed by the model used to build the hierarchy (CodeLlama-7B) mark the same semantic boundaries for the model whose performance is being predicted (DeepSeek-V3).

Editorial extensions

If this is right

  • Traditional complexity metrics should not be treated as proxies for LLM difficulty: after controlling for code length, they lose consistent predictive power.
  • LM-CC can serve as an optimization target: reducing it through semantics-preserving rewriting improves pass@1 across all three tested coding tasks.
  • Benchmarks and datasets can be stratified by LM-CC to compare models on code difficulty as LLMs actually perceive it.
  • High-LM-CC code, even when short, may trigger different inference strategies such as step-by-step reasoning or external tool calls.
  • Training curricula can weight or order code samples by LM-CC to align learning with model-specific processing challenges.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: the entropy source used to build the hierarchy and the target model whose difficulty is measured need not be the same; if the entropy signal is genuinely model-general, LM-CC could be re-estimated for other model families, but the paper only demonstrates one entropy-source/target pairing.
  • My inference: the 20.9% gain is measured on a filtered subset of samples that admitted valid rewrites and had high original LM-CC, so applying this intervention to arbitrary code could show smaller or uneven gains.
  • My inference: a sharper test would be whether LM-CC also predicts difficulty for other programming languages and for models with different tokenizers; the paper's hierarchy construction relies on indentation and syntactic delimiters, which may behave differently across languages.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. This paper studies whether classical code complexity metrics correlate with LLM performance and proposes a new metric, LM-CC, defined as α·TotalBranch + (1−α)·TotalCompLevel over an entropy-guided semantic compositional hierarchy. Using partial correlations controlling for code length, it reports that classical metrics show no consistent correlation, while LM-CC correlates strongly with DeepSeek-V3's pass@1 (Spearman r from −0.92 to −0.97 across program repair, code translation, and execution reasoning). It further reports that semantics-preserving rewrites that lower LM-CC improve pass@1 by up to 20.9%.

Significance. If the results held, LM-CC would be a valuable, model-grounded complexity measure with practical applications in refactoring, benchmark design, and curriculum learning. The paper is commendable for releasing code, providing an explicit algorithm, and combining correlation analysis with a causal intervention. However, the current evidence is weakened by in-sample selection of the feature pair and α, an untested transfer of CodeLlama-7B entropy to DeepSeek-V3, and a correlation protocol that selects the largest of several group-count results. These issues are fixable, and the central idea remains worth pursuing.

major comments (5)
  1. [§3.3–§3.4, §4.1, §4.3] The metric is defined after inspecting the same three task/dataset pairs on which it is then evaluated. Table 2 is used to select TotalBranch/TotalCompLevel, and Fig. 3 is used to set α=0.8. The last column of Table 2 is therefore an in-sample fit, not an independent estimate. Please report correlations on held-out tasks/datasets or use nested selection, and provide confidence intervals for the selected model.
  2. [§3.2, §4.1, App. B.3] Token entropy is computed with CodeLlama-7B, but pass@1 is measured with DeepSeek-V3. The claim that LM-CC captures 'LLM-perceived' difficulty requires that CodeLlama's entropy boundaries transfer to DeepSeek. Proposition B.3 merely assumes a bimodal entropy gap; no empirical check is provided. Please either recompute the hierarchy with the target model's entropy, or demonstrate that the entropy signal adds value beyond the syntactic delimiter component (e.g., by ablating the entropy branch).
  3. [App. C and Table 2] The protocol repeats group count 9–11 and reports the largest significant correlation. With ~10 groups, this is a multiple-comparison selection and inflates the reported |r|. The partial correlations also use group medians/means, reducing N to ~10. Please fix the group count in advance, report all values, and include bootstrap confidence intervals or a multiple-comparison correction.
  4. [Corollary 3.4] The proof shows Φ = δ·TotalCompLevel + γ·TotalBranch − const when all units exceed d*. The metric is LM-CC = α·TotalBranch + (1−α)·TotalCompLevel. These are different linear functions; monotonic relationship requires α/(1−α) = γ/δ, which is neither stated nor derived. Please either prove the relationship under the correct condition or weaken the claim to an approximation.
  5. [Table 1 and §2.2] HC shows partial r = −0.93 for program repair and −0.70 for code translation, both significant. The text says 'all such correlations decline' and 'fail to yield consistently significant results,' while the abstract says classical metrics 'exhibit no consistent correlation.' This overstates the negative result. Please qualify the claim as 'no single metric is significant across all three tasks.'
minor comments (4)
  1. [§4.1 vs. Remark B.4] τ is described as 0.67 in §4.1 but as the 67th percentile in Remark B.4; clarify which one is used.
  2. [Algorithm 1] Line 10 uses B both for the set of boundaries and for the semantic-unit array; rename to avoid confusion.
  3. [Proposition B.1] The argument that indentation whitespace does not affect token length is hand-wavy; state the tokenization assumption explicitly.
  4. [Table 3] The † significance marker is not defined; report the test used and sample sizes for the paired comparison.

Circularity Check

2 steps flagged · score 6.0 of 10

LM-CC's headline correlations are in-sample: features and α are selected on the same three tasks where r≈−0.92/−0.97 is reported, and the theoretical corollary re-expresses the same two aggregates.

  1. fitted input called prediction [§3.3–§4.1, Table 2 (feature selection and evaluation on same tasks)]
    "To examine how these features relate to LLM-perceived difficulty, we perform partial correlation analysis using the same protocol as in Section 2, controlling for code length. ... Using the same datasets and experimental protocol as in our empirical study of existing metrics, we compute partial correlation coefficients between LM-CC and LLM task scores, controlling for code length. ... the weighting factor fixed to α=0.8 based on the ablation results in Section 4.3."

    The two features entering LM-CC (TotalBranch, TotalCompLevel) were selected in §3.3 from the partial-correlation results of Table 2 on these same three tasks; α=0.8 was then tuned in §4.3 on the same three tasks. Table 2's 'LM-CC' column and the §4.1 results are therefore in-sample correlations of a composite built to maximize those very correlations. The r≈−0.92…−0.97 are fitted outcomes; no held-out split or external validation is reported, so they cannot be read as an independent confirmation that LM-CC predicts LLM difficulty.

  2. self definitional [§3.5, Corollary 3.4 (Eq. 5 vs. Eq. 4)]
    "Corollary 3.4 (LM-CC as Structural Complexity Proxy). The structural penalty satisfies Φ(T)≤δ·TotalCompLevel+γ·TotalBranch, with equality (up to constants) when all units exceed the critical depth. Thus, LM-CC =α·TotalBranch+(1−α)·TotalCompLevel is monotonically related to the structural entropy penalty Φ(T)."

    The 'structural entropy penalty' Φ(T)=δΣmax(0,d(v)−d*)+γΣ(b(v)−1)+ from Theorem 3.3 is just another weighted sum of the same aggregate features TotalCompLevel and TotalBranch that define LM-CC in Eq. 4. Since Assumptions 3.1 and 3.2 already posit penalties linear in depth and branching, the corollary restates the definition of LM-CC as a monotone function of Φ; it does not derive LM-CC from measured entropy. The theory is an algebraic identity, not an independent justification.

full rationale

The paper's central empirical claim is that LM-CC has strong partial correlations with DeepSeek-V3 pass@1. The design chain makes this partly self-fulfilling: §3.3 screens six candidate features by partial correlation on the three evaluation tasks and selects TotalCompLevel/TotalBranch; §3.4 forms LM-CC as a weighted sum of just those two chosen features; §4.3 tunes α on those same tasks; §4.1 then reports the correlation of the resulting composite on the same datasets. No held-out split is described, so Table 2's LM-CC column is the optimization criterion, not an independent prediction. Separately, the theoretical support in Corollary 3.4 does not add content: Φ is defined as δΣmax(0,d−d*)+γΣ(b−1)+, which is a linear aggregate of the same depth/branching sums that constitute LM-CC, so the claimed monotonic relationship is definitional. I do not count the CodeLlama-7B→DeepSeek-V3 entropy-transfer mismatch or the untested bimodality of Proposition B.3 as circularity; those are empirical validity threats, not identity reductions. The self-citation to Shi et al. (2025) supports a premise but is backed by an external citation (Cooper & Scholak, 2024), so it is not load-bearing. Overall: partial circularity in the validation protocol and definitional theory yields score 6.

Assumptions & free parameters 5 free parameters · 6 assumptions · 2 invented entities

The central claim rests on two tuned parameters (α, τ), three unmeasured theory constants (δ, γ, d*), and a chain of assumed links between token entropy, syntactic structure, model transferability, and LLM difficulty. The authors flag Assumptions 3.1 and 3.2 explicitly, but they are load-bearing and unmeasured.

free parameters (5)
  • α (weighting factor) = 0.8
    Balances TotalBranch versus TotalCompLevel in LM-CC (Eq. 4); chosen by ablation on the three evaluation tasks (Fig. 3, §4.3), with program repair specifically sensitive to it.
  • τ (entropy boundary threshold) = 0.67 or 67th percentile (stated inconsistently)
    Segmentation threshold for semantic units (§3.2, §4.1, App. B.4); imported from Wang et al. 2025b; the two descriptions define different segmentations and are not reconciled.
  • δ (per-level depth degradation rate) = unmeasured
    Constant in Assumption 3.1 and in Φ(T) (§3.5); never estimated; Corollary 3.4 needs it to map Φ to LM-CC 'up to constants'.
  • γ (branching penalty rate) = unmeasured
    Constant in Assumption 3.2 (§3.5); never estimated; the theory does not constrain the metric without it.
  • d* (critical depth) = unmeasured
    Depth beyond which context degradation begins (Assumption 3.1); never instantiated empirically.
assumptions (6)
  • domain assumption Assumption 3.1: predictive uncertainty increases monotonically with compositional depth at rate δ beyond critical depth d*
    Stated in §3.5 and justified by long-context citations (Liu et al. 2024b; Xiao et al. 2024; Wang et al. 2026); not measured for code hierarchies in this paper.
  • domain assumption Assumption 3.2: divergence points with b≥2 branches add uncertainty γ·(b−1)
    Stated in §3.5 with citations (Li et al. 2025; Sepidband et al. 2025; Basu et al. 2025); asserted, not estimated.
  • domain assumption Token-level entropy of CodeLlama-7B is a valid proxy for the difficulty of the evaluated model DeepSeek-V3
    LM-CC's entropy is computed with CodeLlama-7B (§4.1) while task performance is measured on DeepSeek-V3; cross-model transferability of the entropy landscape is assumed without evidence.
  • domain assumption Token entropy is bimodal with a separability gap (interior vs boundary), per Proposition B.3
    The proposition assumes a bimodal entropy sequence with gap >2η; the paper cites Cooper & Scholak and Shi et al. 2025 but does not test bimodality on its own datasets.
  • domain assumption Indentation and syntactic delimiters define the semantic compositional hierarchy that LLMs actually track
    Algorithm 1 partitions by boundaries and indentation; the mapping from this mechanical tree to the model's internal compositional processing is assumed, not demonstrated.
  • domain assumption Passing (or, for repair, failing) the held-out test suite establishes semantic equivalence of rewrites
    Appendix D uses test outcomes as the semantic-preservation check; test-suite adequacy for semantic equivalence is not assessed.
invented entities (2)
  • Semantic compositional hierarchy (with entropy-defined semantic units)
    purpose: Representation of code as a tree of 'semantic units' whose depth d(v) and branching b(v) define LM-CC (§3.2).
    The only evidence linking this tree to LLM cognition is the in-sample correlation table and the selected rewriting results; no out-of-sample or mechanistic validation is provided.
  • LM-CC score
    purpose: A metric of LLM-perceived code complexity, α·TotalBranch + (1−α)·TotalCompLevel (Eq. 4).
    Its predictive value is reported on the same tasks used to pick α and the features; no held-out benchmark, pre-registered parameter, or formal verification is supplied; the GitHub repository could not be inspected for this review.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rethinking Code Complexity Through the Lens of Large Language Models." pith.science (2026). https://pith.science/paper/LBTS4EPE

@misc{pith2026260207882,
  author       = {Pith},
  title        = {Pith review of: Rethinking Code Complexity Through the Lens of Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LBTS4EPE}},
  note         = {Machine review of arXiv:2602.07882}
}
read the original abstract

Code complexity metrics such as cyclomatic complexity have long been used to assess software quality and maintainability. With the rapid advancement of large language models (LLMs) on coding tasks, an important yet underexplored question arises: do traditional complexity metrics meaningfully characterize the coding difficulty that LLMs perceive? In this work, we empirically demonstrate that classical complexity metrics exhibit no consistent correlation with LLM performance, revealing a fundamental mismatch with model-perceived difficulty. To address this gap, we propose LM-CC, a novel code complexity metric tailored for LLMs, grounded in the hypothesis that model-perceived code difficulty is fundamentally driven by semantic nonlinearity. LM-CC quantifies complexity through an entropy-guided semantic compositional hierarchy, capturing the cumulative uncertainty encountered by LLMs during code understanding. Our experimental results demonstrate that LM-CC exhibits strong and consistent partial correlations with LLM performance, while semantics-preserving reductions in LM-CC consistently lead to improved downstream task performance. The source code is available at: https://github.com/xchen121/lm-cc.

Figures

Figures reproduced from arXiv: 2602.07882 by the authors.

Figure 1
Figure 1. Comparison between Cyclomatic Complexity (CC) and our proposed LM-CC. While CC assigns identical values to code snippets with significantly different cognitive loads for LLMs (top), LM-CC effectively distinguishes them by capturing the model uncertainty on non-linear code semantics (bottom). erties, such metrics aim to approximate the effort required to comprehend, test, and maintain code. Consequently, they have be… view at source ↗
Figure 2
Figure 2. Hierarchical Semantic Decomposition Example. Left: source code with token-entropy annotations, where color-coded regions indicate elevated LLM uncertainty. Right: the induced hierarchical semantic representation, with elements color-aligned to their semantic units of source code [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Ablation on the weighting factor α in LM-CC. Per￾formance peaks at intermediate α values, while hierarchy-only (α→0) and branching-only (α→1) configurations perform sub￾stantially worse. To investigate the impact of the weighting factor α, we conduct an ablation study by computing LM-CC under a range of α values. The results are shown in [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 10 linked inside Pith

  1. [1]

    U., Tushar, M

    Ahmad, W. U., Tushar, M. G. R., Chakraborty, S., and Chang, K.-W. A V ATAR: A parallel corpus for java-python program translation. InFindings of the Association for Computational Linguistics: ACL 2023, pp. 2268–2281. Association for Computational Linguistics,

    2023

  2. [7]

    DeepSeek-V3 technical report.arXiv preprint arXiv:2412.19437,

    DeepSeek-AI. DeepSeek-V3 technical report.arXiv preprint arXiv:2412.19437,

    arXiv

  3. [8]

    B., Galstyan, A., Wells, A., Schwartz, R., Huerta, E

    Du, Y ., Tian, M., Ronanki, S., Rongali, S., Bodapati, S. B., Galstyan, A., Wells, A., Schwartz, R., Huerta, E. A., and Peng, H. Context length alone hurts LLM performance despite perfect retrieval. InFindings of the Association for Computational Linguistics: EMNLP 2025, pp. 23281– 23298. Association for Computational Linguistics,

    2025

  4. [10]

    K., Luo, F., Xiong, Y ., and Liang, W

    Guo, D., Zhu, Q., Yang, D., Xie, Z., Dong, K., Zhang, W., Chen, G., Bi, X., Wu, Y ., Li, Y . K., Luo, F., Xiong, Y ., and Liang, W. DeepSeek-Coder: When the large language model meets programming – the rise of code intelligence. arXiv preprint arXiv:2401.14196,

    arXiv

  5. [13]

    LiveCodeBench: Holistic and contamination free evalu- ation of large language models for code.arXiv preprint arXiv:2403.07974,

    Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., and Stoica, I. LiveCodeBench: Holistic and contamination free evalu- ation of large language models for code.arXiv preprint arXiv:2403.07974,

    arXiv

  6. [14]

    C., Vinh, H

    Le, C. C., Vinh, H. C. T., Phan, H. N., Le, D. D., Nguyen, T. N., and Bui, N. D. Q. Visualcoder: Guiding large language models in code execution with fine-grained mul- timodal chain-of-thought reasoning. InFindings of the Association for Computational Linguistics: NAACL 2025, Albuquerque, New Mexico, USA, April 29 - May 4, 2025, volume NAACL 2025 ofFindin...

    2025

  7. [15]

    Entropy-gated branching for efficient test-time reasoning.arXiv preprint arXiv:2503.21961,

    Li, X., Callanan, E., Ghassel, A., and Zhu, X. Entropy-gated branching for efficient test-time reasoning.arXiv preprint arXiv:2503.21961,

    arXiv

  8. [16]

    D., Ibrahimzada, A

    Liu, C., Zhang, S. D., Ibrahimzada, A. R., and Jabbar- vand, R. CodeMind: A framework to challenge large language models for code reasoning.arXiv preprint arXiv:2402.09664, 2024a. Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. Lost in the middle: How language models use long contexts.Transactions of the Associat...

    arXiv

Show all 26 references
  1. [18]

    E., Adi, Y ., Liu, J., Remez, T., Rapin, J., et al

    Rozi`ere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y ., Liu, J., Remez, T., Rapin, J., et al. Code Llama: Open foundation models for code.arXiv preprint arXiv:2308.12950,

    arXiv

  2. [19]

    Enhancing llm-based code generation with complexity metrics: A feedback-driven approach

    Sepidband, M., Taherkhani, H., Wang, S., and Hemmati, H. Enhancing llm-based code generation with complexity metrics: A feedback-driven approach. In49th IEEE An- nual Computers, Software, and Applications Conference, COMPSAC 2025, Toronto, ON, Canada, July 8-11, 2025, pp. 1416...

    2025

  3. [20]

    From code to correctness: Closing the last mile of code gen- eration with hierarchical debugging.arXiv preprint arXiv:2410.01215, 2024a

    Shi, Y ., Wang, S., Wan, C., Wang, M., and Gu, X. From code to correctness: Closing the last mile of code gen- eration with hierarchical debugging.arXiv preprint arXiv:2410.01215, 2024a. Shi, Y ., Zhang, H., Wan, C., and Gu, X. Between lines of code: Unraveling the distinct pa...

  4. [21]

    Evoc2rust: A skeleton- guided framework for project-level c-to-rust translation

    Wang, C., Yu, T., Xie, C., Wang, J., Chen, D., Zhang, W., Shi, Y ., Gu, X., and Shen, B. Evoc2rust: A skeleton- guided framework for project-level c-to-rust translation. arXiv preprint arXiv:2508.04295, 2025a. Wang, S., Yu, L., Gao, C., Zheng, C., Liu, S., Lu, R., Dang, K., Ch...

  5. [22]

    Epicoder: Encompassing diversity and complexity in code generation

    Wang, Y ., Li, H., Zhang, X., Wu, J., Liu, X., Hu, W., Guo, Z., Huang, Y ., Xin, Y ., Yang, Y ., Su, J., Chen, Q., and Li, S. Epicoder: Encompassing diversity and complexity in code generation. InForty-second International Confer- ence on Machine Learning, ICML 2025, Vancouver...

    2025

  6. [23]

    DeepSeek-Coder- V2: Breaking the barrier of closed-source models in code intelligence.arXiv preprint arXiv:2406.11931,

    Zhu, Q., Guo, D., Shao, Z., Yang, D., Wang, P., Xu, R., Wu, Y ., Li, Y ., Gao, H., Ma, S., et al. DeepSeek-Coder- V2: Breaking the barrier of closed-source models in code intelligence.arXiv preprint arXiv:2406.11931,

    arXiv

  7. [24]

    lost in the middle

    11 Rethinking Code Complexity Through the Lens of Large Language Models Appendix A. Algorithm Algorithm 1 summarizes the computation of LM-CC. The procedure first preprocesses the source code and computes token- level entropy using a pretrained language model (Lines 1–4). Sema...

    2024

  8. [25]

    In the context of code, control-flow constructs (conditionals, loops) introduce structural ambiguity requiring the model to reason about multiple execution paths

    show that high-entropy tokens correspond to critical decision points where prediction uncertainty is elevated, and that concentrating computation budget on such high-uncertainty moments improves reasoning quality. In the context of code, control-flow constructs (conditionals, ...

    2025

  9. [26]

    Chain structure ( c2).Units are arranged in a linear compositional chain with levels 1,2,

    =k= Θ( √ L). Chain structure ( c2).Units are arranged in a linear compositional chain with levels 1,2, . . . , k, yielding TotalCompLevel(T2) = Pk i=1 i=k(k+ 1)/2 = Θ(L). Thus,|TotalCompLevel(T 1)−TotalCompLevel(T 2)|= Θ(L)−Θ( √ L) = Ω(L). These results explain why cyclomatic ...

    2024

  10. [1976]

    Y ., Singh, S., Tang, X., von Werra, L., and Long- pre, S

    Muennighoff, N., Liu, Q., Zebaze, A., Zheng, Q., Hui, B., Zhuo, T. Y ., Singh, S., Tang, X., von Werra, L., and Long- pre, S. Octopack: Instruction tuning code large language models.arXiv preprint arXiv:2308.07124,

    arXiv

  11. [2018]

    Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H

    URL https://www.sonarsource.com/ docs/CognitiveComplexity.pdf. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y ., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374,

    arXiv

  12. [2019]

    Alqadi, B. S. and Maletic, J. I. Slice-based cognitive com- plexity metrics for defect prediction. In27th IEEE In- ternational Conference on Software Analysis, Evolution and Reengineering, SANER 2020, London, ON, Canada, February 18-21, 2020, pp. 411–422. IEEE,

    2020

  13. [2020]

    A critical study of what code-LLMs (do not) learn

    Anand, A., Verma, S., Narasimhan, K., and Mezini, M. A critical study of what code-LLMs (do not) learn. InFind- ings of the Association for Computational Linguistics: ACL 2024, pp. 15869–15889. Association for Computa- tional Linguistics,

    2024

  14. [2021]

    Cooper, N. A. and Scholak, T. Perplexed: Understanding when large language models are confused.arXiv preprint arXiv:2404.06634,

    arXiv

  15. [2023]

    DynaCode: A dynamic complexity-aware code bench- mark for evaluating large language models in code gener- ation

    Hu, W., Duan, J., Wei, C., Zhang, L., Zhang, Y ., and Xu, K. DynaCode: A dynamic complexity-aware code bench- mark for evaluating large language models in code gener- ation. InFindings of the Association for Computational Linguistics: ACL 2025, pp. 21980–21997. Association for...

    2025

  16. [2024]

    Nestful: A benchmark for evaluating llms on nested sequences of api calls

    Basu, K., Abdelaziz, I., Kate, K., Agarwal, M., Crouse, M., Rizk, Y ., Bradford, K., Munawar, A., Kumaravel, S., Goyal, S., et al. Nestful: A benchmark for evaluating llms on nested sequences of api calls. InProceedings of the 2025 Conference on Empirical Methods in Natural La...

    2025

  17. [2025]

    Qwen2.5-Coder technical report.arXiv preprint arXiv:2409.12186,

    Hui, B., Yang, J., Cui, Z., Yang, J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., Lu, K., et al. Qwen2.5-Coder technical report.arXiv preprint arXiv:2409.12186,

    arXiv

  18. [2026]

    Girjoaba, A. V . and Capiluppi, A. Refactoring legacy code using cleaning up cycles: An experience report. InIEEE International Conference on Software Maintenance and Evolution, ICSME 2024, Flagstaff, AZ, USA, October 6-11, 2024, pp. 753–764. IEEE,

    2024

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.