Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

Human-Like Code Quality Evaluation through LLM-based Recursive Semantic Comprehension

T0 review · 5 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that recursively decomposing code into dependency-aware sub-code chunks lets an LLM evaluate generated code quality with correlations up to 0.853 against human experts and 0.753 against code execution, outperforming…

desk verdict Interesting method, compromised evaluation. read the letter →

arxiv 2412.00314 v1 pith:JW5AC4MW submitted 2024-11-30 cs.SE

Fangzhou Xu , Sai Zhang , Zhenchang Xing , Xiaowang Zhang , Yahong Han , Zhiyong Feng This is my paper · ORCID
classification cs.SE
keywords codequalityevaluationlargelanguagemodelsrecursivesemanticcomprehensiondependencydecouplingLLMasevaluatorgenerationHumanEvalCode-Pair
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that code quality can be evaluated automatically by having a large language model recursively decompose code into small sub-code chunks, store and update the semantics of variables and functions in a dependency dictionary, and then compare the reconstructed semantics of the generated code against a reference code. On a new 428-pair dataset called Code-Pair, this method (HuCoSC) achieves Pearson correlations of 0.769 with GPT-3.5 Turbo and 0.853 with GPT-4 Turbo against human expert scores, and 0.594 and 0.753 against code execution on HumanEval, surpassing all tested token-matching and direct-prompt baselines. If the result holds, it means LLM-based evaluators can be made much more reliable for judging generated code without large test suites, and the recursive decomposition also produces explanations of the score.

What carries the argument

The load-bearing mechanism is a recursive code-semantic-comprehension pipeline built from three parts: (1) Code Recursive Decomposition, which splits the AST into sub-codes under predefined node types (For, While, Assign, If, ClassDef, FunctionDef, Switch, Call) and recurses until sub-code nesting depth falls below a threshold; (2) Semantic Dependency Decoupling Storage, a dictionary mapping dependency names to textual semantic descriptions that is updated after each sub-code is analyzed, so each sub-code can be understood independently with the needed context; and (3) Code Semantic Summarization and Comparison, where the sub-code semantics are merged into a whole-code semantic and then compared with the reference code's semantic to produce a score and explanation.

What would settle it

Run HuCoSC on a pair of programs that compute the same function but with different control flow (for example, an iterative loop versus recursion) and identical variable names; if the score differs by more than one point on the 0–4 scale, the semantic comparison is being influenced by surface structure rather than captured semantics. A more direct check: delete a state-changing line inside a deeply nested sub-code and see whether the dependency-storage description for the affected variable changes; if the description is unchanged, the storage has failed to track the state update.

Watch

Extended reading notes

Core claim

The central discovery is that recursively decomposing code into manageable sub-code units and maintaining a Semantic Dependency Decoupling Storage—a dictionary that stores textual semantic descriptions of each external dependency and updates those descriptions as analysis proceeds—lets an LLM capture code semantics accurately enough to score generated code against a reference. The semantic comparison unit then identifies differences and produces a score plus explanation, with the problem statement introduced only when input statements are analyzed to avoid hallucination. The paper reports that this pipeline outperforms all baselines and that the advantage grows with code complexity: at a code depth of 4, the simplified single-pass variant loses 0.7 points (with GPT-3.5 Turbo) or 0.19 points (with GPT-4 Turbo) in expert-rated semantic quality compared with the recursive version.

Load-bearing premise

The whole pipeline assumes the textual semantic descriptions stored for each variable, function, and class preserve the actual state changes of the running code; if a stored description loses or distorts a state change, every downstream semantic comparison and score inherits that error.

Editorial extensions

If this is right

  • On Code-Pair, HuCoSC reaches Spearman/Pearson correlations of 0.769/0.853 with human experts using GPT-3.5 Turbo/GPT-4 Turbo, higher than every baseline tested.
  • On HumanEval, it reaches 0.594/0.753 Pearson correlation with functional correctness, again above all baselines, showing the semantic description carries execution-relevant information.
  • Removing the problem statement entirely drops Code-Pair correlation by 0.03 (GPT-3.5) and 0.12 (GPT-4), so the method still works without problem statements, for tasks like code translation.
  • Recursive decomposition mitigates LLM hallucination on complex code: expert-rated semantic quality declines more slowly with code depth than in the simplified single-pass variant.
  • Adding problem statements at every comprehension step lowers correlation and inflates scores, so selective problem-statement injection is a design choice that matters.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: The dependency-storage design suggests the method's ceiling is the fidelity of the stored semantic descriptions; if a variable's state change is missed, the final score inherits that error, so tracking mutable state more explicitly (e.g., by simulating updates) is a natural next step.
  • Editorial inference: Because the method already works without a problem statement, the same recursive comprehension and comparison could be applied to tasks such as code translation, code clone detection, or unit-test generation, where a reference semantic is available but no natural-language problem description is.
  • Editorial inference: The reported drop in correlation with code depth implies that for very long or deeply nested programs, further gains would require either a lower decomposition threshold or a second pass that verifies consistency between sub-code summaries.
  • Editorial inference: A direct test of semantic fidelity would be to feed HuCoSC a pair of programs that are semantically identical but written with different control flow (e.g., a loop versus recursion); high scores would confirm the storage captures meaning rather than surface structure.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes HuCoSC, an LLM-based framework for code quality evaluation that recursively decomposes code into sub-codes, stores and updates dependency semantics in a Semantic Dependency Decoupling Storage, and finally scores generated code by semantic comparison against a reference. The authors evaluate HuCoSC on a new Code-Pair dataset using human expert scores and on HumanEval using execution correctness, and report correlations that they claim surpass all existing match-based and LLM-based baselines. The central claim is that recursive semantic comprehension yields substantially higher agreement with human judgment and with code execution than direct LLM prompting or token-based matching.

Significance. The recursive semantic comprehension idea is plausible and, if the reported results hold under a properly controlled evaluation, would be a useful contribution to LLM-based code evaluation. The construction of the Code-Pair dataset with expert ratings is a valuable resource, and the paper includes illustrative examples where token-based metrics fail while HuCoSC succeeds. However, the current evidence does not establish the headline claim: the metric labels are inconsistent, the HumanEval sample size as stated is impossible, design choices appear to have been made after seeing the evaluation data, and no statistical uncertainty is reported. The paper is therefore not yet ready for publication, though the approach is sufficiently promising that the issues may be fixable with a substantially revised evaluation.

major comments (5)
  1. [Section I, Section IV.E, Table I] The correlation metric reported for the headline results is inconsistent. The Introduction reports 0.769 and 0.853 as Spearman correlations, while Table I labels the same values as Pearson rs and also reports Kendall tau values, and Section IV.E states that only Kendall-Tau and Pearson are used. Since every central comparison in the paper rests on these coefficients, the authors must state which metric was actually computed and recompute all tables under a single, clearly specified convention.
  2. [Section IV.B.2] The paper states that 'Due to the extensive size of the HumanEval dataset, we randomly selected 500 cases to serve as our validation set,' but the HumanEval dataset contains 164 problems. This makes the reported HumanEval correlations impossible as written. The authors must clarify what was sampled (for example, 500 generated programs drawn from the 164 problems) and rerun or relabel the analysis accordingly.
  3. [Section IV.C, Section V.C, Table I] The headline results appear to be in-sample after design choices were made on the evaluation data. Section IV.C fixes the recursion threshold at 3 without a sensitivity study, and Section V.C compares PFull, PLack, and PInitial on the Code-Pair dataset and then adopts PInitial, with Table I reporting results for that selected policy. Because there is no train/validation/test split and no separate development corpus, the reported advantage over baselines (e.g., GPT-4 Turbo few-shot .853 vs .746 on Code-Pair) can reflect selection rather than a general property. A genuinely held-out evaluation, or a nested validation procedure, is required to support the claim that HuCoSC surpasses existing methods.
  4. [Table I, Section V.A] No statistical uncertainty is reported for any correlation in Table I. There are no confidence intervals, p-values, or tests of whether the difference between HuCoSC and the closest baseline is significant, and the stochasticity of the LLM outputs is not quantified despite the temperature being set to 0.2 rather than 0. The claim that HuCoSC 'surpasses all existing methods' is therefore not supported by significance testing.
  5. [Section III.B, Algorithm 1, Section V.B] The fidelity of the Semantic Dependency Decoupling Storage is load-bearing but unvalidated. Algorithm 1 (lines 5-18) assumes that every external dependency has already been analyzed and stored when it is needed, and the updating step rewrites dependency semantics after each sub-code; however, no experiment measures whether these stored descriptions preserve or corrupt state changes. RQ2 (Fig. 7) shows that semantic quality declines with code depth, which is consistent with storage errors accumulating. A concrete check, such as comparing stored dependency descriptions against ground-truth variable semantics on a small set of programs, would substantially reduce this correctness risk.
minor comments (6)
  1. [Section I vs. Section IV.B.1] The Introduction says the authors 'selected 200 problems' from CodeNet, while Section IV.B.1 says the Code-Pair dataset is built from 85 CP tasks. This discrepancy should be resolved.
  2. [Table I] The row 'GPT-4 Turbo (CoT)' appears twice with identical values, once after 'GPT-4 Turbo (Zhuo w/R.)' and once after 'GPT-4 Turbo (few-shot)'; one of these rows is likely mislabeled.
  3. [Table I] The 'Expert' row reports values 0.97 and 0.91, but the text does not explain how these were computed or what quantity they represent; clarify this row or remove it.
  4. [Section V.A.3] The text says the Pearson correlations for CoT and few-shot prompts on HumanEval were 0.142 and 0.166, but Table I reports 0.142 and 0.186 for GPT-3.5 Turbo few-shot; please correct the mismatch.
  5. [Section IV.D] The description of the Chain-of-Thought baseline says it 'Prompts the LLM to explain the reasoning or steps for a problem before coding,' which appears to describe code generation rather than code evaluation; clarify the actual evaluation-time CoT prompt.
  6. [General] No code, prompts, or the Code-Pair dataset are released, which makes the reported numbers difficult to reproduce; releasing these artifacts would strengthen the paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: HuCoSC is an empirical pipeline and no reported correlation is an input or fit by construction.

full rationale

HuCoSC is an empirical system paper rather than an analytic derivation, so the only circularity question is whether a headline number is an input, a self-citation, or a fitted parameter renamed as a prediction. I find none. The recursion threshold is fixed in Section IV.C ('The threshold of recursive decomposition is set to 3') as a stated implementation choice, and no equation connects that choice to the reported correlations; it is therefore a hyperparameter-validity concern, not a construction-level reduction. The problem-statement policy PInitial is described in the approach (Section III.C) as the design, and RQ3 (Section V.C) is an ablation of that already-specified design rather than a reported model-selection step; the paper does not say that PFull or PLack were tried and PInitial then adopted for Table I. The AST node types are adopted from an external prior work [27], not from the authors' own prior results. There is no load-bearing self-citation, no imported uniqueness theorem, and no renamed known result. The claimed correlations (e.g., 0.769 and 0.853 on Code-Pair; 0.594 and 0.753 on HumanEval) are empirical correlations between HuCoSC scores and human/expert or execution labels; they are not forced by construction. The main legitimate caveats are external-validity issues: the threshold lacks a sensitivity study, the RQ3 analysis uses 200 Code-Pair pairs without an explicitly separate development set, and the Introduction's 'Spearman' label conflicts with Section IV.E and Table I's 'Pearson/Kendall' framing. These should be fixed and would temper the strength of the empirical claim, but they are not instances of circularity under the definitions used here.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The method relies on several domain assumptions: syntax-tree decomposition preserves semantic boundaries, stored dependency semantics remain faithful under updates, and expert ratings are reliable ground truth. The only numeric free parameter is the recursion depth threshold (3), with the LLM temperature (0.2) as a secondary operational setting. The problem-statement injection policy is an empirical design choice, not a theoretically derived one.

free parameters (3)
  • recursive decomposition depth threshold = 3
    Controls when sub-codes are split further; no sensitivity analysis is reported and the value appears to be chosen for the evaluation datasets.
  • LLM temperature = 0.2
    Set to make outputs more deterministic; standard sampling parameter, not fitted to the evaluation data.
  • predefined AST node types = For, While, Assign, If, ClassDef, FunctionDef, Switch, Call
    Chooses which syntax nodes become sub-code boundaries; no ablation shows whether this set is optimal or robust.
assumptions (4)
  • domain assumption AST decomposition with the eight predefined node types preserves semantic boundaries.
    The method assumes splitting at these nodes yields independently comprehensible sub-codes whose combination covers the program semantics (Section III-A, Algorithm 1).
  • domain assumption Semantic Dependency Decoupling Storage faithfully tracks variable and function semantics as code is analyzed.
    If stored descriptions drift or omit state changes, the recursive comprehension and final comparison inherit the error (Section III-B).
  • domain assumption Expert ratings on Code-Pair are reliable ground truth.
    Four experts with Kappa around 0.8; no adjudication of disagreements is described (Section IV-B).
  • ad hoc to paper Problem statement should be injected only at input sub-codes.
    Selected empirically because PFull lowered correlation and PLack hurt GPT-3.5; no theoretical justification is provided (Section V-C).
invented entities (1)
  • Semantic Dependency Decoupling Storage
    purpose: Store textual semantic descriptions of variables, functions, and classes so sub-codes can be analyzed independently.
    A new software component with no empirical evidence outside the paper; its correctness is assumed and is a core part of the method.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Human-Like Code Quality Evaluation through LLM-based Recursive Semantic Comprehension." pith.science (2026). https://pith.science/paper/JW5AC4MW

@misc{pith2026241200314,
  author       = {Pith},
  title        = {Pith review of: Human-Like Code Quality Evaluation through LLM-based Recursive Semantic Comprehension},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JW5AC4MW}},
  note         = {Machine review of arXiv:2412.00314}
}
read the original abstract

Code quality evaluation involves scoring generated code quality based on a reference code for a specific problem statement. Currently, there are two main forms of evaluating code quality: match-based evaluation and execution-based evaluation. The former requires the collection of a large number of test cases, making a huge cost. The latter relies on superficial code matching as an evaluation metric, which fails to accurately capture code semantics. Moreover, extensive research has demonstrated that match-based evaluations do not truly reflect code quality. With the development of large language models (LLMs) in recent years, studies have proven the feasibility of using LLMs as evaluators for generative tasks. However, due to issues like hallucinations and uncertainty in LLMs, their correlation with human judgment remains at a lower level, making the direct use of LLMs for code quality evaluation challenging. To address these issues, we propose Human-Like Code Quality Evaluation through LLM-based Recursive Semantic Comprehension (HuCoSC). We employ a recursive approach to enable LLMs to comprehend portions of code semantics independently each time, obtaining the code semantics through multiple interactions with LLMs. We designed a Semantic Dependency Decoupling Storage to make independent analysis feasible, allowing LLMs to achieve more accurate semantics by breaking down complex problems. Finally, the generated code is scored based on a semantic comparison between the reference code and itself. Experimental results indicate that HuCoSC surpasses existing state-of-the-art methods in terms of correlation with human experts and correlation with code execution.

Figures

Figures reproduced from arXiv: 2412.00314 by the authors.

Figure 1
Figure 1. A comparison of scores from different code quality evaluation [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. A comparison of scores from different code quality evaluation [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Overall Framework of HuCoSC a recursive code semantic comprehension unit. Subsequently, the code semantic comparison unit determines the differences in semantics. Ultimately, the generated code’s score and the explanation of the score are derived by analyzing these semantic differences through an LLM, which considers the problem statement. In order to allow the LLM to analyze the code independently without interfere… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Demonstrates how code is recursively decomposed into sub-codes. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Demonstrates how the Semantic Dependency Decoupling Storage eliminates external dependencies, as well as the process of updating its internal [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: The Kappa coefficients between the four experts. [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: The average expert point for code semantic comprehension at different [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: The impact of incorporating problem statements to different extents on [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: The distribution of expert scores of the scoring explanations generated [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FALCON: Transforming Cyber Threat Intelligence into Deployable IDS Rules with Self-Reflection

    cs.CR 2025-08 conditional novelty 5.0 of 10

    FALCON automates the generation of Snort and YARA intrusion detection rules from cyber threat intelligence using an LLM agent pipeline with a contrastively trained CTI-rule semantic scorer as a ground-truth-free validator.

Reference graph

Works this paper leans on

44 extracted references · 17 canonical work pages · cited by 1 Pith paper

  1. [1]

    Who judges the judge: An empirical study on online judge tests,

    K. Liu, Y . Han, J. M. Zhang, Z. Chen, F. Sarro, M. Harman, G. Huang, and Y . Ma, “Who judges the judge: An empirical study on online judge tests,” in Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis , 2023

    work page 2023

  2. [2]

    Automatic source code evaluation test develop- ment in programming education using grey-box methods,

    J. Skalka and M. Drl ´ık, “Automatic source code evaluation test develop- ment in programming education using grey-box methods,” IEEE Access, 2023

    work page 2023

  3. [3]

    Codexglue: A machine learning benchmark dataset for code understanding and generation,

    S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang et al., “Codexglue: A machine learning benchmark dataset for code understanding and generation,” arXiv preprint arXiv:2102.04664 , 2021

    arXiv 2021

  4. [4]

    Coderl: Mastering code generation through pretrained models and deep reinforcement learning,

    H. Le, Y . Wang, A. D. Gotmare, S. Savarese, and S. C. H. Hoi, “Coderl: Mastering code generation through pretrained models and deep reinforcement learning,” Advances in Neural Information Processing Systems, vol. 35, pp. 21 314–21 328, 2022

    2022

  5. [5]

    Large language models meet nl2code: A survey,

    D. Zan, B. Chen, F. Zhang, D. Lu, B. Wu, B. Guan, W. Yongji, and J.-G. Lou, “Large language models meet nl2code: A survey,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers) , 2023, pp. 7443–7464

    work page 2023

  6. [6]

    Codescore: Evaluating code generation by learning code execution,

    Y . Dong, J. Ding, X. Jiang, Z. Li, G. Li, and Z. Jin, “Codescore: Evaluating code generation by learning code execution,” arXiv preprint arXiv:2301.09043, 2023. 10

    arXiv 2023

  7. [7]

    Spoc: Search-based pseudocode to code,

    S. Kulal, P. Pasupat, K. Chandra, M. Lee, O. Padon, A. Aiken, and P. S. Liang, “Spoc: Search-based pseudocode to code,” Advances in Neural Information Processing Systems , vol. 32, 2019

    2019

  8. [8]

    Measuring coding challenge competence with apps,

    D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song et al., “Measuring coding challenge competence with apps,” arXiv preprint arXiv:2105.09938 , 2021

    arXiv 2021

Show all 44 references
  1. [9]

    Out of the bleu: how should we assess quality of the code generation models?

    M. Evtikhiev, E. Bogomolov, Y . Sokolov, and T. Bryksin, “Out of the bleu: how should we assess quality of the code generation models?” Journal of Systems and Software , vol. 203, p. 111741, 2023

    2023

  2. [10]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting of the Association for Computational Linguistics , 2002, pp. 311–318

    2002

  3. [11]

    A package for automatic evaluation of summaries,

    L. C. ROUGE, “A package for automatic evaluation of summaries,” in Proceedings of Workshop on Text Summarization of ACL, Spain , vol. 5, 2004

    2004

  4. [12]

    Does bleu score work for code migration?

    N. Tran, H. Tran, S. Nguyen, H. Nguyen, and T. Nguyen, “Does bleu score work for code migration?” in 2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC) . IEEE, 2019, pp. 165– 176

    2019

  5. [13]

    Codebertscore: Evaluating code gener- ation with pretrained models of code,

    S. Ren, D. Guo, S. Lu, L. Zhou, S. Liu, D. Tang, N. Sundaresan, M. Zhou, A. Blanco, and S. Ma, “Codebertscore: Evaluating code gener- ation with pretrained models of code,” arXiv preprint arXiv:2302.05527, 2023

    2023 arXiv

  6. [14]

    Codebleu: a method for automatic evaluation of code synthesis,

    Zhou, Shuyan and Alon, Uri and Agarwal, Sumit and Neubig, Graham, “Codebleu: a method for automatic evaluation of code synthesis,” arXiv preprint arXiv:2009.10297, 2020

    2009 arXiv

  7. [15]

    chrf: character n-gram f-score for automatic mt evalu- ation,

    M. Popovi ´c, “chrf: character n-gram f-score for automatic mt evalu- ation,” in Proceedings of the tenth workshop on statistical machine translation, 2015, pp. 392–395

    2015

  8. [16]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  9. [17]

    Large language models are state-of-the-art evaluators of code generation,

    T. Y . Zhuo, “Large language models are state-of-the-art evaluators of code generation,” arXiv preprint arXiv:2304.14317 , 2023

    2023 arXiv

  10. [18]

    Learning to mine aligned code and natural language pairs from stack overflow,

    P. Yin, B. Deng, E. Chen, B. Vasilescu, and G. Neubig, “Learning to mine aligned code and natural language pairs from stack overflow,” in Proceedings of the 15th international conference on mining software repositories, 2018, pp. 476–486

    2018

  11. [19]

    Latent predictor networks for code gen- eration,

    W. Ling, E. Grefenstette, K. M. Hermann, T. Ko ˇcisk`y, A. Senior, F. Wang, and P. Blunsom, “Latent predictor networks for code gen- eration,” arXiv preprint arXiv:1603.06744 , 2016

    2016 arXiv

  12. [20]

    Towards a unified multi-dimensional evaluator for text generation,

    M. Zhong, Y . Liu, D. Yin, Y . Mao, Y . Jiao, P. Liu, C. Zhu, H. Ji, and J. Han, “Towards a unified multi-dimensional evaluator for text generation,” arXiv preprint arXiv:2210.07197 , 2022

    2022 arXiv

  13. [21]

    Survey of hallucination in natural language generation,

    Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y . Xu, E. Ishii, Y . J. Bang, A. Madotto, and P. Fung, “Survey of hallucination in natural language generation,” ACM Computing Surveys , vol. 55, no. 12, pp. 1–38, 2023

    2023

  14. [22]

    Faithful reasoning using large language models,

    A. Creswell and M. Shanahan, “Faithful reasoning using large language models,” arXiv preprint arXiv:2208.14271 , 2022

    2022 arXiv

  15. [23]

    Codenet: A large- scale ai for code dataset for learning a diversity of coding tasks,

    R. Puri, D. S. Kung, G. Janssen, W. Zhang, G. Domeniconi, V . Zolotov, J. Dolby, J. Chen, M. Choudhury, L. Decker et al. , “Codenet: A large- scale ai for code dataset for learning a diversity of coding tasks,” arXiv preprint arXiv:2105.12655, 2021

    2021 arXiv

  16. [24]

    Increasing employability of indian engineering graduates through experiential learning programs and competitive programming: Case study,

    P. R. Nair, “Increasing employability of indian engineering graduates through experiential learning programs and competitive programming: Case study,” Procedia Computer Science , vol. 172, pp. 831–837, 2020

    2020

  17. [25]

    Evaluating large language models trained on code,

    J. T. H. J. Q. Y . H. P. d. O. P. J. K. H. E. e. a. Chen, Mark, “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  18. [26]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou et al. , “Chain-of-thought prompting elicits reasoning in large language models,” Advances in Neural Information Processing Systems , vol. 35, pp. 24 824–24 837, 2022

    2022

  19. [27]

    Fine-grained code clone detection with block-based splitting of abstract syntax tree,

    T. Hu, Z. Xu, Y . Fang, Y . Wu, B. Yuan, D. Zou, and H. Jin, “Fine-grained code clone detection with block-based splitting of abstract syntax tree,” in Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis , 2023, pp. 89–100

    2023

  20. [28]

    Cocoast: Representing source code via hierarchical splitting and re- construction of abstract syntax trees,

    E. Shi, Y . Wang, L. Du, H. Zhang, S. Han, D. Zhang, and H. Sun, “Cocoast: Representing source code via hierarchical splitting and re- construction of abstract syntax trees,” Empirical Software Engineering , vol. 28, no. 6, pp. 1–41, 2023

    2023

  21. [29]

    Blocsum: Block scope-based source code summarization via shared block representation,

    Y . Choi, H. Kim, and J.-H. Lee, “Blocsum: Block scope-based source code summarization via shared block representation,” in Findings of the Association for Computational Linguistics: ACL 2023 , 2023, pp. 11 427– 11 441

    2023

  22. [30]

    Openai gpt-3.5 turbo,

    OpenAI., “Openai gpt-3.5 turbo,” https://platform.openai.com/docs/ guides/text-generation/chat-completions-api, 2022

    2022

  23. [31]

    Exploring security vulnerabilities in competitive programming: An empirical study,

    D. Das, N. S. Mathews, and S. Chimalakonda, “Exploring security vulnerabilities in competitive programming: An empirical study,” in Proceedings of the 26th International Conference on Evaluation and Assessment in Software Engineering , 2022, pp. 110–119

    2022

  24. [32]

    Multipl-e: A scalable and polyglot approach to benchmarking neural code generation,

    F. Cassano, J. Gouwar, D. Nguyen, S. Nguyen, L. Phipps-Costin, D. Pinckney, M.-H. Yee, Y . Zi, C. J. Anderson, M. Q. Feldman, A. Guha, M. Greenberg, and A. Jangda, “Multipl-e: A scalable and polyglot approach to benchmarking neural code generation,” IEEE Transactions on Softwa...

    2023

  25. [33]

    Gpt-4: Language models at scale,

    OpenAI, “Gpt-4: Language models at scale,” https://openai.com/gpt-4/, 2023

    2023

  26. [34]

    Openai gpt-4 turbo,

    OpenAI., “Openai gpt-4 turbo,” https://platform.openai.com/docs/ models/gpt-4-and-gpt-4-turbo, 2023

    2023

  27. [35]

    Meteor: An automatic metric for mt evalua- tion with improved correlation with human judgments,

    S. Banerjee and A. Lavie, “Meteor: An automatic metric for mt evalua- tion with improved correlation with human judgments,” in Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization , 2005, pp. 65–72

    2005

  28. [36]

    Large lan- guage models are zero-shot reasoners,

    T. Kojima, S. S. Gu, M. Reid, Y . Matsuo, and Y . Iwasawa, “Large lan- guage models are zero-shot reasoners,” Advances in neural information processing systems, vol. 35, pp. 22 199–22 213, 2022

    2022

  29. [37]

    Language mod- els are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language mod- els are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020

    1901

  30. [38]

    A new measure of rank correlation,

    M. G. Kendall, “A new measure of rank correlation,” Biometrika, vol. 30, no. 1/2, pp. 81–93, 1938

    1938

  31. [39]

    Pearson correlation coefficient,

    I. Cohen, Y . Huang, J. Chen, J. Benesty, J. Benesty, J. Chen, Y . Huang, and I. Cohen, “Pearson correlation coefficient,” Noise reduction in speech processing, pp. 1–4, 2009

    2009

  32. [40]

    Likert scale: Explored and explained,

    A. Joshi, S. Kale, S. Chandel, and D. K. Pal, “Likert scale: Explored and explained,” British journal of applied science & technology , vol. 7, no. 4, pp. 396–403, 2015

    2015

  33. [41]

    Unsu- pervised translation of programming languages,

    B. Roziere, M.-A. Lachaux, L. Chanussot, and G. Lample, “Unsu- pervised translation of programming languages,” Advances in Neural Information Processing Systems , vol. 33, pp. 20 601–20 611, 2020

    2020

  34. [42]

    An empirical study of auto- mated unit test generation for python,

    S. Lukasczyk, F. Kroiß, and G. Fraser, “An empirical study of auto- mated unit test generation for python,” Empirical Software Engineering , vol. 28, no. 2, p. 36, 2023

    2023

  35. [43]

    Can large language models be an alterna- tive to human evaluations?

    C.-H. Chiang and H.-y. Lee, “Can large language models be an alterna- tive to human evaluations?” arXiv preprint arXiv:2305.01937 , 2023

    2023 arXiv

  36. [44]

    G-eval: Nlg evaluation using gpt-4 with better human alignment, may 2023,

    Y . Liu, D. Iter, Y . Xu, S. Wang, R. Xu, and C. Zhu, “G-eval: Nlg evaluation using gpt-4 with better human alignment, may 2023,” arXiv preprint arXiv:2303.16634. 11

    2023 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.