Pith. sign in

REVIEW 3 major objections 4 minor 52 references

Zero-Shot Detection of LLM-Generated Code via Approximated Task Conditioning

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A zero-shot detector that reconstructs the coding task from a snippet distinguishes LLM-written code from human code with roughly 94 percent AUROC.

desk verdict ATC is a genuinely new zero-shot code-detection trick—reconstruct the task, then score conditional entropy—and the empirical results are strong enough to take seriously despite a soft spot around task-recoverability generalization. read the letter →

arxiv 2506.06069 v1 pith:JVANQY3X submitted 2025-06-06 cs.CL cs.LG

Maor Ashkenazi , Ofir Brenner , Tal Furman Shohet , Eran Treister This is my paper
classification cs.CLcs.LG
keywords LLM-generatedcodedetectionzero-shottokenentropytaskconditioningsyntheticgenerationLlamaAUROC
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the reason LLM-generated code is hard to spot is that detectors have been looking at the wrong distribution: unconditional token entropy barely differs between human and machine code, but entropy computed under the task that produced the snippet separates them cleanly. It introduces ATC, a zero-shot detector that prompts a small open-source LLM to invent a task description for a given code snippet, then scores the snippet by mean token entropy conditioned on that description, averaging over several sampled tasks. No access to the generator model or the original prompt is required. On the MBPP and APPS benchmarks the method reports state-of-the-art AUROC values of 94.22 and 93.82 with CodeLlama-13B and four approximated tasks, and it generalizes to C++ and Java. If right, the work turns task reconstruction into a practical detection signal for AI-written code.

What carries the argument

The load-bearing mechanism is the approximated task t produced by prompting the detector LLM with a fixed instruction ('create a simple one line task description that, when given to an LLM, would likely result in the generation of a similar piece of code'), combined with mean token entropy over the conditional distribution P(x|t) computed on code tokens only. The approximation identity P(x|t) ≈ P(x|t*) is what transfers the task-conditional separation, which the paper first demonstrates in the original-prompt setting, to the practical zero-shot setting where the prompt is unknown. Excluding comments from the entropy sum while keeping them in the conditioning context is a deliberate component, because comments act as a localized task description inside the snippet.

What would settle it

Take the benchmark settings and obfuscate function names, remove comments, and shorten the prompts, or apply ATC to a sample of domain-specific repository code with no clear one-line task; if the AUROC drops to the unconditional-entropy level (around 55 on the paper's own baselines), the approximation assumption is what carried the result.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that conditioning on the task reveals a distributional gap that unconditional scoring hides: LLM-generated code has lower mean token entropy than human code when the detector conditions on a reconstructed task, while the unconditional entropies overlap. ATC operationalizes this by (1) using a fixed prompt to ask the detector LLM to generate N concise task descriptions that would plausibly yield the input snippet, and (2) scoring the snippet as the average over tasks of the mean per-token entropy of its code tokens (comments excluded) under the conditional distribution P(x|t). The paper argues that the approximation P(x|t) ≈ P(x|t*), where t* is the true latent task, connects ATC to rewriting-based detectors, which implicitly sample from P(x'|x) as a proxy for P(x'|t*), and that entropy under the approximated task outperforms likelihood, rank, and log-rank scoring within the same framework. The reported results are state-of-the-art AUROC on MBPP (94.22) and APPS (93.82) with CodeLlama-13B and N=4, with a single approximated task already beating prior baselines.

Load-bearing premise

The load-bearing premise is that the detector LLM's generated task description is close enough to the true latent task that the conditional token distribution stays discriminative; for short, obfuscated, or domain-specific code this equivalence could break down.

Editorial extensions

If this is right

  • Detection requires neither the generator model nor the original prompt, so ATC can be applied to arbitrary code snippets after the fact.
  • A small open-source detector (CodeLlama-7B) already outperforms prior zero-shot baselines, lowering the practical barrier for deployment.
  • Averaging over N approximated tasks improves AUROC with diminishing returns from N=4, providing an explicit accuracy-latency trade-off.
  • The method generalizes across programming languages (Python, C++, Java) and across generator models from CodeLlama-7B to GPT-4o-mini.
  • Within the ATC framework, entropy scoring beats mean log-likelihood, LogRank, and LRR, suggesting that global uncertainty is the right signal under task conditioning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A likely failure mode the paper does not test: on obfuscated, domain-specific, or very short code where the detector LLM cannot reconstruct a faithful task, ATC should degrade toward the unconditional-entropy baseline, which would confirm that task approximation is what carries the result.
  • The same task-conditioning signal might transfer to detecting partially edited AI code, since even small human edits would shift conditional entropy; the paper lists this as future work.
  • The large margin with a single approximation suggests the entropy gap is wide, so cheaper detectors or even a single forward pass may be sufficient in many practical settings.
  • Task conditioning may be a general marker for machine-written structured artifacts beyond code, such as SQL queries, configuration files, or formal proofs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes Approximated Task Conditioning (ATC), a zero-shot detector for LLM-generated code. Given a code snippet, ATC first prompts a detector LLM (CodeLlama-7B or 13B) to generate N one-line task descriptions, then computes the mean token entropy of the code tokens under each approximated task and averages over tasks. The paper reports AUROC values on MBPP, APPS, and CodeContest (CPP and Java) across seven generator LLMs, claiming state-of-the-art results (94.22 on MBPP and 93.82 on APPS with CodeLlama-13B and N=4) and robustness to comment removal, decoding temperature, prompt style, and code length. Section 3.2 provides a 'mathematical intuition' relating ATC to unconditional entropy baselines and to code-rewriting detectors such as Ye et al.

Significance. If the empirical claims hold, ATC is a simple and practical advance: it requires no access to the generator LLM, no training, and fewer generated tokens than the code-rewriting state of the art, while using an open-source detector. The paper's empirical scope is a strength: two Python benchmarks, CPP and Java, seven generator models, and several ablations. The release of code and dataset-collection scripts supports reproducibility. However, the strength of the state-of-the-art claim is limited by the comparison protocol to Ye et al. (published numbers from a different subsample) and by the absence of any variance or significance estimation, and the central task-approximation assumption is only validated on well-formed benchmark tasks.

major comments (3)
  1. [§4.1, Tables 1–2] The headline 'state-of-the-art' comparison to Ye et al. is confounded: Ye et al.'s APPS numbers are taken from their paper, which used a random subsample of about 1,500 instances, whereas ATC is evaluated on the full 3,765-instance test set. The authors acknowledge this ('differences in dataset size and sampling procedures may limit direct comparisons'), but the consequence is that the reported margins—for example 93.82 vs 83.25 on APPS—may be driven by dataset composition rather than method superiority. A valid state-of-the-art claim requires either re-running Ye et al. on the same full test set or at least providing confidence intervals or significance tests for the reported AUROC differences.
  2. [§3.2, Eq. (5); §4.4; Conclusion] The method's entire advantage over unconditional entropy rests on the approximation P(x|t) ≈ P(x|t*). The paper tests task approximation only on MBPP and APPS, where tasks are short, well-specified, and strongly determine the solution; Section 4.4 shows the approximated task's conditional distribution is similar to the original's precisely for these benchmarks. The authors themselves state in the Conclusion that 'task approximation in domain-specific repositories may pose additional challenges and warrants further study,' and no experiment measures detection performance as a function of task recoverability (e.g., obfuscated, domain-specific, or very short snippets). As Figure 1 shows, the unconditional entropy baseline is barely separable, so if approximation quality degrades, ATC could collapse toward that baseline. This needs either a bound or an empirical stress test.
  3. [Tables 1–8 generally] None of the AUROC tables reports variance, confidence intervals, or significance tests, and the paper does not state the number of random seeds used for task sampling or code generation. Consequently claims such as 'N=4 further enhances performance' (e.g., 94.22 vs 93.53 on MBPP) and the ordering of ATC against Shi et al. are not shown to be statistically reliable. At minimum, the authors should report standard errors or confidence intervals over multiple seeds and significance tests for the main comparisons.
minor comments (4)
  1. [Algorithm 1, Tables 1–3] The abbreviation is inconsistent: 'AT C' should be 'ATC' throughout (e.g., 'AT CN =1' in Tables 1–3 and 'AT Cw/Task' in Table 4).
  2. [Reference [25]] Reference [25] contains a typo: 'Detecting fack content' should read 'Detecting fake content'.
  3. [§3.2, Eq. (3)] The approximation P(x'|x) ≈ P(x'|t*) is stated without justification; since this section is explicitly an intuition, please label it as an assumption or provide a small supporting experiment.
  4. [Figure 6] The figure would benefit from error bars or per-method variance, since generation length is stochastic.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ATC's score is computed directly from detector-LLM probabilities on held-out test sets, with no fitted parameters or self-citation chains.

full rationale

ATC's derivation chain is self-contained and empirically anchored. The only input to the score is the code snippet x and the fixed detector-LLM distribution G: Algorithm 1 queries G to generate task descriptions t_i and then computes the mean token entropy H(x_jk | x_<jk, t_i) directly from G's output distribution (Eq. 1). No parameter is fitted to labels, no model is trained, and no result is imported from the authors' prior work; the comparison to Ye et al. [33] is an external baseline, and the 'mathematical intuition' in Section 3.2 (Eqs. 2-5) is explicitly an approximation motivating the method, not a derivation that feeds back into the score. The leading assumption P(x|t) ≈ P(x|t*) is a stated approximation whose failure modes (domain-specific snippets) are acknowledged in the Conclusion; this is a robustness limitation, not a circular step. Because the central claim is evaluated on held-out APPS, MBPP, and CodeContest test sets and the reported AUROC sweeps the threshold rather than fitting it, the prediction does not reduce to its inputs by construction. Score 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or mathematical entities. Its free parameters are the number of task approximations N, the sampling parameters, and the prompt style. The key assumptions are heuristic approximations about task reconstruction and the predictability of human versus machine code; these are empirically motivated but not formally proven.

free parameters (3)
  • N (number of approximated tasks) = 1-10, recommended 4
    Ensemble size for task approximation; authors show AUROC improves with N and use N=4 in main results; selected based on test-set performance.
  • Temperature and top_p for task sampling = 0.7, 0.95
    Standard sampling parameters used both for generating code in experiments and for task approximation; chosen following prior work [33].
  • Task approximation prompt style = Regular
    The prompt used to query the detector LLM for task descriptions; authors test 7 styles and use 'Regular' for all main experiments. Choice affects performance (Figure 4).
assumptions (4)
  • domain assumption A latent task variable t* underlies both code and task descriptions, and P(x) = ∫ P(x|t*) Pt(t*) dt* (Eq. 2).
    Introduced in Section 3.2 as the foundation for the conditional entropy argument; not empirically verified separately.
  • ad hoc to paper The detector LLM's conditional distribution P(x|t) with an approximated task t is close to P(x|t*) with the true task (Eq. 5).
    This is the load-bearing approximation; the paper provides examples but no systematic validation across all generators/languages.
  • domain assumption CodeLlama's entropy scores separate human and LLM code when conditioned on the task, with LLM code having lower entropy.
    Central empirical premise, demonstrated in Figure 1 for same-model generation and indirectly across generators in Tables 1-2; not proven for arbitrary code.
  • domain assumption Comments, when present, act as localized task descriptions and should be excluded from entropy scoring while remaining in the context.
    Stated in Section 'Handling Comment Tokens'; supported by the ablation in Table 8 where including comments degrades AUROC.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Zero-Shot Detection of LLM-Generated Code via Approximated Task Conditioning." pith.science (2026). https://pith.science/paper/JVANQY3X

@misc{pith2026250606069,
  author       = {Pith},
  title        = {Pith review of: Zero-Shot Detection of LLM-Generated Code via Approximated Task Conditioning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JVANQY3X}},
  note         = {Machine review of arXiv:2506.06069}
}
read the original abstract

Detecting Large Language Model (LLM)-generated code is a growing challenge with implications for security, intellectual property, and academic integrity. We investigate the role of conditional probability distributions in improving zero-shot LLM-generated code detection, when considering both the code and the corresponding task prompt that generated it. Our key insight is that when evaluating the probability distribution of code tokens using an LLM, there is little difference between LLM-generated and human-written code. However, conditioning on the task reveals notable differences. This contrasts with natural language text, where differences exist even in the unconditional distributions. Leveraging this, we propose a novel zero-shot detection approach that approximates the original task used to generate a given code snippet and then evaluates token-level entropy under the approximated task conditioning (ATC). We further provide a mathematical intuition, contextualizing our method relative to previous approaches. ATC requires neither access to the generator LLM nor the original task prompts, making it practical for real-world applications. To the best of our knowledge, it achieves state-of-the-art results across benchmarks and generalizes across programming languages, including Python, CPP, and Java. Our findings highlight the importance of task-level conditioning for LLM-generated code detection. The supplementary materials and code are available at https://github.com/maorash/ATC, including the dataset gathering implementation, to foster further research in this area.

Figures

Figures reproduced from arXiv: 2506.06069 by the authors.

Figure 1
Figure 1. Box plot of mean token entropy values for human and LLM-generated [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. We use topp = 0.95 and a temperature of 0.7 for sampling, similar to how we generate the code solutions for the experiments. Setting topp will limit sampling to the most probable tokens whose cumulative probability reaches 0.95, and the temperature controls the randomness of the sampling. Additional details are in Appendix B. While a single approximated task already outperforms cur￾rent SOTA, our experiments show th… view at source ↗
Figure 2
Figure 2. Overview of ATC. Given an input code snippet x, we (1) query the detector LLM (CodeLlama) with a fixed prompt to generate task descriptions t1, .., tN for which x might be a valid solution, and (2) compute the conditional entropy of the input code tokens given each approximated task. Given the prob￾ability distribution P(x | ti) for each task ti , the final score is obtained by averaging the mean token entropy only … view at source ↗
Figures from the paper (13 more)
Figure 3
Figure 3. Figure 3: Approximated tasks examples. Left is MBPP, right [PITH_FULL_IMAGE:figures/full_fig_p012_3.png]
Figure 4
Figure 4. Figure 4: ATC with different prompting styles using CodeLlama7b. [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: Temperature effects on MBPP with CodeLlama7b [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Number of generated tokens in dif￾ferent methods. 0 100 200 300 400 Number of Tokens in Code Snippet 0 200 400 600 800 Number of Tokens Generated ATC (Ours) Ye et al [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Impact of code length on APPS. 2 4 6 8 10 Number of Approximated Tasks (m) 0.89 0.90 0.91 0.92 0.93 0.94 0.95 0.96 AUROC CodeLlama 7B CodeLlama 13B CodeGemma 7B StarChat Claude 3 GPT-3.5 GPT-4o Mini Average [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 1
Figure 1. Figure 1: Distribution of the comments-to-code ratio across human-written and [PITH_FULL_IMAGE:figures/full_fig_p023_1.png]
Figure 2
Figure 2. Figure 2: Heatmap of the conditional probability distribution given the [PITH_FULL_IMAGE:figures/full_fig_p024_2.png]
Figure 3
Figure 3. Figure 3: Heatmap of the conditional probability distribution given the [PITH_FULL_IMAGE:figures/full_fig_p025_3.png]
Figure 4
Figure 4. Figure 4: Heatmap of the unconditional probability distribution for the MBPP [PITH_FULL_IMAGE:figures/full_fig_p025_4.png]
Figure 5
Figure 5. Figure 5: Heatmap of the conditional probability distribution given the [PITH_FULL_IMAGE:figures/full_fig_p026_5.png]
Figure 6
Figure 6. Figure 6: Heatmap of the conditional probability distribution given the [PITH_FULL_IMAGE:figures/full_fig_p027_6.png]
Figure 7
Figure 7. Figure 7: Heatmap of the unconditional probability distribution for the APPS ex [PITH_FULL_IMAGE:figures/full_fig_p027_7.png]
Figure 8
Figure 8. Figure 8: ATC with different prompting styles using CodeLlama7b. [PITH_FULL_IMAGE:figures/full_fig_p031_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

52 extracted references · 38 canonical work pages

  1. [1]

    Anthropic (2024), https://www.anthropic.com/

    work page 2024

  2. [2]

    Openai (2024), https://openai.com/api

    work page 2024

  3. [3]

    Stackoverflow developer survey (2024), https://survey.stackoverflow.co/2024/

    work page 2024

  4. [4]

    Meta ai, llama 3.1 (2024), https://llama.meta.com/

    work page 2024

  5. [5]

    arXiv preprint arXiv:2108.07732 (2021)

    Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al.: Program synthesis with large language models. arXiv preprint arXiv:2108.07732 (2021)

    arXiv 2021

  6. [6]

    arXiv preprint arXiv:1906.03351 (2019)

    Bakhtin, A., Gross, S., Ott, M., Deng, Y., Ranzato, M., Szlam, A.: Real or fake? learning to discriminate machine from human generated text. arXiv preprint arXiv:1906.03351 (2019)

    arXiv 2019

  7. [7]

    arXiv preprint arXiv:2310.05130 (2023)

    Bao, G., Zhao, Y., Teng, Z., Yang, L., Zhang, Y.: Fast-detectgpt: Efficient zero-shot detection of machine-generated text via conditional probability curvature. arXiv preprint arXiv:2310.05130 (2023)

    arXiv 2023

  8. [8]

    arXiv preprint arXiv:2108.07258 (2021)

    Bommasani, R., Hudson, D.A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M.S., Bohg, J., Bosselut, A., Brunskill, E., et al.: On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 (2021)

    arXiv 2021

Show all 52 references
  1. [9]

    arXiv preprint arXiv:2412.16594 (2024)

    Demirok, B., Kutlu, M.: Aigcodeset: A new annotated dataset for ai generated code detection. arXiv preprint arXiv:2412.16594 (2024)

    2024 arXiv

  2. [10]

    In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations

    Gehrmann, S., Strobelt, H., Rush, A.: Gltr: Statistical detection and visualization of generated text. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. Association for Computa- tional Linguistics (2019)

    2019

  3. [11]

    In: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) (2021)

    Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., Steinhardt, J.: Measuring coding challenge com- petence with APPS. In: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks T...

    2021

  4. [12]

    In: International Conference on Learning Representations

    Holtzman, A., Buys, J., Du, L., Forbes, M., Choi, Y.: The curious case of neural text degeneration. In: International Conference on Learning Representations

  5. [13]

    arXiv preprint arXiv:2406.16767 (2024)

    Huang, X.Y., Vishnubhotla, K., Rudzicz, F.: The gpt-writingprompts dataset: A comparative analysis of character portrayal in short stories. arXiv preprint arXiv:2406.16767 (2024)

    2024 arXiv

  6. [14]

    In: Proceedings of the 28th International Conference on Computational Linguistics

    Jawahar, G., Abdul-Mageed, M., Laks Lakshmanan, V.: Automatic detection of machine generated text: A critical survey. In: Proceedings of the 28th International Conference on Computational Linguistics. pp. 2296–2309 (2020)

    2020

  7. [15]

    In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    Lee, T., Hong, S., Ahn, J., Hong, I., Lee, H., Yun, S., Shin, J., Kim, G.: Who wrote this code? watermarking for code generation. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 4890–4911 (2024)

    2024

  8. [16]

    Science378(6624), 1092–1097 (2022)

    Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al.: Competition-level code generation with alphacode. Science378(6624), 1092–1097 (2022)

    2022

  9. [17]

    In: International Conference on Machine Learning

    Mitchell, E., Lee, Y., Khazatsky, A., Manning, C.D., Finn, C.: Detectgpt: Zero-shot machine-generated text detection using probability curvature. In: International Conference on Machine Learning. pp. 24950–24962. PMLR (2023)

    2023

  10. [18]

    arXiv preprint arXiv:2301.13852 (2023) 18 M

    Mitrović,S.,Andreoletti,D.,Ayoub,O.:Chatgptorhuman?detectandexplain.ex- plaining decisions of machine learning model for detecting short chatgpt-generated text. arXiv preprint arXiv:2301.13852 (2023) 18 M. Ashkenazi and O. Brenner et al

    2023 arXiv

  11. [19]

    In: 2023 IEEE Symposium on Security and Privacy (SP)

    Pu, J., Sarwar, Z., Abdullah, S.M., Rehman, A., Kim, Y., Bhattacharya, P., Javed, M., Viswanath, B.: Deepfake text detection: Limitations and opportunities. In: 2023 IEEE Symposium on Security and Privacy (SP). pp. 1613–1630 (2023)

    2023

  12. [20]

    Journal of machine learning research21(140), 1–67 (2020)

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research21(140), 1–67 (2020)

    2020

  13. [21]

    arXiv preprint arXiv:2308.12950 (2023)

    Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X.E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al.: Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 (2023)

    2023 arXiv

  14. [22]

    In: Proceedings of the 47th International Conference on Software Engineering (ICSE 2025)

    Shi, Y., Zhang, H., Wan, C., Gu, X.: Between lines of code: Unraveling the dis- tinct patterns of machine and human programmers. In: Proceedings of the 47th International Conference on Software Engineering (ICSE 2025). IEEE (2025)

    2025

  15. [23]

    arXiv preprint arXiv:2306.05540 (2023)

    Su, J., Zhuo, T.Y., Wang, D., Nakov, P.: Detectllm: Leveraging log rank in- formation for zero-shot detection of machine-generated text. arXiv preprint arXiv:2306.05540 (2023)

    2023 arXiv

  16. [24]

    arXiv preprint arXiv:2406.11409 (2024)

    Team, C., Zhao, H., Hui, J., Howland, J., Nguyen, N., Zuo, S., Hu, A., Choquette- Choo, C.A., Shen, J., Kelley, J., et al.: Codegemma: Open code models based on gemma. arXiv preprint arXiv:2406.11409 (2024)

    2024 arXiv

  17. [25]

    In: CEUR Work- shop Proceedings, ECAI’08 Workshop on Plagiarism Analysis, Authorship Identi- fication and Near-Duplication Detection, November

    Thomas, L.: Detecting fack content with relative entropy scoring. In: CEUR Work- shop Proceedings, ECAI’08 Workshop on Plagiarism Analysis, Authorship Identi- fication and Near-Duplication Detection, November. vol. 377, pp. 27–31 (2008)

    2008

  18. [26]

    Hugging Face Blog (2023), https://huggingface.co/blog/starchat-alpha

    Tunstall, L., Lambert, N., Rajani, N., Beeching, E., Le Scao, T., von Werra, L., Han, S., Schmid, P., Rush, A.: Creating a coding assistant with starcoder. Hugging Face Blog (2023), https://huggingface.co/blog/starchat-alpha

    2023

  19. [27]

    Advances in neural information pro- cessing systems 30 (2017)

    Vaswani,A.,Shazeer,N.,Parmar,N.,Uszkoreit,J.,Jones,L.,Gomez,A.N.,Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information pro- cessing systems 30 (2017)

    2017

  20. [28]

    Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing (2021)

    Wang, Y., Wang, W., Joty, S., Hoi, S.C.: CodeT5: Identifier-aware unified pre- trained encoder-decoder models for code understanding and generation. Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing (2021)

    2021

  21. [29]

    arXiv preprint arXiv:2412.16525 (2024)

    Xu, J., Zhang, H., Yang, Y., Cheng, Z., Lyu, J., Liu, B., Zhou, X., Yang, L., Bacchelli, A., Chiam, Y.K., et al.: Investigating efficacy of perplexity in detecting llm-generated code. arXiv preprint arXiv:2412.16525 (2024)

    2024 arXiv

  22. [30]

    Proceedings of the aaai conference on artificial intelligence 38(21), 23155–23162 (2024)

    Xu, Z., Sheng, V.S.: Detecting ai-generated code assignments using perplexity of large language models. Proceedings of the aaai conference on artificial intelligence 38(21), 23155–23162 (2024)

    2024

  23. [31]

    The Twelfth International Conference on Learning Representations (ICLR) (2024)

    Yang, X., Cheng, W., Wu, Y., Petzold, L., Wang, W.Y., Chen, H.: DNA-GPT: Divergent n-gram analysis for training-free detection of gpt-generated text. The Twelfth International Conference on Learning Representations (ICLR) (2024)

    2024

  24. [32]

    arXiv preprint arXiv:2310.05103 (2023)

    Yang, X., Zhang, K., Chen, H., Petzold, L., Wang, W.Y., Cheng, W.: Zero-shot detection of machine-generated codes. arXiv preprint arXiv:2310.05103 (2023)

    2023 arXiv

  25. [33]

    arXiv preprint arXiv:2405.16133 (2024)

    Ye, T., Du, Y., Ma, T., Wu, L., Zhang, X., Ji, S., Wang, W.: Uncovering llm- generated code: A zero-shot synthetic code detector via code rewriting. arXiv preprint arXiv:2405.16133 (2024)

    2024 arXiv

  26. [34]

    CoRR (2023)

    Yu, X., Qi, Y., Chen, K., Chen, G., Yang, X., Zhu, P., Zhang, W., Yu, N.: Gpt paternity test: Gpt generated text detection with gpt genetic inheritance. CoRR (2023)

    2023

  27. [35]

    Advances in neural information processing systems 32 (2019) Detection of LLM-Generated Code via Approximated Task Conditioning 19

    Zellers, R., Holtzman, A., Rashkin, H., Bisk, Y., Farhadi, A., Roesner, F., Choi, Y.: Defending against neural fake news. Advances in neural information processing systems 32 (2019) Detection of LLM-Generated Code via Approximated Task Conditioning 19

    2019

  28. [36]

    You serve as a writing assistant. I will first give you a prompt. You need to tell me a story about the prompt. <PROMPT>

    Zhong, W., Tang, D., Xu, Z., Wang, R., Duan, N., Zhou, M., Wang, J., Yin, J.: Neural deepfake detection with factual structure of text. In: Conference on Empirical Methods in Natural Language Processing (EMNLP) (2020) Appendix A Initial Experiment Setup and Methodology We prov...

    2020

  29. [37]

    Conditional probability distribution given theoriginal task

  30. [38]

    Conditional probability distribution given theapproximated task

  31. [39]

    Yes") else: print(

    Unconditional probability distribution (no task conditioning). D.1 MBPP Example As discussed in Section 4.4, MBPP tasks are short and simple, resulting in approximated tasks that closely resemble the originals. Unsurprisingly, the con- ditional probability distributions for th...

  32. [40]

    Write a function that takes two arguments,x and y, and returns True if both arguments have opposite signs (positive and negative, or negative and positive), and False otherwise

  33. [41]

    Create a Python function that takes in two integersx and y and returns True if the signs ofx and y are opposite, and False otherwise

  34. [42]

    Write a Python function that takes two integersx and y as input and returns a boolean value indicating whether the signs ofx and y are opposite

  35. [43]

    Yes") else: print(

    Write a function calledopposite_Signs that takes two input values (x and y) and returns a boolean value indicating whether the two inputs have op- posite signs (i.e., one is positive and the other is negative, or one is zero and the other is non-zero). E.2 APPS Example Code: a...

  36. [44]

    Yes", otherwise print

    Write a Python program that takes two integers as input and checks if they are equal. If they are equal, print "Yes", otherwise print "No"

  37. [45]

    Yes" if they are equal, and

    Write a Python program that takes two integers as input and prints "Yes" if they are equal, and "No" otherwise

  38. [46]

    Yes" if a is equal to b, and

    Write a Python program that reads two integers from the user, stores them in variables a and b, and then prints "Yes" if a is equal to b, and "No" otherwise

  39. [47]

    Yes" otherwise print

    Write a Python program that takes two integers as input and checks if they are equal. If they are equal, print "Yes" otherwise print "No". Appendix 29 F Exploring Different Task Approximation Prompts F.1 Prompts Inthissection,weexaminedifferentstylesoftaskapproximationprompts....

  40. [48]

    Read the input list from the user

  41. [49]

    Split the input list into two elements

  42. [50]

    Convert the two elements to integers using theint() function

  43. [51]

    Check if the first and second elements are equal

  44. [52]

    Yes" if they are equal, otherwise print

    Print "Yes" if they are equal, otherwise print "No". Storytelling Alice was a developer working on a project where she needed to compare two numbers,a and b. However, she noticed that some users were inputting extra spaces or special characters at the start and end of their in...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.