Pith. sign in

REVIEW 5 major objections 8 minor 52 references

Leveraging Metamemory Agent for Enhanced Data-Free Code Generation in Large Language Models

T0 review · 5 major / 8 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a confidence-gated metamemory workflow lets LLMs improve one-time code generation using only self-recalled examples, outperforming normal, chain-of-thought, analogical, and retrieval-based few-shot prompting on…

desk verdict A useful prompting pipeline with consistent gains, but the confidence-selection mechanism is untested; the paper needs a random-selection control and reported hyperparameters before the central claim is solid. read the letter →

arxiv 2501.07892 v3 pith:5CFP2QB4 submitted 2025-01-14 cs.SE cs.AI

Shengsheng Zhou , Shuai Wang , Liang Ding , Yibing Zhan , Yong Luo , Zheng He , Fu Lin , Dapeng Tao This is my paper · ORCID
classification cs.SEcs.AI
keywords codegenerationlargelanguagemodelsmetamemoryin-contextlearningdata-freepromptingself-recallpass@1
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a large language model can guide its own one-time code generation without any external reference examples by acting as a metamemory agent: the model recalls similar programming problems from its own knowledge, scores its confidence in each recalled problem and its code, keeps the most confident ones, and uses them to build an implementation plan before writing the final answer. The authors claim this four-stage workflow, executed in a single prompt, improves pass@1 on benchmarks without training sets (HumanEval, HumanEval+, StudentEval, Codeforces, MultiPL-E) across open-source and closed-source LLMs, sometimes by more than 29.43%. The point of the work is that self-recall with a confidence gate can replace retrieval-based few-shot prompting in settings where no curated examples exist. A sympathetic reader would take the contribution to be the evaluation-and-selection step that makes self-recalled examples trustworthy enough to help.

What carries the argument

The load-bearing object is the confidence evaluation stage: after recall, the LLM is prompted to rate each recalled problem, its steps, and its code with a score C_i in [0,100] and to select the top M examples. This gate is what distinguishes M2WF from recitation and analogical prompting; the paper claims it makes recalled content reliable enough to guide planning and final code generation.

What would settle it

A calibration experiment would settle it: collect the recalled examples and confidence scores the model produces for a set of HumanEval prompts, check whether the examples that actually pass their own doctests receive higher confidence than those that do not, and compare M2WF against a version that selects the top M examples at random. If random selection matches confidence selection, or if confidence does not track correctness, the evaluation stage is not doing the claimed work.

Watch

Extended reading notes

Core claim

M2WF is a prompted workflow that turns the LLM into a metamemory agent. In the recall stage the model is asked to produce K related programming problems complete with implementation steps and Python code; in the evaluation stage it assigns each recalled item a confidence score from 0 to 100 and keeps the top M; in the planning stage it writes a tutorial and implementation plan for the original problem based on those selected examples; in the guidance stage it writes the final code. The paper's central claim is that this self-contained pipeline reliably improves one-time code generation quality over normal prompting, chain-of-thought, analogical prompting, and a retrieval-based few-shot method, with the confidence-based selection being the mechanism that filters out invented or inaccurate recalled content.

Load-bearing premise

The method assumes that an LLM's self-reported confidence score for a recalled problem reflects whether that recalled problem and its code are actually correct and useful, and the paper provides no calibration evidence connecting confidence scores to correctness.

Editorial extensions

If this is right

  • LLM code generation can be improved purely from the model's own knowledge, with no need for a training set or external retrieval.
  • The workflow transfers across models and languages, since reported improvements appear for models from 7B to 236B parameters and for six programming languages in MultiPL-E.
  • Benchmarks without any training set (HumanEval, HumanEval+, StudentEval, Codeforces) become addressable by self-recall prompting, with pass@1 gains up to 29.43% in the reported settings.
  • The four-stage one-prompt design keeps the number of API calls at one per problem, at the cost of substantially larger input and output token counts.
  • Against retrieval-based few-shot prompting (AceCoder) on Codeforces, M2WF reaches a higher average accuracy than several configurations of the same model with retrieved examples.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The confidence-score gate is a proxy for verification; replacing it with execution-based checks, such as running recalled code against its own doctests, would likely strengthen reliability, though the paper does not test this.
  • Because the whole workflow runs in one prompt, the model's own output is fed back as context, which can be seen as a cheap form of self-conditioning that may degrade as context length grows, making token budgets a practical concern.
  • The same recall-evaluate-plan-guidance pattern could be tested on natural-language reasoning or summarization tasks where recalled content is also hard to verify; the paper only evaluates code generation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 8 minor

Summary. The paper proposes M2WF, a four-stage prompting workflow (recall, evaluation, planning, guidance) for one-time code generation in data-free settings. The LLM first recalls K related programming problems with implementation steps and code, then assigns each a confidence score C_i in [0,100], selects the top M examples, produces an implementation plan for the target problem, and finally generates Python code under that plan. The method is evaluated against normal prompting, CoT prompting, analogical prompting, and (on Codeforces) the retrieval-based AceCoder baseline, using GPT-4, ChatGPT, DeepSeek-Coder-V2, and Mistral-7B-Instruct-v0.2 on HumanEval, StudentEval, HumanEval+, MultiPL-E, and Codeforces. The reported results show consistent average improvements over the baselines, with a headline relative gain of 29.43% on StudentEval, and the central claimed contribution is that the evaluation stage makes self-recalled examples reliable through confidence-based selection.

Significance. The proposed workflow is simple, model-agnostic, and targets a genuine limitation of retrieval-based few-shot prompting in scenarios without training sets. If the evaluation-stage mechanism were properly validated, the method would be a practical plug-and-play alternative. The breadth of models and benchmarks is a strength, and the paper includes ablation studies and an explicit limitations section. However, the central distinctive mechanism—confidence-based selection—is not directly validated, the exact hyperparameter settings for the main results are not disclosed, and several experimental claims lack statistical support, so the current evidence does not yet substantiate the reliability claim. The code is announced but not yet publicly available.

major comments (5)
  1. [§4.2, §5.5 (Eqs. 2-3, Table 2)] The paper's central claim is that the evaluation stage makes self-recalled examples reliable through confidence-based selection (Eqs. 2-3). The only ablation relevant to this claim is Table 2, but it contrasts the full pipeline with conditions that remove the entire evaluation stage, not with a control that selects M examples at random or uses all K examples. Moreover, no calibration analysis is reported: the paper never shows that a recalled example's confidence score C_i predicts its actual correctness or usefulness, e.g., by correlating C_i with whether the recalled code compiles or passes tests on a validation sample. Without such a control or calibration check, the experimental gains cannot be attributed to the confidence-selection mechanism as opposed to the extra self-recall, planning, or prompt length. Please add a random-selection control, an all-examples control, and a calibration analysis (or an oracle-selection analysis) to isolate the contribution of the evaluation stage.
  2. [§5.5 (Figure 5), Tables 3-6] The exact values of K and M used to produce the main results in Tables 3-6 are never reported. Figure 5 is a sensitivity study of K and M on HumanEval, but the paper does not say which (K, M) pair was used for each model and benchmark in the main tables, nor whether these were fixed in advance. If the (K, M) pairs were selected by inspecting HumanEval results and then the same HumanEval numbers are reported as the method's performance, the improvement on HumanEval is optimistically biased. Please disclose the (K, M) settings per model and benchmark, state whether they were tuned on a validation split, and show that the reported improvements are stable across a range of reasonable (K, M) values rather than peaked at a single chosen point.
  3. [§5.5 (Table 2)] The stage-level ablation is under-specified: the text says noise is added 'every 10 characters, with a noise level set to 0.5' but never defines the noise operation (e.g., random character substitution, insertion, deletion, or token-level replacement) or the random seed, so the experiment is not reproducible. In addition, with n=1 on 164 problems, the pass@1 differences between conditions (e.g., 32.32 vs. 30.11 vs. 28.45) correspond to 3-7 problems and are close to the binomial standard error; the paper reports no error bars, no repeated runs, and no statistical test. The conclusion that 'each stage plays a crucial role' is therefore stronger than the evidence supports.
  4. [§4.2, §5.5] Because the method runs in one input/output pass, the 'selection of the top M examples' is an instruction to the LLM, not a deterministically enforced filter: §5.5 states that M2WF 'involves inputting all instructions at once and producing results for each stage at one-time.' The paper does not parse the generated confidence scores and verify that the examples used in planning are actually those with the highest C_i, nor does it check how often the model follows the selection instruction. As a result, the implemented mechanism may deviate from the formalization in Eqs. (2)-(3). Please either enforce selection with a deterministic post-processing step or provide evidence that the model's in-prompt selection matches the stated top-M criterion.
  5. [Abstract, Section 1 (Contributions), Section 5.6 (RQ4, Table 4)] The headline claim 'pass@1 score sometimes increasing by over 29.43%' is not accurately described. The 29.43% figure (Table 4, ChatGPT on StudentEval) is the relative improvement in the average of the four StudentEval subcategories (First Failure, First Success, Last Failure, Last Success), not a pass@1 score; StudentEval is not scored with pass@1 in that table. The abstract and the contributions bullet in Section 1 should state the metric precisely (e.g., 'average StudentEval subcategory score') to avoid overstating the result.
minor comments (8)
  1. [Table 1] Table 1 contains garbled text in the 'Source' column for DeepSeek-Coder-V2 (a long string of '/uni0000...' tokens) and the 'Open-source' column uses an ambiguous '!' character; the table should be regenerated so all model attributes and URLs are legible.
  2. [Table 5 caption] The caption of Table 5 says 'StudentEval benchmark,' but the table reports results on HumanEval+ (as stated in the Section 5.6 text); please correct the caption.
  3. [Title and metadata] The paper uses inconsistent names for the proposed method: the arXiv title says 'Metamemory Agent,' while the manuscript title, abstract, and Section 4 use 'Metamemory Workflow (M2WF)'; please align the terminology across the paper and the metadata.
  4. [Throughout] Several typos and inconsistent notations remain: 'metamemory morkflow' in Section 1, 'Evalution examples' in Section 4.2, 'codeforces' vs. 'Codeforces' in Section 5.1, and the undefined 'n@k' metric in the caption of Table 6.
  5. [§5.1, Table 6] The description of the Codeforces experiment is incomplete: Section 5.1 says Level-A problems are used as the test set following [48], but Table 6 only mentions Level B and Level C as retrieval data; please state explicitly what the test set is and where the Level-A results are reported.
  6. [Figure 5] Figure 5 has three subfigures labeled (a), (b), (c), but the main text never refers to them individually; please add subfigure references and clarify which model and which M value each panel shows.
  7. [§5.4, Eq. (6)] Equation (6) is the standard pass@k estimator, but the surrounding text should define n and c exactly once and note that n=15, 3, or 1 depending on the table; currently the reader must infer the sample size from each table caption.
  8. [References and Table 1] Some references are incomplete or mis-cited, e.g., the model row for DeepSeek-Coder-V2 cites [10], but the reference list entry gives the 2024 preprint for DeepSeek-Coder (not V2); please verify all citations against the reference list.

Circularity Check

1 steps flagged · score 4.0 of 10

The benchmark pass@1 results are externally measured, but the central reliability claim equates reliability with the LLM's self-assigned confidence by construction.

  1. self definitional [Section 4.2 (Eqs. 2-3) and Section 1 Contributions]
    "Subsequently, the evaluation stage assesses the confidence level of each recalled programming problem and its associated code, selecting the top M examples with the highest confidence. ... the M2WF ensures reliability by carefully evaluating the recalled examples and adaptively tailoring such recall&evaluation process for each programming problem."

    Eq. (2) defines evaluation as asking the same LLM to output a confidence score C_i in [0,100] for each recalled problem, and Eq. (3) selects the top M by that self-score; no independent correctness check, calibration, or external correctness signal is reported. The contribution claims this evaluation stage 'ensures reliability' of recalled examples. Under the method's own operationalization, 'reliable' examples are by construction the examples the LLM is most confident about, so the reliability guarantee restates the selection criterion rather than validating it.

full rationale

The paper is an empirical prompting study, not a derivation, and its headline pass@1 numbers are measured against external benchmarks (HumanEval, StudentEval, HumanEval+, Codeforces, MultiPL-E) with normal/CoT/analogical baselines, so the central quantitative claim is not a fitted constant renamed as a prediction. Self-citations [43,44] appear only in related-work surveys and are not load-bearing. The main circularity concern is narrower: the paper's distinctive mechanism—evaluation making self-recalled examples 'reliable'—is operationalized entirely as the same LLM's self-assigned confidence, with no external correctness signal or calibration; the claim that evaluation ensures reliability is therefore true by construction if 'reliable' means 'high self-confidence.' This affects the interpretation of the contribution but not the benchmark comparisons, which remain externally falsifiable. A secondary risk, that K and M may have been selected on HumanEval before reporting HumanEval results, is not stated explicitly and is therefore not counted as a demonstrated circular step.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The method introduces no new physical or mathematical entities. The free parameters are the hyperparameters K and M, which are not reported and appear to have been investigated on HumanEval. The three axioms listed are the implicit assumptions that make the self-recall and self-evaluation pipeline work.

free parameters (2)
  • K (number of recalled programming problems) = not reported; swept on HumanEval in Figure 5
    The recall stage asks the LLM for K examples (Eq. 1). Section 5.5 studies different K values, but the paper never states which K was used in Tables 3-6.
  • M (number of selected high-confidence examples) = not reported; swept on HumanEval in Figure 5
    The evaluation stage selects the top M examples by confidence (Eq. 3). The final M is not reported, and the selection procedure appears to have been examined on HumanEval, creating a risk of test-set tuning.
assumptions (3)
  • domain assumption LLM self-assigned confidence scores correlate with the actual correctness of recalled programming examples and code.
    Section 4.2 makes the LLM score each recalled item from 0 to 100 and selects the top M. No calibration analysis is provided, and the entire benefit of the evaluation stage depends on this assumption.
  • domain assumption A single one-time prompt can make an LLM perform recall, evaluation, planning, and final code generation without format errors or cross-contamination.
    The framework is 'one-time input and output' (Section 4 and token costs in Table 7). The limitations section acknowledges that the model may refuse to recall and that generated code can fail to match benchmark formats, so this assumption is not fully guaranteed.
  • domain assumption LLMs can recall relevant, accurate programming problems from their parametric memory in data-free settings.
    Section 4.1 relies on the LLM producing K related problems with correct steps and Python code. The motivation section itself notes that inaccurate analogy examples can mislead generation, so the accuracy of self-recalled content is a load-bearing premise.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Leveraging Metamemory Agent for Enhanced Data-Free Code Generation in Large Language Models." pith.science (2026). https://pith.science/paper/5CFP2QB4

@misc{pith2026250107892,
  author       = {Pith},
  title        = {Pith review of: Leveraging Metamemory Agent for Enhanced Data-Free Code Generation in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5CFP2QB4}},
  note         = {Machine review of arXiv:2501.07892}
}
read the original abstract

Large language models (LLMs) have shown strong performance in automated code generation, with few-shot prompting widely used for its simplicity and effectiveness. However, few-shot methods depend on curated or manually crafted reference examples, limiting their applicability in data-free coding scenarios such as real-world data-free coding scenarios and benchmarks without training sets. Existing methods that generate reference examples via recitation or analogy cannot guarantee their authenticity or accuracy. Inspired by human metamemory, we propose a novel metamemory agent to enhance one-time code generation in data-free coding scenarios. The agent guides LLMs to recall relevant prior knowledge, evaluate confidence in recalled information, and selectively exploit reliable content for problem solving. This agent removes the need for external reference examples, improves the authenticity and accuracy of recalled knowledge, and adaptively tailors the recall\&evaluation process to each task. Extensive experiments demonstrate that the proposed metamemory agent significantly improves one-time code generation quality across data-free coding scenarios. The AI contribution is the metamemory agent, which makes self-recalled examples reliable through confidence evaluation and selection; the engineering application is data-free automated code generation, validated on eight public benchmarks.

Figures

Figures reproduced from arXiv: 2501.07892 by the authors.

Figure 1
Figure 1. A comparison of AceCoder method [18] (top) and M2WF (bottom). We can clearly see that the AceCoder method requires retrieving relevant examples from the train￾ing set to guide the LLM’s code generation, whereas our M2WF method uses the knowledge of the LLM itself to pro￾vide guidance. 30, 32, 38, 51, 52]. By pretraining on vast amounts of textual data, the general LLMs can process language in ways that closely resem… view at source ↗
Figure 2
Figure 2. Metamemory workflow. 2.2 Code Generation for LLMs The rise of LLMs has significantly advanced the development of automated code generation. However, due to programming lan￾guages’ unique syntax and semantic structures, it is particularly challenging for LLMs to generate high-quality code. To improve the performance of automated code generation, existing research has primarily focused on pre-training or fine-tuning c… view at source ↗
Figure 3
Figure 3. Analysis of the performance of the ChatGPT based on analogical prompting on the HumanEval benchmark [6]. We can clearly observe that due to the incorrect analogy ex￾amples, the proportion of errors in the code generated by ChatGPT has reached 15.24%. Moreover, analogical prompt￾ing is not specifically tailored for code generation tasks. prompting methods are simpler and more flexible and do not re￾quire secondary or… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The overall framework of metamemory workflow ( [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: The performance of the models (i.e., Mistral-7B-Instruct-v0.2, DeepSeek-Coder-V2, and GPT-4) based on [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Performance of the Mistral-7B-Instruct-v0.2 model [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 24 canonical work pages

  1. [1]

    Table 7: An overview of input and output tokens for LLMs using different methods on the HumanEval benchmark

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal 2https://www.tiobe.com/tiobe-index/ Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Trovato et al. Table 7: An overview of input and output tokens for LLMs using different methods on the HumanEval b...

    arXiv 2023

  2. [2]

    Hannah McLean Babe, Sydney Nguyen, Yangtian Zi, Arjun Guha, Molly Q Feldman, and Carolyn Jane Anderson. 2023. StudentEval: a benchmark of student-written prompts for large language models of code. arXiv preprint arXiv:2306.04556 (2023)

    arXiv 2023

  3. [3]

    John Backus. 1973. Programming language semantics and closed applicative languages. In Proceedings of the 1st annual ACM SIGACT-SIGPLAN symposium on principles of programming languages . 71–86

    work page 1973

  4. [4]

    Patrick Bareiß, Beatriz Souza, Marcelo d’Amorim, and Michael Pradel. 2022. Code generation tools (almost) for free? a study of few-shot, pre-trained language models on code. arXiv preprint arXiv:2206.01335 (2022)

    arXiv 2022

  5. [5]

    Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps- Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. 2023. MultiPL-E: a scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions on Software Engineering 49, 7 (2023), 3675–3691

    work page 2023

  6. [6]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)

    arXiv 2021

  7. [7]

    Yangruibo Ding, Marcus J Min, Gail Kaiser, and Baishakhi Ray. 2024. Cycle: Learn- ing to self-refine the code generation. Proceedings of the ACM on Programming Languages 8, OOPSLA1 (2024), 392–418

    work page 2024

  8. [8]

    John H Flavell and Henry M Wellman. 1975. Metamemory. (1975)

    work page 1975

Show all 52 references
  1. [9]

    Mingyang Geng, Shangwen Wang, Dezun Dong, Haotian Wang, Ge Li, Zhi Jin, Xiaoguang Mao, and Xiangke Liao. 2024. Large language models are few- shot summarizers: Multi-intent comment generation via in-context learning. In Proceedings of the 46th IEEE/ACM International Conference...

    2024

  2. [10]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, YK Li, et al. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming–The Rise of Code Intelligence. arXiv preprint arXiv:2401.14196 (2024)

    2024 arXiv

  3. [11]

    SU Hongjin, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, et al. 2022. Selective annotation makes language models better few-shot learners. In The Eleventh International Conference on Learning Representations

    2022

  4. [12]

    Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2023. Large language models for software engineering: A systematic literature review. ACM Transactions on Software Engineering and Methodology (2023)

    2023

  5. [13]

    Xue Jiang, Yihong Dong, Lecheng Wang, Zheng Fang, Qiwei Shang, Ge Li, Zhi Jin, and Wenpin Jiao. 2024. Self-planning code generation with large language models. ACM Transactions on Software Engineering and Methodology 33, 7 (2024), 1–30

    2024

  6. [14]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems 35 (2022), 22199–22213

    2022

  7. [15]

    Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy S Liang. 2019. Spoc: Search-based pseudocode to code.Advances in Neural Information Processing Systems 32 (2019)

    2019

  8. [16]

    Hongxin Li, Jingran Su, Yuntao Chen, Qing Li, and ZHAO-XIANG ZHANG. 2024. SheetCopilot: Bringing software productivity to the next level through large language models. Advances in Neural Information Processing Systems 36 (2024)

    2024

  9. [17]

    Jia Li, Ge Li, Yongmin Li, and Zhi Jin. 2023. Structured chain-of-thought prompt- ing for code generation. arXiv preprint arXiv:2305.06599 (2023)

    2023 arXiv

  10. [18]

    Jia Li, Yunfei Zhao, Yongmin Li, Ge Li, and Zhi Jin. 2023. Acecoder: Utilizing existing code to enhance code generation. arXiv preprint arXiv:2303.17780 (2023)

    2023 arXiv

  11. [19]

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022. Competition-level code generation with alphacode. Science 378, 6624 (2022), 1092–1097

    2022

  12. [20]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems 36 (2024)

    2024

  13. [21]

    Eugene A Lovelace. 1984. Metamemory: monitoring future recallability during study. Journal of Experimental Psychology: Learning, Memory, and Cognition 10, 4 (1984), 756

    1984

  14. [22]

    Ali Madani, Ben Krause, Eric R Greene, Subu Subramanian, Benjamin P Mohr, James M Holton, Jose Luis Olmos, Caiming Xiong, Zachary Z Sun, Richard Socher, et al. 2023. Large language models generate functional protein sequences across diverse families. Nature Biotechnology 41, 8...

    2023

  15. [23]

    Justin M Mittelstädt, Julia Maier, Panja Goerke, Frank Zinn, and Michael Her- mes. 2024. Large language models can outperform humans in social situational judgments. Scientific Reports 14, 1 (2024), 27449

    2024

  16. [24]

    Kentaro Miyamoto, Takahiro Osada, Rieko Setsuie, Masaki Takeda, Keita Tamura, Yusuke Adachi, and Yasushi Miyashita. 2017. Causal neural network of metamem- ory for retrospection in primates. Science 355, 6321 (2017), 188–193

    2017

  17. [25]

    Noor Nashid, Mifta Sintaha, and Ali Mesbah. 2023. Retrieval-based prompt selection for code-related few-shot learning. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 2450–2462

    2023

  18. [26]

    Thomas O Nelson. 1990. Metamemory: A theoretical framework and new findings. In Psychology of learning and motivation . Vol. 26. Elsevier, 125–173

    1990

  19. [27]

    Jasmeet K Pannu and Alfred W Kaszniak. 2005. Metamemory experiments in neurological populations: A review. Neuropsychology review 15 (2005), 105–130

    2005

  20. [28]

    Arkil Patel, Siva Reddy, Dzmitry Bahdanau, and Pradeep Dasigi. 2023. Eval- uating In-Context Learning of Libraries for Code Generation. arXiv preprint arXiv:2311.09635 (2023)

    2023 arXiv

  21. [29]

    Keqin Peng, Liang Ding, Yancheng Yuan, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. 2024. Revisiting demonstration selection strategies in in- context learning. arXiv preprint arXiv:2401.12087 (2024)

    2024 arXiv

  22. [30]

    Keqin Peng, Liang Ding, Qihuang Zhong, Li Shen, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. 2023. Towards Making the Most of ChatGPT for Machine Translation. In Findings of the Association for Computational Linguistics: EMNLP 2023. 5622–5633

    2023

  23. [31]

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiao- qing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 (2023)

    2023 arXiv

  24. [32]

    Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2023. An empirical evaluation of using large language models for automated unit test generation. IEEE Transactions on Software Engineering (2023)

    2023

  25. [33]

    Bennett L Schwartz and Anastasia Efklides. 2012. Metamemory and memory efficiency: Implications for student learning. Journal of Applied Research in Memory and Cognition 1, 3 (2012), 145–151

    2012

  26. [34]

    Zhenwei Shao, Zhou Yu, Meng Wang, and Jun Yu. 2023. Prompting large language models with answer heuristics for knowledge-based visual question answering. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition . 14974–14983

    2023

  27. [35]

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2024. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems 36 (2024)

    2024

  28. [36]

    James WA Strachan, Dalila Albergo, Giulia Borghini, Oriana Pansardi, Eugenio Scaliti, Saurabh Gupta, Krati Saxena, Alessandro Rufo, Stefano Panzeri, Guido Leveraging Metamemory Mechanisms for Enhanced Data-Free Code Generation in LLMs Conference acronym ’XX, June 03–05, 2018, ...

    2024

  29. [37]

    Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, and Denny Zhou. 2023. Recitation-Augmented Language Models. InThe Eleventh International Conference on Learning Representations. https://openreview.net/forum?id=-cqvvvb-NkI

    2023

  30. [38]

    Alex Tamkin, Miles Brundage, Jack Clark, and Deep Ganguli. 2021. Understanding the capabilities, limitations, and societal impact of large language models. arXiv preprint arXiv:2102.02503 (2021)

    2021 arXiv

  31. [39]

    Robert D. Tennent. 1976. The denotational semantics of programming languages. Commun. ACM 19, 8 (1976), 437–453

    1976

  32. [40]

    Eva AM Van Dis, Johan Bollen, Willem Zuidema, Robert Van Rooij, and Claudi L Bockting. 2023. ChatGPT: five priorities for research. Nature 614, 7947 (2023), 224–226

    2023

  33. [41]

    Dave Van Veen, Cara Van Uden, Louis Blankemeier, Jean-Benoit Delbrouck, Asad Aali, Christian Bluethgen, Anuj Pareek, Malgorzata Polacin, Eduardo Pontes Reis, Anna Seehofnerová, et al. 2024. Adapted large language models can outperform medical experts in clinical text summariza...

    2024

  34. [42]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017)

    2017

  35. [43]

    Shuai Wang, Liang Ding, Li Shen, Yong Luo, Bo Du, and Dacheng Tao. 2024. OOP: Object-Oriented Programming Evaluation Benchmark for Large Language Models. arXiv preprint arXiv:2401.06628 (2024)

    2024 arXiv

  36. [44]

    Shuai Wang, Liang Ding, Li Shen, Yong Luo, Zheng He, Wei Yu, and Dacheng Tao. 2024. USCD: Improving Code Generation of LLMs by Uncertainty-Aware Selective Contrastive Decoding. arXiv preprint arXiv:2409.05923 (2024)

    2024 arXiv

  37. [45]

    Zejun Wang, Jia Li, Ge Li, and Zhi Jin. 2023. ChatCoder: Chat-based Refine Requirement Improves LLMs’ Code Generation. arXiv preprint arXiv:2311.00272 (2023)

    2023 arXiv

  38. [46]

    Zhiyong Wu, Yaoxiang Wang, Jiacheng Ye, and Lingpeng Kong. 2023. Self- Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Ordering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics...

    2023

  39. [47]

    Zhen Yang, Jacky Wai Keung, Zeyu Sun, Yunfei Zhao, Ge Li, Zhi Jin, Shuo Liu, and Yishu Li. 2024. Improving domain-specific neural code generation with few-shot meta-learning. Information and Software Technology 166 (2024), 107365

    2024

  40. [48]

    Chi, and Denny Zhou

    Michihiro Yasunaga, Xinyun Chen, Yujia Li, Panupong Pasupat, Jure Leskovec, Percy Liang, Ed H. Chi, and Denny Zhou. 2024. Large Language Models as Analogical Reasoners. In The Twelfth International Conference on Learning Repre- sentations. https://openreview.net/forum?id=AgDICX1h50

    2024

  41. [49]

    Xiao Yu, Baolin Peng, Michel Galley, Jianfeng Gao, and Zhou Yu. 2024. Teaching Language Models to Self-Improve through Interactive Demonstrations. In Pro- ceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lang...

    2024

  42. [50]

    Biao Zhang, Barry Haddow, and Alexandra Birch. 2023. Prompting large language model for machine translation: A case study. In International Conference on Machine Learning. PMLR, 41092–41110

    2023

  43. [51]

    Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2023. Can chatgpt understand too? a comparative study on chatgpt and fine-tuned bert. arXiv preprint arXiv:2302.10198 (2023)

    2023 arXiv

  44. [52]

    Qihuang Zhong, Kang Wang, Ziyang Xu, Juhua Liu, Liang Ding, Bo Du, and Dacheng Tao. 2024. Achieving> 97% on GSM8K: Deeply Understanding the Problems Makes LLMs Perfect Reasoners. arXiv preprint arXiv:2404.14963 (2024)

    2024 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.