{"id":"26087822-223a-4ce0-97d6-335102d8f4ad","arxiv_id":"2501.07892","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"M2WF improves one-time LLM code generation by having the model recall, evaluate, and selectively exploit its own remembered coding examples.","lead":"The paper introduces a prompting framework (M2WF) that makes a large language model recall its own coding knowledge, score it for confidence, and use the best remembered examples to plan and write code. It is a data-free alternative to few-shot prompting, which normally needs hand-picked or retrieved reference examples.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"","rationale":"The paper reports consistent pass@1 improvements across multiple models and benchmarks, so the broad empirical claim is plausible and I do not challenge its honesty. The reader's CONDITIONAL verdict is appropriate. My stress-test focuses on the mechanism that supposedly makes M2WF novel: the evaluation stage in §4.2, where the LLM assigns confidence scores C_i in [0,100] and selects the top M recalled examples. This is the only stage that, per the abstract and contributions, 'ensures reliability' and 'makes self-recalled examples reliable.' The paper provides no evidence that these confidence scores are calibrated or even useful: there is no comparison against random selection, no ablation where the selected examples are deliberately swapped for lower-confidence ones, and no correlation analysis between C_i and whether the recalled code is actually correct or relevant. Because all four stages are executed in a single prompt, the 'selection' is also not a deterministic algorithm but another generated output, which makes the reliability claim even harder to verify. If the evaluation stage is inert, M2WF's gains could come entirely from self-recall plus the planning stage, and the method would be closer to analogical prompting with an extra planning step. That would not necessarily invalidate the reported improvements, but it would remove the paper's central conceptual contribution. The proposed concrete test settles this by comparing confidence-based selection with random selection and no selection; a null result would force a recharacterization of the method even if the headline pass@1 numbers stand. I therefore agree with the reader's weakest assumption and see no reason to change the CONDITIONAL verdict based on this pass.","tokens_in":21913,"tokens_out":5168,"duration_ms":58294,"concrete_test":"Re-run M2WF on HumanEval with GPT-4 and Mistral-7B, fixing K=5 and M=2, under three selection policies: (a) top-M by stated confidence C_i as in §4.2, (b) M examples chosen uniformly at random from the K recalled examples, and (c) all K examples passed to the planning stage. Keep temperature 0.8, top-p=0.95, n=15, and report pass@1 with confidence intervals across at least 3 seeds. If policy (a) is not statistically separable from (b) or (c), the confidence-evaluation mechanism is not load-bearing; additionally, independently execute each recalled code against simple test cases and correlate pass/fail with C_i as a calibration check.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing gap is that §4.2's confidence-based selection (Eqs. 2–3) is never compared against a random-selection or no-selection control. Table 2 and Figure 5 vary K, M, and injected noise, but they never demonstrate that the self-assigned confidence scores C_i are informative; no calibration analysis or correlation with actual correctness is reported. Moreover, §5.5 states that M2WF runs all four stages in one input/output pass, so \"selecting the top M examples\" is LLM-generated text rather than an externally enforced deterministic filter, which further weakens the claimed reliability mechanism. If confidence selection is inert, M2WF reduces to analogical prompting with self-generated examples plus a planning step, and the paper's distinctive contribution—making self-recalled examples reliable through evaluation—is not established, even though the overall prompting pipeline may still yield better code. This is the single most load-bearing concern because the evaluation stage is the only component claimed to separate M2WF from plain self-recall or analogical prompting.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes M2WF, a four-stage prompting workflow (recall, evaluation, planning, guidance) for one-time code generation in data-free settings. The LLM first recalls K related programming problems with implementation steps and code, then assigns each a confidence score C_i in [0,100], selects the top M examples, produces an implementation plan for the target problem, and finally generates Python code under that plan. The method is evaluated against normal prompting, CoT prompting, analogical prompting, and (on Codeforces) the retrieval-based AceCoder baseline, using GPT-4, ChatGPT, DeepSeek-Coder-V2, and Mistral-7B-Instruct-v0.2 on HumanEval, StudentEval, HumanEval+, MultiPL-E, and Codeforces. The reported results show consistent average improvements over the baselines, with a headline relative gain of 29.43% on StudentEval, and the central claimed contribution is that the evaluation stage makes self-recalled examples reliable through confidence-based selection.","tokens_in":22073,"tokens_out":10061,"duration_ms":89162,"significance":"The proposed workflow is simple, model-agnostic, and targets a genuine limitation of retrieval-based few-shot prompting in scenarios without training sets. If the evaluation-stage mechanism were properly validated, the method would be a practical plug-and-play alternative. The breadth of models and benchmarks is a strength, and the paper includes ablation studies and an explicit limitations section. However, the central distinctive mechanism—confidence-based selection—is not directly validated, the exact hyperparameter settings for the main results are not disclosed, and several experimental claims lack statistical support, so the current evidence does not yet substantiate the reliability claim. The code is announced but not yet publicly available.","major_comments":[{"comment":"The paper's central claim is that the evaluation stage makes self-recalled examples reliable through confidence-based selection (Eqs. 2-3). The only ablation relevant to this claim is Table 2, but it contrasts the full pipeline with conditions that remove the entire evaluation stage, not with a control that selects M examples at random or uses all K examples. Moreover, no calibration analysis is reported: the paper never shows that a recalled example's confidence score C_i predicts its actual correctness or usefulness, e.g., by correlating C_i with whether the recalled code compiles or passes tests on a validation sample. Without such a control or calibration check, the experimental gains cannot be attributed to the confidence-selection mechanism as opposed to the extra self-recall, planning, or prompt length. Please add a random-selection control, an all-examples control, and a calibration analysis (or an oracle-selection analysis) to isolate the contribution of the evaluation stage.","section":"§4.2, §5.5 (Eqs. 2-3, Table 2)"},{"comment":"The exact values of K and M used to produce the main results in Tables 3-6 are never reported. Figure 5 is a sensitivity study of K and M on HumanEval, but the paper does not say which (K, M) pair was used for each model and benchmark in the main tables, nor whether these were fixed in advance. If the (K, M) pairs were selected by inspecting HumanEval results and then the same HumanEval numbers are reported as the method's performance, the improvement on HumanEval is optimistically biased. Please disclose the (K, M) settings per model and benchmark, state whether they were tuned on a validation split, and show that the reported improvements are stable across a range of reasonable (K, M) values rather than peaked at a single chosen point.","section":"§5.5 (Figure 5), Tables 3-6"},{"comment":"The stage-level ablation is under-specified: the text says noise is added 'every 10 characters, with a noise level set to 0.5' but never defines the noise operation (e.g., random character substitution, insertion, deletion, or token-level replacement) or the random seed, so the experiment is not reproducible. In addition, with n=1 on 164 problems, the pass@1 differences between conditions (e.g., 32.32 vs. 30.11 vs. 28.45) correspond to 3-7 problems and are close to the binomial standard error; the paper reports no error bars, no repeated runs, and no statistical test. The conclusion that 'each stage plays a crucial role' is therefore stronger than the evidence supports.","section":"§5.5 (Table 2)"},{"comment":"Because the method runs in one input/output pass, the 'selection of the top M examples' is an instruction to the LLM, not a deterministically enforced filter: §5.5 states that M2WF 'involves inputting all instructions at once and producing results for each stage at one-time.' The paper does not parse the generated confidence scores and verify that the examples used in planning are actually those with the highest C_i, nor does it check how often the model follows the selection instruction. As a result, the implemented mechanism may deviate from the formalization in Eqs. (2)-(3). Please either enforce selection with a deterministic post-processing step or provide evidence that the model's in-prompt selection matches the stated top-M criterion.","section":"§4.2, §5.5"},{"comment":"The headline claim 'pass@1 score sometimes increasing by over 29.43%' is not accurately described. The 29.43% figure (Table 4, ChatGPT on StudentEval) is the relative improvement in the average of the four StudentEval subcategories (First Failure, First Success, Last Failure, Last Success), not a pass@1 score; StudentEval is not scored with pass@1 in that table. The abstract and the contributions bullet in Section 1 should state the metric precisely (e.g., 'average StudentEval subcategory score') to avoid overstating the result.","section":"Abstract, Section 1 (Contributions), Section 5.6 (RQ4, Table 4)"}],"minor_comments":[{"comment":"Table 1 contains garbled text in the 'Source' column for DeepSeek-Coder-V2 (a long string of '/uni0000...' tokens) and the 'Open-source' column uses an ambiguous '!' character; the table should be regenerated so all model attributes and URLs are legible.","section":"Table 1"},{"comment":"The caption of Table 5 says 'StudentEval benchmark,' but the table reports results on HumanEval+ (as stated in the Section 5.6 text); please correct the caption.","section":"Table 5 caption"},{"comment":"The paper uses inconsistent names for the proposed method: the arXiv title says 'Metamemory Agent,' while the manuscript title, abstract, and Section 4 use 'Metamemory Workflow (M2WF)'; please align the terminology across the paper and the metadata.","section":"Title and metadata"},{"comment":"Several typos and inconsistent notations remain: 'metamemory morkflow' in Section 1, 'Evalution examples' in Section 4.2, 'codeforces' vs. 'Codeforces' in Section 5.1, and the undefined 'n@k' metric in the caption of Table 6.","section":"Throughout"},{"comment":"The description of the Codeforces experiment is incomplete: Section 5.1 says Level-A problems are used as the test set following [48], but Table 6 only mentions Level B and Level C as retrieval data; please state explicitly what the test set is and where the Level-A results are reported.","section":"§5.1, Table 6"},{"comment":"Figure 5 has three subfigures labeled (a), (b), (c), but the main text never refers to them individually; please add subfigure references and clarify which model and which M value each panel shows.","section":"Figure 5"},{"comment":"Equation (6) is the standard pass@k estimator, but the surrounding text should define n and c exactly once and note that n=15, 3, or 1 depending on the table; currently the reader must infer the sample size from each table caption.","section":"§5.4, Eq. (6)"},{"comment":"Some references are incomplete or mis-cited, e.g., the model row for DeepSeek-Coder-V2 cites [10], but the reference list entry gives the 2024 preprint for DeepSeek-Coder (not V2); please verify all citations against the reference list.","section":"References and Table 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears not to be in camera-ready form: the ACM reference format still contains the placeholder '2018' date, Table 1 contains corrupted strings, and the title/abstract terminology is inconsistent. Before resubmission, the authors should also address the missing disclosure of K and M in the main experiments, as this is essential for assessing possible selection bias on HumanEval. The paper's scope and topic are appropriate for the journal, but the evidence supporting the central evaluation-stage mechanism needs substantial strengthening."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take on arXiv:2501.07892 (M2WF). The paper describes a four-stage prompting pipeline for data-free code generation: the model recalls related problems with code, assigns confidence scores, picks the top M, plans, then generates. The composite is new—recitation, confidence scoring, and planning have been explored separately, but not in this one-prompt workflow—and the empirical story is broadly consistent: M2WF beats normal and CoT prompting on HumanEval, StudentEval, and HumanEval+ in almost every model/benchmark cell, and the gains are not tiny (up to ~29% relative on ChatGPT/StudentEval). The paper also deserves credit for reporting a noisy-stage ablation and for listing the token-cost overhead in the limitations.\n\nThe soft spots are real, though. Most importantly, the confidence-selection mechanism is never actually evaluated. The whole pipeline runs in one input/output pass, so \"selecting the top M examples\" is the LLM talking to itself, not an externally enforced filter. There is no comparison against random selection or no-selection. The stage ablation in Table 2 shows that corrupting the evaluation stage hurts, but that just tells you the model uses the prompt text; it does not show that confidence scores are informative. Without a calibration check or a random-selection control, the paper's distinctive claim—that self-recalled examples become reliable through evaluation—is not established.\n\nA second reproducibility gap: the final K and M used in the main tables are never reported. Figure 5 sweeps them, but the numbers in Tables 3–5 are just labeled \"M2WF.\" If K/M were chosen on HumanEval, that is circular for that benchmark, and we cannot check transfer to StudentEval without the values. There are also no error bars or statistical tests, and the code is not actually available yet despite the abstract saying it will be.\n\nThe metamemory framing is a bit loose and the \"adaptive\" claim is overstated—the prompts are fixed. But the method itself is clear and the experiments are extensive.\n\nWho this is for: people working on prompting strategies for code generation, especially in training-set-free settings. It is not a capability breakthrough, and the extra token cost is substantial (4–5x output tokens).\n\nBottom line: worth a serious referee, with the expectation that the authors add a random-selection control, report K/M, and give at least one calibration or error-bar analysis. Without that, the mechanism stays unverified even if the recipe works.","headline":"A useful prompting pipeline with consistent gains, but the confidence-selection mechanism is untested; the paper needs a random-selection control and reported hyperparameters before the central claim is solid.","tokens_in":22613,"tokens_out":5407,"would_cite":false,"duration_ms":49640,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a confidence-gated metamemory workflow lets LLMs improve one-time code generation using only self-recalled examples, outperforming normal, chain-of-thought, analogical, and retrieval-based few-shot prompting on…","keywords":["code generation","large language models","metamemory","in-context learning","data-free prompting","self-recall","pass@1"],"falsifier":"A calibration experiment would settle it: collect the recalled examples and confidence scores the model produces for a set of HumanEval prompts, check whether the examples that actually pass their own doctests receive higher confidence than those that do not, and compare M2WF against a version that selects the top M examples at random. If random selection matches confidence selection, or if confidence does not track correctness, the evaluation stage is not doing the claimed work.","tokens_in":21726,"feed_emoji":"🧠","tokens_out":3954,"duration_ms":36630,"temperature":0.7,"pith_summary":"This paper argues that a large language model can guide its own one-time code generation without any external reference examples by acting as a metamemory agent: the model recalls similar programming problems from its own knowledge, scores its confidence in each recalled problem and its code, keeps the most confident ones, and uses them to build an implementation plan before writing the final answer. The authors claim this four-stage workflow, executed in a single prompt, improves pass@1 on benchmarks without training sets (HumanEval, HumanEval+, StudentEval, Codeforces, MultiPL-E) across open-source and closed-source LLMs, sometimes by more than 29.43%. The point of the work is that self-recall with a confidence gate can replace retrieval-based few-shot prompting in settings where no curated examples exist. A sympathetic reader would take the contribution to be the evaluation-and-selection step that makes self-recalled examples trustworthy enough to help.","feed_headline":"Self-recalled examples lift code generation without training data","feed_subtitle":"Confidence-gated self-recall lifts pass@1 by up to 29.43% on benchmarks with no training sets.","key_machinery":"The load-bearing object is the confidence evaluation stage: after recall, the LLM is prompted to rate each recalled problem, its steps, and its code with a score C_i in [0,100] and to select the top M examples. This gate is what distinguishes M2WF from recitation and analogical prompting; the paper claims it makes recalled content reliable enough to guide planning and final code generation.","core_discovery":"M2WF is a prompted workflow that turns the LLM into a metamemory agent. In the recall stage the model is asked to produce K related programming problems complete with implementation steps and Python code; in the evaluation stage it assigns each recalled item a confidence score from 0 to 100 and keeps the top M; in the planning stage it writes a tutorial and implementation plan for the original problem based on those selected examples; in the guidance stage it writes the final code. The paper's central claim is that this self-contained pipeline reliably improves one-time code generation quality over normal prompting, chain-of-thought, analogical prompting, and a retrieval-based few-shot method, with the confidence-based selection being the mechanism that filters out invented or inaccurate recalled content.","pith_inferences":["The confidence-score gate is a proxy for verification; replacing it with execution-based checks, such as running recalled code against its own doctests, would likely strengthen reliability, though the paper does not test this.","Because the whole workflow runs in one prompt, the model's own output is fed back as context, which can be seen as a cheap form of self-conditioning that may degrade as context length grows, making token budgets a practical concern.","The same recall-evaluate-plan-guidance pattern could be tested on natural-language reasoning or summarization tasks where recalled content is also hard to verify; the paper only evaluates code generation."],"forward_implications":["LLM code generation can be improved purely from the model's own knowledge, with no need for a training set or external retrieval.","The workflow transfers across models and languages, since reported improvements appear for models from 7B to 236B parameters and for six programming languages in MultiPL-E.","Benchmarks without any training set (HumanEval, HumanEval+, StudentEval, Codeforces) become addressable by self-recall prompting, with pass@1 gains up to 29.43% in the reported settings.","The four-stage one-prompt design keeps the number of API calls at one per problem, at the cost of substantially larger input and output token counts.","Against retrieval-based few-shot prompting (AceCoder) on Codeforces, M2WF reaches a higher average accuracy than several configurations of the same model with retrieved examples."],"supporting_citations":[{"why":"Supplies the cognitive metamemory model of recall, evaluation, and regulation that M2WF adapts into a prompted workflow.","marker":"[8]"},{"why":"Provides the analogical prompting baseline and the Codeforces benchmark, including the failure mode of incorrect analogies that motivates confidence evaluation.","marker":"[48]"},{"why":"Provides the retrieval-based few-shot method AceCoder that M2WF is compared against on Codeforces.","marker":"[18]"},{"why":"Supplies the HumanEval benchmark and the pass@k evaluation metric used throughout the experiments.","marker":"[6]"},{"why":"Provides the zero-shot chain-of-thought prompting baseline that M2WF is compared with on all main benchmarks.","marker":"[14]"},{"why":"Represents prior recitation-based self-recall prompting, which the paper claims lacks authenticity guarantees and is improved by the confidence evaluation stage.","marker":"[37]"},{"why":"Supplies the StudentEval benchmark of student-written prompts, a data-free test set where the largest reported pass@1 improvement appears.","marker":"[2]"},{"why":"Supplies the MultiPL-E multilingual benchmark used to test whether M2WF transfers across six programming languages.","marker":"[5]"},{"why":"Provides DeepSeek-Coder-V2, one of the evaluated open-source models, used to test the workflow on a larger-parameter code LLM.","marker":"[10]"}],"fun_headline_variants":["Confidence-gated self-recall lifts data-free code generation","LLM metamemory agent improves code generation without training data","Self-recalled examples with confidence check boost coding","Data-free codegen: LLM recalls, scores, then codes","Metamemory agent makes self-recalled coding examples reliable"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that an LLM's self-reported confidence score for a recalled problem reflects whether that recalled problem and its code are actually correct and useful, and the paper provides no calibration evidence connecting confidence scores to correctness.","fun_headline_variants_meta":{"raw":{"variants":["Confidence-gated self-recall lifts data-free code generation","LLM metamemory agent improves code generation without training data","Self-recalled examples with confidence check boost coding","Data-free codegen: LLM recalls, scores, then codes","Metamemory agent makes self-recalled coding examples reliable"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000513,"raw_usage":{"total_tokens":2468,"prompt_tokens":896,"completion_tokens":1572,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":512,"completion_tokens_details":{"reasoning_tokens":1490}},"tokens_in":512,"tokens_out":1572,"duration_ms":12033,"temperature":1.0,"reasoning_tokens":1490,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:30:05.073686+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A calibration experiment would settle it: collect the recalled examples and confidence scores the model produces for a set of HumanEval prompts, check whether the examples that actually pass their own doctests receive higher confidence than those that do not, and compare M2WF against a version that selects the top M examples at random. If random selection matches confidence selection, or if confidence does not track correctness, the evaluation stage is not doing the claimed work.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the cognitive metamemory model of recall, evaluation, and regulation that M2WF adapts into a prompted workflow."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Represents prior recitation-based self-recall prompting, which the paper claims lacks authenticity guarantees and is improved by the confidence evaluation stage."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MultiPL-E multilingual benchmark used to test whether M2WF transfers across six programming languages."}],"review_version":1}