{"id":"7ff9eb9b-6f31-4b72-a284-f75d90e7eec6","arxiv_id":"2608.10504","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"MEGA reports higher benchmark scores than prior agent optimizers, but its central claim of a self-evolving wisdom graph is not actually demonstrated by the presented experiments.","lead":"This paper describes MEGA, a three-layer infrastructure that aims to make AI coding agents improve themselves by distilling past sessions into reusable knowledge, reasoning over that knowledge as a graph, and feeding optimization results back into the graph. The authors report that MEGA outperforms prior prompt optimizers on four benchmarks, but the experiments do not test the claimed self-evolving loop.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The +7.03-point headline gain is not attributable to the self-evolving Wisdom Graph because the benchmark seeds the graph with golden reference skills and never exercises cross-project accumulation.","rationale":"The reader's weakest assumption identified exactly the evaluation gap: the static 4,207-asset pool with golden skills cannot stand in for a graph built by MEGA's own distillation and feedback loop, and Section 7.2 never tests cross-project accumulation. I agree this is the load-bearing soft spot. The paper's architecture is detailed and the component designs are plausible, but the empirical claims that carry the paper—the +7.03 aggregate gain and the self-evolution property—rest on an evaluation that is misaligned with the claim. The SkillsBench comparison is internally somewhat informative about retrieval quality, but the inclusion of golden skills in the pool weakens even that interpretation. The optimization benchmark has several additional concrete problems: single-run results with no variance, smaller validation sets for MEGA than the cited baselines, no ablation isolating the Wisdom Graph from the other Layer 3 mechanisms, and no matched-condition rerun of baselines. The Scope of Disclosure explicitly withholds algorithmic details, which prevents independent verification of the exact inference and feedback mechanisms. Together these are correctness risks, not stylistic issues. I concur with REJECT on correctness grounds: the central claim is not established by the presented evidence. I did not find a separate more load-bearing concern than the evaluation-loop gap; the misalignment is the concern that matters most.","tokens_in":23516,"tokens_out":1473,"duration_ms":17007,"concrete_test":"Run a controlled cross-project experiment: optimize one held-out project with MEGA, then freeze the resulting Wisdom Graph and measure performance on a second unseen project against (a) the empty-graph cold start and (b) the golden-skill-seeded pool. If the frozen graph from Project 1 does not improve Project 2's baseline or optimization trajectory, the compounding claim is unsupported. Additionally, re-run Table 5 with at least 5 seeds and equal validation splits, and report mean±std for MEGA and all baselines under identical splits.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim is that MEGA forms a self-evolving loop in which optimization evidence refines the Wisdom Graph, so that new projects begin from validated wisdom rather than from scratch. The evaluation in Section 7 does not test this loop at all. Section 7.1 uses a static pool of 4,207 assets that explicitly includes SkillsBench's golden skills; Section 7.2 then runs Layer 3 on the same pool. A pool containing the test tasks' golden reference skills is a best-case retrieval condition for the curation comparison, and it completely bypasses Layer 1 distillation and any cross-project evidence accumulation. The +7.03 aggregate gain over GEPA in Table 5 is reported as a single run with no variance, no seeds, no error bars, and no ablation separating the Wisdom Graph contribution from the multi-agent optimizer, the data augmentation, or the Seed-Epoch attribution discipline. Since baseline scores are cited from prior papers while MEGA uses smaller validation sets (100 vs. 300 for HotpotQA), the comparison is not matched. The paper's self-described Scope of Disclosure says algorithmic details are selectively disclosed, meaning the exact retrieval, reasoning, and feedback derivations cannot be independently checked. The self-evolution premise—evidence feeding back to improve curation and future optimization—is thereby assumed rather than demonstrated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MEGA, a three-layer infrastructure for agent-system optimization. Layer 1 distills agent sessions into validated 'wisdom assets' via clustering and behavioral A/B testing; Layer 2 decomposes these assets into atomic PCR (Primary-Context-Resultant) units stored in a typed Wisdom Graph, performs deductive, abductive, and inductive inference to expand the graph, and retrieves execution plans via a Prize-Collecting Steiner Tree formulation; Layer 3 runs a multi-agent optimization loop over heterogeneous workflows, attributes performance deltas to specific changes via a fixed-seed 'Seed-Epoch' regime, and feeds verdicts back into the graph. The central claim is that this closed loop lets optimization knowledge compound across projects, so new projects start from previously validated wisdom. The empirical sections report a SkillsBench curation study (Table 4) and a four-benchmark optimization study against prior optimizers (Table 5), with an aggregate score of 76.55 versus GEPA's 69.52 on GPT-4.1 Mini.","tokens_in":23817,"tokens_out":2934,"duration_ms":29123,"significance":"If the self-evolving loop were actually demonstrated, the paper would address a real gap in the agent-optimization literature: current optimizers do not accumulate transferable knowledge, and current skill libraries do not reason compositionally over that knowledge. The architecture is specified in unusual detail, including the PCR formalism (Definition 1), the Seed-Epoch attribution scheme (Definition 2), verdict feedback (Definition 3), and the PCST retrieval objective (Eq. 5). The paper also ships a repository link for benchmark configuration and results, which is a genuine strength, and it compares against published baselines rather than only internal ablations. However, the empirical sections do not test the mechanism that defines the paper's contribution: the cross-project, evidence-driven self-evolution of the Wisdom Graph. The evaluation is set up so that the graph already contains the golden skills for the test tasks, and the optimization study measures a single pass over one static pool, so the reported gains cannot be attributed to the claimed loop.","major_comments":[{"comment":"The skill pool for the curation evaluation 'comprises 4,207 assets including SkillsBench's golden skills for all curation systems.' This means MEGA's WG-DB retrieval is evaluated under an oracle condition in which the target task's reference skills are present in the pool, and the same is true for the comparison systems. As a result, the 46.5% pass rate measures retrieval and plan assembly from a pool that already contains the correct solutions; it does not measure Layer 1's distillation of reusable wisdom from raw sessions, nor does it test the cross-project accumulation that the paper's central claim requires. An ablation with the golden skills withheld, or with a pool built only from MEGA's own Layer 1 output, would be needed to attribute the result to the Wisdom Graph rather than to the oracle pool.","section":"Section 7.1, Table 4"},{"comment":"The optimization experiment does not exercise the self-evolving loop that Eq. (1) formalizes. MEGA's Layer 3 is run on the same static 4,207-asset graph evaluated in Section 7.1, and the verdicts produced during a single benchmark are fed back into that same pool; there is no measurement over multiple projects, no sequence of graph updates, and no evidence that cross-project optimization trajectories change the starting point of a later optimization. The abstract's claim that 'optimizing an agent system and evolving the knowledge that guides optimization are one and the same process' is therefore assumed rather than demonstrated. A minimal test would run MEGA on a first project, then a second project with the updated graph, and compare against a graph that received no feedback; the paper reports no such comparison.","section":"Section 7.2 and Eq. (1)"},{"comment":"The headline +7.03 aggregate improvement over GEPA is reported as a single set of point values with no error bars, no multiple-seed variance, and no significance tests. Baseline scores for MIPROv2, TextGrad, GEPA, and Feedback Descent are cited from the respective prior papers rather than reproduced under identical conditions, while MEGA is run with smaller validation sets (100 vs. 300 for HotpotQA, as stated in the setup). These differences make the comparison unmatched: the reported gain could result from evaluation-protocol differences, validation-set size, or random seed rather than from MEGA's mechanisms. The claim that 'MEGA's optimization loop requires less validation data to converge' is not supported by any controlled comparison that varies validation size.","section":"Table 5, Section 7.2"},{"comment":"The Scope of Disclosure states that 'algorithmic details are selectively disclosed at the level of design objectives and formal properties,' and Section 5.6 explicitly says that 'detailed parameter settings and statistical assumptions are omitted.' The architecture relies on a large set of free parameters and unspecified functions: the BIRCH radius threshold T (Eq. 3), the cold/warm gating thresholds θ_cold and θ_warm (Eq. 7), the blending threshold k (Eq. 8), the promotion thresholds n_min and r_min (Eq. 9), the evolution threshold θ_evo (Eq. 10), and the prize/cost functions of the PCST objective (Eq. 5). Because none of these are given concrete values or estimation procedures, the central derivation cannot be independently checked, and the reported experimental results cannot be reproduced from the manuscript alone.","section":"Sections 1.2 and 5.6"}],"minor_comments":[{"comment":"The manuscript contains repeated OCR-like artifacts, including 'with-out' in the abstract, 'relations;' in the abstract, and missing spaces in Section 1; these should be corrected throughout.","section":"General"},{"comment":"The efficiency metric is defined in the text as 'pass-rate improvement per additional megatoken consumed relative to the NoSkills baseline' but the table reports 'Efficiency (score/Mtok)' and the caption defines it as 'pass rate per megatoken consumed'; these definitions are inconsistent and should be unified.","section":"Table 4 and Section 7.1"},{"comment":"Pass rates are computed from 5 attempts per task across 84 tasks with a stochastic agent, yet no error bars or confidence intervals are reported; at minimum, a binomial confidence interval would clarify whether the differences among the three curation methods are meaningful.","section":"Figure 10 and Table 4"},{"comment":"The statement that HotpotQA and PUPA 'approach the ceiling reachable through workflow and prompt optimization' is presented without evidence; the paper does not provide an oracle or ceiling analysis to justify this interpretation.","section":"Section 7.2"},{"comment":"Several references list only partial author lists or use placeholder years (e.g., some 2026 arXiv preprints), which is acceptable for a preprint but should be checked for consistency with the journal's citation requirements.","section":"References"}],"recommendation":"reject","confidential_remarks":"This manuscript reads as a proprietary tech report rather than a scientific evaluation of its central hypothesis. The Scope of Disclosure explicitly reserves the algorithmic details that would be necessary for verification, and the experimental design does not test the self-evolution loop that defines the contribution. The comparison against published baseline numbers without matched reproduction also raises comparability concerns that would need substantial new experiments to resolve. In my assessment, these issues are not local fixes; they require a different evaluation design, so I would not encourage resubmission in the current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead this one for the architecture, not for the numbers. The paper is unusually clear about what it is trying to do: unify skill distillation, compositional reasoning, and attributed optimization feedback into one infrastructure. The PCR decomposition, role-fluid typed graph, PCST-based retrieval, and Seed-Epoch attribution are described in enough detail that you can see the design logic even where the math is informal. I believe the authors know what they want to build, and the related work section is genuinely useful: it correctly identifies that SkillNet and AgentSkillOS still compose whole skills statically, and that prompt/workflow optimizers do not accumulate transferable rationale.\n\nThe weaknesses are real and they are load-bearing. Section 7.1 builds a 4,207-asset pool that explicitly includes SkillsBench's golden skills, and Section 7.2 feeds that same pool into the optimization loop. That is a best-case retrieval setting, and it completely bypasses Layer 1 distillation and any cross-project evidence accumulation. The central claim—that optimization evidence feeds back and refines the Wisdom Graph so future projects start from validated wisdom—is never measured. The +7.03 aggregate gain over GEPA on GPT-4.1 Mini comes from a single run with no variance, no error bars, and no ablation separating the Wisdom Graph contribution from the multi-agent optimizer, the data augmentation, or the Seed-Epoch discipline. Baseline scores are cited from prior papers while MEGA uses smaller validation sets (100 vs. 300 for HotpotQA), so the comparison is not matched. The Scope of Disclosure admits algorithmic details are selectively disclosed, which prevents independent reproduction. These are not minor gaps: the experiments test a static, gold-enriched pool, not the self-evolving loop the title promises.\n\nThat said, I would not call the paper incoherent. The architecture is described honestly enough that a careful reader can see what would need to be tested: a multi-cycle, cross-project study with a pool that excludes golden skills, ablations, and variance reporting. The authors are not hiding the evaluation's shape—it is all in the text. But the text also overclaims the closed loop in the abstract and conclusion, and the reader's stress-test note lands squarely.\n\nWho is this for? Researchers working on skill libraries or agent optimization infrastructure will find the PCR + PCST + Seed-Epoch combination worth discussing. It is a serious design proposal, not a finished empirical result. I would not reject it out of hand; I would send it to referees who can demand the missing longitudinal evaluation and matched baselines. If those are added, this could be a solid systems paper. As it stands, the evidence does not support the self-evolution claim.\n\nRecommendation: engage with it, but treat the headline numbers as unverified. Would I take it to reading group? Maybe, mostly to talk about what a proper cross-project evaluation would look like.","headline":"A genuinely thoughtful architecture paper whose core self-evolution claim is not actually tested by its own experiments; the headline gains come from a setup that seeds the Wisdom Graph with golden skills and never closes the loop.","tokens_in":24335,"tokens_out":717,"would_cite":false,"duration_ms":9284,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MEGA proposes a single infrastructure where optimizing an agent and evolving the knowledge that guides optimization are the same process, and reports a 7.03-point aggregate gain over GEPA on four benchmarks.","keywords":["agent optimization infrastructure","Wisdom Graph","PCR decomposition","compositional retrieval","self-evolving knowledge","Seed-Epoch attribution","skill curation","multi-agent optimization"],"falsifier":"Run MEGA's Layer 1 and Layer 2 on SkillsBench with a skill pool generated only from ordinary agent sessions, withholding the benchmark's golden skills: if pass rate does not exceed the No Skills baseline or stays far below the pre-seeded-pool result, the claim that operational evidence alone grows the graph is unsupported. Alternatively, optimize a project, then start a second project with a cold graph versus the warmed graph; equal performance would refute the compounding claim.","tokens_in":23306,"feed_emoji":"🧠","tokens_out":8231,"duration_ms":67023,"temperature":0.7,"pith_summary":"MEGA tries to establish that the bottleneck in agent development is not any single agent or optimizer but the absence of an infrastructure that both optimizes agent systems and accumulates the resulting knowledge. It proposes a closed three-layer cycle: distill durable wisdom from sessions, reason over that wisdom in a typed graph to assemble execution plans, and run evaluation-driven optimization whose evidence feeds back to refine the graph. The paper reports that the full loop outperforms existing skill-curation pipelines on SkillsBench and existing prompt and workflow optimizers on four benchmarks. If the paper is right, the significance is that each optimization run leaves behind validated strategies, curation patterns, and optimization trajectories, so future projects begin from prior evidence rather than from scratch.","feed_headline":"MEGA's self-evolving loop beats GEPA by 7.03 points on four benchmarks","feed_subtitle":"A Wisdom Graph stores verified skills, composes them into plans, and feeds optimization evidence back into the graph.","key_machinery":"The central object is the typed Wisdom Graph (WG-DB), a directed multi-graph whose nodes are atomic PCR triplets and whose edges carry sufficiency/necessity scores; a role-fluid node pool lets one concept act as the action in one triplet and as the condition in another. Layer 2 reasons over the graph with deduction, abduction, and induction to add unrecorded relations, then solves a Prize-Collecting Steiner Tree formulation to retrieve a connected subgraph that includes low-similarity bridging knowledge. The Seed-Epoch fixed-seed evaluation procedure in Layer 3 is the attribution mechanism that turns optimization outcomes into trustworthy evidence. Together these pieces make the graph self-correcting: verified plans raise evidence confidence, failed combinations are demoted, and contradictions are resolved through graph maintenance.","core_discovery":"The central claim is that one infrastructure can unify three capabilities that current systems keep separate: optimizing agent workflows, accumulating transferable knowledge, and evolving that knowledge from operational evidence. The mechanism is a typed Wisdom Graph in which every wisdom asset is decomposed into atomic PCR (Primary-Context-Resultant) units with typed dependencies and sufficiency/necessity scores; logical inference expands implicit relations among these units, and Prize-Collecting Steiner Tree retrieval assembles role-differentiated execution plans. In Layer 3, a Seed-Epoch regime fixes the evaluation set within each epoch so performance deltas are attributable to strategy changes, and each verdict updates evidence confidence and transfer rates in the graph. On the paper's measurements, the integrated system reaches 46.5 percent pass rate on SkillsBench with the best token efficiency, and an aggregate score of 76.55 on GPT-4.1 Mini across four benchmarks, 7.03 points above GEPA.","pith_inferences":["A testable extension the paper leaves implicit: if the loop truly compounds, warmed graphs should require fewer optimization epochs or less validation data per new project; a longitudinal study tracking convergence time against cumulative verdict count would test this directly.","Because Layer 1 distills wisdom from sessions before any model-specific optimization, the same Wisdom Graph might transfer strategic guidance across different base LLMs; the paper does not test cross-model transfer.","Explicit accumulation of negative evidence, encoded as 'when not to invoke,' suggests a stronger claim than ordinary skill libraries: the graph could steer future retrieval away from known failure combinations, which would show up as reduced error rates on repeated task families.","Layer 3's ability to synthesize evaluation data when none exists implies that the infrastructure also shifts the human role from dataset builder to objective-setter and data-strategy approver; the paper keeps this at the level of pipeline design and marks text-only generation as the current limit."],"forward_implications":["New agent projects could begin with previously validated skills, strategies, curation patterns, and optimization trajectories instead of restarting strategy search from scratch.","Optimization would target the entire heterogeneous workflow, including code nodes, LLM calls, and tool-using agents, rather than prompts alone, because Layer 3 jointly optimizes all node types.","Retrieval quality should improve with use: every attributed verdict updates evidence confidence and historical transfer rates, and maintenance merges duplicates, resolves contradictions, and updates stale content.","The reported results imply that even with smaller validation sets than baseline optimizers use, the closed loop can converge to higher scores by composing wisdom rather than only rewriting prompts."],"supporting_citations":[{"why":"Supplies the four benchmark workflows and reported baseline scores that MEGA's optimization comparison must beat.","marker":"[2]"},{"why":"Provides the Prize-Collecting Steiner Tree formulation that Layer 2 uses for compositional retrieval.","marker":"[5]"},{"why":"Supplies Feedback Descent baseline scores for the optimization comparison.","marker":"[14]"},{"why":"Supplies the AgentSkillOS hierarchical tree-search baseline compared on SkillsBench.","marker":"[15]"},{"why":"Supplies the SkillNet typed relation-graph baseline compared on SkillsBench.","marker":"[16]"},{"why":"Supplies the 84-task benchmark and golden skills used in the curation-quality evaluation.","marker":"[17]"},{"why":"Supplies the MIPROv2 baseline for instruction and demonstration optimization in the comparison.","marker":"[21]"},{"why":"Supplies the sufficiency and necessity causal scoring that edges of the Wisdom Graph carry.","marker":"[25]"},{"why":"Supplies the TextGrad baseline for the optimization comparison.","marker":"[37]"},{"why":"Supplies the single-pass BIRCH clustering method used for behavioral-pattern discovery in Layer 1.","marker":"[40]"}],"fun_headline_variants":["MEGA's Wisdom Graph self-evolves to beat GEPA by 7.03","Self-evolving agent optimizer: MEGA beats GEPA by 7.03","MEGA unifies agent optimization and knowledge evolution","MEGA's loop: optimize agents, evolve wisdom, beat GEPA","MEGA: one infrastructure, self-evolving, 7.03 ahead of GEPA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a pre-seeded skill pool containing the benchmark's own golden skills can stand in for a Wisdom Graph that MEGA's distillation and feedback loop builds and refines itself, so the promised self-evolution is assumed rather than demonstrated.","fun_headline_variants_meta":{"raw":{"variants":["MEGA's Wisdom Graph self-evolves to beat GEPA by 7.03","Self-evolving agent optimizer: MEGA beats GEPA by 7.03","MEGA unifies agent optimization and knowledge evolution","MEGA's loop: optimize agents, evolve wisdom, beat GEPA","MEGA: one infrastructure, self-evolving, 7.03 ahead of GEPA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000675,"raw_usage":{"total_tokens":3100,"prompt_tokens":1004,"completion_tokens":2096,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":1997}},"tokens_in":620,"tokens_out":2096,"duration_ms":13065,"temperature":1.0,"reasoning_tokens":1997,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:18:25.014423+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run MEGA's Layer 1 and Layer 2 on SkillsBench with a skill pool generated only from ordinary agent sessions, withholding the benchmark's golden skills: if pass rate does not exceed the No Skills baseline or stays far below the pre-seeded-pool result, the claim that operational evidence alone grows the graph is unsupported. Alternatively, optimize a project, then start a second project with a cold graph versus the warmed graph; equal performance would refute the compounding claim.","supporting_citations":[{"cited_title":"GEPA:Reflectivepromptevolutioncanoutperformreinforcementlearning","cited_arxiv_id":null,"evidence_quote":"Supplies the four benchmark workflows and reported baseline scores that MEGA's optimization comparison must beat."},{"cited_title":"A general approximation technique for constrained forestproblems","cited_arxiv_id":null,"evidence_quote":"Provides the Prize-Collecting Steiner Tree formulation that Layer 2 uses for compositional retrieval."},{"cited_title":"Optimizinginstructionsanddemonstrationsformulti-stagelanguagemodelprograms","cited_arxiv_id":null,"evidence_quote":"Supplies the MIPROv2 baseline for instruction and demonstration optimization in the comparison."},{"cited_title":"Causality: Models, Reasoning, and Inference.CambridgeUniversityPress,2ndedition, 2009","cited_arxiv_id":null,"evidence_quote":"Supplies the sufficiency and necessity causal scoring that edges of the Wisdom Graph carry."},{"cited_title":"TextGrad: Automaticdifferentiationviatext","cited_arxiv_id":null,"evidence_quote":"Supplies the TextGrad baseline for the optimization comparison."},{"cited_title":"BIRCH: An efficient data clustering method forverylargedatabases","cited_arxiv_id":null,"evidence_quote":"Supplies the single-pass BIRCH clustering method used for behavioral-pattern discovery in Layer 1."}],"review_version":1}