{"id":"8ff93589-8d6e-4677-9361-838fb587e13c","arxiv_id":"2505.20670","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"MIRROR combines before-execution self-reflection inside each agent with after-execution memory-based reflection, and reports improved pass rates over the baselines tested on StableToolBench and TravelPlanner.","lead":"MIRROR adds a pre-action self-check step to each agent in a multi-agent tool-use pipeline, so plans, tool calls, and answers are critiqued before and after execution. It reports higher pass rates than the baseline methods it compares against on two tool-use benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Intra-reflection's quality-gating is unvalidated: self-scores are not shown to predict execution success, and the ablation cannot separate score-based filtering from extra reasoning compute; a calibration and matched-revision-rate test is needed.","rationale":"I read the paper in good faith and find the empirical results plausible: MIRROR consistently outperforms the chosen baselines across four LLM cores on StableToolBench, and the ablations show each component contributes. The central claim, however, is not just the numbers but the mechanism: intra-reflection as a preventive, score-gated quality filter. That mechanism is load-bearing for the paper's novelty, and it is exactly where the evidence is thinnest. The reader's weakest_assumption identifies the same issue: no calibration study links self-scores to success. My refinement is that the ablation design cannot isolate calibration from the extra compute of the revision loop, and the threshold tuning on the benchmark adds a mild circularity. A matched random-gate experiment would settle the attribution cleanly. If the concern lands, the paper becomes a solid empirical study of a cheaper 'self-refinement with retries' method rather than a demonstration of calibrated pre-execution reflection; that distinction matters for the SOTA/novelty framing. The conditional verdict remains appropriate: strengthen the mechanism evidence, add held-out tuning, and the claim can be accepted. I therefore recommend no change to the reader's conditional verdict.","tokens_in":15651,"tokens_out":5916,"duration_ms":59808,"concrete_test":"On a held-out StableToolBench category with GPT-4o Mini, log each intra-reflection score and the corresponding downstream tool-call/final success; compute the AUC of scores as quality predictors. Then replace the score gate with a random gate that triggers revision at exactly the same per-agent frequency while matching token budget and total revision count. If the random-gate variant matches full MIRROR's Pass Rate within error bars, or if the AUC is near 0.5, the threshold mechanism is not what drives the gains and the central novelty claim needs revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.4 makes the framework's novel component hinge on each agent's self-assigned 1–10 intra-reflection score: outputs scoring below θp/θt/θa are revised, while high-scoring outputs proceed. The central claim that this pre-execution 'intra-reflection' yields state-of-the-art results requires these scores to be informative about downstream quality. No calibration evidence is given. Figure 3 only shows score distributions (Planner and Tool cluster high, Answer spreads), not whether scores correlate with plan, tool-call, or answer success. The ablation in Table 3 removes the entire intra-reflection block, conflating three effects: the self-critique prompt, the score value, and the threshold-triggered revision. Since the revision loop is additional sampled reasoning, the observed gains could come from a compute/self-refinement effect rather than genuine error anticipation. The threshold selection paragraph also tunes θ on StableToolBench itself (Planner θ=7/8/9 → Pass Rates 82.3/83.3/85.7), so the operating point may be overfit to this benchmark. If self-scores are uncalibrated or the threshold is just a compute dial, the paper's distinctive contribution—preventive quality gating—is not supported, even though the headline numbers may still hold as an empirical result.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MIRROR, a multi-agent framework for tool learning that combines intra-reflection (each agent self-scores its intended output on a 1–10 scale and revises it below a learned threshold before execution or handoff) with inter-reflection (post-execution learning via short-term and long-term memory). The method is evaluated on StableToolBench and TravelPlanner across five base LLMs, reporting Pass Rate and Win Rate on StableToolBench and Delivery/Commonsense/Hard-Constraint/Final Pass Rates on TravelPlanner. The paper reports consistent improvements over ReAct, DFSDT, Reflexion, Smurfs, ToolLlama-2, and ToolGen, and ablations show that removing intra-reflection components degrades performance. The central claim is that preventive intra-reflection, on top of post-hoc inter-reflection, yields state-of-the-art tool-learning performance.","tokens_in":15891,"tokens_out":3177,"duration_ms":34609,"significance":"If validated, MIRROR's intra-reflection is a conceptually simple and potentially useful complement to existing post-hoc reflection methods, and the dual-memory inter-reflection mechanism is a reasonable way to propagate lessons within a task. The paper has several strengths: it evaluates across five diverse base LLMs on StableToolBench, reports standard deviations for Pass Rates, includes systematic ablations of each reflection component, and provides a token-cost comparison. The core weakness is that the mechanism's load-bearing assumption—that self-assigned intra-reflection scores reliably predict execution success—is not directly tested, and the TravelPlanner evidence is statistically thin. The contribution is therefore plausible but not yet fully established.","major_comments":[{"comment":"The entire framework's novel component, intra-reflection, depends on each agent's self-assigned 1–10 score being predictive of the quality of the plan, tool call, or answer. No calibration evidence is provided: Figure 3 shows only score distributions, not score conditional on downstream success. The ablation in Table 3 removes the entire intra-reflection block, which simultaneously removes the self-critique prompt, the score assignment, and the threshold-triggered revision. Since the revision loop is additional sampled reasoning, the observed gain (85.7 vs. 78.7 average Pass Rate) could come from extra compute or a generic self-refinement effect rather than from genuine error anticipation via score-based filtering. Please provide a calibration study (e.g., success rate binned by intra-reflection score) and a compute-matched control (e.g., always revise once, or revise with a revision rate matched to the threshold policy but with scores replaced by random values).","section":"Section 3.4, Table 3"},{"comment":"The TravelPlanner evaluation uses only the 180-query validation set, with no standard deviations or significance tests, and the Final Pass Rates are 0, 0, 1.7, and 2.2 across the four rows. Differences of 0 versus 1.7 or 1.7 versus 2.2 on 180 queries are not meaningful, and the comparison includes only ReAct as a baseline. This evidence cannot support the abstract's claim of 'superior performance' on 'both benchmarks.' Please report results on the full test set or at least provide confidence intervals, and compare against at least one reflection-based or multi-agent baseline beyond ReAct.","section":"Section 4.1, Table 2"},{"comment":"The thresholds θp, θt, θa and the number of inter-reflection rounds (5) are selected based on StableToolBench performance: the text reports that Planner thresholds of 7, 8, and 9 yield Pass Rates of 82.3%, 83.3%, and 85.7%, and that 5 rounds outperform 3 and 7 rounds. Because the same benchmark is used for final evaluation, this constitutes test-set tuning and risks overfitting to StableToolBench. The claim that these thresholds are 'generalizable' is unsubstantiated. Please describe how the thresholds were chosen (e.g., on a development split), report sensitivity across models and benchmarks, and ideally show that the selected operating point is not critical to the conclusion.","section":"Section 4.3, 'Score threshold selection' and 'Rounds of Inter-reflection'"}],"minor_comments":[{"comment":"The Win Rate for ReAct is shown as '/'; please state explicitly that ReAct is the reference method and therefore its Win Rate is undefined or 50% by construction.","section":"Table 1"},{"comment":"The sentence 'This superior performance extends to TravelPlanner' overstates the evidence given that Final Pass Rates are near zero for all systems; please temper the claim to match the results.","section":"Section 4.2"},{"comment":"Figure 3 would be far more informative if it displayed threshold lines and success rates conditioned on score, rather than only raw score distributions; this would directly address the calibration question.","section":"Section 4.3, Figure 3"},{"comment":"The phrase 'threshold scores (e.g., Planner Agent at the 90th percentile)' is ambiguous; please specify what distribution the percentile refers to and how it maps to absolute thresholds of 7, 8, and 9.","section":"Section 4.3, 'Score threshold selection'"},{"comment":"The paper does not mention whether code or prompts are released; providing the code and all prompts would substantially aid reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The core idea is interesting and the StableToolBench results are internally consistent, but the calibration issue is central: the claimed mechanism may simply be an extra-compute effect. I would be willing to accept after the authors demonstrate that the intra-reflection score itself is informative, for example through a score-conditioned success-rate analysis and a compute-matched control. The TravelPlanner evidence should also be strengthened or made more modest in its claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"MIRROR is a sensible engineering contribution: it adds pre-execution self-reflection to a multi-agent tool-use pipeline and shows consistent gains on StableToolBench across four base models. The paper deserves a real referee, and the reader's conditional verdict is about right.\n\nThe new thing here is the specific combination: each agent scores its own intended output 1-10 and revises if the score falls below a threshold, while a dual-memory mechanism handles post-execution learning. The authors build directly on Self-refine, Reflexion, and ToT, and the ablations in Table 3 support the broad claim that the reflection components matter — removing all intra-reflection drops average pass rate from 85.7 to 78.7. The token-cost analysis is a plus; they don't hide the price of reflection.\n\nThe main soft spot is exactly what the stress-test note flags: the load-bearing assumption that self-scores predict downstream quality is never validated. No calibration study links the 1–10 scores to actual execution success. The ablations remove the whole intra-reflection block, so they conflate the self-critique prompt, the score, and threshold-triggered revision with added reasoning compute. The gain could come from simply giving the model more chances to think, not from the gate. A matched-revision-rate control or a score-to-success correlation would settle this. It's a fixable weakness, but it means the paper's distinctive mechanism is under-supported.\n\nTwo smaller issues. Thresholds appear tuned on StableToolBench: the planner example shows 7/8/9 yielding 82.3/83.3/85.7, and they adopt the best value. That's a mild circularity, not a fatal one, but it should be acknowledged. TravelPlanner evidence is thin: 180 validation queries, no error bars, final pass rates near zero, and only compared to ReAct. The abstract's 'state-of-the-art' overstates what the baseline set can support.\n\nFor anyone working on LLM agents or tool learning, this is a useful baseline and worth reading. It does not open a new direction, but the empirical results are consistent and the framing is honest about limitations. I'd send it to review. With a calibration study and released code, it could become a solid paper.","headline":"MIRROR is a solid engineering contribution with a real soft spot: the self-score gate that makes 'intra-reflection' distinctive is never shown to predict success.","tokens_in":16443,"tokens_out":3031,"would_cite":true,"duration_ms":30072,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MIRROR claims that making each agent in a tool-use pipeline score and revise its intended plan, tool call, and answer before execution—on top of post-execution memory—raises performance to state-of-the-art on StableToolBench and…","keywords":["multi-agent systems","tool learning","LLM reflection","intra-reflection","inter-reflection","StableToolBench","TravelPlanner","self-assessment"],"falsifier":"A calibration study on StableToolBench would settle it: if outputs that the agent scores 9-10 do not succeed noticeably more often than outputs it scores 6-7, or if force-executing low-scored outputs succeeds as often as high-scored ones, then the threshold is not filtering bad outputs and the gains must be attributed to extra revision computation rather than to the intra-reflection gate.","tokens_in":15433,"feed_emoji":"🧠","tokens_out":7645,"duration_ms":67870,"temperature":0.7,"pith_summary":"The paper argues that LLM agents can reflect before acting, not only after executing a tool call. It presents MIRROR, a three-agent pipeline—planner, tool selector, and answer synthesizer—in which every agent scores its own intended output on a 1-to-10 scale and rewrites it if the score falls below an agent-specific threshold, before the output is executed or passed along. This pre-action 'intra-reflection' is combined with post-execution 'inter-reflection' memories that carry failure lessons forward. Across StableToolBench and TravelPlanner, and with five different base LLMs, MIRROR reports the highest average pass rates among the compared methods, with ablations showing that removing the intra-reflection gates costs several points. The broader claim is that preventive self-critique and retrospective learning are complementary, and together they make tool use more reliable without fine-tuning.","feed_headline":"Self-scoring before acting beats post-hoc reflection in tool agents","feed_subtitle":"By grading and revising each next step before executing it, MIRROR tops prior reflection methods on tool and planning benchmarks.","key_machinery":"The load-bearing mechanism is the intra-reflection gate: a prompt-embedded self-evaluation step in each agent that asks the model to critique its own tentative output, assign a score from 1 to 10, and regenerate the output whenever the score is below that agent's threshold ($\\theta_p$, $\\theta_t$, or $\\theta_a$). The gate runs before the output is executed or handed to the next agent, so bad plans, tool calls, and answers are filtered at the source. It is paired with a dual-memory inter-reflection loop in which Short-Term Memory feeds execution failures and prior self-critiques back into the Tool Agent's next selection for the current subtask, and Long-Term Memory stores full failed trajectories so the Planner Agent can re-decompose the task on a later round.","core_discovery":"On the paper's own terms, MIRROR's central claim is that pre-execution self-assessment by each agent—grading the proposed task decomposition, the chosen tool and parameters, and the assembled final answer, then redoing any output that scores below its threshold—reduces error propagation in multi-agent tool learning and delivers state-of-the-art results. The reported StableToolBench average pass rates are 83.7% with GPT-3.5 Turbo, 83.1% with GPT-4o, 82.9% with Qwen2.5-72B, and 80.9% with Claude 3 Haiku, each above every baseline tested on the same core, and TravelPlanner Delivery Rates rise by 14 to 21 percentage points over ReAct depending on the core. The ablation on GPT-4o Mini attributes most of the benefit to the intra-reflection stage, especially the Answer Agent's final self-check; removing all intra-reflection drops the average pass rate from 85.7% to 78.7%.","pith_inferences":["Beyond the paper: the self-score calibration question is the natural next experiment—if scores cluster near the top, the observed gains could come from the extra revision computation rather than from filtering, and a variant that always rewrites once regardless of score would separate the two.","Beyond the paper: the same gate could apply to any agent emitting a structured action before observing its effect—code edits, database writes, or email drafts—where cheap pre-execution self-checking may prevent costly rollbacks.","Beyond the paper: thresholds could be learned per task type, since the paper only fixes them by percentile; an adaptive threshold would trade token cost against pass rate in a principled way."],"forward_implications":["Reflection does not have to wait for execution: a self-score threshold can act as a model-agnostic quality gate on any structured output before it touches a tool.","The Answer Agent's self-check appears to carry the largest share of the gain, so end-of-trajectory quality control deserves as much attention as upstream planning.","Because MIRROR outperforms the supervised fine-tuning baselines without any parameter update, prompt-level self-assessment can substitute for some SFT in tool use.","The same pre-execution gating transfers from API calls to multi-constraint planning, where commonsense and hard-constraint pass rates increase over ReAct in every tested core."],"supporting_citations":[{"why":"Supplies Reflexion, the post-execution reflection baseline that MIRROR extends and must outperform on both benchmarks.","marker":"[Shinn et al., 2024]"},{"why":"Supplies DFSDT, the decision-tree reflection baseline, and the ToolLLM/ToolBench line from which StableToolBench derives.","marker":"[Qin et al., 2023]"},{"why":"Defines ReAct, the reasoning-acting baseline used for TravelPlanner comparisons and as the Win Rate reference core.","marker":"[Yao et al., 2022]"},{"why":"Provides StableToolBench, including its six evaluation sets and LLM-simulated API executor used for the main pass-rate results.","marker":"[Guo et al., 2024]"},{"why":"Provides TravelPlanner, the multi-constraint planning benchmark with the Delivery, Commonsense, and Hard Constraint metrics.","marker":"[Xie et al., 2024]"},{"why":"Supplies Smurfs, the strongest multi-agent tool-planning baseline that MIRROR compares against on StableToolBench.","marker":"[Chen et al., 2024]"},{"why":"Supplies ToolGen, a supervised fine-tuning baseline for direct tool-call generation that MIRROR outperforms without training.","marker":"[Wang et al., 2024a]"}],"fun_headline_variants":["Pre-action self-scoring lifts tool agents to SOTA","MIRROR grades each step before it runs","Anticipate tool errors before they happen","Pre-check beats post-mortem in tool agents","Self-critique before execution improves tool learning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that an LLM's self-assigned score on a 1-to-10 scale predicts how good its plan, tool call, or answer will actually be, so that discarding and rewriting low-scoring outputs genuinely filters out errors rather than merely spending extra computation.","fun_headline_variants_meta":{"raw":{"variants":["Pre-action self-scoring lifts tool agents to SOTA","MIRROR grades each step before it runs","Anticipate tool errors before they happen","Pre-check beats post-mortem in tool agents","Self-critique before execution improves tool learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000244,"raw_usage":{"total_tokens":1536,"prompt_tokens":951,"completion_tokens":585,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":512}},"tokens_in":567,"tokens_out":585,"duration_ms":6258,"temperature":1.0,"reasoning_tokens":512,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:48:47.397282+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A calibration study on StableToolBench would settle it: if outputs that the agent scores 9-10 do not succeed noticeably more often than outputs it scores 6-7, or if force-executing low-scored outputs succeeds as often as high-scored ones, then the threshold is not filtering bad outputs and the gains must be attributed to extra revision computation rather than to the intra-reflection gate.","supporting_citations":[],"review_version":1}