AgentCheck: A Reproduce-Intervene-Mitigate Workbench for LLM Agents over MCP

Aritra Mazumder; Nusrat Jahan Lia

AgentCheck turns MCP tool servers into a workbench that records real responses, injects twelve fault types, and lets developers confirm mitigations on the identical failure path before deployment.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-15 09:00 UTC pith:NSP33S55

load-bearing objection Practical MCP fault-injection workbench for agent tool failures; numbers look useful but the cache-then-live confirm step may not isolate the original fault after mitigation divergence. the 3 major comments →

arxiv 2607.11098 v3 pith:NSP33S55 submitted 2026-07-13 cs.SE cs.AIcs.CL

AgentCheck: A Reproduce-Intervene-Mitigate Workbench for LLM Agents over MCP

Aritra Mazumder , Nusrat jahan Lia This is my paper

classification cs.SE cs.AIcs.CL

keywords LLM agentsMCPtool failurefault injectionagent reliabilitymitigationreproduce-intervene-confirmworkbench

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Tool-using LLM agents are usually scored as if every tool call succeeds. AgentCheck is an open-source web workbench that instead treats an MCP server as an intervention surface: it first runs the agent against its real tools and records every response, then re-runs the agent under a twelve-type fault injector while replaying matching calls from cache and sending later calls live once the agent diverges. That cache-then-live policy yields a reproduce–intervene–confirm loop—the developer toggles a mitigation, re-runs against the same fault, and sees whether the failure disappears. Scoring combines deterministic pass/fail rules with an LLM judge validated against human labels. Across five agents the best passes 105 of 120 scenarios and the weakest only 77; failures are typically silent, confident use of incorrect tool outputs rather than crashes. On the weakest agent a simple retry lifts timeout success from as low as 30 percent to 100 percent, while stale-data faults stay near three to four of ten regardless of mitigation.

Core claim

A reproduce–intervene–confirm loop over MCP—recording live tool traffic, replaying matching calls under twelve injected fault types, and going live after divergence—makes agent tool-failure modes reproducible and verifiable before deployment. Best of five agents passes 105/120 scenarios, weakest 77/120; the dominant failure is silent confident use of bad tool output, and retry fully repairs timeouts on the weakest agent while stale-data faults remain largely unmitigated.

What carries the argument

Cache-then-live replay under a twelve-type fault injector: matching tool calls are served from a recorded cache so the failure path is identical, and later calls go live after the agent diverges. Together with dual scoring (deterministic rules plus an LLM judge) this carries the reproduce–intervene–confirm workbench loop.

Load-bearing premise

That the twelve injected fault types plus the cache-matching and post-divergence live-call policy are a faithful enough model of real MCP deployment failures for the measured pass rates and mitigation effects to transfer outside the workbench.

What would settle it

Deploy the same five agents against a production MCP service that experiences natural timeouts, stale values, and description drift, then check whether pass rates still cluster near 105/120 and 77/120 and whether retry still drives timeout success to 100 percent while stale-data success stays near 3–4 of 10.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Developers can verify a mitigation against an identical fault path before shipping rather than discovering silent failures in production.
Agent reliability becomes comparable under a shared 120-scenario fault suite instead of ad-hoc anecdotes.
Timeout-class faults are highly mitigable by retry; stale-data faults are not, so freshness checks need a different design.
Silent confident misuse of incorrect tool outputs is the primary risk surface, not crashes or exceptions.
Teams can score their own agents on the same open fault types before deployment.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The cache-then-live intervention pattern could generalize beyond MCP to other tool protocols if the recording surface is abstracted.
Stubborn stale-data failures suggest tool-side versioning or client-side freshness metadata may be required rather than agent-side retry alone.
An LLM judge validated against human annotations points toward hybrid deterministic-plus-judge scoring as a practical standard for agent reliability suites.
Dominance of silent incorrect-output use implies production monitors should track output consistency and provenance, not only exception rates.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Practical MCP fault-injection workbench for agent tool failures; numbers look useful but the cache-then-live confirm step may not isolate the original fault after mitigation divergence.

read the letter

The one thing to know is that AgentCheck is a concrete open-source workbench that turns an MCP server into a fault-injection surface so you can reproduce tool failures, toggle a mitigation, and re-run. The second is that the abstract’s own description of cache-then-live replay leaves open whether the “confirm” step still hits the identical injected fault once a mitigation changes the call sequence.

What is new is the end-to-end loop over real MCP tools: record live responses, inject one of twelve faults on matching calls, let later calls go live after divergence, then score with deterministic rules plus a human-validated LLM judge. They report five agents on 120 scenarios (best 105, weakest 77), mostly silent confident misuse of bad outputs rather than crashes, and a clear split in mitigation effect—retry lifts timeout success from ~30 % to 100 % on the weakest agent while stale-data stays stuck near 3–4/10. That differential is useful and the problem (production tool faults) is real.

The soft spot is load-bearing for every mitigation number. If retry or another fix makes the agent diverge before the originally faulted call, the second run either escapes the injector or receives a different live fault. The abstract claims “re-runs against the identical fault,” yet the mechanism described does not guarantee it. That is an internal-validity gap, not a minor implementation detail. Everything else—scenario construction, judge–human agreement numbers, baselines, exact fault definitions—is invisible from the abstract alone, so the empirical claims stay provisional.

This is for people building or evaluating tool-using agents on MCP who need a reproducible way to stress tool failures before deployment. A serious referee should see it; the systems idea is clear enough and the measurements, if they hold, are worth having. I would send it out for review with instructions to pressure-test the replay identity and demand the agreement stats and scenario details. Not a desk reject.

Referee Report

3 major / 3 minor

Summary. The manuscript introduces AgentCheck, an open-source web workbench that turns an MCP server into an intervention surface for tool-using LLM agents. It records live tool responses, then re-runs the agent under a 12-type fault injector while replaying matching calls from cache and sending post-divergence calls live, yielding a reproduce–intervene–confirm loop for testing mitigations before deployment. Scoring combines deterministic pass/fail rules with an LLM judge validated against human annotations. Across five agents and 120 scenarios the best agent passes 105/120 and the weakest 77/120; failures are typically silent confident use of incorrect tool outputs. On the weakest agent, retry raises timeout success from as low as 30% to 100%, while stale-data faults remain near 3–4 of 10 regardless of mitigation.

Significance. If the workbench truly isolates identical faults under mitigation and the measured patterns transfer, AgentCheck would be a practical contribution to agent reliability engineering: it makes tool-failure modes reproducible, comparable across agents, and checkable before deployment. Strengths claimed in the abstract include an open-source implementation, a multi-agent empirical comparison, an explicit distinction between timeout and stale-data mitigation efficacy, and dual scoring (deterministic rules plus a human-validated LLM judge). Those elements, if substantiated in the full methods and artifacts, would be useful to the MCP/agent-tooling community.

major comments (3)

[Abstract (cache-then-live / reproduce–intervene–confirm)] The abstract’s central mechanism for the confirm step—“Matching tool calls are replayed from cache, and later tool calls go live after the agent diverges”—does not guarantee re-application of the identical injected fault once a mitigation changes the tool-call sequence. Early divergence (e.g., under retry) can skip the originally faulted call; subsequent live calls then escape the injector or receive uncontrolled faults. Measured recovery (timeout success 30%→100%; stale-data remaining ~3–4/10) therefore cannot be attributed solely to the mitigation, which is load-bearing for every quantitative efficacy claim and for the assertion that failures are “verifiable before deployment.” The manuscript needs either a stronger identity invariant (e.g., fault keyed to logical tool/purpose rather than call order) or an explicit characterization of when confirm remains valid.
[Abstract (120 scenarios / 12 fault types)] Pass rates (105/120 best, 77/120 weakest) and mitigation deltas rest on an unspecified set of 12 fault types and 120 scenarios. Without a clear account of scenario construction, success-rule definition, and whether the same authors co-designed scenarios and scoring, selection bias and limited external validity cannot be ruled out. This is load-bearing for the cross-agent comparison and for transfer claims outside the workbench.
[Abstract (LLM judge / human validation)] Interpretive scoring relies on an LLM judge “validated against human annotations,” but the abstract reports no agreement metric, sample size, or disagreement analysis. Because many failures are described as silent confident misuse rather than crashes, judge reliability is load-bearing for the failure-mode characterization; quantitative validation must be reported.

minor comments (3)

[Abstract] The abstract is dense and packs mechanism, evaluation design, and results into a single block; a clearer separation of workbench design vs. experimental protocol would help readers assess internal validity.
[Abstract (timeout mitigation result)] “As few as 30%” for timeout success is imprecise; report exact per-agent or per-fault counts (and denominators) so the 30%→100% claim can be checked.
[Abstract (five agents)] Name the five agents and the MCP tools/servers used, even briefly, so the 77–105/120 range is interpretable.

Circularity Check

0 steps flagged

No significant circularity: empirical systems paper with measured pass rates under injected faults, not a derivation that reduces to its inputs by construction.

full rationale

This is an abstract-only systems/empirical paper. Its central claims are measured outcomes (pass rates such as 105/120 and 77/120 across five agents; timeout success rising from ~30% to 100% under retry; stale-data remaining ~3–4/10) obtained by running agents under a controlled fault injector with a cache-then-live replay policy, plus an LLM judge validated against human annotations. Nothing in the abstract equates a claimed prediction or first-principles result to a fitted parameter, a self-definition, or a load-bearing self-citation uniqueness theorem. The reproduce–intervene–confirm loop is a methodological design choice whose internal-validity limitations (e.g., divergence before the original faulted call) are correctness/transfer concerns, not circularity: the reported numbers are still observations under that design, not tautologies forced by how a parameter was defined. No equations, no fitted-then-predicted quantities, and no uniqueness claims imported from the same authors appear. Per the hard rules, honest non-finding is the correct outcome; score 0 with empty steps.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 1 invented entities

Abstract-only systems paper. No fitted physical constants. Load-bearing pieces are domain assumptions about MCP tool agents, the chosen fault taxonomy, the cache/live replay policy, and the dual scoring scheme. AgentCheck itself is the engineered artifact, not a postulated physical entity. Free parameters are design choices (fault set size, scenario count, agent set) that define the evaluation surface rather than hidden fit knobs for a closed-form claim.

free parameters (3)

fault_type_set_size = 12
Twelve fault types are chosen by the authors; the abstract does not derive the set from a completeness theorem, so coverage of real failures depends on this design choice.
scenario_count = 120
Headline scores (e.g., 105/120) are relative to an author-defined suite of 120 scenarios whose construction rules are not given in the abstract.
agent_set = 5
Results are reported over five agents; selection criteria and representativeness are not specified in the abstract.

axioms (3)

domain assumption MCP tool servers and tool-using LLM agents are the right abstraction for the targeted deployment failures.
The entire intervention surface is defined over MCP; transfer beyond MCP is assumed, not shown.
ad hoc to paper Cache-matched replay of tool responses until agent divergence, then live calls, preserves a meaningful identical-fault comparison.
Core experimental control in the abstract; validity of mitigation A/B depends on this policy.
domain assumption Deterministic pass/fail rules plus an LLM judge validated against human annotations suffice to score agent success under faults.
Scoring stack is stated as validated; abstract does not give agreement metrics or rule definitions.

invented entities (1)

AgentCheck workbench no independent evidence
purpose: Provide a reproduce–intervene–confirm surface over MCP with fault injection and dual scoring.
Engineered system, not a new physical particle or force; independent evidence would be public code, scenarios, and third-party reuse.

pith-pipeline@v1.1.0-grok45 · 6168 in / 2681 out tokens · 34310 ms · 2026-07-15T09:00:03.797122+00:00 · methodology

0 comments

read the original abstract

Tool-using LLM agents are mostly evaluated assuming all tools work. When a tool times out, returns a week-stale value, or has its description poisoned in deployment, the developer needs a controlled way to reproduce the failure, test a fix, and confirm the fix worked before deployment. We present AgentCheck, an open-source web workbench that turns an MCP server into an intervention surface. AgentCheck runs an agent against its real tools and records every tool response, then re-runs the agent with the response perturbed by a fault (12 types) injector. Matching tool calls are replayed from cache, and later tool calls go live after the agent diverges. This yields a reproduce-intervene-confirm loop: the developer toggles a mitigation, re-runs against the identical fault, and sees if the failure goes away. Scoring has two parts: deterministic pass/fail rules, plus an LLM judge for interpretive labels, validated against human annotations. Across five agents, the best passes 105/120 scenarios and the weakest only 77. The failures are usually silent, confident use of incorrect tool outputs rather than crashes. On the weakest agent, a retry mitigation raises success on timeout error faults from as few as 30% of cases to 100%, whereas stale-data faults remain near 3-4 of 10 regardless of the mitigation. AgentCheck makes these failure modes reproducible, comparable, and verifiable before deployment.