Adaptive Monitoring and Real-World Evaluation of Agentic AI Systems

Manish Shukla

REVIEW 5 major objections 5 minor 8 references

Adaptive Monitoring and Real-World Evaluation of Agentic AI Systems

T0 review · 5 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adaptive monitoring across five evaluation axes detects agentic-AI goal drift in 5.6 seconds versus 12.3 seconds for static thresholds, while cutting false positives from 4.5% to 0.9%.

desk verdict Good plumbing, oversold title: AMDM is a sensible monitoring pipeline, but the headline numbers come from simulations only, and the promised real-world evaluation never appears. read the letter →

arxiv 2509.00115 v3 pith:XRQCEPBV submitted 2025-08-28 cs.AI cs.CLcs.MA

Manish Shukla This is my paper · ORCID

classification cs.AIcs.CLcs.MA

keywords AgenticAIMulti-agentsystemsEvaluationframeworkAdaptiveMulti-DimensionalMonitoring(AMDM)OnlineanomalydetectionGoaldriftSafetyandethics

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that evaluations of agentic AI systems—multi-agent setups that plan, call tools, and execute long tasks—still over-weight capability metrics: a review of 84 papers from 2023–2025 finds 83% report capability numbers while only 30% consider human-centred or economic axes. To close that gap it offers AMDM, an online monitor that normalizes heterogeneous metrics, adapts per-axis thresholds via exponentially weighted moving averages, and flags joint anomalies with the Mahalanobis distance. On simulated workflows with injected goal drift, safety violations, trust shocks, and cost spikes, AMDM detects anomalies in about 5.6 seconds versus 12.3 seconds for static thresholds, cutting false positives from 4.5% to 0.9%. The practical upshot is that this turns a conceptual evaluation framework into a runnable tool at under 3% overhead, and it re-reads three reported industry deployments to show which evaluation axes their productivity numbers omit.

What carries the argument

AMDM (Adaptive Multi-Dimensional Monitoring), an online anomaly-detection algorithm with three coupled parts. Equation (1) computes rolling z-scores so metrics as different as latency, toxicity rate, and cost per interaction live on one scale; Equation (2) maintains a per-axis exponentially weighted moving average that acts as a drifting threshold, the part that tracks gradual goal drift; Equation (3) computes the Mahalanobis distance of the five axis scores, thresholded by the chi-square quantile, the part that flags joint anomalies such as an efficiency spike paired with a safety drop. The load-bearing idea is the combination: adaptive thresholds alone miss multi-axis shocks, and a joint t

What would settle it

Run AMDM on the released two-week production log set (8,400 events, two annotators, Cohen's κ = 0.82) and compare detection latency and false-positive rate against the static-threshold baseline on the same events; if the ~5.6 s / 0.9% advantage does not reproduce on real logs, the simulated claim was favorable by construction. A second check is the simulation code itself: whether each injected anomaly's ground truth was defined by the same z-score/EWMA/Mahalanobis criteria that AMDM monitors.

Watch

Extended reading notes

Core claim

The paper's central claim is that one algorithm can operationalize the five-axis evaluation framework: rolling z-score normalization puts heterogeneous metrics on a common scale, per-axis exponentially weighted moving averages act as adaptive thresholds that track slow shifts, and a Mahalanobis distance over the five axis scores—thresholded by a chi-square quantile—catches multi-axis shocks. AMDM is reported to detect injected goal drift in 5.6 ± 0.9 s versus 12.3 ± 1.4 s for static thresholds, to cut false positives from 4.5% to 0.9%, and to reach a 7.5% false-positive rate at a 95% true-positive rate, versus 18.6% for static thresholds. The division of labor carries the argument: the EWMA

Load-bearing premise

The load-bearing premise is that the simulated workflows—with anomalies injected to match the same five axes AMDM monitors—faithfully represent how real agentic systems fail; the paper reports two weeks of production logs but never gives AMDM's performance on them, so the transfer of the 5.6 s and 0.9% numbers to real deployments is unvalidated.

Editorial extensions

If this is right

Operators could run AMDM on existing agent logs and get both per-axis alerts and a joint anomaly signal, with monitoring overhead under 3% of workflow cost.
If the simulation numbers transfer, goal drift is caught roughly twice as fast (12.3 s → 5.6 s) with five times fewer false alarms (4.5% → 0.9%) than static thresholds.
The ablations imply that adaptive thresholds alone and joint detection alone are each insufficient; the gain comes from running both mechanisms together.
The case-study reanalysis implies that headline productivity claims (20–60% gains, ~30% faster credit decisions) are incomplete without trust, fairness, and energy metrics, giving deployment teams a checklist for balanced reporting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The headline numbers come from simulations whose anomalies were injected to align with the five axes AMDM monitors; the paper describes two weeks of production logs (8,400 events, Cohen's κ = 0.82) but never reports AMDM's results on them, so whether 5.6 s / 0.9% holds in real deployment is untested—and directly checkable with the released code and data.
The decisive experiment would be running AMDM on those real logs against the same static-threshold baseline: reproducing the latency and false-positive advantage would strongly support the claim; failing to do so would indicate the simulation was favorable by construction.
The joint detector's design suggests a governance role the paper leaves implicit: an efficiency spike paired with a safety drop is precisely the signature a human-oversight review queue should surface, so AMDM could double as escalation triage.
Because the anomaly threshold is calibrated for exactly five axes, any deployment that adds or merges evaluation dimensions would need to re-tune the chi-square cutoff; the paper's quiet-period calibration recipe (λ around 0.25, window of 80, ~1% joint false-alarm rate) is a template for that re-tuning.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a reasonable engineering paper with a clear algorithm and an honest limitations section, but the title's "Real-World Evaluation" is not supported by the reported evidence. The claimed latency/FPR wins are simulation-only; the two weeks of real logs in §5.1 are never scored with AMDM.

What's new: the specific combination of rolling z-scores, per-axis EWMA thresholds, and joint Mahalanobis detection for agentic-AI monitoring. Each component is standard, but the integration is clean, the pseudo-code is concrete, and the complexity analysis is useful. The paper ships code and data, and the reproducibility checklist is specific. The ablations and the ROC/PR comparison against multiple baselines, including Isolation Forest and a Kalman filter, show a genuine attempt to situate the method.

Soft spots: the central empirical assertion is Table 1, which reports only simulated goal-drift runs. The real-world logs described in §5.1 have rater labels with high inter-rater agreement, but no AMDM result is reported on them. The contributions and conclusion claim validation "through simulations and real-world logs," which overstates what is demonstrated. Hyperparameters are fixed on the same simulated data without a holdout; the paper says performance is stable across ranges, but that is still in-sample tuning. The broader baseline comparisons in §5.6 are described without numbers, so the headline table only pits AMDM against weaker variants. The systematic-review statistics are cited from another paper, which is fine, but they are not original evidence.

The paper itself concedes in §7.5 that simulations may not capture real deployments, yet the conclusion still asserts real-world validation. That gap is load-bearing: the only numbers supporting the paper's practical value come from a favorable setup where anomalies are injected on the exact axes AMDM monitors.

Verdict: this deserves a serious referee, but the referee should require either AMDM's performance on the 8,400 real events or a title and claims scaled back to simulation results. The algorithm is not the problem; the evidence is. If the real-log numbers are as good as the simulated ones, this becomes a useful practical contribution. As it stands, it is an incremental but honest engineering proposal with an overstated claim.

Referee Report

5 major / 5 minor

Summary. The paper proposes AMDM, an online anomaly-detection algorithm for agentic AI systems that normalizes metrics via rolling z-scores, applies per-axis EWMA-based adaptive thresholds, and performs joint anomaly detection via Mahalanobis distance. It reports simulations for goal drift, safety violations, trust shocks, and cost spikes, claiming reductions in detection latency (12.3 s to 5.6 s) and false-positive rate (4.5% to 0.9%) versus static thresholds. The paper also describes two weeks of real-world logs with rater labels, reanalyzes three industry case studies, and releases code/data with a reproducibility checklist.

Significance. The algorithmic contribution is clear and the code/data release is a strength; the five-axis framing is useful for broadening evaluation practice. However, the headline empirical claims rest entirely on simulated data with anomalies injected along the same axes AMDM monitors, and the promised real-world evaluation is never reported. The contribution is therefore potentially valuable but not yet demonstrated in real deployments; significance is conditional on closing that gap.

major comments (5)

[§5.1, §8, Abstract] The paper states in §5.1 that AMDM was evaluated on two weeks of production logs (8,400 events, Cohen's κ=0.82) and the conclusion claims validation 'through simulations and real-world logs.' However, no AMDM result on these logs appears anywhere; Table 1 and Figures 2–3 report only simulated anomalies. The central claim of real-world evaluation is unsupported. Please either include the real-log results (latency/FPR or ROC/PR) or remove/rewrite the real-world claims.
[§5.1, §5.3, Table 2] Hyperparameters λ=0.25, w=80, and k=χ²_5(0.99) are fixed on the same simulated data used for evaluation, with no explicit train/validation/test split. Injected anomalies are generated along the five axes AMDM monitors, making the setup favorable by construction. The sensitivity analysis in §5.3 is qualitative ('remained stable') and does not address overfitting. Use held-out data or nested cross-validation, and discuss explicitly how the injection protocol might inflate AMDM's apparent advantage.
[§4.2, Algorithm 1, Table 2] The symbol k is used both as a per-axis sensitivity multiplier in Eq. (2)/Algorithm 1 and as the joint chi-square threshold in Table 2 ('k = χ²_5(0.99)'). This makes the algorithm ambiguous and the reported defaults unreproducible. Define separate symbols (e.g., k_axis and α) and clarify which parameter controls per-axis vs. joint alarms.
[§5.2, Table 1] The headline latency/FPR comparison is based on 10 runs and reported only as mean ± standard error, with no significance tests or confidence intervals for the differences. Given the small sample and multiple baselines, the claim that AMDM 'cuts' latency and FPR needs statistical support (e.g., paired bootstrap or Wilcoxon tests) or explicit effect-size reporting.
[§7.5] The limitation that 'simulations are simplified abstractions and may not capture the full complexity of real deployments' directly undercuts the generalization of the quantitative claims. Combined with the absence of real-log results, the abstract's assertions overstate what is demonstrated. Please temper the claims to match the evidence or supply the missing real-world data.

minor comments (5)

[§1] Typo: 'measurement imbalanceundermines' should be 'measurement imbalance undermines'.
[§4.3] Formatting: 'theMahalanobis' should be 'the Mahalanobis'. Also define A as the number of axes before using χ²_A.
[§5.5, Figure 3] AUC and PR-AUC values are not reported; 'AMDM dominates baselines' should be quantified numerically.
[§5.4] The axis attribution plot is 'not shown due to space'; since interpretability is claimed, include it in an appendix or supplement.
[§5.6] Classical detectors (Isolation Forest, One-Class SVM, Kalman filter) are mentioned but no results are shown; add a comparison table or remove the claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: AMDM's latency/FPR results are empirical simulation outcomes, not derived by construction from the self-cited five-axis framework.

full rationale

The paper's central claim (12.3 s to 5.6 s latency; 4.5% to 0.9% FPR, Table 1) is a measured comparison from simulated runs, not a derivation. The AMDM algorithm (rolling z-scores, EWMA thresholds, Mahalanobis distance) uses standard signal-processing components; none of its equations is defined in terms of the anomaly labels or the reported outcome, so there is no self-definitional reduction. The five-axis framework is self-cited from the author's 'Basic' paper, but it is an input taxonomy; it does not formally imply the latency/FPR numbers, so the self-citation is not load-bearing. The nearest concern is that the injected anomaly types (goal drift, safety violations, trust shocks, cost spikes) are aligned with the five axes AMDM monitors; this makes the simulation favorable, but the anomalies are independent ground-truth injections and the comparison against static thresholds is a genuine empirical result, so this is an evaluation-design limitation rather than equation-level circularity. The paper also announces real-world log evaluation in Section 5.1 but reports no AMDM results on those logs, and Section 7.5 concedes 'simulations are simplified abstractions and may not capture the full complexity of real deployments'; this is a missing-evidence / sim-to-real gap, not a circular reduction. Accordingly, no circular step is established under the required standard.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or conceptual entities; it combines known statistical techniques. The main assumptions are the validity of the self-cited five-axis framework and the representativeness of the simulated anomaly streams. The free parameters are standard hyperparameters, but they are tuned on the same simulation data used to produce the headline results, which weakens the evidence.

free parameters (5)

lambda (EWMA smoothing) = 0.25 default, tuned in [0.15, 0.35]
Controls the balance between responsiveness and stability of per-axis thresholds. Chosen by hand during calibration on simulated data, not derived from first principles.
w (rolling window length) = 80 default, tuned in [40, 120]
Window for rolling mean, standard deviation, and z-score normalization. Selected based on 'typical cycle length' in simulations.
k (per-axis anomaly sensitivity multiplier) = Not explicitly given
Used in the condition |S_A(t) - theta_A(t)| > k * sigma_S_A(t). The paper says it should be chosen based on the desired per-axis false-positive rate, but no default value is reported.
Joint threshold alpha (chi-square quantile) = 0.99 (chi-square 5 df)
Sets the joint anomaly false-alarm rate to approximately 1%. Chosen to yield a low false-positive rate in the reported experiments.
Shrinkage parameter for covariance estimator = Not specified
The paper mentions 'shrinkage covariance estimators improved robustness under small sample sizes' but does not specify the shrinkage intensity or how it was selected.

assumptions (4)

domain assumption The five-axis evaluation framework from the author's prior 'Basic' paper is a valid and complete decomposition of agentic AI evaluation.
The entire AMDM design and simulation are built around these five axes. The paper refers to 'Basic' for the conceptual foundation and does not independently justify the axes.
domain assumption Anomalies in agentic systems produce measurable shifts in the metric streams that rolling z-scores, EWMA thresholds, and Mahalanobis distance will detect.
The central claim assumes that the defined metrics and their aggregate axis scores are sensitive to goal drift, safety violations, trust shocks, and cost spikes. The simulation injects anomalies into these metric streams, which is an optimistic construction.
standard math The Mahalanobis distance threshold from the chi-square distribution requires approximate multivariate normality of the axis scores.
Equation (3) and the chi-square threshold rely on the joint distribution of S(t) being multivariate normal, which is not established for agentic AI metric streams.
ad hoc to paper The simulation experiment setup is representative of real agentic AI deployments.
The paper's conclusion that AMDM reduces latency and false positives in practice depends on the simulated workflows being representative. No quantitative real-world validation is presented.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Adaptive Monitoring and Real-World Evaluation of Agentic AI Systems." pith.science (2026). https://pith.science/paper/XRQCEPBV

@misc{pith2026250900115,
  author       = {Pith},
  title        = {Pith review of: Adaptive Monitoring and Real-World Evaluation of Agentic AI Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XRQCEPBV}},
  note         = {Machine review of arXiv:2509.00115}
}

read the original abstract

Agentic artificial intelligence (AI) -- multi-agent systems that combine large language models with external tools and autonomous planning -- are rapidly transitioning from research laboratories into high-stakes domains. Our earlier "Basic" paper introduced a five-axis framework and proposed preliminary metrics such as goal drift and harm reduction but did not provide an algorithmic instantiation or empirical evidence. This "Advanced" sequel fills that gap. First, we revisit recent benchmarks and industrial deployments to show that technical metrics still dominate evaluations: a systematic review of 84 papers from 2023--2025 found that 83% report capability metrics while only 30% consider human-centred or economic axes [2]. Second, we formalise an Adaptive Multi-Dimensional Monitoring (AMDM) algorithm that normalises heterogeneous metrics, applies per-axis exponentially weighted moving-average thresholds and performs joint anomaly detection via the Mahalanobis distance [7]. Third, we conduct simulations and real-world experiments. AMDM cuts anomaly-detection latency from 12.3 s to 5.6 s on simulated goal drift and reduces false-positive rates from 4.5% to 0.9% compared with static thresholds. We present a comparison table and ROC/PR curves, and we reanalyse case studies to surface missing metrics. Code, data and a reproducibility checklist accompany this paper to facilitate replication. The code supporting this work is available at https://github.com/Manishms18/Adaptive-Multi-Dimensional-Monitoring.

Figures

Figures reproduced from arXiv: 2509.00115 by the authors.

**Figure 2.** Average anomaly detection latency across methods. AMDM detects goal drift, safety [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Left: ROC curves for anomaly detection. Right: Precision– Recall curves. AMDM (blue) dominates baselines across a wide range of operating points. true positive rate at a 10 % false positive rate with 22 % lower latency than EWMA-only monitoring. Under shocks AMDM reached the same true positive rate with 35 % lower latency than Mahalanobis-only monitoring. Axis attribution. To aid interpretability we compute the rel… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

8 extracted references · 5 canonical work pages

[1]

AI agents vs. agentic AI: A concep- tual taxonomy, applications and challenges,

R. Sapkota, K. I. Roumeliotis, and M. Karkee, “AI agents vs. agentic AI: A concep- tual taxonomy, applications and challenges,” Information Fusion, 2025. Available on arXiv:2505.10468

arXiv 2025
[2]

The measurement imbalance in agentic AI evaluation undermines industry productivity claims,

K. J. Meimandi, G. Ar´ anguiz-Dias, G. R. Kim, L. Saadeddin, and M. J. Kochenderfer, “The measurement imbalance in agentic AI evaluation undermines industry productivity claims,” 2025. arXiv:2506.02064. This work documents that capability metrics dominate agentic AI evaluations (around 83% of surveyed studies) while human-centred and economic metrics are ...

arXiv 2025
[3]

Technical report: Evaluating goal drift in language model agents,

R. Arike, E. Donoway, H. Bartsch, and M. Hobbhahn, “Technical report: Evaluating goal drift in language model agents,” 2025. arXiv:2505.02709. The authors demonstrate that agents given a goal and then exposed to competing objectives exhibit gradual drift

arXiv 2025
[4]

Seizing the agentic AI advantage,

B. Heger, “Seizing the agentic AI advantage,” 2025. Blog post summarising the McKinsey report. The article reports productivity gains of 20–60 % and approximately 30 % faster credit-turnaround times for agentic AI deployments

work page 2025
[5]

Theory of trust and acceptance of artificial intelligence technology (TrAAIT): An instrument to assess clinician trust and acceptance of artificial intelligence,

A. F. Stevens, P. Stetson, et al., “Theory of trust and acceptance of artificial intelligence technology (TrAAIT): An instrument to assess clinician trust and acceptance of artificial intelligence,” Journal of Biomedical Informatics, vol. 148, 2023. The TrAAIT model mea- sures trust through perceived information credibility, system reliability and applica...

work page 2023
[6]

Large language model evaluation in 2025: 10+ metrics and methods,

C. Dilmegani, “Large language model evaluation in 2025: 10+ metrics and methods,”

work page 2025
[7]

Evaluating agentic AI systems: A balanced framework for perfor- mance, robustness, safety and beyond,

M. A. Shukla, “Evaluating agentic AI systems: A balanced framework for perfor- mance, robustness, safety and beyond,” 2025. Preprint available at Preprints.org. DOI: 10.20944/preprints202508.1847.v1 11

work page arXiv 2025
[2025]

AIMultiple article advocating multidimensional evaluation integrating automated scores with human assessments and tests for bias, fairness and energy consumption. 10

work page

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

AI agents vs. agentic AI: A concep- tual taxonomy, applications and challenges,

R. Sapkota, K. I. Roumeliotis, and M. Karkee, “AI agents vs. agentic AI: A concep- tual taxonomy, applications and challenges,” Information Fusion, 2025. Available on arXiv:2505.10468

arXiv 2025

[2] [2]

The measurement imbalance in agentic AI evaluation undermines industry productivity claims,

K. J. Meimandi, G. Ar´ anguiz-Dias, G. R. Kim, L. Saadeddin, and M. J. Kochenderfer, “The measurement imbalance in agentic AI evaluation undermines industry productivity claims,” 2025. arXiv:2506.02064. This work documents that capability metrics dominate agentic AI evaluations (around 83% of surveyed studies) while human-centred and economic metrics are ...

arXiv 2025

[3] [3]

Technical report: Evaluating goal drift in language model agents,

R. Arike, E. Donoway, H. Bartsch, and M. Hobbhahn, “Technical report: Evaluating goal drift in language model agents,” 2025. arXiv:2505.02709. The authors demonstrate that agents given a goal and then exposed to competing objectives exhibit gradual drift

arXiv 2025

[4] [4]

Seizing the agentic AI advantage,

B. Heger, “Seizing the agentic AI advantage,” 2025. Blog post summarising the McKinsey report. The article reports productivity gains of 20–60 % and approximately 30 % faster credit-turnaround times for agentic AI deployments

work page 2025

[5] [5]

Theory of trust and acceptance of artificial intelligence technology (TrAAIT): An instrument to assess clinician trust and acceptance of artificial intelligence,

A. F. Stevens, P. Stetson, et al., “Theory of trust and acceptance of artificial intelligence technology (TrAAIT): An instrument to assess clinician trust and acceptance of artificial intelligence,” Journal of Biomedical Informatics, vol. 148, 2023. The TrAAIT model mea- sures trust through perceived information credibility, system reliability and applica...

work page 2023

[6] [6]

Large language model evaluation in 2025: 10+ metrics and methods,

C. Dilmegani, “Large language model evaluation in 2025: 10+ metrics and methods,”

work page 2025

[7] [7]

Evaluating agentic AI systems: A balanced framework for perfor- mance, robustness, safety and beyond,

M. A. Shukla, “Evaluating agentic AI systems: A balanced framework for perfor- mance, robustness, safety and beyond,” 2025. Preprint available at Preprints.org. DOI: 10.20944/preprints202508.1847.v1 11

work page arXiv 2025

[8] [2025]

AIMultiple article advocating multidimensional evaluation integrating automated scores with human assessments and tests for bias, fairness and energy consumption. 10

work page