EARS: Explanatory Abstention for Reliable Sub-Agent Modeling in Large-scale Multi-Agent Systems

Han Li; Lingyun Wang; Shuang Xie; Yunan Lu

arxiv: 2606.18668 · v1 · pith:SR57JIDKnew · submitted 2026-06-17 · 💻 cs.MA · cs.CL

EARS: Explanatory Abstention for Reliable Sub-Agent Modeling in Large-scale Multi-Agent Systems

Shuang Xie , Yunan Lu , Han Li , Lingyun Wang This is my paper

Pith reviewed 2026-06-26 18:47 UTC · model grok-4.3

classification 💻 cs.MA cs.CL

keywords multi-agent systemsabstentionreliabilityLLM-as-a-Judgefine-tuningfailure taxonomye-commerce workflows

0 comments

The pith

EARS trains sub-agents to return rationales for failures, increasing response pass rate from 68.5% to 78.9% in multi-agent systems.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces EARS to address reliability issues in large-scale multi-agent systems where sub-agents often over-answer requests they cannot handle. It reframes abstention as an inter-agent protocol where sub-agents provide structured explanations of failure states based on a taxonomy of modes like ambiguity or unsupported requests. Data for training is created by an ensemble of LLM judges that label interactions with abstention rationales. Fine-tuning sub-agents on this data allows them to detect these conditions and communicate them to the coordinator for better handling. Evaluation in a production e-commerce assistant shows the pass rate improvement.

Core claim

By curating human-agent interaction data with an ensemble of calibrated LLM-as-a-Judge models to produce structured abstention labels and rationales under a taxonomy of sub-agent failure modes, and using this data to fine-tune sub-agents, EARS enables sub-agents to expose actionable failure states, thereby improving the overall response pass rate from 68.5% to 78.9% in a large-scale production e-commerce assistant.

What carries the argument

The EARS framework that uses an ensemble of LLM-as-a-Judge models to generate labeled data on failure modes for fine-tuning sub-agents to produce explanatory abstentions.

If this is right

Sub-agents avoid hallucinated outputs by abstaining with reasons on ambiguous or unsupported requests.
The coordinator receives actionable information for clarification, rerouting, or fallback decisions.
Overall MAS reliability increases while preserving modularity and cost efficiency.
Failure mode detection becomes a learned capability of the sub-agents rather than relying on post-hoc checks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Explanatory abstention could be integrated into other types of agent systems to handle capability limits more transparently.
The taxonomy of failure modes might be reusable across different enterprise domains for standardizing agent calibration.
Further gains might come from iterating the LLM judge ensemble with feedback from the fine-tuned sub-agents.

Load-bearing premise

The ensemble of LLM judges produces abstention labels and rationales that match what the sub-agents actually need to learn for the target domain.

What would settle it

If replacing the LLM-generated labels with random labels or human labels that disagree leads to no improvement or a decrease in the pass rate when fine-tuning.

Figures

Figures reproduced from arXiv: 2606.18668 by Han Li, Lingyun Wang, Shuang Xie, Yunan Lu.

**Figure 1.** Figure 1: Overview of EARS. The framework starts by sampling a subset of human-agent interaction data for human annotation, then uses the annotated seed set to calibrate LLM-as-a-Judge models under an abstention taxonomy. The calibrated judges then curate large-scale interaction data for sub-agent fine-tuning. The updated sub-agent is finally redeployed into the multi-agent system, forming an online data flywheel fo… view at source ↗

**Figure 2.** Figure 2: Ablation results on the effect of abstention data under different curation strategies. Adding human [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Figure 3: Abstention judge confusion matrices for final-model abstention-category predictions. The y-axis gives [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗

read the original abstract

In large-scale enterprise settings, centralized multi-agent systems (MAS) are increasingly adopted, in which a coordinator delegates user requests to lightweight, domain-specialized sub-agents. While this architecture improves modularity, scalability, and cost efficiency, its reliability depends not only on accurate routing but also on sub-agents' ability to calibrate their responses to capability constraints. In particular, sub-agents built on smaller fine-tuned models often struggle with such calibration, leading them to over-answer ambiguous, underspecified, misrouted, or unsupported requests and produce hallucinated outputs instead of actionable feedback. To address this challenge, we present EARS (Explanatory Abstention for Reliable Sub-Agent Modeling), a production-oriented framework that reframes sub-agent abstention as an inter-agent communication protocol: a sub-agent does not merely abstain, but exposes an actionable failure state to the coordinator. EARS curates human-agent interaction data using an ensemble of calibrated LLM-as-a-Judge models, producing structured abstention labels and rationales under a taxonomy of sub-agent failure modes. These data are used to fine-tune sub-agents to detect failure conditions and return rationales for coordinator-level clarification, rerouting, or fallback. We evaluate EARS in a large-scale production e-commerce assistant supporting enterprise business intelligence workflows. EARS improves the overall response pass rate from 68.5% to 78.9%, demonstrating that sub-agent-side explanatory abstention improves MAS reliability.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

EARS offers a practical protocol for sub-agent abstention in enterprise MAS but the 10-point pass-rate gain rests on unvalidated LLM-judge labels with no shown agreement stats or controls.

read the letter

The main point is that EARS reframes abstention as an explicit message from sub-agent to coordinator that includes a failure rationale, then uses an LLM-judge ensemble to label training data under a failure-mode taxonomy before fine-tuning. In their e-commerce production run this is said to move the pass rate from 68.5% to 78.9%.

The framework itself is a straightforward engineering extension: it takes existing abstention and LLM-as-judge ideas and packages them for a centralized MAS where the coordinator can act on the returned state. The taxonomy and curation steps are concrete enough that someone running similar systems could copy the structure.

The evidence is the soft spot. The abstract states the numerical lift but supplies none of the usual details on measurement, baselines, statistical tests, or error breakdown. More importantly, the labels come from the same class of models being fine-tuned, yet the text gives no human agreement numbers, inter-judge reliability, or out-of-distribution checks on real production traffic. If the judges are mostly echoing their own priors, the reported gain could be an artifact rather than a genuine change in sub-agent behavior. The stress-test note captures this exactly.

The work is aimed at engineers who already run large-scale MAS in production and need a way to reduce over-answering from smaller fine-tuned agents. A reader in that setting might pick up the protocol and taxonomy even if the numbers need more support.

It is worth sending for peer review because the applied setting is real and the core idea is simple to test; the authors would just have to supply the missing validation steps.

Referee Report

2 major / 0 minor

Summary. The manuscript presents EARS, a framework that reframes sub-agent abstention in centralized multi-agent systems as an inter-agent communication protocol. It uses an ensemble of calibrated LLM-as-a-Judge models to curate human-agent interaction data with structured abstention labels and rationales based on a taxonomy of sub-agent failure modes. These data are used to fine-tune sub-agents to detect failures and return rationales for coordinator action. The paper reports an evaluation in a large-scale production e-commerce assistant where EARS improves the overall response pass rate from 68.5% to 78.9%.

Significance. If the reported improvement is robustly validated with appropriate controls and judge validation, EARS could provide a practical method for enhancing reliability in large-scale enterprise multi-agent systems by enabling sub-agents to communicate actionable failure states rather than producing hallucinations. The production deployment setting and focus on fine-tuning smaller models for calibration address real-world scalability and cost concerns in MAS.

major comments (2)

The abstract states a numerical improvement in response pass rate from 68.5% to 78.9% but supplies no details on experimental controls, baseline systems, statistical tests, data splits, or how the 78.9% figure was measured. This absence is load-bearing for the central empirical claim and prevents verification of whether the gains reflect genuine abstention improvements.
The method depends on the ensemble of calibrated LLM-as-a-Judge models producing reliable structured abstention labels and rationales that generalize to the target sub-agents under the failure-mode taxonomy. No human-agreement statistics, inter-judge kappa, or out-of-distribution validation on production interactions are provided, which is critical to the claim that fine-tuning yields the reported reliability lift.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive review and for highlighting the need for greater transparency around the central empirical claims and the labeling process. We address each major comment below and commit to revisions that strengthen verifiability without altering the reported results.

read point-by-point responses

Referee: The abstract states a numerical improvement in response pass rate from 68.5% to 78.9% but supplies no details on experimental controls, baseline systems, statistical tests, data splits, or how the 78.9% figure was measured. This absence is load-bearing for the central empirical claim and prevents verification of whether the gains reflect genuine abstention improvements.

Authors: We agree that the abstract is too concise on this point. The full manuscript's Evaluation section describes the baseline (sub-agents without EARS fine-tuning), the production e-commerce dataset, the pass-rate metric (fraction of requests receiving correct non-hallucinated responses), and the data collection protocol. To make the claim more verifiable at a glance, we will revise the abstract to include a one-sentence summary of the evaluation setting and controls. We will also ensure that any statistical tests (e.g., significance of the lift) are explicitly stated in the main text. revision: yes
Referee: The method depends on the ensemble of calibrated LLM-as-a-Judge models producing reliable structured abstention labels and rationales that generalize to the target sub-agents under the failure-mode taxonomy. No human-agreement statistics, inter-judge kappa, or out-of-distribution validation on production interactions are provided, which is critical to the claim that fine-tuning yields the reported reliability lift.

Authors: We acknowledge that quantitative validation of the judge ensemble is not reported. Section 3 details the calibration procedure and taxonomy, but agreement metrics are absent. We will add a new subsection reporting inter-judge agreement (Cohen's or Fleiss' kappa) on a human-validated sample of labels together with out-of-distribution results on held-out production interactions. These additions will directly support the reliability of the training labels. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical production evaluation independent of labeling process

full rationale

The paper presents a framework for curating labeled data via LLM judges and then fine-tuning sub-agents, with the central claim resting on a measured production pass-rate lift (68.5% to 78.9%). No equations, self-citations, or derivations are present that reduce the reported improvement to the input labels by construction. The evaluation metric is described as an external production outcome rather than a judge-derived quantity, leaving the result self-contained against the described inputs.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review yields no identifiable free parameters, axioms, or invented entities. The central claim rests on the unexamined reliability of the LLM-as-Judge ensemble and the transferability of the resulting labels.

pith-pipeline@v0.9.1-grok · 5801 in / 1311 out tokens · 18905 ms · 2026-06-26T18:47:17.922430+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

20 extracted references · 4 canonical work pages · 3 internal anchors

[1]

Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks

Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks.arXiv preprint. ArXiv:2411.04468 [cs.AI]. Google. 2025. Gemini 2.5 flash. Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryu- taro Tanno, Khaled Saab, Dan Popovici, Jacob Blum, Fan Zhang, Kather...

work page internal anchor Pith review Pith/arXiv arXiv 2025
[2]

arXiv preprint (2024), https://arxiv.org/abs/2407.18418, arXiv:2407.18418

Know Your Limits: A Survey of Absten- tion in Large Language Models.arXiv preprint. ArXiv:2407.18418 [cs.CL]. Feijie Wu, Zitao Li, Fei Wei, Yaliang Li, Bolin Ding, and Jing Gao. 2026. Talk to Right Specialists: It- erative Routing in Multi-agent Systems for Ques- tion Answering.arXiv preprint. ArXiv:2501.07813 [cs.MA]. Qingyun Wu, Gagan Bansal, Jieyu Zhan...

work page arXiv 2026
[3]

AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation

AutoGen: Enabling Next-Gen LLM Applica- tions via Multi-Agent Conversations.arXiv preprint arXiv:2308.08155. Yanwei Yue, Guibin Zhang, Boyang Liu, Guancheng Wan, Kun Wang, Dawei Cheng, and Yiyan Qi. 2025. MasRouter: Learning to Route LLMs for Multi- Agent Systems.arXiv preprint. ArXiv:2502.11133 [cs.LG]. Skylar Zhai, Jingcheng Liang, and Dongyeop Kang

work page internal anchor Pith review Pith/arXiv arXiv 2025
[4]

Abstain-R1: Calibrated Abstention and Post-Refusal Clarification via Verifiable RL

Abstain-R1: Calibrated Abstention and Post-Refusal Clarification via Verifiable RL.arXiv preprint. ArXiv:2604.17073 [cs.CL] version: 1. Wentao Zhang, Liang Zeng, Yuzhen Xiao, Yongcong Li, Ce Cui, Yilei Zhao, Rui Hu, Yang Liu, Yahui Zhou, and Bo An. 2026. AgentOrchestra: Or- chestrating Multi-Agent Intelligence with the Tool- Environment-Agent(TEA) Protoco...

work page internal anchor Pith review Pith/arXiv arXiv 2026
[5]

Validate that all metrics, dimensions, and models are allowed by [ANALYTICS_SCHEMA_RESTRICTIONS]
[6]

Check that the query matches the user's intent, including filters, aggregation level, requested entity type, and requested time range
[7]

If the user asks for a visualization, require a supported visualization directive from [SUPPORTED_VISUALIZATION_TYPES]
[8]

For trend requests, require an ordering or grouping that makes the trend interpretable
[9]

For top, highest, maximum, or best requests, require descending ordering by an appropriate metric
[10]

For bottom, lowest, minimum, or least requests, require ascending ordering by an appropriate metric
[11]

If no date is specified, treat all-time data as the default
[12]

Interpret relative date ranges using the request timestamp in context
[13]

Apply domain-specific semantic rules for reversal metrics, traffic-source dimensions, entity-name matching, and exclusion filters, with all platform-specific field names masked
[14]

Reject outputs that attempt unsupported reports listed in [UNSUPPORTED_REQUEST_TYPES]. A.2 Segmentation Judge Prompt Input: user_message: the original user request context: request metadata, including request time when available model_output: a generated [SEGMENTATION_FILTER_LANGUAGE] expression or structured abstention signal Task: Evaluate whether model...
[15]

Confirm intent alignment before judging syntax-level details
[16]

Treat segmentation as audience-list construction, not aggregate analytics
[17]

Reject requests that require aggregate reports, ranking, visualization, mutation, or unsupported capabilities in [UNSUPPORTED_REQUEST_TYPES]
[18]

Use request time from context for relative dates
[19]

Prefer lifecycle/value-group filters for vague high-value or loyalty requests when no numeric threshold is provided
[20]

9 B Human Evaluation Details This appendix reports the response-quality rubric used for the full MAS human evaluation in Section 3

Do not accept invented tags, unsupported attributes, or proxy filters that create unsupported behavior. 9 B Human Evaluation Details This appendix reports the response-quality rubric used for the full MAS human evaluation in Section 3. Domain annotators assessed session-level assistant responses according to the rubric in Table 5. A rating of 1 is counted...

[1] [1]

Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks

Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks.arXiv preprint. ArXiv:2411.04468 [cs.AI]. Google. 2025. Gemini 2.5 flash. Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryu- taro Tanno, Khaled Saab, Dan Popovici, Jacob Blum, Fan Zhang, Kather...

work page internal anchor Pith review Pith/arXiv arXiv 2025

[2] [2]

arXiv preprint (2024), https://arxiv.org/abs/2407.18418, arXiv:2407.18418

Know Your Limits: A Survey of Absten- tion in Large Language Models.arXiv preprint. ArXiv:2407.18418 [cs.CL]. Feijie Wu, Zitao Li, Fei Wei, Yaliang Li, Bolin Ding, and Jing Gao. 2026. Talk to Right Specialists: It- erative Routing in Multi-agent Systems for Ques- tion Answering.arXiv preprint. ArXiv:2501.07813 [cs.MA]. Qingyun Wu, Gagan Bansal, Jieyu Zhan...

work page arXiv 2026

[3] [3]

AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation

AutoGen: Enabling Next-Gen LLM Applica- tions via Multi-Agent Conversations.arXiv preprint arXiv:2308.08155. Yanwei Yue, Guibin Zhang, Boyang Liu, Guancheng Wan, Kun Wang, Dawei Cheng, and Yiyan Qi. 2025. MasRouter: Learning to Route LLMs for Multi- Agent Systems.arXiv preprint. ArXiv:2502.11133 [cs.LG]. Skylar Zhai, Jingcheng Liang, and Dongyeop Kang

work page internal anchor Pith review Pith/arXiv arXiv 2025

[4] [4]

Abstain-R1: Calibrated Abstention and Post-Refusal Clarification via Verifiable RL

Abstain-R1: Calibrated Abstention and Post-Refusal Clarification via Verifiable RL.arXiv preprint. ArXiv:2604.17073 [cs.CL] version: 1. Wentao Zhang, Liang Zeng, Yuzhen Xiao, Yongcong Li, Ce Cui, Yilei Zhao, Rui Hu, Yang Liu, Yahui Zhou, and Bo An. 2026. AgentOrchestra: Or- chestrating Multi-Agent Intelligence with the Tool- Environment-Agent(TEA) Protoco...

work page internal anchor Pith review Pith/arXiv arXiv 2026

[5] [5]

Validate that all metrics, dimensions, and models are allowed by [ANALYTICS_SCHEMA_RESTRICTIONS]

[6] [6]

Check that the query matches the user's intent, including filters, aggregation level, requested entity type, and requested time range

[7] [7]

If the user asks for a visualization, require a supported visualization directive from [SUPPORTED_VISUALIZATION_TYPES]

[8] [8]

For trend requests, require an ordering or grouping that makes the trend interpretable

[9] [9]

For top, highest, maximum, or best requests, require descending ordering by an appropriate metric

[10] [10]

For bottom, lowest, minimum, or least requests, require ascending ordering by an appropriate metric

[11] [11]

If no date is specified, treat all-time data as the default

[12] [12]

Interpret relative date ranges using the request timestamp in context

[13] [13]

Apply domain-specific semantic rules for reversal metrics, traffic-source dimensions, entity-name matching, and exclusion filters, with all platform-specific field names masked

[14] [14]

Reject outputs that attempt unsupported reports listed in [UNSUPPORTED_REQUEST_TYPES]. A.2 Segmentation Judge Prompt Input: user_message: the original user request context: request metadata, including request time when available model_output: a generated [SEGMENTATION_FILTER_LANGUAGE] expression or structured abstention signal Task: Evaluate whether model...

[15] [15]

Confirm intent alignment before judging syntax-level details

[16] [16]

Treat segmentation as audience-list construction, not aggregate analytics

[17] [17]

Reject requests that require aggregate reports, ranking, visualization, mutation, or unsupported capabilities in [UNSUPPORTED_REQUEST_TYPES]

[18] [18]

Use request time from context for relative dates

[19] [19]

Prefer lifecycle/value-group filters for vague high-value or loyalty requests when no numeric threshold is provided

[20] [20]

9 B Human Evaluation Details This appendix reports the response-quality rubric used for the full MAS human evaluation in Section 3

Do not accept invented tags, unsupported attributes, or proxy filters that create unsupported behavior. 9 B Human Evaluation Details This appendix reports the response-quality rubric used for the full MAS human evaluation in Section 3. Domain annotators assessed session-level assistant responses according to the rubric in Table 5. A rating of 1 is counted...