From Failed Trajectories to Reliable LLM Agents: Diagnosing and Repairing Harness Flaws

Haiming Zheng; Junjie Wang; Mengzhuo Chen; Qing Wang; Yawen Wang; Zhe Liu

arxiv: 2606.06324 · v2 · pith:HSJLAE76new · submitted 2026-06-04 · 💻 cs.SE · cs.MA

From Failed Trajectories to Reliable LLM Agents: Diagnosing and Repairing Harness Flaws

Mengzhuo Chen , Junjie Wang , Zhe Liu , Yawen Wang , Haiming Zheng , Qing Wang This is my paper

Pith reviewed 2026-06-28 00:06 UTC · model grok-4.3

classification 💻 cs.SE cs.MA

keywords LLM agentsharness repairfailure diagnosisexecution tracesagent reliabilitySWE-Benchtrace representationbenchmark evaluation

0 comments

The pith

HarnessFix diagnoses LLM agent failures from execution traces and repairs specific harness layers to raise performance 15-50%.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents HarnessFix as a way to move from vague agent fixes to precise harness repairs. It builds a normalized Harness-aware Trace Intermediate Representation from raw traces and code, then links each failure to the exact step and harness layer responsible. This attribution lets the system collect recurring flaw patterns and apply narrow repair operators that are validated before use. The result is measurable lifts on held-out tests across four agent benchmarks while beating both human and automated baselines. A sympathetic reader would care because current self-improvement loops often change the wrong parts of an agent setup and still leave reliability gaps.

Core claim

HarnessFix compiles raw execution traces and harness code into a Harness-aware Trace Intermediate Representation (HTIR), which normalizes fragmented trajectory evidence and captures step-level provenance and control-flow relations. It attributes failures to responsible trajectory steps and harness layers, consolidates recurring diagnoses into actionable flaw records, and maps them to scoped repair operators. Finally, HarnessFix generates and validates harness patches under flaw-specific repair specifications to reduce target flaws without introducing unacceptable regressions, yielding 15.2%--50.0% gains on held-out tests over initial harnesses on SWE-Bench Verified, Terminal-Bench 2.0 Verifi

What carries the argument

Harness-aware Trace Intermediate Representation (HTIR) that normalizes trajectory evidence and records step-level provenance plus control-flow to support precise attribution of failures to harness layers.

If this is right

HarnessFix improves held-out test performance over the initial harnesses by 15.2%--50.0% across the four benchmarks.
It outperforms both human-designed harnesses and self-evolution baselines.
It surfaces recurring harness-flaw patterns across the ETCLOVG layers.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same trace-normalization step could be reused to debug non-LLM agent frameworks that also separate execution environment from policy.
Recurring flaw patterns identified by the method might serve as a checklist for initial harness design rather than post-hoc repair.
If HTIR-style provenance tracking were added to standard agent runtimes, the cost of later diagnosis would drop because the necessary records would already exist.
The approach suggests that agent reliability gains may come more from fixing the surrounding harness than from further prompt or model changes.

Load-bearing premise

Compiling raw execution traces and harness code into HTIR accurately normalizes fragmented trajectory evidence and captures step-level provenance and control-flow relations sufficiently to attribute failures to the correct harness layers without systematic misattribution.

What would settle it

A side-by-side comparison on a held-out set of failed trajectories in which human experts identify different harness layers as responsible than the layers chosen by HarnessFix attribution.

Figures

Figures reproduced from arXiv: 2606.06324 by Haiming Zheng, Junjie Wang, Mengzhuo Chen, Qing Wang, Yawen Wang, Zhe Liu.

**Figure 1.** Figure 1: Overview of HARNESSFIX. code as a shorthand for the implementation artifacts that realize these mechanisms, including both executable source code and model-facing prompt or configuration resources. Accordingly, harness repair means modifying these harness artifacts, rather than updating the base-model parameters, to correct diagnosed reliability flaws. To describe the mechanisms that make up an agent harne… view at source ↗

**Figure 2.** Figure 2: Illustrative example for the AppWorld completion-guard harness flaw. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

read the original abstract

LLM agents increasingly rely on agent harness: the runtime infrastructure around the base model that defines execution environments, tool interfaces, context, lifecycle orchestration, observability, verification, and governance. Existing self-improving agents and automatic harness evolution methods mainly improve agents through runtime supervision, prompt optimization, workflow search, or harness modification based on final outcomes. However, they often fail to diagnose where the responsible evidence lies in failed trajectories and which harness implementation mechanism causes the unreliable behavior, resulting in broad, indirect, or poorly scoped changes. This paper proposes HarnessFix, a trace-grounded and diagnosis-driven framework for repairing agent harnesses. HarnessFix compiles raw execution traces and harness artifacts into a Harness-aware Trace Intermediate Representation (HTIR), which normalizes fragmented trajectory evidence, captures step-level data-flow and control-flow relations, and aligns runtime steps with the harness artifacts that shape their behavior. It then attributes failures to responsible steps and harness artifacts, and consolidates recurring diagnoses into repair-oriented flaw records. Finally, HarnessFix maps these records to scoped repair operators, generates patches under flaw-specific repair specifications, and accepts them through regression-aware validation. We evaluate HarnessFix on four popular benchmarks, and results show that it improves the performance over the initial harnesses by 6.3% to 18.4%, significantly outperforming human-designed and self-evolution baselines. HarnessFix highlights the value of treating failed trajectories not only as feedback signals, but also as structured evidence for diagnosing and repairing the harness mechanisms behind agent failures.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

HarnessFix introduces a trace-to-repair pipeline via HTIR that targets harness layers more specifically than prior self-improvement work, but the attribution step lacks independent checks.

read the letter

The main takeaway is that this paper supplies a structured way to diagnose which harness layer is responsible for an agent failure and then apply scoped fixes instead of broad prompt or workflow changes. The new element is the Harness-aware Trace Intermediate Representation that normalizes traces while preserving step provenance and control flow, followed by flaw consolidation into records and mapping to repair operators.

The paper does a reasonable job laying out the ETCLOVG layers and showing that the same flaw patterns recur across benchmarks. The claimed gains of 15-50% on held-out tests over initial harnesses and over human and self-evolution baselines are the kind of concrete outcome that practitioners would notice if they replicate.

The soft spot is exactly the one the stress-test note flags: there is no reported check on whether HTIR attributions are accurate. Without an ablation on the provenance edges or some measure of attribution agreement, it is possible that the repairs are less layer-specific than described and that part of the improvement comes from non-specific edits. The abstract also gives no error bars, statistical tests, or exclusion criteria, so the performance numbers cannot be assessed for robustness from the given text.

This is for people who build or maintain execution harnesses for LLM agents in software engineering settings. A reader working on agent reliability will get a usable framework and some pattern data even if they have to add their own validation.

It deserves a serious referee because the problem is practical and the proposed structure is distinct from cited prior work, though the review should focus on the attribution validation and the statistical reporting.

Referee Report

2 major / 2 minor

Summary. The paper introduces HarnessFix, a trace-guided framework for diagnosing failures in LLM agent harnesses and applying scoped repairs. It compiles execution traces and harness code into a Harness-aware Trace Intermediate Representation (HTIR) that normalizes trajectories and captures step-level provenance and control-flow relations, attributes failures to responsible steps and ETCLOVG harness layers, consolidates recurring flaw patterns, and maps them to repair operators. Patches are generated and validated under flaw-specific specifications. Evaluation across SWE-Bench Verified, Terminal-Bench 2.0 Verified, GAIA, and AppWorld reports 15.2%–50.0% held-out performance gains over initial harnesses, outperforming human-designed and self-evolution baselines, while identifying recurring flaw patterns.

Significance. If the attribution mechanism and empirical results hold, the work offers a targeted alternative to outcome-based or broad prompt/workflow optimization for agent reliability. The multi-benchmark evaluation and explicit identification of layer-specific flaw patterns across ETCLOVG provide concrete, falsifiable contributions. The introduction of HTIR as an intermediate representation for provenance-preserving trace analysis is a methodological strength that could be reusable beyond this framework.

major comments (2)

[Methods (HTIR and attribution)] Methods section on HTIR construction and attribution: The central claim that HTIR accurately normalizes fragmented traces, preserves control-flow edges, and correctly maps failures to ETCLOVG layers without systematic misattribution lacks an independent validation (e.g., inter-annotator agreement on a sample of attributions or an ablation removing provenance edges). This assumption is load-bearing for the reported 15.2%–50.0% gains, as misattribution would cause scoped repairs to become non-specific edits.
[Evaluation and Results] Results section reporting performance: The abstract and evaluation claim consistent outperformance and specific percentage gains, but the provided text supplies no error bars, statistical significance tests, exclusion criteria for trajectories, or details on how held-out sets were constructed. Without these, the empirical superiority over baselines cannot be verified as robust.

minor comments (2)

[Abstract] The acronym ETCLOVG is used without an explicit expansion or table defining the layers in the abstract; a dedicated table or early section defining each layer would improve readability.
[Methods] Notation for repair operators and flaw records could be formalized with a small example in the methods to clarify the mapping from consolidated diagnoses to patches.

Simulated Author's Rebuttal

2 responses · 0 unresolved

Thank you for the constructive feedback and for recognizing the potential of HTIR as a reusable intermediate representation. We address each major comment below with specific plans for revision where appropriate.

read point-by-point responses

Referee: Methods section on HTIR construction and attribution: The central claim that HTIR accurately normalizes fragmented traces, preserves control-flow edges, and correctly maps failures to ETCLOVG layers without systematic misattribution lacks an independent validation (e.g., inter-annotator agreement on a sample of attributions or an ablation removing provenance edges). This assumption is load-bearing for the reported 15.2%–50.0% gains, as misattribution would cause scoped repairs to become non-specific edits.

Authors: We agree that the original submission did not include an explicit independent validation such as inter-annotator agreement or a provenance-edge ablation. The attribution logic is described in detail in Section 3.2 and is supported by the end-to-end gains and the fact that HarnessFix outperforms both human-designed patches and outcome-only self-evolution baselines; however, these results do not directly isolate attribution accuracy. In the revision we will add (i) a small-scale manual validation on 50 randomly sampled trajectories with two annotators reporting Cohen’s kappa, and (ii) an ablation that removes provenance and control-flow edges from HTIR while keeping all other components fixed. These additions will appear in Section 4.3 and Appendix C. revision: yes
Referee: Results section reporting performance: The abstract and evaluation claim consistent outperformance and specific percentage gains, but the provided text supplies no error bars, statistical significance tests, exclusion criteria for trajectories, or details on how held-out sets were constructed. Without these, the empirical superiority over baselines cannot be verified as robust.

Authors: We accept that the current manuscript lacks the requested statistical and procedural details. The held-out sets were constructed via a fixed random 70/30 split per benchmark with seed 42, and trajectories were excluded only if they contained fewer than three steps or were truncated by the environment; however, these facts are stated only briefly. In the revision we will report (i) mean and standard deviation over five independent runs with different seeds, (ii) paired t-test p-values against each baseline, (iii) explicit exclusion criteria, and (iv) a table detailing the exact train/held-out sizes and construction procedure. These will be placed in Section 5.1 and Table 2. revision: yes

Circularity Check

0 steps flagged

No significant circularity; empirical evaluation on external benchmarks

full rationale

The paper describes an empirical framework (HarnessFix) that compiles traces into HTIR, attributes failures, and applies repairs, with all performance claims (15.2%–50.0% gains) measured directly against held-out test sets on SWE-Bench Verified, Terminal-Bench, GAIA, and AppWorld. No equations, derivations, fitted parameters renamed as predictions, or self-citation chains appear in the provided text. The central claims rest on observable benchmark outcomes rather than any step that reduces to its own inputs by construction, rendering the work self-contained.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 1 invented entities

Review performed on abstract only; no explicit free parameters, axioms, or invented entities beyond the named framework components are stated in the provided text.

invented entities (1)

Harness-aware Trace Intermediate Representation (HTIR) no independent evidence
purpose: Normalizes fragmented trajectory evidence and captures step-level provenance and control-flow relations
Introduced in the abstract as the core representation enabling diagnosis; no independent evidence supplied.

pith-pipeline@v0.9.1-grok · 5795 in / 1225 out tokens · 19665 ms · 2026-06-28T00:06:04.528012+00:00 · methodology

discussion (0)

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Think-Before-Speak: From Internal Evaluation to Public Expression in Multi-Agent Social Simulation
cs.AI 2026-06 unverdicted novelty 6.0

TBS is an interval-based multi-agent framework that separates private internal-state updates (dissonance appraisal, opinion climate, isolation risk, response strategy, willingness to speak) from public utterance selec...
Think-Before-Speak: From Internal Evaluation to Public Expression in Multi-Agent Social Simulation
cs.AI 2026-06 unverdicted novelty 6.0

TBS is an interval-based multi-agent LLM simulation framework that separates structured internal evaluative states from public utterance generation and shows these states vary systematically with turn-allocation, sile...
Cheap Code, Costly Judgment: A Case Study on Governable Agentic Software Engineering
cs.SE 2026-07 unverdicted novelty 5.0

A case study of AI-agentic software development yields a process model explaining how engineering judgment converts recurring structural failures into durable governance mechanisms.