{"id":"8fdc8367-e059-4c87-b582-8d1d628299af","arxiv_id":"2608.06909","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A benchmark of 1,351 annotated LLM-agent trajectories that standardizes attribution to a primary cause and optional attack and execution chains across task-aligned, unsafe, and refusal behaviors.","lead":"This paper builds a benchmark for explaining why an LLM agent took a particular action, using more than 1,300 annotated execution traces. It gives researchers a common way to test attribution methods across successful, unsafe, and refusal behaviors.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark's load-bearing artifact is its ground-truth attribution labels, yet the paper reports no inter-annotator agreement and does not release the labels, leaving the single-primary-cause assumption in §3.2/§4.3 unverified.","rationale":"The reader's weakest assumption correctly identifies the single-primary-component assumption and the reliability of LLM-assisted annotation as the load-bearing point. My stress-test concurs and refines it: the paper's own limitations section (§9) already flags that 'some trajectories may admit alternative plausible attribution explanations,' and Section 4.4 offers no human agreement data. Because the benchmark's value rests on ground-truth labels being trustworthy enough to score attribution methods, the lack of inter-annotator agreement and the absence of the released dataset mean the correctness of every headline number in Tables 5-7 is conditional. I found no formal inconsistency in the problem formulation or evaluation metrics; the weakness is empirical verifiability. A concrete IAA study on the current annotations would settle whether the forced single-primary schema is stable or whether multicausal labels introduce noise. If agreement is high, the concern is resolved and the benchmark claim can be accepted; if agreement is low, the difficulty measurements and the claimed benchmark utility would need substantial revision. Since the authors have the ability to run this check and report it, the appropriate verdict remains CONDITIONAL rather than REJECT or ACCEPT.","tokens_in":13730,"tokens_out":3433,"duration_ms":37757,"concrete_test":"Select a random stratified sample of 150 trajectories (50 per target type) from the 1,351. Have two independent expert annotators, blinded to the paper's labels and to each other, annotate each trajectory using the same protocol but with the forced-single-primary constraint removed, allowing them to mark 0-3 primary components and to select chain components. Compute Cohen's kappa on exact primary-component identity and on chain membership, and record the fraction of trajectories where annotators choose more than one primary or disagree on the primary. If kappa < 0.8 or more than 15% of trajectories yield multiple primary components, the single-primary ground truth is not stable, and Tables 5-7 must be re-derived under a multicausal annotation scheme or accompanied by label-noise bounds.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that the 1,351 trajectories form a valid attribution benchmark with component-level ground truth. For this to hold, two conditions must be satisfied: (i) the mapping from a trajectory plus target behavior to a unique primary component P* and optional chain H* is well-defined; and (ii) the annotation procedure actually instantiates that mapping consistently. The paper provides no evidence for (ii). Section 4.4 describes deterministic structural checks and an LLM-based semantic reviewer with iterative repair, but no agreement statistics are reported, and the contributions section's claim of 'verified through human review' is not quantified anywhere. Section 9 explicitly concedes that 'some trajectories may admit alternative plausible attribution explanations.' Since Sections 3.2 and 4.3 force exactly one primary component, any trajectory that is genuinely multicausal receives an arbitrary label; the 'attribution distance' statistics in Table 4 and the Hit@1/MRR and Recall/MAP results in Tables 5-7 then reflect annotation preference as much as attribution difficulty. Moreover, the repository description refers to 'future releases,' so the annotated dataset—the benchmark's core artifact—is not currently available for independent audit. This is an empirical gap rather than a mathematical error; the attribution formulation itself appears internally consistent.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces trajectory attribution as an evaluation problem for LLM agents and proposes a benchmark plus annotation framework. It standardizes heterogeneous execution traces from AgentDojo and two Agent3Sigma settings into a unified component schema, annotates one primary attribution component per target behavior, and adds attack/execution chains for a subset of unsafe-action trajectories. The claimed instantiation contains 1,351 trajectories across task-aligned actions, unsafe actions, and safety refusals. Two reference baselines (incremental trajectory contribution and leave-one-out perturbation) are evaluated with Hit@1/MRR and Recall@K/MAP, and the paper reports substantial difficulty variation across target types, source benchmarks, attribution distance, and chain structure. A reusable annotation skill is promised for future trajectories. The central artifact is the annotated ground truth, and the paper's validity hinges on the consistency and availability of those labels.","tokens_in":13911,"tokens_out":3299,"duration_ms":34389,"significance":"If the annotation protocol is reliable and the labels are released, this would be a useful contribution: the unified component schema, the explicit distinction between primary attribution and structured chains, and the cross-behavior coverage (task-aligned, unsafe, refusal) address a real gap in agent evaluation, which currently focuses on outcomes rather than process. The two evaluation tasks and the reference baselines provide a concrete starting point for comparing attribution methods. The paper also gives credit for proposing a reusable protocol rather than a fixed dataset. However, the contribution is currently empirical rather than formal: there are no machine-checked proofs or parameter-free derivations, and the benchmark's value depends on the validity of human/LLM-produced labels, which is not yet demonstrated.","major_comments":[{"comment":"The load-bearing claim is that the 1,351 trajectories carry reliable component-level ground-truth attribution labels, but the paper reports no inter-annotator agreement, no error analysis, and no quantitative assessment of annotation reliability. Section 4.4 describes only deterministic structural checks and an LLM-based semantic reviewer with iterative repair, and Section 9 explicitly concedes that 'some trajectories may admit alternative plausible attribution explanations.' Because Sections 3.2 and 4.3 force exactly one primary attribution component per target, genuinely multicausal trajectories would receive arbitrary labels under this protocol. I ask the authors to report agreement statistics (e.g., Cohen's kappa or Krippendorff's alpha) on primary-component selection and chain membership, to provide an error analysis of the validation stage, and to quantify how often the LLM-based reviewer changed or rejected annotations.","section":"§4.4, §9"},{"comment":"The contributions section states that annotations are 'verified through human review,' but Section 4.4 describes no human review step; the semantic validation is performed by an LLM-based reviewer. This is a discrepancy in a claim that directly supports the benchmark's credibility. The authors should either document the human review procedure (number of annotators, sample size, adjudication process, and measured agreement with the LLM-assisted annotations) or remove the 'human review' claim from the contributions.","section":"§1, §4.4"},{"comment":"The paper contains numeric inconsistencies that undermine the reproducibility of the reported statistics. Table 3 reports 1,351 total trajectories, but Table 5's 'All Targets Overall' N is 1,349. For Agent3Sigma Stage unsafe actions, Table 4 lists 187 trajectories while Table 5 lists N=186; for safety refusals, Table 4 lists 309 Stage trajectories while Table 5 lists N=308. In Table 7, Stage execution-chain coverage is 71 in Table 4 but 70 in Table 7, making the summed execution-chain N (214) disagree with Table 4 (215). The authors should reconcile these counts and state explicitly which trajectories are excluded from the evaluation and why.","section":"Tables 4, 5, and 7"},{"comment":"The benchmark's core artifact, the annotated dataset, is not currently available: the abstract and Section 10 refer to 'project resources and future releases,' and the GitHub description mentions future releases. Without the actual labels, independent audit of the ground truth is impossible, and the paper's benchmark claims cannot be verified. I ask the authors to release the full annotated dataset, the standardized trajectories, and the annotation prompts/skill as part of the review or revision process, or to clearly state a concrete release date and embargo policy.","section":"§4.5, repository description"}],"minor_comments":[{"comment":"The notation is inconsistent: the prose defines the primary attribution as the set P = {c*}, but Eq. (4) writes P = c*. Please align the notation, since the evaluation metrics use rank(c*) and the set notation clarifies that P contains exactly one element.","section":"§3.2, Eq. (4)"},{"comment":"The description of the semantic validation criteria is too vague to be reproducible; the authors should provide the actual evaluation criteria or the prompt template used by the LLM reviewer, including how 'root cause' is distinguished from 'necessary context' and 'supporting evidence.'","section":"§4.4"},{"comment":"In the leave-one-out score, the first average log-likelihood term is identical for every candidate component and therefore does not affect the ranking. The authors could simplify the formula or state explicitly that ranking depends only on the second term.","section":"§6.2, Eq. (11)"},{"comment":"The perfect Hit@1 and MRR of 1.000 for local task-aligned cases are described as a calibration setting, but the paper does not report whether these cases are excluded from the aggregate difficulty claims; please clarify how these degenerate cases affect the overall conclusions.","section":"§7.2, Table 6"}],"recommendation":"major_revision","confidential_remarks":"The central formulation is internally consistent, and the paper addresses a timely problem, but the empirical basis is currently insufficient: no inter-annotator agreement, no released labels, and several unreconciled table counts. These issues are fixable within the manuscript's scope, so I recommend major revision rather than rejection. I would also encourage the editor to require the dataset release as a condition of acceptance, since the benchmark's value is wholly dependent on the quality and availability of its ground-truth annotations."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — the short version: this is a decent benchmark-design paper whose central artifact, the 1,351 annotated trajectories, is not yet available and whose label reliability is not established. The protocol itself is well thought out; the gap is empirical, not mathematical.\n\nWhat's actually new: the paper proposes trajectory attribution as a distinct evaluation problem and provides a unified component schema, a behavior-aware annotation process with one primary cause plus optional attack/execution chains, two evaluation tasks, and two likelihood-based baselines. That combination is not present in the prior work they cite. They also reuse execution-derived trajectories from AgentDojo and Agent3Sigma, which is good practice. The examples in Figures 3 and 4 are clear, and the distance-based analysis (local vs long-range) is a sensible way to characterize difficulty. The math is simple and internally consistent; there are no fitted parameters or circular derivations. The baselines do what they claim.\n\nThe soft spots are where the stress-test note lands. The labels — the benchmark's ground truth — come from an LLM-assisted annotation pipeline with only deterministic structural checks and an LLM-based semantic review. No inter-annotator agreement is reported, and the 'human review' in the contributions is not quantified anywhere. The limitations section admits that alternative plausible explanations can exist, yet Section 3.2 forces exactly one primary component per trajectory. That is acceptable only if the annotation procedure is consistent, and we have no evidence of it. Worse, the repository says 'future releases,' so the annotated data is not currently available to audit. That is a deal-breaker for immediate use as a benchmark.\n\nThe circularity concern is minor: yes, both the labels and baselines use LLM likelihoods, but the annotation is based on reasoning over causal roles, not on the same scoring function, and the baselines are only provided as references. I would not call that a fatal flaw.\n\nWho this is for: researchers working on agent interpretability, failure diagnosis, or AI safety evaluation. If they ship the data and report agreement, this could become a useful standard. Right now it's a promising protocol in search of evidence.\n\nI'd send it to peer review — the problem is timely and the framework is clean enough to deserve referee time — but I'd make release of data and reliability statistics a precondition for acceptance. A serious referee should ask for that.\n\nWould I cite it in the next year? Not until the artifacts are out. Worth reading now, citing later.","headline":"A sensible protocol for trajectory attribution; the benchmark's validity currently rests on unreleased, unquantified LLM-produced labels.","tokens_in":14470,"tokens_out":2715,"would_cite":false,"duration_ms":26737,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces trajectory attribution as a benchmarkable evaluation problem for LLM agents, with a unified component schema and over 1,300 annotated trajectories spanning task-aligned actions, unsafe actions, and safety refusals.","keywords":["trajectory attribution","LLM agents","component-level schema","attribution benchmark","attack chains","execution chains","safety refusal","long-horizon agent evaluation"],"falsifier":"Take a random sample of trajectories from the published benchmark, have independent annotators who have never seen the labels re-apply the protocol, and measure agreement on the primary component and on chain membership; if pairwise agreement on primary attribution is not high, for example below roughly 80 percent, the labels and the performance gaps reported in Tables 5 to 7 cannot be treated as a stable ground truth.","tokens_in":13472,"feed_emoji":"🎯","tokens_out":7457,"duration_ms":61218,"temperature":0.7,"pith_summary":"This paper argues that explaining an LLM agent's behavior requires looking at the whole execution trail, not just the final answer, and that this act—trajectory attribution—can be a standardized evaluation problem. It builds a benchmark by converting heterogeneous agent traces into one component-level schema, labeling each target behavior with exactly one primary cause and, where relevant, an ordered attack or execution chain. Across three existing agent environments this yields 1,351 annotated trajectories spanning task-aligned actions, unsafe actions, and safety refusals. The paper also defines two evaluation tasks, primary-attribution localization and attribution-chain recovery, and shows that reference baselines find long-range and chain-structured cases measurably harder. If the framework holds, future agent models can be scored on how well their actions can be explained, alongside how well the actions succeed.","feed_headline":"Attribution benchmark turns agent behavior into a measurable question","feed_subtitle":"More than 1,300 standardized trajectories label the root cause behind task success, unsafe actions, and safety refusals.","key_machinery":"The load-bearing object is the reference attribution structure $A=(P,H^*)$ defined over an ordered component trajectory $\\tau=(c_1,\\dots,c_T)$: for a target component $c_t$, the candidate set is all earlier components, $P=\\{c^*\\}$ is the single annotated root cause, and $H^*=(c_{i_1},\\dots,c_{i_m})$ is the optional ordered chain of contributing components, with attack and execution chains as two specialized chain types for unsafe actions. The construction protocol carries the argument: it standardizes raw traces into this schema, identifies the target behavior as task-aligned, unsafe, or a safety refusal, annotates the primary cause and chains, then validates the annotations structurally and semantically with an LLM reviewer and a repair loop. This protocol is what lets trajectories from different agent systems be compared under one attribution standard.","core_discovery":"The central claim is that trajectory attribution can be made into a reproducible evaluation task: any agent trajectory is an ordered sequence of components, such as system instructions, user messages, reasoning steps, tool calls, observations, and memory, and for a chosen target behavior the job is to rank the earlier components by causal contribution. The reference annotation gives every target one primary attribution component, the annotated root cause, and, for unsafe actions with distributed causation, an ordered attack chain showing how adversarial influence propagated and an execution chain showing how the unsafe act was carried out. The paper instantiates this protocol on trajectories from three existing agent environments, producing more than 1,300 standardized trajectories, and evaluates incremental and leave-one-out baselines under two metrics: Hit@1 and MRR for root-cause localization, and Recall@K and MAP for chain recovery. The substantial performance differences across local and long-range settings are offered as evidence that the benchmark captures real attribution difficulty rather than a single easy level.","pith_inferences":["If real agent behaviors are often multicausal, the one-primary-cause assumption may make the benchmark systematically easier than real-world attribution; a natural extension is to allow weighted or multiple primary causes and measure how baselines behave on them.","The same protocol could be reused as a quality gate during agent development: trajectories whose attributed cause is far from the intended instruction could be flagged for debugging before deployment.","A direct test of the annotation's reliability would be double-annotation of a sample by independent annotators; agreement statistics would show how much of the benchmark's difficulty is inherent versus an artifact of labeling choices.","The chain annotations suggest a path toward counterfactual intervention: replacing or removing the primary component should predictably change the target behavior, which would convert static labels into testable causal claims."],"forward_implications":["Attribution becomes a continuous evaluation axis for LLM agents, so a model's decisions can be scored for explainability rather than only for success or safety.","The benchmark's local-versus-long-range gap quantifies a concrete failure mode: methods that rely on immediately preceding context will miss root causes separated by many steps.","Because chain recovery is consistently harder than root-cause localization, unsafe behaviors that arise through multi-step influence are flagged as a distinct challenge for attribution methods.","The reusable annotation protocol means trajectories from future agent models can be standardized, annotated, and evaluated under the same schema, making benchmarks comparable as agent systems evolve.","The reported baseline scores provide a reference point for comparing future trajectory attribution methods on the same trajectories."],"supporting_citations":[{"why":"Supplies the AgentDojo execution trajectories that form one of the three benchmark environments.","marker":"Debenedetti et al. (2024)"},{"why":"Supplies the Agent3Sigma Stage trajectories used in the benchmark instantiation.","marker":"Ma et al. (2026)"},{"why":"Supplies the Agent3Sigma Canary trajectories used in the benchmark instantiation.","marker":"Li et al. (2026a)"},{"why":"Defines the agentic attribution problem and provides the incremental scoring formula used as a baseline.","marker":"Qian et al. (2026)"},{"why":"Provides the leave-one-out component-attribution strategy used as the second baseline.","marker":"Kim et al. (2026)"},{"why":"Prior failure-attribution benchmark whose single-agent scope motivates the unified cross-behavior design.","marker":"Zhang et al. (2025)"},{"why":"Prior multi-agent failure-attribution benchmark contrasted as lacking structured chains and cross-behavior coverage.","marker":"Chen et al. (2026)"}],"fun_headline_variants":["Why did the agent do that? New benchmark pinpoints root causes","Attribution benchmark labels 1300+ trajectories with root cause","Unified framework for tracing LLM agent behavior to its cause","Benchmark traces agent decisions to causes across 1300 cases"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes every target behavior has exactly one primary cause and that LLM-assisted annotation, after structural and semantic validation, reliably finds that cause and the optional chains; if behaviors are often jointly caused or the labels are inconsistent, the ground-truth annotations and all difficulty measurements lose their foundation.","fun_headline_variants_meta":{"raw":{"variants":["Why did the agent do that? New benchmark pinpoints root causes","Attribution benchmark labels 1300+ trajectories with root cause","Unified framework for tracing LLM agent behavior to its cause","Benchmark traces agent decisions to causes across 1300 cases"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000535,"raw_usage":{"total_tokens":2590,"prompt_tokens":982,"completion_tokens":1608,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":1545}},"tokens_in":598,"tokens_out":1608,"duration_ms":11337,"temperature":1.0,"reasoning_tokens":1545,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:33:20.886126+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of trajectories from the published benchmark, have independent annotators who have never seen the labels re-apply the protocol, and measure agreement on the primary component and on chain membership; if pairwise agreement on primary attribution is not high, for example below roughly 80 percent, the labels and the performance gaps reported in Tables 5 to 7 cannot be treated as a stable ground truth.","supporting_citations":[{"cited_title":"2026 , url =","cited_arxiv_id":null,"evidence_quote":"Defines the agentic attribution problem and provides the incremental scoring formula used as a baseline."},{"cited_title":"and Pfister, Tomas , journal =","cited_arxiv_id":null,"evidence_quote":"Provides the leave-one-out component-attribution strategy used as the second baseline."},{"cited_title":"Which Agent Causes Task Failures and When? On Automated Failure Attribution of","cited_arxiv_id":null,"evidence_quote":"Prior failure-attribution benchmark whose single-agent scope motivates the unified cross-behavior design."},{"cited_title":"Seeing the Whole Elephant: A Benchmark for Failure Attribution in","cited_arxiv_id":null,"evidence_quote":"Prior multi-agent failure-attribution benchmark contrasted as lacking structured chains and cross-behavior coverage."}],"review_version":1}