{"id":"d1a2eb8f-dfe7-4142-ac0c-035b71c57e4c","arxiv_id":"2607.18816","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"AgentTrails reconstructs provenance graphs from raw agent logs and aligns multiple executions into a joined graph that exposes shared workflows and divergent branches.","lead":"AgentTrails turns the raw step-by-step logs that AI agents leave behind into structured dependency graphs, showing which tool calls produced which results and how later calls reuse earlier outputs. It compares multiple runs side by side, aligning similar steps so developers can spot the efficient workflow, the detours, and the reusable patterns.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Dependency reconstruction accuracy is unvalidated: no precision/recall is reported, and the LLM refinement step may inject hallucinated edges, yet every downstream claim (joined graph, pattern extraction) inherits this step.","rationale":"The reader's weakest assumption is precisely that dependency reconstruction from raw traces is the load-bearing, least-secure step. I agree: all downstream claims depend on it, and the paper provides no quantitative evidence that this step works. The paper is a well-scoped prototype paper with explicit limitation statements, so a CONDITIONAL verdict is appropriate: it should be accepted only if the staged evaluation delivers the promised accuracy numbers and the LLM refinement is shown not to be the source of discovered structure. My proposed concrete test directly targets this by comparing against instrumented ground truth, which is the only way to validate post-hoc reconstruction. The concern does not rise to REJECT because the paper is explicitly preliminary and the claims are framed as demonstrations, not as fully validated results. Therefore, the verdict should remain CONDITIONAL (represented as UNCHANGED).","tokens_in":6990,"tokens_out":2893,"duration_ms":27916,"concrete_test":"Construct ground truth with instrumentation: run a set of agent tasks (e.g., SciAgentGym or a synthetic tool-calling environment) where each tool call is wrapped to record the actual input objects it receives and the output objects it returns, yielding true producer-consumer edges. Strip the instrumentation from the logs to obtain raw chronological traces. Run AgentTrails on these raw traces and compute precision/recall/F1 of the recovered dependency edges against the instrumented ground truth, separately for exact-key, semantic-candidate, and LLM-refined edge subsets. If F1 for the full reconstruction is below, say, 0.8, or if LLM-refined edges have markedly lower precision than exact edges, the central claim weakens substantially. Also run an ablation with the LLM refinement disabled; a large difference in recovered edges indicates the LLM is driving the structure rather than trace evi","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that AgentTrails 'reveals hidden dependencies' from raw, uninstrumented logs (abstract; §1, S1). The load-bearing premise is that the evidence-based reconstruction in §3 (exact key matching, semantic candidates, then LLM refinement) actually recovers true dataflow. This premise is unsupported: the only evaluation is 'we hand-annotated 10 traces with 234 gold dependency edges; preliminary results are promising' (§3), with no precision, recall, or F1 numbers, no edge-type breakdown, and no error analysis. The paper itself admits 'provenance quality needs gold dependency benchmarks' (§3) and 'provenance quality lacks ground-truth benchmarks' (§5). Because there is no instrumented ground truth, the recovered graphs cannot be distinguished from plausibly coherent but incorrect structures. A specific sub-concern is the constrained LLM refinement step: it can add or remove edges beyond exact evidence, and while patches must preserve temporal order and schema, nothing prevents the LLM from introducing a 'dependency' that looks plausible but is not actual dataflow. The two usage scenarios (§4) are post-hoc narrative reconstructions; they show the system can produce a neat graph, not that the graph is correct. If reconstruction accuracy is low, the joined quotient graph alignment and pattern extraction are built on unreliable topology, and the claimed 'hidden dependencies' become partially artifacts of the reconstruction heuristic. That makes the central claim insecure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AgentTrails, a prototype system that takes raw chronological tool-calling agent logs and reconstructs a bipartite provenance graph in which tool calls are activities and their inputs/outputs are data artifacts. Reconstruction proceeds in three stages: a deterministic skeleton from exact artifact-key matches (identifiers, paths, filenames, URLs), optional semantic dependency candidates, and a constrained LLM refinement step that outputs patch operations. For multi-trace analysis, activities are abstracted into 'activity capsules' and clustered across traces using a weighted similarity measure; each cluster becomes a joined anchor, producing a joined quotient graph that highlights recurring structures and trace-specific deviations. A visual analytics front end and an LLM-based copilot support inspection and comparison. The paper demonstrates the approach on two usage scenarios (a SciAgentGym physics trace and Discovera gene-set traces) and claims that AgentTrails reveals hidden dependencies, aligns divergent executions, and surfaces recurring tool-use patterns.","tokens_in":7233,"tokens_out":3374,"duration_ms":36002,"significance":"If the reconstruction is accurate, AgentTrails addresses a real and timely gap: recovering dependency topology from unstructured agent logs without framework instrumentation. The design has several commendable properties: the deterministic skeleton is grounded in exact trace evidence, the final graph keeps exact, semantic, and LLM-refined edges distinguishable, and the multi-trace alignment via activity capsules is a plausible way to handle stochastic executions. These features make the paper's central claims falsifiable and the approach potentially reproducible. However, the paper provides no quantitative evidence that the recovered provenance graphs are correct. The two usage scenarios are illustrative; they show that the system can produce visually coherent graphs, not that those graphs correspond to true dataflow. Because every downstream claim—joined alignment, pattern extraction, workflow reuse—inherits the accuracy of the reconstruction step, the missing evaluation is load-bearing rather than cosmetic.","major_comments":[{"comment":"The central claim that AgentTrails 'reveals hidden dependencies' rests entirely on the reconstruction step, yet the only evaluation reported is: 'we hand-annotated 10 traces with 234 gold dependency edges; preliminary results are promising.' No precision, recall, F1, edge-type breakdown, or error analysis is given. The paper itself admits 'provenance quality needs gold dependency benchmarks.' This is not a minor omission: if the reconstructed graphs contain false or missed edges, the joined quotient graph and extracted patterns are built on unreliable topology. The authors should report full metrics on the 234-edge gold set, ideally with a per-edge-type breakdown (exact key, semantic candidate, LLM-refined), and compare against baselines such as exact matching alone or direct LLM inference. If possible, they should also test on instrumented frameworks where ground-truth provenance is ava","section":"§3, 'Provenance Graph Construction' and 'Two questions remain'"},{"comment":"The LLM refinement step can add or remove edges beyond exact evidence, constrained only by valid node references, temporal order, and the activity/entity schema. These constraints do not prevent the LLM from introducing plausible-looking but incorrect dependencies. Because the final graph is a co-production of evidence and LLM judgment, the paper's auditability claim ('exact edges, semantic candidates, and LLM-refined edits remain distinguishable') is good, but it is not enough. The evaluation must quantify the accuracy of each edge source separately and include an ablation that removes the LLM refinement to show whether it improves or degrades reconstruction. Without this, the 'hidden dependencies' shown in §4 could be partially artifacts of the LLM rather than recovered dataflow.","section":"§3, 'Provenance Graph Construction' (constrained LLM refinement)"},{"comment":"The alignment quality depends on the capsule similarity weights w_m, which are free parameters. The paper does not state how the weights were chosen, reports no sensitivity analysis, and §5 concedes that 'the capsule similarity function and its weights require principled tuning and evaluation.' Since the joined graph and the claimed alignment of divergent executions are central contributions, the paper should at least report the chosen weights, test robustness to reasonable perturbations, and ideally evaluate clustering quality against a gold alignment of traces for the same task. Without this, it is unclear whether the alignment results in §4 are meaningful or an artifact of parameter choice.","section":"§3, 'Multi-Trace Provenance Graph', Eq. for capsule similarity"},{"comment":"Both usage scenarios are post-hoc narrative reconstructions. They demonstrate that AgentTrails can produce a neat graph and that the graph matches the authors' scientific interpretation of the task, but they do not validate that the recovered dependencies are correct. For the SciAgentGym scenario, the paper mentions comparing with the benchmark's expected tool use, but provides no quantitative comparison. For the Discovera scenario, the graph is shown to align with the expected data-processing pipeline, but again without ground-truth dependency edges. The authors should provide a systematic evaluation on a larger set of traces where the true dependency structure is known (e.g., from an instrumented workflow-aligned agent), reporting edge-level accuracy and, where possible, demonstrating that the joined graph's high-support paths correspond to reusable workflows and its low-support paths","section":"§4, usage scenarios"}],"minor_comments":[{"comment":"The notation K(e_j) and refs(c_i.args) is introduced informally. A precise definition of what counts as a 'key' (e.g., exact string, normalized identifier, filename prefix) and what counts as a 'reference' would make the deterministic skeleton reproducible. Similarly, the 'informedBy' edge type is mentioned but not defined; its semantics should be clarified.","section":"§3, notation"},{"comment":"The activity capsule is described verbally as containing tool identity, input/output structure, local graph context, and upstream root lineage, but the feature set and the similarity functions s_m are not specified. A table or pseudocode listing the features and the Jaccard/cosine variants would improve clarity and enable independent implementation.","section":"§3, activity capsule"},{"comment":"The term 'quotient graph' is used but never formally defined. It would be helpful to state explicitly how the original nodes/edges are mapped to joined anchors and how the aggregated edges preserve trace-specific multiplicity and support.","section":"§3, 'Multi-Trace Provenance Graph'"},{"comment":"The text references panels (A), (B), and (C) of Figure 4 but does not explain all visual encodings (e.g., node size, edge width, color) in the figure caption. Adding a legend or explicit caption text would help readers interpret the claimed support and score summaries.","section":"§4, Figure 4"},{"comment":"The provenance copilot is said to 'help users interpret trace-derived, auditable graphs,' but the copilot itself is an LLM and could introduce interpretive errors. A brief note on how copilot outputs are grounded or bounded would be useful, even if the copilot is not the focus of the paper.","section":"§3, 'Interactive Visualization'"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest about its limitations, and the workshop framing explains the preliminary nature. For a full archival venue, however, the absence of quantitative reconstruction evaluation is a blocking issue. The authors should be encouraged to conduct the staged evaluation they describe and to make the annotated gold traces and system code available for reproducibility. I would not reject the paper, because the core idea and design choices are sound and the limitations are explicitly acknowledged; with the missing evaluation supplied, the paper could become a solid systems contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid, honest prototype paper with one genuinely new idea — recovering dependency graphs from raw, uninstrumented agent logs and then aligning multiple traces through a joined quotient graph. The problem is real: chronological logs hide the dataflow that developers need for debugging and reuse. No one in the cited 24 references does post-hoc reconstruction plus cross-trace alignment; PROV-AGENT instruments the runtime, TRAIL/AgenTracer target failures, and the rest focus on single-trace visualization. The joined quotient graph, built from activity capsules clustered across traces, is the most original piece and reads as a clear design.\n\nWhat it does well: the reconstruction pipeline is not a black box. The deterministic skeleton is grounded in exact key matches — artifact IDs, paths, URLs — and the semantic/LLM steps produce candidates or constrained patches that remain distinguishable. The two usage scenarios are internally consistent and show the system producing sensible structure. The paper also explicitly flags its own open problems: no gold dependency benchmarks, unvalidated capsule similarity weights, no scalability assessment. That is the right level of honesty for a workshop prototype.\n\nThe soft spot is exactly where the stress-test lands: the central claim, 'reveals hidden dependencies,' is supported by ten hand-annotated traces with 234 gold edges and the words 'preliminary results are promising.' No precision, recall, or F1; no error analysis; no breakdown of exact vs. semantic vs. LLM edges. Since every downstream step — the joined graph, pattern extraction, the copilot — inherits this topology, the whole contribution currently rests on an unvalidated reconstruction. The specific worry about LLM refinement injecting plausible-but-false edges is real; the constraints on patches (valid node refs, temporal order, schema) reduce but do not eliminate it. Also, the alignment weights are unpublished, no code or data is shipped, and the Discovera scenario comes from the authors' own group. These are proportionate concerns for a prototype: they do not make the paper wrong, they make it early.\n\nWho this is for: anyone working on agent observability, provenance, or trace comparison. It deserves a serious referee. The right outcome is conditional acceptance with a request for the staged evaluation: P/R/F1 on gold benchmarks, an error and hallucination analysis of the LLM refinement step, and artifact release.","headline":"A genuinely new post-hoc provenance reconstruction plus multi-trace alignment prototype, honestly scoped but unvalidated at its load-bearing step.","tokens_in":7807,"tokens_out":2450,"would_cite":false,"duration_ms":21550,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AgentTrails reconstructs dependency graphs from raw chronological agent logs, making the hidden dataflow of tool-calling agents explicit for debugging, comparison, and reuse.","keywords":["agent provenance","provenance graph","dependency reconstruction","activity capsules","joined quotient graph","multi-trace alignment","LLM agents","trace analysis"],"falsifier":"Hand-annotate a gold set of agent traces with true dependency edges and compute precision and recall of the reconstructed graph. If coincidental token overlaps produce false edges, or if references without shared keys cause missed edges in numbers that materially change the workflow, the claim that hidden dependencies are revealed would be undermined. The paper itself notes that provenance quality needs gold dependency benchmarks and reports only preliminary hand-annotation on 10 traces (234 edges) without numbers.","tokens_in":6733,"feed_emoji":"🕸️","tokens_out":5405,"duration_ms":48775,"temperature":0.7,"pith_summary":"AgentTrails argues that the raw chronological logs produced by tool-calling AI agents obscure the real structure of what the agent did: which outputs fed which later calls, which intermediate artifacts were reused, and where work branched or converged. The paper's central claim is that this dependency topology can be recovered after the fact, without instrumenting the agent framework, by treating evidence in the log—artifact identifiers, paths, URLs, argument values, and semantic references—as clues to producer-consumer relationships. To compare multiple executions of the same task, AgentTrails abstracts each tool call into an 'activity capsule' and clusters similar capsules across traces, producing a joined quotient graph that aligns shared workflow structure while keeping trace-specific branches visible. If this works at scale, developers could debug, compare, and reuse agent behaviors from logs alone, and recurring patterns could be lifted into reusable skills.","feed_headline":"AgentTrails reconstructs the hidden dataflow inside raw agent logs","feed_subtitle":"Turning chronological traces into provenance graphs reveals dependencies, aligns runs, and surfaces reusable workflows.","key_machinery":"The central object is the provenance graph G_T = (A_T ∪ E_T, R_T), a bipartite graph whose nodes are tool-call activities and data artifacts and whose edges are generatedBy, usedBy, and informedBy relations. Reconstruction relies on key matching between recovered entity keys (storage ids, paths, filenames, URLs, names) and references in later tool-call arguments, plus semantic candidate matching and a constrained LLM refinement. The multi-trace mechanism is the activity capsule: each activity is summarized by tool identity, input/output structure, local graph context, and upstream root lineage, and capsules are clustered using a weighted Jaccard/cosine similarity S(κ_i, κ_j) = Σ w_m s_m(κ_i,","core_discovery":"The paper's core discovery is that a bipartite provenance graph can be reconstructed from a raw trace by matching key evidence from outputs to references in later inputs. Tool calls become activities, and the entities they produce or consume—inputs, outputs, intermediate artifacts, returned values—become data artifacts. Edges are assigned deterministically for exact matches (e.g., a filename or URL appears in a later call's arguments) and as auditable candidates for weaker semantic matches, with a constrained LLM refinement step that may patch the graph but only within schema, temporal, and reference constraints. This graph exposes fan-in/fan-out structures and artifact reuse that chronologi","pith_inferences":["A reliable reconstruction method would let teams retroactively add provenance to existing agent systems whose frameworks cannot be instrumented, as long as logs contain enough structural artifacts.","The capsule-clustering alignment could be reused as a preprocessing step for automated workflow mining: clusters with high support across many traces could seed libraries of reusable agent skills, turning raw trajectory corpora into structured training data.","One testable extension is to use support patterns in joined graphs as a weak signal for execution quality: if low-scoring runs systematically exhibit more off-path branches, then provenance-level structure could serve as a debugging heuristic without gold labels.","The approach's dependence on visible artifact identifiers suggests a boundary condition: agents that pass data only through semantic content with no shared keys may require much more aggressive semantic matching, and the system's confidence should be exposed to users."],"forward_implications":["Reconstructed provenance graphs expose hidden dependency structure—fan-in/fan-out patterns and artifact reuse—that is invisible in raw chronological logs.","The joined quotient graph aligns multiple stochastic executions, revealing high-support workflow motifs and low-support trace-specific branches that are candidates for debugging or pruning.","Because reconstruction is trace-agnostic and requires no framework instrumentation, the approach can be applied to heterogeneous agent logs from different frameworks and domains.","The auditable separation of exact edges, semantic candidates, and LLM edits supports trust: users can see which dependencies are grounded in direct evidence and which are inferred.","Patterns extracted from joined graphs can feed skill abstraction and workflow reuse, turning trajectory corpora into structured assets."],"fun_headline_variants":["Turn agent logs into provenance graphs to expose dependencies","AgentTrails rebuilds dataflow from raw agent traces","Compare and reuse agent tasks with AgentTrails' provenance graphs","AgentTrails surfaces reusable patterns in agent logs"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that producer-consumer dependencies can be reliably recovered from raw trace evidence such as artifact IDs, paths, filenames, URLs, and semantic token reuse without instrumenting the agent framework; if reconstruction yields false or missed edges, the hidden dependencies and joined alignments built on top inherit those errors.","fun_headline_variants_meta":{"raw":{"variants":["Turn agent logs into provenance graphs to expose dependencies","AgentTrails rebuilds dataflow from raw agent traces","Compare and reuse agent tasks with AgentTrails' provenance graphs","AgentTrails surfaces reusable patterns in agent logs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000808,"raw_usage":{"total_tokens":3358,"prompt_tokens":697,"completion_tokens":2661,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":441,"completion_tokens_details":{"reasoning_tokens":2597}},"tokens_in":441,"tokens_out":2661,"duration_ms":17895,"temperature":1.0,"reasoning_tokens":2597,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T14:13:20.498703+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hand-annotate a gold set of agent traces with true dependency edges and compute precision and recall of the reconstructed graph. If coincidental token overlaps produce false edges, or if references without shared keys cause missed edges in numbers that materially change the workflow, the claim that hidden dependencies are revealed would be undermined. The paper itself notes that provenance quality needs gold dependency benchmarks and reports only preliminary hand-annotation on 10 traces (234 edges) without numbers.","supporting_citations":[],"review_version":1}