{"id":"a523c99c-71ef-4704-8736-7c43d86ebc25","arxiv_id":"2607.06873","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":6,"one_line_summary":"Mining a directly-follows workflow graph from black-box conversations with an LLM agent enables structurally targeted boundary testing that covers 23–38 distinct stateful boundaries per agent, roughly doubling prompt-only baselines.","lead":"This paper presents AgentEval, a black-box testing framework that mines a workflow graph from conversations with an LLM agent and uses its structure to generate boundary tests. It matters because deployed conversational agents can cause real-world harm when stateful guards like confirmation gates fail, and this provides a repeatable method to find those failures without source code access.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"The headline metric—distinct-boundary count—depends on unvalidated LLM-based boundary clustering, and structural enumeration may systematically bias the auditor toward over-splitting tests from different graph locations.","rationale":"The reader correctly identified that unvalidated LLM steps underlying graph construction and metric computation are the soft spot. However, the reader focused on event abstraction (Section IV-A) as the load-bearing concern, whereas I find the boundary clustering step (Section VI-B) more directly threatening to the headline number. Abstraction noise affects graph quality, which indirectly affects target count; clustering noise directly determines the distinct-boundary count that IS the headline metric. Both are unvalidated and both could systematically favor Ours, but clustering is the more immediate threat. That said, the ablation design in RQ3 is genuinely well-constructed: Graph-context and Ours share the same mined graph, discovery, and budget, differing only in whether the graph is used as prompt text or for structural target enumeration. This means abstraction noise (which affects graph quality) is held constant across Graph-context and Ours, so it cannot explain the Ours > Graph-context gap. The concern about clustering bias remains, but it would need to be quite large to erase a 23-vs-9 gap (Ours vs Graph-context, same graph). The reader's CONDITIONAL verdict is appropriate: the directional finding is well-supported by the ablation, but the absolute numbers and their magnitude need independent validation of the clustering step. The paper's own threats-to-validity section is honest about these limitations, which is a credit to the work. Code and prompts are promised upon acceptance but not shipped, which limits independent replication of the clustering procedure specifically.","tokens_in":19480,"tokens_out":4084,"duration_ms":179366,"concrete_test":"Have a domain expert independently cluster all valid boundary tests from both Ours and Prompt-only on the airline domain, blind to (a) which configuration generated each test and (b) the graph-location label each test carries. Recompute distinct-boundary counts from the human clustering. If the Ours/Prompt-only ratio narrows substantially—for instance, Ours drops below ~18 or Prompt-only rises above ~15—the headline magnitude claim weakens, though the directional finding (structural targeting > prompt-only) likely survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is '23 distinct boundaries vs 12.' This count comes from LLM-proposed same-boundary clusters (Section VI-B), and the paper itself acknowledges: 'stricter independent pairwise judgments are left to future work' (Section VI-E). The concern is specific and systematic: Ours generates tests by enumerating structural locations (nodes, edges, starts, ends—Table I), so each test carries a distinct graph-location label. When the LLM auditor clusters tests by boundary similarity, it may be biased toward treating tests from different structural locations as distinct boundaries even when they test semantically equivalent guards. Prompt-only tests, lacking structural labels, would not trigger this bias. This would inflate Ours's distinct count relative to Prompt-only. The reader's concern about abstraction consistency compounds this: if the LLM fragments semantically identical activities into separate nodes (Section IV-A, no quantitative validation), Ours gets more targets and potentially more 'distinct' boundaries, while Prompt-only is unaffected since it never uses the graph structure. Neither the clustering step nor the abstraction step has quantitative validation. The ablation design (RQ3) is strong—it isolates structural targeting as the cause since Graph-context and Ours share the same graph—but the magnitude of the effect (23 vs 12) could be partly an artifact of these two unvalidated LLM steps systematically favoring structural enumeration.","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"This paper presents AGENTEVAL, a black-box testing framework for conversational LLM agents. The framework operates in two phases: Discovery and Execution. In the Discovery phase, AGENTEVAL interacts with the agent via its chat interface to collect session traces. It abstracts these traces into agent activities and user actions, then constructs a 'conversational workflow graph' using the directly-follows graph technique from process mining. This graph is used to enumerate structural locations (nodes, edges, starts, ends) as boundary targets. For each target, an LLM generates a boundary test that replays the conversational path to that location before applying a perturbation (e.g., skipping a prerequisite). In the Execution phase, an LLM-driven runner executes each test against a fresh session, and a separate LLM judge evaluates the outcome based solely on the visible conversation. The authors evaluate AGENTEVAL on four τ³-bench domains using a privileged, white-box LLM auditor (with access to the source code) as the ground-truth oracle. The results show that graph-guided boundary generation yields 23–38 distinct boundaries per agent, outperforming a prompt-only baseline (23 vs. 12 distinct boundaries on the airline domain) at lower duplicate and false-alarm rates.","tokens_in":19695,"tokens_out":1468,"duration_ms":364444,"significance":"The paper addresses a practically important problem: testing stateful workflow failures in black-box conversational LLM agents, where critical boundaries (e.g., confirmation gates) are hidden behind multi-turn prerequisites. The adaptation of process mining (directly-follows graphs) to abstract unstructured conversation turns into a behavioral model for test generation is a novel and well-motivated contribution. The benchmark design, which uses a privileged white-box auditor to score a black-box tester, is a significant methodological strength. The central ablation (Table V) is well-designed: by comparing 'Graph-context' (graph as prompt text) against 'Ours' (structural target enumeration), both using the identical mined graph, the paper cleanly isolates structural target selection as the cause of the boundary-testing gain. The framework produces reusable regression test plans, adding practical value for CI/CD pipelines.","major_comments":[{"comment":"The headline metric—distinct-boundary count—is derived from LLM-based clustering, which is unvalidated and potentially biased by the independent variable. Section VI-B states the auditor extracts a 'boundary card' and clusters cards describing the same boundary. However, because AGENTEVAL generates tests by enumerating structural locations (Table I), each test carries a distinct graph-location label. When the LLM auditor clusters tests by boundary similarity, it may be biased toward treating tests from different structural locations as distinct boundaries even when they test semantically equivalent guards. This would systematically inflate the 'Ours' distinct count relative to the 'Prompt-only' baseline (which lacks structural labels). The paper acknowledges 'stricter independent pairwise judgments are left to future work' (Section VI-E), but this threat is load-bearing for the central R","section":null},{"comment":"The LLM-driven event abstraction step (Section IV-A) lacks quantitative validation, yet the entire graph construction depends on it. The paper states 'The model produces labels that fit the trace and assigns the same label to all surface variants' but provides no measure of abstraction consistency (e.g., inter-run agreement, stability under rephrasing). If the LLM fragments semantically identical activities into separate nodes, the directly-follows graph will contain spurious nodes and missing edges. This directly affects the structural target enumeration that drives the boundary-testing gain: more nodes yield more targets, potentially inflating the distinct-boundary count for 'Ours' without a corresponding increase in actual boundary coverage. A quantitative check of abstraction stability is needed to ensure the mined graph is a reliable foundation for test generation.","section":null},{"comment":"The evaluation relies on a single run per configuration (Section VI-E, Internal threats). The paper notes that 'small differences... should not be overread,' but the boundary-count gap (23 vs. 12) is the central claim. Given the inherent randomness of LLM-driven generation and judging, a single run provides no confidence intervals or variance estimates. It is possible that the 23 vs. 12 gap is partly an artifact of a single favorable run. To solidify the claim, the ablation should be repeated across multiple seeds, reporting mean and standard deviation for the distinct-boundary count and duplicate rate.","section":null}],"minor_comments":[{"comment":"Section VI-D, RQ2: The lower boundary validity on airline and retail (0.78 and 0.79) is attributed to the generator pairing the authenticated caller with an order belonging to a different customer. This is a significant limitation that effectively wastes ~20% of the boundary test budget. The paper should discuss whether this is a fundamental limitation of the LLM generator or a prompt engineering issue that could be mitigated.","section":null},{"comment":"Table II: The auditor model selection is based on agreement with a single expert human reference on 26 airline scenarios. It would strengthen the benchmark to validate the auditor on a larger, more diverse set of scenarios across all four domains, not just airline.","section":null},{"comment":"Section VI-D, RQ4: The judge reliability study on value/accuracy faults (Table VII) shows that a doubled stored reservation price is caught in only 1 of 10 relevant tests. The paper explains this is because the agent recomputes the correct total. This suggests the judge is comparing the displayed value rather than the executed outcome, which may limit its ability to detect certain classes of faults. This limitation should be discussed more prominently.","section":null},{"comment":"Figure 1: The graph excerpt is helpful, but the full graph (46 activities, 54 transitions) is not shown. Providing access to the full graph, perhaps in a supplementary artifact, would aid reproducibility and allow readers to assess the graph's quality.","section":null},{"comment":"Section III-A: The warm-up schedule parameters (m_cap=1, m_hp=2, m_cc=1) are fixed. The paper should briefly discuss whether the system is sensitive to these choices, as they determine the initial exploration coverage before the LLM planner takes over.","section":null}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern regarding the clustering bias is valid and is the most significant issue. The design of the ablation (Table V) is actually quite strong in isolating the structural targeting mechanism, but the measurement of the dependent variable (distinct boundaries) is compromised by the lack of independent clustering validation. If the authors can demonstrate that the clustering is not biased by structural labels—perhaps by having the auditor cluster tests without their structural location metadata, or by providing human validation of a sample of clusters—the central claim would be much more convincing. The abstraction consistency issue is also important but somewhat secondary to the clustering bias. I recommend major revision to address these empirical validation gaps."},"author_rebuttal":null,"desk_editor":{"model":"glm-5.2","letter":"The paper you want to look at is AgentEval—a black-box testing framework for conversational LLM agents. The core idea: interact with an agent, mine a directly-follows graph from the conversation traces (borrowed from process mining), then use the graph's structural locations—nodes, edges, starts, ends—as deterministic targets for boundary test generation. Each test replays the conversational route to a boundary before perturbing it. The headline result: 23 distinct boundaries on the airline agent with structural targeting, versus 12 with prompt-only generation, at a lower duplicate rate (0.26 vs 0.56) and near-zero false alarms. Four τ³-bench domains, consistent results across all of them.","headline":"Solid empirical contribution to black-box testing of LLM agents. The ablation design is genuinely clean, but two intermediate LLM steps lack quantitative validation, which is where the skepticism should focus.","tokens_in":20213,"tokens_out":1619,"would_cite":true,"duration_ms":58470,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Mined workflow graphs double the boundaries a black-box agent test can reach","keywords":[],"falsifier":"If the LLM-driven event abstraction step is noisy—assigning different labels to the same activity or the same label to different activities—the directly-follows graph would fragment or collapse, producing spurious nodes and missing edges. The structural target enumeration would then operate on an unreliable model, and the 23-vs-12 boundary gain could be an artifact of graph topology rather than genuine boundary discovery. A direct falsification would show that abstraction consistency is low (e.g., re-labeling the same traces produces different graphs) and that the boundary gain disappears when","tokens_in":19712,"feed_emoji":"🔍","tokens_out":1128,"duration_ms":152272,"temperature":0.7,"pith_summary":"The paper claims that the most dangerous failures of conversational AI agents—canceling a booking before the user confirms, acting for an unverified caller, accepting a value that should be refused—sit at stateful boundaries hidden behind multi-turn conversational prerequisites. A tester who sends single prompts cannot reach these boundaries because the agent only exposes them after a sequence of turns has navigated to the right state. The paper presents AgentEval, a framework that solves this by first interacting with the agent as a black box to mine a conversational workflow graph: a directly-follows graph whose nodes are abstracted agent activities and whose edges carry the user actions observed between them. Once the graph is built, AgentEval does not prompt blindly for tests. Instead it deterministically enumerates every structural location in the graph—every node, edge, entry point, and terminal state—as a potential boundary target, asks an LLM to rank each target by how likely stressing it would expose a guard or prerequisite, and then generates a perturbation test for the highest-ranked targets. Each test inherits the conversational route to its target location, so the runner walks the agent through the prerequisite turns before applying the perturbation, reaching state-dependent behavior that a single prompt cannot. The central empirical claim is that this structural enumeration is what drives the gain: on an airline agent, graph-guided generation covers 23 distinct boundaries versus 12 with a prompt-only baseline, at a lower duplicate rate (0.26 vs 0.56) and near-zero false-alarm rate. Ablation shows that merely feeding the graph as prompt context does not help—only using it to enumerate structural targets does.","feed_headline":"Mined workflow graphs double the boundaries a black-box agent test can reach","feed_subtitle":"Structurally enumerating every node and edge in a conversational graph uncovers 23 distinct boundaries vs 12 with blind prompting, at near","key_machinery":"The conversational workflow graph is a directly-follows graph built from black-box session traces. An LLM first abstracts each conversation turn into a user action and an agent activity label, merging surface variants of the same activity into one node. Then a deterministic frequency-counting algorithm (Algorithm 1) adds an edge between any two activities observed in direct succession, recording the user actions that drove each transition. The resulting graph's structural locations—nodes, edges, entry points, terminal states—are then enumerated as boundary targets. Each target inherits the observed route (the sequence of user actions from session start to that location), and each generated边界","core_discovery":"The paper's central discovery is that the gain in boundary-test coverage comes not from having a graph but from using its structure to enumerate targets. When the same mined graph is passed to an LLM only as textual context, the generator produces 9 distinct boundaries—no better than the 12 from a no-graph baseline, because a free-form prompt still clusters on a few salient flows and repeats them. Only when every node, edge, start, and end of the graph becomes an explicit, deterministic test target does coverage jump to 23 distinct boundaries. The graph's structural locations force the generator to spread tests across the entire observed workflow rather than re-prompting the same few high-sy","pith_inferences":[],"forward_implications":["If structural enumeration of a mined behavioral model is the key driver of boundary-test diversity, then any conversational agent with a recurring interaction structure—routing, slot-filling, validation, confirmation gates—can be tested this way without source code access, making regression testing of deployed agents feasible as a CI/CD artifact.","The separation between the tester (black-box, graph-driven) and the auditor (white-box, source-code-reading) establishes a protocol for validating black-box test quality against ground truth, which other agent-testing frameworks could adopt.","The finding that graph-as-context fails but graph-as-structure succeeds suggests a broader principle: LLM test generators benefit from deterministic structural scaffolding that constrains where they look, rather than from being given more information in their prompt.","The identity-mismatch problem—where the generator pairs an authenticated caller with another customer's reservation, making the boundary unreachable—points to a need for state-aware test grounding that goes beyond copying values from observed traces."],"fun_headline_variants":["Graph structure, not graph context, drives black-box boundary test coverage","Structural enumeration of workflow nodes doubles black-box agent test coverage","AgentEval: workflow graph structure doubles reachable boundaries over blind prompts","Using graph structure as targets beats text context 23-to-9 for agent boundaries","Mined workflow graphs double boundary coverage when structure drives test targeting"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The entire graph construction depends on an LLM reliably assigning the same activity label to semantically identical agent replies across different conversations. If the LLM labels 'What is your reservation code?' and 'Could you give me the booking reference?' as different activities, the graph fragments into spurious nodes and missing edges, and the structural target enumeration that drives the boundary-testing gain operates on an unreliable model. The paper states that theL","fun_headline_variants_meta":{"raw":{"variants":["Graph structure, not graph context, drives black-box boundary test coverage","Structural enumeration of workflow nodes doubles black-box agent test coverage","AgentEval: workflow graph structure doubles reachable boundaries over blind prompts","Using graph structure as targets beats text context 23-to-9 for agent boundaries","Mined workflow graphs double boundary coverage when structure drives test targeting","Black-box testing reaches 23 boundaries vs 12 when graph structure enumerates targets","Graph structure as deterministic targets, not textual context, expands boundary coverage","Workflow graph structure forces test spread, doubling boundaries vs blind prompting","Textual graph context yields 9 boundaries; structural enumeration yields 23","Black-box agent testing rivals white-box auditor via mined workflow graph structure"]},"model":"glm-5.2","effort":"low","cost_usd":0.0,"raw_usage":{"total_tokens":1818,"prompt_tokens":559,"completion_tokens":1259,"prompt_tokens_details":null},"tokens_in":559,"tokens_out":1259,"duration_ms":67242,"temperature":1.0,"reasoning_tokens":1150,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-09T23:47:47.265342+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If the LLM-driven event abstraction step is noisy—assigning different labels to the same activity or the same label to different activities—the directly-follows graph would fragment or collapse, producing spurious nodes and missing edges. The structural target enumeration would then operate on an unreliable model, and the 23-vs-12 boundary gain could be an artifact of graph topology rather than genuine boundary discovery. A direct falsification would show that abstraction consistency is low (e.g., re-labeling the same traces produces different graphs) and that the boundary gain disappears when","supporting_citations":[],"review_version":1}