Impatient Users Confuse AI Agents: High-fidelity Simulations of Human Traits for Testing Agents

Anand Kumar; James Zou; Meghana Rajeev; Muyu He; Nazneen Rajani; Tsach Mackey

REVIEW 4 major objections 5 minor 1 cited by

Impatient Users Confuse AI Agents: High-fidelity Simulations of Human Traits for Testing Agents

T0 review · 4 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read User traits like impatience are linearly encoded in LLM activation space, and steering test users with them drops frontier agents by 2–30% (up to 46%).

desk verdict Useful idea and a real benchmark, but the trait vectors come from contaminated contrastive pairs and the tuning shares the evaluation signal — referee it, but expect major revision. read the letter →

arxiv 2510.04491 v3 pith:3GXSM765 submitted 2025-10-06 cs.AI cs.CL

Muyu He , Anand Kumar , Tsach Mackey , Meghana Rajeev , James Zou , Nazneen Rajani This is my paper · ORCID

classification cs.AIcs.CL

keywords activationsteeringtraitvectorsusersimulationAIagentrobustnessconversationalagentsτ-Traitbenchmarkpersonacomposition

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that current conversational AI agents are brittle to small shifts in user behavior, but benchmarks miss this because they model users with static prompts. It introduces TraitBasis, a method that extracts trait directions from contrastive responses in a model's activation space and steers a user simulator at inference time without retraining. On realism, fidelity, stability, and compositionality, TraitBasis beats prompting, full fine-tuning, and LoRA in human evaluations. Using it to extend τ-Bench into τ-Trait across four domains, the paper reports that three frontier models degrade by an average of 2–30%—and up to 46%—when the simulated user becomes impatient, confused, skeptical, or incoherent, even though the underlying tasks are unchanged.

What carries the argument

TraitBasis: a matrix of trait vectors extracted as P_T(z) = (1/n) Σ_i (P_i,pos(z) − P_i,neg(z)), where each P is the per-token mean of hidden activations at layer z for contrastive conversation pairs. At inference the hidden state is updated as h(z) ← h(z) + α P_t(z). The τ-Trait benchmark extends τ-Bench's POMDP with a trait-vector space V, changing the transition to S × A × V → S × O.

What would settle it

Compute an impatience vector from airline dialogues and apply it to a banking domain; if blinded raters cannot tell the steered user is more impatient than a neutral user, the linear-direction claim is false. Alternatively, subtracting the impatience vector from an already impatient user should make them measurably calmer; if it does not, the vector is not the causal driver of the observed 2–30% performance drops.

Watch

Extended reading notes

Core claim

The central claim is that human-like traits are linearly encoded in the activation space of a language model, so that the direction obtained by averaging the differences between trait-positive and trait-negative conversations, P_T = (1/n)Σ(P_pos − P_neg), acts as a controllable trait vector. Adding a scaled version of this vector to hidden states at a chosen layer during inference produces user behavior that human raters judge as more realistic, more finely adjustable in intensity, more stable over long dialogues, and more easily composable across multiple traits than prompt-based or fine-tuned baselines. When these vectors are applied to the user simulator in τ-Bench, the resulting τ-Trait

Load-bearing premise

The whole method stands on the belief that a human trait lives in one linear direction inside a single layer of a model's internal states, so that the same steering vector works across any new conversation, intent, or domain.

Editorial extensions

If this is right

If trait vectors are stable and composable, agent evaluation can avoid expensive human role-play and instead generate diverse, behaviorally varied users on demand for any domain.
Because τ-Trait degrades frontier agents even though tasks and intents are unchanged, the reported performance drops quantify agent brittleness attributable purely to user behavior.
TraitBasis can be applied at inference time without fine-tuning or extra data, making robustness testing lightweight and model-agnostic.
Composing traits via linear combination produces blended personas without the trait suppression or imbalance seen in prompt-based and fine-tuned baselines.
The open-sourced τ-Trait environments across four domains allow the community to QA agents under realistic trait variation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the linear-direction hypothesis holds across architectures, TraitBasis could be extended to other traits (e.g., politeness, urgency, anxiety) without collecting new data, just by writing contrastive exemplars.
The 2–30% degradation numbers likely underestimate real-world risk because real users combine traits with memory and adaptivity; a natural next test is mixing trait vectors with user attributes to see if degradation compounds.
A calibration procedure could select the steering strength α per trait and domain automatically using the benchmark's verifiable tasks as a criterion, rather than relying on human selection of the best layer.
A falsifiable prediction: if the trait vector is causally responsible for the degradation, then subtracting it from already-traitful users should reduce perceived trait intensity and recover agent performance; this reverse-steering experiment is not reported in the paper.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this paper is worth refereeing but not yet worth trusting. The idea—extracting steerable vectors for user traits from contrastive examples and using them to stress-test agents—is genuinely useful, and the τ-Trait benchmark with open data is a real artifact. But the central mechanism has a load-bearing weakness that the paper does not address, and the headline robustness numbers are shakier than the text claims.

What's new: applying activation steering to multi-turn user-trait simulation, composing trait vectors, and packaging it as a benchmark with two new domains (telecom, telehealth). The human-eval protocol (Elo with 100 shuffles, fidelity with abstain) is careful, and the paper is honest that LLM-as-judge fails on compositionality, which is a point in its favor.

Soft spots, in rough order of severity. First, the contrastive pairs in App. A.5 are not content-matched. The skeptical 'trait' responses begin with "Let me be very skeptical here"—explicit trait labels—and the confused responses are outright hostile ("what the hell?? ... Are you retarded?"), which is abuse, not confusion. So the averaged difference vector PT is encoding surface markers and unrelated affect, not a clean trait direction. The paper claims the pairs "differ only in the intensity of the trait," and that is simply not what the appendix shows. Until the vectors are re-derived from clean pairs (same content, same length, no trait labels), the realism and fidelity numbers can be explained by keyword mimicry. The τ-Trait degradation numbers inherit this.

Second, the layer selection and α calibration use five annotators picking the "most obvious steering result," and the evaluation RQs ask humans to rank realism and intensity. Same signal on both sides. That is at least partly circular.

Third, Table 3 claims degradation across all models and domains, but two cells are positive (GLM-4.5 retail skepticism +0.2, telecom/telehealth skepticism +0.8). And the numbers are averages over 3 rollouts with no error bars. That's minor but should be fixed.

Who this is for: people building agent robustness benchmarks and using activation steering for user simulation. If the method is cleaned up, it deserves a place. As it stands, the empirical case does not establish that TraitBasis captures realistic trait shifts rather than style artifacts. I'd send it to peer review—the idea is timely and the artifact is real—with a strong request for cleaner contrastive pairs, a pre-registered tuning/eval split, and error bars. I would not cite it in my own work until those changes land.

Referee Report

4 major / 5 minor

Summary. The paper introduces TraitBasis, an activation-steering method that induces user traits (impatience, confusion, skepticism, incoherence) in LLM-based user simulators by adding contrastively learned trait vectors to hidden states at inference. The authors evaluate TraitBasis against prompt-based steering, SFT, and LoRA on realism, fidelity, stability, and compositionality using both human annotators and an LLM judge, reporting large gains over baselines. They then apply TraitBasis to extend τ-Bench into τ-Trait across airline, retail, telecom, and telehealth domains, and report an average 2%–30% performance degradation for GPT-4o, Kimi K2, and GLM-4.5, which they interpret as evidence that frontier agents are brittle to realistic user-behavior shifts.

Significance. If the trait vectors genuinely encode the intended human traits rather than surface-level markers, TraitBasis would be a data-efficient and composable robustness-testing tool, and τ-Trait would be a valuable benchmark for agent evaluation. The paper has concrete strengths: it releases the benchmark, uses pairwise human evaluation with 100 shuffles for realism, compares against three baseline families, and spans multiple domains and models. However, the central interpretive claim is not yet established: the contrastive pairs used to derive trait vectors appear to differ in content, register, and explicit trait self-labeling, and the headline τ-Trait degradation numbers lack variance estimates, contain a contradictory cell, and are averaged over only three rollouts. The contribution is potentially significant, but the current evidence is insufficient to support the strong claims in the abstract and introduction.

major comments (4)

[Appendix A.5 and §3.2] The method's core assumption is that the contrastive pairs (Y_pos, Y_neg) “differ only in the intensity of the trait” (§3.2). The exemplars in App. A.5 violate this. For skepticism, every trait response begins with the explicit self-label “Let me be very skeptical here,” while the normal responses differ in content (“Yes I have asked, and they said there is a balance of 100.”). For confusion, the trait turns are hostile and abusive (“what the hell?? I don't understand. I DO NOT NEED ANYONE TO COME HERE.... Are you retarded?”, “Why for the sake of god is it taking so long?”) rather than confused, and differ from normal turns in length, register, and content. P_T = (1/n)Σ(P_pos − P_neg) therefore cancels only pair-specific incidental features, not auxiliary attributes; it may encode explicit trait labels, anger, and response style. This undermines the claim that τ-Trait degradation measure
[§6.2 and Table 3] The headline “2%–30% performance degradation across frontier models” is not supported by the reported numbers. Table 3 reports deltas averaged over only 3 rollouts with no variance, confidence intervals, or baseline pass rates; one cell (GLM-4.5, Retail, Skepticism) is +0.2, directly contradicting the text in §6.2 that performance degrades “across all three models and all four domains.” In addition, the table groups Telecom and Telehealth into a single row even though the abstract and §5 describe four domains. Please report per-domain results with CIs/statistical tests, clarify the direction of every cell, and revise the abstract and introduction accordingly.
[§3.2 and §4.2] The layer and vector selection procedure — five annotators choose the z*(T) and vector producing “the most obvious steering result” — and the α-calibration are not described in detail, and they are not separated from the subsequent fidelity/realism evaluations. Selecting for obviousness can favor exactly the explicit trait markers identified above, making RQ1/RQ2 partly a measure of how well the method matches the calibration criterion rather than an independent assessment of trait realism. Please specify the α-calibration algorithm, report annotator agreement and the exact calibration instructions, and confirm that RQ1/RQ2 evaluators were distinct from calibration annotators and did not see the contrastive exemplars.
[§3.2 and §5] The paper calls TraitBasis “model-agnostic,” but activation vectors are defined for a specific model's hidden states (P_T^(z) depends on LLM parameters θ). Nowhere do the authors specify which model the vectors are extracted from or which user simulator is steered in RQ1–RQ4 and in τ-Trait. Without this, the experiments are not reproducible, and the applicability to the three agent models (GPT-4o, Kimi K2, GLM-4.5) is unclear: either separate vectors were extracted per agent model, or vectors from one model were applied to another. This needs to be stated and justified.

minor comments (5)

[Throughout] There are several typos and inconsistencies: “we select the select” (§3.2), “soley” (§5), “In this casem an agent” (Figure 3 caption), “TauBench” vs. τ-Bench, and inconsistent use of “Consistency” vs. “Stability” across Table 2, Table 5, and the text. Please copy-edit carefully.
[§6.1 and Table 2] TraitBasis is said to achieve the “highest consistency rate” at 24.8% (human), but 24.8% is low in absolute terms. The stability definition — consistency plus realistic escalation — should be stated before citing these numbers, and the low absolute consistency should be discussed.
[Figure 2 and Table 2] Elo scores are reported with ± values (e.g., 1530.08±45). Please state what these intervals represent (e.g., standard deviation across 100 shuffles) and how they were computed.
[Appendix A.5] The confusion contrastive exemplars contain offensive language (“Are you retarded?”) and are not appropriate as trait demonstrations. Even aside from the validity issue raised above, they should be replaced.
[§4.2] The number of annotators is given as “at least 3” for each instance, but no inter-annotator agreement metrics (e.g., Cohen's κ or Krippendorff's α) are reported. Please add agreement statistics for the human evaluations.

Circularity Check

2 steps flagged · score 6.0 of 10

Fidelity evaluation is partly self-fulfilling via calibration on the same criterion, and trait vectors are contrastive differences that encode surface markers rather than traits alone.

fitted input called prediction [Section 3.2, paragraph on layer/strength selection; Section 4.2, RQ2]
"To select the select the most effective layer z∗(T) and vector P (z) t for each trait T , we generate a conversation of 10 turns using each of [z ∗(T), P (z) t ] to measure the quality of their influence on outputs. We then ask five annotators to select the conversation that sees the most obvious steering result. ... For each pair, we only choose the multi-turn conversations with intensity C ∈ {low,high} because their difference represents the largest shift in trait intensity. ... Annotators are tasked to select the conversation that better conveys the intended trait."

The layer z* and calibrated strengths α are chosen by asking five annotators which 10-turn conversation shows 'the most obvious steering result.' The RQ2 fidelity test then asks annotators to pick which conversation 'better conveys the intended trait' / shows higher intensity. These are the same construct evaluated on the same kind of output. Thus TraitBasis's high fidelity score (97.5% human) is partly guaranteed by construction: the steering parameters were selected to maximize the very obviousness that fidelity measures. Baselines did not receive an equivalent selection loop, so the reported margins over SFT/prompting on fidelity are not a fair, independent comparison.
self definitional [Section 3.2, contrastive-pair construction; Appendix A.5]
"To find the vector for a trait T , we need a pair of contrastive responses (Ypos, Yneg) to the same prompts X={x 1 . . . xn} that differ only in the intensity of the trait exhibited ... By generating such n pairs of responses, we are able to cancel out the effect of auxiliary attributes and model the vector for T. [A.5] 'Let me be very skeptical here. I'm not fully sure about that.' ... 'what the hell?? I don't understand. I DO NOT NEED ANYONE TO COME HERE.... Are you retarded?'"

P_T is defined as P(z)_T := (1/n) Σ(P(z)_i,pos − P(z)_i,neg). The appendix pairs that define this vector do not 'differ only in the intensity of the trait': the skeptical examples contain the explicit self-label 'Let me be very skeptical here,' and the confusion examples are hostile/abusive and differ in content, length, register, and cooperativeness. Therefore the 'trait direction' is by construction the average of all these differences, not a direction isolating the trait. Applying h ← h + α P_T then reproduces surface markers and unrelated affect rather than independently inducing the trait. Calling this 'steerable user traits' is a definitional restatement of the contrast, and τ-Trait degradation may reflect keyword-laden or abusive turns instead of realistic user-behavior shifts.

full rationale

The central τ-Trait benchmark has independent content: it measures downstream agent success rates under trait-perturbed user models, and the degradation numbers are not logically forced by the construction of the benchmark. That part is not circular. However, two load-bearing components of the paper's central claim are partly self-fulfilling. First, the layer and strength calibration in §3.2 uses the same 'most obvious steering' criterion that RQ2 later calls fidelity, so the headline fidelity advantage over baselines is partially a restatement of the selection procedure. Second, the trait vectors are computed from contrastive pairs that, by the paper's own appendix, differ in far more than trait intensity, and the paper asserts without evidence that averaging cancels auxiliary attributes. Since P_T is defined as that average difference, the claimed inference-time 'trait control' reduces to reproducing the contrast-pair differences, including explicit marker phrases and unrelated affect. Self-citation is not a major issue here: the activation-steering premise cites independent prior work (Chen et al. 2025) as well as same-author work (Liu et al. 2024), and the premise is also empirically tested in this paper. Overall, the derivation is not fully circular because the comparisons, stability, compositionality, and τ-Trait robustness numbers contain genuine independent measurement; but the fidelity claim and the construct validity of the trait directions are substantially circular, warranting a partial-circularity score of 6.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The method rests on activation-space linearity, contrastive isolation, and calibrated intensities, plus the correctness of LLM-generated benchmark environments. No formal verification or independent replication is presented.

free parameters (3)

trait direction PT = mean contrastive activation difference per trait; 4 pairs per trait in Appendix A.5
The central steering object is estimated from a small set of hand-written contrastive conversations; it is not derived from first principles.
layer z*(T) = one layer per trait chosen by five annotators
Section 3.2: the layer is picked by generating 10-turn conversations and asking annotators to select the 'most obvious steering result'.
trait strength α(T, level) = calibrated scalars for low/medium/high per trait
Section 3.2 says the strengths are 'calibrated'; the calibration procedure is not specified, and these values directly determine the intensity contrast measured in RQ2.

assumptions (5)

domain assumption There exists a linear direction in a model's activation space that encodes each human trait, and adding it to hidden states steers behavior (Section 3.2, 'we work under the assumption...').
The paper relies on this to justify the entire method; it cites prior work but does not prove it for these traits.
domain assumption Averaging contrastive activation differences (positive minus negative) isolates the trait and cancels all other attributes such as intent and context (Section 3.2).
The method assumes that auxiliary attributes cancel in the difference; this is plausible but not demonstrated per trait.
domain assumption Human annotators' and LLM judges' preferences are reliable ground truth for realism, fidelity, stability, and compositionality (Section 4.2, Appendix A.2).
All four headline comparisons rest on subjective judgments; no inter-annotator agreement statistics are reported.
domain assumption Synthetic telecom/telehealth databases, tools, and policies generated by Claude Sonnet 4 and manually verified are correct enough to measure agent success (Section 5).
The new benchmark domains are entirely synthetic; errors in schemas, tools, or policies would change the measured degradation.
ad hoc to paper The intensity levels {low, medium, high} are meaningfully ordered and are realized by calibrated α values (Section 3.2).
The existence of distinguishable intensity levels is assumed by construction and then measured in RQ2.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Impatient Users Confuse AI Agents: High-fidelity Simulations of Human Traits for Testing Agents." pith.science (2026). https://pith.science/paper/3GXSM765

@misc{pith2026251004491,
  author       = {Pith},
  title        = {Pith review of: Impatient Users Confuse AI Agents: High-fidelity Simulations of Human Traits for Testing Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3GXSM765}},
  note         = {Machine review of arXiv:2510.04491}
}

abstract

Despite rapid progress in building conversational AI agents, robustness is still largely untested. Small shifts in user behavior, such as being more impatient, incoherent, or skeptical, can cause sharp drops in agent performance, revealing how brittle current AI agents are. Today's benchmarks fail to capture this fragility: agents may perform well under standard evaluations but degrade spectacularly in more realistic and varied settings. We address this robustness testing gap by introducing TraitBasis, a lightweight, model-agnostic method for systematically stress testing AI agents. TraitBasis learns directions in activation space corresponding to steerable user traits (e.g., impatience or incoherence), which can be controlled, scaled, composed, and applied at inference time without any fine-tuning or extra data. Using TraitBasis, we extend $\tau$-Bench to $\tau$-Trait, where user behaviors are altered via controlled trait vectors. We observe on average a 2%-30% performance degradation on $\tau$-Trait across frontier models, highlighting the lack of robustness of current AI agents to variations in user behavior. Together, these results highlight both the critical role of robustness testing and the promise of TraitBasis as a simple, data-efficient, and compositional tool. By powering simulation-driven stress tests and training loops, TraitBasis opens the door to building AI agents that remain reliable in the unpredictable dynamics of real-world human interactions. We have open-sourced $\tau$-Trai across four domains: airline, retail, telecom, and telehealth, so the community can systematically QA their agents under realistic, behaviorally diverse intents and trait scenarios: https://github.com/collinear-ai/tau-trait.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

When Should Service Agents Reconsider? Difficulty-Routed Control in Customer-Service Operations
cs.AI 2026-07 unverdicted novelty 5.0 of 10

A difficulty-routed architecture routes conflicted customer-service requests to escalated workflows with conflict-aware communication and write-triggered reconsideration, improving reliability on operational conflicts...

Reference graph

Works this paper leans on

16 extracted references · 1 linked inside Pith · cited by 1 Pith paper

[1]

Conversation 1 — shows the trait more realistically
[2]

Conversation 2 — shows the trait more realistically
[3]

•Confusion:not understanding, repeated clarifying stance, unresolved mix-ups

Neither — neither shows the trait realistically Trait Reference: •Impatience:more pressure to act, quicker push, noticeable escalation. •Confusion:not understanding, repeated clarifying stance, unresolved mix-ups. •Skepticism:challenging/testing of claims, withholding acceptance. •Incoherence:harder to follow, poor grammar, disorganized. RQ2 Instructions ...
[4]

•Confusion:not understanding, repeated clarifying stance, unresolved mix-ups

Not present — the trait is absent in both Trait Reference: •Impatience:more pressure to act, quicker push, noticeable escalation. •Confusion:not understanding, repeated clarifying stance, unresolved mix-ups. •Skepticism:challenging/testing of claims, withholding acceptance. •Incoherence:harder to follow, poor grammar, disorganized. RQ3 Instructions You wi...
[5]

Conversation 1 — shows the trait more strongly
[7]

Neither — both show the trait with equal strength
[9]

Conversation 1 — shows the trait more strongly 13
[10]

Conversation 2 — shows the trait more strongly

Show all 16 references

[11]

Same Intensity — both show the trait with equal strength
[12]

impatience

Not present — the trait is absent in both Trait Reference: •Impatience:more pressure to act, quicker push, noticeable escalation. •Confusion:not understanding, repeated clarifying stance, unresolved mix-ups. •Skepticism:challenging/testing of claims, withholding acceptance. •I...
[13]

Take the existing persona and enhance it by layering in the specific {{ trait_name }} trait at intensity level {{ trait_intensity }}
[14]

Keep all the original persona characteristics intact
[15]

Add the trait-specific behavior as a natural extension of their existing personality
[16]

Make it feel like one cohesive personality, not separate traits bolted together
[17]

"") context_bot_template = Template(

Focus on how this trait level would manifest in their communication style and approach CRITICAL REQUIREMENTS: - Keep the original persona's context, situation, and core characteristics - Seamlessly blend in the {{ trait_name }} trait at the specified intensity - Use natural, c...
[2025]

yes” if you’d like me to move forward with this change. User Yes, please go ahead and downgrade all of them. Tool Call 1{ 2

URLhttps://arxiv.org/abs/2508.20453. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ-bench: A benchmark for tool-agent-user interaction in real-world domains, 2024. URLhttps://arxiv.org/abs/2406.12045. Ming Yin, Dinghan Shen, Silei Xu, Jianbing Han, Sixun Dong...

2024 arXiv

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.

[1] [1]

Conversation 1 — shows the trait more realistically

[2] [2]

Conversation 2 — shows the trait more realistically

[3] [3]

•Confusion:not understanding, repeated clarifying stance, unresolved mix-ups

Neither — neither shows the trait realistically Trait Reference: •Impatience:more pressure to act, quicker push, noticeable escalation. •Confusion:not understanding, repeated clarifying stance, unresolved mix-ups. •Skepticism:challenging/testing of claims, withholding acceptance. •Incoherence:harder to follow, poor grammar, disorganized. RQ2 Instructions ...

[4] [4]

•Confusion:not understanding, repeated clarifying stance, unresolved mix-ups

Not present — the trait is absent in both Trait Reference: •Impatience:more pressure to act, quicker push, noticeable escalation. •Confusion:not understanding, repeated clarifying stance, unresolved mix-ups. •Skepticism:challenging/testing of claims, withholding acceptance. •Incoherence:harder to follow, poor grammar, disorganized. RQ3 Instructions You wi...

[5] [5]

Conversation 1 — shows the trait more strongly

[6] [7]

Neither — both show the trait with equal strength

[7] [9]

Conversation 1 — shows the trait more strongly 13

[8] [10]

Conversation 2 — shows the trait more strongly