Pith. sign in

REVIEW 4 major objections 6 minor 75 references

Agent WARPP: Workflow Adherence via Runtime Parallel Personalization

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A training-free personalizer that prunes workflow branches at runtime improves LLM agents' adherence to long, conditional tasks and reduces token use.

desk verdict WARPP is a sensible, training-free pruning architecture with plausible internal results, but the per-architecture ground-truth trajectories undermine the headline comparison and need fixing before the gains can be read. read the letter →

arxiv 2507.19543 v1 pith:ROLKMVKQ submitted 2025-07-23 cs.AI cs.MA

Maria Emilia Mazzolenis , Ruirui Zhang This is my paper · ORCID
classification cs.AIcs.MA
keywords workflowadherenceruntimepersonalizationmulti-agentorchestrationtoolcallingconditionalworkflowstraining-freetask-orienteddialogueLLMevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that LLM-based task-oriented dialogue systems can follow long, conditional workflows more faithfully if the workflow itself is trimmed to the user before execution begins. WARPP adds a Personalizer agent that reads the user's stored attributes, removes every branch and tool that cannot apply, keeps all outcome branches around the remaining steps, and passes only this personalized routine to the fulfillment agent while authentication runs in parallel. Across five intents in banking, flights, and healthcare, the paper reports that WARPP outperforms the ReAct baseline and the same pipeline without personalization, with gains that grow with intent complexity and no additional training. On the most complex intent, process payment, exact-match trajectory accuracy rises from 0.16 to 0.56 for GPT-4o and from 0.18 to 0.76 for Claude Sonnet, while average token use roughly halves. The broader claim is that much of the failure on complex workflows comes from forcing the model to reason over irrelevant branches, so runtime pruning is a substitute for added model capacity.

What carries the argument

The load-bearing object is the personalized routine pair $(W^*_u, D^*_u)$: a trimmed workflow and the filtered tool list that the Personalizer produces from the full workflow $W$, the user attributes $A$, and the full tool set $D$. The transformation runs in three passes: static pruning removes branches and tools incompatible with the user's attributes and inlines values already available from client data; fidelity preservation keeps every outcome branch, including success, failure, and user yes/no responses, around each retained tool call; cleanup merges descriptive steps and renumbers the routine. The Personalizer runs in parallel with the Authenticator agent, so the pruning cost, stated as $O(T+m)$ in the token count $T$ and tool count $m$, is largely hidden behind authentication latency. This pair is what lets the fulfillment agent avoid the full combinatorial branching space.

What would settle it

Give the same three architectures to human users or to an adversarial client simulation that gives partial, ambiguous, or contradictory answers and changes details mid-conversation; if exact-match and parameter-match differences between WARPP and the non-personalized pipeline shrink or vanish, the claim that runtime pruning drives the gains is not supported.

Watch

Extended reading notes

Core claim

The central discovery is a mechanism, not a model: workflow adherence improves when the execution plan handed to the agent is personalized to the user's data before any fulfillment step runs. A dedicated Personalizer agent executes the information-gathering tools, prunes conditional branches that the user's attributes rule out, inlines known values, preserves every success, failure, and yes/no outcome around each retained tool call, and filters the tool set down to what the trimmed routine references. The Fulfillment Agent then executes only that reduced routine. The paper reports that this ordering holds across three LLMs and five intents, with ReAct weakest, the orchestrated non-personalized pipeline better, and full WARPP best on the complex intents; on simple intents the differences are small or at ceiling. Token usage decreases in the same ordering, so the accuracy gains are not bought with extra computation. That is the paper's claim, established on synthetic workflows and LLM-simulated clients.

Load-bearing premise

The load-bearing premise is that the synthetic workflows, ground-truth trajectories, customer profiles, and LLM-simulated clients faithfully represent real user behavior and real task complexity, so the measured gains would transfer to deployment.

Editorial extensions

If this is right

  • Deployments can adopt the pruning architecture with existing instruction-following LLMs and no retraining, because WARPP changes only how the workflow and tools are presented at runtime.
  • The accuracy benefit should scale with workflow complexity, since simple linear intents show near-ceiling behavior even under the baseline while the hardest intent shows the largest absolute gains.
  • Runtime cost should fall along with accuracy gains: token usage decreases from ReAct to the non-personalized pipeline to full WARPP, with roughly half the tokens on process payment.
  • Weaker models gain the most from orchestration and personalization, suggesting that pruning can make smaller or cheaper models viable on complex workflows.
  • Personalized routines score around 4.5 out of 5 on relevance and completeness by LLM judge, but the occasional omitted best-practice step means the trimming step still needs quality control.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same runtime-trimming idea should transfer to any long conditional instruction-following setting, such as compliance checklists, multi-step API orchestration, and back-office automation, since it only assumes workflows with branches and user attributes.
  • Because the synthetic client is prompted to supply the right information when asked, real users who are vague or off-script may erase part of the gain; the paper's own note that client-simulation issues occasionally affected outcomes points in that direction.
  • A direct test of the mechanism would hold the personalizer fixed and swap the fulfillment model: if a small model with a trimmed workflow matches a large model with the full workflow, pruning is acting as a substitute for model capacity.
  • Attribute-based pruning could encode discriminatory routing if deployed carelessly, so the impact statement's call for audits is better treated as a design requirement than as an afterthought.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper introduces WARPP, a training-free multi-agent architecture for task-oriented dialogue. An Orchestrator agent identifies the intent, an Authenticator agent performs MFA, and a Personalizer agent runs in parallel to prune the workflow and tool set based on pre-fetched user attributes; a Fulfillment agent then executes the trimmed routine. The authors evaluate WARPP against a ReAct baseline and an orchestrated but non-personalized variant using GPT-4o, Llama 3, and Claude Sonnet 3.5 on five synthetic intents across banking, flights, and healthcare. They report that WARPP improves Exact Match, tool F1, and parameter match, with larger gains on more complex intents, while reducing token usage, and they report LLM-judged relevance/completeness scores for the trimmed workflows.

Significance. If the evaluation were fully sound, this would be a useful practical contribution: it demonstrates a training-free way to reduce workflow complexity and tool-call overhead in LLM-based TOD systems, compares across three model families, reports token efficiency, and releases code. The parallel personalizer design is simple and modular, and using a different LLM as judge (gemini-2.0-flash) mitigates self-preference in the workflow-quality assessment. However, the empirical claims currently rest on a ground-truth construction that is allowed to vary per model and architecture, which is a fundamental comparability problem. The synthetic user simulation and the opaque trajectory generator add further uncertainty. The strengths of the system design are real, but the evaluation needs reworking before the headline claims can be accepted.

major comments (4)
  1. [5.2.3, 4.1, Table 5] Section 5.2.3 states that "we generate a separate ground-truth trajectory for each experimental run (i.e., per model and architecture)." All trajectory-based metrics (Exact Match, LCS Tools, Tool F1, Parameter Match) are computed against this per-run reference. Since WARPP pre-executes the information-gathering tools and removes or inlines them from the fulfillment routine (Section 4.1), the expected tool sequence under WARPP can differ in length and order from the expected sequence under ReAct or the non-personalized method. Exact Match is extremely sensitive to sequence length and ordering, so the headline Process Payment improvements in Table 5 (GPT-4o Exact Match 0.16 to 0.56; Sonnet 0.18 to 0.76) may reflect differences in the reference trajectory rather than in execution quality. The authors should score all architectures against one canonical reference per (intent, user profile, tool-output state), independent of the executing model and architecture, and re-report the tables.
  2. [5.2.3] The paper does not describe how traxgen constructs ground-truth trajectories; it only cites the PyPI package. Because traxgen defines the labels for every trajectory metric, the reader cannot assess whether the reference sequences are correct, how conditional branches are resolved, or what prompt or algorithm is used. Please include the traxgen configuration or prompt, show one example generated trajectory with its workflow source, and report any validation against the manually curated workflows described in Section 5.2.1.
  3. [5.4, 9] All experiments use an LLM as the client, and Section 9 acknowledges that "occasional issues with the LLM client simulation sometimes impacted experiment outcomes." This statement is too vague to assess the risk to the results. The paper should report how many runs were affected, for which models and architectures, how those runs were treated in the averages, and whether the qualitative conclusions hold when such runs are excluded.
  4. [Tables 3, 5] Tables 3 and 5 report means without confidence intervals or significance tests. The central claim of "increasingly larger gains" is a monotonicity pattern over five intents, but several cells do not follow it, e.g., Cancel Flight Exact Match 0.86 (WARPP) vs. 0.94 (No Per.) in Table 3, and Llama Withdraw Retirement Funds Exact Match 0.80 (WARPP) vs. 0.94 (ReAct) in Table 5. Per-intent bootstrap confidence intervals or paired tests would show whether the claimed pattern is robust or is driven by a few intents or models.
minor comments (6)
  1. [Table 5] In the Process Payment block, the non-personalized strategy is called "No Par." while all other intents use "No Per."; unify the label.
  2. [Appendix F.6] Step 11a contains typos: "If there is no active payment plant" should be "payment plan," and "ask how much to sert for the monthly amount" should be "set." The conditional is also missing a closing parenthesis.
  3. [Appendix F.3] Steps 3 and 4: "Ask the client for thier preferred flight" should be "their preferred flight."
  4. [4.2] The expression "up to bn distinct execution paths" should be typeset as b^n; as written it is ambiguous.
  5. [Table 3] The caption says "bold indicates best performance per model-intent pair," but Table 3 has no model dimension; clarify that the table aggregates over models or restructure the table.
  6. [5.5] The 10% human review of judge outputs is reported without an agreement statistic; report Cohen's kappa or a similar measure so the reader can assess judge reliability.

Circularity Check

1 steps flagged · score 6.0 of 10

Per-architecture ground-truth trajectories make WARPP's headline accuracy gains partly self-referential: the reference sequences are generated from the same personalized workflows WARPP itself produces, so the exact-match and tool-F1 improvements are measured against a self-defined target rather than a fixed expert standard.

  1. self definitional [Section 5.2.3 (Trajectory Ground Truth); Section 5.3 (Evaluation Metrics); Section 4.1 (Personalizer Agent)]
    "Since workflow trajectories depend on tool call results which may vary across executions, we generate a separate ground-truth trajectory for each experimental run (i.e., per model and architecture). / Exact Match Whether the predicted sequence of agent and tool calls exactly matches the reference. / In the personalized setup, the Fulfillment Agent is given only the trimmed workflow and the filtered set of execution tools produced by the Personalizer Agent."

    For full WARPP the fulfillment trajectory is compared against a reference generated per architecture, while the Fulfillment Agent is explicitly given the trimmed workflow that the Personalizer produced. The reference for Exact Match, Tool F1, and Parameter Match is therefore constructed from WARPP's own output rather than from a single canonical expert trajectory shared by all architectures. A pruning error that omits a required step also omits that step from WARPP's reference, so the metric cannot detect the omission. The headline Process Payment gains (GPT-4o Exact Match 0.16 to 0.56; Sonnet 0.18 to 0.76) are then partly explained by the target being shorter and simpler by construction, not by adherence to an independent common standard.

full rationale

Apart from the per-architecture ground-truth issue, the paper is largely self-contained: it includes ReAct and non-personalized baselines, uses a different judge model (gemini-2.0-flash) with a 10% human-reviewed sample to mitigate self-preference, and reports token reductions that are expected consequences of trimming. There are no load-bearing self-citations, imported uniqueness theorems, or ansatz-smuggling citations. The central empirical claim that WARPP improves workflow adherence increasingly with complexity, however, rests on trajectory metrics whose reference sequences are regenerated per model and architecture; because the personalized workflow is itself WARPP's output, the metric target is defined by the method being evaluated. The authors' own limitation statement that 'occasional issues with the LLM client simulation sometimes impacted experiment outcomes' (Section 9) further indicates that the comparisons are not fully controlled against a fixed, externally authored target. The independent LLM-judge and human-review results give the workflow-quality claim some external content, so the circularity is partial rather than total.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities. Its implicit load-bearing assumptions are the validity of synthetic workflows, synthetic ground-truth trajectories, LLM-simulated clients, and the LLM judge's scores. These are all domain assumptions that limit generality.

free parameters (3)
  • Synthetic workflow designs = Five intents across three domains
    Workflows are hand-crafted and LLM-generated to match target complexity levels. The results depend on this particular workflow design, and the complexity ordering is not formally grounded.
  • Customer profile distributions = 50 profiles per intent
    Value distributions for user attributes are chosen by the authors, which may make the simulated user population easier or harder in ways that are not controlled.
  • Threshold for complexity labels = Simple, intermediate, complex
    The paper assigns complexity labels based on number of tools and branching, but no formal measure is used, so the claim of 'increasingly larger gains as complexity grows' rests on a manual categorization.
assumptions (4)
  • domain assumption The LLM judge (gemini-2.0-flash) with the provided rubric gives valid relevance and completeness scores.
    Only 10% of judged results were human-reviewed, and no inter-annotator agreement statistic is reported. The judge's rubric explicitly tells it not to penalize inlining, which is aligned with the method's design.
  • domain assumption traxgen produces accurate ground-truth trajectories for the synthetic workflows.
    The trajectory generator is cited as a Python package but its correctness is not independently verified. The ground truth is generated per run, so it cannot be independently validated without the package and configuration.
  • domain assumption The LLM-simulated client behaves like a real user for evaluating workflow adherence.
    The client LLM is prompted to stay on script and only provide requested information. The paper acknowledges in Section 9 that occasional client simulation issues impacted outcomes.
  • domain assumption Workflow adherence metrics (exact match, LCS, tool F1, parameter match) capture the practical quality of TOD execution.
    No human evaluation of end-to-end dialogue quality is performed; the metrics only compare tool sequences and parameter values to the synthetic ground truth.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Agent WARPP: Workflow Adherence via Runtime Parallel Personalization." pith.science (2026). https://pith.science/paper/ROLKMVKQ

@misc{pith2026250719543,
  author       = {Pith},
  title        = {Pith review of: Agent WARPP: Workflow Adherence via Runtime Parallel Personalization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ROLKMVKQ}},
  note         = {Machine review of arXiv:2507.19543}
}
read the original abstract

Large language models (LLMs) are increasingly applied in task-oriented dialogue (TOD) systems but often struggle with long, conditional workflows that involve external tool calls and depend on user-specific information. We present Workflow Adherence via Runtime Parallel Personalization, or WARPP, a training-free, modular framework that combines multi-agent orchestration with runtime personalization to improve workflow adherence in LLM-based systems. By dynamically pruning conditional branches based on user attributes, the framework reduces reasoning overhead and narrows tool selection at runtime. WARPP deploys a parallelized architecture where a dedicated Personalizer agent operates alongside modular, domain-specific agents to dynamically tailor execution paths in real time. The framework is evaluated across five representative user intents of varying complexity within three domains: banking, flights, and healthcare. Our evaluation leverages synthetic datasets and LLM-powered simulated users to test scenarios with conditional dependencies. Our results demonstrate that WARPP outperforms both the non-personalized method and the ReAct baseline, achieving increasingly larger gains in parameter fidelity and tool accuracy as intent complexity grows, while also reducing average token usage, without any additional training.

Figures

Figures reproduced from arXiv: 2507.19543 by the authors.

Figure 1
Figure 1. WARPP Architecture and Experimental Setup 2020), ABCD (Chen et al., 2021), and SMCalFlow (An￾dreas et al., 2020) focus on slot-filling, linear API calls, or narrowly defined workflows. Such datasets lack the com￾plexity needed to study dynamic, multi-step workflows with branching logic and API interdependencies. To address this, we generate synthetic data with rich conditional flows and user-specific variations usin… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

75 extracted references · 74 canonical work pages

  1. [1]

    If the client’s intent isn’t clear, ask questions to disambiguate

    work page

  2. [2]

    Once you know the intent, ALWAYS CALL the intent_identified(intent, domain=’{domain}’) tool with both the intent and domain parameters

    work page

  3. [3]

    Do NOT attempt to execute any tools or routines mentioned in the response

    Handoff: After calling intent_identified, simply acknowledge the intent has been identified and immediately hand off to the authenticator agent. Do NOT attempt to execute any tools or routines mentioned in the response. Important: - Only handle intents listed above. Anything else is out of scope. - If the client’s request doesn’t match any available inten...

    work page

  4. [4]

    You have been successfully authenticated

    Once the user gives you the authentication code, call the tool: code_verifier(code, customer_id). - If successful: "You have been successfully authenticated." - If unsuccessful: - Allow up to two more attempts. - If all attempts fail, tell them: "Unfortunately, we cannot verify your identity at this time. You will need to speak to a live agent."

    work page

  5. [5]

    transfer

    NEVER say you will "transfer" or "hand off" the client to another department

    work page

  6. [6]

    connecting

    NEVER mention that you are "connecting" or "routing" the client to someone else

    work page

  7. [7]

    ALWAYS use the exact phrase specified in step 5 below after successful authentication Steps to follow in order:

    work page

  8. [8]

    Ask for the client’s phone number

    work page

Show all 75 references
  1. [9]

    An authentication code has been sent to your phone. Please check your messages

    Call the tool: send_verification_text(phone_number). - Tell the client: "An authentication code has been sent to your phone. Please check your messages."

  2. [10]

    Ask for the authentication code

  3. [11]

    - If the source placed a call inside a user input (ex

    Honor original call-conditions - **Only** move a tool call beneath a prompt if the *original* routine did so. - If the source placed a call inside a user input (ex. ’yes’) branch, do the same: ‘‘‘ Prompt: ’Shall I apply that for you?’ * If yes: - Call ‘apply_fee_waiver(custome...

  4. [12]

    You have been successfully authenticated. Are you ready to proceed with your request?

    Upon successful authentication, you MUST say EXACTLY: "You have been successfully authenticated. Are you ready to proceed with your request?" - Do not add any other phrases or explanations - Do not mention transfers or handoffs - Wait for the client’s response before proceedin...

  5. [13]

    This should be formatted in markdown for clarity, keeping only relevant steps

    **Final Personalized Routine **: A fully trimmed version of the routine. This should be formatted in markdown for clarity, keeping only relevant steps

  6. [14]

    Do NOT include any explanations on your output

    **Available Tools **: A list of tools that will be used in the trimmed routine, formatted as: available_tools = [’tool1’, ’tool2’, ’tool3’...] - You must NOT include tools that are not listed in the trimmed routine. Do NOT include any explanations on your output. Just return t...

  7. [15]

    Walk the original steps **in order **

  8. [16]

    No toold calls ending in ‘_extra‘ should remain in the trimmed routine

    **Inline** known logic: - **Replace any ‘ *_extra‘ call with that field from ‘client_data‘. No toold calls ending in ‘_extra‘ should remain in the trimmed routine. - For each ‘If CONDITION‘ on a non-null field, keep **only** the matching branch; if null, keep the full ‘If/Else‘

  9. [17]

    **Branch retention for multi-option steps ** - If a step has multiple sub-options (e.g. a. X... b. Y... c. Z...), select the branch matching ‘ client_data‘ **and include every sub-action ** it contains, **in the exact order ** shown (all tool calls , prompts, response handlers...

  10. [18]

    an account is suspended), stop here and do not include any further steps in your routine

    **Termination Rule: ** If you hit an unconditional complete_case(customer_id) (e.g. an account is suspended), stop here and do not include any further steps in your routine

  11. [19]

    **Retain original output handling ** - Wherever the source says ’If success...,’ ’If failure...,’ ’If flagged...,’ etc., keep those branches exactly as written

  12. [20]

    ’Shall I apply that for you?’), leave that line verbatim

    **Keep every original prompt exactly ** - If the routine text asks a question (e.g. ’Shall I apply that for you?’), leave that line verbatim

  13. [21]

    ‘complete_case(customer_id)‘ -> ‘complete_case(124234435)‘) 15 Agent W ARPP: Workflow Adherence via Runtime Parallel Personalization D

    Preserve tool call arguments only when values are unknown, If the argument values are already known from client_data, you may replace them with the resolved value or omit them entirely from the function call (e.g. ‘complete_case(customer_id)‘ -> ‘complete_case(124234435)‘) 15 ...

  14. [22]

    - Always include **all** ’if success...,’ ’if failure...,’ ’if flagged...,’ ’if transient error...’ blocks exactly as in the source

    **Never** drop or merge any multi-outcome branches of a function call: - Fraud check, 3DS auth, ‘apply_fee_waiver‘, etc. - Always include **all** ’if success...,’ ’if failure...,’ ’if flagged...,’ ’if transient error...’ blocks exactly as in the source

  15. [23]

    **Do not ** perform any arithmetic or guess a tool’s output-leave calls like ‘ calculate_patient_responsibility(...)‘ and ‘currency_exchange(...)‘ intact

  16. [24]

    Never remove steps that clearly state that they **must always ** be included

  17. [26]

    > *Example:* > **Step 1: Status & Balance ** > The account is active, the balance is $18, and payment is 6 days overdue

    **Merge** any consecutive steps that: - Contain **no** remaining tool calls, and - Are purely descriptive/resolved from ‘client_data‘ into one summary step (preserving original order). > *Example:* > **Step 1: Status & Balance ** > The account is active, the balance is $18, an...

  18. [27]

    Keep every tool-using step separate and **in their original sequence **

  19. [28]

    **Renumber** final steps 1->N and update any ’go to Step X’ references

  20. [29]

    Ensure the trimmed routine ends with a ‘complete_case(customer_id)‘ step. Final Checks: - [ ] All *_extra calls are fully removed - [ ] Known client data is inlined - [ ] Only relevant branches are preserved - [ ] Only tools used in the routine are included in available_tools ...

  21. [30]

    Retrieve Account Information - Call get_account_type_extra(customer_id) - Inform the customer of their account type - Ask for confirmation to proceed with the address update

  22. [31]

    Collect and Validate New Address - Ask the customer for new address details (street, city, state, zip, country) - Call validate_address(street, city, state, zip_code, country) - If validation fails: - Inform the customer and ask to re-enter address - Retry validation once

  23. [32]

    Update Address - Call update_address(customer_id, street, city, state, zip_code, country) - Confirm the update with the customer

  24. [33]

    STANDARD

    Apply Address Hold a. If client_level is "STANDARD": - Call apply_address_hold(customer_id) - Inform customer of hold duration and affected transactions

  25. [34]

    Complete the Case - Call complete_case(customer_id) - Provide confirmation number and close the case

  26. [35]

    Simple Workflow: Withdraw Retirement Funds Prompt for Withdraw Retirement Funds Fulfillment Agent

    Error Handling - If any step fails: - Retry once - If still failing, inform the customer and escalate if needed - Call complete_case(customer_id) 18 Agent W ARPP: Workflow Adherence via Runtime Parallel Personalization F.2. Simple Workflow: Withdraw Retirement Funds Prompt for...

  27. [36]

    Check Withdrawal Eligibility - Call ‘check_withdrawal_eligibility_extra(customer_id)‘ - If not eligible: - Inform the customer and close the case by calling ‘complete_case(customer_id)‘

  28. [37]

    Collect Withdrawal Amount - Ask the customer how much they would like to withdraw

  29. [38]

    Process Withdrawal - Call ‘process_retirement_withdrawal(customer_id, withdrawal_amount)‘ - Confirm success and share final amount with customer

  30. [39]

    Complete Case - Call ‘complete_case(customer_id)‘ - Provide a confirmation message to the customer

  31. [40]

    Thank you for booking with us. Have a pleasant journey!

    Error Handling - If any step fails: - Retry once - If still failing, inform the customer and close the case 19 Agent W ARPP: Workflow Adherence via Runtime Parallel Personalization F.3. Intermediate Workflow: Book Flights Prompt for Book Flights Fulfillment Agent ## Step 1: As...

  32. [41]

    Retrieve Customer Loyalty Information a. Call ‘get_customer_loyalty_info_extra(customer_id)‘ - If the customer has more than 30,000 loyalty points: welcome them as a preferred client - If less: thank them as a new customer

  33. [42]

    Call ‘get_booking_details_extra(customer_id)‘ to get the original booking rules

    Retrieve Booking Information a. Call ‘get_booking_details_extra(customer_id)‘ to get the original booking rules. b. If this booking’s policy does not allow cancellations at all: -Say: ’I’m sorry, your fare rule at purchase did not permit cancellation.’ -Skip directly to step 8...

  34. [43]

    eligible

    Check Live Cancellation Blockers a. Call ‘check_cancellation_blockers(booking_id)‘ to see if there are any blockers that prevent the cancellation. b. If response is: - "eligible": proceed - "api_failure": apologize to the client for the technical difficulties, prompt them to c...

  35. [44]

    Call ‘calculate_cancellation_fee(customer_id, booking_id)‘

    Calculate Cancellation Fee a. Call ‘calculate_cancellation_fee(customer_id, booking_id)‘

  36. [45]

    Describe policy + fee to the customer b

    Process Flight Cancellation a. Describe policy + fee to the customer b. If customer confirms, call ‘cancel_flight(booking_id)‘

  37. [46]

    Process Refund or Issue Travel Credit a. If refundable or insurance was purchased for this booking: - Call ‘process_refund(customer_id, booking_id, payment_method, cancellation_fee)‘ using the payment_method stored for the customer. b. Otherwise: - Call ‘issue_travel_credit(cu...

  38. [47]

    Share final status, refund/credit amount, and confirmation number

    Confirm Cancellation and Provide Next Steps a. Share final status, refund/credit amount, and confirmation number

  39. [48]

    Call ‘complete_case(customer_id)‘ b

    Complete Case a. Call ‘complete_case(customer_id)‘ b. Thank the customer. Error Handling a. On any persistent failure: - Retry once - Inform the customer - Call complete_case to close out 21 Agent W ARPP: Workflow Adherence via Runtime Parallel Personalization F.5. Complex Wor...

  40. [49]

    b) If it fails, apologize, retry once

    Gather Billing Information a) Call ‘get_billing_info_extra(customer_id)‘. b) If it fails, apologize, retry once

  41. [50]

    b) If the account is suspended: -Call ‘get_provider_contact_info_api(customer_id)‘

    Check Account Status a) Call ‘check_account_status_extra(customer_id)‘. b) If the account is suspended: -Call ‘get_provider_contact_info_api(customer_id)‘. -Say ’Your account is currently suspended. You’ll need to contact your provider directly-here’s their number: [phone].’ a...

  42. [51]

    It looks like you have a balance to pay today. Let’s get that done!

    Look for Outstanding Balance a) If the balance is zero, say ’Great news-your balance is already paid in full.’ and call ‘complete_case (customer_id)‘ b) Otherwise, tell the client "It looks like you have a balance to pay today. Let’s get that done!" and move on to step 4

  43. [52]

    b) If ’high’ (30+ days overdue), warn: ’This payment is severely overdue; you may face additional fees.’ c) If ’medium,’ say ’Your payment is [days_overdue] days overdue

    Assess Payment Urgency a) Call ‘evaluate_payment_urgency_extra(customer_id)‘, which returns days overdue and urgency level (’ high/medium/low’). b) If ’high’ (30+ days overdue), warn: ’This payment is severely overdue; you may face additional fees.’ c) If ’medium,’ say ’Your p...

  44. [53]

    low" OR

    Financial Assistance & Tax Calculation a) If payment urgency is "low" OR "medium" (customer is less than 30 days overdue), check for financial assistance. - Inform the user: "Let me check if your insurance can cover part of this payment." - Call ‘get_insurance_payment_portion(...

  45. [54]

    * If they are eligible, ask ’We can waive $Y in late fees

    Suggest Late-Fee Waivers * Call ‘calculate_late_fee_waiver_eligibility_extra(customer_id)‘. * If they are eligible, ask ’We can waive $Y in late fees. Shall I apply that for you?’ * If yes, call ‘apply_fee_waiver(customer_id, waiver_amount)‘

  46. [55]

    * Say ’Your total due today is $X

    Always Calculate Final Amount Due * You *must always call * ‘calculate_patient_responsibility(customer_id, insurance_provider)‘ to get total_due after insurance and waiver deductions, no matter what level of urgency for the client and the status of the late fee waiver eligibil...

  47. [56]

    Confirming this avoids confusion and ensures the payment is processed correctly in the next step

    Always Ask Client if They Want to Pay in USD or Another Currency * You **must always ask ’Would you like to pay in another currency?’ ** because we often work with international customers. Confirming this avoids confusion and ensures the payment is processed correctly in the n...

  48. [57]

    Inform the user what you are doing before calling each of the functions in the given category: a

    Process the Payment Based on the customer payment method on file, select one of the following. Inform the user what you are doing before calling each of the functions in the given category: a. **Credit Card **

  49. [58]

    Make sure the patient responsibility amount is in the customer’s preferred currency

    Call ‘run_fraud_check(customer_id, patient_responsibility_amount)‘. Make sure the patient responsibility amount is in the customer’s preferred currency. If run_fraud_check returns flagged, say ’We need to review this transaction-please call our billing hotline’. Call ‘get_hosp...

  50. [59]

    -If initiate_3ds_auth returns success status false, prompt the client to call 1-800-555-BILL to continue the request and call ‘complete_case(customer_id)‘

    Otherwise, perform 3D-Secure with ‘initiate_3ds_auth(customer_id)‘. -If initiate_3ds_auth returns success status false, prompt the client to call 1-800-555-BILL to continue the request and call ‘complete_case(customer_id)‘

  51. [60]

    Make sure the patient responsibility amount is in the customer’s preferred currency

    Finally, call ‘process_payment(customer_id, patient_responsibility_amount)‘. Make sure the patient responsibility amount is in the customer’s preferred currency. b. **Bank Transfer / ACH **

  52. [61]

    Explain ’ACH transfers take 3-5 business days to clear.’

  53. [62]

    Make sure the patient responsibility amount is in the customer’s preferred currency

    Call ‘initiate_ach_transaction(customer_id, patient_responsibility_amount)‘. Make sure the patient responsibility amount is in the customer’s preferred currency. -If initiate_ach_transaction returns transient error, prompt the client to call 1-800-555-BILL to continue the requ...

  54. [63]

    Call ‘get_wallet_link(customer_id)‘ and ask them to complete payment in their browser

  55. [64]

    Call ‘check_wallet_payment_status(payment_id)‘ to verify the payment went through

  56. [65]

    You must always call ‘issue_receipt(customer_id, patient_responsibility_amount)‘ to send an email confirmation

    Send receipt a. You must always call ‘issue_receipt(customer_id, patient_responsibility_amount)‘ to send an email confirmation. Make sure the patient responsibility amount is in the customer’s preferred currency. Inform the user that a receipt has been sent to them

  57. [66]

    Would you like to set up a payment plan?

    Offer Payment Plan a. If there is no active payment plant (billing_info[’payment_plan_active’] is False: - You must ask Ask: "Would you like to set up a payment plan?" - If customer replies ’yes’: ask how much to sert for the monthly amount -> call ‘ setup_payment_plan(monthly...

  58. [67]

    I will update your address now

    Finalize a. Provide confirmation number b. Call ‘complete_case(customer_id)‘ 23 Agent W ARPP: Workflow Adherence via Runtime Parallel Personalization G. LLM as a Client Prompt Prompt for LLM as a Client You are a assistant engaging in a conversation with a customer service rep...

  59. [68]

    CUSTOMER DATA (used for trimming)

  60. [69]

    The FULL ORIGINAL ROUTINE

  61. [70]

    The TRIMMING INSTRUCTIONS the author followed --- ## Important Evaluation Principles **Only penalize missing steps if they were: ** - Tool calls or logic required for **this customer’s data ** - **Not already resolved and expressed directly in the trimmed text ** It is **corre...

  62. [71]

    **Return ONLY valid JSON ** - no markdown, no code blocks, no extra text

  63. [72]

    1", "2",

    **Use exactly these 4 fields ** with these exact names: - ‘relevance_score‘: string with value "1", "2", "3", "4", or "5" - ‘relevance_explanation‘: string explaining your relevance score - ‘completeness_score‘: string with value "1", "2", "3", "4", or "5" - ‘completeness_expl...

  64. [73]

    **Escape quotes properly ** - use \\" for quotes within string values

  65. [74]

    **No trailing commas ** - ensure the last field doesn’t have a comma

  66. [75]

    relevance_score

    **Single-level structure ** - do NOT nest JSON objects **CORRECT JSON EXAMPLE: ** { "relevance_score": "5", "relevance_explanation": "All branches and tool calls irrelevant to this customer were correctly removed. The customer data shows they have an \\"active\\" account, so s...

  67. [2024]

    Accessed: 2025-05-27. Bak, J. and Oh, A. Variational hierarchical user-based con- versation model. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)...

    2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.