Arbor: A Framework for Reliable Navigation of Critical Conversation Flows

Catarina Farinha; Clara Matos; Diogo Gon\c{c}alves; Lu\'is Silva; Lu\'is Ungaro

REVIEW 3 major objections 4 minor 7 references

Decomposing decision-tree navigation into per-node retrieval, transition evaluation, and message generation lets small language models match or exceed much larger ones in clinical triage, raising mean turn accuracy by 29.4 percentage points

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 23:06 UTC pith:FV36V44G

load-bearing objection A clean engineering paper with a big accuracy claim, but the baseline may be silently broken by context overflow for several models, so the headline numbers need a rerun before I'd trust them. the 3 major comments →

arxiv 2602.14643 v4 pith:FV36V44G submitted 2026-02-16 cs.AI

Arbor: A Framework for Reliable Navigation of Critical Conversation Flows

Lu\'is Silva , Diogo Gon\c{c}alves , Catarina Farinha , Clara Matos , Lu\'is Ungaro This is my paper

classification cs.AI

keywords decision tree navigationlarge language modelsconversational AIclinical triageprompt decompositionstateful agentscontext windowcost efficiency

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Monolithic prompting — stuffing an entire 449-node, 980-edge decision tree into a single prompt — forces a model to simultaneously track state, evaluate all possible branches, and write a natural reply, a burden that degrades accuracy as trees grow and that heavily favors the largest, most expensive models. The paper argues that this is the wrong architecture for high-stakes structured workflows and proposes Arbor, which standardizes the tree into an edge list, retrieves only the current node's outgoing edges at each turn, uses one dedicated LLM call to evaluate transitions, and a separate call to generate the user-facing message. Across ten foundation models on real clinical triage conversations, Arbor improves mean turn accuracy from 58.80% to 88.23%, cuts mean per-turn latency from 33.84s to 14.51s, and drops mean per-turn cost from $0.166 to $0.012. The central claim is that architectural decomposition, not raw model capability, is what determines reliable navigation — so smaller, cheaper models become viable in safety-critical conversational systems.

Core claim

The paper claims that encoding a full decision tree in a single prompt is both inefficient and fundamentally unreliable for multi-turn navigation: accuracy is tightly coupled to the model's intrinsic reasoning strength, latency and cost scale with total tree size, and failures are opaque. In contrast, Arbor's decomposition — a standardized edge-list representation, dynamic retrieval of only the outgoing edges of the current node, an iterative transition-evaluation loop that can traverse multiple nodes per turn and outputs 'stay' when information is insufficient, and a separate message-generation step that receives the evaluation's reasoning — makes navigation accuracy primarily a function of

What carries the argument

The core mechanism is the edge-list representation combined with dynamic node-level retrieval and a two-phase inference loop: each decision tree is normalized offline into a list of transitions (source node, target node, question, answer, extra context, flags), validated for orphan nodes, broken references, and unescapable cycles; at runtime, only the current node's outgoing edges are retrieved and passed to an LLM evaluator that produces a 'stay' or a target node; if a transition is taken, the loop repeats; once the tree is exhausted, a separate LLM call generates the user-facing message using the evaluator's chain-of-thought reasoning. This mechanism enforces contextual isolation — the mod

Load-bearing premise

The single-prompt baseline is assumed to be a fair and feasible representation of monolithic navigation — specifically, that every evaluated model can actually ingest the full 119,990-token serialized tree without truncation or silent context-window overflow; if any model fails to process the whole prompt, the baseline is artificially degraded and Arbor's margin is overstated.

What would settle it

Run the same head-to-head comparison on a decision tree small enough (e.g., under 8,000 tokens) to fit comfortably in every model's context window; if the single-prompt baseline's turn accuracy rises to match Arbor's ~90% level, the claimed architectural advantage is largely a consequence of context overflow rather than decomposition of reasoning.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Smaller and open-weight language models become practical for high-stakes structured workflows such as clinical triage, since navigation accuracy decouples from model scale.
Per-turn latency and cost stop scaling with decision-tree size, so workflows can grow large without proportional operational overhead.
Failures become localizable to specific nodes or edges, easing debugging and enabling targeted updates without redesigning the entire prompt.
The evaluation and generation steps can be assigned to different models, allowing hybrid strategies that reserve expensive reasoning models for the decision step and use cheap models for wording.
The same decomposition principle should extend to any conversation that must follow a predefined protocol, such as compliance checklists, troubleshooting guides, or eligibility workflows.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The measured accuracy advantage may be partially inflated if any baseline model truncated the 119,990-token serialized tree; a fairness check would be to compare on a smaller tree that comfortably fits all context windows, where the gap should narrow if the mechanism is truly local reasoning rather than context overflow.
The single-reference-path annotation treats clarification requests as errors even when they are clinically reasonable; real-world gains in ambiguous cases could be lower than the reported 29-point spread, though the cost and latency advantages would still hold.
The framework's 'stay' mechanism and iterative traversal could be combined with confidence estimation or ensemble voting to flag low-certainty transitions for human review before committing a state change — a natural safety extension the paper mentions as future work but does not validate.
The abstraction of 'decision tree as data' suggests a marketplace or library of shareable, validated clinical workflows, since the agent logic is agnostic to tree authorship — an implication the authors gesture at but do not develop.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A clean engineering paper with a big accuracy claim, but the baseline may be silently broken by context overflow for several models, so the headline numbers need a rerun before I'd trust them.

read the letter

This is a solid engineering write-up of an LLM-based decision tree navigator, and the empirical comparison across ten models is genuinely useful. The two-step eval/generation split is clean, the prompts are documented, and the paper honestly credits prior work with the externalized-branching pattern. If the results hold, the framework is a practical deployable pattern for clinical triage.

But there's a load-bearing problem in the evaluation. The single-prompt baseline serializes the full tree to ~120k tokens, then appends conversation history. Several of the models (Qwen3-30B and GPT-4.1 among them) have 128k context windows. At later turns the combined input will exceed the window, and the paper never reports truncation checks or per-model context handling. If a model silently drops the tail of the tree, the baseline is failing not because of lost-in-the-middle but because the relevant edge is literally invisible. That would inflate the 29.4-point accuracy gain, the 57.1% latency improvement, and the 14.4x cost reduction. The stress-test note is right: this is not a minor footnote, it's the basis of the central claim.

Beyond that, the dataset is small — 20 conversations, 174 turns, one tree — and the cost numbers for open-weights models are normalized rather than actual prices, so they're only meaningful comparatively within the setup. The message-quality evaluation with three clinicians is a nice touch, but the sample of 50 aligned turns is underpowered for anything beyond ruling out a large effect. The paper's own limitations section is honest about forward-only traversal and state drift.

Still, the architecture itself is sound and the direction is sensible. The accuracy gains are large and consistent across models, even if the absolute numbers may shrink with a feasible baseline. This paper is for practitioners building structured conversational agents; researchers will find it a reasonable case study rather than a breakthrough.

My recommendation: send it to peer review, but require the authors to rerun the baseline with explicit truncation checks and per-model context usage at every turn. If the baseline is infeasible for some models, they need to either exclude those models or use a chunked baseline. Without that, the headline numbers are not trustworthy.

Referee Report

3 major / 4 minor

Summary. The paper proposes Arbor, a framework for LLM-based navigation of clinical decision trees. A decision tree is converted offline into an edge-list representation; at runtime, a language model evaluates only the outgoing edges of the current node in a dedicated transition-evaluation call, and a separate LLM call generates the user-facing message. The framework is evaluated against a single-prompt baseline that serializes the full 449-node/980-edge tree (about 119,990 tokens) into the system prompt. Using 174 annotated turns from real triage conversations and 10 model configurations, the paper reports that Arbor improves mean turn accuracy by 29.4 percentage points, reduces per-turn latency by 57.1%, and reduces per-turn cost by an average of 14.4x, while preserving message quality as judged by physical therapists.

Significance. If the empirical claims hold, the paper makes a useful practical contribution: it shows that a relatively simple decomposition of decision-tree navigation into local transition evaluation and response generation can substantially improve accuracy, latency, and cost relative to a monolithic single-prompt approach. The strengths of the study are that it uses real clinical conversations with expert annotations, evaluates a broad set of proprietary and open-weight models, repeats each configuration five times, and reports per-model standard deviations. The architecture is straightforward and provider-agnostic. However, the headline numbers depend on the single-prompt baseline being a fair and feasible representation of monolithic navigation. The paper does not verify that all evaluated models can ingest the 119,990-token serialized tree plus conversation history, and it mixes normalized open-weight costs with actual proprietary prices in the aggregate cost claim. These issues are load-bearing for the main claims and need to be resolved before the results can be fully accepted.

major comments (3)

[§4.1.1, §4.1.2, §6] The serialized tree alone is approximately 119,990 tokens with the o200k_base tokenizer, and the baseline appends the conversation history at every turn. The paper never reports per-model context windows, total per-turn token counts including history and instructions, or what happens when the API input exceeds the window. Several evaluated models (GPT-4.1, DeepSeek V3.1, Qwen3-30B, Qwen3-235B) have public context windows around 128k tokens, so the tree alone leaves little room for history, and later turns are likely to exceed the limit. If a model silently truncates the prompt, the baseline loses either the relevant part of the tree or the recent conversation, and the reported 29.4-point accuracy gain, 57.1% latency reduction, and 14.4x cost reduction are inflated by an infeasible baseline. The manuscript should either (a) report per-model token counts and verify against documented conte
[§4.1.3, §4.1.4, Table 3, Table 2] The cost metric mixes actual public prices for proprietary models with 'normalized inference costs under a fixed deployment configuration' for open-weight models. The per-model cost ratios in Table 6 range from about 5.7x to 302x, with the largest ratio coming from a normalized DeepSeek figure. The aggregate Table 2 hides this heterogeneity and reports a single 14.4x 'reduction in per-turn cost' as a headline result. Since the open-weights costs are not actual prices, this aggregate number is not an economically meaningful claim. Please report proprietary and open-weights costs separately, or state the deployment assumptions explicitly and provide sensitivity analysis.
[§3.2.1, §4.1.2, Figure 8] Section 3.2.1 states that Arbor's transition evaluator receives 'external context' such as eligibility constraints, risk flags, and profile attributes. Section 4.1.2 claims both architectures receive 'equivalent informational context,' but the single-prompt system prompt shown in Figure 8 only lists member name, birthdate, local time, and eligible/enrolled programs. If the baseline did not include the same non-conversational state variables, then the comparison is confounded: the baseline could fail because it lacks clinically relevant flags, not because monolithic prompting is intrinsically inferior. Please provide a complete list of fields supplied to each strategy, or explicitly confirm that all external-context fields were included in the baseline prompt.

minor comments (4)

[Abstract] The abstract reports '14.4x' in the full-text version but '13.8x' in the front-matter abstract; Table 2 reports 14.4x, though the ratio of the table's rounded mean costs is 13.8x. These numbers should be reconciled.
[§4.2.2, Table 7] The text says the message-quality evaluation used 'a sample of 50 real triage use cases' and that each case yielded two responses. Table 7 sums to 145 responses for Arbor and 150 for the single-prompt baseline. The acceptance rates (97.3% and 100%) also appear inconsistent with these counts. Please clarify the number of cases, turns, and accepted messages.
[§4.1.3] There is a typo in the bullet heading: 'T urn accuracy' should be 'Turn accuracy.' Also, 'perceptuallimitation' in Section 1 should be 'perceptual limitation.'
[§4.2.1] The message-quality evaluation uses only GPT-5-minimal. This is reasonable for a controlled comparison, but the paper should state explicitly that the message-quality conclusion is limited to that model and may not generalize to other models or to turns where the two strategies choose different nodes.

Circularity Check

0 steps flagged

No significant circularity: Arbor's reported gains are empirical measurements against a fixed baseline, not derivations from its own inputs.

full rationale

The paper's central claims—29.4-point accuracy gain, 57.1% latency reduction, and 14.4x cost reduction—are empirical measurements from head-to-head evaluations of Arbor against a single-prompt baseline across ten models. No model parameter is fitted to the evaluation data and then renamed as a prediction; the framework is fully specified before evaluation. The turn-level ground truth is manually annotated from recorded conversations, and the comparison is defined by a fixed baseline protocol. The only concerns raised by reviewers concern the feasibility of the single-prompt baseline (e.g., whether all models can ingest the full ~119,990-token serialized tree) and the use of self-annotated private data, but these are external-validity or measurement-validity issues, not circular derivation. The paper contains no load-bearing self-citations: prior work is cited for background (lost-in-the-middle, chain-of-thought, graph-of-thoughts, etc.), but Arbor's design and evaluation do not depend on any uniqueness theorem or prior result by the same authors. The limitations section candidly discusses remaining issues such as latency sensitivity, forward-only traversal, and state drift. Accordingly, no step in the derivation reduces to its own inputs by construction.

Axiom & Free-Parameter Ledger

1 free parameters · 5 axioms · 0 invented entities

The central claim rests on several domain assumptions about data representativeness and annotation quality, and on a single hand-chosen cost normalization. No scientific entities are postulated.

free parameters (1)

Normalized token rates for open-weights models = Table 3 (e.g., $0.60/$1.70 per M tokens for DeepSeek V3.1)
Authors state these are normalized inference costs under a fixed deployment configuration, not actual market prices; the cost-reduction claim depends on these chosen rates.

axioms (5)

domain assumption The manually annotated current-node and target-node labels for all 174 turns are correct.
Ground truth for turn accuracy; annotation procedure is not described (who annotated, inter-annotator agreement). Section 4.1.1.
domain assumption The 20 selected conversations are representative of the triage workflow.
Selection criteria are vague ('reflects a realistic distribution') and there is no statistical justification; Section 4.1.1.
domain assumption Every evaluated model can fully ingest the 119,990-token single-prompt baseline without truncation.
No context-window handling or truncation checks are reported; if false, baseline accuracy is artificially depressed. Section 4.1.2.
domain assumption Chain-of-thought prompting in the evaluation step improves transition selection.
The paper cites Wei et al. but does not ablate CoT vs. no-CoT within Arbor; the accuracy gains could come from context reduction alone. Section 3.2.1.
domain assumption Repeated runs at temperature 0 with five seeds capture the relevant nondeterminism.
They acknowledge nondeterminism but 5 runs may be insufficient for stable variance estimates. Section 4.1.2.

pith-pipeline@v1.3.0-alltime-deepseek · 14866 in / 11845 out tokens · 111534 ms · 2026-08-02T23:06:18.103667+00:00 · methodology

0 comments

read the original abstract

Large language models struggle to maintain strict adherence to structured workflows in high-stakes domains such as healthcare triage. Monolithic approaches that encode entire decision structures within a single prompt are prone to instruction-following degradation as prompt length increases, including lost-in-the-middle effects and context window overflow. To address this gap, we present Arbor, a framework that decomposes decision tree navigation into specialized, node-level tasks. Decision trees are standardized into an edge-list representation and stored for dynamic retrieval. At runtime, a directed acyclic graph (DAG)-based orchestration mechanism iteratively retrieves only the outgoing edges of the current node, evaluates valid transitions via a dedicated LLM call, and delegates response generation to a separate inference step. The framework is agnostic to the underlying decision logic and model provider. Evaluated against single-prompt baselines across 10 foundation models using annotated turns from real clinical triage conversations. Arbor improves mean turn accuracy by 29.4 percentage points, reduces per-turn latency by 57.1%, and achieves an average 13.8x reduction in per-turn cost. These results indicate that architectural decomposition reduces dependence on intrinsic model capability, enabling smaller models to match or exceed larger models operating under single-prompt baselines.

discussion (0)

Reference graph

Works this paper leans on

7 extracted references · 3 linked inside Pith

[1]

Traverse the decision tree deciding based on the current chat history in which node you are
[2]

Decide if the current node is fully addressed; if not, remain here and generate the message for this node
[3]

Available Triage Paths

URLhttps://arxiv.org/abs/2411.07037. An Yang, Anfeng Li, Baosong Yang, and Beichen Zhang et al. Qwen3 technical report, 2025. URL https://arxiv.org/ abs/2505.09388. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023. URL http...

Pith/arXiv arXiv 2025
[4]

Output both: - message: what to send to the member

Craft the message for the final node you land on. Output both: - message: what to send to the member. - new_current_node: the node key you ended at. # Traversal Instructions - Use the conversation history to check if a node’s question is already answered. - Always check children of the current node in the full DT. - Never guess or assume an answer; if uns...
[6]

Traverse to the next child node whose answer cannot yet be inferred
[2024]

15 Arbor: A Framework for Reliable Navigation of Critical Conversation Flows OpenAI

URLhttps://arxiv.org/abs/2401.11120. 15 Arbor: A Framework for Reliable Navigation of Critical Conversation Flows OpenAI. Introducing gpt-4.1 in the api, apr 2025a. URLhttps://openai.com/index/gpt-4-1/. OpenAI. Gpt-5 system card. System card, OpenAI, aug 2025b. URL https://cdn.openai.com/gpt-5-system-card.pdf . Fatma ˝Ozcan, Abdul Quamar, Jaydeep Sen, Chu...

Pith/arXiv arXiv 2020
[2025]

Annica Ernesäter, Inger Holmström, and Maria Engström

URLhttps://openreview.net/forum?id=XMb9poL2Mo. Annica Ernesäter, Inger Holmström, and Maria Engström. Telenurses’ experiences of working with computerized decision support: supporting, inhibiting and quality improving.Journal of advanced nursing, 65(5):1074–1083, 2009. Kelly Hong. Context rot: How increasing input tokens impacts llm performance, 2025. URL...

Pith/arXiv arXiv 2009

[1] [1]

Traverse the decision tree deciding based on the current chat history in which node you are

[2] [2]

Decide if the current node is fully addressed; if not, remain here and generate the message for this node

[3] [3]

Available Triage Paths

URLhttps://arxiv.org/abs/2411.07037. An Yang, Anfeng Li, Baosong Yang, and Beichen Zhang et al. Qwen3 technical report, 2025. URL https://arxiv.org/ abs/2505.09388. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023. URL http...

Pith/arXiv arXiv 2025

[4] [4]

Output both: - message: what to send to the member

Craft the message for the final node you land on. Output both: - message: what to send to the member. - new_current_node: the node key you ended at. # Traversal Instructions - Use the conversation history to check if a node’s question is already answered. - Always check children of the current node in the full DT. - Never guess or assume an answer; if uns...

[5] [6]

Traverse to the next child node whose answer cannot yet be inferred

[6] [2024]

15 Arbor: A Framework for Reliable Navigation of Critical Conversation Flows OpenAI

URLhttps://arxiv.org/abs/2401.11120. 15 Arbor: A Framework for Reliable Navigation of Critical Conversation Flows OpenAI. Introducing gpt-4.1 in the api, apr 2025a. URLhttps://openai.com/index/gpt-4-1/. OpenAI. Gpt-5 system card. System card, OpenAI, aug 2025b. URL https://cdn.openai.com/gpt-5-system-card.pdf . Fatma ˝Ozcan, Abdul Quamar, Jaydeep Sen, Chu...

Pith/arXiv arXiv 2020

[7] [2025]

Annica Ernesäter, Inger Holmström, and Maria Engström

URLhttps://openreview.net/forum?id=XMb9poL2Mo. Annica Ernesäter, Inger Holmström, and Maria Engström. Telenurses’ experiences of working with computerized decision support: supporting, inhibiting and quality improving.Journal of advanced nursing, 65(5):1074–1083, 2009. Kelly Hong. Context rot: How increasing input tokens impacts llm performance, 2025. URL...

Pith/arXiv arXiv 2009