Pith. sign in

REVIEW 4 major objections 5 minor 3 cited by

Can a Single Model Master Both Multi-turn Conversations and Tool Use? CoALM: A Unified Conversational Agentic Language Model

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that a single model trained on a mixture of dialogue, tool-use, and ReAct data can outperform specialized models, including GPT-4o, at both multi-turn task-oriented dialogue and function calling.

desk verdict A genuinely useful dataset and model family, undermined by an abstract overclaim and an unaddressed contamination risk. read the letter →

arxiv 2502.08820 v3 pith:7ZLOKH42 submitted 2025-02-12 cs.AI cs.CL

Emre Can Acikgoz , Jeremiah Greer , Akul Datta , Ze Yang , William Zeng , Oussama Elachqar , Emmanouil Koukoumidis , Dilek Hakkani-Tür
show 1 more author
Gokhan Tur
This is my paper · ORCID
classification cs.AIcs.CL
keywords conversationalagentstask-orienteddialoguefunctioncallingReActmulti-turninstructiontuningAPIusagezero-shotevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to answer a question the field has treated as two separate problems: can one language model carry a multi-turn, goal-oriented conversation while also calling a wide range of external APIs correctly? The authors say yes, and their evidence is the CoALM model family, trained on a new mixture called CoALM-IT that interleaves dialogue state tracking, single-turn function calling, and multi-turn ReAct-style dialogue with API calls. Evaluated without using any evaluation-benchmark training data, CoALM 8B, 70B, and 405B achieve strong results on MultiWOZ 2.4, API-Bank, and BFCL V3, and the two larger variants are claimed to outperform GPT-4o and specialized baselines across these benchmarks. If the claim holds, a single open model can replace the current pattern of pairing a dialogue system with a separate agentic tool-caller.

What carries the argument

The load-bearing mechanism is the training mixture CoALM-IT and, within it, the Conversational ReAct API (CRA) format. CRA turns each dialogue turn into a fixed sequence: User, Thought1 (deciding whether an API call is needed), Action, Observation, Thought2 (deciding whether the information suffices), and System response. This format teaches the model when to call an external API and when to answer directly, inside a multi-turn conversation, rather than treating function calling as a single-turn classification. The non-CRA components supply complementary skills: SNIPS-derived samples teach dialogue state tracking, and Hammer and ToolACE samples teach complex single-turn and multi-turn function calling over thousands of APIs. Interleaving all three in one fine-tuning pass is what the paper credits for avoiding overfitting to any one skill.

What would settle it

Run a systematic contamination audit of CoALM-IT against the MultiWOZ 2.4 test set, API-Bank dialogues, and BFCL V3 cases, using exact and near-duplicate matching on paraphrased utterances, API names, and argument values. If a substantial share of test items has a matching training sample, the reported zero-shot gains would be inflated; a weaker check is to retrain CoALM after removing the ToolACE and Hammer subsets and re-measure BFCL V3 to see how much of the gain depended on benchmark-style training data.

Watch

Extended reading notes

Core claim

The central claim is that conversational skill and agentic tool use do not require separate models: a single instruction-tuned model can do both, provided the training data mixes the right skills. CoALM-IT combines SNIPS-derived dialogue state tracking, Hammer and ToolACE function-calling samples, and a newly generated Conversational ReAct API dataset built from SGD dialogues, where each agent turn is expanded into Thought1-Action-Observation-Thought2-Response. Fine-tuning Llama base models on this interleaved mixture produces CoALM 8B, 70B, and 405B. On MultiWOZ 2.4, CoALM 70B reaches a Success rate of 69.4 and Joint Goal Accuracy of 43.8, beating GPT-4o on the state-tracking metric; on API-Bank and BFCL V3, the larger CoALM models surpass GPT-4o on overall function-calling accuracy, with CoALM 405B reaching 100 percent on the relevance-detection subtask. The authors' conclusion is that the apparent trade-off between multi-turn dialogue mastery and advanced API use is an artifact of how models are trained, not an inherent limitation.

Load-bearing premise

The load-bearing premise is that none of the evaluation-benchmark test items, in original or paraphrased form, appear in the CoALM-IT training mixture, a premise the paper asserts but does not verify with a contamination analysis.

Editorial extensions

If this is right

  • One open model can serve as both the dialogue manager and the tool caller, removing the need to combine a task-oriented dialogue system with a separate function-calling agent.
  • Because MultiWOZ training data was not used, the MultiWOZ results are presented as evidence that the recipe generalizes to unseen dialogue domains rather than memorized API flows.
  • The ablation results imply that each training ingredient is load-bearing: removing the language-agent data sharply degrades API-Bank Rouge-L, removing the state-tracking data lowers Joint Goal Accuracy, and removing the CRA data hurts MultiWOZ Success and multi-turn API-Bank performance.
  • Scaling from 8B to 70B to 405B consistently improves overall BFCL V3 accuracy, with the 405B model reaching 100 percent on relevance detection, suggesting the unified training objective benefits from scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the zero-shot result is real, the CoALM-IT recipe is a candidate template for turning any instruction-tuned base model into a conversational agent, but the paper only tests Llama-family bases, so the breadth of the recipe remains untested.
  • Because the ToolACE and Hammer subsets are the same style of data used to build function-calling leaderboards, a contamination audit is the natural next check; test-set leakage, including paraphrased examples, is the main alternative explanation for the reported gains.
  • The paper's own limitation note leaves general reasoning ability unmeasured after fine-tuning, so a follow-up could test CoALM on standard reasoning benchmarks to see whether unified training causes catastrophic forgetting.
  • A testable extension is to replace static fine-tuning with reinforcement learning over live API rollouts, which would show whether the unified conversational-agent skill can improve beyond the curated CoALM-IT data.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes CoALM, a family of LLMs fine-tuned on a new multi-task dataset (CoALM-IT) that combines task-oriented dialogue (TOD) state tracking from SNIPS, function-calling data from Hammer and ToolACE, and a newly generated multi-turn ReAct-style dataset (CRA) derived from SGD. The authors evaluate CoALM 8B, 70B, and 405B on MultiWOZ 2.4, API-Bank, and BFCL V3 in a zero-shot setting, and claim that the larger models outperform GPT-4o across all three benchmarks, demonstrating the feasibility of a single open-source model for both conversation and tool use.

Significance. If the central claim were fully supported, the paper would be a valuable contribution: it would show that a single open-source model can jointly handle multi-turn task completion and complex API calling, and the released code, weights, datasets, and training configurations would enable reproduction and follow-up work. The CRA dataset is a novel resource, and the ablations in Table 5 provide useful evidence that each training component contributes. However, the current evidence is weakened by an overbroad headline claim, a lack of contamination analysis between training and evaluation data, missing variance estimates, incomplete baselines on API-Bank, and the provisional status of the 405B results.

major comments (4)
  1. [Abstract, §1, and Table 2] The abstract and introduction claim that CoALM 70B and 405B outperform GPT-4o 'across all three benchmarks,' but Table 2 shows GPT-4o achieves a MultiWOZ Success of 75.5 versus 69.4 for CoALM 70B and 66.7 for CoALM 405B; CoALM surpasses GPT-4o only on JGA. This overclaim should be corrected, and the paper should state precisely which metrics are improved.
  2. [§4.1, Table 1, and §5.1] The zero-shot definition in §5.1 (no evaluation benchmark train split) is insufficient to rule out contamination. CoALM-IT includes Hammer and ToolACE training data, both of which were constructed for BFCL-style function calling, yet no deduplication, n-gram overlap, or embedding-based analysis is reported between these corpora and the BFCL V3 or API-Bank test sets. Without such an analysis, the claimed zero-shot gains over GPT-4o on the LA benchmarks may be inflated by memorization, directly affecting the central feasibility claim.
  3. [Tables 2–4] All results are single runs without error bars or significance tests, and some margins are small (e.g., BFCL V3 Overall Acc: CoALM 70B 60.49 vs GPT-4o 59.83). Additionally, the CoALM 405B results are from a checkpoint after one epoch while training is still ongoing, so the reported numbers are provisional. The authors should provide multiple seeds or confidence intervals, and either finalize the 405B training or clearly label the results as preliminary.
  4. [Table 3 and §5.3] API-Bank results in Table 3 do not include GPT-4o or GPT-4o-mini as baselines, so the claim that the larger CoALM models outperform GPT-4o on 'function calling benchmarks' is only supported for BFCL V3. The authors should add a GPT-4o baseline on API-Bank or restrict the claim to BFCL V3.
minor comments (5)
  1. [§4.1 vs Table 1] Section 4.1 states that the SNIPS transformation yields 24,542 samples, but Table 1 reports 13,028 samples for SNIPS; the table total of 311,583 matches the 13,028 figure, so the text appears to contain a numerical error.
  2. [Table 3] Table 3 is difficult to read because several cells concatenate numbers without clear column separators (e.g., the CoALM 70B row shows '92.783.294.582.7'); please reformat the table.
  3. [Related Work] There are minor typos, including 'simple simple reasoning' and inconsistent spelling 'GPT4o'/'GPT-4o'.
  4. [Appendix D] The human validation of 100 CRA dialogues reports a 9% error rate, but no details on inter-annotator agreement or the selection of the evaluator are given; a brief description would help.
  5. [Limitations] The limitations section is candid about the Llama-only family and computational costs, but it does not address the contamination risk discussed above; adding a discussion would strengthen the paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation; the empirical claims rest on external benchmarks, with a contamination risk that is a validity concern but not a demonstrated circular reduction.

full rationale

This paper is an empirical fine-tuning study rather than a formal derivation. It constructs CoALM-IT from SNIPS, Hammer, ToolACE, and a GPT-4o-generated CRA corpus, fine-tunes Llama models, and evaluates on MultiWOZ 2.4, API-Bank, and BFCL V3. There is no fitted parameter renamed as a prediction, no uniqueness theorem imported from the authors' prior work, and no self-citation chain that forces the central claim. The central comparison is against external third-party baselines, including GPT-4o, so the headline result is not equivalent to the training objective by construction. The most serious concern is that CoALM-IT includes Hammer and ToolACE, which are function-calling corpora oriented toward BFCL-style evaluation, while the paper's 'zero-shot' definition in footnote 3 only excludes the official train splits of the evaluation benchmarks. The absence of a contamination analysis is a real data-hygiene threat, but the paper provides no evidence that BFCL V3 or API-Bank test cases actually appear in CoALM-IT, so this is not a demonstrated circular reduction. Separately, the abstract's claim that CoALM outperforms GPT-4o across all three benchmarks is inconsistent with Table 2, where GPT-4o attains higher MultiWOZ Success than both CoALM 70B and CoALM 405B; that is a reporting and consistency issue, not circularity. Accordingly, no circular steps meeting the required evidentiary standard are identified.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the quality and separation of the CoALM-IT mixture. Free parameters are standard fine-tuning hyperparameters chosen by hand. The key domain assumptions are benchmark validity, GPT-4o trace quality, and training/evaluation disjointness, the last of which is not verified.

free parameters (5)
  • LoRA rank r = 16
    Chosen by hand for all models; not tuned per benchmark, but affects capacity.
  • LoRA alpha = 32
    Scaling factor paired with rank; hand-picked.
  • Learning rate = 1e-4
    Standard choice for LoRA fine-tuning; not optimized.
  • Number of epochs = 3
    Fixed for all sizes except 405B which is at one epoch; not justified by validation.
  • Global batch size = 8
    Set by compute constraints rather than tuned.
assumptions (4)
  • domain assumption The three benchmarks (MultiWOZ 2.4, API-Bank, BFCL V3) are valid measures of conversational and agentic ability.
    The paper's contributions are evaluated solely on these benchmarks, so their validity is load-bearing.
  • domain assumption The SGD-derived CRA data generated by GPT-4o is sufficiently clean and representative.
    Human validation on 100 dialogues found a 9% error rate; errors are acknowledged in restaurant domains (Appendix D), yet the full 82K set is used without filtering errors.
  • domain assumption No overlap exists between CoALM-IT training data and the test sets of the evaluation benchmarks.
    The zero-shot claim rests on this; the paper defines zero-shot as not using the evaluation train-sets but does not analyze dataset overlap or paraphrasing.
  • domain assumption LoRA/QLoRA fine-tuning on Llama retains enough base knowledge to support the new tasks.
    The models are only adapter-trained on top of Llama; any catastrophic forgetting or capacity loss would distort results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can a Single Model Master Both Multi-turn Conversations and Tool Use? CoALM: A Unified Conversational Agentic Language Model." pith.science (2026). https://pith.science/paper/7ZLOKH42

@misc{pith2026250208820,
  author       = {Pith},
  title        = {Pith review of: Can a Single Model Master Both Multi-turn Conversations and Tool Use? CoALM: A Unified Conversational Agentic Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7ZLOKH42}},
  note         = {Machine review of arXiv:2502.08820}
}
read the original abstract

Large Language Models (LLMs) with API-calling capabilities enabled building effective Language Agents (LA), while also revolutionizing the conventional task-oriented dialogue (TOD) paradigm. However, current approaches face a critical dilemma: TOD systems are often trained on a limited set of target APIs, requiring new data to maintain their quality when interfacing with new services, while LAs are not trained to maintain user intent over multi-turn conversations. Because both robust multi-turn management and advanced function calling are crucial for effective conversational agents, we evaluate these skills on three popular benchmarks: MultiWOZ 2.4 (TOD), BFCL V3 (LA), and API-Bank (LA), and our analyses reveal that specialized approaches excel in one domain but underperform in the other. To bridge this chasm, we introduce CoALM (Conversational Agentic Language Model), a unified approach that integrates both conversational and agentic capabilities. We created CoALM-IT, a carefully constructed multi-task dataset that interleave multi-turn ReAct reasoning with complex API usage. Using CoALM-IT, we train three models CoALM 8B, CoALM 70B, and CoALM 405B, which outperform top domain-specific models, including GPT-4o, across all three benchmarks. This demonstrates the feasibility of a single model approach for both TOD and LA, setting a new standard for conversational agents.

Figures

Figures reproduced from arXiv: 2502.08820 by the authors.

Figure 1
Figure 1. Unifying Capabilities of TOD Systems and LAs. TOD systems excel in multi-turn conversations and task completion but lack advanced API capabilities, while LA handle APIs well but struggle with coherent multi-turn dialogue. (Schick et al., 2024) while transforming modular di￾alogue systems into end-to-end solutions (Hudecek ˇ and Dusek, 2023). Despite sharing LLM founda￾tions, they are typically focused and analyzed s… view at source ↗
Figure 2
Figure 2. Overview of the CoALM Pipeline. This figure illustrates our dataset generation and fine-tuning framework. The top three rows depict the data transformation processes, along with a corresponding sample shown on the right. In each training sample, green text highlights the input components of the instruction sample, while purple text indicates the target outputs optimized during fine-tuning. For detailed examples, ref… view at source ↗
Figure 3
Figure 3. Error Analysis of Function-Calling Results. Illustrated performance comparison on function calling benchmarks API-Bank L1 (top) and BFCL V3 parallel function call (bottom). Results demonstrate CoALM’s consistent performance compared to other baselines [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: SNIPS fine-tuning sample example [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]
Figure 5
Figure 5. Figure 5: Hammer fine-tuning sample example [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: SGD fine-tuning sample example targeting function optimization. [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: SGD fine-tuning sample example targeting system response optimization. [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DICE-BENCH: Evaluating the Tool-Use Capabilities of Large Language Models in Multi-Round, Multi-Party Dialogues

    cs.CL 2025-06 conditional novelty 7.0 of 10

    A new benchmark and metric show that large language models still struggle to call tools when the needed details are scattered across multi-party, multi-round group dialogues.

  2. Lost in Execution: On the Multilingual Robustness of Tool Calling in Large Language Models

    cs.CL 2026-01 conditional novelty 6.0 of 10

    A new benchmark shows multilingual tool-calling errors in LLMs are mostly parameter-language mismatches at the execution boundary, not failures of intent understanding.

  3. MUA-RL: Multi-turn User-interacting Agent Reinforcement Learning for agentic tool use

    cs.AI 2025-08 conditional novelty 6.0 of 10

    MUA-RL adds an LLM-simulated user to the RL rollout loop for multi-turn tool use, improving small Qwen3 models on TAU2, BFCL-V3 Multi Turn, and ACEBench Agent.

Reference graph

Works this paper leans on

17 extracted references · 15 canonical work pages · cited by 3 Pith papers

  1. [1]

    Make one or more function/tool calls to meet the request based on the question

    work page

  2. [2]

    Advances in Neural Information Processing Systems, 36

    Toolformer: Language models can teach them- selves to use tools. Advances in Neural Information Processing Systems, 36. Yixuan Su, Lei Shu, Elman Mansimov, Arshit Gupta, Deng Cai, Yi-An Lai, and Yi Zhang. 2022. Multi-task pre-training for plug-and-play task-oriented dialogue system. In Proceedings of the 60th Annual Meeting of the Association for Computat...

    arXiv 2022

  3. [3]

    name": "LxOm64zLyg

    If the given question lacks the parameters required by the function, also point it out. [END OF TASK INSTRUCTION] [BEGIN OF A V AILABLE TOOLS] ["name": "LxOm64zLyg", "description": "Gets hourly weather forecast information for given geographical coordinates using the RapidAPI service.", "parameters": "TDpjPd": "description": "The latitude of the geographi...

    work page

  4. [5]

    If none of the function can be used, point it out and refuse to answer

    work page

  5. [7]

    Analyze the user’s query within the given dialogue context to identify their intent and relevant details

    work page

  6. [8]

    Make a function/tool call and provide the necessary arguments to meet the request based on the user query

    work page

  7. [9]

    [END OF TASK INSTRUCTION] [BEGIN OF A V AILABLE TOOLS]

    Formulate a natural and coherent response, guiding the conversation towards resolving the user’s request. [END OF TASK INSTRUCTION] [BEGIN OF A V AILABLE TOOLS]

    work page

  8. [10]

    FindBus(from_location, to_location, leaving_date)

    work page

Show all 17 references
  1. [11]

    ReserveCar(pickup_location, pickup_date, pickup_time, dropoff_date, type)

  2. [14]

    CheckBalance(account_type)

  3. [15]

    TransferMoney(account_type, amount, recipient_account_name)

  4. [50]

    FindProvider(city, type)

  5. [51]

    GetRide(destination, number_of_riders, shared_ride) [END OF A V AILABLE TOOLS] [BEGIN OF FORMAT INSTRUCTION] The output MUST strictly adhere to the following structured text format. Example Output API Call Format: function_name(argument1=value1, argument2=value2, ...) [END OF ...

    2023

  6. [52]

    FindAttractions(location)

  7. [53]

    Meditate to Sounds of Nature

    GetWeather(city) # Output Format: - If an API Call is Needed: User: [User Input] Thought1: [I need to call an API] API Name: [API Call Name: CheckBalance(), TransferMoney(),..., FindAttractions(), GetWeather()] API Input: [The input parameters for the API] API Result: [API out...

  8. [2022]

    mango”, portion_amount=8.0, portion_unit=“piece

    MultiWOZ 2.4: A multi-domain task-oriented dialogue dataset with essential annotation corrections to improve state tracking evaluation. In Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 351–360, Edinburgh, UK. Association ...

    2002

  9. [2024]

    Preprint, arXiv:2402.10466

    Large language models as zero-shot dia- logue state tracker through function calling. Preprint, arXiv:2402.10466. Qiqiang Lin, Muning Wen, Qiuying Peng, Guanyu Nie, Junwei Liao, Jun Wang, Xiaoyun Mo, Ji- amu Zhou, Cheng Cheng, Yin Zhao, and Weinan Zhang. 2024. Hammer: Robust f...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.