REVIEW 3 major objections 5 minor 15 references
TOOL-ED: Enhancing Empathetic Response Generation with the Tool Calling Capability of LLM
T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Fine-tuning an LLM to selectively call a commonsense knowledge tool improves empathetic responses over always injecting the knowledge.
desk verdict First tool-calling framing for empathetic knowledge infusion, with a useful new dataset, but the headline gains are undermined by golden-response lookahead in dataset construction. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the EKTC tool-calling loop: the model outputs an "Action: EmotionKnowledgeBase" with an "Action Input" containing the dialogue context, receives five commonsense relations (xIntent, xNeed, xWant, xEffect, xReact from COMET, or the equivalent relations from CICERO) as an observation, and then generates the response. The dataset construction uses LLaMA3-70B as an Annotator to decide whether a tool call is appropriate and as a Reflector to filter calls whose results are judged highly relevant to the golden response, yielding a training set where only 26.46% of instances contain tool calls. The ReAct-style (action, observation) format carries the mechanism, and LoRA fine-tuning adapts the base models to the new task.
What would settle it
Force the fine-tuned tool-calling model at inference to always answer directly, or replace the tool observation with random text, and measure the same empathy metrics; if performance does not drop below the tool-calling version, the observed improvement is not caused by the tool itself.
Extended reading notes
Core claim
The paper establishes that a dialogue model can be trained end-to-end to invoke a commonsense knowledge base as a tool, and that this active, selective invocation outperforms both no-knowledge baselines and unconditioned knowledge injection. The evidence is that Qwen1.5-14B and Vicuna-7B fine-tuned on TOOL-ED with COMET or CICERO as the tool score higher across BLEU-1/2/3/4, BERTScore, ROUGE-1/2/L, and in A/B human and GPT-4 evaluations against the same models fine-tuned on the original ED dataset. The framework, called EKTC, thus demonstrates that the timing of knowledge access is itself a learnable skill that improves response quality.
Load-bearing premise
The training data only contains tool calls that a strong LLM judged, after seeing the golden response, to produce knowledge relevant to that response; the model must then learn to decide when to call the tool without ever seeing the golden response.
Editorial extensions
If this is right
- If correct, LLMs can benefit from model-initiated, selective use of commonsense knowledge rather than unconditional knowledge injection, reducing noise from irrelevant or redundant information.
- Knowledge bases become swappable tools: the same fine-tuned policy works with COMET or CICERO by replacing the tool API, suggesting generalizability across commonsense resources.
- The TOOL-ED dataset provides training traces for tool-calling empathetic dialogue, enabling other models to be adapted to this paradigm via supervised fine-tuning.
- The reflection-based filtering of tool calls against golden responses is a viable method for creating tool-use demonstrations from existing dialogue corpora without manual annotation.
Reading between the lines
- The approach could extend beyond knowledge bases to other auxiliary modules such as sentiment classifiers, persona retrievers, or safety filters, letting the model decide when to consult each one.
- Because the Reflector filters training calls using the golden response, the learned policy at inference may be more conservative than optimal; a reward model or test-time filtering could close that gap.
- The lower tool-calling ratio observed in the fine-tuned models suggests they learn to reserve tool calls for emotionally intense contexts, which might be even better exploited with an adjustable threshold.
- The same selective-tool framework could apply to other generation tasks where external knowledge is sometimes helpful and sometimes noisy, such as counseling, negotiation, or open-domain QA.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Emotional Knowledge Tool Calling (EKTC), a framework for empathetic response generation that wraps commonsense knowledge bases (COMET and CICERO) as callable tools and fine-tunes LLMs (Vicuna-7B, Qwen1.5-14B) with LoRA on a newly constructed dataset TOOL-ED. TOOL-ED is created from EmpatheticDialogues by using LLaMA3-70B as an Annotator to decide whether to call the tool and as a Reflector to filter tool outputs based on their relevance to the golden assistant response. The authors report automatic metrics (BLEU, BERTScore, ROUGE, Distinct) and human/LLM-based A/B evaluations showing improvements over models fine-tuned on the original ED dataset, and they run ablations including a 'noref' variant without the reflection step.
Significance. If the central claim holds—that selectively invoking an external commonsense knowledge base via tool calling improves empathetic response quality over always injecting knowledge or never using it—this would be a useful contribution to empathetic dialogue and tool-augmented LLMs. The paper provides a reproducible code repository, a plug-and-play tool design that works with two knowledge bases, and a new dataset. However, the experimental evidence for the central claim is weakened by a training-data confound and a lack of statistical rigor, as detailed in the major comments.
major comments (3)
- [§3.3, Appendix A.1] The training data construction uses the golden response in both the Annotator prompt ('your response: {assistant_response}' and asking whether the tool would be used to produce that response) and the Reflector (keeping only tool outputs judged relevant to that same golden response). This introduces a reference-conditioned selection bias: the model is trained on tool calls that are specifically beneficial for the ground-truth response, but at inference no reference is available and tool outputs are unfiltered. As a result, the reported gains of qwen_tool_comet and vicuna_tool_comet over qwen_lora and vicuna_lora (Tables 1, 2, 3) may stem from the curated training distribution rather than from the model's learned ability to call tools selectively. The 'noref' ablation in §4.5.2 removes only the Reflector, not the Annotator's access to the golden response, so it does not control for this confound. To support the central claim, the authors should either construct training data without reference lookahead (e.g., an Annotator that sees only dialogue context) or provide evidence that the inference-time tool-calling policy generalizes beyond the reference-filtered training distribution.
- [Tables 1-4] No variance, confidence intervals, or statistical significance tests are reported for any automatic metric. Many differences are small in absolute terms (e.g., BERTScore 0.8765 vs. 0.8760 for qwen models; ROUGE-L 0.1793 vs. 0.1696), so without repeated runs or paired significance tests the observed improvements cannot be distinguished from noise. The human evaluation (§4.4) uses only 100 randomly selected conversations and three annotators, with no inter-annotator agreement reported, and the GPT-4-based evaluation gives no sample size or agreement analysis. Quantitative claims of improvement should be accompanied by significance levels or at least standard deviations across multiple runs.
- [§4.3] The paper re-splits the ED dataset into training, validation, and test sets with a random 8:1:1 ratio. The original EmpatheticDialogues dataset has a predetermined split; using a different split makes the comparisons with prior published numeric results (e.g., CEM, KEMP, MIME in Table 1) invalid, because those baselines were evaluated on the original test partition. The authors should use the standard split or explicitly state that all baselines were re-evaluated on the same re-split; otherwise the claimed improvements over SOTA are not established.
minor comments (5)
- [Table 4] In Table 4, qwen_noref_comet reports BLEU-4 = 0.3130 and qwen_noref_cicero reports 0.3190; these are likely typos for 0.0313 and 0.0319, since other BLEU-4 values are around 0.03. Please correct them.
- [Table 1] Table 1 reports qwen_oneshot BLEU-3 as 0.1780, which is likely a typo (should be 0.0178). Also, the Dist-1/Dist-2 values for qwen_base (0.2322/0.1720) are implausible as Dist-1 is usually lower than or comparable to Dist-2; please verify these numbers.
- [§4.4] In the human evaluation description, 'Informativity (Inf.) meatures' should be 'measures'. Additionally, the paper should state whether the 100 samples are drawn from the test set and whether the three annotators were blinded to the model identities.
- [Abstract] The abstract contains a formatting error: 'EMPATHETICMPATHETIC DIALOGUE' should be 'EMPATHETIC DIALOGUE'.
- [§3.3] The paper reports a 26.46% tool-call insertion rate but does not specify the resulting dataset size or the filtering rate of the Reflector. This information is useful for assessing the dataset quality and the potential selection bias.
Circularity Check
No significant circularity: the empirical evaluation is self-contained, though the reference-conditioned dataset construction is a validity concern rather than a circular step.
full rationale
The paper makes an empirical claim that fine-tuning on TOOL-ED improves empathetic response generation, and it supports that claim with automatic metrics, human A/B testing, and an LLM judge against external baselines. No step in the method derives a prediction from its own input by construction: the COMET and CICERO observations are computed from the dialogue context by fixed external knowledge models, the LoRA objective in Eq. 5 is standard supervised next-token prediction, and the test-time tool-call decisions are made by the fine-tuned model rather than by the Annotator. The only self-citation, Zhang et al. (2024) for the tool-learning prompt template in Appendix A.2, is not load-bearing. One genuine concern is that the Annotator and Reflector prompts in Figures 6 and 7 include the golden assistant response, so the training tool-call labels are reference-conditioned; this is a leakage or validity confound for the claimed benefit of selective tool calling, but it is not a circular derivation because the evaluated responses are generated at inference without access to the golden response. The Limitation section notes only the absence of multi-tool support and does not address this leakage, which is a correctness risk rather than a circularity risk.
Assumptions & free parameters
free parameters (3)
- LoRA configuration =
not reported
- Tool-call insertion rate =
26.46% of training examples
- Dataset split ratio =
80/10/10
assumptions (4)
- domain assumption COMET's five commonsense relations (xIntent, xNeed, xWant, xEffect, xReact) provide useful information for empathetic response generation.
- domain assumption The assistant responses in the ED dataset are correct targets for training and evaluation.
- domain assumption LLaMA3-70B annotations of tool-use timing and relevance are reliable.
- ad hoc to paper Filtering tool outputs by relevance to the golden response does not introduce a train/test distribution shift.
invented entities (1)
-
EmotionKnowledgeBase tool
independent evidence
Cite this review
Pith. "Pith review of TOOL-ED: Enhancing Empathetic Response Generation with the Tool Calling Capability of LLM." pith.science (2026). https://pith.science/paper/PFK3VZD3
@misc{pith2026241203096,
author = {Pith},
title = {Pith review of: TOOL-ED: Enhancing Empathetic Response Generation with the Tool Calling Capability of LLM},
year = {2026},
howpublished = {\url{https://pith.science/paper/PFK3VZD3}},
note = {Machine review of arXiv:2412.03096}
}
read the original abstract
Empathetic conversation is a crucial characteristic in daily conversations between individuals. Nowadays, Large Language models (LLMs) have shown outstanding performance in generating empathetic responses. Knowledge bases like COMET can assist LLMs in mitigating illusions and enhancing the understanding of users' intentions and emotions. However, models remain heavily reliant on fixed knowledge bases and unrestricted incorporation of external knowledge can introduce noise. Tool learning is a flexible end-to-end approach that assists LLMs in handling complex problems. In this paper, we propose Emotional Knowledge Tool Calling (EKTC) framework, which encapsulates the commonsense knowledge bases as empathetic tools, enabling LLMs to integrate external knowledge flexibly through tool calling. In order to adapt the models to the new task, we construct a novel dataset TOOL-ED based on the EMPATHETICMPATHETIC DIALOGUE (ED) dataset. We validate EKTC on the ED dataset, and the experimental results demonstrate that our framework can enhance the ability of LLMs to generate empathetic responses effectively.
Figures
Figures from the paper (10 more)
Reference graph
Works this paper leans on
-
[1]
Understand the depth of the other person's emotions
-
[2]
Based on what you say in the conversation and emotional intensity of the user, please check if you have untilized this tool to answer the user's conversation, and tell me the reason. Your output should adhere to the format: ##Result one of [Yes, No] Annotator prompt Figure 6: Prompt for Anotator There are two roles in the conversation,including user and a...
-
[3]
Whether the expression is natural or not If the fluency of statement 1 is higher than that of statement 2, please output 1; If the fluency of statement 1 is lower than that of statement 2, output 0. You don't need to output the analysis process, just output 1 or 0. ##Result one of [1, 0] LLM-based evaluation prompt (Fluency) Figure 12: Prompt Template for...
-
[5]
arXiv preprint arXiv:2309.17452
Tora: A tool-integrated reasoning agent for mathematical problem solving. arXiv preprint arXiv:2309.17452. 9 Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirkpatrick, and Graham Neubig. 2021. Towards a unified view of parameter-efficient transfer learning. arXiv preprint arXiv:2110.04366. Cheng-Yu Hsieh, Si-An Chen, Chun-Liang Li, Yasuhisa Fujii, Al...
arXiv 2021
-
[7]
Multiview Contextual Commonsense Inference: A New Dataset and Task
Multiview contextual commonsense infer- ence: A new dataset and task. arXiv preprint arXiv:2210.02890. Zhengliang Shi, Shen Gao, Xiuyi Chen, Lingyong Yan, Haibo Shi, Dawei Yin, Zhumin Chen, Pengjie Ren, Suzan Verberne, and Zhaochun Ren. 2024. Learning to use tools via cooperative and interactive agents. arXiv preprint arXiv:2403.03031. Zhenqiao Song, Xiao...
-
[8]
Don’t lose yourself! empathetic response generation via explicit self-other awareness. In Find- ings of the Association for Computational Linguistics: ACL 2023, pages 13331–13344, Toronto, Canada. As- sociation for Computational Linguistics. Yuyue Zhao, Jiancan Wu, Xiang Wang, Wei Tang, Dingxian Wang, and Maarten de Rijke. 2024. Let me do it for you: Towa...
work page 2023
-
[11]
Whether the response to the other person's emotions is sincere and warm If the empathy level of statement 1 is higher than that of statement 2, output 1; If the empathy level of statement1 is weaker than statement2, output 0. You don't need to output the analysis process, just output 1 or 0 ##Result one of [1, 0] LLM-based evaluation prompt (Empathy) Figu...
-
[12]
The relevance of sentence content to the theme of dialogue history
Show all 15 references
-
[13]
The correlation between sentences, including key points of the dialogue, and accurate expression If the relevance of statement 1 is higher than that of statement 2, output 1; If the correlation of statement 1 is weaker than that of statement 2, output 0. You don't need to outp...
-
[14]
Coherence of language
-
[2019]
In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4762–4779, Flo- rence, Italy
COMET: Commonsense transformers for auto- matic knowledge graph construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4762–4779, Flo- rence, Italy. Association for Computational Linguis- tics. Tom B. Brown, Benjamin Man...
-
[2020]
In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY , USA
Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY , USA. Curran Associates Inc. Hua Cai, Xuli Shen, Qing Xu, Weilin Shen, Xiaomei Wang, Weifeng Ge, Xiaoqing Zheng, and Xia...
2023
-
[2022]
In Proceedings of the 29th International Conference on Computational Linguistics, pages 669– 683
Does gpt-3 generate empathetic dialogues? a novel in-context example selection method and au- tomatic evaluation metric for empathetic dialogue generation. In Proceedings of the 29th International Conference on Computational Linguistics, pages 669– 683. Jiwei Li, Michel Galley...
2016 arXiv
-
[2023]
See https://vicuna
Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3):6. Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. 2023. Eva: Exploring the l...
2023
-
[2024]
In ICASSP 2024-2024 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 10631–10635
Pecer: Empathetic response generation via dynamic personality extraction and contextual emo- tional reasoning. In ICASSP 2024-2024 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 10631–10635. IEEE. Hyungjoo Chae, Yongho Song, Kai Tzu ...
2024 arXiv
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.