Pith. sign in

REVIEW 3 major objections 4 minor 35 references

UserToolBench: A User-Profile-Hidden Benchmark for Personalized Decision Making in Tool-Use LLMs

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read UserToolBench claims that no tested tool-use LLM can reliably recover a profile-conditioned decision trajectory when the explicit user profile is hidden, with the best model scoring 49.36% exact accuracy.

desk verdict A genuinely new profile-hidden evaluation protocol for personalized tool-use LLMs, with an honest but fixable blind spot around the undisclosed reference generator. read the letter →

arxiv 2608.10042 v1 pith:T5N423WT submitted 2026-08-10 cs.LG cs.AI

Xuexiong Yin , Zechuan Chen , Yongsen Zheng , Yuxiang Zhang , Jingyuan Yang , Bin Wang , Yubin Wang , Keze Wang This is my paper · ORCID
classification cs.LGcs.AI
keywords personalizeddecisionmakingtool-useLLMsprofile-hiddenevaluationpreferenceinferenceclarificationbehaviortool-calltrajectoriesmulti-toolcoordinationlong-horizonconsistency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces UserToolBench, a benchmark that asks whether tool-using large language models can act as personalized delegates: infer a persistent user's latent preferences from interaction history, judge when a request is too incomplete to act on, and emit tool-call trajectories that match what a profile-aware planner would do. The benchmark's signature move is profile-hidden evaluation: reference trajectories are built with the explicit user profile visible and human-verified, while the tested model sees only history, the current request, and tool schemas. Across nine models the best exact-trajectory accuracy is 49.36%, and relaxed task-completion accuracy runs far higher, so models routinely complete the generic task while missing the user-specific decision. The paper's conclusion is that personalization evaluation should measure whether an assistant makes the right executable decision for the represented user, not whether its output sounds user-specific.

What carries the argument

The load-bearing mechanism is the profile-hidden asymmetry: reference trajectories $y^\star$ are generated as $y^\star = f_{\text{ref}}(p, h, q, T)$ with the persistent profile $p$ visible to the generator and validators, while the evaluated model produces $\hat{y} = f_\theta(h, q, T)$ with $p$ withheld, forcing preference recovery from interaction history alone. Rounding out the machinery are a milestone-based synthesis pipeline (a persona-conditioned user simulator that deliberately omits decision-critical constraints, a profile-aware planner that fills them from stable preferences, and human verification of tool choice, arguments, clarification behavior, and dependencies) plus a dual scoring scheme (exact trajectory matching and relaxed task-completion accuracy) and failure diagnostics that separate harmless ordering variation from wrong tools, missing calls, violated constraints, and broken dependencies.

What would settle it

Ask the users behind the sanitized profiles (or blind judges applying the same preference rules) to rate whether the model predictions that pass relaxed accuracy but fail exact matching are genuinely misaligned with the profile; if a large share are judged equally preference-aligned, the claim that exact-match failures are mostly substantive decision deviations would collapse. A complementary check: regenerate the reference trajectories with a different underlying generator model and measure how much exact accuracy of the same evaluated models changes, since large shifts would show the benchmark tracks one planner's style rather than personalized decision quality.

Watch

Extended reading notes

Core claim

The paper's central claim is that personalized delegation is currently unsolved for tool-use LLMs, and that this failure is visible only when evaluation hides the user profile: with $p$ hidden, the best tested model reproduces the profile-conditioned reference trajectory just 49.36% of the time, while relaxed task-completion accuracy reaches up to 72.55% for the same models. The gap between the two metrics is the paper's key evidence that executable task completion does not imply personalized decision alignment. From failure diagnostics, the authors further argue that exact-match errors are mostly substantive rather than cosmetic, with sequence and dependency errors in 82.1-96.5% of failures, wrong-tool decisions in 48.2-90.3%, and user-constraint violations in 40.9-53.5%, and that multi-tool coordination, missing-constraint inference, and long-horizon behavioral consistency are the binding bottlenecks. A preliminary dynamic-preference split shows the same construction framework can represent preference updates, though models still track the latest applicable preference poorly.

Load-bearing premise

The benchmark treats its LLM-generated reference trajectories, built with the profile visible and screened by human verifiers, as the definition of the correct personalized decision, and nothing independently confirms those trajectories match what the real users would actually want.

Editorial extensions

If this is right

  • No tested LLM can yet act as a reliable personalized delegate: exact trajectory accuracy caps at 49.36%, so the capability should be treated as open rather than nearly solved.
  • Task completion is not a proxy for personalization: because relaxed accuracy runs 20-36 points above exact accuracy on frontier models, benchmarks that score only whether the job got done will systematically overstate personalized alignment.
  • Multi-tool delegation is the hardest regime (about 25.24% exact accuracy versus 51.30% for single-tool tasks), so personalization difficulty concentrates in sequential decision control, where small early errors propagate through later calls.
  • Longer interaction history does not by itself improve personalization: several models degrade in later trajectory thirds, indicating that selective retrieval and updating of user state is a distinct capability from accumulating context.
  • Missing-constraint handling is a separate skill from generic tool use: model rankings on lack-of-information tasks diverge from rankings on single-tool tasks, so benchmarks need dedicated underspecification probes.
  • A direct test of the benchmark's premise would show the same hidden profiles to blind judges and ask whether reference trajectories match stated preferences better than the models' relaxed-compatible alternatives; a high rate of ties would weaken the exact-match interpretation.
  • The single-reference scoring design suggests an equivalence-class variant: scoring against a set of preference-compatible reference paths, or against a learned preference-satisfaction oracle, would convert the benchmark from agreement-with-one-planner into closer-to-true utility measurement.
  • The profile-hidden protocol could transfer to other agentic settings such as web navigation, mobile-device control, and code generation, wherever a persistent user's stable constraints must be recovered from history; the paper's own dynamic-preference split points in this direction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the benchmark's premise would show the same hidden profiles to blind judges and ask whether reference trajectories match stated preferences better than the models' relaxed-compatible alternatives; a high rate of ties would weaken the exact-match interpretation.
  • The single-reference scoring design suggests an equivalence-class variant: scoring against a set of preference-compatible reference paths, or against a learned preference-satisfaction oracle, would convert the benchmark from agreement-with-one-planner into closer-to-true utility measurement.
  • The profile-hidden protocol could transfer to other agentic settings such as web navigation, mobile-device control, and code generation, wherever a persistent user's stable constraints must be recovered from history; the paper's own dynamic-preference split points in this direction.
  • Because the identity of the reference generator is not disclosed, published scores may bound what current models achieve relative to that particular planner's style rather than the ceiling of personalized delegation; disclosing the generator would let readers calibrate the absolute numbers.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces UserToolBench, a benchmark for evaluating personalized decision-making in tool-use LLMs under a profile-hidden protocol. Reference trajectories are constructed with access to persistent user profiles, a persona-conditioned user simulator, and an unspecified planner; evaluated models receive only interaction history, the current request, and tool schemas. The benchmark includes 10 profiles, 36 toolsets, 1,065 turns, 170 unique tools, and 799 task instances. Experiments with nine models report best average exact accuracy of 49.36% and a large gap between relaxed task completion and exact trajectory matching, which the authors interpret as evidence that executable task completion does not imply personalized decision alignment. The paper also includes trajectory-diversity diagnostics, multi-label failure analysis, and a preliminary dynamic-preference split.

Significance. The benchmark addresses a genuine gap by jointly requiring persistent profiles, executable tool-call trajectories, long-horizon interaction, and history-based inference without explicit profile access. The profile-hidden protocol is a solid methodological contribution because it rules out direct profile copying, and the diversity audits for personas and trajectories support internal validity. The paper makes useful falsifiable claims: current frontier tool-use models do not reliably match profile-conditioned reference trajectories, and this inability is not explained by generic tool-use competence. The release of code and prompts is a concrete strength. The central risk is external validity of the LLM-generated reference trajectories; the headline accuracy numbers inherit this risk.

major comments (3)
  1. [3.2, Appendix E, Eq. (1)] The reference trajectory y* = f_ref(p, h, q, T) is generated by the unspecified 'planner component of the assistant pipeline' (Appendix E), and the human verification interface (Appendix E.1) checks internal consistency, tool validity, and persona consistency rather than agreement with an independently elicited expert decision or with the originating users' preferences. Because exact accuracy is the paper's central metric, the low absolute numbers in Table 4 may partly reflect agreement with this particular generator's planning conventions. The authors should disclose the generator model and version, add a profile-visible oracle condition, and validate a sample of references against independent human judgments with reported inter-annotator agreement.
  2. [3.4, Appendix D.2] The paper properly concedes in Section 3.4 that single-reference exact matching can penalize equally valid preference-compatible trajectories. It then argues from Appendix D that most mismatches are substantive rather than cosmetic. However, the manuscript does not state how the Slight/Major labels in Table 13 were assigned, by whom, or whether annotators agreed, and the multi-label analysis in Table 12 has the same gap. Without annotation protocol and agreement statistics, the diagnostic does not fully separate harmless variation from personalization failure. The authors should publish the labeling protocol, report inter-annotator agreement, or validate a sample of mismatches with independent evaluators.
  3. [4.2] All evaluated models are tested only in the profile-hidden condition; there is no profile-visible oracle or upper-bound condition. Such a condition would show whether the reference trajectories are reproducible when the explicit profile is provided and would calibrate the claim that the 49.36% average reflects difficulty of latent preference inference rather than ambiguity or idiosyncrasy of the single reference path. This is a small experiment that is clearly within the scope of the paper and should be added.
minor comments (4)
  1. [4.1, Tables 4-5] Several numerical entries in Tables 4 and 5 are typeset without column separators (e.g., '13.4842.22' and '78.5366.80'), making the results difficult to read; the tables should be re-typeset.
  2. [Title, Appendix A] The title on the first page renders 'UserToolBench' as 'USERTOOLBENCH' and 'profile-hidden' as 'HIDDENBENCHMARK' with a missing space; also, Appendix A refers to 'GPT prediction' without specifying which of the evaluated GPT models produced the shown trajectory.
  3. [Appendix D.1] The multi-label failure percentages in Table 12 are described as jointly assigned, but the manuscript does not define how the labels were derived from failed trajectories; a short description of the labeling procedure would improve reproducibility.
  4. [3.4] The exact-accuracy definition for lack-of-information tasks credits both clarification and inference, but the evaluation protocol does not state how a partial clarification (e.g., asking for one of several missing constraints) is scored; please clarify this scoring rule.

Circularity Check

1 steps flagged · score 2.0 of 10

No fitted-parameter or self-citation circularity; only a mild, acknowledged self-definitional residue from LLM-generated reference trajectories.

  1. self definitional [Section 3.2 (Trajectory Collection) and Section 3.4 (Evaluation Protocol), Eq. (1)]
    "The reference trajectory generator is implemented as the planner component of the assistant pipeline. It is also given access to the same profile and the available tool schemas, and is required to produce executable tool-call plans rather than free-form conversational replies whenever external actions are needed. ... Therefore, exact tool-call correctness is a high-precision operational proxy for personalized decision alignment in our benchmark."

    The ground-truth trajectory y* is defined by Eq. (1) as f_ref(p,h,q,T), where f_ref is itself an LLM planner. Exact Acc. is then computed as agreement between the evaluated model's trajectory and y*. Consequently, the headline result that no model exceeds 49.36% exact accuracy is, by construction, a statement about how well other LLMs reproduce this particular generator's planning behavior, not about independently established user preferences. The paper does not fit any parameter from this target, and the hidden-profile split prevents answer copying, so this is not a statistical forcing; it is a mild definitional dependence. The paper's own single-reference caveat in Section 3.4 and the 'controlled evidence' caveat in Section 5 are honest mitigations.

full rationale

UserToolBench is a benchmark construction, not a fitted model. The reference trajectories are generated once by an LLM planner with profile access and human verification; the nine evaluated models are scored by exact and relaxed matching to those fixed references. No parameter is fitted from the model outputs, no prediction is derived from a fitted input, and no load-bearing claim rests on a self-citation. The profile-hidden split prevents direct copying of the persona. The one residual is self-definitional in a weak sense: the target y* = f_ref(p,h,q,T) is itself an LLM planner's output, so exact accuracy measures agreement with that generator's planning behavior. The paper explicitly concedes the single-reference limitation ('Because most instances contain a single verified reference, exact matching can penalize an alternative trajectory that is also preference-compatible') and limits interpretation ('benchmark performance should be interpreted as controlled evidence about personalized tool decisions, not as a complete estimate of deployment performance'). Because the paper does not claim external ground truth and provides relaxed and severity diagnostics, this is an acknowledged validity limitation rather than a circular derivation. Score 2 reflects no significant circularity, with only this mild self-definitional residue.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The benchmark introduces no fitted parameters or new entities. Its claims rest on four domain assumptions, the most fragile being that LLM-generated references are valid ground truth for personalized decisions.

assumptions (4)
  • domain assumption Stable user preferences are recoverable from interaction history.
    Profile-hidden evaluation (Section 3.4) depends on the premise that history contains enough signal to infer the user's constraints; if preferences are not stable or not expressed, the task is underdetermined.
  • domain assumption LLM-generated reference trajectories, after human verification, are valid ground truth for personalized decisions.
    Reference generator is an LLM planner (Section 3.2) and the generator model is not identified; the benchmark's target decisions are therefore defined by that generator.
  • domain assumption Exact match to a single reference trajectory is a high-precision proxy for personalized alignment.
    The paper itself states that exact match can penalize alternative preference-compatible trajectories (Section 3.4) and treats multi-reference evaluation as an open problem.
  • domain assumption Sanitized LLM abstraction of real traces preserves decision-relevant preferences.
    Persona profiles are built via LLM-assisted abstraction from privacy-sanitized traces (Section 3.2, Appendix B); no fidelity check against original traces is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of UserToolBench: A User-Profile-Hidden Benchmark for Personalized Decision Making in Tool-Use LLMs." pith.science (2026). https://pith.science/paper/T5N423WT

@misc{pith2026260810042,
  author       = {Pith},
  title        = {Pith review of: UserToolBench: A User-Profile-Hidden Benchmark for Personalized Decision Making in Tool-Use LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T5N423WT}},
  note         = {Machine review of arXiv:2608.10042}
}
read the original abstract

Tool-use LLMs are increasingly asked to act on users' behalf, but existing benchmarks usually focus on profile recall, style imitation, generic tool use, or response-level personalization. We introduce UserToolBench , a benchmark for personalized decision making in tool-use LLMs. UserToolBench tests whether a model can infer latent user preferences from interaction history, recognize when clarification is needed, and produce user-aligned tool-call trajectories under incomplete information. The benchmark is built from privacy-sanitized real interaction traces and combines structured persona profiles, public API-style tool ecosystems, and long-horizon multi-turn trajectories. It includes 10 user profiles, 36 tool sets, 1,065 turns, 170 unique tools, and evaluation-focused task types covering lack-of-information, single-tool, and multi-tool settings. Experiments with strong tool-use LLMs show that current models still have difficulty with personalized delegation. Multi-tool coordination, missing-constraint inference, and long-horizon behavioral consistency remain major bottlenecks. These results suggest that personalization evaluation should move beyond asking whether outputs sound user-specific and instead ask whether LLMs make correct decisions for the users they represent.

Figures

Figures reproduced from arXiv: 2608.10042 by the authors.

Figure 1
Figure 1. Overview of the UserToolBench construction pipeline. A persistent user profile is visible to both the user [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Dataset composition of UserToolBench. Left: major field groups covered by structured persona profiles. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Heatmap visualization of the complete pairwise full-profile similarity. Left: token Jaccard similarity. Right: [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 23 canonical work pages

  1. [1]

    Tuesday Retail Notes

    "Tuesday Retail Notes" LinkedIn column: publish every two weeks, with each piece under 500 words, using anonymized operating data to comment on industry trends. Example topics include "Why I no longer look at GMV and instead focus on sales per square foot" and "How a 12-person team manages three stores and one online shop." Reference trajectory.The refere...

    work page arXiv 2024

  2. [3]

    Personalized language modeling from personalized human feedback.arXiv preprint arXiv:2402.05133,

    Xinyu Li, Ruiyang Zhou, Zachary C Lipton, and Liu Leqi. Personalized language modeling from personalized human feedback.arXiv preprint arXiv:2402.05133,

    arXiv

  3. [4]

    Aligning LLMs by Predicting Preferences from User Writing Samples

    Stéphane Aroca-Ouellette, Natalie Mackraz, Barry-John Theobald, and Katherine Metcalf. Aligning llms by predicting preferences from user writing samples.arXiv preprint arXiv:2505.23815,

    work page Pith review arXiv

  4. [5]

    API-bank: A comprehensive benchmark for tool-augmented LLMs

    Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. API-bank: A comprehensive benchmark for tool-augmented LLMs. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3102–3116, Singapore, December

    work page 2023

  5. [7]

    Benchmarking llm tool-use in the wild.arXiv preprint arXiv:2604.06185,

    Peijie Yu, Wei Liu, Yifan Yang, Jinjian Li, Zelong Zhang, Xiao Feng, and Feng Zhang. Benchmarking llm tool-use in the wild.arXiv preprint arXiv:2604.06185,

    arXiv

  6. [8]

    10 UserToolBench Zidi Xiu, David Q Sun, Kevin Cheng, Maitrik Patel, Yizhe Zhang, Jiarui Lu, Omar Attia, Raviteja Vemulapalli, Oncel Tuzel, Meng Cao, et al

    URLhttps://arxiv.org/abs/2406.12045. 10 UserToolBench Zidi Xiu, David Q Sun, Kevin Cheng, Maitrik Patel, Yizhe Zhang, Jiarui Lu, Omar Attia, Raviteja Vemulapalli, Oncel Tuzel, Meng Cao, et al. Astra-bench: Evaluating tool-use agent reasoning and action planning with personal user context.arXiv preprint arXiv:2603.01357,

    arXiv

  7. [10]

    Advancing and benchmarking personalized tool invocation for llms.arXiv preprint arXiv:2505.04072,

    Xu Huang, Yuefeng Huang, Weiwen Liu, Xingshan Zeng, Yasheng Wang, Ruiming Tang, Hong Xie, and Defu Lian. Advancing and benchmarking personalized tool invocation for llms.arXiv preprint arXiv:2505.04072,

    arXiv

  8. [11]

    Tool- spectrum: Towards personalized tool utilization for large language models

    Zihao Cheng, Hongru Wang, Zeming Liu, Yuhang Guo, Yuanfang Guo, Yunhong Wang, and Haifeng Wang. Tool- spectrum: Towards personalized tool utilization for large language models. InFindings of the Association for Computational Linguistics: ACL 2025, pages 20679–20699,

    work page 2025

Show all 35 references
  1. [12]

    doi:10.18653/v1/2026.acl-long.370

    Association for Computational Linguistics. doi:10.18653/v1/2026.acl-long.370. URL https://aclanthology. org/2026.acl-long.370/. Hongru Cai, Yongqi Li, Wenjie Wang, Fengbin Zhu, Xiaoyu Shen, Wenjie Li, and Tat-Seng Chua. Large language models empowered personalized web agents. ...

    2026 doi

  2. [13]

    Fingertip 20k: A benchmark for proactive and personalized mobile llm agents.arXiv preprint arXiv:2507.21071,

    Qinglong Yang, Haoming Li, Haotian Zhao, Xiaokai Yan, Jingtao Ding, Fengli Xu, and Yong Li. Fingertip 20k: A benchmark for proactive and personalized mobile llm agents.arXiv preprint arXiv:2507.21071,

  3. [14]

    Persona2web: Benchmarking personalized web agents for contextual reasoning with user history.arXiv preprint arXiv:2602.17003, 2026a

    Serin Kim, Sangam Lee, and Dongha Lee. Persona2web: Benchmarking personalized web agents for contextual reasoning with user history.arXiv preprint arXiv:2602.17003, 2026a. Sunghwan Kim, Ryang Heo, Yongsik Seo, Jinyoung Yeo, and Dongha Lee. Agenticshop: Benchmarking agentic pro...

    2026 arXiv

  4. [15]

    Me-agent: A personalized mobile agent with two-level user habit learning for enhanced interaction.arXiv preprint arXiv:2601.20162, 2026a

    Shuoxin Wang, Chang Liu, Gowen Loo, Lifan Zheng, Kaiwen Wei, Xinyi Zeng, Jingyuan Zhang, and Yu Tian. Me-agent: A personalized mobile agent with two-level user habit learning for enhanced interaction.arXiv preprint arXiv:2601.20162, 2026a. Harmanpreet Singh, Nikhil Verma, Yixi...

    2024

  5. [16]

    Valuepilot: A two-phase framework for value-driven decision-making.arXiv preprint arXiv:2503.04569,

    Yitong Luo, Hou Hei Lam, Ziang Chen, Zhenliang Zhang, and Xue Feng. Valuepilot: A two-phase framework for value-driven decision-making.arXiv preprint arXiv:2503.04569,

    arXiv

  6. [17]

    Shopsimulator: Evaluating and exploring rl-driven llm agent for shopping assistants.arXiv preprint arXiv:2601.18225, 2026b

    Pei Wang, Yanan Wu, Xiaoshuai Song, Weixun Wang, Gengru Chen, Zhongwen Li, Kezhong Yan, Ken Deng, Qi Liu, Shuaibing Zhao, et al. Shopsimulator: Evaluating and exploring rl-driven llm agent for shopping assistants.arXiv preprint arXiv:2601.18225, 2026b. Jiho Kim, Junseong Choi,...

  7. [18]

    PersonaLLM: Investigating the abil- ity of large language models to express personality traits

    11 UserToolBench Hang Jiang, Xiajie Zhang, Xubo Cao, Cynthia Breazeal, Deb Roy, and Jad Kabbara. PersonaLLM: Investigating the abil- ity of large language models to express personality traits. InFindings of the Association for Computational Linguistics: NAACL 2024, pages 3605–...

    2024

  8. [19]

    URLhttps://aclanthology.org/2024.findings-naacl.229/

    doi:10.18653/v1/2024.findings- naacl.229. URLhttps://aclanthology.org/2024.findings-naacl.229/. Moonshot AI. Kimi K2.6.https://www.moonshot.ai/,

    2024 doi

  9. [20]

    Accessed: 2026-05-25. OpenAI. Introducing GPT-5.4. https://openai.com/index/introducing-gpt-5-4/ , March

    2026

  10. [21]

    Qwen Team

    Accessed: 2026-05-25. Qwen Team. Qwen3.6-Plus: Towards real world agents. https://qwen.ai/blog?id=qwen3.6, April

    2026

  11. [22]

    DeepSeek-AI

    Accessed: 2026-05-25. DeepSeek-AI. DeepSeek V4 Preview Release. https://api-docs.deepseek.com/news/news260424, April

    2026

  12. [23]

    Google DeepMind

    Accessed: 2026-05-25. Google DeepMind. Gemini 3.5 Flash Model Card. https://deepmind.google/models/model-cards/ gemini-3-5-flash/, May

    2026

  13. [24]

    Accessed: 2026-05-25. Z.ai. GLM-5: From vibe coding to agentic engineering. https://z.ai/blog/glm-5, February

    2026

  14. [25]

    Qiqiang Lin, Muning Wen, Qiuying Peng, Guanyu Nie, Junwei Liao, Jun Wang, Xiaoyun Mo, Jiamu Zhou, Cheng Cheng, Yin Zhao, et al

    Accessed: 2026-05-25. Qiqiang Lin, Muning Wen, Qiuying Peng, Guanyu Nie, Junwei Liao, Jun Wang, Xiaoyun Mo, Jiamu Zhou, Cheng Cheng, Yin Zhao, et al. Hammer: Robust function-calling for on-device language models via function masking. arXiv preprint arXiv:2410.04587,

    2026

  15. [26]

    Accessed: 2026-05-25

    URL https://huggingface.co/ MadeAgents/Hammer2.1-7b. Accessed: 2026-05-25. Weiwen Liu, Xu Huang, Xingshan Zeng, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Zhengying Liu, Yuanqing Yu, Zezhong WANG, et al. Toolace: Winning the points of llm function calling. InInternational Con...

    2026

  16. [27]

    co/Team-ACE/ToolACE-2.5-Llama-3.1-8B

    URL https://huggingface. co/Team-ACE/ToolACE-2.5-Llama-3.1-8B. Accessed: 2026-05-25. Watt AI. watt-tool-8B. Hugging Face model card, June

    2026

  17. [28]

    Accessed: 2026-05-25

    URL https://huggingface.co/watt-ai/ watt-tool-8B. Accessed: 2026-05-25. Wentao Shi, Mengqi Yuan, Junkang Wu, Qifan Wang, and Fuli Feng. Direct multi-turn preference optimization for language agents,

    2026

  18. [29]

    Han Luo and Guy Laban

    URLhttps://arxiv.org/abs/2406.14868. Han Luo and Guy Laban. SPASM: Stable persona-driven agent simulation for multi-turn dialogue generation. In Findings of the Association for Computational Linguistics: ACL 2026, pages 8455–8475. Association for Com- putational Linguistics,

    2026 arXiv

  19. [30]

    URL https://aclanthology.org/2026

    doi:10.18653/v1/2026.findings-acl.412. URL https://aclanthology.org/2026. findings-acl.412/. Yilin Jiang, Fei Tan, Xuanyu Yin, Leng Jing, and Aimin Zhou. HACHIMI: Scalable and controllable student persona generation via orchestrated agents. InFindings of the Association for Co...

    2026 doi

  20. [31]

    URL https://aclanthology.org/2026.findings-acl.1080/

    doi:10.18653/v1/2026.findings-acl.1080. URL https://aclanthology.org/2026.findings-acl.1080/. Gauri Kambhatla, Chantal Shaib, and Venkata S. Govindarajan. Measuring lexical diversity of synthetic data generated through fine-grained persona prompting. InFindings of the Associat...

    2026 doi

  21. [32]

    36Kr GreenLeaf retail rising star

    doi:10.18653/v1/2025.findings-emnlp.1146. URLhttps://aclanthology.org/2025.findings-emnlp.1146/. A Qualitative Case Study This appendix provides a qualitative case study illustrating how UserToolBench evaluates personalized decision making beyond generic tool execution. The ca...

    2025 doi

  22. [34]

    B.3 Quantitative Persona Diversity Audit We complement the field-coverage analysis with a quantitative audit of the ten selected profiles

    Profile-specific fields are summarized at the category level to avoid exposing sensitive or identifying details. B.3 Quantitative Persona Diversity Audit We complement the field-coverage analysis with a quantitative audit of the ten selected profiles. Demographic descriptors s...

    2026

  23. [35]

    Left: token Jaccard similarity

    Figure 3: Heatmap visualization of the complete pairwise full-profile similarity. Left: token Jaccard similarity. Right: TF–IDF cosine similarity. Lower values indicate greater diversity. Diagonal cells are masked because self-similarity is not part of the pairwise audit. Toge...

    2025

  24. [2023]

    doi:10.18653/v1/2023.emnlp-main.187

    Association for Computational Linguistics. doi:10.18653/v1/2023.emnlp-main.187. URL https://aclanthology.org/2023. emnlp-main.187/. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu ...

    2023 doi

  25. [2024]

    Know me, respond to me: Benchmarking llms for dynamic user profiling and personalized responses at scale.arXiv preprint arXiv:2504.14225,

    Bowen Jiang, Zhuoqun Hao, Young-Min Cho, Bryan Li, Yuan Yuan, Sihao Chen, Lyle Ungar, Camillo J Taylor, and Dan Roth. Know me, respond to me: Benchmarking llms for dynamic user profiling and personalized responses at scale.arXiv preprint arXiv:2504.14225,

  26. [2025]

    Personalens: A benchmark for personalization evaluation in conversational ai assistants

    Zheng Zhao, Clara Vania, Subhradeep Kayal, Naila Khan, Shay B Cohen, and Emine Yilmaz. Personalens: A benchmark for personalization evaluation in conversational ai assistants. InFindings of the Association for Computational Linguistics: ACL 2025, pages 18023–18055,

    2025

  27. [2026]

    Petoolllm: Towards personalized tool learning in large language models

    Qiancheng Xu, Yongqi Li, Heming Xia, Fan Liu, Min Yang, and Wenjie Li. Petoolllm: Towards personalized tool learning in large language models. InFindings of the Association for Computational Linguistics: ACL 2025, pages 21488–21503,

    2025

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.