Unified Agent: Managing Interactions across Devices

Runfa Blark Li; Shaoxiu Wei; Truong Nguyen; Xin Lin; Xinshuang Liu

REVIEW 3 major objections 5 minor 62 references

Unified Agent: Managing Interactions across Devices

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A compact carried state, organized as engagement evidence, stated facts, and the standing request, is the decisive factor for cross-device, cross-time agent requests.

desk verdict A genuinely new problem and a careful benchmark, but the headline gains over published baselines are likely input-feature artifacts; the honest effect size vs the full-record control is small, and the benchmark's targets encode the method's own state categories. read the letter →

arxiv 2608.05729 v1 pith:PLDMWETD submitted 2026-08-06 cs.AI cs.CLcs.CVcs.HC

Xinshuang Liu , Runfa Blark Li , Shaoxiu Wei , Xin Lin , Truong Nguyen This is my paper · ORCID

classification cs.AIcs.CLcs.CVcs.HC

keywords cross-deviceagentscarriedstateengagementevidencestandingrequestagentbenchmarkmultimodallargelanguagemodelsdevice-unspecifiedrequests

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that an agent serving one user across devices and moments fails not because the model is weak or the history is missing, but because the state it carries forward is badly organized. It proposes Unified Agent, which compresses each observation into three streams — engagement evidence, stated facts, and the standing request — and reads that compact state together with the current view to act. On the new UA-BENCH benchmark, this design beats adapted versions of four published agent designs by 0.194 to 0.408 points and the full-history control by 0.055, with the advantage holding across four multimodal large-language-model settings. The point of the paper is that the representation of carried state, not raw history or model strength, is the decisive factor for cross-device, cross-time requests.

What carries the argument

The carried state $S_t$ is the load-bearing mechanism: a compact tuple $(C_t, P_t, K_t, r_t)$ updated by folding each observation ($S_t = U(S_{t-1}, O_t)$) and read together with the current observation to act ($a_t = D(S_t, O_t)$). Engagement evidence accumulates per-device activity and attended-device cues as tallies; stated facts keep the first fact per device–topic pair; the standing request stores the latest actionable request. The state is what lets a later decision recover evidence after the original cue is no longer visible; the paper's claim is that this specific organization — not raw history, not decoded answers, not per-agent memory — is what carries the cross-device advantage.

What would settle it

Build a version of UA-BENCH where the five downstream targets come from an independent protocol (e.g., human annotators watching the full device-camera history) rather than from the construction script, and check whether Unified Agent's margin over the Full-context control persists; if it shrinks or vanishes, the reported advantage reflects the benchmark's construction rather than a general property of state design.

Watch

Extended reading notes

Core claim

Unified Agent is a stateful agent design for the setting where one agent serves one user across multiple devices over time, with observations arriving one device-camera view at a time and no replay of earlier moments. At each step it folds the current observation into a carried state $S_t = (C_t, P_t, K_t, r_t)$, where $C_t$ and $P_t$ store engagement evidence (activity and attended-device cues, with per-device tallies), $K_t$ stores the first stated fact for each device–topic (or topic-only) pair, and $r_t$ stores the latest standing request. The agent then acts from the updated state together with the current observation, producing the five downstream decisions: identify the engaged device, infer intent, recall relevant information, pick the responding device, and decide the next action. On UA-BENCH, 100 matched pairs (200 episodes) of rendered 3D interactions in which the later device-unspecified request and recall questions depend on swapped earlier engagement roles, Unified Agent achieves the highest overall score in the default GPT-5.6-Luna low-effort setting (0.668), leading the full-context control (0.613) and four adapted published designs (0.474, 0.474, 0.288, 0.260), with paired-bootstrap gaps whose 95% intervals are all above zero; it remains ahead in all four MLLM settings tested. The paper presents this as evidence that the way carried state is represented and used, not added system complexity or model strength, is what makes cross-device, cross-time requests answerable.

Load-bearing premise

If the benchmark's ground-truth rules already encode the very categories the method stores, the reported advantage may come from the benchmark's design rather than from the state design itself.

Editorial extensions

If this is right

Designers of cross-device agents should prioritize a compact, action-ready carried state over retaining the full interaction transcript; the paper shows a bounded state (11 times smaller than full context after 12 frames) yields higher overall accuracy.
The three-stream organization (engagement evidence, stated facts, standing request) sets the agenda for what an agent should persist: evidence for device attribution, topic-keyed facts, and the pending request, rather than raw frames or cached answers.
The advantage generalizes across MLLM families, capabilities, and reasoning efforts, so the state-design lesson transfers to different foundation models without fine-tuning or extra machinery.
Ablations predict which state element matters for which decision: engagement counts and pointing for identifying the engaged device, prior state for recall, and per-device ability clauses for routing.
Because the state is explicit, it can be inspected and selectively revised, which is a privacy-relevant property: the agent keeps a minimal record rather than the full interaction history.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: the same state-design principle may apply to single-device long-horizon tasks (e.g., a browser agent that must remember a fact from an earlier tab), where the deciding cue also passes before the later request; the compact three-stream state is a candidate generalization.
Beyond the paper: a testable extension is to replace UA-BENCH's scripted engagement cues with real behavioral signals (gaze, touch, app usage) and check whether the engagement-tally mechanism still resolves device-unspecified references; the current real-photo case uses hand-on-device contact as a stand-in.
Beyond the paper: the paper's benchmark construction defines ground truth directly from the same categories the method stores; an independent, human-annotated variant of UA-BENCH would clarify whether the measured advantage is a property of the state design or of the benchmark's construction rules.
Beyond the paper: if the state-design advantage holds in production, one implication is that agent memory systems should expose a small, revision-friendly state to the user (what the agent believes about engagement, facts, and pending requests) rather than a raw transcript, which would also make privacy review and data minimization more tractable.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this paper has a real new problem formulation and a carefully built benchmark, but the strongest numbers do not isolate what the authors claim. The 0.38–0.41 gaps over Mem0 and MM-DST are probably input-feature artifacts. Appendix D.2 says compared systems receive the shared perception record "restricted to the input fields their designs specify," and the paper never reports which fields each baseline actually got. If Mem0 and MM-DST never see the attended-device, activity-count, or pointing entries, their near-floor Eng and Inf scores tell us nothing about state representation. The only control that receives the complete record is Full context, and the gap there is 0.055—below the paper's own 0.10 effect-size criterion for the published-baseline family. So the robust state-design advantage is, on current evidence, a modest advantage over full raw history, not the large advantage the abstract implies.

What is genuinely good: the observe–think–act task decomposition for cross-device, cross-time interaction is useful and well-motivated. UA-BENCH's matched-pair construction—shared request/recall frames with role-swapped engagement—is a clever way to isolate carried evidence, and the statistical protocol (clustered bootstrap, Holm correction, aligned endpoints) is more careful than most of this literature. The ablations coherently show that engagement counts, pointing, prior state, and device abilities each contribute to their targeted decisions. The real-photo case study is thin—one prepared pair, no human agreement on the intent judge—but at least it is an attempt at external validation, and the authors admit the limitation.

The circularity concern is real but not fatal. The benchmark's ground truths are deterministic functions of the same three streams—engagement evidence, stated facts, standing request—that Unified Agent's state stores. That tests whether an agent explicitly built to track those streams can exploit them, which is a legitimate design question, but it inflates the apparent advantage over methods not built that way. The authors need to report exactly what each baseline received, add a full-record baseline that is not just raw history (or justify why raw history is the right control), and either pre-register or justify the 0.10 criterion. The intent judge being a single LLM without human agreement is a minor issue for relative comparison but should be disclosed.

The citation pattern looks fine; the related work is current and relevant. Code and data are promised but not yet available, so independent verification is impossible right now.

Bottom line: a serious paper worth engaging, and the benchmark could become a useful resource once code and data are out. I would send it to peer review, expecting major revision: fix the baseline-input reporting, temper the central claim, and address the benchmark's built-in advantage for the proposed state design. It is not a paradigm shift, but it is a solid step for agent-state research.

Referee Report

3 major / 5 minor

Summary. The paper proposes Unified Agent, a stateful agent for cross-device, cross-time interaction that carries a compact state composed of three streams: per-device engagement evidence, stated facts, and the standing request. The authors introduce UA-BENCH, a rendered 3D benchmark with 100 matched pairs (200 episodes) in which later device-unspecified requests depend on earlier interaction evidence, plus a small real-photo matched-pair case study. The agent is evaluated on five downstream decision tasks (Eng, Int, Inf, Rsp, Nxt) under four MLLM settings. The authors report that Unified Agent significantly outperforms adapted published baselines (gaps 0.194-0.408) and the full-context control (gap 0.055), with statistical significance assessed by clustered bootstrap and Holm correction, and that the advantage persists across MLLM families, capabilities, and reasoning efforts.

Significance. If the central claim holds, the paper makes a useful and simple contribution: a compact, inspectable carried-state design that is action-ready and bounded, in contrast to full-history retention. The strengths are the explicit state specification, the carefully controlled matched-pair evaluation, clustered bootstrap uncertainty quantification, Holm-adjusted multiple comparisons, the inclusion of strong state controls such as Full context, and the promise of public code and data. The paper also presents ablations separating the roles of engagement counts, pointing, prior state, and device abilities, which are informative. However, the significance is conditional on resolving two issues: the claimed large advantages over published baselines may be driven by asymmetric input features rather than state design, and the benchmark's ground truth is generated from the same categories that Unified Agent's state explicitly tracks, which may make the measured advantage partly construction-internal. The external evidence is limited to one author-prepared real-photo matched pair.

major comments (3)

[Section 5 / Appendix D.2 / Table 1] The manuscript states in Section 5 that all MLLM-based methods 'are evaluated using a shared per-frame perception output,' but Appendix D.2 clarifies that compared systems 'begin from the shared perception record, restricted to the input fields their designs specify.' The paper never reports the specific input-field subset given to each baseline. If Mem0 and MM-DST are restricted to text/fact fields and are not given the attended-device, activity-count, or pointing entries, their near-floor Eng scores (0.082 and 0.195) and Inf scores (0.395 and 0.025) are explained by missing inputs rather than by a failure of state design. The only comparison that truly holds inputs fixed is Unified Agent versus Full context, and that gap is 0.055 (CI +0.035 to +0.075), which is below the 0.10 effect-size threshold the paper itself uses for deciding that a published-baseline gap supports a conclusion. Please provide the exact fields each baseline received, justify each restriction as a faithful adaptation of the published design, and ideally re-run the baselines on the complete shared record. Without this, the headline gaps of 0.380-0.408 do not isolate the contribution of carried-state representation.
[Appendix B.3 / Appendix A.1] The benchmark's ground truths for Eng, Inf, Rsp, and Nxt are, by the authors' own description, deterministic functions of the construction script, device metadata, and controlled role assignment, which specify engagement schedules, stated facts, and standing requests. Unified Agent's state is defined to store exactly these categories: engagement counts, pointing cues, first stated facts per device-topic, and the standing request. Consequently, the evaluation targets are generated from the same conceptual categories that Unified Agent tracks, and its strong performance may reflect agreement with the construction protocol rather than a general property of stateful agents. The real-photo matched pair (Appendix F) is a single author-prepared example and does not establish external validity. I request either an evaluation on episodes whose ground truth is not derived from the method's state categories (e.g., human-annotated or independently generated targets) or a clear and explicit scoping of the claims as benchmark-internal rather than general.
[Table 1 / Section 6.2] The advantage over the full-record control is small and concentrated in one decision. Comparing Unified Agent with Full context: Eng +0.012, Int -0.031, Inf +0.010, Rsp -0.023, and Nxt +0.305. Thus the overall gap of 0.055 is almost entirely driven by Nxt, while Full context is actually slightly better on Int and Rsp. Because Full context receives all the same information, the paper should analyze whether the Nxt difference is a robust state-design effect or an artifact of the specific Nxt output space and scoring rule (exact action-device match on designated frames). Otherwise the broader statement that a well-organized carried state provides a performance advantage is stronger than the evidence supports.

minor comments (5)

[Figure 5] The caption and axis labels are difficult to parse ('131 1,442'); please clarify whether the numbers are characters at specific frames or a range, and ensure the figure is readable in grayscale.
[Section 8 (Limitations)] The Limitations section reads partly as a defense of the design rather than a limitation statement; please acknowledge more directly that an explicit carried state is itself a privacy-sensitive artifact and that the real-photo evidence is a single matched pair with no uncertainty quantification.
[Abstract / Section 1] The phrase 'significant outperforms' is used in the abstract and contributions; in the main text significance is defined only in terms of bootstrap p-values, not an effect-size criterion for the state controls. Please align the wording with the statistical definitions used in Section 6 and Appendix E.
[Appendix D.2] The baseline adaptations are described only briefly; please add a sentence for each baseline about which input fields it receives and how the adaptation maps the original method's interface to the shared perception record, so that the fairness of the comparison is verifiable.
[General] The paper states that code and data will be publicly released but provides no repository link or availability artifact; include a URL or an availability statement at the time of publication.

Circularity Check

1 steps flagged · score 6.0 of 10

UA-BENCH defines Eng/Inf/Rsp/Nxt targets from the same construction-script semantics that Unified Agent's carried state is hard-wired to accumulate, so the default-setting 'predictions' are partly round-trip read-outs of the benchmark's own annotation record.

self definitional [Appendix A.1-A.2 (state representation/update) and Appendix B.3 (episode construction and admission)]
"'Eng, Inf, Rsp, and Nxt are deterministic functions of the construction script, device metadata, and controlled role assignment' (B.3); 'In Ct, mt counts nonempty engagement observations ... and litt(d) counts those containing d ... The decision interface summarizes these cues as per-device tallies' (A.1); 'Eng uses accumulated activity and attended-device cues; ... Inf retrieves the fact for the queried topic' (A.2)."

Eng target is the device the script assigns as engaged; Eng output is an argmax over Ct/Pt tallies that count the scripted engagement observations. Inf target is the stated value assigned to a device–topic pair; Inf output is a lookup of Kt, which stores the first stated fact for that pair. Rsp and Nxt likewise re-read the card-declared abilities and standing request that define their targets. High scores therefore verify that the hand-written fold/read-out round-trips the benchmark annotation record, not an independent inference problem. Baselines denied these fields start from a strictly smaller input space, so the largest reported gaps conflate feature availability with state design.

full rationale

The paper is not built on self-citations; its references are external, and no uniqueness theorem is imported from the authors' prior work. The main circularity is benchmark-level: UA-BENCH defines Eng/Inf/Rsp/Nxt as deterministic functions of the same scripted roles, values, requests, and ability cards that Unified Agent's state (Ct, Pt, Kt, rt) is designed to carry, and Appendix A.2 states each decision is read directly from those carried fields. Thus the default-setting 'predictions' are partly read-outs of the annotation record. Two pieces of evidence keep this from being fully circular: Full context receives the complete record and still trails on Nxt (gap 0.055), and the real-photo matched pair tests the pipeline on non-rendered inputs. However, the full-record advantage is small relative to the paper's own 0.10 effect-size criterion, and the real-photo evidence is a single author-prepared pair. The published-baseline gaps (0.194–0.408) are additionally confounded by Appendix D.1's statement that baselines receive the shared perception record 'restricted to the input fields their designs specify,' so near-floor Mem0/MM-DST Eng/Inf scores may reflect withheld visual engagement fields rather than state design. This is a validity concern separate from formal circularity, but it compounds the benchmark-alignment problem. Overall score 6: the central Eng/Inf/Rsp/Nxt results reduce partly by construction, while Full context and one real-photo pair provide limited independent support.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No numeric free parameters are fitted to data; the state design is hand-crafted but not fitted. No new physical or conceptual entities (particles, forces, dimensions) are introduced. The central claim rests on domain assumptions about benchmark validity, the reliability of engagement cues, the adequacy of the shared perception output, and the validity of the model-based intent judge.

assumptions (4)

domain assumption UA-BENCH's matched-pair construction and deterministic ground truth faithfully represent the cross-device interaction problem (Appendix B).
The entire evaluation rests on the benchmark being a valid operationalization of the target scenario. If the benchmark is not representative, the comparative results do not transfer.
domain assumption Engagement evidence (activity counts, pointing, hand-on-device contact) is a reliable cue for resolving device-unspecified references (Appendix A.1).
The method and the benchmark both assume that engagement strength identifies the device the user refers to. This is the mechanism that makes the state useful.
domain assumption The shared per-frame perception record provides sufficient information for downstream decisions (Appendix C.2).
All compared systems depend on the shared MLLM perception output being good enough. Perception errors propagate and are only partially diagnosed in Table 8.
domain assumption The fixed same-meaning MLLM judge reliably evaluates intent equivalence (Appendix C.4).
The Intent scores are produced by a text-only GPT-5.6-Sol judge without reported human agreement. Relative comparisons are fair because the judge is fixed, but absolute scores may be noisy.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Unified Agent: Managing Interactions across Devices." pith.science (2026). https://pith.science/paper/PLDMWETD

@misc{pith2026260805729,
  author       = {Pith},
  title        = {Pith review of: Unified Agent: Managing Interactions across Devices},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PLDMWETD}},
  note         = {Machine review of arXiv:2608.05729}
}

read the original abstract

As capabilities rapidly increase, AI agents can move from running inside one app to acting across a user's devices over time. Yet existing agent systems still fall short in this scenario. This is because observations are scattered across devices and moments, but mainstream systems are not designed around this fact: a single agent that treats devices as tools lacks effective state management for all devices across time, and multi-agent systems coordinate across agents but do not maintain the compact carried state a cross-device, cross-time request needs. We argue that the agent should maintain an effectively designed state that organizes engagement evidence, stated facts, and the standing request in a compact, action-ready form for deciding its action given the current observation. To compare state designs, we construct a benchmark of user-agent interaction across devices and time. We instantiate this principle in Unified Agent, a stateful agent that carries interaction evidence across devices and moments and uses it with the current observation to act. In the default setting, it significantly outperforms our adaptations of four published designs. Across changes in multimodal large language model (MLLM) family, capability, and reasoning effort, it remains ahead of all compared systems, demonstrating that the state-design advantage is robust across MLLM settings. Our code and data will be publicly available on GitHub.

Figures

Figures reproduced from arXiv: 2608.05729 by the authors.

**Figure 1.** One agent serving one user across devices and over time. After comparing restaurants on several devices, the user later decides to make a reservation but has forgotten the restaurant’s name. The user therefore asks to book “the restaurant I’ve been looking at” without specifying a device. A stateless agent may need to ask, “Which device was it on?” In contrast, Unified Agent carries a compact, action-ready state an… view at source ↗

**Figure 2.** Six tasks organized as observe–think–act. The agent perceives the current interaction and identifies the engaged device, reasons with information carried across moments, and determines how to act across devices. provide durable stores, typically within a channel, scene, profile, or task, while the benchmarks test whether agents can retain and use information. Unified Agent instead addresses the cross-device case, w… view at source ↗

**Figure 3.** Overview of Unified Agent. At moment t, St−1 carries evidence accumulated across devices over moments 1, . . . , t − 1. The agent folds the current observation Ot into St−1 to produce St, which stores engagement evidence, stated facts, and the standing request. It then acts by reading St together with Ot, and carries St forward to the next moment. 4 Method An agent acting across a user’s devices must decide each ac… view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Rendered UA-BENCH frame. (a) An agentvisible device-camera view shows user–device interaction in a scene containing screen devices and a fetch robot. (b) The paired semantic ground truth supports construction-time validation of object identity and visibility and is …

**Figure 5.** Figure 5: Carried-state size. Characters of retained values after each frame, averaged over the 200 episodes; fixed template text is excluded identically for every system. Unified Agent stays bounded; Full context grows linearly to 11 times the size of Unified Agent’s state. re…

**Figure 6.** Figure 6: Real-photo matched pair. The user engages with the laptop in episode A and the tablet in episode B; the shared request frame is identical across the pair while the earlier engagement is swapped. 6.4 Generalization across MLLM settings Finding 3: The benefit of a well-o…

**Figure 7.** Figure 7: Matched-pair design. The paired episodes exchange earlier engagement roles and associated values while sharing the later request and recall frames. The paired construction makes earlier interaction evidence the distinguishing context for the later decisions. fies devic…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 42 canonical work pages

[1]

Adnen Abdessaied, Manuel Hochmeister, and Andreas Bulling. 2024. Olvit: Multi-modal state tracking via attention-based embeddings for video-grounded dialog. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 12348--12358

work page 2024
[2]

Mouad Abrini and Mohamed Chetouani. 2026. Clue: Crossmodal disambiguation via language-vision understanding with attention. arXiv preprint arXiv:2602.08999

arXiv 2026
[3]

Emre Can Acikgoz, Jeremiah Greer, Akul Datta, Ze Yang, William Zeng, Oussama Elachqar, Emmanouil Koukoumidis, Dilek Hakkani-Tur, and Gokhan Tur. 2025. Can a single model master both multi-turn conversations and tool use? coalm: A unified conversational agentic language model. In Proceedings of the 63rd Annual Meeting of the Association for Computational L...

work page 2025
[4]

Saaket Agashe, Kyle Wong, Vincent Tu, Jiachen Yang, Ang Li, and Xin Eric Wang. 2025. Agent s2: A compositional generalist-specialist framework for computer use agents. arXiv preprint arXiv:2504.00906

arXiv 2025
[5]

Riku Arakawa, Prasoon Patidar, Will Page, Jill Lehman, and Mayank Goel. 2025. Scaling context-aware task assistants that learn from demonstration and adapt through mixed-initiative dialogue. In Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology, pages 1--19

work page 2025
[6]

Pawe Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, I \ n igo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. 2018. Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 5016--5026

work page 2018
[7]

Matthew Chang, Gunjan Chhablani, Alexander Clegg, Mikael Dallaire Cote, Ruta Desai, Michal Hlavac, Vladimir Karashchuk, Jacob Krantz, Roozbeh Mottaghi, Priyam Parashar, et al. 2025. Partnr: A benchmark for planning and reasoning in embodied multi-agent tasks. In International Conference on Learning Representations, volume 2025, pages 65205--65268

work page 2025
[8]

Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. 2025. Mem0: Building production-ready ai agents with scalable long-term memory. arXiv preprint arXiv:2504.19413

arXiv 2025

Show all 62 references

[9]

Hyeong Kyu Choi, Xiaojin Zhu, and Sharon Li. 2025. Debate or vote: Which yields better decisions in multi-agent large language models? In Advances in Neural Information Processing Systems

2025
[10]

Google DeepMind . 2026. https://deepmind.google/models/model-cards/gemini-3-1-flash-lite/ Gemini 3.1 Flash-Lite . Technical report, Google DeepMind

2026
[11]

Qijun Han, Haoqin Tu, Zijun Wang, Haoyue Dai, Yiyang Zhou, Nancy Lau, Alvaro A Cardenas, Yuhui Xu, Ran Xu, Caiming Xiong, et al. 2026. Vlaa-gui: Knowing when to stop, recover, and search, a modular framework for gui automation. arXiv preprint arXiv:2604.21375

2026 arXiv
[12]

Shaid Hasan, Breenice Lee, Sujan Sarker, and Tariq Iqbal. 2026. M2hri: An llm-driven multimodal multi-agent framework for personalized human-robot interaction. arXiv preprint arXiv:2604.11975

2026 arXiv
[13]

Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. 2022. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608

2022 arXiv
[14]

Brian Ichter, Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, Dmitry Kalashnikov, Sergey Levine, Yao Lu, Carolina Parada, Kanishka Rao, Pierre Sermanet, Alexander Toshev, Vincent Vanho...

2022
[15]

Koji Inoue, Divesh Lala, Mikey Elmers, Keiko Ochi, and Tatsuya Kawahara. 2025. An llm benchmark for addressee recognition in multi-modal multi-party dialogue. In Proceedings of the 15th International Workshop on Spoken Dialogue Systems Technology, pages 330--334

2025
[16]

Lawrence Zitnick, and Ross B

Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei - Fei, C. Lawrence Zitnick, and Ross B. Girshick. 2017. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition,...

2017
[17]

Nikhita Joshi, Richard Li, Jiannan Li, Leonardo Pavanatto, Michel Pahud, Jatin Sharma, Bongshin Lee, Hugo Romat, William Buxton, Nicolai Marquardt, et al. 2024. Opportunistic nudges for task migration between personal devices. In Extended Abstracts of the CHI Conference on Hum...

2024
[18]

Chang, and Manolis Savva

Mukul Khanna, Yongsen Mao, Hanxiao Jiang, Sanjay Haresh, Brennan Shacklett, Dhruv Batra, Alexander Clegg, Eric Undersander, Angel X. Chang, and Manolis Savva. 2024. Habitat synthetic scenes dataset (HSSD-200): an analysis of 3d scene scale and realism tradeoffs for objectgoal ...

2024
[19]

Satwik Kottur, Seungwhan Moon, Alborz Geramifard, and Babak Damavandi. 2021. Simmc 2.0: A task-oriented dialog dataset for immersive multimodal conversations. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4903--4912

2021
[20]

Sorokin, and Mikhail Burtsev

Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Y. Sorokin, and Mikhail Burtsev. 2024. Babilong: Testing the limits of llms with long context reasoning-in-a-haystack. In Advances in Neural Information Processing Systems 37: Annual Conference on N...

2024
[21]

Kenneth Kwok, Basura Fernando, Qianli Xu, Vigneshwaran Subbaraju, Dongkyu Choi, and Boon Kiat Quek. 2026. Explicit world models for reliable human-robot collaboration. arXiv preprint arXiv:2601.01705

2026
[22]

Hung Le, Nancy Chen, and Steven Hoi. 2022. Multimodal dialogue state tracking. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3394--3415

2022
[23]

Geonsun Lee, Min Xia, Nels Numan, Xun Qian, David Li, Yanhe Chen, Achin Kulshrestha, Ishan Chatterjee, Yinda Zhang, Dinesh Manocha, et al. 2025. Sensible agent: A framework for unobtrusive interaction with proactive ar agents. In Proceedings of the 38th Annual ACM Symposium on...

2025
[24]

Jaewook Lee, Jun Wang, Elizabeth Brown, Liam Chu, Sebastian S Rodriguez, and Jon E Froehlich. 2024. Gazepointar: A context-aware multimodal voice assistant for pronoun disambiguation in wearable augmented reality. In Proceedings of the 2024 CHI Conference on Human Factors in C...

2024
[25]

Xinpeng Li, Shijian Deng, Bolin Lai, Weiguo Pian, James M Rehg, and Yapeng Tian. 2026. Towards online multi-modal social interaction understanding. Transactions on Machine Learning Research (TMLR)

2026
[26]

Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. 2023. Code as policies: Language model programs for embodied control. In 2023 IEEE International conference on robotics and automation (ICRA), pages 9493--9500. IEEE

2023
[27]

Kehui Liu, Zixin Tang, Dong Wang, Zhigang Wang, Xuelong Li, and Bin Zhao. 2025. Coherent: Collaboration of heterogeneous multi-robot system with large language models. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 10208--10214. IEEE

2025
[28]

Chenyang Ma, Kai Lu, Ruta Desai, Xavier Puig, Andrew Markham, and Niki Trigoni. 2025. Coopera: Continual open-ended human-robot assistance. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS)

2025
[29]

Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. 2024. Evaluating very long-term conversational memory of llm agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pap...

2024
[30]

Fanqing Meng, Lingxiao Du, Zijian Wu, Guanzheng Chen, Xiangyan Liu, Jiaqi Liao, Chonghe Jiang, Zhenglin Wan, Jiawei Gu, Pengfei Zhou, et al. 2026. Clawmark: A living-world benchmark for multi-turn, multi-day, multimodal coworker agents. arXiv preprint arXiv:2604.23781

2026 arXiv
[31]

So Yeon Min, Xavi Puig, Devendra Singh Chaplot, Tsung-Yen Yang, Akshara Rai, Priyam Parashar, Ruslan Salakhutdinov, Yonatan Bisk, and Roozbeh Mottaghi. 2024. Situated instruction following. In European Conference on Computer Vision, pages 202--228. Springer

2024
[32]

OpenAI . 2026. https://deploymentsafety.openai.com/gpt-5-6 GPT-5.6 system card

2026
[33]

Patil, Kevin Lin, Sarah Wooders, and Joseph E

Charles Packer, Vivian Fang, Shishir G. Patil, Kevin Lin, Sarah Wooders, and Joseph E. Gonzalez. 2023. Memgpt: Towards llms as operating systems. CoRR, abs/2310.08560

2023 arXiv
[34]

Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Tsung-Yen Yang, Ruslan Partsey, Ruta Desai, Alexander Clegg, Michal Hlavac, So Yeon Min, et al. 2024. Habitat 3.0: A co-habitat for humans, avatars, and robots. In International Conference on Learning Representa...

2024
[35]

Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. 2025. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326

2025 arXiv
[36]

Preston Rasmussen, Pavlo Paliychuk, Travis Beauvais, Jack Ryan, and Daniel Chalef. 2025. Zep: a temporal knowledge graph architecture for agent memory. arXiv preprint arXiv:2501.13956

2025 arXiv
[37]

Alireza Rezazadeh, Zichao Li, Ange Lou, Yuying Zhao, Wei Wei, and Yujia Bao. 2025. Collaborative memory: Multi-user memory sharing in llm agents with dynamic access control. arXiv preprint arXiv:2505.18279

2025 arXiv
[38]

Dmitriy Rivkin, Francois Hogan, Amal Feriani, Abhisek Konar, Adam Sigal, Steve Liu, and Greg Dudek. 2023. Sage: smart home agent with grounded execution. arXiv preprint arXiv:2311.00772

2023 arXiv
[39]

Timo Schick, Jane Dwivedi - Yu, Roberto Dess \` , Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. In Advances in Neural Information Processing Systems 36: A...

2023
[40]

Omar Shaikh, Shardul Sapkota, Shan Rizvi, Eric Horvitz, Joon Sung Park, Diyi Yang, and Michael S Bernstein. 2025. Creating general user models from computer use. In Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology, pages 1--23

2025
[41]

Yiheng Shu, Saisri Padmaja Jonnalagedda, Xiang Gao, Bernal Jim \'e nez Guti \'e rrez, Weijian Qi, Kamalika Das, Huan Sun, and Yu Su. 2026. Remem: Reasoning with episodic memory in language agent. arXiv preprint arXiv:2602.13530

2026
[42]

Huajie Tan, Cheng Chi, Xiansheng Chen, Yuheng Ji, Zhongxia Zhao, Xiaoshuai Hao, Yaoxu Lyu, Mingyu Cao, Junkai Zhao, Huaihai Lyu, et al. 2025. Roboos-next: A unified memory-based framework for lifelong, scalable, and robust multi-robot collaboration. arXiv preprint arXiv:2510.26536

2025
[43]

Zineng Tang, Lingjun Mao, and Alane Suhr. 2024. Grounding language in multi-perspective referential communication. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 19727--19741

2024
[44]

Ravi Tejwani, Boris Katz, and Cynthia Breazeal. 2021. Migratable ai: Investigating users' affect on identity and information migration of a conversational ai agent. In International Conference on Social Robotics, pages 257--267. Springer

2021
[45]

Zhuyu Teng, Pei Chen, Yichen Cai, Ruoqing Lu, Zhaoqu Jiang, Jiayang Li, Weitao You, and Lingyun Sun. 2026. Seeing eye to eye: Enabling cognitive alignment through shared first-person perspective in human-ai collaboration: Seeing eye to eye. In Proceedings of the 2026 CHI Confe...

2026
[46]

Takuma Udagawa and Akiko Aizawa. 2019. A natural language corpus of common grounding under continuous and partially-observable context. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7120--7127

2019
[47]

Hannah VanderHoeven, Brady Bhalla, Ibrahim Khebour, Austin C Youngren, Videep Venkatesha, Mariah Bradford, Jack Fitzgerald, Carlos Mabrey, Jingxuan Tu, Yifan Zhu, et al. 2025. Trace: Real-time multimodal common ground tracking in situated collaborative dialogues. In Proceeding...

2025
[48]

Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. 2025. Mixture-of-agents enhances large language model capabilities. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025 . OpenReview.net

2025
[49]

Zhiyuan Wang, Erzhen Hu, Mark Rucker, and Laura E Barnes. 2026. Psi: Shared state as the missing layer for coherent ai-generated instruments in personal ai agents. arXiv preprint arXiv:2604.08529

2026 arXiv
[50]

Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai - Wei Chang, and Dong Yu. 2025. Longmemeval: Benchmarking chat assistants on long-term interactive memory. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025 . OpenReview.net

2025
[51]

Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. 2025. A-mem: Agentic memory for llm agents. arXiv preprint arXiv:2502.12110

2025 arXiv
[52]

Karmesh Yadav, Yusuf Ali, Gunshi Gupta, Yarin Gal, and Zsolt Kira. 2025. Findingdory: A benchmark to evaluate memory in embodied agents. arXiv preprint arXiv:2506.15635

2025
[53]

Narasimhan, and Yuan Cao

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . Ope...

2023
[54]

Yiqun Yao, Naitong Yu, Xiang Li, Xin Jiang, Xuezhi Fang, Wenjia Ma, Xuying Meng, Jing Li, Aixin Sun, and Yequan Wang. 2025. Egomem: Lifelong memory agent for full-duplex omnimodal models. arXiv preprint arXiv:2509.11914

2025
[55]

Albert Yu, Chengshu Li, Luca Macesanu, Arnav Balaji, Ruchira Ray, Raymond Mooney, and Roberto Mart \' n-Mart \' n. 2025 a . Mixed-initiative dialog for human-robot collaborative manipulation. arXiv preprint arXiv:2508.05535

2025
[56]

Yangchen Yu, Yin Chen, Jia Li, Peng Jia, Yu Zhang, Li Dai, Zhenzhen Hu, Meng Wang, and Richang Hong. 2025 b . Generalizable engagement estimation in conversation via domain prompting and parallel attention. In Proceedings of the 33rd ACM International Conference on Multimedia,...

2025
[57]

Kevin Zakka, Yuval Tassa, and MuJoCo Menagerie Contributors . 2022. http://github.com/google-deepmind/mujoco_menagerie MuJoCo Menagerie: A collection of high-quality simulation models for MuJoCo

2022
[58]

Zhonghao Zhan, Krinos Li, Yefan Zhang, and Hamed Haddadi. 2026. Hearthnet: Edge multi-agent orchestration for smart homes. In Proceedings of the ACM Conference on AI and Agentic Systems, CAIS 2026, San Jose, CA, USA, May 26-29, 2026 , pages 1123--1127. ACM

2026
[59]

Chaoyun Zhang, Liqun Li, He Huang, Chiming Ni, Bo Qiao, Si Qin, Yu Kang, Minghua Ma, Qingwei Lin, Saravan Rajmohan, et al. 2025 a . UFO ^ 3 : Weaving the digital agent galaxy. arXiv preprint arXiv:2511.11332

2025
[60]

Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B Tenenbaum, Tianmin Shu, and Chuang Gan. 2024. Building cooperative embodied agents modularly with large language models. In International Conference on Learning Representations, volume 2024, pages 19373--19401

2024
[61]

Wentao Zhang, Liang Zeng, Yuzhen Xiao, Yongcong Li, Ce Cui, Yilei Zhao, Rui Hu, Yang Liu, Yahui Zhou, and Bo An. 2025 b . Agentorchestra: Orchestrating multi-agent intelligence with the tool-environment-agent (tea) protocol. arXiv preprint arXiv:2506.12508

2025 arXiv
[62]

Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. 2023. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, pages 2165--2183. PMLR

2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Adnen Abdessaied, Manuel Hochmeister, and Andreas Bulling. 2024. Olvit: Multi-modal state tracking via attention-based embeddings for video-grounded dialog. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 12348--12358

work page 2024

[2] [2]

Mouad Abrini and Mohamed Chetouani. 2026. Clue: Crossmodal disambiguation via language-vision understanding with attention. arXiv preprint arXiv:2602.08999

arXiv 2026

[3] [3]

Emre Can Acikgoz, Jeremiah Greer, Akul Datta, Ze Yang, William Zeng, Oussama Elachqar, Emmanouil Koukoumidis, Dilek Hakkani-Tur, and Gokhan Tur. 2025. Can a single model master both multi-turn conversations and tool use? coalm: A unified conversational agentic language model. In Proceedings of the 63rd Annual Meeting of the Association for Computational L...

work page 2025

[4] [4]

Saaket Agashe, Kyle Wong, Vincent Tu, Jiachen Yang, Ang Li, and Xin Eric Wang. 2025. Agent s2: A compositional generalist-specialist framework for computer use agents. arXiv preprint arXiv:2504.00906

arXiv 2025

[5] [5]

Riku Arakawa, Prasoon Patidar, Will Page, Jill Lehman, and Mayank Goel. 2025. Scaling context-aware task assistants that learn from demonstration and adapt through mixed-initiative dialogue. In Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology, pages 1--19

work page 2025

[6] [6]

Pawe Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, I \ n igo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. 2018. Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 5016--5026

work page 2018

[7] [7]

Matthew Chang, Gunjan Chhablani, Alexander Clegg, Mikael Dallaire Cote, Ruta Desai, Michal Hlavac, Vladimir Karashchuk, Jacob Krantz, Roozbeh Mottaghi, Priyam Parashar, et al. 2025. Partnr: A benchmark for planning and reasoning in embodied multi-agent tasks. In International Conference on Learning Representations, volume 2025, pages 65205--65268

work page 2025

[8] [8]

Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. 2025. Mem0: Building production-ready ai agents with scalable long-term memory. arXiv preprint arXiv:2504.19413

arXiv 2025