Pith. sign in

REVIEW 4 major objections 5 minor 34 references

Sharingan: Extract User Action Sequence from Desktop Recordings

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper demonstrates that vision-language models can extract user action sequences from desktop screen recordings with 70–80% accuracy on operation type, and that feeding frames directly outperforms explicitly computing UI changes.

desk verdict First VLM task formulation for desktop action extraction with two benchmarks, but the paper's own replay data contradicts its semantic metric, so the headline accuracy claims need caution. read the letter →

arxiv 2411.08768 v1 pith:SRSYAZ25 submitted 2024-11-13 cs.CV cs.AI

Yanting Chen , Yi Ren , Xiaoting Qin , Jue Zhang , Kehong Yuan , Lu Han , Qingwei Lin , Dongmei Zhang
show 2 more authors
Saravan Rajmohan Qi Zhang
This is my paper · ORCID
classification cs.CVcs.AI
keywords desktopscreenrecordingsuseractionextractionvision-languagemodelsoperationtypeclassificationRoboticProcessAutomationUIchangedetectionsequencebenchmarksemanticmatching
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a vision-language model can turn a desktop screen recording into a structured sequence of user actions — click, select, scroll, drag, type — with operation-type accuracy around 70%–80% and with sequences that are executable by a robotic-process-automation tool. It compares two pipelines: one (DF) feeds sampled frames directly to the model, the other (DiffF) first computes explicit pixel-level UI changes and feeds those to the model. Across a curated dataset and a realistic benchmark, the direct pipeline performs best, and explicit UI-change extraction tends to degrade accuracy instead of helping it. If correct, this makes video a practical source of behavioral data for automation, tutorials, and personalization, and it argues that current models are better left to infer actions from raw frames.

What carries the argument

The machinery is the action tuple $(O,D,C)$ — operation type (one of five), the detail (UI element), and the context (application) — combined with a three-module VLM pipeline: an Action Proposer that outputs candidate actions with reasoning, an Action Corrector that deletes redundant and invalid actions and fills missing information, and an Action Merger that stitches overlapping sliding windows. In DiffF, a Frame Difference Localizer computes pixel-difference masks using computer-vision filters and thresholds, and a Frame Difference Descriptor narrates those changes before the same proposer/corrector modules run. The evaluation device is a semantic matching procedure that accepts a prediction when the operation type matches exactly and the BERT-cosine similarity of the detail and context descriptions lies above a manually tuned threshold of 0.7, followed by greedy chronological alignment.

What would settle it

Take a sample of predicted action sequences from ACTONE and ACTREAL, have independent human judges mark each predicted action as correct/incorrect against the ground truth, and compare those labels with the paper's semantic-matching verdicts; if the agreement is low, the 70–80% figure is an artifact of the metric. A second, direct check is to replay predicted sequences on a larger, unbiased set of cases and see whether replay success tracks the semantic precision/recall.

Watch

Extended reading notes

Core claim

The central discovery is that the straightforward DF pipeline works: given a desktop recording, uniformly sampled frames, and a large vision-language model prompted to propose, correct, and merge actions, the model identifies the operation type with recall and precision around 0.8 (GPT-4o: 0.83/0.81 on ACTONE, 0.82/0.70 on ACTREAL). When the full action tuple — operation, UI element, and application context — is evaluated, performance drops to 0.71/0.68 and 0.53/0.45 respectively, but the paper still reports the operation-level numbers as the headline accuracy because RPA replay succeeded in 6 of 9 tested cases. A second discovery is that DiffF, which adds an explicit computer-vision stage to localize and describe UI changes before action inference, does not improve over DF and often hurts precision, which the paper attributes to the model over-focusing on localized boxes and losing global context.

Load-bearing premise

The reported accuracy numbers assume that a BERT-cosine similarity score above a hand-tuned 0.7 threshold, combined with the paper's greedy chronological matching, correctly identifies when a predicted action matches the ground-truth action; if that match rule misclassifies actions, the 70–80% accuracy collapses.

Editorial extensions

If this is right

  • Operation-type predictions are accurate enough to drive RPA replay: 6 of 9 tested cases succeeded, and the successful cases all had perfect semantic precision/recall.
  • The DF pipeline is the better default: the two ablation studies show that adding frame differences to DF's input or adding all frames to DiffF's proposer yields only marginal or negative gains.
  • Smaller VLMs (4o-mini, Flash) lose 0.2–0.5 in precision, showing that model capacity is a major factor in this task.
  • The error breakdown points to visual hallucination/blindness as the dominant failure mode for DF and reasoning problems for DiffF, which tells future work where to invest.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 70–80% figure applies to operation-type recognition only; the full-tuple numbers on ACTREAL (0.53 recall) show that recovering the exact UI element and context is still the hard part of the task.
  • The 0.7 BERT-cosine threshold is tuned by hand on these datasets; a principled alignment method, or a learned threshold, could change the reported accuracies even if the relative ranking of DF and DiffF holds.
  • DiffF's poor result may be an artifact of giving the VLM precomputed pixel-diff boxes; a model that computes temporal differences internally, rather than relying on an external CV stage, might capture the same signal without the context loss.
  • Because replay validation covers only nine click/type cases from the curated dataset, a direct next experiment is to replay a larger, more diverse sample (including drag, scroll, select) to see whether the semantic metric remains predictive.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes two vision-language-model (VLM) based methods for extracting user action sequences from desktop screen recordings. The Direct Frame-Based approach (DF) feeds sampled frames directly to a VLM and uses sliding-window processing with modules for action proposal, correction, and merging. The Differential Frame-Based approach (DiffF) first computes explicit frame differences via computer-vision techniques, then asks a VLM to interpret textual descriptions of localized UI changes. The authors evaluate both methods on two datasets: ACTONE, a self-curated set of 40 short videos each containing one or two simple actions of a single type, and ACTREAL, a set of 41 videos adapted from GUI-World with longer, multi-action sequences. Evaluation uses precision and recall computed both at the operation-type level and over the full (operation, detail, context) tuple, with semantic matching based on BERT cosine similarity and a manually tuned threshold of 0.7. The paper reports that GPT-4o with DF achieves operation-level recall of 0.83 and precision of 0.81 on ACTONE, and 0.82 recall and 0.70 precision on ACTREAL, while the full-tuple metrics are substantially lower. A replay-based validation on nine ACTONE cases using a modified version of the authors' own UFO tool succeeded in six cases. The central claim is that the DF approach achieves 70%–80% accuracy in identifying operation types and that the extracted sequences are re-playable through RPA-like processes.

Significance. If the claims are validated, this work would be an early demonstration of VLM-based extraction of user actions from desktop video, with potential downstream applications in Robotic Process Automation, tutorial generation, and user modeling. The two datasets, the modular method design, and the detailed prompt templates are potentially useful resources for the community. However, the current evaluation is not yet at the standard needed to support the headline claims: the datasets are small (40 and 41 videos), no variance estimates or significance tests are provided, the primary metric is a semantic proxy whose validity is partially contradicted by the paper's own replay data, and the abstract overstates the level of accuracy by omitting the 'operation type' qualifier. The strengths are the clear task formulation and the explicit, reproducible experimental setup, but these need to be matched by more rigorous evaluation and more carefully hedged claims.

major comments (4)
  1. [Section IV-B, Table VII] The paper states that 'Precision and Recall metrics derived from semantic matching are consistent with actual success rates of RPA replay,' but Table VII contains a direct counterexample: the case 'click/text icon/tab' has semantic Precision = 1 and Recall = 1 yet the replay result is 'no'. This means that a perfect semantic score does not guarantee executable correctness, so the metric can certify a sequence as correct even when an RPA tool cannot replay it. Since the 70%–80% accuracy figures in Section V-B and the abstract are computed with this metric, the replayability claim is not adequately supported. Please either reconcile this discrepancy, provide a refined metric, or temper the claim that extracted sequences are re-playable through RPA-like processes.
  2. [Section V-B and Abstract] The abstract says the DF approach achieves 'an accuracy of 70% to 80% in identifying user actions,' but the reported numbers in Table II are for operation-type identification only: GPT-4o with DF has operation recall 0.83 and precision 0.81, while the full operation-detail-context ('All') metrics are 0.71 recall and 0.68 precision. On ACTREAL (Table III), the 'All' metrics are even lower, with recall 0.53. The phrase 'identifying user actions' without the 'operation type' qualifier is therefore misleading; the claims in Sections I and V-B should be explicitly scoped to operation-type identification, and the full-tuple accuracy should be reported wherever the headline accuracy is cited.
  3. [Sections IV-A/IV-B and Tables II/III] The evaluation is based on only 40 and 41 videos, and no standard errors, confidence intervals, or significance tests are reported. The conclusion that DF is more reliable than DiffF in real-world scenarios relies on differences such as GPT-4o 'All' recall of 0.53 versus 0.27 on ACTREAL, but without variance estimates these differences could be due to chance. Please provide per-video results or bootstrap confidence intervals, at least for the primary DF-versus-DiffF comparisons. Additionally, the replay validation covers only nine ACTONE cases and none from ACTREAL, so the claim that extracted sequences are generally re-playable through RPA is only weakly supported.
  4. [Section IV-B] The semantic matching procedure relies on a manually tuned BERT cosine-similarity threshold of 0.7, but the paper does not provide a sensitivity analysis or justification for this value. Since every reported precision and recall number depends on this threshold, the choice is load-bearing for the central claim. Please include a threshold sweep showing how the metrics vary with the threshold, or provide an alternative justification for why 0.7 is the correct operating point.
minor comments (5)
  1. [Abstract] There is a typo: 're-playable though Robotic Process Automation' should read 're-playable through Robotic Process Automation.'
  2. [Section III-A, footnote] The footnote says the prompts 'are not included in current draft' and will be open-sourced, but the full prompts are actually provided in the Appendix (Tables VIII–XXI). Please remove or update this note.
  3. [Section IV-B] The replay validation paragraph states that 'two of the three unsuccessful cases exhibited lower Precision and Recall values,' but does not mention that one unsuccessful case ('click/text icon/tab') achieved perfect scores on both metrics. This omission makes the validation seem more favorable than it is; please be explicit about the exception.
  4. [Table VII] The column headers are confusing: the table lists 'Semantic Matching' and 'Successful Replay?' but the columns are actually 'Precision (All)', 'Recall (All)', and a yes/no replay result. Please restructure the table or the headers for clarity.
  5. [Section II] The related work section could better differentiate the paper from GUI agent works such as UFO and OmniParser, which also process screen content with VLMs; the claim of being the 'first' attempt should be verified and hedged accordingly.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: the reported accuracy is a direct empirical measurement; the only self-referential element is a minor self-cited replay harness that does not force the results.

full rationale

The paper contains no derivation chain in which a prediction is defined in terms of its own inputs. The headline 70-80% operation-type accuracy (Section V-B, Tables II and III) is computed by exact matching of predicted operation types against manually annotated ground truth; it does not depend on the manually tuned BERT threshold of 0.7, which only affects the separate 'All' metrics via the semantic similarity matrices described in Section IV-B. ACTONE is self-curated and ACTREAL is adapted from the external GUI-World dataset, so the evaluation is anchored outside the method's own prompt design. The one self-citational element is the replay validator: Section IV-B states 'We employed a modified version of the VLM-based UI navigation tool [33]', and the authors of [33] overlap with the present paper. However, this tool is used as an execution harness rather than as a source of the semantic scores, and it is not a load-bearing premise for the operation-type accuracy numbers. The paper's own Table VII even contains a counterexample to the claim that semantic scores match replay success (case 'click/text icon/tab' has Precision=1 and Recall=1 but replay='no'), which is a correctness and validity concern about the metric, not a circularity: the metric is not equivalent to the replay outcome by construction. No self-definitional equations, fitted-input-as-prediction steps, uniqueness imports, or ansatz-smuggling steps are present; the score of 2 reflects only the minor self-citation in the validation harness.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new scientific entities, particles, or forces; its free parameters are evaluation and preprocessing hyperparameters. The most consequential free parameter is the BERT similarity threshold of 0.7, because it directly defines the reported accuracy numbers. The core domain assumption is the five-action taxonomy, which bounds the scope of the claim.

free parameters (6)
  • frame difference threshold = 0.15
    Used in Frame Difference Localizer (Section III-B) to threshold L2 pixel difference; described as tuned without a tuning procedure.
  • minimum object size = 10 pixels
    Objects smaller than 10 pixels removed from the binary mask; chosen as below human perception.
  • bounding box expansion = 100 pixels
    Each bounding box expanded by 100 pixels on all sides and overlapping boxes merged; chosen to provide visual context.
  • BERT similarity threshold = 0.7
    Manually tuned threshold for semantic matching in Section IV-B; directly controls reported Precision and Recall.
  • sliding window size and overlap = 10 frames, 5 overlap
    Set based on GPT-series 10-image limit; not tuned for performance.
  • frame sampling rate = 1 fps (ACTONE), 2 fps (ACTREAL)
    Chosen per dataset; no sensitivity analysis is reported.
assumptions (4)
  • domain assumption User actions can be fully described by five operation types: click, select, scroll, drag, and type.
    The entire annotation and evaluation is built on this taxonomy (Section III).
  • domain assumption Frame differences detected by thresholding and connected components correspond to user actions.
    Foundation of the DiffF method (Section III-B).
  • ad hoc to paper BERT embedding cosine similarity with a threshold of 0.7 is a valid semantic equivalence measure for UI element descriptions.
    Used to compute ground-truth matches; only validated on 9 replay cases (Section IV-B).
  • domain assumption VLM outputs follow the requested JSON structure reliably enough that parsing errors are negligible.
    No analysis of parse failures is reported in the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Sharingan: Extract User Action Sequence from Desktop Recordings." pith.science (2026). https://pith.science/paper/SRSYAZ25

@misc{pith2026241108768,
  author       = {Pith},
  title        = {Pith review of: Sharingan: Extract User Action Sequence from Desktop Recordings},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SRSYAZ25}},
  note         = {Machine review of arXiv:2411.08768}
}
read the original abstract

Video recordings of user activities, particularly desktop recordings, offer a rich source of data for understanding user behaviors and automating processes. However, despite advancements in Vision-Language Models (VLMs) and their increasing use in video analysis, extracting user actions from desktop recordings remains an underexplored area. This paper addresses this gap by proposing two novel VLM-based methods for user action extraction: the Direct Frame-Based Approach (DF), which inputs sampled frames directly into VLMs, and the Differential Frame-Based Approach (DiffF), which incorporates explicit frame differences detected via computer vision techniques. We evaluate these methods using a basic self-curated dataset and an advanced benchmark adapted from prior work. Our results show that the DF approach achieves an accuracy of 70% to 80% in identifying user actions, with the extracted action sequences being re-playable though Robotic Process Automation. We find that while VLMs show potential, incorporating explicit UI changes can degrade performance, making the DF approach more reliable. This work represents the first application of VLMs for extracting user action sequences from desktop recordings, contributing new methods, benchmarks, and insights for future research.

Figures

Figures reproduced from arXiv: 2411.08768 by the authors.

Figure 1
Figure 1. Architectures of Direct Frame-Based Approach (left) and Differential Frame-Based Approach (right). [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. An example output from Frame Difference Localizer. Left and right are the changed regions with red bounding boxes [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. An example output of Frame Difference Descriptor. It is derived from the identified UI changes in Figure 2. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 29 canonical work pages

  1. [1]

    Florian Bordes, Richard Yuanzhe Pang, Anurag Ajay, Alexander C. Li, Adrien Bardes, Suzanne Petryk, Oscar Ma ˜nas, Zhiqiu Lin, Anas Mahmoud, Bargav Jayaraman, Mark Ibrahim, Melissa Hall, Yunyang Xiong, Jonathan Lebensold, Candace Ross, Srihari Jayakumar, Chuan Guo, Diane Bouchacourt, Haider Al-Tahan, Karthik Padthe, Vasu Sharma, Hu Xu, Xiaoqing Ellen Tan, ...

    work page 2024

  2. [2]

    G. Bradski. The OpenCV Library. Dr . Dobb’s Journal of Software Tools, 2000

    2000

  3. [3]

    Gui-world: A dataset for gui-oriented multimodal llm-based agents, 2024

    Dongping Chen, Yue Huang, Siyuan Wu, Jingyu Tang, Liuyi Chen, Yilin Bai, Zhigang He, Chenlong Wang, Huichi Zhou, Yiqiang Li, Tianshuo Zhou, Yue Yu, Chujie Gao, Qihui Zhang, Yi Gui, Zhen Li, Yao Wan, Pan Zhou, Jianfeng Gao, and Lichao Sun. Gui-world: A dataset for gui-oriented multimodal llm-based agents, 2024

    work page 2024

  4. [4]

    Vg4d: Vision-language model goes 4d video recog- nition

    Zhichao Deng, Xiangtai Li, Xia Li, Yunhai Tong, Shen Zhao, and Mengyuan Liu. Vg4d: Vision-language model goes 4d video recog- nition. In 2024 IEEE International Conference on Robotics and Automation (ICRA) , pages 5014–5020, 2024

    work page 2024

  5. [5]

    Internlm-xcomposer2-4khd: A pioneering large vision-language model handling resolutions from 336 pixels to 4k hd, 2024

    Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Zhe Chen, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang, Kai Chen, Conghui He, Xingcheng Zhang, Jifeng Dai, Yu Qiao, Dahua Lin, and Jiaqi Wang. Internlm-xcomposer2-4khd: A pioneering large vision-language mode...

    work page 2024

  6. [6]

    Videoagent: A memory-augmented multimodal agent for video understanding, 2024

    Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. Videoagent: A memory-augmented multimodal agent for video understanding, 2024

    work page 2024

  7. [7]

    Video-of-thought: Step-by-step video reasoning from perception to cognition

    Hao Fei, Shengqiong Wu, Wei Ji, Hanwang Zhang, Meishan Zhang, Mong-Li Lee, and Wynne Hsu. Video-of-thought: Step-by-step video reasoning from perception to cognition. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st Inter- national Conference on M...

    work page 2024

  8. [8]

    Gemini models

    Google. Gemini models. https://deepmind.google/ technologies/gemini/, 2024. Accessed: 2024-09-15

    work page 2024

Show all 34 references
  1. [9]

    Humane. Ai pin. https://humane.com/aipin, 2024. Accessed: 2024-09-15

    2024

  2. [10]

    Smartflow: Robotic process automation using llms, 2024

    Arushi Jain, Shubham Paliwal, Monika Sharma, Lovekesh Vig, and Gautam Shroff. Smartflow: Robotic process automation using llms, 2024

    2024

  3. [11]

    Dream2real: Zero-shot 3d object rearrangement with vision-language models

    Ivan Kapelyukh, Yifei Ren, Ignacio Alzugaray, and Edward Johns. Dream2real: Zero-shot 3d object rearrangement with vision-language models. In 2024 IEEE International Conference on Robotics and Automation (ICRA) , pages 4796–4803, 2024

    2024

  4. [12]

    Wolf: Captioning everything with a world summarization framework, 2024

    Boyi Li, Ligeng Zhu, Ran Tian, Shuhan Tan, Yuxiao Chen, Yao Lu, Yin Cui, Sushant Veer, Max Ehrlich, Jonah Philion, Xinshuo Weng, Fuzhao Xue, Andrew Tao, Ming-Yu Liu, Sanja Fidler, Boris Ivanovic, Trevor Darrell, Jitendra Malik, Song Han, and Marco Pavone. Wolf: Captioning ever...

    2024

  5. [13]

    Videochat: Chat-centric video understanding, 2023

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding, 2023

    2023

  6. [14]

    Llava-next: Improved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024

    2024

  7. [15]

    Visualwebbench: How far have multi- modal llms evolved in web page understanding and grounding?, 2024

    Junpeng Liu, Yifan Song, Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li, and Xiang Yue. Visualwebbench: How far have multi- modal llms evolved in web page understanding and grounding?, 2024

    2024

  8. [16]

    Online robot navigation and manipulation with distilled vision-language models, 2024

    Kangcheng Liu. Online robot navigation and manipulation with distilled vision-language models, 2024

    2024

  9. [17]

    Robollm: Robotic vision tasks grounded on multimodal large language models

    Zijun Long, George Killick, Richard McCreadie, and Gerardo Aragon- Camarasa. Robollm: Robotic vision tasks grounded on multimodal large language models. In 2024 IEEE International Conference on Robotics and Automation (ICRA) , pages 12428–12435, 2024

    2024

  10. [18]

    Omniparser for pure vision based gui agent, 2024

    Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. Omniparser for pure vision based gui agent, 2024

    2024

  11. [19]

    OBS. Obs. https://obsproject.com/, 2024. Accessed: 2024-09-15

    2024

  12. [20]

    OpenAI. Gpt-4o. https://platform.openai.com/docs/models/ gpt-4o, 2024. Accessed: 2024-09-15

    2024

  13. [21]

    OpenAI. Gpt-4v. https://platform.openai.com/docs/models/ gpt-4-turbo-and-gpt-4 , 2024. Accessed: 2024-09-15

    2024

  14. [22]

    Owl - always-on wearable ai

    Owl. Owl - always-on wearable ai. https://github.com/ OwlAIProject/Owl, 2024. Accessed: 2024-09-15

    2024

  15. [23]

    bert-embedding

    pypi. bert-embedding. https://pypi.org/project/ bert-embedding/, 2024. Accessed: 2024-09-15

    2024

  16. [24]

    Learning transferable visual models from natural language supervision, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021

    2021

  17. [25]

    Beltran-Hernandez, Masashi Hamaya, Atsushi Hashimoto, Shohei Tanaka, Kento Kawaharazuka, Kazutoshi Tanaka, Yoshitaka Ushiku, and Shinsuke Mori

    Keisuke Shirai, Cristian C. Beltran-Hernandez, Masashi Hamaya, Atsushi Hashimoto, Shohei Tanaka, Kento Kawaharazuka, Kazutoshi Tanaka, Yoshitaka Ushiku, and Shinsuke Mori. Vision-language interpreter for robot task planning. In 2024 IEEE International Conference on Robotics an...

    2024

  18. [26]

    Karlsson, Bo An, Shuicheng Yan, and Zongqing Lu

    Weihao Tan, Wentao Zhang, Xinrun Xu, Haochong Xia, Ziluo Ding, Boyu Li, Bohan Zhou, Junpeng Yue, Jiechuan Jiang, Yewen Li, Ruyi An, Molei Qin, Chuqiao Zong, Longtao Zheng, Yujie Wu, Xiaoqiang Chai, Yifei Bi, Tianbao Xie, Pengjie Gu, Xiyun Li, Ceyao Zhang, Long Tian, Chaojie Wa...

    2024

  19. [27]

    Sch ¨onberger, Juan Nunez-Iglesias, Franc ¸ois Boulogne, Joshua D

    St ´efan van der Walt, Johannes L. Sch ¨onberger, Juan Nunez-Iglesias, Franc ¸ois Boulogne, Joshua D. Warner, Neil Yager, Emmanuelle Gouil- lart, Tony Yu, and the scikit-image contributors. scikit-image: image processing in Python. PeerJ, 2:e453, 2014

    2014

  20. [28]

    Drive anywhere: Generalizable end-to-end autonomous driving with multi- modal foundation models, 2023

    Tsun-Hsuan Wang, Alaa Maalouf, Wei Xiao, Yutong Ban, Alexander Amini, Guy Rosman, Sertac Karaman, and Daniela Rus. Drive anywhere: Generalizable end-to-end autonomous driving with multi- modal foundation models, 2023

    2023

  21. [29]

    Videoagent: Long-form video understanding with large language model as agent, 2024

    Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. Videoagent: Long-form video understanding with large language model as agent, 2024

    2024

  22. [30]

    Videotree: Adaptive tree- based video representation for llm reasoning on long videos, 2024

    Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin, Jaehong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. Videotree: Adaptive tree- based video representation for llm reasoning on long videos, 2024

    2024

  23. [31]

    Incorporating scene graphs into pre- trained vision-language models for multimodal open-vocabulary action recognition

    Chao Wei and Zhidong Deng. Incorporating scene graphs into pre- trained vision-language models for multimodal open-vocabulary action recognition. In 2024 IEEE International Conference on Robotics and Automation (ICRA) , pages 440–447, 2024

    2024

  24. [32]

    Vlfm: Vision-language frontier maps for zero- shot semantic navigation

    Naoki Yokoyama, Sehoon Ha, Dhruv Batra, Jiuguang Wang, and Bernadette Bucher. Vlfm: Vision-language frontier maps for zero- shot semantic navigation. In 2024 IEEE International Conference on Robotics and Automation (ICRA) , pages 42–48, 2024

    2024

  25. [33]

    Ufo: A ui-focused agent for windows os interaction, 2024

    Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Qin Zhang. Ufo: A ui-focused agent for windows os interaction, 2024

    2024

  26. [34]

    global_description

    Zhen Zhang, Anran Lin, Chun Wai Wong, Xiangyu Chu, Qi Dou, and K. W. Samuel Au. Interactive navigation in environments with traversable obstacles using large language and vision-language models. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 786...

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.