Proactive Assistant Dialogue Generation from Streaming Egocentric Videos

Andrea Madotto; Anuj Kumar; Babak Damavandi; Joyce Chai; Seungwhan Moon; Xin Luna Dong; Yichi Zhang; Zhaojiang Lin

REVIEW 4 major objections 5 minor 27 references

Proactive Assistant Dialogue Generation from Streaming Egocentric Videos

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Synthetic dialogues from egocentric videos can match human-collected ones, enabling scalable training and evaluation of proactive assistants that guide users through real tasks in real time.

desk verdict A genuinely useful synthetic dataset and task formulation, but the benchmark claim is validated inside the generator's distribution, not against real task guidance. read the letter →

arxiv 2506.05904 v1 pith:3FSL73J4 submitted 2025-06-06 cs.AI cs.CLcs.CVcs.HC

Yichi Zhang , Xin Luna Dong , Zhaojiang Lin , Andrea Madotto , Anuj Kumar , Babak Damavandi , Joyce Chai , Seungwhan Moon This is my paper · ORCID

classification cs.AIcs.CLcs.CVcs.HC

keywords proactiveassistantegocentricvideostreamingdialoguegenerationsyntheticdatasettaskguidanceLLM-as-a-judgeresponsetimingnegativeframesub-sampling

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that proactive task-guidance assistants—systems that watch streaming egocentric video and decide when and what to say—can be trained and evaluated without expensive human-collected interaction data. The claim, supported by human studies, is that a large synthetic dialogue dataset generated by an LLM from timestamped video annotations matches or outperforms human-collected dialogues on quality, and that two automatic metrics (pairwise sentence matching and LLM-as-a-judge) correlate with human judgment. If true, the field gains a reusable 30,135-dialogue dataset across cooking, assembly, object manipulation, and laboratory domains, plus evaluation tools that make rapid model comparison feasible.

What carries the argument

The data curation pipeline uses LLaMA-3.1-70B-Instruct to convert standardized timestamped video annotations into task goals, recipes, and multi-round user-assistant dialogues, generating 10 dialogues per video across three user types (no talk, talk some, talk more), followed by refinement, intent labeling, quality scoring, and filtering. The evaluation uses pairwise F1 computed via bipartite matching with semantic and temporal costs, plus an LLM-as-a-judge scoring four dimensions; the model extends VideoLLM-Online with negative frame sub-sampling for speaking decisions and iterative progress summarization for long videos.

What would settle it

A human-in-the-loop experiment where a model trained on PROASSIST assists people performing real physical tasks (e.g., cooking or assembly in a new kitchen or workspace) and is compared against a system trained on human-collected dialogues, measuring task completion time, error rate, and user ratings of intervention timing; if the PROASSIST-trained model performs substantially worse on real-time guidance despite scoring well on PROASSIST test sets, the benchmark's reliability as a proxy for real assistance would be refuted.

Watch

Extended reading notes

Core claim

The central claim is that PROASSIST is a reliable benchmarking resource for proactive assistant dialogue generation: the synthetic dialogues match or outperform human-collected ones in human evaluation (Table 2), and the proposed automatic metrics (pairwise F1 and LLM-as-a-judge) correlate with human judgment (Table 3). If correct, the field gains a large-scale, multi-domain dataset and reusable evaluation tools.

Load-bearing premise

The load-bearing premise is that dialogues synthesized by LLaMA-3.1-70B from timestamped text descriptions are a valid proxy for real human-AI task guidance; if these synthetic dialogues systematically diverge from genuine interaction (alignment score 2.91/4, as the paper concedes), training and evaluation built on them may favor matching the generator's style over providing correct, well-timed assistance.

Editorial extensions

If this is right

If PROASSIST is reliable, training data for proactive assistants can be synthesized from existing annotated egocentric videos at scale across multiple domains, avoiding costly Wizard-of-Oz collection.
The pairwise F1 metric can be applied to other streaming video-to-text tasks requiring joint timing and content evaluation, such as online action narration.
The LLM-as-a-judge metric with knowledge-conditioned evaluation (providing recipes) is recommended as the standard setup, since it aligns with human judgment and accounts for multiple valid guidance strategies.
Iterative progress summarization enables handling of arbitrary-length video streams during training and inference, potentially enabling hour-long task guidance.
The finding that better visual perception helps action narration but not dialogue generation identifies higher-level reasoning, progress tracking, and knowledge application as the bottlenecks for task guidance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's validation of metrics is limited to its own models and data; a natural next step is to test pairwise F1 and LLM-as-a-judge on independently collected human-human or human-AI task-guidance dialogues to see if the correlations hold across distribution shifts.
Because the weakest known point is dialogue-video alignment (2.91/4), training models on PROASSIST may reward matching the generator's style rather than providing genuinely well-timed assistance; this could be tested by measuring whether models trained on PROASSIST transfer to real-time guidance in unseen environments.
The dataset's dependence on existing timestamped action annotations limits scalability; the paper notes future LLMs could generate dialogues directly from raw video, which would remove the annotation bottleneck but raise new questions about factual grounding.
The negative frame sub-sampling technique suggests that speaking decisions in streaming dialogue can be treated as a class-imbalance problem; this idea could generalize to other event-triggered streaming generation tasks such as live sports commentary or real-time navigation instructions.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's the short version: this is a resource paper, and the resource is real—a 30k-dialogue synthetic dataset for proactive assistant dialogue from streaming egocentric video, plus paired evaluation metrics and a baseline model. The benchmark claim ("reliable resource for the field") is the part I'd push back on: it's validated almost entirely inside the distribution of the LLM that generated the data.

What's good: the task formulation is new and well specified. PROASSIST spans six egocentric datasets, 479 hours, four domains—that's a large, useful resource for the streaming-assistant and AR crowd. The data pipeline is careful: chunked generation, refinement, post-filtering, intent annotation, progress summaries. The human evaluation of dataset quality is substantive, not a token check. The metrics work does what most metric papers don't: it runs human correlation studies and reports the correlations honestly. The model is a baseline, not a breakthrough, but negative frame sub-sampling and iterative progress summarization are sensible, and the ablations are clearly reported.

Now the soft spots. The stress-test note lands. Dialogues are generated by LLaMA-3.1-70B from timestamped text; the LLM judge is the same model family; the trained model is LLaMA-3.1-8B. Training and evaluation both live in LLaMA-3.1's simulation of assistant behavior. The paper's own human eval shows video-dialogue alignment at 2.91/4—the lowest dimension—and the Limitations section concedes it "requires enhancement." The direct comparison to human-collected dialogues covers only HoloAssist and WTaG, with WTaG at 7 test dialogues and no significance testing, so "matches or outperforms" is not established. The metric validation uses synthetic references from the same generator, so the metrics track human preferences within PROASSIST's distribution, not whether that distribution matches real task guidance. The θ selection on validation F1 is fine as a practical choice; it just ties another knot to the synthetic reference.

How soft is this, really? The circularity doesn't sink the dataset as a training resource—people building streaming task-guidance models can still train on it and test later on real interaction. But the central claim that PROASSIST is a reliable proxy for real human-AI task guidance is asserted more strongly than tested. I'd want either a human-in-the-loop evaluation with actual users or a genuinely held-out set of human-collected dialogues used as test references across all domains, not two with one tiny. Practical caveat: data and code are promised but not yet released, so the resource value is currently potential.

Who it's for: researchers working on proactive assistants from egocentric video, and anyone building evaluation for streaming dialogue. It deserves a serious referee. I'd send it to review, conditionally: require artifact release and a softer, evidence-matching claim about the proxy.

Referee Report

4 major / 5 minor

Summary. The paper introduces PROASSIST, a large-scale synthetic dialogue dataset for proactive assistant dialogue generation from streaming egocentric videos, built by using LLaMA-3.1-70B to synthesize timestamped assistant-user dialogues from annotated egocentric videos across six datasets. It also proposes two automatic evaluation metrics—pairwise F1 via sentence matching and LLM-as-a-judge—and an end-to-end model based on VideoLLM-Online with negative frame sub-sampling and iterative progress summarization. The central claim is that PROASSIST and the proposed metrics constitute a reliable benchmarking resource for this new task, supported by human evaluation of synthetic dialogue quality and by studies showing that the metrics correlate with human judgment.

Significance. If the benchmarking claim holds, the paper provides a valuable multi-domain resource (30,135 dialogues over 479 hours of video) and a reusable offline evaluation protocol for a difficult streaming task. The strengths include the scale and diversity of the dataset, the concretely described pipeline with prompts and post-filtering, the human-evaluation effort (81% inter-rater agreement), and the honest reporting that model performance is still below acceptable thresholds (overall scores below 3/5). The metric-validation study is a useful step for timing-aware dialogue evaluation. However, the significance is conditional on demonstrating that synthetic LLaMA-generated dialogues transfer to real task guidance and that the metrics measure task-guidance quality rather than stylistic similarity to the generator. These conditions are not yet established, as detailed in the major comments.

major comments (4)

[§6.2, Table 2] The load-bearing premise that synthetic dialogues are a valid proxy for real human-AI task guidance is not established. The overall dialogue-video alignment score is 2.91/4.00, the lowest dimension, and the Limitations section concedes that alignment 'requires enhancement.' The side-by-side comparison with human dialogues covers only HoloAssist and WTaG, and the WTaG subset contains only 7 test videos and 21 test dialogues (Table 1); no significance tests are reported, so the claim that synthetic dialogues 'match or outperform' human-collected ones is not statistically supported. To support the benchmark claim, the authors should report significance tests on the human evaluation, enlarge the human-collected comparison, and ideally run an interaction-based study (e.g., users performing tasks with assistant guidance) to measure whether synthetic-dialogue-trained systems actually improve task success.
[§6.3, Table 3] The metric validation is circular in an important sense: the ground-truth references are generated by LLaMA-3.1-70B, the LLM judge is the same model family, and the evaluated model backbone is LLaMA-3.1-8B. Correlations computed on this closed loop may reward stylistic similarity to the generator rather than correct, well-timed assistance. The reported correlations (F1 vs human 0.35, Overall vs human 0.47) are moderate, and the paper does not validate the metrics on human-collected dialogues (e.g., HoloAssist or WTaG held-out) or against task-outcome measures. Please add metric validation on human-collected references and a concrete test such as measuring correlation between metric scores and task success or completion time; this would break the circularity and strengthen the benchmark claim.
[§5.2, §6.1, Table 4] The speaking threshold θ and negative sampling ratio ρ are tuned on the validation F1 score, which is the same metric used for the reported results. Table 4 shows that F1-based threshold selection agrees with human preference in only 0.67 of dialogue-generation cases, so the tuning procedure can introduce optimistic bias and does not fully align with human judgment. Please report results across the range of θ and ρ (e.g., sensitivity curves or error bars), and consider selecting hyperparameters based on metric-human agreement or on a separate held-out set that is not used for metric development.
[§4, Appendix B.1] The pairwise matching metric has several hand-set parameters—semantic similarity threshold (0.5), temporal cost exponent p=1.5, and per-dataset time cutoffs R and L—chosen heuristically without a robustness analysis. Since the F1 metric is used to select model variants and thresholds, variation in these parameters could change model rankings. Please include an ablation showing that the reported conclusions (e.g., Table 5 comparisons) are stable under reasonable variation in these parameters, or specify how the parameters were chosen from data without overfitting.

minor comments (5)

[§6.4] There is a typo: 'Becides' should be 'Besides.'
[Figure 1] The example dialogue text appears duplicated in the figure/caption; please clean up the typesetting so that the example conversation is shown once.
[Appendix A.4] The sentence 'See Figure 6-11 or example dialogues' should read 'See Figure 6-11 for example dialogues.'
[References] In the LAPJV reference, the author name is misspelled as 'V olgenant'; it should be 'Volgenant.'
[Table 3 caption] The caption should state explicitly that Pearson and Spearman coefficients are reported, and should clarify the sample size (50 tasks) and the number of annotators used for the human ranking.

Circularity Check

1 steps flagged · score 2.0 of 10

Mostly self-contained with independent human validation; a partial generator-judge loop affects the automatic metric but not the central claim.

other [Section 4 (End-to-End Evaluation via LLM-as-a-Judge); Appendix B.2; Section 3.2 (data curation)]
"Given timestamped predictions and reference dialogues, we prompt an LLM to evaluate system performance across four dimensions: correctness of guidance and feedback, appropriateness of response timing, efficiency of information delivery, and overall helpfulness. ... We use LLaMA-3.1-70B-Instruct as the evaluator in our experiments. ... We leverage LLaMA-3.1-70B-Instruct (Dubey et al., 2024) as the LLM for all the aforementioned steps."

The gold-standard reference dialogues against which predictions are judged were generated by LLaMA-3.1-70B-Instruct, and the judge is the same LLaMA-3.1-70B-Instruct. The judge rubric asks whether a prediction is correct 'based on the context and the gold-standard reference,' so the automatic Overall score partly measures agreement with the generator's own dialogue style. Because the trained assistant (LLaMA-3.1-8B) is trained on those same LLaMA-generated references, model development and automatic evaluation share one distribution, creating a closed loop for the automatic metric itself.

full rationale

The paper's central claims are supported by external human judgment rather than by self-definition. The synthetic dialogue quality claim rests on human ratings of correctness, helpfulness, alignment, and naturalness (Table 2), including a direct comparison against human-collected dialogues. The metric-validation claim rests on human rankings of model outputs and human selection of the best speaking threshold (Tables 3 and 4). Hyperparameters such as the speaking threshold and NFS sampling ratio are chosen on validation F1 and then reported on test F1, which is standard practice rather than a fitted parameter being renamed as a prediction. I found no load-bearing self-citation, no imported uniqueness theorem, and no ansatz smuggled in via citation. The only notable circular aspect is that the LLM-as-a-judge evaluator and the synthetic ground-truth dialogues share the same generator family (LLaMA-3.1-70B-Instruct), which can reward stylistic mimicry of the generator; however, because the paper validates the metric and data against human judgments, this loop does not reduce the central derivation to its inputs. Score 2 reflects this minor, non-load-bearing circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claims rest on the synthetic dialogues being a valid proxy for real task guidance and on the evaluation metrics correlating with human judgment. The human studies provide independent grounding, but the LLM-generated ground truth, the hand-tuned metric parameters, and the selection of thresholds on validation F1 add assumptions a reader should track.

free parameters (5)

Speaking threshold theta = 0.2 to 0.5 depending on subset and model (Table 11)
Controls the silence/speak decision at inference; selected per subset using validation F1, and the paper notes high sensitivity.
Negative frame sub-sampling ratio rho = 0.1
Proportion of negative (silent) frames used for gradient computation; chosen based on validation F1 in Table 6.
Semantic similarity threshold = 0.5
Threshold in the pairwise matching metric for considering a prediction matched to a reference (Appendix B.1).
Temporal cost parameters (p, R, L) = p=1.5; R=2.5s action narration, 1.5-6.0s dialogue; L=R/2
Defines the allowed timing offset in the pairwise matching cost (Appendix B.1).
Post-filtering quality thresholds = score < 3 removed from training; score < 5 removes videos from validation
Hand-set cutoffs in the automatic dialogue quality filter (Section 3.2, Appendix A.1).

assumptions (5)

domain assumption Timestamped egocentric video annotations from the six source datasets are accurate enough to serve as the basis for dialogue synthesis.
The pipeline converts action descriptions and error labels into dialogues; errors in annotations propagate to dialogues. Section 3.2.
domain assumption LLM-generated dialogues from text video descriptions can serve as a reliable training signal for video-grounded assistance.
Central to PROASSIST; the model is trained to match these synthetic dialogues. Section 3.2 and 5.
domain assumption Human ratings on a 4-point scale are a valid measure of dialogue quality.
Used to validate data and metrics. Appendix D.
ad hoc to paper The pairwise matching metric with hand-set thresholds approximates human judgment of assistance quality.
The cost matrix combines cosine similarity and temporal distance with hand-set R, L, p, threshold 0.5. Section 4 and Appendix B.1.
ad hoc to paper The LLM-as-a-judge (LLaMA-3.1-70B) provides unbiased quality scores for dialogues of a related model family.
Used to compute Correctness, Promptness, Efficiency, Overall; not independently calibrated against human scores per item. Section B.2.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Proactive Assistant Dialogue Generation from Streaming Egocentric Videos." pith.science (2026). https://pith.science/paper/3FSL73J4

@misc{pith2026250605904,
  author       = {Pith},
  title        = {Pith review of: Proactive Assistant Dialogue Generation from Streaming Egocentric Videos},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3FSL73J4}},
  note         = {Machine review of arXiv:2506.05904}
}

read the original abstract

Recent advances in conversational AI have been substantial, but developing real-time systems for perceptual task guidance remains challenging. These systems must provide interactive, proactive assistance based on streaming visual inputs, yet their development is constrained by the costly and labor-intensive process of data collection and system evaluation. To address these limitations, we present a comprehensive framework with three key contributions. First, we introduce a novel data curation pipeline that synthesizes dialogues from annotated egocentric videos, resulting in \dataset, a large-scale synthetic dialogue dataset spanning multiple domains. Second, we develop a suite of automatic evaluation metrics, validated through extensive human studies. Third, we propose an end-to-end model that processes streaming video inputs to generate contextually appropriate responses, incorporating novel techniques for handling data imbalance and long-duration videos. This work lays the foundation for developing real-time, proactive AI assistants capable of guiding users through diverse tasks. Project page: https://pro-assist.github.io/

Figures

Figures reproduced from arXiv: 2506.05904 by the authors.

**Figure 2.** A cooking task example from [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Left: Streaming video-to-dialogue generation using VideoLLM-Online. The model processes live video frames and optional textual inputs to decide whether to speak or remain silent at designated decision points (yellow stars), and autogressively generates assistant responses as needed. Learning when to speak faces significant class imbalance due to the sparsity of speaking frames. Right: Illustration of iterative progr… view at source ↗

Figures from the paper (8 more)

**Figure 4.** Figure 4: Model performance under different speaking [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: PROASSIST dataset statistics overview including task domain distribution, dialogue structure, duration variability, and assistant interaction patterns. Mean values for each distribution are provided in the legend. - Correctness: does each generated instruction/feedback…

**Figure 6.** Figure 6: Example of a cooking task in PROASSIST [PITH_FULL_IMAGE:figures/full_fig_p023_6.png]

**Figure 7.** Figure 7: Example of a cooking task in PROASSIST. 10 [PITH_FULL_IMAGE:figures/full_fig_p023_7.png]

**Figure 8.** Figure 8: Example of a cooking task in PROASSIST [PITH_FULL_IMAGE:figures/full_fig_p024_8.png]

**Figure 9.** Figure 9: Example of a assembly task in PROASSIST. 11 [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]

**Figure 10.** Figure 10: Example of a laboratory task in PROASSIST [PITH_FULL_IMAGE:figures/full_fig_p025_10.png]

**Figure 11.** Figure 11: Example of a cooking task in PROASSIST. 12 [PITH_FULL_IMAGE:figures/full_fig_p025_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 22 canonical work pages

[1]

perform the task roughly following the {knowledge_type} (**no** need to be strict), OR

work page
[2]

In Proceedings of the 2023 Conference on Empiri- cal Methods in Natural Language Processing, pages 10462–10479

Gazevqa: A video question answering dataset for multiview eye-gaze task-oriented collaborations. In Proceedings of the 2023 Conference on Empiri- cal Methods in Natural Language Processing, pages 10462–10479. Roy Jonker and Ton V olgenant. 1988. A shortest aug- menting path algorithm for dense and sparse linear assignment problems. In DGOR/NSOR: Papers of...

arXiv 2023
[3]

Make them more clear and helpful when necessary, but keep them concise and to the point in general

Decide the length of assistant messages smartly. Make them more clear and helpful when necessary, but keep them concise and to the point in general

work page
[4]

make sure

Avoid repeating the same talking patterns or phrases. For example, do not say "make sure ..." for every instruction

work page
[5]

as I have mentioned this earlier

Rephrase impolite or inappropriate language, such as "as I have mentioned this earlier ...", to be more friendly and helpful. But keep concise and to the point

work page
[6]

Do not just copy paste the original dialog! Listing 7: Prompt for dialogue refinement and intent labeling for assistant turns

Remove anything other than the dialog itself, such as the user’s actions or explanations of how the dialog is generated. Do not just copy paste the original dialog! Listing 7: Prompt for dialogue refinement and intent labeling for assistant turns. Dialogue Annotation To facilitate the analysis of our generated dialogues, we use LLM to anno- tate the initi...

work page
[8]

In Euro- pean Conference on Computer Vision , pages 485–

Assistq: Affordance-centric question-driven task completion for egocentric assistant. In Euro- pean Conference on Computer Vision , pages 485–

work page
[10]

Okay." - [15.1s] assistant:

with GoalStep annotations (Song et al., 2024b), EpicKitchen (Damen et al., 2020, 2022), HoloAssist (Wang et al., 2023b), Assembly101 (Sener et al., 2022), EgoExoLearn (Huang et al., 2024), and WTaG (Bao et al., 2023). Detailed statistics and label types are summarized in Table 9. When generating timestamped video descriptions for these videos, we leverage...

work page 2020

Show all 27 references

[11]

Here is a video description of an experienced user working on the task - {goal_description}: {video_descriptions} Try to infer the **high-level** recipe from the descriptions

Next, we refine these recipes into a single cohe- sive and integrated version by calling the LLM one more time with the prompt in List 3. Here is a video description of an experienced user working on the task - {goal_description}: {video_descriptions} Try to infer the **high-l...

2023
[13]

Final answer: 1, 2 or 0

perform other tasks (or another trial of the same task) simultaneously in a multi-tasking manner? Answer with your analysis, and end your response with "Final answer: 1, 2 or 0" (0 denotes that the activity is not related to {domain}). Listing 4: Prompt for video pre-filtering...
[14]

Merge turns that are close in time (less than 1 second apart) into a single turn, when the content is similar or related
[15]

Use more coreference and pronouns to make the dialog more coherent and human-like
[20]

The task goal mentioned by the user
[21]

Other topics mentioned by the user in the conversation, if any
[22]

Be faithful and try to include all the relevant information

The current state/step of the task. Be faithful and try to include all the relevant information. Give your response in plain text of a single line in the following format: SUMMARY: <progress summary> Listing 8: Prompt for progress summary generation Automatic Quality Evaluatio...
[23]

The model is hosted locally using vLLM (Kwon et al., 2023), running a FP8-quantized version5 on four H100 GPUs

as the LLM for all the steps described above. The model is hosted locally using vLLM (Kwon et al., 2023), running a FP8-quantized version5 on four H100 GPUs. Although we use a specific LLM for data generation, our pipeline is model-agnostic and can be readily adapted to more a...

2023
[24]

Keep you analysis concise and to the point

Read dialogues carefully and compare them line by line. Keep you analysis concise and to the point
[25]

Evaluate the following aspects: 6https://huggingface.co/sentence-transformers/ all-mpnet-base-v2 5 Cooking 58.3% Object Manipulation 25.4% Assembly 12.0% Laboratory T asks 4.3% (a) Task Domain Distribution. 0 50 100 150 200 Number of T urns 0 200 400 600Number of Dialogues epi...
[26]

C Model Implementation Details C.1 VideoLLM-Online Model Implementation We use LLaMA-3.1-8B-Instruct (Dubey et al.,

For each aspect, give a score from 1 to 5 based on the following criteria: - 1=very poor: most of utterances are incorrect, irrelevant, mistimed, inefficient etc - 2=poor: bad utterances that are incorrect, irrelevant, mistimed are more than good ones - 3=average: the number o...
[27]

Assis- tant: C opens the fridge

as the base LLM and the pretrained SigLIP- SO400M-14-3847 model (Zhai et al., 2023) as the frame encoder. To extract frame features, we use the embeddings from the second last layer of the [CLS] token, and N × N patch features obtained through average pooling of the correspond...

2023
[501]

Te-Lin Wu, Satwik Kottur, Andrea Madotto, Mahmoud Azab, Pedro Rodriguez, Babak Damavandi, Nanyun Peng, and Seungwhan Moon

Springer. Te-Lin Wu, Satwik Kottur, Andrea Madotto, Mahmoud Azab, Pedro Rodriguez, Babak Damavandi, Nanyun Peng, and Seungwhan Moon. 2023. SIMMC-VR: A task-oriented multimodal dialog dataset with sit- uated and immersive VR streams. In Proceedings of the 61st Annual Meeting of...

2023 arXiv
[2016]

Advanced Engi- neering Informatics, 30(3):406–421

Multi-modal augmented-reality assembly guid- ance based on bare-hand interface. Advanced Engi- neering Informatics, 30(3):406–421. Zhanyu Wang, Longyue Wang, Zhen Zhao, Minghao Wu, Chenyang Lyu, Huayang Li, Deng Cai, Lup- ing Zhou, Shuming Shi, and Zhaopeng Tu. 2024. Gpt4video...

2024
[2018]

arXiv preprint arXiv:1801.04871

Building a conversational agent overnight with dialogue self-play. arXiv preprint arXiv:1801.04871. Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. 2024a. Moviechat: From dense token to sparse...

arXiv
[2020]

In Proceedings of the 28th International Con- ference on Computational Linguistics, pages 1103– 1121

Situated and interactive multimodal conversa- tions. In Proceedings of the 28th International Con- ference on Computational Linguistics, pages 1103– 1121. Seungwhan Moon, Andrea Madotto, Zhaojiang Lin, Tushar Nagarajan, Matt Smith, Shashank Jain, Chun- Fu Yeh, Prakash Murugesa...

2024
[2022]

Advances in neural in- formation processing systems, 35:27730–27744

Training language models to follow instruc- tions with human feedback. Advances in neural in- formation processing systems, 35:27730–27744. Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Mered- ith Ringel Morris, Percy Liang, and Michael S Bern- stein. 2023. Generative agents...

2023 arXiv
[2023]

something something

Can foundation models watch, talk and guide you step by step to make a cake? arXiv preprint arXiv:2311.00738. Tom B Brown. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165. Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Ga...

2020 arXiv
[2024]

arXiv preprint arXiv:2403.05530

Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv preprint arXiv:2403.05530. Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. 2024....

2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

perform the task roughly following the {knowledge_type} (**no** need to be strict), OR

work page

[2] [2]

In Proceedings of the 2023 Conference on Empiri- cal Methods in Natural Language Processing, pages 10462–10479

Gazevqa: A video question answering dataset for multiview eye-gaze task-oriented collaborations. In Proceedings of the 2023 Conference on Empiri- cal Methods in Natural Language Processing, pages 10462–10479. Roy Jonker and Ton V olgenant. 1988. A shortest aug- menting path algorithm for dense and sparse linear assignment problems. In DGOR/NSOR: Papers of...

arXiv 2023

[3] [3]

Make them more clear and helpful when necessary, but keep them concise and to the point in general

Decide the length of assistant messages smartly. Make them more clear and helpful when necessary, but keep them concise and to the point in general

work page

[4] [4]

make sure

Avoid repeating the same talking patterns or phrases. For example, do not say "make sure ..." for every instruction

work page

[5] [5]

as I have mentioned this earlier

Rephrase impolite or inappropriate language, such as "as I have mentioned this earlier ...", to be more friendly and helpful. But keep concise and to the point

work page

[6] [6]

Do not just copy paste the original dialog! Listing 7: Prompt for dialogue refinement and intent labeling for assistant turns

Remove anything other than the dialog itself, such as the user’s actions or explanations of how the dialog is generated. Do not just copy paste the original dialog! Listing 7: Prompt for dialogue refinement and intent labeling for assistant turns. Dialogue Annotation To facilitate the analysis of our generated dialogues, we use LLM to anno- tate the initi...

work page

[7] [8]

In Euro- pean Conference on Computer Vision , pages 485–

Assistq: Affordance-centric question-driven task completion for egocentric assistant. In Euro- pean Conference on Computer Vision , pages 485–

work page

[8] [10]

Okay." - [15.1s] assistant:

with GoalStep annotations (Song et al., 2024b), EpicKitchen (Damen et al., 2020, 2022), HoloAssist (Wang et al., 2023b), Assembly101 (Sener et al., 2022), EgoExoLearn (Huang et al., 2024), and WTaG (Bao et al., 2023). Detailed statistics and label types are summarized in Table 9. When generating timestamped video descriptions for these videos, we leverage...

work page 2020