Pith. sign in

REVIEW 5 major objections 5 minor 59 references

LEGO Co-builder: Exploring Fine-Grained Vision-Language Modeling for Multimodal LEGO Assembly Assistants

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper builds a 35,612-sample LEGO assembly benchmark and shows current vision-language models cannot reliably verify assembly states, with the best state-detection F1 at 40.54%.

desk verdict Useful new benchmark, but the headline T3 result is indistinguishable from a trivial baseline, so the central claim needs significant rework before the paper is citable. read the letter →

arxiv 2507.05515 v2 pith:XBT7QP7R submitted 2025-07-07 cs.AI cs.CLcs.CV

Haochen Huang , Jiahuan Pei , Mohammad Aliannejadi , Xin Sun , Moonisa Ahsan , Chuang Yu , Zhaochun Ren , Pablo Cesar
show 1 more author
Junxiao Wang
This is my paper
classification cs.AIcs.CLcs.CV
keywords LEGOassemblyvision-languagemodelsfine-grainedvisualunderstandingstatedetectionobjectproceduralinstructionfollowingmultimodalbenchmarksyntheticdatageneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that fine-grained procedural understanding, not just coarse object recognition, is the missing capability in current vision-language models, and that a controlled benchmark can expose it. It introduces LEGO Co-builder, a dataset of 35,612 vision-language samples built from 65 official LEGO instruction manuals, spanning three tasks: scene understanding, object detection, and state detection. Across zero-shot and fine-tuned evaluations, the best state-detection result reaches only 40.54% F1, with several models below random guessing. The paper argues this gap shows that step-by-step assembly assistance needs its own benchmark and training resources.

What carries the argument

The central object is the LEGO Co-builder generation pipeline. It takes 65 official LEGO manuals, splits them into object-seeking and object-assembly sessions, and produces three task datasets: scene-description pairs directly from manual steps; a detection task that composes the seeking image with three randomly sampled images and adjusts ground-truth coordinates; and a state-detection task that segments the part to be assembled, shifts it by at least 5% to create three negative variants per positive state, and labels each image correct or incorrect. Evaluation runs through a shared vision-language architecture with task-specific prompt tokens '[grounding]', '[object]', and '[state]', so a single model can be tested on all three tasks under the same interface.

What would settle it

Have independent human annotators draw bounding boxes on a random sample of the object-detection images and recompute F1-Object and IOU against those human boxes instead of the generated boxes; if model rankings or scores change materially, the benchmark measures alignment with the generator rather than true localization. A complementary check would run the state-detection task on real photographs of the same LEGO steps; if fine-tuned models that pass the synthetic negatives fail there, the task captures perturbation detection rather than assembly verification.

Watch

Extended reading notes

Core claim

The central claim is that current vision-language models cannot reliably verify whether an assembly step was completed correctly from a single image. On the proposed benchmark, the best state-detection score is 40.54% F1, and object localization also remains weak: the highest intersection-over-union after fine-tuning is 47.20%, despite object-identification F1 scores above 98%. The authors read this as evidence that fine-grained spatial and state understanding is a distinct bottleneck, and they offer the dataset, generation pipeline, and a one-for-all prompting architecture as a foundation for measuring and improving it.

Load-bearing premise

The load-bearing premise is that the automatically generated ground-truth bounding boxes for object detection are accurate; they were produced by an earlier vision-language model, and only 100 samples were human-validated, with moderate inter-annotator agreement.

Editorial extensions

If this is right

  • Any assistant that claims to guide step-by-step physical assembly should be measured on state-detection F1 and object IOU, not only on entity recognition.
  • Fine-tuning on this dataset improves open-resource models on scene understanding and object detection, so the benchmark doubles as a training resource.
  • Current commercial APIs, even at their best, are not reliable enough to verify an assembly state from an image without further fine-grained supervision.
  • The modular synthetic generation pipeline can be reused to build similar fine-grained benchmarks for other manual-guided assembly tasks without new human annotation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: the state-detection bottleneck is probably about relational verification rather than object knowledge; even a model that perfectly locates a part still has to decide whether its position relative to the build is correct.
  • My inference: the low state-detection F1s may partly reflect a calibration issue, since false-positive rates vary widely across models; re-scoring with balanced class priors would separate perception errors from decision-threshold effects.
  • My inference: because the negative states are synthetic shifts of the same render, the benchmark measures sensitivity to displacement, not full assembly-error detection; real photographs with occlusion and lighting variation would be a stricter follow-up.
  • My inference: the human-quality check sampled only 100 items per task, so a larger independent annotation study would be the natural first extension to pin down how much of the measured gap is real visual difficulty versus label noise.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces LEGO Co-builder, a hybrid benchmark for fine-grained vision-language assembly assistance, with three tasks: scene understanding (T1), object detection (T2), and state detection (T3). The dataset is generated from 65 LEGO instruction manuals with programmatic negative variants, and the paper evaluates nine vision-language models in zero-shot and LoRA fine-tuned settings. The headline result is that GPT-4o achieves only 40.54% F1-State on T3, which the authors interpret as evidence that current VLMs lack fine-grained visual understanding. The paper also reports moderate F1-Object but low IOU scores on T2, and releases the benchmark, codebase, and generation pipeline.

Significance. If the benchmark's labels and metrics are valid, the resource would be a useful controlled testbed for procedural instruction-following, and the public release of the dataset, code, and generation pipeline is a real strength. The three-task decomposition is sensible, and the inclusion of a human quality audit, though small, is good practice. However, the central quantitative claim is not currently supported: the T3 F1-State of 40.54% sits essentially at the always-positive baseline for the 1:3 positive/negative split, and the T2 ground truth is produced by one of the evaluated models, MiniGPT-v2. With trivial baselines and human ceilings reported, the benchmark could still be valuable; as it stands, the paper's evidence for a fine-grained VLM deficit is not established.

major comments (5)
  1. [Variant State Generation / Table 2] The T3 task pairs each positive assembly state with three automatically generated negative variants, so the positive class is 25% of the 10,864 T3 samples. Under this imbalance, a policy that always predicts 'correct' achieves precision 0.25, recall 1.0, and F1 = 0.40, which is within 0.54 points of GPT-4o's reported 40.54% F1-State. The text's statement that all F1-State scores below 50% are 'worse than random guessing' is incorrect for this distribution: a fair-coin guess has expected F1 around 33%, and the maximum achievable F1 for any constant-label policy is 40%. The fine-tuned InstructBLIP result (F1-State 0.02%, FPR 0.00%) is the opposite degenerate all-negative policy. The paper should report always-positive, always-negative, and random-guess baselines, a human ceiling, and confusion matrices or balanced metrics before claiming a VLM perception gap.
  2. [Object Position Inference / Table 2] The T2 ground-truth bounding boxes are 'initially generated by querying the image using MiniGPT-v2' and then adjusted to fit the composite images. MiniGPT-v2 is itself benchmarked against these labels in Table 2, so its F1-Object and IOU scores measure agreement with the labeling model rather than independent object localization. The human validation covers only 100 samples, with boundary precision 1.25/2 and Fleiss Kappa 0.61. Please provide an independent annotation set for at least a held-out subset, report agreement between the evaluated models and the labeling model, and discuss how label-generation bias affects the model ranking.
  3. [Data Quality Assessment] The human audit for T3 reports average state identifiability of 0.74/1.00 with Kappa 0.59, i.e., moderate agreement on whether a negative variant is recognizable. This means a substantial fraction of the 'incorrect' labels may be ambiguous even to human annotators, which directly weakens the validity of F1-State as a measure of model perception. Please report the human ceiling on T3, exclude or re-label low-identifiability samples, and show that the main results are robust to this filtering.
  4. [Dataset Statistics / Table 1] The text says '5,614 instruction steps' but Table 1 reports # Step Overall 10,428; the following sentence then gives 4,814 seeking and 5,614 assembly steps, whose sum matches the table but not the earlier number. The same section says 'covering 3,172 states' while Table 1 reports # State 2,716. These inconsistencies make the dataset description unreliable and must be corrected before the benchmark can be used by others.
  5. [Evaluation Metrics / Table 2] FPR alone is not a sufficient companion to F1 under extreme class imbalance, and the reported FPR values are not comparable across models with different output formats and decision thresholds. Please report full confusion matrices or ROC/AUC for T3, and state whether the reported F1 values are computed per-sample, per-state, or with any thresholding on the model's output probability.
minor comments (5)
  1. [Throughout] There are several typos: 'pertubations' in the Variant State Generation section, 'Compariable' in Table 4, 'after fine-tuing' in Benchmarks, and inconsistent rendering of 'LlaV A'. These should be fixed in a copyedit pass.
  2. [References / Data Quality Assessment] The Fleiss Kappa computation is cited as Scikit-learn, but the URL points to Statsmodels; please correct the citation or the URL.
  3. [Table 2] The superscript symbols ▲, ▽, and − are defined but appear as suffix markers rather than superscripts in the table; please clarify the notation and ensure each model row has a consistent marker.
  4. [Abstract] The abstract's 'maximum F1 score of just 40.54%' should be qualified as F1-State on T3 and should be accompanied by the class distribution and trivial baselines, since the number is meaningless without them.
  5. [Experimental Setup / Benchmarks] The list says nine models but actually includes twelve entries; please reconcile the count and the table contents.

Circularity Check

2 steps flagged · score 6.0 of 10

T2 ground truth is MiniGPT-v2's own output (then MiniGPT-v2 is scored on it), and the T3 40.54% F1 headline coincides with the always-positive baseline forced by the 1:3 split.

  1. fitted input called prediction [Object Position Inference; Table 2]
    "This is initially generated by querying the image using MiniGPT-v2 (Chen et al. 2023), the state-of-the-art model at the time of this work. The composite image ˜Vt′ is created by combining the current image Vt′ with three randomly sampled images. Last, the initial coordinates are adjusted to fit the composite image."

    The T2 reference coordinates (the ground-truth boxes) are produced by MiniGPT-v2. When Table 2 reports MiniGPT-v2's F1-Object/IOU (84.95/85.91 and 26.98/25.94), the score measures how well MiniGPT-v2 reproduces its own generated boxes, not how accurately it localizes objects. For every other model, T2 also measures alignment with this one model's output rather than with independently verified object locations. The only human check covers 100 samples, with boundary precision 1.25/2 and Kappa 0.61, so the remaining 19,000+ T2 labels remain machine-generated. The prediction therefore reduces to reproducing the label generator by construction.

  2. other [Variant State Generation; Benchmark Results]
    "We treated each state from the manual as a positive sample and generated three variant states as negative samples. ... all evaluated VLMs have F1-State scores below 50%, indicating that their predictions are even worse than random guessing."

    The 1:3 positive/negative split is fixed by the generation procedure, so an always-answer-correct policy has precision 0.25, recall 1.0, and F1 = 0.40. GPT-4o's headline 40.54% is within 0.54 points of this trivial policy; the paper's assertion that sub-50% F1 means 'worse than random guessing' is arithmetically incorrect for this class distribution. The headline evidence for a fine-grained VLM deficit therefore coincides with the class-ratio baseline by construction, and the T3 numbers do not independently establish a perceptual gap.

full rationale

The paper's most concrete circularity is in T2: the object-detection ground truth is generated by MiniGPT-v2, and MiniGPT-v2 is then benchmarked against those same labels in Table 2. This is not a fitted parameter renamed as a prediction, but it is the same reduction-by-construction pattern: the evaluated model is one of the label generators. The small human audit (100 samples, moderate Kappa) does not break the self-reference for the full dataset. The T3 headline suffers from a different but related construction artifact: the 1:3 generation rule makes the always-positive policy achieve F1 = 0.40, so GPT-4o's 40.54% is indistinguishable from a trivial baseline, and the paper's 'worse than random guessing' claim is not supported. Apart from these two benchmark-construction issues, I find no self-citation chain, no imported uniqueness theorem, and no ansatz-smuggling; the proposed architecture and the other evaluation results are independent of each other. The central contribution is a new benchmark, and its headline number is substantially compromised by the two reductions above, hence a partial circularity score of 6.

Assumptions & free parameters 2 free parameters · 6 assumptions · 0 invented entities

The central claim rests on the reliability of the generated labels and the validity of the task design. The T2 labels are model-generated, the T3 negative samples are defined by a hand-set shift threshold, and no full-dataset human validation is provided. These are the key assumptions a reader must accept before trusting the benchmark numbers.

free parameters (2)
  • minimum perturbation shift = 5%
    Hand-set threshold that defines T3 negative samples; no sensitivity analysis is shown. If the threshold were smaller, the state-detection task would be harder.
  • number of negative variants per positive state = 3
    Creates a 1:3 positive-to-negative ratio, giving an always-positive F1 baseline of 40%; this choice drives why the headline score is close to chance.
assumptions (6)
  • domain assumption LEGO Audio Instructions manuals provide accurate ground truth for assembly steps and states.
    The entire dataset inherits correctness from these manuals for the blind; errors in the manuals would propagate into all labels.
  • domain assumption MiniGPT-v2's generated bounding boxes are accurate enough to serve as T2 ground truth.
    Object Position Inference uses MiniGPT-v2 output as the label; only 100 samples were human-checked, with boundary precision 1.25/2 and Kappa 0.61.
  • domain assumption K-means color detection plus hue filtering reliably finds the part to be assembled.
    Used in Variant State Generation to segment the part; manual correction is mentioned but not quantified.
  • ad hoc to paper A shifted part (at least 5%) constitutes an incorrect assembly state.
    This operationalizes 'incorrect state' for T3; no user study or physics-based check that such shifts are plausible errors.
  • ad hoc to paper T1, T2, and T3 span the capabilities a LEGO assembly assistant needs.
    The paper states this without evidence; other abilities such as inventory tracking and error recovery are omitted.
  • domain assumption Moderate Kappa on 100 samples generalizes to the full dataset.
    Quality assessment sampled 100 of 19,136 T2 and 10,864 T3 samples; no full-data validation was performed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LEGO Co-builder: Exploring Fine-Grained Vision-Language Modeling for Multimodal LEGO Assembly Assistants." pith.science (2026). https://pith.science/paper/XBT7QP7R

@misc{pith2026250705515,
  author       = {Pith},
  title        = {Pith review of: LEGO Co-builder: Exploring Fine-Grained Vision-Language Modeling for Multimodal LEGO Assembly Assistants},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XBT7QP7R}},
  note         = {Machine review of arXiv:2507.05515}
}
read the original abstract

Vision-language models (VLMs) are facing the challenges of understanding and following multimodal assembly instructions, particularly when fine-grained spatial reasoning and precise object state detection are required. In this work, we explore LEGO Co-builder, a hybrid benchmark combining real-world LEGO assembly logic with programmatically generated multimodal scenes. The dataset captures stepwise visual states and procedural instructions, allowing controlled evaluation of instruction-following, object detection, and state detection. We introduce a unified framework and assess leading VLMs such as GPT-4o, Gemini, and Qwen-VL, under zero-shot and fine-tuned settings. Our results reveal that even advanced models like GPT-4o struggle with fine-grained assembly tasks, with a maximum F1 score of just 40.54\% on state detection, highlighting gaps in fine-grained visual understanding. We release the benchmark, codebase, and generation pipeline to support future research on multimodal assembly assistants grounded in real-world workflows.

Figures

Figures reproduced from arXiv: 2507.05515 by the authors.

Figure 1
Figure 1. An example of incorrect state detection by GPT [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An illustration of the proposed three vision-language model (VLM) tasks, highlighting the core capabilities required [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The architecture for one-for-all vision-language [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Human assessment of data quality. As shown in [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

59 extracted references · 44 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al

    Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  4. [4]

    K.; Afouras, T.; and Grauman, K

    Ashutosh, K.; Ramakrishnan, S. K.; Afouras, T.; and Grauman, K. 2024. Video-mined task graphs for keystep recognition in instructional videos. In NeurIPS

    work page 2024

  5. [5]

    Bai, J.; Bai, S.; Yang, S.; Wang, S.; Tan, S.; Wang, P.; Lin, J.; Zhou, C.; and Zhou, J. 2023. Qwen-VL: A versatile vision-language model for understanding, localization. Text Reading, and Beyond, 2

    work page 2023

  6. [6]

    Bazi, Y.; Rahhal, M. M. A.; Bashmal, L.; and Zuair, M. 2023. Vision-language model for visual question answering in medical imagery. Bioengineering, 10(3): 380

    work page 2023

  7. [7]

    Bellalouna, F.; Luimula, M.; Markopoulos, P.; Markopoulos, E.; and Zipperling, F. 2020. FiAAR: an augmented reality firetruck equipment assembly and configuration assistant technology. In CogInfoCom, 000237--000244. IEEE

    work page 2020

  8. [8]

    Chen, B.; Xu, Z.; Kirmani, S.; Ichter, B.; Sadigh, D.; Guibas, L.; and Xia, F. 2024 a . Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. In CVPR, 14455--14465

    work page 2024

Show all 59 references
  1. [9]

    Chen, F.; Zhang, D.; Chen, X.; Shi, J.; Xu, S.; and Xu, B. 2022. Unsupervised and pseudo-supervised vision-language alignment in visual dialog. In MM, 4142--4153

    2022

  2. [10]

    Chen, J.; Zhu, D.; Shen, X.; Li, X.; Liu, Z.; Zhang, P.; Krishnamoorthi, R.; Chandra, V.; Xiong, Y.; and Elhoseiny, M. 2023. Minigpt-v2: Large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478

    2023 arXiv

  3. [11]

    Chen, L.; Li, J.; Dong, X.; Zhang, P.; Zang, Y.; Chen, Z.; Duan, H.; Wang, J.; Qiao, Y.; Lin, D.; et al. 2024 b . Are we on the right way for evaluating large vision-language models? NeurIPS, 37: 27056--27087

    2024

  4. [12]

    Cheng, A.-C.; Yin, H.; Fu, Y.; Guo, Q.; Yang, R.; Kautz, J.; Wang, X.; and Liu, S. 2024. Spatialrgpt: Grounded spatial reasoning in vision-language models. NeurIPS, 37: 135062--135093

    2024

  5. [13]

    Chi, T.-C.; Shen, M.; Eric, M.; Kim, S.; and Hakkani-Tur, D. 2020. Just ask: An interactive learning framework for vision and language navigation. In AAAI, 2459--2466

    2020

  6. [14]

    Chu, M.; Tsuji, Y.; and Kato, S. 2014. Hue-based object detection in color images. In CVPR, 148--153

    2014

  7. [15]

    Comanici, G.; Bieber, E.; Schaekermann, M.; Pasupat, I.; Sachdeva, N.; Dhillon, I.; Blistein, M.; Ram, O.; Zhang, D.; Rosen, E.; et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv ...

    2025 arXiv

  8. [16]

    Dai, W.; Li, J.; Li, D.; Tiong, A. M. H.; Zhao, J.; Wang, W.; Li, B.; Fung, P.; and Hoi, S. 2023. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. arXiv:2305.06500

    2023 arXiv

  9. [17]

    M.; Fidler, S.; Furnari, A.; Kazakos, E.; Moltisanti, D.; Munro, J.; Perrett, T.; Price, W.; and Wray, M

    Damen, D.; Doughty, H.; Farinella, G. M.; Fidler, S.; Furnari, A.; Kazakos, E.; Moltisanti, D.; Munro, J.; Perrett, T.; Price, W.; and Wray, M. 2018. Scaling egocentric vision: The epic-kitchens dataset. In ECCV

    2018

  10. [18]

    Du, Y.; Liu, Z.; Li, J.; and Zhao, W. X. 2022. A survey of vision-language pre-trained models. arXiv preprint arXiv:2202.10936

    2022 arXiv

  11. [19]

    Fan, X.; Ji, T.; Li, S.; Jin, S.; Song, S.; Wang, J.; Hong, B.; Chen, L.; Zheng, G.; Zhang, M.; et al. 2024. Poly-visual-expert vision-language models. In COLM

    2024

  12. [20]

    a chler, A.; B \

    Funk, M.; B \"a chler, A.; B \"a chler, L.; Kosch, T.; Heidenreich, T.; and Schmidt, A. 2017. Working with augmented reality? A long-term analysis of in-situ instructions at the assembly workplace. In PETRA, 222--229

    2017

  13. [21]

    Gan, Z.; Li, L.; Li, C.; Wang, L.; Liu, Z.; Gao, J.; et al. 2022. Vision-language pre-training: Basics, recent advances, and future trends. Foundations and Trends in Computer Graphics and Vision , 14(3--4): 163--352

    2022

  14. [22]

    Gao, J.; Sarkar, B.; Xia, F.; Xiao, T.; Wu, J.; Ichter, B.; Majumdar, A.; and Sadigh, D. 2024. Physically grounded vision-language models for robotic manipulation. In ICRA, 12462--12469. IEEE

    2024

  15. [23]

    Gwinnup, J.; and Duh, K. 2023. A survey of vision-language pre-training from the lens of multimodal machine translation. arXiv preprint arXiv:2306.07198

    2023 arXiv

  16. [24]

    Hong, W.; Yu, W.; Gu, X.; Wang, G.; Gan, G.; Tang, H.; Cheng, J.; Qi, J.; Ji, J.; Pan, L.; et al. 2025. GLM-4.1v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning. arXiv preprint arXiv:2507.01006

    2025 arXiv

  17. [25]

    Hu, X.; Gan, Z.; Wang, J.; Yang, Z.; Liu, Z.; Lu, Y.; and Wang, L. 2022. Scaling up vision-language pre-training for image captioning. In CVPR, 17980--17989

    2022

  18. [26]

    Lauren c on, H.; Tronchon, L.; Cord, M.; and Sanh, V. 2024. What matters when building vision-language models? NeurIPS, 37: 87874--87907

    2024

  19. [27]

    LEGO. 2024. LEGO audio instructions. https://legoaudioinstructions.com/instructions. Accessed: 12-12-2024

    2024

  20. [28]

    Li, B.; Zhang, Y.; Chen, L.; Wang, J.; Pu, F.; Yang, J.; Li, C.; and Liu, Z. 2023 a . Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306.05425

    2023 arXiv

  21. [29]

    Li, B.; Zhang, Y.; Chen, L.; Wang, J.; Yang, J.; and Liu, Z. 2023 b . Otter: A multi-modal model with in-context instruction tuning. arXiv:2305.03726

    2023 arXiv

  22. [30]

    Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023 c . Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML

    2023

  23. [31]

    Li, X.; Yin, X.; Li, C.; Zhang, P.; Hu, X.; Zhang, L.; Wang, L.; Hu, H.; Dong, L.; Wei, F.; et al. 2020. Oscar: Object-semantics aligned pre-training for vision-language tasks. In ECCV, 121--137. Springer

    2020

  24. [32]

    Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023. Visual instruction tuning. In NeurIPS

    2023

  25. [33]

    Miech, A.; Alayrac, J.-B.; Laptev, I.; Sivic, J.; and Zisserman, A. 2020. RareAct: A video dataset of unusual interactions. arxiv:2008.01018

    2020 arXiv

  26. [34]

    Miech, A.; Zhukov, D.; Alayrac, J.-B.; Tapaswi, M.; Laptev, I.; and Sivic, J. 2019. HowTo100M: Learning a text-video embedding by watching hundred million narrated video clips. In CVPR

    2019

  27. [35]

    OpenAI. 2024. GPT-4o. https://openai.com/index/hello-gpt-4o/

    2024

  28. [36]

    Padmakumar, A.; Thomason, J.; Shrivastava, A.; Lange, P.; Narayan-Chen, A.; Gella, S.; Piramuthu, R.; Tur, G.; and Hakkani-Tur, D. 2021. TEACh: Task-driven embodied agents that chat. In AAAI

    2021

  29. [37]

    QwenTeam. 2024. Qwen2.5: A party of foundation models

    2024

  30. [38]

    W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al

    Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021. Learning transferable visual models from natural language supervision. In ICML, 8748--8763. PMLR

    2021

  31. [39]

    R.; and Nguyen, A

    Rahmanzadehgervi, P.; Bolton, L.; Taesiri, M. R.; and Nguyen, A. T. 2024. Vision language models are blind. In ACCV, 18--34

    2024

  32. [40]

    Scikit-learn. 2024. Fleiss Kappa. https://www.statsmodels.org/dev/generated/statsmodels.stats.inter_rater.fleiss_kappa.html

    2024

  33. [41]

    Sener, F.; Chatterjee, D.; Shelepov, D.; He, K.; Singhania, D.; Wang, R.; and Yao, A. 2022. Assembly101: A large-scale multi-view video dataset for understanding procedural activities. In CVPR, 21096--21106

    2022

  34. [42]

    Shen, H.; Liu, P.; Li, J.; Fang, C.; Ma, Y.; Liao, J.; Shen, Q.; Zhang, Z.; Zhao, K.; Zhang, Q.; et al. 2025. Vlm-r1: A stable and generalizable r1-style large vision-language model. arXiv preprint arXiv:2504.07615

    2025 arXiv

  35. [43]

    Stanescu, A.; Mohr, P.; Kozinski, M.; Mori, S.; Schmalstieg, D.; and Kalkofen, D. 2023 a . State-aware configuration detection for augmented reality step-by-step tutorials. In ISMAR, 157--166. IEEE

    2023

  36. [44]

    Stanescu, A.; Mohr, P.; Kozinski, M.; Mori, S.; Schmalstieg, D.; and Kalkofen, D. 2023 b . State-aware configuration detection for augmented reality step-by-step tutorials. In ISMAR, 157--166

    2023

  37. [45]

    Statsmodels. 2024. KMeans. https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html

    2024

  38. [46]

    Subramanian, S.; Merrill, W.; Darrell, T.; Gardner, M.; Singh, S.; and Rohrbach, A. 2022. ReCLIP: A strong zero-shot baseline for referring expression comprehension. In ACL

    2022

  39. [47]

    Tang, Y.; Ding, D.; Rao, Y.; Zheng, Y.; Zhang, D.; Zhao, L.; Lu, J.; and Zhou, J. 2019. Coin: A large-scale dataset for comprehensive instructional video analysis. In CVPR, 1207--1216

    2019

  40. [48]

    Wang, B.; Li, Y.; Lv, Z.; Xia, H.; Xu, Y.; and Sodhi, R. 2024 a . LAVE: LLM-powered agent assistance and language augmentation for video editing. arXiv preprint arXiv:2402.10294

    2024 arXiv

  41. [49]

    Wang, T.; Zhou, W.; Zeng, Y.; and Zhang, X. 2023 a . EfficientVLM: Fast and accurate vision-language models via knowledge distillation and modal-adaptive pruning. In ACL, 13899--13913

    2023

  42. [50]

    Wang, W.; Chen, Z.; Chen, X.; Wu, J.; Zhu, X.; Zeng, G.; Luo, P.; Lu, T.; Zhou, J.; Qiao, Y.; et al. 2024 b . Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. Advances in Neural Information Processing Systems, 36

    2024

  43. [51]

    V.; Joshi, N.; and Pollefeys, M

    Wang, X.; Kwon, T.; Rad, M.; Pan, B.; Chakraborty, I.; Andrist, S.; Bohus, D.; Feniello, A.; Tekin, B.; Frujeri, F. V.; Joshi, N.; and Pollefeys, M. 2023 b . HoloAssist: An egocentric human interaction dataset for interactive ai assistants in the real world. In CVPR

    2023

  44. [52]

    Wei, H.; Kong, L.; Chen, J.; Zhao, L.; Ge, Z.; Yang, J.; Sun, J.; Han, C.; and Zhang, X. 2024. Vary: Scaling up the vision vocabulary for large vision-language model. In ECCV, 408--424. Springer

    2024

  45. [53]

    Ye, Q.; Xu, H.; Ye, J.; Yan, M.; Hu, A.; Liu, H.; Qian, Q.; Zhang, J.; and Huang, F. 2024. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. In CVPR, 13040--13051

    2024

  46. [54]

    You, Y.; Ji, Z.; Yang, X.; and Liu, Y. 2022. From human-human collaboration to human-robot collaboration: automated generation of assembly task knowledge model. In ICAC, 1--6. IEEE

    2022

  47. [55]

    Zhang, J.; Huang, J.; Jin, S.; and Lu, S. 2024. Vision-language models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence

    2024

  48. [56]

    Zhong, Y.; Yu, L.; Bai, Y.; Li, S.; Yan, X.; and Li, Y. 2023. Learning procedure-aware video representation from instructional videos and their narrations. In CVPR

    2023

  49. [57]

    Zhou, L.; Palangi, H.; Zhang, L.; Hu, H.; Corso, J.; and Gao, J. 2020. Unified vision-language pre-training for image captioning and vqa. In AAAI, 13041--13049

    2020

  50. [58]

    Zhu, D.; Chen, J.; Shen, X.; Li, X.; and Elhoseiny, M. 2023. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. arXiv:2304.10592

    2023 arXiv

  51. [59]

    G.; Fouhey, D.; Laptev, I.; and Sivic, J

    Zhukov, D.; Alayrac, J.-B.; Cinbis, R. G.; Fouhey, D.; Laptev, I.; and Sivic, J. 2019. Cross-task weakly supervised learning from instructional videos. In CVPR

    2019

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.