REVIEW 3 major objections 4 minor 34 references
Decomposing assembly actions into verb-object-tool questions lets vision-language models beat standard video classifiers on fine-grained hand-object tasks.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-14 09:11 UTC pith:EYTEARKZ
load-bearing objection Solid empirical systems paper that makes VLMs usable for assembly actions via element-wise VQA and alternating LoRA; the layer-partition story is the softest part but does not sink the result. the 3 major comments →
Compositional Context Fine-Tuning Vision-Language Model for Complex Assembly Action Understanding from Videos
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Compositional Context Fine-Tuning (CCFT) together with Layer-Partitioned Alternating Training (LP-AT) turns a general vision-language model into a more accurate and more interpretable recognizer of fine-grained assembly actions than purpose-built video classifiers, by forcing the model to answer separate templated questions for verb, object and tool and by isolating each question’s low-rank adapters to distinct layer groups that are updated in alternation.
What carries the argument
Layer-Partitioned Alternating Training (LP-AT): each action element is given its own LoRA adapter attached only to a hand-chosen contiguous block of language-model layers; the adapters are optimized and merged one after another inside every epoch, so gradient updates stay inside element-specific subspaces and each adapter can receive its own learning-rate and rank.
Load-bearing premise
The particular layer blocks chosen for each element (and the per-block learning rates) are assumed to be near-optimal; if a different partition erased the reported gains, the method’s claimed advantage would disappear.
What would settle it
Re-run the identical fine-tuning protocol on HA-ViD-VQA and IKEA-ASM-VQA but with a single shared adapter across all layers, or with randomly reassigned layer blocks of the same sizes; if holistic Top-1 accuracy then matches or exceeds the partitioned alternating results, the central claim that layer partitioning is necessary collapses.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper adapts vision-language models (specifically Qwen2.5-VL-7B) to fine-grained assembly action understanding by introducing Compositional Context Fine-Tuning (CCFT). Assembly actions are decomposed into semantic elements (Verb, Object/Tool) and the VLM is fine-tuned on templated VQA pairs so that outputs are near-deterministic and element-level. To support multi-task learning under limited data, Layer-Partitioned Alternating Training (LP-AT) assigns disjoint layer groups to element-specific LoRA adapters and alternates their updates within each epoch. Two new VQA reformulations of existing assembly datasets (HA-ViD-VQA, IKEA-ASM-VQA) are released. On held-out test splits the method reports higher holistic Top-1 accuracy than TSM, UniFormerV2 and VideoMAE V2 (Table I) while also supplying per-element accuracies (Table II). Ablations contrast compositional vs. holistic fine-tuning, partitioned vs. shared adapters, and alternating vs. independent/sequential training (Tables III–VI).
Significance. If the gains hold under more rigorous controls, the work supplies a practical recipe for turning generalist VLMs into near-deterministic, interpretable action recognizers for human-robot collaborative assembly—an application domain that currently lacks strong VLM baselines. The release of HA-ViD-VQA and IKEA-ASM-VQA, the public code repository, and the three ablations that isolate compositionality, layer partitioning and alternating updates are concrete contributions that other groups can build on. The element-level outputs are immediately usable for downstream HRC modules (skill learning, assistance triggering). The main limitation on significance is the modest absolute accuracies and the small test sets, which leave open how far the approach generalizes beyond the two chosen datasets.
major comments (3)
- §IV-B and Discussion §IV-E.3: the layer partitions (HA-ViD: 0–9 verb / 10–23 objects / 24–27 tool; IKEA: 0–13 verb / 14–27 object) and the associated per-adapter learning rates are justified only by a developmental-trajectory analogy and “preliminary experiments” whose numbers are not shown. All LP-AT ablations (Tables IV–VI) use exactly these partitions. Without at least one alternative partition (or a random-partition control) it remains possible that the reported gains of alternating training over shared/independent/sequential schemes are an artifact of a lucky allocation rather than of the alternating principle itself. A short sensitivity study is needed to make the central LP-AT claim load-bearing.
- Tables I–II: test sets contain only 67–86 clips per view/hand. No standard deviations, confidence intervals or statistical tests are reported. With such small N the 2–14 point margins over VideoMAE V2 / UniFormerV2 could easily reverse under a different split. Either bootstrap/error-bar results or an explicit statement that the differences are not claimed to be statistically significant should be added before the “consistently outperforms” claim can be taken as robust.
- §III-E, Eqs. (6)–(7): evaluation relies on MiniLM cosine similarity ≥ τ = 0.95 between free-form strings. While the threshold is stated to be robust, no human agreement study or exact-match baseline is provided. Because the model is free to generate any string, a non-negligible fraction of “correct” predictions under this metric may still be unusable for deterministic HRC pipelines. Reporting exact string match (or a constrained decoding accuracy) alongside the soft metric would strengthen the near-determinism claim.
minor comments (4)
- §IV-B: the temporal sampling policy (max 76 frames @ 2 fps) is mentioned only in passing; a short ablation or justification relative to the original video lengths would help reproducibility.
- Fig. 2 qualitative examples are useful but the base-model and prompted outputs are truncated; showing the full generated strings (or at least the extracted elements) would make the comparison clearer.
- Related Work §II-c cites Zheng et al. [27] for compositional assembly segmentation; a more explicit contrast of how CCFT differs from that earlier heuristic recombination would help position the contribution.
- Typographical inconsistencies: “V erb”, “T ool”, “F ormulation” appear with stray spaces in several places; “IKEA ASM” vs. “IKEA-ASM” is used interchangeably.
Circularity Check
No significant circularity: empirical train/test evaluation of a fine-tuning method; self-citations only supply source datasets, not load-bearing uniqueness or forced predictions.
full rationale
The paper is a standard empirical ML contribution. CCFT and LP-AT are defined as a multi-task LoRA fine-tuning procedure (Eqs. 1–5) that is trained on the train splits of the newly constructed HA-ViD-VQA / IKEA-ASM-VQA datasets and scored by string-similarity accuracy (Eqs. 6–7) on held-out test splits. Reported gains over TSM, UniFormerV2 and VideoMAE V2 (Table I) and over shared/independent/sequential adapters (Tables IV–VI) are therefore ordinary out-of-sample measurements, not quantities that reduce by construction to fitted constants or to the layer-partition ansatz. The hand-chosen partitions (Sec. IV-B) and developmental-trajectory analogy are acknowledged as heuristic in the Discussion (Sec. IV-E.3); they affect the magnitude of the ablation deltas but do not make any accuracy equal to an input by definition. Self-citations ([7], [27]) merely identify the source video collections that were re-annotated; they do not supply a uniqueness theorem or an ansatz that is then re-used as a “first-principles” result. Consequently the derivation chain contains no self-definitional step, no fitted-input-called-prediction, and no load-bearing self-citation circularity.
Axiom & Free-Parameter Ledger
free parameters (5)
- layer partition boundaries (HA-ViD 0–9 / 10–23 / 24–27; IKEA 0–13 / 14–27)
- per-adapter learning rates (5e-5, 5e-5, 2e-5)
- LoRA rank r=256 and scaling α=256
- evaluation similarity threshold τ=0.95
- temporal sampling (max 76 frames @ 2 fps)
axioms (4)
- domain assumption Assembly actions factor into independent semantic elements (Verb, Object, Tool) that can be queried and scored separately then concatenated.
- domain assumption Freezing the pretrained vision encoder of Qwen2.5-VL and fine-tuning only the language decoder is sufficient for assembly action recognition on the given data scale.
- ad hoc to paper Alternating merge of element-specific LoRA adapters within each epoch reduces cross-task interference relative to independent or sequential training.
- ad hoc to paper Cosine similarity ≥ 0.95 between MiniLM embeddings of predicted and ground-truth strings is an adequate proxy for semantic correctness.
invented entities (3)
-
Compositional Context Fine-Tuning (CCFT)
no independent evidence
-
Layer-Partitioned Alternating Training (LP-AT)
no independent evidence
-
HA-ViD-VQA and IKEA-ASM-VQA datasets
no independent evidence
read the original abstract
Assembly action understanding is a key enabler for effective human-robot collaborative assembly, yet it remains challenging due to subtle motions and fine-grained hand-object interactions. We adapt vision-language models (VLMs) to this challenging domain with Compositional Context Fine-Tuning (CCFT), a method that decomposes assembly actions into semantic elements (Verb, Object, Tool) and fine-tunes VLMs to recognize each action element using templated question-answering pairs. This approach ensures near-deterministic outputs. To enable efficient and effective multi-task learning under limited data, a Layer-Partitioned Alternating Training (LP-AT) method is presented, which assigns distinct model layers to recognize specific action elements through element-specific low-rank adapters. LP-AT alternates weight updates across element-specific adapters, reducing cross-task interference while enabling per-adapter hyperparameter optimization. Furthermore, we create HA-ViD-VQA and IKEA-ASM-VQA datasets from existing assembly video datasets. Extensive experiments on these datasets demonstrate that our method consistently outperforms strong action recognition baselines while providing interpretable element-level predictions that can support diverse downstream applications.
Figures
Reference graph
Works this paper leans on
-
[1]
Vision-language models for vision tasks: A survey,
J. Zhang, J. Huang, S. Jin, and S. Lu, “Vision-language models for vision tasks: A survey,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 8, pp. 5625–5644, 2024
2024
-
[2]
Z. Li, X. Wu, H. Du, F. Liu, H. Nghiem, and G. Shi, “A survey of state of the art large vision language models: Alignment, benchmark, evaluations and challenges,” 2025. [Online]. Available: https://arxiv.org/abs/2501.02189
Pith/arXiv arXiv 2025
-
[3]
Attach dataset: Annotated two-handed assembly actions for human action understanding,
D. Aganian, B. Stephan, M. Eisenbach, C. Stretz, and H.-M. Gross, “Attach dataset: Annotated two-handed assembly actions for human action understanding,” in2023 IEEE International Conference on Robotics and Automation (ICRA), 2023, pp. 11 367–11 373
2023
-
[4]
A human-robot collaborative assembly framework with quality checking based on real-time dual-hand action segmentation,
H. Zheng, W. Xia, and X. Xu, “A human-robot collaborative assembly framework with quality checking based on real-time dual-hand action segmentation,”Robotics and Computer-Integrated Manufacturing, vol. 94, p. 102976, 2025. [Online]. Available: https: //www.sciencedirect.com/science/article/pii/S0736584525000304
2025
-
[5]
Video-language understanding: A survey from model architecture, model training, and data perspectives,
T. Nguyen, Y . Bin, J. Xiao, L. Qu, Y . Li, J. Z. Wu, C.-D. Nguyen, S.-K. Ng, and L. A. Tuan, “Video-language understanding: A survey from model architecture, model training, and data perspectives,”
-
[6]
Available: https://arxiv.org/abs/2406.05615
[Online]. Available: https://arxiv.org/abs/2406.05615
-
[7]
Lora: Low-rank adaptation of large language models,
E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,”
-
[8]
Available: https://arxiv.org/abs/2106.09685
[Online]. Available: https://arxiv.org/abs/2106.09685
-
[9]
Ha-vid: A human assembly video dataset for comprehensive assembly knowledge understanding,
H. Zheng, R. Lee, and Y . Lu, “Ha-vid: A human assembly video dataset for comprehensive assembly knowledge understanding,” inAdvances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36. Curran Associates, Inc., 2023, pp. 67 069–67 081. [Online]. Available: https://proceedings.neurip...
2023
-
[10]
The ikea asm dataset: Understanding people assembling furniture through actions, objects and pose,
Y . Ben-Shabat, X. Yu, F. Saleh, D. Campbell, C. Rodriguez-Opazo, H. Li, and S. Gould, “The ikea asm dataset: Understanding people assembling furniture through actions, objects and pose,” in2021 IEEE Winter Conference on Applications of Computer Vision (WACV), 2021, pp. 846–858
2021
-
[13]
Temporal segment networks for action recognition in videos,
L. Wang, Y . Xiong, Z. Wang, Y . Qiao, D. Lin, X. Tang, and L. Van Gool, “Temporal segment networks for action recognition in videos,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 11, pp. 2740–2755, 2019
2019
-
[14]
Tsm: Temporal shift module for efficient and scalable video understanding on edge devices,
J. Lin, C. Gan, K. Wang, and S. Han, “Tsm: Temporal shift module for efficient and scalable video understanding on edge devices,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 5, pp. 2760–2774, 2022
2022
-
[15]
Hierarchically Decomposed Graph Convolutional Networks for Skeleton-Based Action Recognition ,
J. Lee, M. Lee, D. Lee, and S. Lee, “ Hierarchically Decomposed Graph Convolutional Networks for Skeleton-Based Action Recognition ,” in2023 IEEE/CVF International Conference on Computer Vision (ICCV). Los Alamitos, CA, USA: IEEE Computer Society, Oct. 2023, pp. 10 410–10 419. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/ICCV51070.2023.00958
-
[16]
Uniformerv2: Unlocking the potential of image vits for video under- standing,
K. Li, Y . Wang, Y . He, Y . Li, Y . Wang, L. Wang, and Y . Qiao, “Uniformerv2: Unlocking the potential of image vits for video under- standing,” in2023 IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 1632–1643
2023
-
[17]
Videomae v2: Scaling video masked autoencoders with dual masking,
L. Wang, B. Huang, Z. Zhao, Z. Tong, Y . He, Y . Wang, Y . Wang, and Y . Qiao, “Videomae v2: Scaling video masked autoencoders with dual masking,” in2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 14 549–14 560
2023
-
[18]
Duha: a dual-hand action segmentation method for human-robot collaborative assembly,
H. Zheng, R. Lee, Y . Lu, and X. Xu, “Duha: a dual-hand action segmentation method for human-robot collaborative assembly,” in 2024 IEEE 20th International Conference on Automation Science and Engineering (CASE), 2024, pp. 522–527
2024
-
[19]
Skeleton- rgb integrated highly similar human action prediction in human–robot collaborative assembly,
Y . Zhang, K. Ding, J. Hui, S. Liu, W. Guo, and L. Wang, “Skeleton- rgb integrated highly similar human action prediction in human–robot collaborative assembly,”Robotics and Computer-Integrated Manufacturing, vol. 86, p. 102659, 2024. [Online]. Available: https: //www.sciencedirect.com/science/article/pii/S0736584523001345
2024
-
[20]
Learning transferable visual models from natural language supervision,
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” 2021. [Online]. Available: https: //arxiv.org/abs/2103.00020
Pith/arXiv arXiv 2021
-
[21]
Scaling up visual and vision-language representation learning with noisy text supervision,
C. Jia, Y . Yang, Y . Xia, Y .-T. Chen, Z. Parekh, H. Pham, Q. Le, Y .-H. Sung, Z. Li, and T. Duerig, “Scaling up visual and vision-language representation learning with noisy text supervision,” inProceedings of the 38th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, M. Meila and T. Zhang, Eds., vol. 139. PMLR...
2021
-
[22]
Llava-video: Video instruction tuning with synthetic data,
Y . Zhang, J. Wu, W. Li, B. Li, Z. Ma, Z. Liu, and C. Li, “Llava-video: Video instruction tuning with synthetic data,” 2025. [Online]. Available: https://arxiv.org/abs/2410.02713
Pith/arXiv arXiv 2025
-
[23]
Incorporating scene graphs into pre-trained vision-language models for multimodal open-vocabulary action recog- nition,
C. Wei and Z. Deng, “Incorporating scene graphs into pre-trained vision-language models for multimodal open-vocabulary action recog- nition,” in2024 IEEE International Conference on Robotics and Automation (ICRA), 2024, pp. 440–447
2024
-
[24]
Qlora: Efficient finetuning of quantized llms,
T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient finetuning of quantized llms,” 2023. [Online]. Available: https://arxiv.org/abs/2305.14314
Pith/arXiv arXiv 2023
-
[25]
Dora: Weight-decomposed low-rank adaptation,
S.-Y . Liu, C.-Y . Wang, H. Yin, P. Molchanov, Y .-C. F. Wang, K.-T. Cheng, and M.-H. Chen, “Dora: Weight-decomposed low-rank adaptation,” 2024. [Online]. Available: https://arxiv.org/abs/2402. 09353
2024
-
[26]
Assembly101: A large-scale multi-view video dataset for understanding procedural activities,
F. Sener, D. Chatterjee, D. Shelepov, K. He, D. Singhania, R. Wang, and A. Yao, “Assembly101: A large-scale multi-view video dataset for understanding procedural activities,” in2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 21 064–21 074
2022
-
[27]
@ CREPE: Can Vision-Language Foundation Models Reason Compositionally? ,
Z. Ma, J. Hong, M. O. Gul, M. Gandhi, I. Gao, and R. Krishna, “ @ CREPE: Can Vision-Language Foundation Models Reason Compositionally? ,” in2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Los Alamitos, CA, USA: IEEE Computer Society, Jun. 2023, pp. 10 910–10 921. [Online]. Available: https://doi.ieeecomputersociety. org/10.1109...
-
[28]
Compositional Chain-of-Thought Prompting for Large Multimodal Models ,
C. Mitra, B. Huang, T. Darrell, and R. Herzig, “ Compositional Chain-of-Thought Prompting for Large Multimodal Models ,” in 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Los Alamitos, CA, USA: IEEE Computer Society, Jun. 2024, pp. 14 420–14 431. [Online]. Available: https: //doi.ieeecomputersociety.org/10.1109/CVPR52733.2024.01367
-
[29]
Causal graphical models for vision- language compositional understanding,
F. Parascandolo, N. Moratelli, E. Sangineto, L. Baraldi, and R. Cucchiara, “Causal graphical models for vision- language compositional understanding,” 2025. [Online]. Available: https://arxiv.org/abs/2412.09353
Pith/arXiv arXiv 2025
-
[30]
Ducas: a knowledge- enhanced dual-hand compositional action segmentation method for human-robot collaborative assembly,
H. Zheng, R. Lee, H. Liang, Y . Lu, and X. Xu, “Ducas: a knowledge- enhanced dual-hand compositional action segmentation method for human-robot collaborative assembly,” in2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2024, pp. 7175– 7180
2024
-
[31]
Qwen2.5-vl technical report,
S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y . Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y . Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin, “Qwen2.5-vl technical report,”
-
[32]
Available: https://arxiv.org/abs/2502.13923
[Online]. Available: https://arxiv.org/abs/2502.13923
-
[33]
Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers,
W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, and M. Zhou, “Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers,” 2020. [Online]. Available: https: //arxiv.org/abs/2002.10957
Pith/arXiv arXiv 2020
-
[34]
A. W. Needham and E. L. Nelson, “How babies use their hands to learn about objects: Exploration, reach-to-grasp, manipulation, and tool use,” vol. 14, no. 6, p. e1661. [Online]. Available: https://wires.onlinelibrary.wiley.com/doi/10.1002/wcs.1661
-
[35]
Hierarchical planning for long-horizon manipulation with geometric and symbolic scene graphs,
Y . Zhu, J. Tremblay, S. Birchfield, and Y . Zhu, “Hierarchical planning for long-horizon manipulation with geometric and symbolic scene graphs,” in2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE Press, 2021, p. 6541–6548. [Online]. Available: https://doi.org/10.1109/ICRA48506.2021.9561548
-
[36]
Self-refine: Itera- tive refinement with self-feedback,
A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y . Yanget al., “Self-refine: Itera- tive refinement with self-feedback,”Advances in Neural Information Processing Systems, vol. 36, pp. 46 534–46 594, 2023
2023
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.