Pith. sign in

REVIEW 3 major objections 6 minor 67 references

IKEA Manuals at Work: 4D Grounding of Assembly Instructions on Internet Videos

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read IKEA Video Manuals is the first dataset to ground assembly instructions in videos with dense spatio-temporal alignments among 3D part models, step-by-step manuals, and real-world internet videos.

desk verdict A useful, well-documented assembly dataset that deserves peer review; the pose ground truth lacks an independent accuracy check, and the paper should be revised to supply one. read the letter →

arxiv 2411.11409 v1 pith:LH6BFDZS submitted 2024-11-18 cs.CV cs.AIcs.LGcs.RO

Yunong Liu , Cristobal Eyzaguirre , Manling Li , Shubh Khanna , Juan Carlos Niebles , Vineeth Ravi , Saumitra Mishra , Weiyu Liu
show 1 more author
Jiajun Wu
This is my paper · ORCID
classification cs.CVcs.AIcs.LGcs.RO
keywords 4Dgroundingassemblydataset6-DoFposeestimationinstructionsvideopartsegmentationIKEAfurnitureshape
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces IKEA Video Manuals, a dataset built to connect every level of the furniture-assembly problem: the step-by-step instruction manual, the 3D geometry of the parts, and the messy real-world video of a person actually doing the assembly. Its central claim is that it is the first resource to provide 4D grounding of assembly instructions, meaning dense spatial and temporal alignment across these modalities, delivered as 34,441 annotated frames from 98 internet videos covering 36 IKEA furniture models. The paper argues that existing datasets each cover only a slice of the problem: some align manual steps without visual grounding, some provide 3D poses only in controlled labs, and internet video datasets lack correspondence with 3D models. If the dataset is sound, it gives the community a common benchmark on which task planning, visual grounding, and robot assembly can be studied together against real-world difficulties such as occlusions, changing viewpoints, and long assembly sequences, with five baseline tasks already defined.

What carries the argument

The central object is the 4D grounding itself: a dense spatio-temporal alignment in which 3D furniture parts, manual steps, and video frames are tied together, with each frame's sub-assemblies given a segmentation mask and a 6-DoF pose in the camera coordinate frame, plus per-segment camera intrinsics. The machinery that produces it is a semi-automated annotation pipeline: videos are split into substeps defined by the appearance of a new part or the formation of a sub-assembly; part identities are assigned by watching the whole video; masks are generated interactively with the Segment Anything Model and manually corrected; and poses come from hand-labeled 2D-3D keypoints solved with PnP and RANSAC to estimate intrinsics per video segment, then refined by annotators in a 3D interface that checks relative poses, coplanarity, and cross-frame consistency from multiple viewpoints. The load-bearing link is the mapping from each manual image to a contiguous span of video frames, so that the same assembly step has both a pictorial and a physical demonstration.

What would settle it

Take a random sample of annotated frames, project the annotated 3D part poses into the image using the provided camera intrinsics, and measure the silhouette overlap against the annotated segmentation masks; if the projected outlines systematically drift from the masks in frames with camera motion or across segment boundaries, the intrinsic estimation is biased. A stronger test is to independently calibrate a subset of videos using structure-from-motion on static background or known planar surfaces such as tabletops and compare the recovered intrinsics and poses; if the independent calibration disagrees with the dataset's estimates beyond a few degrees or centimeters, the claimed 6-DoF ground truth is not yet established.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is the data itself: a benchmark in which each furniture item is represented by its 3D parts with 6-DoF poses, and each sampled video frame carries the identity, segmentation mask, and camera-frame 6-DoF pose of every sub-assembly in view, together with camera intrinsics estimated per video segment and a mapping from each manual step to a span of video substeps. The authors claim this is the first dataset to supply 6-DoF pose annotations for furniture assembly from internet videos, combining the 3D models and manuals of the IKEA-Manual dataset with in-the-wild part trajectories that prior datasets only approximated. The annotation process assembles these correspondences through temporal segmentation into substeps, part identity tracking, Segment Anything Model assisted mask annotation, keypoint-based camera estimation with PnP and RANSAC, and manual 3D refinement verified from multiple viewpoints. On this resource the paper defines five applications, assembly plan generation, part-conditioned segmentation, part-conditioned pose estimation, video object segmentation, and furniture assembly from instruction videos, and reports baselines showing that all of them remain difficult.

Load-bearing premise

The dataset's pose annotations are treated as ground truth, but the camera parameters they depend on are estimated from hand-labeled keypoints and manual refinement rather than measured or independently validated, so any systematic error in the cameras would silently bias every downstream pose number.

Editorial extensions

If this is right

  • Because the dataset anchors manual steps, 3D poses, and video pixels to the same events, a system that masters its five benchmark tasks could watch an assembly video and produce an executable assembly plan grounded in real object poses rather than in 2D annotations alone.
  • The reported baselines show that zero-shot part segmentation and pose estimation methods such as CNOS, SAM-6D, and MegaPose degrade substantially on internet video compared with lab-based benchmarks, quantifying a measurable gap between current methods and in-the-wild assembly understanding.
  • The video-based assembly task demonstrates that with ground-truth keyframe and pose annotations a modular pipeline assembles furniture to a Chamfer distance of 0.33, while replacing keyframe detection with GPT-4o degrades this to 0.55, indicating that automatic step detection is currently the weakest link in video-guided assembly.
  • The dataset's comparison with existing resources such as Assembly101, IKEA ASM, IKEA-Manual, and IKEA in the Wild positions it as the only one combining internet-sourced video, 3D object models, and dense 6-DoF annotations, enabling evaluation of generalization across environments and assembly sequences.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The camera intrinsics estimated per video segment are the hidden load-bearing assumption: the paper states that correct camera parameters are a prerequisite for pose accuracy, yet reports no independent calibration check or inter-annotator agreement study, so a systematic bias in those estimates would transfer directly into every 6-DoF pose number in the benchmark.
  • Because the annotation pipeline is built on Segment Anything Model prompts plus keypoint PnP, the marginal cost of extending the dataset to new furniture categories is plausibly much lower than for fully manual datasets; the same pipeline could be reapplied to other branded assembly products.
  • All pose baselines run from monocular RGB with estimated depth, even though the dataset supplies true 3D geometry and camera parameters; a natural next experiment, not run in the paper, is to measure how much of the reported gap closes when methods are allowed to use the provided intrinsics and multi-frame consistency.
  • The manual refinement step that corrects 2D-projection errors by checking part poses from top and side views suggests that pure automatic pose recovery from keypoints is insufficient for ground-truth quality, which implies the dataset's accuracy depends on human effort that future scaling of the resource will need to control.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces IKEA Video Manuals, a multimodal dataset that aligns 3D furniture models from IKEA-Manual with Internet assembly videos from IAW. For 98 videos and 36 furniture models it provides temporal step and substep alignments, per-frame part segmentation masks, per-frame 6-DoF part poses, and per-segment camera intrinsic estimates. The authors report five applications—assembly plan generation, part-conditioned segmentation, part-conditioned pose estimation, video object segmentation, and video-based shape assembly—with baseline evaluations showing that current methods perform poorly on this real-world data. The central claim is that this is the first dataset offering dense spatio-temporal (4D) grounding of assembly instructions in Internet videos.

Significance. If the annotation quality is adequate, this is a valuable benchmark contribution: it fills a real gap by combining real-world assembly videos with 3D models and manuals, and it includes a substantial annotation pipeline, detailed appendices, a datasheet, a public release under CC-BY-4.0, and a DOI. The five baseline tasks and the qualitative error analysis in Appendix H give the community concrete starting points. The paper's main weakness is that the novel 6-DoF pose and camera-intrinsic annotations are not validated quantitatively, which matters because the poses are the distinguishing asset of the dataset and are used as ground truth in Tables 4 and 5 and in Section 5.5.

major comments (3)
  1. [§4.4, App. C.6, Fig. A7] The load-bearing assumption is that the per-segment camera intrinsics estimated from manual 2D-3D keypoints with PnP and RANSAC, followed by manual pose refinement, are accurate enough to serve as benchmark ground truth. This is not checked. The focal-length/depth coupling in uncalibrated monocular video means near-perfect 2D reprojection can coexist with biased 3D poses, and a missed camera change contaminates all poses in a segment. Fig. A7 explicitly shows that correct 2D projection does not imply correct 3D relative pose, yet the refinement interface in App. E.3 re-checks against the same 2D video frames. The manuscript reports no inter-annotator agreement, no comparison against calibrated cameras or EXIF intrinsics, no synthetic-reprojection validation, and no final-assembly closure residual. I would like to see a quantitative validation study, even on a subset, that establishes a noise floor for the pose annotations; without it, the 4D-grounding claim and the numbers in Table 4 rest on an unmeasured quantity.
  2. [App. F] The 'Error Bar' appendix reports standard deviations from running the evaluated models with three random seeds. This measures model stochasticity, not annotation uncertainty. For a dataset paper, the relevant error bar is the inter-annotator or inter-method variability of the ground-truth poses and masks. Since the pose annotations are the novel annotation type, I ask the authors to add annotation-level agreement statistics (e.g., re-annotating a subset by a second annotator and reporting pose/rotation errors and mask IoU).
  3. [§5.5] The shape-assembly experiment in Setting 1 uses the annotated poses directly and reports a Chamfer Distance of 0.33. As written, this number conflates two things: the accuracy of the annotation pipeline and the quality of the assembly protocol. In particular, 'parts are not fully connected in the last frames of the substeps' is an annotation-consistency issue, not an algorithm failure. The discussion should separate annotation-induced error from method error, and ideally be tied to the validation requested above.
minor comments (6)
  1. [§4.1] The sentence ending 'as illustrated in Fig. 4a.' is grammatically incomplete; it should be merged with the preceding sentence or rewritten.
  2. [Table 1 / Table A1] The 'IKEA in the Wild' row lists '# Object' as 420; please clarify whether this is the number of videos, the number of object instances, or the number of distinct object types, since the same row reports 14 object classes.
  3. [Fig. 4 caption] The caption uses a full-width parenthesis in '（c) Parts in Frame Annotation'; please use consistent ASCII parentheses.
  4. [§5.5] The symbol K is used for the number of parts in a sub-assembly in the assembly task, whereas K was earlier used for the number of parts in a plan-graph node in Section 5.1; please use distinct notation or define it again at first use.
  5. [Fig. A2] The field value 'firstPerson' in the example JSON should be 'first-person' for consistency with the Person View description.
  6. [§5.5] The Chamfer Distance values are reported without units or normalization. Please state whether the distance is in the model's native scale and how the ground-truth furniture is normalized across different furniture sizes.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's dataset construction and benchmark evaluations do not reduce a claimed output to its own inputs; the camera-estimation validation gap is a correctness concern, not a circular step.

full rationale

The paper's central contribution is a dataset with manually constructed annotations, and it does not claim a theoretical derivation or a prediction obtained by fitting a parameter and then re-predicting the same quantity. The closest candidate is Section 4.4, where camera intrinsics and 6-DoF part poses are estimated from manually labeled 2D-3D keypoints using PnP and RANSAC, and these poses are later used as ground truth in the part-conditioned pose estimation experiments of Section 5.3. This is standard dataset construction rather than circularity: the evaluated methods (SAM-6D, MegaPose, and differentiable rendering) are independent algorithms scored against those annotations, and the annotations are not produced by those methods. The paper's reliance on the IKEA-Manual dataset for 3D models is a transparent reuse of an external published resource by overlapping authors, but it does not import the paper's conclusions; the reused content is the geometry of the furniture models, not the 4D-grounding result being claimed. The absence of independent calibrated-camera or inter-annotator validation for the camera estimates in Section 4.4 and Appendix C.6 is a legitimate quality and correctness risk for any benchmark built on these poses, and Appendix F reports only model-seed variance rather than annotation noise. However, under the stated standards, lack of validation or a weak assumption is not circularity unless a claimed output is shown to be equivalent to its input by construction or by a self-citation chain. No such reduction appears in the paper, so the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new free parameters or theoretical entities. The central assumptions are that the upstream 3D models and video-step alignments are correct, and that the manual pose-refinement process produces trustworthy ground truth. Camera intrinsics are estimated per segment rather than calibrated, which is the main source of potential bias.

assumptions (3)
  • domain assumption The 3D furniture models from IKEA-Manual accurately represent the real products in the videos.
    The dataset reuses 36 segmented 3D models from IKEA-Manual (Section 4.1) and uses them as geometric ground truth for segmentation and pose annotations.
  • domain assumption The IAW dataset supplies correct video-to-model correspondences and manual-step timestamps.
    The authors select 98 assembly videos from IAW based on matching instruction-manual IDs (Section 4.1, Appendix C), so errors in IAW matching propagate into the new dataset.
  • domain assumption Manual refinement by annotators yields physically consistent 3D poses.
    Pose annotations are refined by annotators in a 3D interface and verified by reviewing projections (Section 4.4, Appendix D), but accuracy is not independently quantified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of IKEA Manuals at Work: 4D Grounding of Assembly Instructions on Internet Videos." pith.science (2026). https://pith.science/paper/LH6BFDZS

@misc{pith2026241111409,
  author       = {Pith},
  title        = {Pith review of: IKEA Manuals at Work: 4D Grounding of Assembly Instructions on Internet Videos},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LH6BFDZS}},
  note         = {Machine review of arXiv:2411.11409}
}
read the original abstract

Shape assembly is a ubiquitous task in daily life, integral for constructing complex 3D structures like IKEA furniture. While significant progress has been made in developing autonomous agents for shape assembly, existing datasets have not yet tackled the 4D grounding of assembly instructions in videos, essential for a holistic understanding of assembly in 3D space over time. We introduce IKEA Video Manuals, a dataset that features 3D models of furniture parts, instructional manuals, assembly videos from the Internet, and most importantly, annotations of dense spatio-temporal alignments between these data modalities. To demonstrate the utility of IKEA Video Manuals, we present five applications essential for shape assembly: assembly plan generation, part-conditioned segmentation, part-conditioned pose estimation, video object segmentation, and furniture assembly based on instructional video manuals. For each application, we provide evaluation metrics and baseline methods. Through experiments on our annotated data, we highlight many challenges in grounding assembly instructions in videos to improve shape assembly, including handling occlusions, varying viewpoints, and extended assembly sequences.

Figures

Figures reproduced from arXiv: 2411.11409 by the authors.

Figure 1
Figure 1. Dataset Overview. (a) Manual images showing the assembly steps. (b) Video frames from the corresponding assembly videos. Temporal alignment between the video frames and each assembly step is also provided. (c) Segmentation masks for individual parts and sub-assemblies that are being constructed in each frame. When two parts are assembled, their masks are combined. (d) 6-DoF poses for parts and sub-assemblies in each… view at source ↗
Figure 2
Figure 2. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Dataset Statistics. (a) Distribution of the number of assembly steps in videos. (b) Distribution of the number of sub-assembly steps (substeps) in videos. (c) Distribution of the number annotations in videos. assembly complexity, with the longest video spanning 49 minutes and the average duration per video being six minutes. Fig. 3a shows the distribution of the number of steps in the videos. Fig. 3b presents the di… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Data Collection and Annotation Pipeline. (a) Collecting 3D furniture models, associated assembly manuals and videos. (b) Annotating coarse temporal segmentation of videos into segments showing each assembly step. (c) Tracking identities of 3D parts throughout each vide…
Figure 5
Figure 5. Figure 5: Example of Hierarchical Assembly Trees. (a) Assembly tree structure derived from the high-level steps in the IKEA manual. (b) More detailed assembly tree structure extracted from the fine-grained substeps annotated in the assembly videos. aligned controls, enabling the…
Figure 6
Figure 6. Figure 6: Qualitative Examples. Examples of part-conditioned segmentation (left) and part-conditioned pose estimation (right) on the IKEA Video Manuals dataset. For segmentation, the ground truth masks are shown along with predicted masks from CNOS [37] and SAM-6D [38]. The oran…
Figure 7
Figure 7. Figure 7: Qualitative Examples. Examples of shape assembly on the IKEA Video Manuals dataset. Results and Analysis [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

67 extracted references · 58 canonical work pages

  1. [1]

    Designing effective step-by-step assembly instructions.ACM Transactions on Graphics (TOG), 22(3):828–837, 2003

    Maneesh Agrawala, Doantam Phan, Julie Heiser, John Haymaker, Jeff Klingner, Pat Hanrahan, and Barbara Tversky. Designing effective step-by-step assembly instructions.ACM Transactions on Graphics (TOG), 22(3):828–837, 2003. 1

    work page 2003

  2. [2]

    Ikea-manual: Seeing shape assembly step by step

    Ruocheng Wang, Yunzhi Zhang, Jiayuan Mao, Ran Zhang, Chin-Yi Cheng, and Jiajun Wu. Ikea-manual: Seeing shape assembly step by step. Advances in Neural Information Processing Systems, 35:28428–28440, 2022. 1, 3, 5, 7, 19, 29

    work page 2022

  3. [3]

    The ikea asm dataset: Understanding people assembling furniture through actions, objects and pose

    Yizhak Ben-Shabat, Xin Yu, Fatemeh Saleh, Dylan Campbell, Cristian Rodriguez-Opazo, Hongdong Li, and Stephen Gould. The ikea asm dataset: Understanding people assembling furniture through actions, objects and pose. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 847–859, 2021. 1, 2, 5, 29

    work page 2021

  4. [4]

    IKEA Object State Dataset: A 6DoF object pose estimation dataset and benchmark for multi-state assembly objects

    Yongzhi Su, Mingxin Liu, Jason Rambach, Antonia Pehrson, Anton Berg, and Didier Stricker. Ikea object state dataset: A 6dof object pose estimation dataset and benchmark for multi-state assembly objects. arXiv preprint arXiv:2111.08614, 2021. 1

    work page Pith review arXiv 2021

  5. [5]

    Aligning step-by-step instructional diagrams to video demonstrations

    Jiahao Zhang, Anoop Cherian, Yanbin Liu, Yizhak Ben-Shabat, Cristian Rodriguez, and Stephen Gould. Aligning step-by-step instructional diagrams to video demonstrations. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 1, 19

    work page 2023

  6. [6]

    Towards automatic learning of procedures from web instructional videos

    Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018. 2

    work page 2018

  7. [7]

    Coin: A large-scale dataset for comprehensive instructional video analysis

    Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. Coin: A large-scale dataset for comprehensive instructional video analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1207–1216, 2019. 2

    work page 2019

  8. [8]

    Scaling egocentric vision: The epic-kitchens dataset

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic-kitchens dataset. In Proceedings of the European conference on computer vision (ECCV), pages 720–736, 2018. 2

    2018

Show all 67 references
  1. [9]

    Human action forecasting by learning task grammars

    Tengda Han, Jue Wang, Anoop Cherian, and Stephen Gould. Human action forecasting by learning task grammars. arXiv:1709.06391, 2017. 2

    2017 arXiv

  2. [10]

    Human pose forecasting via deep Markov models

    Sam Toyer, Anoop Cherian, Tengda Han, and Stephen Gould. Human pose forecasting via deep Markov models. In DICTA, 2017. 2

    2017

  3. [11]

    Assembly101: A large-scale multi-view video dataset for understanding procedural activities

    Fadime Sener, Dibyadip Chatterjee, Daniel Shelepov, Kun He, Dipika Singhania, Robert Wang, and Angela Yao. Assembly101: A large-scale multi-view video dataset for understanding procedural activities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco...

    2022

  4. [12]

    Cross-task weakly supervised learning from instructional videos

    Dimitri Zhukov, Jean-Baptiste Alayrac, Ramazan Gokberk Cinbis, David Fouhey, Ivan Laptev, and Josef Sivic. Cross-task weakly supervised learning from instructional videos. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3537–3545,

  5. [13]

    Howto100m: Learning a text-video embedding by watching hundred million narrated video clips

    Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF international conference on computer vision, pages ...

    2019

  6. [14]

    Learning fine-grained view-invariant representa- tions from unpaired ego-exo videos via temporal alignment

    Zihui Sherry Xue and Kristen Grauman. Learning fine-grained view-invariant representa- tions from unpaired ego-exo videos via temporal alignment. Advances in Neural Information Processing Systems, 36:53688–53710, 2023. 2

    2023

  7. [15]

    Timeception for complex action recognition

    Noureldien Hussein, Efstratios Gavves, and Arnold WM Smeulders. Timeception for complex action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 254–263, 2019. 2

    2019

  8. [16]

    Videograph: Recognizing minutes-long human activities in videos

    Noureldien Hussein, Efstratios Gavves, and Arnold WM Smeulders. Videograph: Recognizing minutes-long human activities in videos. arXiv preprint arXiv:1905.05143, 2019

    1905 arXiv

  9. [17]

    Graph-based high-order relation modeling for long-term action recognition

    Jiaming Zhou, Kun-Yu Lin, Haoxin Li, and Wei-Shi Zheng. Graph-based high-order relation modeling for long-term action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8984–8993, 2021. 2

    2021

  10. [18]

    Unsupervised learning from narrated instruction videos

    Jean-Baptiste Alayrac, Piotr Bojanowski, Nishant Agrawal, Josef Sivic, Ivan Laptev, and Simon Lacoste-Julien. Unsupervised learning from narrated instruction videos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4575–4583, 2016. 2

    2016

  11. [19]

    Modeling by example

    Thomas Funkhouser, Michael Kazhdan, Philip Shilane, Patrick Min, William Kiefer, Ayellet Tal, Szymon Rusinkiewicz, and David Dobkin. Modeling by example. ACM transactions on graphics (TOG), 23(3):652–663, 2004. 3

    2004

  12. [20]

    Proba- bilistic reasoning for assembly-based 3d modeling

    Siddhartha Chaudhuri, Evangelos Kalogerakis, Leonidas Guibas, and Vladlen Koltun. Proba- bilistic reasoning for assembly-based 3d modeling. In ACM SIGGRAPH, pages 1–10, 2011

    2011

  13. [21]

    A proba- bilistic model for component-based shape synthesis

    Evangelos Kalogerakis, Siddhartha Chaudhuri, Daphne Koller, and Vladlen Koltun. A proba- bilistic model for component-based shape synthesis. Acm Transactions on Graphics (TOG), 31(4):1–11, 2012

    2012

  14. [22]

    Assembly-based conceptual 3d modeling with unlabeled components using probabilistic factor graph

    Prakhar Jaiswal, Jinmiao Huang, and Rahul Rai. Assembly-based conceptual 3d modeling with unlabeled components using probabilistic factor graph. Computer-Aided Design, 74:45–54,

  15. [23]

    Learning 3d part assembly from a single image

    Yichen Li, Kaichun Mo, Lin Shao, Minhyuk Sung, and Leonidas Guibas. Learning 3d part assembly from a single image. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VI 16, pages 664–682. Springer, 2020. 3

    2020

  16. [24]

    Componet: Learning to generate the unseen by part synthesis and composition

    Nadav Schor, Oren Katzir, Hao Zhang, and Daniel Cohen-Or. Componet: Learning to generate the unseen by part synthesis and composition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8759–8768, 2019

    2019

  17. [25]

    Pq-net: A generative part seq2seq network for 3d shapes

    Rundi Wu, Yixin Zhuang, Kai Xu, Hao Zhang, and Baoquan Chen. Pq-net: A generative part seq2seq network for 3d shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 829–838, 2020. 3

    2020

  18. [26]

    Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding

    Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas J Guibas, and Hao Su. Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogn...

    2019

  19. [27]

    Generative 3d part assembly via dynamic graph learning

    Guanqi Zhan, Qingnan Fan, Kaichun Mo, Lin Shao, Baoquan Chen, Leonidas J Guibas, Hao Dong, et al. Generative 3d part assembly via dynamic graph learning. Advances in Neural Information Processing Systems, 33:6315–6326, 2020. 3 12

    2020

  20. [28]

    Rgl-net: A recurrent graph learning framework for progressive part assembly

    Abhinav Narayan, Rajendra Nagar, and Shanmuganathan Raman. Rgl-net: A recurrent graph learning framework for progressive part assembly. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 78–87, 2022. 3

    2022

  21. [29]

    3d part assembly genera- tion with instance encoded transformer.IEEE Robotics and Automation Letters, 7(4):9051–9058,

    Rufeng Zhang, Tao Kong, Weihao Wang, Xuan Han, and Mingyu You. 3d part assembly genera- tion with instance encoded transformer.IEEE Robotics and Automation Letters, 7(4):9051–9058,

  22. [30]

    Ha-vid: a human assembly video dataset for com- prehensive assembly knowledge understanding

    Hao Zheng, Regina Lee, and Yuqian Lu. Ha-vid: a human assembly video dataset for com- prehensive assembly knowledge understanding. Advances in Neural Information Processing Systems, 36, 2024. 5, 29

    2024

  23. [31]

    Ikea ego 3d dataset: Understanding furniture assembly actions from ego-view 3d point clouds

    Yizhak Ben-Shabat, Jonathan Paul, Eviatar Segev, Oren Shrout, and Stephen Gould. Ikea ego 3d dataset: Understanding furniture assembly actions from ego-view 3d point clouds. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 4355–4364, 2...

    2024

  24. [32]

    Aligning step-by-step instructional diagrams to video demonstrations

    Jiahao Zhang, Anoop Cherian, Yanbin Liu, Yizhak Ben-Shabat, Cristian Rodriguez, and Stephen Gould. Aligning step-by-step instructional diagrams to video demonstrations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2483–2492,

  25. [33]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.02643, 2023. 5

    2023 arXiv

  26. [34]

    Ep n p: An accurate o (n) solution to the p n p problem

    Vincent Lepetit, Francesc Moreno-Noguer, and Pascal Fua. Ep n p: An accurate o (n) solution to the p n p problem. International journal of computer vision, 81:155–166, 2009. 6

    2009

  27. [35]

    Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography

    Martin A Fischler and Robert C Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6):381–395, 1981. 6

    1981

  28. [36]

    Dynamic graph cnn for learning on point clouds

    Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph cnn for learning on point clouds. ACM Transactions on Graphics (tog), 38(5):1–12, 2019. 7

    2019

  29. [37]

    Cnos: A strong baseline for cad-based novel object segmentation

    Van Nguyen Nguyen, Thibault Groueix, Georgy Ponimatkin, Vincent Lepetit, and Tomas Hodan. Cnos: A strong baseline for cad-based novel object segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2134–2140, 2023. 8, 9

    2023

  30. [38]

    Sam-6d: Segment anything model meets zero-shot 6d object pose estimation

    Jiehong Lin, Lihua Liu, Dekun Lu, and Kui Jia. Sam-6d: Segment anything model meets zero-shot 6d object pose estimation. arXiv preprint arXiv:2311.15707, 2023. 8, 9

    2023 arXiv

  31. [39]

    Megapose: 6d pose estimation of novel objects via render & compare

    Yann Labbé, Lucas Manuelli, Arsalan Mousavian, Stephen Tyree, Stan Birchfield, Jonathan Tremblay, Justin Carpentier, Mathieu Aubry, Dieter Fox, and Josef Sivic. Megapose: 6d pose estimation of novel objects via render & compare. arXiv preprint arXiv:2212.06870, 2022. 8, 9

    2022 arXiv

  32. [40]

    Midas v3

    Reiner Birkl, Diana Wofk, and Matthias Müller. Midas v3. 1–a model zoo for robust monocular relative depth estimation. arXiv preprint arXiv:2307.14460, 2023. 8

    2023 arXiv

  33. [41]

    Perceiving 3d human-object spatial arrangements from a single image in the wild

    Jason Y Zhang, Sam Pepose, Hanbyul Joo, Deva Ramanan, Jitendra Malik, and Angjoo Kanazawa. Perceiving 3d human-object spatial arrangements from a single image in the wild. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Par...

    2020

  34. [42]

    Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes

    Stefan Hinterstoisser, Vincent Lepetit, Slobodan Ilic, Stefan Holzer, Gary Bradski, Kurt Kono- lige, and Nassir Navab. Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes. In Computer Vision–ACCV 2012: 11th Asian Conferenc...

    2012

  35. [43]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024. 9

    2024 arXiv

  36. [44]

    Mose: A new dataset for video object segmentation in complex scenes

    Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, Philip HS Torr, and Song Bai. Mose: A new dataset for video object segmentation in complex scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20224–20234, 2023. 9

    2023

  37. [45]

    The 2017 davis challenge on video object segmentation

    Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung, and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint arXiv:1704.00675, 2017. 9

    2017 arXiv

  38. [46]

    Lvos: A benchmark for long-term video object segmentation

    Lingyi Hong, Wenchao Chen, Zhongying Liu, Wei Zhang, Pinxue Guo, Zhaoyu Chen, and Wenqiang Zhang. Lvos: A benchmark for long-term video object segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13480–13492, 2023. 9

    2023

  39. [47]

    The 2nd large-scale video object segmentation challenge - video object segmentation track, October 2019

    Linjie Yang, Yuchen Fan, and Ning Xu. The 2nd large-scale video object segmentation challenge - video object segmentation track, October 2019. 9

    2019

  40. [48]

    Putting the object back into video object segmentation

    Ho Kei Cheng, Seoung Wug Oh, Brian Price, Joon-Young Lee, and Alexander Schwing. Putting the object back into video object segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3151–3161, 2024. 9

    2024

  41. [49]

    not visible

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 10 14 Supplementary Material for IKEA Manuals at ...

    2023 arXiv

  42. [50]

    Select the appropriate category, subcategory, object, and step for the video you want to annotate

  43. [51]

    • Use the provided tools, such as a brush or eraser, to refine the mask based on the feedback provided

    In the Segmentation Mask mode: • Select points that best represent the overall shape and area of the part to ensure optimal performance of the Segment Anything Model (SAM). • Use the provided tools, such as a brush or eraser, to refine the mask based on the feedback provided

  44. [52]

    • Review the rendered image and adjust the selected points if necessary to improve the alignment between the 3D model and the 2D video frame

    In the 2D-3D Points Correspondence mode: • Select corresponding points on the 3D model and the 2D video frame that represent key features or edges of the furniture parts. • Review the rendered image and adjust the selected points if necessary to improve the alignment between t...

  45. [53]

    Navigate frames using the ’Next Frame’ button and review the predicted points from the TAPIR model, modifying any unsatisfactory points

  46. [54]

    E.2 Mask Re-Annotation Interface (Fig

    Review the segmented images and estimated poses for accuracy and consistency, and submit the annotations. E.2 Mask Re-Annotation Interface (Fig. A9)

  47. [55]

    mask was annotated to the wrong part,

    Review Previous Mask: • The interface will display the previously annotated mask in the bottom left corner of the screen, along with the reason for the decline. Reviewing the previous mask and the reason for the decline helps the annotator understand the required corrections. ...

  48. [56]

    These tools allow precise modifications to the mask

    Refine Mask: • Use the provided tools, such as a brush or eraser, to refine the mask based on the feedback provided. These tools allow precise modifications to the mask. • Ensure the refined mask accurately captures the entire part while excluding any neigh- bouring parts or b...

  49. [57]

    Additional Buttons: (Same as in the Segmentation Mask Annotation Interface.)

  50. [58]

    This final review step verifies that the necessary corrections have been made

    Review and Submit: Review the refined mask for accuracy and completeness, ensuring it addresses the reason for the decline. This final review step verifies that the necessary corrections have been made. Submit the updated mask using the provided submission functionality to sav...

  51. [59]

    Review the initial poses of all parts in the frame, estimated from the previous annotation

  52. [60]

    Use the provided controls to adjust the position and orientation of each part in the camera frame

  53. [61]

    Ensure that the relative positions and orientations of the parts are consistent with the assembly process

  54. [62]

    Review the refined poses for accuracy and submit the updated poses. By following these instructions and leveraging the provided video demonstrations, annotators can effectively use the annotation interfaces to generate high-quality segmentation masks, 2D-3D point correspondenc...

    2023

  55. [63]

    For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? [Yes] Justification: The dataset and experiments match our main claims. See Section 3 and Section 5 for details. (b) Did you describe the lim...

  56. [64]

    If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results? [N/A] (b) Did you include complete proofs of all theoretical results? [N/A] Justification: The paper does not include theoretical results; it is primarily focu...

  57. [65]

    for benchmarks)

    If you ran experiments (e.g. for benchmarks)... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] Justification: The datasets we build upon are cited and discussed i...

  58. [66]

    (a) If your work uses existing assets, did you cite the creators? [Yes] Justification: The datasets we build upon are cited and discussed in Section 4

    If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators? [Yes] Justification: The datasets we build upon are cited and discussed in Section 4. (b) Did you mention the licen...

  59. [67]

    If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable? [Yes] Justification: Details for the annotation process and compensation for annotators are in Appendi...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.