{"id":"5b45f96f-9049-4598-9032-af40d2f17866","arxiv_id":"2504.21749","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Common3D learns deformable 3D morphable models for common objects from object-centric videos using neural features, and solves pose, segmentation, and semantic correspondence in a self-supervised, zero-shot manner.","lead":"Common3D learns category-specific 3D morphable models for common objects from casual object-centric videos, without manual 3D annotations. Once trained, a single model estimates 3D pose, instance segmentation, and semantic correspondences from one image.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Canonical poses inherited from UOP3D (Sec. 3.3) entangle the training signal with the baseline; the +6.3/+4.4 pose gains could reflect distillation rather than an independent 3D prior. A retraining with independent canonicalization would settle it.","rationale":"I read the paper in good faith. The method combines well-established components (DMTet, DINOv2, affine deformation, contrastive surface probabilities) in a coherent training loop, and the paper provides external-benchmark results, ablations, and released code, which are real supporting evidence. The strongest claim, however, is the superiority over UOP3D and the 'completely self-supervised' framing. The critical assumption is that Common3D learns an independent category-level 3D prior; but the canonical coordinate frame is bootstrapped from UOP3D (Sec. 3.3). This makes the comparison against UOP3D a teacher-student comparison: the baseline supplies the geometry used to train the student. The +6.3/+4.4 point pose improvements and the segmentation/correspondence results could therefore reflect distillation plus better inverse rendering, rather than a categorically better self-supervised prior. I am not claiming the method is unsound: the reported numbers are plausible, the code is public, and the ablation study supports the design choices. The concern is about independence and framing; a retraining with a different canonicalization is the direct way to test it. Because this concern matches the reader's weakest assumption and supports the CONDITIONAL verdict, I do not recommend changing the verdict.","tokens_in":18375,"tokens_out":14606,"duration_ms":159992,"concrete_test":"Retrain the car and chair models using the same released code, but replace UOP3D canonicalization with an independent unsupervised alignment: for each CO3D video, run COLMAP to get the point cloud, then align all instances to a shared canonical frame by Procrustes fitting to a category-level template initialized from the mean of the SFM point clouds (or, if available, use a different unsupervised viewpoint method). Evaluate PASCAL3D+ 30° accuracy exactly as in Tables 1-2, keeping the same masks, backbone, and hyperparameters. If the 30° gains over UOP3D persist at similar magnitude, the dependency is benign; if they shrink or reverse, the reported improvement is substantially inherited from UOP3D rather than from Common3D's representation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Common3D's central claim (Sec. 5) has two parts: it is the first completely self-supervised method to estimate shape, pose, correspondence, and segmentation for common objects, and it improves 30° pose accuracy over UOP3D by 6.3 points on PASCAL3D+ and 4.4 points on ObjectNet3D. The load-bearing assumption on which both parts rest is that the training signal is independent of the baseline. In Sec. 3.3 the authors state: 'We achieve this by adopting the method proposed in [52] to get initial camera pose annotations in an unsupervised manner.' The template, the deformation model, and the contrastive appearance loss (Eqs. 11-12) are all optimized in the coordinate frame that UOP3D defines. At inference, the pose is found by inverse rendering (Eq. 13) against the features learned in that frame. Therefore the geometry and the correspondence features are not independent of UOP3D; they inherit any bias in UOP3D's canonical poses, including possible reflected or azimuth-shifted conventions. The headline comparison is then a teacher-student setup: the baseline provides the pseudo-labels that define the geometry, and the reported gains may measure a more robust optimization or smoother features rather than a more accurate category-level 3D prior. This is a concern about evaluation independence and about the word 'completely' (PointRend masks and SFM point clouds are also used, and PointRend is a supervised mask estimator), not about internal consistency. The method is coherent and code is released; the concern is that the key superiority claim is entangled with the baseline it is compared against.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Common3D, a method for learning category-specific 3D morphable models (3DMMs) from collections of object-centric videos without manual 3D annotations. The model combines a DMTet-based hybrid volumetric-mesh template, an instance-level affine deformation field conditioned on an image latent code, and a neural feature field on the mesh surface. Appearance is represented by DINOv2 features refined by a learned adapter, and training uses geometric losses (mask, chamfer, SDF, deformation regularization) together with a contrastive appearance objective based on surface probabilities. At inference, pose is estimated by inverse rendering in feature space and shape by the latent encoder. The paper evaluates single-image 3D pose, instance segmentation, and semantic correspondence on PASCAL3D+, ObjectNet3D, and SPair-71k, reporting improvements over the unsupervised baseline UOP3D and claims to be the first completely self-supervised method to solve these tasks jointly. Code is released.","tokens_in":18688,"tokens_out":4653,"duration_ms":47120,"significance":"If the central claims are sustained, this is a meaningful contribution: it extends 3DMMs beyond faces and bodies to common object categories without manual 3D supervision, demonstrates that feature-space appearance representations can make inverse rendering tractable, and shows improvements on external benchmarks. The evaluation is conducted on held-out in-the-wild datasets, so the main results are not fitted to the test sets, and the ablation study in Table 6 supports the importance of the adapter and the dense appearance loss. The release of code is a further strength. However, the headline 'completely self-supervised' claim is partially undermined by the use of PointRend masks, precomputed structure-from-motion point clouds, and UOP3D-derived canonical camera poses, and the comparison against UOP3D is entangled with the training coordinate frame. The contribution would be strengthened substantially by an independent canonicalization experiment and by uncertainty quantification on the aggregate metrics.","major_comments":[{"comment":"The canonical camera poses used to train Common3D are obtained by adopting UOP3D [52], as stated in Sec. 3.3 and Sec. 4.1. The template shape, the deformation model, and the contrastive appearance objective (Eqs. 11-12) are all optimized in the coordinate frame defined by those poses, and inference in Eq. (13) is performed against features learned in that same frame. Consequently, the reported pose improvements over UOP3D (69.2% to 75.3% on PASCAL3D+ and 52.4% to 56.8% on ObjectNet3D) may reflect a better optimization or smoother correspondence features operating on UOP3D's pseudo-labels rather than an independent category-level 3D prior. The paper should either retrain with an independent canonicalization, such as a different unsupervised pose aligner or a canonical frame derived directly from the SFM point clouds, or demonstrate robustness to the choice of canonical frame. Without this, the comparison is closer to a teacher-student evaluation than a head-to-head one, and the 'completely self-supervised' claim is overstated.","section":"Sec. 3.3, Sec. 4.1, Tables 1-2"},{"comment":"The claim that Common3D is 'completely self-supervised' is contradicted by the training data dependencies described in Sec. 3.3 and Sec. 4.1: masks come from PointRend [26] in Eq. (2), a supervised instance segmentation method; 3D point clouds come from a precomputed SFM pipeline [48] in Eq. (4); and canonical camera poses come from UOP3D [52] in Eqs. (11)-(12). While these are not manual 3D shape or pose annotations, they are precomputed geometric pseudo-labels that shape the learned model. Please qualify the self-supervision claim at the outset, for example by stating explicitly that the method uses no manual keypoint, pose, or 3D shape annotations but does rely on precomputed masks, SFM reconstructions, and an unsupervised canonicalization from a prior method.","section":"Abstract, Sec. 1, Conclusion"},{"comment":"The paper reports only point estimates for the main quantitative results, yet the text states that the improvements are 'significant' (Sec. 4.2). Given the per-category variance visible in Tables 1-2 (e.g., ObjectNet3D suitcase: UOP3D 44.7 vs. Ours 38.5; keyboard: 26.9 vs. 34.5) and the modest per-category sample sizes, the aggregate gains need uncertainty quantification, such as confidence intervals from bootstrapping over test images or error bars across training seeds. At minimum, this would make the headline claims about the average improvement over UOP3D interpretable.","section":"Tables 1-5, Sec. 4.2"}],"minor_comments":[{"comment":"The denominator of the surface probability in Eq. (9) appears to be missing a negative sign in the exponent: it should be exp(-||vi - vk||^2 / (2 sigma^2)) to match the numerator and the usual Gaussian form.","section":"Eq. (9)"},{"comment":"The sentence 'We achieve adopt the method proposed in [52]...' contains a grammatical error; it should read 'We adopt the method proposed in [52]...' or 'We achieve this by adopting...'.","section":"Sec. 4.1"},{"comment":"The PCK@0.1 metric is used in Table 5 and Sec. 4.4 without a definition, and the fixed weighting 0.8/0.2 between DINOv2 and adapter features is reported without justification or ablation. Please define the metric and either ablate or justify the weighting.","section":"Sec. 4.4, Table 5"},{"comment":"Farthest point sampling is mentioned as part of the appearance objective, but the main text defers the details; the supplementary Table 7 lists 150 sampled vertices. Please state this number and the purpose of the sampling directly in Sec. 3.3 for readability.","section":"Sec. 3.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is well structured and the method is plausible, but the independence of the evaluation from the UOP3D baseline is the key question that needs to be resolved before publication. I would ask the authors to provide either a retraining with an independent canonicalization or a clear sensitivity analysis of the canonical frame, and to temper the 'completely self-supervised' claim. The point-estimate-only reporting is also worth addressing in the revision. If the independence concern cannot be addressed, the paper should be reframed as improving UOP3D within its own coordinate frame rather than as an independent fully self-supervised method."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: Common3D is a solid, well-engineered step in self-supervised category-level 3D understanding, and the core mechanism — jointly training a deformable DMTet template with a DINOv2 adapter through a mesh-defined contrastive objective — is genuinely new as a combination. The paper is honest about most of its component debts, ships code, and ablates the design choices properly. But the headline 'completely self-supervised' claim is overstated, and the headline numbers against UOP3D are not fully independent: the canonical poses that define the training frame come from the authors' own UOP3D. That does not sink the paper, but it should temper both the claim and the comparison.\n\nWhat is new: prior work either restricts to animals (3D Fauna, MagicPony), uses a rigid template (UOP3D), or needs synthetic data. Common3D is the first to train a deformable DMTet morphable model plus a correspondence adapter from casual video. The gains — +6.3 on PASCAL3D+ 30° pose, +4.4 on ObjectNet3D, plus segmentation and PCK improvements — are consistent across the tables rather than driven by one lucky category. Table 6 is informative: the adapter and dense appearance loss do real work, and instance deformation helps fine-grained pose and segmentation. Evaluation is on external benchmarks and code is released, so I treat the results as reproducible evidence.\n\nSoft spots, in order of size. First, evaluation independence. Section 3.3 adopts UOP3D to obtain canonical camera poses, and the template, deformation field, and appearance features are all optimized in UOP3D's coordinate frame. The main baseline is then the method that defined the frame. The +6.3/+4.4 could partly reflect a smoother optimization surface or feature distillation rather than a more accurate category-level 3D prior. A retraining with independent canonicalization would settle it. This is a genuine limitation of the comparison, not a fatal flaw in the method. Second, 'completely self-supervised' is inaccurate: PointRend masks come from a supervised estimator, SFM point clouds are used, and UOP3D poses are the authors' own unsupervised method. The claim should be something like 'no manual 3D annotations.' Third, minor: Tables 1–5 report point estimates without error bars; with 7–20 categories I would like some sense of variance. The independence concern is the main thing I would push on in review.\n\nWho this is for: people working on category-level pose, 3D morphable models, or correspondence learning, and anyone interested in whether a self-supervised pipeline is genuinely independent of its own scaffolding. It deserves a serious referee. My recommendation: send to review, require a careful discussion (or better, an experiment) addressing the UOP3D entanglement, and ask the authors to soften the 'completely self-supervised' wording.","headline":"Solid self-supervised 3DMM paper with a real but overstated 'completely self-supervised' claim — the head-to-head numbers are entangled with the authors' own UOP3D poses, but the method is coherent, well-ablated, and deserves a serious review.","tokens_in":19296,"tokens_out":7227,"would_cite":true,"duration_ms":61961,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Common3D learns 3D morphable models of everyday objects from casual videos alone, with no manual 3D annotations.","keywords":["self-supervised learning","3D morphable models","neural feature fields","semantic correspondence","3D pose estimation","inverse rendering","object-centric videos","DMTet"],"falsifier":"Train the same recipe on a category with strong mirror or rotational symmetry, such as mugs or bottles, and compare the learned template and predicted poses against independent structure-from-motion reconstructions after reflecting one symmetry-related reconstruction; if predictions flip along the symmetry axis instead of staying consistent, the canonical pose initialization rather than the contrastive correspondence mechanism is carrying the result.","tokens_in":18151,"feed_emoji":"📐","tokens_out":5772,"duration_ms":55876,"temperature":0.7,"pith_summary":"This paper claims that 3D morphable models, deformable category-level shape templates with per-vertex appearance, can be learned for everyday object categories from casual object-centric videos without any manual 3D annotations. It argues that this becomes practical when object appearance is stored as neural features rather than RGB colors, because inverse rendering against a feature representation rewards features that encode 2D-3D correspondences. Trained once per category, the model is said to estimate 3D pose, instance segmentation, shape, and semantic correspondences from a single novel image. The reported gains over the prior self-supervised method are from 69.2% to 75.3% average 30-degree pose accuracy on PASCAL3D+ and from 52.4% to 56.8% on ObjectNet3D.","feed_headline":"Casual videos train 3D models of common objects, no 3D labels","feed_subtitle":"One trained model per category gives pose, shape, and semantic correspondences from a single image.","key_machinery":"The load-bearing mechanism is the coupling between the deformable template mesh and a contrastive appearance objective. The template supplies a shared 3D coordinate frame: after deformation, corresponding vertices across instances are known by construction, and those correspondences are used to supervise the feature adapter. The appearance loss treats each 2D image feature as a query over all vertex features plus a background token, computing a surface-probability distribution via a softmax with temperature $\\kappa$, while the target distribution is the Euclidean-distance surface probability on the mesh. Minimizing their cross-entropy forces the 2D features and the 3D vertex features to agree on which point of the object a pixel belongs to, with farthest-point vertex sampling and the background token preventing collapse to trivial constant features. The deformed mesh vertices also give direct point access for chamfer-distance shape fitting and rasterization-based mask fitting, so geometry and appearance are optimized jointly in the same canonical frame.","core_discovery":"Common3D's central claim is that a deformable 3D morphable model can be trained end-to-end without any 3D supervision and then serve as one prior that solves multiple single-image tasks. The model is a learned category-level template mesh, extracted from a signed distance field via differentiable marching tetrahedra, whose vertices carry semantic feature vectors generated by a feature field. An image-conditioned affine deformation field morphs the template to the instance geometry, and a frozen DINOv2 backbone with a small trainable adapter provides image features. Training combines geometric losses, silhouette overlap, distance-transform expansion, chamfer distance to per-video structure-from-motion point clouds, smoothness and Eikonal regularization, with a contrastive appearance loss in which every image feature must be explained by a vertex feature through surface probabilities derived from the current mesh. The paper argues this makes the adapter learn correspondence features that are more distinct than raw DINOv2 features, and makes inverse rendering on those features reliably recover pose. On this basis, Common3D claims to be the first completely self-supervised method that estimates shape, pose, and semantic correspondence of common objects from a single image.","pith_inferences":["A natural extension is to test whether the same framework scales to categories with large topological variation, since DMTet and the learned SDF should in principle handle topology changes that rigid template meshes cannot.","The reliance on a separately estimated canonical pose suggests a testable variant: replace the UOP3D pose initialization with poses from multi-view triangulation or object symmetry and compare downstream accuracy, isolating how much of the gain comes from the feature-adapter mechanism.","If correspondence features genuinely improve through this loop, the trained adapter could transfer to related tasks such as part segmentation, keypoint discovery, or video object tracking, where the same pixel-to-surface question is asked.","The small training budget, under ten hours per category on a single GPU, raises the possibility of interactive model creation: a user shoots a short video of an object and immediately obtains a category-specific 3D morphable model."],"forward_implications":["A single category model trained on up to 50 videos per category yields zero-shot 3D pose, instance segmentation, and semantic correspondence on unseen in-the-wild images of that category.","The average gains of 6.3 and 4.4 points in 30-degree pose accuracy on PASCAL3D+ and ObjectNet3D imply that the correspondence-aware features transfer across the domain gap from CO3D videos to images in the wild.","Categories with little shape variation across instances, such as cellphone, microwave, and toaster, do not improve, suggesting the deformation model contributes most where instance geometry varies.","Because the adapter is trained jointly with the geometry, correspondence quality can improve as the shape model improves, enabling a self-improving loop between 2D features and the 3D prior.","The same trained model supports multiple tasks, so task-specific heads or per-task training data are not required at test time."],"supporting_citations":[{"why":"Supplies the canonical camera poses used to align all training videos into a shared coordinate frame and serves as the main self-supervised baseline.","marker":"[52]"},{"why":"Provides the differentiably extractable mesh from the signed distance field, enabling mesh-based regularization and rendering.","marker":"[50]"},{"why":"Supplies the frozen DINOv2 backbone whose features the trainable adapter enhances for correspondence learning.","marker":"[41]"},{"why":"Supplies the contrastive surface-probability formulation used by the appearance loss to prevent feature collapse.","marker":"[4]"},{"why":"Provides per-video structure-from-motion point clouds and camera poses that drive the geometric reconstruction losses.","marker":"[48]"},{"why":"Supplies the object-centric videos, point clouds, and category coverage used for training and evaluation.","marker":"[45]"},{"why":"Supplies the object masks used in the silhouette and distance-transform objectives.","marker":"[26]"},{"why":"Supplies the affine transformation field used for instance-level deformation of the category-level template.","marker":"[82]"},{"why":"Supplies the differentiable rasterizer used to render the mesh and its feature field during training and inverse rendering.","marker":"[29]"}],"fun_headline_variants":["Self-supervised 3D models of common objects from casual videos","No 3D labels: learn object 3D shape and pose from videos","Neural features, not RGB, for zero-shot 3D object tasks","One trained model per category: pose, shape, correspondence from one image"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire model is trained in a canonical coordinate frame whose camera poses come from an earlier unsupervised pose-estimation method (UOP3D [52]); if those poses are systematically biased, the learned template, deformations, features, and all downstream evaluations inherit that bias, and the comparison against UOP3D is not fully independent.","fun_headline_variants_meta":{"raw":{"variants":["Self-supervised 3D models of common objects from casual videos","No 3D labels: learn object 3D shape and pose from videos","Neural features, not RGB, for zero-shot 3D object tasks","One trained model per category: pose, shape, correspondence from one image"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000371,"raw_usage":{"total_tokens":2041,"prompt_tokens":1053,"completion_tokens":988,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":669,"completion_tokens_details":{"reasoning_tokens":907}},"tokens_in":669,"tokens_out":988,"duration_ms":9832,"temperature":1.0,"reasoning_tokens":907,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:55:18.499492+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same recipe on a category with strong mirror or rotational symmetry, such as mugs or bottles, and compare the learned template and predicted poses against independent structure-from-motion reconstructions after reflecting one symmetry-related reconstruction; if predictions flip along the symmetry axis instead of staying consistent, the canonical pose initialization rather than the contrastive correspondence mechanism is carrying the result.","supporting_citations":[{"cited_title":"Unsupervised learning of category-level 3d pose 10 from object-centric videos","cited_arxiv_id":null,"evidence_quote":"Supplies the canonical camera poses used to align all training videos into a shared coordinate frame and serves as the main self-supervised baseline."},{"cited_title":"Structure-from-Motion Revisited","cited_arxiv_id":null,"evidence_quote":"Provides per-video structure-from-motion point clouds and camera poses that drive the geometric reconstruction losses."},{"cited_title":"Pointrend: Image segmentation as rendering","cited_arxiv_id":null,"evidence_quote":"Supplies the object masks used in the silhouette and distance-transform objectives."},{"cited_title":"Deep implicit templates for 3d shape representation","cited_arxiv_id":null,"evidence_quote":"Supplies the affine transformation field used for instance-level deformation of the category-level template."},{"cited_title":"Modular primitives for high-performance differentiable rendering","cited_arxiv_id":null,"evidence_quote":"Supplies the differentiable rasterizer used to render the mesh and its feature field during training and inverse rendering."}],"review_version":1}