{"id":"11844ec5-7cc0-4ad6-8aa7-4ad36eeed9b3","arxiv_id":"2411.10275","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A coarse-to-fine neural network learns camera pose and 4D shape of deforming objects from multiple RGB videos, adding a local quadratic deformation model to a BANMo-style neural radiance field.","lead":"This paper builds a computer vision system that reconstructs a 3D shape that changes over time, like a moving pet, directly from ordinary videos. It does this without requiring 3D templates, 3D training data, or precomputed camera positions, and the authors report better detail than a leading prior method on several benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's 'unknown object' claim depends on DensePose CSE from Eq. (17); all experiments use categories with pretrained CSE models, so the method's generality to novel deformable categories is unvalidated.","rationale":"The reader identified the CSE dependence as the weakest assumption, and my reading agrees: it is the most load-bearing concern because it directly targets the abstract's 'unknown object' and 'self-supervised' claims. The paper contributes a plausible coarse-to-fine extension of BANMo, and the ablation supports the value of the fine canonical network and the local quadratic deformation model. However, those contributions are evaluated only on categories with pretrained DensePose CSE models, and the CSE terms are a fixed part of the optimization in Eq. (19). There is also a secondary overstatement in Table 1: on AMA-swing, BANMo reports better CD and F@2% than the proposed method, so 'outperforms state-of-the-art approaches' is not uniformly supported; but that is a reporting issue rather than the core logical weakness. In addition, Eq. (15) appears to have a typo in L^f_sil (the fine silhouette loss compares to the coarse prediction rather than the ground-truth mask), though the surrounding text could be read as intentional teacher-student supervision; this is worth checking but is not the central claim. Because the CSE concern does not contradict the paper's internal logic, it supports the existing CONDITIONAL verdict rather than requiring a different one: the method may work well for CSE-covered categories, but the claimed generality remains unproven. A controlled CSE ablation is the one concrete check that would settle the concern.","tokens_in":1943,"tokens_out":1147,"duration_ms":63555,"concrete_test":"Re-run the full pipeline on the provided casual-cat data with lambda_reg=0 in Eq. (19), which removes both CSE losses L_cse2D and L_cse3D from Eq. (17) while keeping photometric, silhouette, optical-flow, and cycle losses active. Compare the resulting meshes and view-synthesis quality against the reported results; if cross-sequence alignment degrades, duplicate limbs appear, or surfaces split, the DensePose CSE anchor is load-bearing and the method's generality to categories without pretrained CSE models is unsupported. If quality is unchanged, the concern is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the method reconstructs an 'unknown object' in a self-supervised manner with no 3D template or 3D training data. The load-bearing assumption is that DensePose Continuous Surface Embeddings (CSE) used in Eq. (17) are available for the object's category. Section 3.1 says canonical embeddings 'are matched by pixels from different visual conditions, enabling long-range correspondence across sequences,' but Eqs. (17) and (19) supervise those embeddings with a pretrained category-specific DensePose network [26] through L_cse2D and L_cse3D. The losses are not optional in the formulation: they are part of the global objective minimized in Eq. (19). All experiments use categories with existing CSE models (cat, dog, human). For a genuinely novel deformable category, no pretrained CSE model exists, and the cross-sequence anchor disappears. The paper provides no experiment or analysis showing that the remaining photometric, silhouette, optical-flow, and cycle losses (Eqs. 14-16, 18) prevent the optimization from drifting or collapsing when CSE supervision is absent. The only qualitative mention of omitting 3D CSE constraining (Fig. 3 and Section 6.3) is not a controlled ablation, and Table 2 does not isolate the CSE terms. This is an unvalidated generality claim rather than an internal contradiction: the abstract's 'unknown object' and 'no 3D training data' wording goes beyond what the experiments and formulation demonstrate.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes 4DPV, a coarse-to-fine neural deformation model for jointly recovering camera pose and 4D shape of a dynamic object from multiple RGB videos without a 3D template or 3D training data. The method combines a Linear Blend Skinning coarse model, a local quadratic deformation network with spatio-temporal smoothness, and a fine canonical network, trained self-supervised with photometric, silhouette, optical-flow, CSE, and cycle-consistency losses. It is evaluated on four synthetic datasets (AMA-swing, AMA-samba, Eagle, Hands) and three real datasets (casual-cat, adult-5, casual-dog), with comparisons to BANMo and ViSER and an ablation on view synthesis.","tokens_in":13655,"tokens_out":4253,"duration_ms":39982,"significance":"If the claims hold, the contribution is a plausible extension of BANMo that improves fine detail in non-rigid NeRF reconstruction, and the authors release code, which aids reproducibility. The core architectural idea of combining LBS coarse deformation with a local quadratic deformation model is reasonable and is evaluated on both synthetic and in-the-wild datasets. However, the paper's central generality claim ('unknown object') is not fully supported because the method depends on category-specific DensePose CSE supervision, and the empirical support for 'outperforms state-of-the-art' is weakened by inconsistent results in Table 1 and the absence of error bars or significance tests. The manuscript is worth a major revision rather than rejection, since the identified issues are addressable with additional ablations, corrected metrics, and phrase revision.","major_comments":[{"comment":"The claim that the method recovers an 'unknown object' is load-bearing, but the optimization relies on category-specific Continuous Surface Embeddings from a pretrained DensePose network [26] through L_cse2D and L_cse3D in Eq. (17), which are part of the global objective in Eq. (19). All evaluated categories (cat, dog, human) already have pretrained CSE models; for a genuinely novel deformable category without a pretrained embedding model, the cross-sequence correspondence anchor disappears. The paper does not provide a controlled ablation that removes the CSE losses, and the qualitative note in Fig. 3 about 'without 3D CSE constraining' is not quantified in Table 2. Thus the abstract's 'unknown object' and 'no 3D training data' wording goes beyond what the formulation and experiments demonstrate.","section":"Abstract; Section 3.1; Eq. (17); Eq. (19)"},{"comment":"The claim of outperforming state-of-the-art approaches is not consistently supported by Table 1. The full method is worse than its own baseline on Hands (CD 6.0 vs. 5.4), worse than the '+ fcn' variant on Eagle (5.1 vs. 4.7) and Hands (6.0 vs. 4.8), and worse than the original BANMo on AMA-swing (10.0 vs. 9.1). No error bars, multiple-seed runs, or significance tests are reported, so the 'best on average' conclusion is fragile. The authors should report variance and/or a paired comparison to substantiate the superiority claim.","section":"Section 6.3; Table 1"},{"comment":"The fine silhouette loss is defined as ||s_c(\\bar{x}) - s_f(\\bar{x})||^2, comparing the coarse and fine silhouette predictions against each other rather than comparing the fine prediction to the ground-truth mask s(\\bar{x}). This contradicts the preceding sentence, which states the silhouette loss penalizes deviations between the predicted 2D shape and the segmentation mask. If implemented exactly as written, the fine network is not directly supervised by the ground-truth silhouette; the authors should correct the formula or clarify the intended teacher-student behavior.","section":"Eq. (15); Section 5"},{"comment":"The ablation study in Table 2 evaluates view synthesis quality (PSNR/SSIM) rather than 3D reconstruction accuracy, while the main quantitative comparison in Table 1 reports Chamfer distance and F-score. These two rankings can differ, and indeed the '+ fcn' variant already outperforms the full method on two datasets in Table 1. The qualitative claim in Fig. 3 about omitting 3D CSE constraining is not backed by a quantitative ablation isolating the CSE terms. The component analysis therefore does not clearly establish which proposed module is responsible for the reported gains.","section":"Section 6.3; Table 2; Fig. 3"}],"minor_comments":[{"comment":"Several F@2% entries are missing (BANMo on AMA-samba, ViSER on AMA-samba and possibly others); the authors should either report the values or explicitly state why they are omitted.","section":"Table 1"},{"comment":"The sentence 'We obtain the weight coefficients experimentally and fix them for all experiments, obtaining a non-overfitted solution' makes an unsupported claim; reporting a small hyperparameter sweep or a sensitivity check would be more appropriate.","section":"Section 6.1"},{"comment":"The notation A^t_{s,k} appears in the loss formulas but is not defined; clarify that it denotes MLP_Q evaluated at the sampled neighboring point.","section":"Eq. (12) and Eq. (13)"},{"comment":"The phrase 'we also employ this algorithm in our training setup' is ambiguous because it is unclear whether 'this algorithm' refers to BANMo or ViSER; rephrase for clarity.","section":"Section 6.3"},{"comment":"The text contains many OCR-like artifacts (e.g., 'coarse-to-\\ufb01ne', 'de\\ufb01ne', 'con\\ufb01gurations') and inconsistent hyphenation; a careful proofreading pass is needed.","section":"General"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuine but modest improvement over BANMo, and the headline claim of reconstructing an 'unknown object' goes further than the experiments support. The new pieces are a local quadratic deformation model (LQM) with spatial and temporal smoothness plus a fine canonical network on top of BANMo's LBS pipeline. Quadratic deformation was used in classic NRSfM, but the neural, per-point, regularized form is new. The ablation in Table 2 and the retrained BANMo baseline are useful, and the average Chamfer distance on four synthetic sequences drops from 8.8 to 7.6.\n\nSoft spots are real but not disqualifying. No error bars or repeated runs; on AMA-swing and Hands the full model is worse than BANMo or its own baseline. The abstract says 'unknown object' with no 3D template or 3D training data, but every experiment uses categories that already have a pretrained DensePose CSE model, and the CSE losses in Eq. (17) are part of the final objective. There is no test or discussion of what happens for a category without CSE. That is a genuine gap between claim and evidence. The real-data evaluation is qualitative only. Minor presentation glitches, like the '9×n' matrix in Eq. (8) looking like it should be n×9.\n\nNone of this sinks the paper. The work is coherent, the math is sound, and the authors acknowledge the over-constraining issue in Section 4.2. For anyone working on non-rigid reconstruction or animal capture, this is a useful reference and deserves a serious referee. I would send it to peer review, with the expectation that the authors address the CSE dependence and report variance.","headline":"Solid incremental extension of BANMo with a local quadratic deformation model, but the 'unknown object' claim is unvalidated because all experiments depend on pretrained category-specific DensePose CSE.","tokens_in":14168,"tokens_out":4632,"would_cite":false,"duration_ms":39863,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A coarse-to-fine neural deformation model recovers camera pose and 4D shape of unknown deformable objects from casual multi-view videos, without 3D templates or 3D training data.","keywords":["4D reconstruction","non-rigid radiance fields","neural deformation model","coarse-to-fine","camera pose estimation","self-supervised learning","continuous surface embeddings","animal reconstruction"],"falsifier":"Run the pipeline on a deformable category with no pre-trained CSE model (e.g., a horse or a cloth toy captured in several videos) and check whether the recovered canonical shape, per-frame meshes, and camera trajectories stay consistent across sequences; also ablate by removing the CSE losses (Eq. 17) and measuring how much the Chamfer distance and novel-view PSNR degrade on the existing cat and dog datasets.","tokens_in":13082,"feed_emoji":"🐈","tokens_out":2710,"duration_ms":27447,"temperature":0.7,"pith_summary":"The paper claims that simultaneous camera-pose estimation and high-detail 4D reconstruction of an unknown, freely deforming object can be achieved from multiple RGB videos in the wild, with no 3D template and no 3D supervision. The key is a coarse-to-fine neural deformation model: a linear blend skinning component captures articulated motion, while a newly introduced local quadratic deformation model with spatio-temporal smoothness captures fine, non-rigid details. Combined with canonical-space embeddings that provide cross-sequence correspondences, the model is trained self-supervised by rendering and comparing against input images. If correct, this would make detailed 4D capture of animals and other deformable objects practical from ordinary phone video, removing the need for category-specific 3D models or controlled capture setups.","feed_headline":"One network recovers 4D animal shapes from casual phone videos","feed_subtitle":"Coarse-to-fine deformation model estimates camera pose and detailed 3D shape without any 3D template or training data.","key_machinery":"The method combines three learned modules: (1) a coarse neural radiance field with Linear Blend Skinning (LBS) that maps points between canonical and image space, estimating root and bone poses from latent codes; (2) a fine deformation network, the Local Quadratic Model (LQM), which takes the coarse canonical point and a deformation code and outputs a 3x9 matrix of linear, quadratic, and cross-term coefficients, applied to an extended coordinate vector to produce a refined point, with losses enforcing spatial and temporal smoothness of the coefficient matrices; and (3) a fine canonical network with extra samples to refine shape and appearance. Cross-sequence correspondence is anchored by Continuous Surface Embeddings (CSE) learned in canonical space and matched to DensePose predictions in 2D, plus 2D/3D cycle consistency and optical-flow reprojection losses.","core_discovery":"The central claim is that decomposing the deformation into a coarse articulated part (linear blend skinning) and a fine part (a neural local quadratic model with spatial and temporal coherence) lets a single self-supervised pipeline jointly estimate camera poses, an implicit canonical shape, and a time-varying deformation field from multiple monocular RGB videos, even under uncontrolled lighting and unknown camera motion. The fine quadratic model, applied per point via an MLP, captures bending, stretching, and twisting that piecewise-linear models miss, while the fine canonical network avoids over-constraining the surface and preserves detail. The paper reports quantitative and qualitative improvements over BANMo and ViSER on synthetic and real animal and human datasets, including higher mesh fidelity and fewer artifacts such as extra legs or heads.","pith_inferences":["The method is not fully category-agnostic as stated, because the CSE correspondences rely on a pre-trained DensePose model that exists only for a few categories (cat, dog, human); a direct test on a category without such an embedding would show how far the self-supervised pipeline can stretch.","A natural extension would be to replace the fixed CSE with online-learned embeddings from the video collection itself, which would break the dependency on category-specific models and potentially generalize to arbitrary deformable objects.","The local quadratic model is a second-order polynomial deformation field per point; one could view it as a truncated Taylor expansion, and higher-order terms or a learned basis might capture even finer effects such as fur or cloth wrinkles, at the cost of more regularization.","The requirement for silhouettes and optical flow from off-the-shelf networks suggests that end-to-end learning of these prerequisites within the same framework, or making the method robust to their noise, would be a direct testable improvement."],"forward_implications":["If the method works as claimed, detailed 4D reconstruction of pets and other deformable animals becomes possible from casual multi-view phone footage, with no 3D scanning or category-specific template.","The coarse-to-fine decomposition suggests a general recipe: use a physically motivated coarse model (e.g., skinning) to avoid local minima, then add a local higher-order deformation to recover fine surface detail without losing global consistency.","The learned canonical shape and deformation can be reused for novel-view synthesis at arbitrary time steps, enabling free-viewpoint video of the reconstructed object.","The self-supervised nature of the pipeline means it could scale to large video collections of many objects, since it does not depend on 3D ground-truth data.","The ablation study indicates that both the fine deformation model and the fine canonical network contribute to quality, and the spatio-temporal smoothness terms are necessary to avoid artifacts."],"supporting_citations":[{"why":"BANMo is the main baseline and provides the LBS + canonical space + CSE framework that this paper extends with a fine deformation model.","marker":"[48]"},{"why":"NeRF supplies the volume rendering formulation and the implicit representation of appearance and density used throughout.","marker":"[24]"},{"why":"Continuous Surface Embeddings (CSE) from DensePose are used to establish cross-sequence correspondences via the matching losses in Eq. (17).","marker":"[26]"},{"why":"The quadratic deformation model is inspired by this prior work on non-rigid structure from motion, providing the extended-coordinate formulation in Eq. (8).","marker":"[10]"},{"why":"LASR introduces the LBS-based articulated shape reconstruction from monocular video that the coarse component builds upon.","marker":"[46]"},{"why":"ViSER is a state-of-the-art competitor used for quantitative comparison and to motivate the need for finer deformation modeling.","marker":"[47]"}],"fun_headline_variants":["Coarse-to-fine 4D pet from casual videos without templates","Self-supervised radiance fields rebuild animals from phone clips","One model estimates pose and 4D shape from multiple RGB streams","No 3D data, just videos: neural deformation captures fine motion","Quadratic deformation model lifts pets to 4D from monocular clips"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method relies on pre-trained category-specific surface embeddings (DensePose/CSE) to supply dense correspondences across sequences; for an object category without such a model, the cross-sequence anchor disappears and the self-supervised pipeline has no way to align the videos.","fun_headline_variants_meta":{"raw":{"variants":["Coarse-to-fine 4D pet from casual videos without templates","Self-supervised radiance fields rebuild animals from phone clips","One model estimates pose and 4D shape from multiple RGB streams","No 3D data, just videos: neural deformation captures fine motion","Quadratic deformation model lifts pets to 4D from monocular clips"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000165,"raw_usage":{"total_tokens":1207,"prompt_tokens":860,"completion_tokens":347,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":476,"completion_tokens_details":{"reasoning_tokens":255}},"tokens_in":476,"tokens_out":347,"duration_ms":4051,"temperature":1.0,"reasoning_tokens":255,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:46:51.507044+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the pipeline on a deformable category with no pre-trained CSE model (e.g., a horse or a cloth toy captured in several videos) and check whether the recovered canonical shape, per-frame meshes, and camera trajectories stay consistent across sequences; also ablate by removing the CSE losses (Eq. 17) and measuring how much the Chamfer distance and novel-view PSNR degrade on the existing cat and dog datasets.","supporting_citations":[{"cited_title":"In: CVPR (2022) 4DPV 17","cited_arxiv_id":null,"evidence_quote":"BANMo is the main baseline and provides the LBS + canonical space + CSE framework that this paper extends with a fine deformation model."},{"cited_title":"In: ECCV (2020)","cited_arxiv_id":null,"evidence_quote":"NeRF supplies the volume rendering formulation and the implicit representation of appearance and density used throughout."},{"cited_title":"NeurIPS (2020)","cited_arxiv_id":null,"evidence_quote":"Continuous Surface Embeddings (CSE) from DensePose are used to establish cross-sequence correspondences via the matching losses in Eq. (17)."},{"cited_title":"In: BMVC (2009)","cited_arxiv_id":null,"evidence_quote":"The quadratic deformation model is inspired by this prior work on non-rigid structure from motion, providing the extended-coordinate formulation in Eq. (8)."},{"cited_title":"In: CVPR (2021)","cited_arxiv_id":null,"evidence_quote":"LASR introduces the LBS-based articulated shape reconstruction from monocular video that the coarse component builds upon."},{"cited_title":"In: NeurIPS (2021)","cited_arxiv_id":null,"evidence_quote":"ViSER is a state-of-the-art competitor used for quantitative comparison and to motivate the need for finer deformation modeling."}],"review_version":1}