{"id":"430d9bd2-a1ea-4f3b-9961-28d8913df43f","arxiv_id":"2412.19089","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Dynamic 3D scenes can be reconstructed from unsynchronized, uncalibrated multi-view videos by first aligning estimated human motion across views and then refining the alignment during neural field training.","lead":"This paper shows that unsynchronized videos from unknown camera positions can be turned into a dynamic 3D scene model, as long as people appear in the footage. It uses AI-estimated human poses as a built-in calibration pattern, then refines the result with a neural scene network.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Novel-view metrics in Table 3 are computed after optimizing the test camera pose and timestamp on the test view, so the near-oracle rendering claim overstates what the pipeline alone achieves.","rationale":"I considered two candidate load-bearing concerns: the reliance on SLAHMR, which the reader highlighted, and the test-time optimization in the rendering evaluation. The SLAHMR dependence is real but is acknowledged in the Limitations section, is partially stress-tested in Table 2 with noise, image degradation, and mixed framerates, and it represents an external failure mode rather than a flaw in the proposed pipeline's internal logic. The test-time optimization, by contrast, changes the quantity being measured: without it, the reader cannot know whether the estimated calibration alone is sufficient for near-oracle view synthesis. The calibration numbers in Table 1 are strong and would survive this check, so I would not reject the paper. The appropriate outcome remains the reader's conditional acceptance, with the additional condition that the rendering evaluation be repeated, or also reported, without test-time pose and timestamp optimization. I mark agreement as partial because the reader's weakest_assumption pointed to SLAHMR and person correspondence, while the reader's rationale did list test-time optimization as a caveat. The proposed check is a single computational re-evaluation that directly settles whether the concern lands.","tokens_in":16555,"tokens_out":5964,"duration_ms":60924,"concrete_test":"Rerun the Table 3 evaluation with test-time optimization disabled: render the held-out view using only the pipeline's estimated calibration, or, if the held-out camera was not part of the calibration, transform the held-out ground-truth pose to the estimated coordinate frame using the same Procrustes alignment described in Supplement B.4 and then freeze it. Compare PSNR/SSIM/LPIPS to the current numbers. If average PSNR drops by more than about 1 dB or falls below Sync-NeRF on any scene, the near-oracle rendering claim is unsupported without qualification; if the metrics remain within a small margin, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3 explicitly states that before measuring novel-view synthesis errors, the authors freeze the trained NeRF and optimize only the test camera poses and timestamps on the test view, with the protocol detailed in Supplement B.4. Consequently, the PSNR/SSIM/LPIPS values in Table 3 do not measure reconstruction from the calibration estimated by the proposed pipeline; they measure reconstruction after a per-test-view bundle adjustment that can absorb residual pose and time-offset error. The calibration results in Table 1 are unaffected by this protocol, so the spatio-temporal calibration claim is secure within the tested datasets. However, the abstract's headline claim that unsynchronized videos from unknown poses can generate dynamic neural fields is heavily supported by the near-oracle rendering numbers, and those numbers are not a clean test of that claim. With oracle calibration, test-time optimization cannot compensate for anything because the poses and times are already correct; with the proposed method, it can only help, making the gap to the oracle appear smaller than it truly is. This is the most load-bearing weakness because it directly concerns the evidence for the strongest claim. It is not an internal inconsistency, but it is a measurement-protocol gap that should be closed before the rendering claim is taken at face value.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper addresses dynamic 3D neural field reconstruction from multi-view videos that are unsynchronized and have unknown camera poses, using human motion as a calibration pattern. The pipeline extracts per-video human motion (SMPL parameters) with SLAHMR; estimates initial time offsets via pairwise DTW on canonical 3D joint positions followed by a greedy global alignment; estimates initial camera poses by Procrustes alignment of global 3D joints; and then jointly refines time offsets and camera poses with a K-Planes dynamic NeRF, using a progressive curriculum that unfreezes camera poses at s0 steps and time offsets at s0+s1 steps, plus a coarse-to-fine feature weighting schedule. Experiments on CMU Panoptic Studio, Mobile-Stage, and EgoBody report rotation errors around 0.4 degrees after refinement and novel-view synthesis metrics close to an oracle K-Planes baseline with ground-truth poses and timestamps.","tokens_in":16792,"tokens_out":6288,"duration_ms":62882,"significance":"The work is significant because it removes two common hardware assumptions---synchronized capture and calibrated camera poses---for dynamic scene reconstruction. The mid-level representation of humans as a calibration pattern is well motivated, and the initialization appears robust to the tested image degradation, SMPL noise, and mixed frame rates. The paper provides detailed pseudocode for the alignment steps and ablates the progressive training strategy, which strengthens reproducibility. If confirmed with a cleaner evaluation protocol, the method could enable casual multi-view capture of dynamic scenes. The main reservations are the test-time optimization used for the novel-view metrics and the lack of repeated-run statistics, both of which are addressable through additional experiments and reporting.","major_comments":[{"comment":"The novel-view metrics in Table 3 are computed after freezing the trained NeRF and optimizing the test camera pose and timestamp on the test view's own frames. This protocol can absorb residual pose and time-offset errors, so the reported PSNR/SSIM/LPIPS values do not directly measure the reconstruction obtained from the calibration estimated by the proposed pipeline. The paper should report metrics without test-time optimization. If the protocol is retained, it must be applied identically to the oracle and Sync-NeRF baselines, and an ablation should quantify its effect. As written, the \"on par with oracle\" statement overstates the rendering evidence for the headline claim.","section":"Section 4.3 and Supplement B.4"},{"comment":"All quantitative results appear to be from a single run. The optimization involves stochastic sampling, and the initialization includes a random anchor selection in Eq. (7). Without repeated runs, the claims of 0.4 degree rotation error, 0.03-frame offset, and near-oracle rendering lack statistical support. The authors should report mean and standard deviation over at least 3-5 runs, or justify why the quantities are deterministic.","section":"Tables 1-3"},{"comment":"The method assumes known person correspondence across views when multiple humans are present, but the experiments do not exercise difficult correspondence cases. The Panoptic Studio scenes contain a single human, the Mobile-Stage dancers are visually distinct, and the EgoBody evaluation is limited to the initialization stage. To support the claim that the method handles an arbitrary number of humans, either present a multi-person experiment with ambiguous correspondences or qualify the claim in the paper.","section":"Section 3.1 and experiments"},{"comment":"The global alignment of pairwise time offsets is a greedy heuristic, but the paper does not analyze its consistency or failure modes. Since the time-offset initialization is a load-bearing step for the whole pipeline, the authors should provide a synthetic experiment or a theoretical argument showing that the greedy procedure recovers the correct global offsets even under noisy pairwise DTW costs, and discuss possible cycle-consistency issues.","section":"Section 3.2 and Supplement Algorithm 1"}],"minor_comments":[{"comment":"The title in the paper header reads \"Humans as a Calibration:\" while the abstract uses \"Humans as a Calibration Pattern\"; please ensure the title is consistent throughout the manuscript.","section":"Abstract and title"},{"comment":"The time offset Delta-T_ij is described as the \"most frequent warping time\" from DTW, but the paper does not specify how this mode is computed when the 3D joint positions are interpolated for mixed frame rates; a short implementation detail would clarify the procedure.","section":"Section 3.2"},{"comment":"The SVD-based rotation in the Procrustes analysis should handle the reflection case where det(U V*) = -1, so that the estimated transform is a similarity transform rather than a reflection; this standard detail is worth stating explicitly.","section":"Supplement Algorithm 2"},{"comment":"The \"Data\" column header is not defined in the caption; it appears to report the average ground-truth absolute time offset of the input videos, but this should be stated in the caption.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is well within the scope of the journal and the core idea is appealing. The main revision needed is the evaluation protocol: the novel-view metrics should be reported without test-time optimization of the test view, and repeated-run statistics should be provided for the headline numbers. I also recommend asking for a multi-person experiment that exercises the correspondence assumption. The authors' related-work coverage is adequate, and the limitation statement is honest about SLAHMR failures."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi,\n\nQuick take on arXiv:2412.19089. This paper does something genuinely new: it reconstructs a dynamic NeRF from multi-view videos that are both unsynchronized and have unknown camera poses, using human motion as the calibration cue. The method is sensible: SMPL parameters from SLAHMR give per-video human trajectories; DTW on canonical joint positions gives initial time offsets; Procrustes on global joints gives camera poses; then joint refinement with K-Planes using a coarse-to-fine feature weighting and curriculum schedule. The calibration results are strong — 0.4° rotation and 0.03-frame time offset after refinement on Panoptic Studio, with decent robustness to SMPL noise, video degradation, and mixed frame rates. The ablation without progressive training shows that the schedule matters a lot. That part I buy.\n\nThe main soft spot is the rendering evaluation. Section 4.3 and Supplement B.4 disclose that, before computing novel-view metrics, the authors freeze the trained NeRF and optimize the test camera pose and timestamp on the test view. So Table 3 measures reconstruction after per-test-view bundle adjustment, not what the pipeline alone estimates. Calibration Table 1 is unaffected, so the spatio-temporal alignment claim is secure within these datasets. But the abstract's headline claim — that unsynchronized, unknown-pose videos can generate dynamic neural fields — leans heavily on those near-oracle rendering numbers, and those numbers are not a clean test of the pipeline. The fix is straightforward: report the same metrics with and without test-time optimization, or restrict test-time optimization to the oracle baseline only. This should be addressed before the rendering claim is taken at face value.\n\nMinor issues: no error bars over runs; no code; the method inherits SLAHMR's failure modes (acknowledged in Limitations); and known person correspondence when multiple people are present is assumed but never stress-tested in the full pipeline. EgoBody, which has two people and a moving camera, is used only to evaluate initialization, not the joint refinement. Moving-camera refinement is explicitly future work.\n\nWho's this for: anyone working on dynamic scene reconstruction, camera calibration from video, or practical multi-view capture. It deserves a serious referee — the central idea is new and the calibration evidence is solid. Ask for the cleaner evaluation protocol, error bars, and ideally code, but don't desk-reject it.","headline":"Genuinely new dynamic NeRF from unsynchronized, uncalibrated video via human motion; rendering metrics are partly inflated by test-time optimization.","tokens_in":17356,"tokens_out":3886,"would_cite":false,"duration_ms":33191,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that unsynchronized multi-view videos with unknown camera poses can be reconstructed into dynamic 3D scenes whenever they capture human motion, using the moving person as a calibration pattern.","keywords":["dynamic neural radiance fields","unsynchronized videos","camera calibration","human motion estimation","time offset estimation","4D scene reconstruction","progressive learning","multi-view video"],"falsifier":"Take two cameras pointed at the same moving person in a textureless room, but arrange one camera so the person is heavily occluded or viewed from an extreme angle for most of the sequence, causing the human-motion estimator to produce flipped or jittered poses. If the central claim holds, the pipeline should still converge to sub-frame time offsets and sub-degree poses; if the recovered offsets and poses diverge by more than about 2 frames and 5 degrees after the initialization stage, the dependence of the calibration on the human-motion priors is demonstrated.","tokens_in":16368,"feed_emoji":"🎥","tokens_out":6541,"duration_ms":126916,"temperature":0.7,"pith_summary":"This paper claims that unsynchronized multi-view videos with completely unknown camera poses can still be turned into a dynamic 3D scene whenever the videos contain people. The key move is to treat human motion as a live calibration target: per-video estimates of body shape and pose are noisy, but they provide a strong enough starting point to align the videos in time and space. From that initialization, the paper jointly optimizes camera poses, time offsets, and a 4D neural radiance field, reaching sub-millisecond alignment and sub-centimeter pose accuracy. If true, it removes two of the most restrictive hardware requirements for 4D reconstruction and opens the door to casual multi-camera capture of events like sports and concerts.","feed_headline":"A moving person is enough to calibrate multi-view 4D video","feed_subtitle":"Off-the-shelf human pose estimates align time and camera poses, reaching sub-frame and sub-degree accuracy without a sync rig.","key_machinery":"The carrying object is the estimated human body sequence, represented as SMPL-H parameters per frame (root orientation, 22 joint rotations, root position, and shape). Its two derived signals do the calibration work: canonical root-relative joint positions serve as the distance function for dynamic time warping, giving pairwise time offsets, and global joint positions serve as corresponding points for Procrustes alignment, giving camera poses in a common world frame. The refinement stage wraps this in K-Planes, a multiresolution 4D feature-grid representation, with a coarse-to-fine weight schedule on grid resolutions and a curriculum that unfreezes camera poses before time offsets.","core_discovery":"The paper's central claim is that a moving person's estimated skeleton, obtained independently from each camera, is a sufficient calibration pattern for dynamic neural scene reconstruction. It establishes this by first aligning human joint trajectories across pairs of videos with dynamic time warping, assembling pairwise offsets into a global synchronization, and then registering the cameras to a common world frame with Procrustes alignment on 3D joint positions. Starting from those crude but consistent estimates, it refines everything together with a K-Planes dynamic radiance field, using a progressive schedule that stabilizes the non-convex optimization. The result is near-perfect calibration, with about 0.4 degrees of rotation error, 0.2 cm translation error, and 0.03 frames of time offset on Panoptic Studio, and novel-view synthesis on par with an oracle that is handed ground-truth poses and timing.","pith_inferences":["Because the calibration cue is the person rather than scene texture, the same recipe should transfer to any object with a parametric motion model, such as articulated animals, vehicles, or rigid objects in known shape families, whenever a monocular estimator can supply per-frame pose.","Refining time offsets to 0.03 frames effectively means post-hoc synchronization can beat the precision of many hardware sync solutions, which suggests the approach could also resynchronize archival footage where no sync signal survives.","The curriculum order, field first, poses second, time offsets last, is a reusable trick: the same staged unfreezing may stabilize other under-constrained joint estimation problems, such as simultaneous mapping and tracking in scenes with moving objects."],"forward_implications":["Videos shot by ordinary phones at a gathering or sports event, with no sync hardware and no surveyed camera positions, become usable input for photorealistic 4D reconstruction as long as a person is visible.","The initialization alone, before any neural-field refinement, already recovers camera rotation within about 5.5 degrees and time offsets within about 1.3 frames on average, so it can serve as a standalone calibration fallback when structure-from-motion fails on textureless or repetitive scenes.","Joint refinement brings calibration to sub-frame and sub-centimeter precision, reaching accuracy comparable to an oracle calibrated with ground-truth poses and timestamps.","The pipeline is not limited to one person or to static rigs: it is demonstrated with one, two, and three humans, with mixed frame rates, with moving cameras, and with different camera intrinsics."],"supporting_citations":[{"why":"SLAHMR supplies the per-video human shape, pose, and camera trajectory that form the calibration cue for the entire pipeline.","marker":"[48]"},{"why":"K-Planes provides the 4D multiresolution radiance field that is jointly optimized with the calibration parameters.","marker":"[8]"},{"why":"Sync-NeRF is the prior method that handles unsynchronized videos but requires known poses and near-correct initial offsets, making it the comparison baseline.","marker":"[16]"},{"why":"BARF contributes the coarse-to-fine registration principle that the paper adapts to weight K-Planes grid resolutions.","marker":"[21]"},{"why":"SCNeRF provides the curriculum-learning idea of progressively adding camera parameters during neural radiance field optimization.","marker":"[11]"},{"why":"SMPL-H defines the body model used to convert pose and shape parameters into the 3D joint positions used by dynamic time warping and Procrustes alignment.","marker":"[37]"},{"why":"The dynamic time warping algorithm supplies the pairwise sequence alignment used to estimate time offsets between videos.","marker":"[38]"},{"why":"Panoptic Studio supplies multi-view human motion sequences with ground-truth calibration used to evaluate the method's accuracy.","marker":"[13]"}],"fun_headline_variants":["Human skeletons calibrate unsynchronized video arrays","One moving person syncs and calibrates multi-view video","Skeleton-based calibration for dynamic 3D from loose videos","Unsynchronized videos? Human pose aligns time and cameras","Dynamic 3D from uncalibrated video using human calibration pattern"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole pipeline depends on the per-video human motion estimates being roughly correct: if the body pose and shape recovered from a single video are badly wrong, the time offsets and camera poses derived from them are wrong, and the later neural-field refinement cannot repair the calibration.","fun_headline_variants_meta":{"raw":{"variants":["Human skeletons calibrate unsynchronized video arrays","One moving person syncs and calibrates multi-view video","Skeleton-based calibration for dynamic 3D from loose videos","Unsynchronized videos? Human pose aligns time and cameras","Dynamic 3D from uncalibrated video using human calibration pattern"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000196,"raw_usage":{"total_tokens":1355,"prompt_tokens":934,"completion_tokens":421,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":338}},"tokens_in":550,"tokens_out":421,"duration_ms":5347,"temperature":1.0,"reasoning_tokens":338,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:57:15.075613+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take two cameras pointed at the same moving person in a textureless room, but arrange one camera so the person is heavily occluded or viewed from an extreme angle for most of the sequence, causing the human-motion estimator to produce flipped or jittered poses. If the central claim holds, the pipeline should still converge to sub-frame time offsets and sub-degree poses; if the recovered offsets and poses diverge by more than about 2 frames and 5 degrees after the initialization stage, the dependence of the calibration on the human-motion priors is demonstrated.","supporting_citations":[{"cited_title":"Decoupling human and camera motion from videos in the wild","cited_arxiv_id":null,"evidence_quote":"SLAHMR supplies the per-video human shape, pose, and camera trajectory that form the calibration cue for the entire pipeline."},{"cited_title":"K-planes: Explicit radiance fields in space, time, and appearance","cited_arxiv_id":null,"evidence_quote":"K-Planes provides the 4D multiresolution radiance field that is jointly optimized with the calibration parameters."},{"cited_title":"Sync-nerf: Generalizing dy- namic nerfs to unsynchronized videos","cited_arxiv_id":null,"evidence_quote":"Sync-NeRF is the prior method that handles unsynchronized videos but requires known poses and near-correct initial offsets, making it the comparison baseline."},{"cited_title":"Barf: Bundle-adjusting neural radiance fields","cited_arxiv_id":null,"evidence_quote":"BARF contributes the coarse-to-fine registration principle that the paper adapts to weight K-Planes grid resolutions."},{"cited_title":"Self-calibrating neural radiance fields","cited_arxiv_id":null,"evidence_quote":"SCNeRF provides the curriculum-learning idea of progressively adding camera parameters during neural radiance field optimization."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SMPL-H defines the body model used to convert pose and shape parameters into the 3D joint positions used by dynamic time warping and Procrustes alignment."},{"cited_title":"Dynamic programming al- gorithm optimization for spoken word recognition","cited_arxiv_id":null,"evidence_quote":"The dynamic time warping algorithm supplies the pairwise sequence alignment used to estimate time offsets between videos."},{"cited_title":"Panoptic studio: A massively multiview sys- tem for social interaction capture.IEEE Transactions on Pat- tern Analysis and Machine Intelligence, 2017","cited_arxiv_id":null,"evidence_quote":"Panoptic Studio supplies multi-view human motion sequences with ground-truth calibration used to evaluate the method's accuracy."}],"review_version":1}