{"id":"4d7795cd-007f-40da-b463-e6c841184434","arxiv_id":"2501.01715","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Cloth-Splatting couples a graph-network cloth dynamics prior with mesh-constrained 3D Gaussian Splatting to refine 3D cloth state estimates from RGB images, improving accuracy and convergence speed over existing trackers.","lead":"Cloth-Splatting estimates the 3D shape and motion of a cloth from ordinary RGB camera images by combining a learned physics prediction with a visual refinement step using 3D Gaussian Splatting. It is faster and more accurate than current differentiable-rendering trackers, uses only three to four cameras, and can support closed-loop robotic folding.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The accuracy claim rests on an unfair baseline: MD-Splatting is evaluated with 4 views despite the paper stating it needs at least 50, and the reported 9.66% gain contradicts the abstract's '57%'.","rationale":"I identify the baseline fairness as the most load-bearing concern because it attacks the empirical foundation of the central claim rather than an acknowledged limitation. The paper's own text creates the expectation that MD-Splatting requires 50 cameras, yet the experiments use 4; this is a controllable confound. The abstract's '57%' discrepancy makes the result look fragile. Running MD-Splatting under its native conditions would settle whether Cloth-Splatting actually improves on the state of the art. The reader's static-appearance concern is valid but secondary: it is explicitly listed as a limitation and does not invalidate the synthetic evaluation. I therefore partially agree with the reader: we both flag the unfair baseline, but I would rank it as the primary issue rather than the static-appearance assumption. The verdict remains conditional because the paper could still be correct if the re-run confirms the advantage, but the current evidence is insufficient.","tokens_in":17354,"tokens_out":6999,"duration_ms":68819,"concrete_test":"Re-run the Table 1 comparison on the same 75 synthetic scenes with MD-Splatting configured with 50 cameras (or its original multi-view setting) and no other changes. Compute MTE, delta_avg, and convergence time. If MD-Splatting's MTE falls below 3.284 mm or its convergence time changes substantially, the paper's accuracy and speed claims are artifacts of the sparse-view handicap. As a secondary check, recompute the percent improvement over the best baseline from Table 1 and reconcile it with the abstract's '57%' figure.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim depends on Table 1, where Cloth-Splatting (3.284 mm MTE) is compared against MD-Splatting (3.635 mm MTE) using the same 4 camera views. However, Section 2 states that MD-Splatting and DeformGS 'require dense observations (at least 50 cameras).' Evaluating a method outside its design regime handicaps it: with only 4 views, its deformation field is underconstrained, so its inflated MTE may not reflect a real deficiency relative to Cloth-Splatting. Section 5.1 also notes that depth is given to the baselines but not to Cloth-Splatting, further stacking the comparison. The abstract's '57% more accurate' does not match the 9.66% improvement computed from Table 1, indicating that the headline numbers are internally inconsistent. Because the superiority claim is the paper's primary contribution, this comparison is load-bearing; if MD-Splatting at its native camera count matches or beats Cloth-Splatting, the central claim fails. The static-appearance limitation is real but explicitly acknowledged and does not affect the synthetic quantitative comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Cloth-Splatting, a prediction-update framework for 3D cloth state estimation from RGB images. The method couples a GNN-based action-conditioned dynamics model (trained on simulated cloth data) with a mesh-constrained 3D Gaussian Splatting measurement model. At test time, a per-scene residual MLP is optimized by minimizing photometric and regularization losses between rendered and observed RGB images, thereby refining mesh-vertex positions. Experiments on 75 synthetic scenes and a real-world folding setup compare Cloth-Splatting against several baselines, reporting improved median trajectory error (MTE) and faster convergence, and the paper demonstrates a closed-loop manipulation use case.","tokens_in":17627,"tokens_out":7089,"duration_ms":66758,"significance":"If the quantitative claims hold, Cloth-Splatting is a useful step toward RGB-only 3D cloth tracking with sparse camera views. The mesh-constrained Gaussian Splatting formulation is a clean and potentially reusable contribution, and the combination of a learned dynamics prior with differentiable rendering is a sensible design. The paper promises code and videos, which would aid reproducibility. However, the support for the headline accuracy claim is weakened by an internal inconsistency in the reported improvement and by concerns about the fairness of the baseline evaluation protocol.","major_comments":[{"comment":"The abstract and Section 1 state that Cloth-Splatting is \"57 % more accurate\" than the best-performing baseline, but Table 1 reports mean MTE of 3.284 mm for Cloth-Splatting versus 3.635 mm for MD-Splatting, which is a 9.66% improvement. Moreover, on the TSHIRT category Cloth-Splatting is worse than MD-Splatting (3.159 vs. 2.109 mm). The headline number is therefore not supported by the tabulated results, and the paper must either reconcile the 57% figure with Table 1 or remove it.","section":"Abstract and Section 1 vs. Table 1"},{"comment":"Section 2 states that MD-Splatting and DeformGS \"require dense observations (at least 50 cameras)\", yet Table 1 evaluates MD-Splatting with only 4 camera views. This places the main baseline outside its design regime and makes the reported accuracy advantage of Cloth-Splatting difficult to interpret. Please either re-run the comparison with the baselines' required camera count, or explicitly frame the comparison as sparse-view tracking and analyze the effect of view count on all methods. In addition, Section 5.1 says that depth is used only for the baseline methods; please clarify which baselines receive depth and whether the comparison gives Cloth-Splatting an information disadvantage.","section":"Section 2 and Section 5.1 / Table 1"},{"comment":"The central accuracy claim rests on a small average MTE difference (3.284 vs. 3.635 mm) with large reported standard deviations (3.722 vs. 6.235 mm). The paper does not report per-scene paired comparisons or significance tests. Please provide such an analysis, or temper the \"outperforms all baselines\" claim accordingly.","section":"Table 1"}],"minor_comments":[{"comment":"The caption contains the unexplained expression \"t = 16.42 min (627.83%)\" and the axis mixes minutes and seconds; please clarify what is being plotted and correct the units.","section":"Figure 4 caption"},{"comment":"The header \"Survial rate\" is a typo and should read \"Survival rate\".","section":"Table 1"},{"comment":"The GNN input is described as the last m states of the mesh, but m is never defined; the appendix mentions the past k = 3 velocities, yet the relationship between m and k is unclear.","section":"Section 4.1 and Appendix A.1"},{"comment":"Line 6, \"δ ˆMt+1 ∝ ∇Lobs\", is not a valid update rule; please replace it with an explicit gradient descent step such as δ ← δ − λ∇δ Lobs.","section":"Algorithm 1"},{"comment":"The covariance decomposition should be written as Σ = R S S^T R^T (or with explicit matrix products) to avoid ambiguity in the current notation \"RSSTRT\".","section":"Equation (4)"},{"comment":"The oracle baseline is referred to as both OL-ORACLE and MPC-ORACLE; please use a single consistent name.","section":"Section 5.4 and Appendix F"},{"comment":"The manipulation results appear twice as Table 3 in the main text and Table 6 in the appendix; please assign unique numbers and cite each consistently.","section":"Table 3 and Table 6"}],"recommendation":"major_revision","confidential_remarks":"The 9.66% improvement over MD-Splatting is modest, and the comparison is complicated by the paper's own statement that MD-Splatting requires at least 50 cameras while the evaluation uses only 4. I would ask the authors to provide corrected or additional comparisons, or to significantly temper the accuracy claims, before the paper can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's my read. The core idea is genuinely useful: coupling an action-conditioned GNN dynamics model with a mesh-constrained 3D Gaussian Splatting measurement model, plus a per-scene residual MLP, yields a practical RGB-only cloth tracker that works with sparse views. The ablations show each component matters, and the rollout update is clever and makes convergence roughly 85% faster per their time plot. That speed advantage is the paper's strongest and least contested claim.\n\nThe central accuracy claim, however, has problems. The abstract says 57% more accurate than the best baseline, yet Table 1 shows a 9.66% MTE improvement (3.284 vs 3.635 mm). Those numbers are inconsistent unless the 57% refers to something else, and the text does not say what. That is a serious reporting gap. Second, the main comparison runs MD-Splatting with 4 views, even though the paper itself states MD-Splatting needs at least 50 cameras. That puts the baseline in a regime it wasn't designed for, inflating the apparent gap. The authors could have framed the comparison as targeting the sparse-view regime, but they didn't. One detail: the stress-test note says giving depth to baselines stacks the comparison against Cloth-Splatting. Actually it stacks it in Cloth-Splatting's favor, so that point doesn't stick.\n\nThe static appearance assumption is acknowledged and real; changing lighting could push the tracker toward visual artifacts. The real-world evaluation is qualitative only, which is a pity given the sim-to-real claim.\n\nOverall, the method is well-engineered and the direction is promising. The paper deserves a serious referee, but I'd want the headline number corrected, the MD-Splatting comparison rerun or clearly qualified, and ideally quantitative real-world results. As it stands, this is a conditional accept rather than a clean one.","headline":"Promising RGB-only cloth tracker, but the headline accuracy claim doesn't match its own table and the main baseline is evaluated outside its design regime.","tokens_in":18202,"tokens_out":2582,"would_cite":true,"duration_ms":25336,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Cloth-Splatting shows that the 3D state of a cloth can be tracked accurately from a few RGB views by combining a learned dynamics model with a mesh-constrained 3D Gaussian Splatting measurement model, cutting convergence time by about 85…","keywords":["3D state estimation","Gaussian Splatting","cloth tracking","deformable object manipulation","RGB supervision","graph neural network dynamics","closed-loop manipulation"],"falsifier":"Run Cloth-Splatting on a cloth under a moving light source or cast shadow while recording ground-truth mesh state; if median trajectory error grows substantially or the refined mesh tracks the shadow edge rather than the cloth, the static-appearance assumption is violated. A simulator version with time-varying texture or lighting would settle it directly.","tokens_in":17166,"feed_emoji":"🧵","tokens_out":8382,"duration_ms":73739,"temperature":0.7,"pith_summary":"The paper aims to establish that the 3D state of a cloth can be estimated from RGB images alone, without depth sensors, by unrolling a learned cloth dynamics model and refining its predictions with a differentiable renderer. It casts the problem as Bayesian filtering: an action-conditioned graph neural network proposes the next mesh state, and a mesh-constrained 3D Gaussian Splatting model renders that state into the observed camera views, so gradient descent on the photometric error can correct the proposal. The reported experiments on 75 synthetic scenes show a median trajectory error of 3.284 mm versus 3.635 mm for the best baseline, about 85 percent faster convergence, and real-robot demonstrations of closed-loop cloth folding. The reason to care is that accurate RGB-only state estimation is a bottleneck for robotic manipulation of deformable objects, where depth is often sparse, noisy, or unavailable.","feed_headline":"Tracks cloth in 3D from RGB alone, 85% faster","feed_subtitle":"Pairing learned dynamics with mesh-constrained splatting beats depth-based baselines and enables closed-loop folding.","key_machinery":"The load-bearing object is the mesh-constrained 3D Gaussian Splatting measurement model. Each Gaussian lives on a face of the cloth mesh: its mean is expressed in barycentric coordinates of the face's vertices and its rotation is expressed relative to the face, so when the mesh deforms the Gaussians follow the cloth while their colors and barycentric coordinates stay fixed. This makes the image render a differentiable function of the cloth state, providing the measurement likelihood for the update step. A small MLP outputs a per-timestep residual correction, and the photometric loss is combined with SSIM, isometric, and motion-magnitude regularizers to stabilize the refinement.","core_discovery":"The central claim is that the cloth mesh can act as the bridge between a learned dynamics prior and image observations. Each Gaussian is anchored to a mesh face with fixed barycentric coordinates and a face-relative rotation, so the rendered image is a differentiable function of the vertex positions; optimizing the photometric loss between the render and the observed RGB image updates the vertex positions through a learned residual. This gives a measurement model for a prediction-update loop that refines GNN predictions without retraining the dynamics network. In the paper's evaluation the refined estimates beat the prior mesh predictions and the dynamic-Gaussian baselines, and the refinement is what makes closed-loop model-predictive folding approach the accuracy of an oracle.","pith_inferences":["The static-appearance assumption suggests a concrete extension: periodically re-estimating Gaussian colors, or adding an explicit lighting model, would let the tracker survive moving shadows without abandoning the mesh constraint.","Because the residual state update is a per-scene MLP optimized from scratch, an amortized network that predicts residuals directly from images could trade a little accuracy for much faster deployment-time adaptation.","The monocular ablation shows a larger accuracy drop than going from four views to three, which points to view diversity rather than sheer camera count as the main driver; a carefully placed stereo pair may capture most of the benefit.","The paper's own note about occlusion-free initialization could be combined with template-based reconstruction of crumpled cloth to remove the initial-mesh restriction and broaden the method to heavily wrinkled cloths."],"forward_implications":["With only three or four calibrated RGB cameras, and no depth at run time, cloth state can be refined well enough to outperform dense-view dynamic Gaussian baselines in the paper's experiments.","Refining GNN rollouts with the photometric loss corrects error accumulation, so the tracker can run in a fast rollout mode rather than requiring iterative re-prediction at every step.","The same prediction-update architecture transfers to closed-loop manipulation: model-predictive control with Cloth-Splatting state estimates folds cloths to final states close to those achieved by an oracle with ground-truth states.","Faster convergence makes per-scene online adaptation practical, addressing a limitation of prior per-scene optimization approaches that need many camera views and long optimization times."],"supporting_citations":[{"why":"supplies the 3D Gaussian Splatting primitive that the authors adapt into a mesh-constrained measurement model.","marker":"[14]"},{"why":"provides the Graph Network Simulator architecture used for the action-conditioned cloth dynamics prediction.","marker":"[9]"},{"why":"is the strongest baseline, MD-Splatting, whose accuracy and convergence time Cloth-Splatting is compared against.","marker":"[46]"},{"why":"establishes the earlier test-time-optimization approach for cloth state refinement that this work extends from depth to RGB supervision.","marker":"[13]"},{"why":"justifies conditioning the dynamics model on actions by treating the grasped vertex as rigidly attached to the gripper.","marker":"[52]"},{"why":"defines the SSIM loss used in the regularization term that stabilizes the state update.","marker":"[53]"},{"why":"provides the as-rigid-as-possible principle behind the isometric regularization loss.","marker":"[54]"},{"why":"defines the half-folding benchmark task used to evaluate closed-loop manipulation.","marker":"[63]"}],"fun_headline_variants":["RGB-only cloth state via Gaussian splatting refinement","Cloth-Splatting: refine 3D cloth pose with just RGB","Mesh-splatting bridge enables RGB-only cloth tracking","Differentiable splats refine cloth state without depth","Cloth state estimation from RGB using splatting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The tracking assumes the cloth's visual appearance is static: Gaussian colors and the Gaussians' attachment to mesh faces are fixed during refinement, so if lighting changes or shadows move, the photometric loss can drag the mesh toward visual artifacts instead of the cloth's true motion.","fun_headline_variants_meta":{"raw":{"variants":["RGB-only cloth state via Gaussian splatting refinement","Cloth-Splatting: refine 3D cloth pose with just RGB","Mesh-splatting bridge enables RGB-only cloth tracking","Differentiable splats refine cloth state without depth","Cloth state estimation from RGB using splatting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000169,"raw_usage":{"total_tokens":1189,"prompt_tokens":797,"completion_tokens":392,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":413,"completion_tokens_details":{"reasoning_tokens":312}},"tokens_in":413,"tokens_out":392,"duration_ms":4458,"temperature":1.0,"reasoning_tokens":312,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:21:44.851323+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Cloth-Splatting on a cloth under a moving light source or cast shadow while recording ground-truth mesh state; if median trajectory error grows substantially or the refined mesh tracks the shadow edge rather than the cloth, the static-appearance assumption is violated. A simulator version with time-varying texture or lighting would settle it directly.","supporting_citations":[{"cited_title":"Kerbl, G","cited_arxiv_id":null,"evidence_quote":"supplies the 3D Gaussian Splatting primitive that the authors adapt into a mesh-constrained measurement model."},{"cited_title":"Sanchez-Gonzalez, J","cited_arxiv_id":null,"evidence_quote":"provides the Graph Network Simulator architecture used for the action-conditioned cloth dynamics prediction."},{"cited_title":"Huang, X","cited_arxiv_id":null,"evidence_quote":"establishes the earlier test-time-optimization approach for cloth state refinement that this work extends from depth to RGB supervision."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"justifies conditioning the dynamics model on actions by treating the grasped vertex as rigidly attached to the gripper."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"defines the SSIM loss used in the regularization term that stabilizes the state update."},{"cited_title":"Sorkine and M","cited_arxiv_id":null,"evidence_quote":"provides the as-rigid-as-possible principle behind the isometric regularization loss."},{"cited_title":"Garcia-Camacho, M","cited_arxiv_id":null,"evidence_quote":"defines the half-folding benchmark task used to evaluate closed-loop manipulation."}],"review_version":1}