{"id":"64823872-4f3a-4a55-9e3c-616671b35e04","arxiv_id":"2507.11129","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A single 3D Gaussian scene can encode RGB, thermal, and language modalities more accurately and compactly by using per-modality opacities and gradient-difference-based Gaussian decomposition.","lead":"MMOne proposes a 3D Gaussian Splatting framework that represents several modalities (RGB, thermal, language) in a single scene by giving each modality its own opacity and splitting Gaussians when their modality gradients diverge. The method reports better per-modality rendering and up to one-third fewer Gaussians than prior multimodal GS methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 3's gradient-difference conflict metric is scale-dependent and underspecified; if it is not the active ingredient, the decomposition claim is unsupported.","rationale":"The reader's weakest_assumption correctly flags the gradient-difference heuristic in Sec. 4.3 as ad hoc. I agree that the fixed threshold is not derived from a model of modality conflict, and I sharpen this into a more precise technical objection: Eq. 3 is underspecified about which parameter gradients are being compared, and the L2 norm of a gradient difference is not scale-invariant, making the threshold dataset- and optimization-dependent. This is a correctness risk for the central mechanism, not merely a tuning concern. However, I do not think this warrants changing the reader's CONDITIONAL verdict: the paper provides code, threshold ablations, and component ablations that make the empirical claim plausible, and the proposed test, not the current evidence, would settle whether the mechanism is truly load-bearing. The reader's verdict already conditions acceptance on addressing exactly this kind of under-specification, so I leave it UNCHANGED. I mark agreement as partial because the reader emphasized threshold validity while I additionally emphasize the missing isolation of the decomposition signal from the other components (modality indicator, soft prune) in the ablations.","tokens_in":16475,"tokens_out":5809,"duration_ms":83023,"concrete_test":"In the released code, replace Eq. 3 with a random decomposition rule that splits the same number of Gaussians at the same densification iterations, while keeping soft prune, modality indicators, and all other hyperparameters fixed. Re-run the RGB-Thermal-Language experiments (Table 3/5). If RGB/thermal PSNR, SSIM, and language mIoU stay within the small spread of the Table 6 threshold ablation, the gradient-difference criterion is not load-bearing and the decomposition mechanism needs re-support; if random splitting degrades performance substantially, the concern is resolved in favor of the paper.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that MMOne 'consistently enhances' each modality and is 'scalable to additional modalities'—rests on the multimodal decomposition mechanism (Sec. 4.3) deciding when a multi-modal Gaussian should be split. The decision rule, Eq. 3, is gd = norm(g_mi - g_mj), compared to a fixed threshold 0.0002. The paper does not state the parameter space of these gradients: if they are gradients w.r.t. shared geometry/opacity, the L2 norm scales with loss weights (0.5/0.5/0.2), learning rate, pixel footprint, and scene scale, so a fixed threshold is not calibrated across datasets; if they are gradients w.r.t. modality-specific features, the vectors live in different dimensions and the subtraction in Eq. 3 is not well-defined. The threshold ablation in Table 6 only varies 0.0001-0.0004 on one 3-modality setting and does not test whether the criterion tracks true granularity conflict rather than optimization dynamics. Moreover, Table 5 always combines decomposition with soft prune and the modality indicator, so no ablation isolates whether the specific Eq. 3 signal, as opposed to added per-modality capacity and aggressive pruning, drives the gains. If the decomposition decision is effectively interchangeable with a random split, the paper's central mechanism is not validated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"MMOne proposes a general 3D Gaussian Splatting framework that represents multiple modalities (RGB, thermal, language, and optionally depth) in a single scene. The paper identifies property disparity and granularity disparity as two modality-conflict challenges. The method contributes a modality modeling module with modality-specific features and a per-modality indicator opacity, and a multimodal decomposition mechanism that splits multi-modal Gaussians into modality-specific Gaussians based on the L2 norm of per-modality gradient differences (Eq. 3). The evaluation covers RGB-thermal (RGBT-Scenes), RGB-language (LERF/LangSplat), and RGB-thermal-language (four RGBT-Scenes scenes with manually annotated masks), plus ablations. The paper claims consistent per-modality improvements and compactness: roughly one-third to one-quarter the Gaussian count of baselines while improving or matching average PSNR, SSIM, and mIoU.","tokens_in":16777,"tokens_out":6006,"duration_ms":66076,"significance":"If the central claim is validated, MMOne is a step toward a unified multimodal scene representation: the compactness results (Table 8 and Table 9) are substantial, the framework is designed to add modalities with modest code changes, the code is released, and the paper includes ablations and a threshold-sensitivity study in the supplementary material. These are real strengths. However, the evidence is currently mixed: the gradient-difference decomposition rule in Eq. (3) is underspecified, the ablations do not isolate the decision rule, per-scene LPIPS worsens on several scenes, and no error bars or multiple-seed results are provided. The 'consistently enhances' claim is therefore stronger than the presented numbers support.","major_comments":[{"comment":"The conflict metric gd = norm(g_mi - g_mj) is underspecified: the paper does not state whether the gradients are taken with respect to the shared Gaussian attributes (mean, covariance, opacity) or the modality-specific features, nor how they are accumulated across pixels or iterations. Gradient magnitudes depend on the modality loss weights (0.5/0.5/0.2 in the RGB-thermal-language experiments), learning rate, image resolution, and scene scale, so a fixed threshold of 0.0002 (Supplementary, Sec. 7) has no obvious calibration across datasets. If the gradients are with respect to modality-specific feature vectors of different dimensions, the subtraction in Eq. (3) is not even defined. Table 6 only sweeps the threshold over 0.0001-0.0004 on one aggregate setting; it does not show that the criterion tracks true granularity conflict rather than optimization dynamics. Because decomposition is the mechanism that grounds the compactness and granularity claims, this specification gap is load-bearing.","section":"§4.3, Eq. (3)"},{"comment":"The ablation does not isolate the decomposition decision rule. The 'Decomp.' row adds the multimodal decomposition mechanism on top of the modality indicator and soft prune, so gains relative to 'Prune (S)' could come from the added per-modality capacity (separate Gaussians for each modality) or from the modified densification, rather than from the Eq. (3) gradient-difference signal. A control that replaces the gradient-difference criterion with a random split of the same fraction of Gaussians, or with splitting all multi-modal Gaussians, is needed; without such a control, the paper's central mechanistic claim is not supported.","section":"§5.4, Table 5"},{"comment":"The abstract's claim of 'consistently enhances the representation capability for each modality' is not supported by per-scene results. Against ThermalGaussian, MMOne has worse RGB LPIPS on Dim (0.203 vs 0.194), RB (0.235 vs 0.199), and Pt (0.291 vs 0.268), and worse thermal LPIPS on RB (0.213 vs 0.198) and LS (0.272 vs 0.248). The average PSNR gain is 0.5 dB for RGB and 0.4 dB for thermal; with no error bars or multiple seeds, it is unclear whether these average differences are significant relative to per-scene variance. Please either soften the consistency claim or add per-scene significance testing and variance estimates.","section":"§5.1, Table 1"},{"comment":"The three-modality evaluation rests entirely on a self-implemented baseline ('MM-J') and manually annotated ground-truth masks for language queries, as the paper acknowledges. Because no independent baseline or established dataset exists for RGB-thermal-language, the mIoU numbers in Table 3 are not directly comparable with those in Table 2, and the manual annotation (medium-level SAM masks) could bias semantic segmentation results. The paper should release the annotations, report annotation consistency, and hedge any comparative conclusion on this setting.","section":"§5.3, Table 3"}],"minor_comments":[{"comment":"The notation for the per-Gaussian feature m_i is overloaded: in Eq. (1) m_i denotes the modality feature vector, whereas in Eq. (2) the modality indicator uses a superscript m (α_i^m), making the distinction between modality index and Gaussian index confusing.","section":"§3.1, Eq. (1) vs Eq. (2)"},{"comment":"The dashed/solid 'distributions' are not labeled with axes or the exact plotted quantity; please clarify the units and the definition of the difference shown (e.g., α_R - α_T).","section":"Figure 3"},{"comment":"The column headed 'Num×10^4' appears without units in the table body; please state explicitly that the numbers are Gaussian counts in units of 10^4 and align this with the scene-wise counts in the supplementary tables.","section":"Table 5"},{"comment":"The scalability experiment with monocular depth is performed on a single scene ('Dimsum') and is presented as 'additional'; the conclusion that adding depth does not compromise other modalities should be supported by more than one scene or explicitly labeled as a pilot result.","section":"Supplementary Sec. 11"},{"comment":"In the LangSurf reference, 'Dingewn Zhang' appears to be a typo; please verify the author names.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The core idea is worth publishing if the decomposition mechanism can be pinned down. The current version overclaims consistency and underspecifies the conflict metric. I would ask the authors to add a control ablation replacing Eq. (3) with a random or all-split rule, to report multiple seeds, and to make the gradient parameter space explicit; these are within the scope of a revision. I do not see an inherent flaw that demands rejection, but the manuscript's central mechanism is currently not isolated from added capacity and pruning effects."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: MMOne is a useful 3DGS extension with a genuinely new mechanism—per-modality opacity/indicator plus a gradient-difference decomposition that soft-prunes modalities rather than entire Gaussians—and it delivers a large compactness win: roughly one-third the Gaussians of ThermalGaussian and about 25% of LangSplat. The accuracy gains are modest (~0.5 dB PSNR on RGB/thermal, a few mIoU points on language), and the “consistently enhances” claim does not survive per-scene inspection: LPIPS worsens on several scenes in Table 1. Still, the paper is worth a serious referee.\n\nWhat is actually good: the modality indicator serving as a switch is a clean idea, the soft-prune vs hard-prune ablation (Table 5) shows a real benefit, and the Gaussian-count tables in the supplement credibly support the efficiency claim. The threshold sensitivity study (Table 6) is a nice touch, and code is released. The writing is clear and the related work is handled fairly.\n\nThe soft spots are real. Eq. 3 defines the decomposition criterion as gd = norm(g_mi - g_mj), but the paper never states what parameter space these gradients live in. If they are gradients w.r.t. shared geometry/opacity, the L2 norm scales with loss weights (0.5/0.5/0.2), learning rate, pixel footprint, and scene scale, so a fixed threshold of 0.0002 is not calibrated across datasets. If they are gradients w.r.t. modality-specific features, the vectors have different dimensions and the subtraction is not well-defined. The threshold ablation only varies 0.0001–0.0004 on one 3-modality scene and does not test whether the criterion tracks true granularity conflict rather than optimization dynamics. More importantly, Table 5 always combines decomposition with soft prune and the modality indicator, so no ablation isolates whether Eq. 3 as such, rather than simply added per-modality capacity and aggressive pruning, drives the gains. I think the decomposition could be doing useful work, but it is not proven to be better than a random split with equivalent capacity. The lack of error bars or multiple seeds also makes the modest average gains hard to trust.\n\nWho this is for: people working on multimodal 3D scene representation, robotics/AR perception, and Gaussian splatting variants. The efficiency result alone makes it worth reading. But the referee should push for an ablation that isolates Eq. 3, a specification of the gradient parameter space, and per-scene reporting or seeds. I would send it to review, not desk-reject it.","headline":"MMOne is a genuinely new 3DGS-based multimodal representation with impressive compactness but a central decomposition mechanism that is underspecified and not cleanly isolated in the ablations.","tokens_in":17266,"tokens_out":1610,"would_cite":true,"duration_ms":21474,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MMOne claims that modality conflicts inside a shared Gaussian scene—property and granularity differences—can be resolved with a per-modality indicator plus a gradient-difference decomposition, yielding better quality for every modality…","keywords":["3D Gaussian Splatting","multimodal scene representation","modality conflict","granularity disparity","modality indicator","multimodal decomposition","RGB-thermal-language rendering","open-vocabulary segmentation"],"falsifier":"Run MMOne with the multimodal decomposition step disabled, keeping the modality indicator and soft prune: if RGB, thermal, and language quality stay within noise and the Gaussian count does not grow, the decomposition is not what carries the reported gains. Directly measuring the distribution of per-Gaussian gradient differences during training would further show whether the fixed 0.0002 threshold actually separates conflicting Gaussians from non-conflicting ones.","tokens_in":16266,"feed_emoji":"🖼️","tokens_out":7624,"duration_ms":80061,"temperature":0.7,"pith_summary":"MMOne tackles a problem that arises when one 3D scene must represent several modalities at once: RGB, thermal, and language differ in physical properties and in the level of detail they need. The paper claims that the usual approach of attaching extra feature vectors to the same set of Gaussians lets these differences fight each other, degrading every modality. MMOne instead gives each modality its own feature vector and a per-modality opacity 'indicator' that can switch a modality off during rendering. It then adds a decomposition step in training that watches the per-modality gradients on each Gaussian and, when two modalities pull in different directions, replaces the shared Gaussian with separate per-modality Gaussians. On RGB-thermal, RGB-language, and RGB-thermal-language benchmarks, the method reports improved quality for every modality while using roughly one-third of the Gaussians of joint-training baselines.","feed_headline":"Three modalities, one scene, one-third the Gaussians","feed_subtitle":"MMOne splits conflicting Gaussians by modality, improving rendering and semantics while shrinking the scene.","key_machinery":"The load-bearing mechanism is the multimodal decomposition step embedded in 3D Gaussian Splatting's densification loop. Each Gaussian carries modality-specific features and a modality indicator; during training, the framework accumulates per-modality gradients and compares them through the L2 norm of their difference. When that norm exceeds a fixed threshold (0.0002), the Gaussian is decomposed into single-modal Gaussians, each driven by its own modality loss. A companion 'Soft Prune' rule turns off a single modality's indicator instead of deleting the whole Gaussian, and raises the pruning threshold for single-modal Gaussians to keep the scene compact. The modality indicator thus does double duty: it weights each modality's contribution in alpha-blending and acts as a switch that freezes updates for deactivated modalities.","core_discovery":"The central discovery is that modality conflicts in a unified Gaussian scene can be resolved by explicitly separating shared structure from modality-specific structure. MMOne models each modality with its own feature vector plus a modality indicator $\\alpha^m \\in [0,1]$ that acts as a per-modality opacity and as a learnable switch; because the switch can be turned off during rasterization, gradients from one modality no longer corrupt the geometry that another modality needs. The second mechanism watches the L2 norm of the gradient difference $g^d_{ij} = \\mathrm{norm}(g^{m_i} - g^{m_j})$ accumulated on each Gaussian during densification. When that difference exceeds a threshold, the mechanism decomposes the multi-modal Gaussian into single-modal copies, each optimized by its own modality loss. The experiments claim that this yields consistently better rendering and semantics for RGB, thermal, and language, and that the decomposed representation is more compact than joint training on shared Gaussians.","pith_inferences":["The gradient-difference split rule resembles gradient-conflict resolution in multi-task learning, so an adaptive per-region threshold might replace the fixed 0.0002 value; the paper does not test that variant.","The per-modality switch suggests a cheap route to editable or privacy-filtered representations, since thermal or language information could be suppressed at inference time, an application the paper describes in passing but does not explore.","Because the compactness gain comes partly from pruning single-modal Gaussians more aggressively, an ablation that varies only the prune threshold would separate the contributions of decomposition and pruning.","A natural stress test of the scalability claim would be to add a modality with a very different dimensionality or physical unit, such as audio or tactile signals; the paper only demonstrates a fourth modality that is geometrically aligned with the others (monocular depth)."],"forward_implications":["Adding a fourth modality, monocular depth, improves depth rendering on flat surfaces and leaves RGB, thermal, and language quality unchanged, supporting the scalability claim.","Because the modality indicator acts as a switch, the trained scene can render or suppress individual modalities at inference time without retraining.","The scene is more compact as well as better: the method uses about one-third of the Gaussians of joint-training baselines on RGB-thermal scenes and about one-quarter of LangSplat's count on RGB-language scenes.","In the three-modality experiments, adding language degrades RGB and thermal by about 0.5 dB for shared-Gaussian baselines, while MMOne slightly improves them, showing the decomposition removes interference that grows with modality count."],"supporting_citations":[{"why":"3D Gaussian Splatting supplies the Gaussian representation, rendering equation, and densification loop that MMOne modifies.","marker":"[19]"},{"why":"ThermalGaussian provides the RGB-thermal joint-training baseline, the thermal loss setup, and the RGBT-Scenes dataset used for two- and three-modality tests.","marker":"[25]"},{"why":"LangSplat provides the RGB-language baseline, the SAM/CLIP language-feature pipeline, and the extended LERF dataset with ground-truth masks.","marker":"[34]"},{"why":"LERF defines the source dataset for the RGB-language scenes, used through the LangSplat extension, and the open-vocabulary query protocol.","marker":"[20]"},{"why":"SAM generates the medium-level masks whose features act as ground truth for the language modality.","marker":"[21]"},{"why":"CLIP/OpenCLIP produces the ground-truth language features that the rasterized fields are trained to match.","marker":"[37]"},{"why":"COLMAP supplies the sparse point cloud and camera poses that initialize the Gaussians, keeping all compared methods on the same geometry.","marker":"[39]"}],"fun_headline_variants":["MMOne: one scene, multiple modalities, no conflicts","Separate shared and modality-specific Gaussians to end clashes","Modality indicator prevents cross-talk in unified Gaussian scenes","Decompose multimodal Gaussians for compact and accurate scenes","MMOne resolves property and granularity disparities per modality"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes that the L2 norm of the difference between two modalities' gradients on a Gaussian, compared against a fixed threshold of 0.0002, reliably tells when that Gaussian should be split into modality-specific copies; if that heuristic does not track the true granularity needs of each modality, the quality and compactness claims lose their foundation.","fun_headline_variants_meta":{"raw":{"variants":["MMOne: one scene, multiple modalities, no conflicts","Separate shared and modality-specific Gaussians to end clashes","Modality indicator prevents cross-talk in unified Gaussian scenes","Decompose multimodal Gaussians for compact and accurate scenes","MMOne resolves property and granularity disparities per modality"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00064,"raw_usage":{"total_tokens":2927,"prompt_tokens":904,"completion_tokens":2023,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":520,"completion_tokens_details":{"reasoning_tokens":1944}},"tokens_in":520,"tokens_out":2023,"duration_ms":20742,"temperature":1.0,"reasoning_tokens":1944,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:15:36.358227+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run MMOne with the multimodal decomposition step disabled, keeping the modality indicator and soft prune: if RGB, thermal, and language quality stay within noise and the Gaussian count does not grow, the decomposition is not what carries the reported gains. Directly measuring the distribution of per-Gaussian gradient differences during training would further show whether the fixed 0.0002 threshold actually separates conflicting Gaussians from non-conflicting ones.","supporting_citations":[{"cited_title":"3D Gaussian Splatting for Real-Time Radiance Field Rendering","cited_arxiv_id":null,"evidence_quote":"3D Gaussian Splatting supplies the Gaussian representation, rendering equation, and densification loop that MMOne modifies."},{"cited_title":"Thermal- Gaussian: Thermal 3D Gaussian Splatting","cited_arxiv_id":null,"evidence_quote":"ThermalGaussian provides the RGB-thermal joint-training baseline, the thermal loss setup, and the RGBT-Scenes dataset used for two- and three-modality tests."},{"cited_title":"LangSplat: 3D Language Gaussian Splat- ting","cited_arxiv_id":null,"evidence_quote":"LangSplat provides the RGB-language baseline, the SAM/CLIP language-feature pipeline, and the extended LERF dataset with ground-truth masks."},{"cited_title":"LERF: Language Embed- ded Radiance Fields","cited_arxiv_id":null,"evidence_quote":"LERF defines the source dataset for the RGB-language scenes, used through the LangSplat extension, and the open-vocabulary query protocol."},{"cited_title":"Segment Any- thing","cited_arxiv_id":null,"evidence_quote":"SAM generates the medium-level masks whose features act as ground truth for the language modality."},{"cited_title":"Learning Transferable Visual Models From Natural Language Super- vision","cited_arxiv_id":null,"evidence_quote":"CLIP/OpenCLIP produces the ground-truth language features that the rasterized fields are trained to match."},{"cited_title":"Structure- From-Motion Revisited","cited_arxiv_id":null,"evidence_quote":"COLMAP supplies the sparse point cloud and camera poses that initialize the Gaussians, keeping all compared methods on the same geometry."}],"review_version":1}