{"id":"d3678395-5426-4eae-8d7e-87f2a16d8d6a","arxiv_id":"2504.15028","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A self-supervised 6D latent space of material appearance, learned without human labels, enables controllable appearance transfer and editing in a diffusion pipeline.","lead":"This paper trains a self-supervised model to compress how a material looks in a photo into six dials: color, gloss, lightness, and lighting direction. A diffusion pipeline uses those dials to transfer or edit the appearance of real objects, and users can adjust each dial independently.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Geometry-independence of the 6D code is asserted from an inductive bias but tested only on blob/sphere; per-dimension disentanglement on complex geometries is unmeasured.","rationale":"After reading the paper, the central claim is credible and the design is thoughtful: normals in the decoder, the TC penalty, dataset balancing, and the ablations all support the intended direction. The most load-bearing gap is not reconstruction quality or the diffusion pipeline, both of which are clearly presented as proof-of-concept; it is the geometry-exclusion assumption. The quantitative metrics are the strongest evidence for the representation claim, and they are measured only on simple geometries. The supplement's robustness experiment addresses a related but weaker question. My concern is therefore that the headline claim is under-supported, not contradicted. This matches the reader's weakest_assumption, so I agree with the CONDITIONAL verdict; no adjustment is needed beyond requiring the proposed complex-geometry evaluation to be reported or prominently caveated.","tokens_in":23127,"tokens_out":4836,"duration_ms":48136,"concrete_test":"Recompute the per-dimension disentanglement and interpretability metrics (Z-min, MIR, and MIS/GTC) on a test set that includes complex, unseen geometries (e.g., statuette, cat, chair) rendered with the same controlled material and illumination labels, instead of restricting to blob and sphere; if MIR/Z-min degrade materially or any single dimension responds systematically when geometry is varied with appearance held fixed, the geometry-exclusion premise of Sec. 3.1.2 fails and the central claim must be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that each of the six latent dimensions varies only with a single appearance attribute and not with geometry. The only mechanism offered for excluding geometry is architectural: the decoder receives normal maps (Sec. 3.1.2), so the encoder is not forced to encode shape. This is a soft inductive bias, not a constraint, and there is no direct penalty preventing geometry-dependent information from remaining in the code. The quantitative evidence in Table 1 is computed on a test subset deliberately limited to blob and sphere geometries (Sec. 4.1), chosen so that the baselines, which have no geometry-disentanglement mechanism, are not over-penalized. Consequently, the headline disentanglement and interpretability numbers never test the model where the geometry-exclusion assumption is hardest. The only complex-geometry check in the supplement (S3.4) reports aggregate cosine similarity between embeddings of the same material rendered on cylinder and statuette; an aggregate similarity value of 0.879 can remain high even if one latent dimension carries a geometry-dependent signal, so it does not establish per-dimension invariance. Since Section 5's selective transfer relies on recombining individual dimensions across images, any residual geometry dependence would directly corrupt the claimed application. The user study (S6.2) is a two-alternative preference (61.5% vs 38.5% over 20 participants) with no significance test, and it measures transfer quality, not per-dimension disentanglement. Thus the evidence does not yet secure the central claim for the general case stated in the abstract.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a self-supervised method for learning a compact, interpretable appearance representation from images of homogeneous, opaque objects. A FactorVAE-based encoder maps an input image to a six-dimensional latent code, while the decoder is conditioned on normal maps to encourage the code to capture material and illumination rather than geometry. The loss is modified with a norm-based KL regularizer and beta-annealing to mitigate posterior collapse. The authors introduce a synthetic dataset of 98,550 renders (30 geometries, 365 measured BRDFs, 9 lighting conditions) and report quantitative disentanglement, interpretability, and reconstruction comparisons against betaVAE, betaTCVAE, and FactorVAE. They then use the frozen encoder to condition a diffusion pipeline via a lightweight IP-Adapter, enabling appearance transfer, selective attribute transfer, editing, and interpolation. Code, models, and dataset are released.","tokens_in":23389,"tokens_out":5129,"duration_ms":50690,"significance":"If the central claim holds, this is a useful contribution: a six-dimensional, self-supervised appearance code that separates hue, gloss, lightness, and illumination would enable fine-grained control in material transfer and editing without human labels. The paper's strengths include the public release of code, models, and dataset; the use of measured BRDFs; systematic ablations of the loss modifications; and quantitative comparisons to three VAE baselines. The compactness of the representation and the demonstration that it can condition a diffusion pipeline are also valuable. However, the geometry-independence claim--which is load-bearing for selective transfer--is not directly measured on complex geometry, and the application-level evaluation is mostly qualitative. These issues are addressable with additional experiments, so the contribution is defensible but needs revision.","major_comments":[{"comment":"The headline disentanglement and interpretability metrics are computed only on a test subset deliberately limited to blob and sphere geometries (Sec. 4.1). Complex geometry is used only for reconstruction quality, not for disentanglement or interpretability. This leaves the central claim that each latent dimension varies only with material/illumination and not with geometry untested exactly where it is hardest. Supplement S3.4 reports an aggregate cosine similarity of 0.879 between embeddings of the same material rendered on cylinder and statuette, but an aggregate value does not rule out a single latent dimension carrying geometry-dependent signal. Because Section 5's selective transfer recombines individual dimensions across images, any per-dimension geometry leakage would directly corrupt the application. Please provide per-dimension invariance statistics (e.g., per-dimension variance or consistency across geometry renderings) on complex unseen geometries, not only an aggregate similarity.","section":"Sec. 4.1, Table 1; Sec. 5.2"},{"comment":"Conditioning the decoder on normal maps is presented as 'compelling' the latent space to focus on appearance, but it is a soft inductive bias rather than a constraint, and no direct measurement shows that geometry information is absent from the code. The 'Without Normals' ablation in Table 2 shows changes in PSNR and MIR, but it does not measure what geometry-dependent information remains in the six dimensions. I recommend a direct leakage test: train a simple linear classifier on the latent codes to predict geometry identity (with appearance and illumination held fixed), or compute per-dimension classification accuracy / mutual information with geometry. Without such a test, the architecture claim in Sec. 3.1.2 is supported only by indirect evidence.","section":"Sec. 3.1.2, Table 2"},{"comment":"The test set overlaps with the training set in materials: 'some materials are both in our training and test sets' (Sec. 4.1). Since the interpretability metrics (Z-min, MIR) use ground-truth labels on exactly this test set, the model could partially memorize material identity rather than learning general attribute semantics, inflating the reported interpretability scores. The authors note that illuminations and scene configurations differ, which mitigates but does not eliminate the confound. I ask for a hold-out material split, or at least a breakdown of MIR/Z-min separately for materials seen and unseen during training, so readers can gauge generalization to genuinely new materials.","section":"Sec. 4.1"},{"comment":"The diffusion-pipeline evaluation for appearance transfer and editing is largely qualitative. The only quantitative evidence is a two-alternative user study with 20 participants (61.5% vs. 38.5% preference over ZeST), reported in S6.2 without a significance test. Moreover, there is no measurement of whether traversing one latent dimension in the diffusion outputs changes only the intended attribute and leaves others unchanged. Given that the abstract and Section 5 claim 'fine-grained control' and 'strong disentanglement,' I recommend adding a quantitative or at least statistically sound evaluation of the diffusion outputs, for example an attribute-classification test on generated images or a significance test for the user preference data.","section":"Sec. 5.2, S6.2"}],"minor_comments":[{"comment":"The definition of the norm-based KL term is difficult to parse as typeset: D_KL(Q,P,n) appears to be a scalar expression without an explicit index over latent dimensions, so the 'norm of order n' is not reproducible from the equation. Please give the explicit form, e.g., (sum_j KL_j^n)^{1/n} or the equivalent used in the implementation.","section":"Eq. (1)"},{"comment":"The GTC column shows '-' for betaVAE and betaTCVAE. If GTC was not computed for these baselines, state why; otherwise provide the values for completeness.","section":"Table 1"},{"comment":"The KL evolution plots are described as showing how much information each dimension stores. Since the y-axis is not labeled in the figure, please add a caption note that higher values correspond to larger KL distance from the standard normal prior, and clarify the scale used.","section":"Sec. 4.3, Fig. 6"},{"comment":"The semantic names of the dimensions (lightness, hue #1, hue #2, light direction #1/#2, gloss) are assigned after training. This is acceptable, but the figure caption should make explicit that these names are post hoc interpretations of traversal behavior, not learned labels.","section":"Sec. 4.2, Fig. 3"},{"comment":"The diffusion-based posterior traversals omit the two illumination dimensions. This is a reasonable choice, but it should be stated in the main text or figure caption so readers do not infer that illumination traversals were validated in the diffusion pipeline.","section":"S6.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the journal's scope and the central idea is attractive. The main weakness is that the geometry-independence claim, which underpins selective transfer, is not directly validated at the per-dimension level on complex geometry; this is fixable with targeted experiments. I would also encourage the editor to ensure that the supplementary user study's lack of significance testing is addressed in revision, as the current claim of user preference is weaker than presented."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful, carefully built self-supervised appearance representation for homogeneous opaque materials, supported by a new 98k synthetic dataset, sensible ablations, and honest limitation notes. The headline claim of a 6D disentangled and interpretable space is credible for the tested regime. The weak spot is that \"geometry stays out of the code\" is an architectural hope, not a measured guarantee: the quantitative disentanglement metrics run only on blob/sphere test sets, and the supplement's aggregate cosine similarity (0.879 same-material) doesn't rule out per-dimension geometry leakage. The paper itself admits the VAE decoder struggles on unseen complex geometries, which indirectly supports that worry. The diffusion pipeline is a proof-of-concept application; its evaluation is mostly qualitative, and the user study (20 participants, 61.5% vs 38.5%) lacks a significance test. None of this kills the central contribution, but the abstract overstates the general case.\n\nWhat is genuinely new: an adapted FactorVAE with normal-map decoder conditioning, a carefully designed unlabeled dataset, and the finding that six dimensions (hue x2, light dir x2, lightness, gloss) emerge without labels. The ablations on the KL norm, beta annealing, normals, and reconstruction loss are real evidence. The quantitative disentanglement metrics (GTC, MIS, Z-min, MIR) and reconstruction quality beat the baselines. Dataset and code are promised, which is good.\n\nWhere I'd push back: the stress-test concern about geometry is not a nitpick, because Section 5's selective transfer recombines individual dimensions across images. If a dimension carries a geometry-dependent signal, the application breaks. The fix isn't hard: report per-dimension robustness on complex geometries, e.g. measure each latent coordinate's variance across geometries for fixed material/lighting, not just mean cosine similarity. They should also either add a significance test to the user study or drop it to a preference report. Material overlap between train/test is disclosed and mitigated by different illuminations; I don't consider that fatal.\n\nWho this is for: graphics/vision readers working on material appearance, disentangled representations, or diffusion-based editing. It warrants a rigorous referee; I'd recommend accept with major-but-targeted revision. The core method is solid; the evaluation needs to close the geometry gap before the strong claim can stand.","headline":"Solid self-supervised 6D appearance space with honest caveats, but geometry-independence is only tested on simple shapes and the diffusion part is proof-of-concept.","tokens_in":23939,"tokens_out":2102,"would_cite":true,"duration_ms":19373,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that the appearance of a homogeneous object in a single image can be compressed without labels into a six-dimensional code — two hue axes, two light-direction axes, lightness, and gloss — that then conditions a diffusion…","keywords":["material appearance","disentangled representation learning","self-supervised learning","latent space interpretability","appearance transfer","appearance editing","FactorVAE","compact latent representation"],"falsifier":"A concrete check is to render the same measured material and lighting on several complex unseen shapes, encode each image, and compare the six-dimensional codes: if codes drift with geometry, or if stepping along the gloss or hue dimension changes shadows or shape-dependent highlights, then geometry has leaked into the appearance code. The supplement already reports a template for this, with mean cosine similarity 0.879 between same-material pairs on two different geometries versus 0.321 for different materials; repeating that measurement on complex geometries with a full latent-traversal inspection would settle the claim.","tokens_in":22915,"feed_emoji":"🎨","tokens_out":11342,"duration_ms":95712,"temperature":0.7,"pith_summary":"This paper tries to establish that the visual appearance of a homogeneous, opaque object in a single image can be compressed, without any human labels, into a six-dimensional code in which two dimensions encode hue, two encode illumination direction, one encodes lightness, and one encodes gloss. The authors adapt a FactorVAE: they feed normal maps into the decoder so geometry is supplied rather than encoded, and they reshape the KL regularizer with an order-3 norm and an annealed weight to stop posterior collapse. They train on a self-built set of 98,550 synthetic renders combining 30 geometries, 365 measured BRDFs, and nine lighting conditions, then use the frozen encoder to condition a pretrained latent diffusion model through a lightweight cross-attention adapter. If the claim is right, ordinary images become editable in material terms, and appearance can be transferred from one or several exemplars with slider-like control over individual attributes.","feed_headline":"One 6D code stores a material's look and lighting","feed_subtitle":"A label-free autoencoder splits a material's look into six independent knobs for diffusion-based transfer and editing.","key_machinery":"The load-bearing device is the six-dimensional latent vector $f\\in\\mathbb{R}^6$ produced by an adapted FactorVAE. The encoder maps an input image of a homogeneous object to this vector, while the decoder concatenates a normal map of the target geometry into its layers, so geometry is handed to the model rather than stored in the code. The training loss combines a smooth-L1 reconstruction term, a dimension-wise KL regularization term $\\beta\\,D_{KL}(q_\\theta(z|x),p(z),n)$ with an order-$n$ norm and annealed weight, and a total-correlation term computed by a discriminator to encourage independent dimensions. A dimensionality sweep from 3 to 10 justifies the choice of 6, balancing the Mutual Information Ratio (interpretability) and Mutual Information Score (disentanglement). In the diffusion stage, the frozen encoder becomes the appearance-conditioning branch, with a lightweight cross-attention adapter translating $f$ into generation guidance and an edge-plus-depth conditioning branch supplying the target geometry.","core_discovery":"The central claim, stated by the authors, is that a variational autoencoder with a total-correlation disentanglement objective can learn an interpretable and disentangled image-space code for material appearance without explicit supervision, provided the decoder is given the target geometry as normal maps. The learned six dimensions turn out to be: lightness; two hue axes that the model aligns approximately perpendicular in the chromatic circle; two light-direction axes; and gloss. This code lives in a 6D space selected by sweeping dimensionality from three to ten and balancing interpretability and disentanglement metrics. The paper further shows that the same encoder can drive a diffusion-based inpainting pipeline, enabling appearance transfer, selective transfer of chosen dimensions from different images, interpolation between materials, and fine-grained editing of individual attributes in real photographs, even though training is synthetic. The authors acknowledge limits: the autoencoder alone degrades on geometries very different from training, and the representation covers only homogeneous opaque materials under moderate lighting.","pith_inferences":["If the code is as geometry-free as claimed, the same encoder could serve as a retrieval descriptor for material search across arbitrary object shapes; the paper mentions retrieval only as future work, so this is an extension rather than a paper claim.","The same self-supervised recipe, applied to render sets that vary texture, roughness, anisotropy, or translucency, could plausibly yield interpretable dimensions for those attributes; the paper does not test this.","Because only two dimensions encode illumination direction, selective transfer between images with different environment colors or high-frequency lighting structures will likely fail; the paper itself reports degradation on high-frequency lighting, so the practical scope of the code is limited to moderate, roughly neutral lighting.","The reported preference for the proposed transfer over a semantic-image-embedding baseline (61.5% versus 38.5% in a user study) suggests geometry leakage in the baseline is a real practical issue, but a larger study that varies geometry complexity would be needed to know how much the advantage generalizes."],"forward_implications":["A single photograph of a homogeneous object can be compressed to a 6D vector carrying material and lighting, so appearance transfer no longer needs a text prompt or human-annotated labels.","Because the dimensions are independent, users can move one attribute at a time — gloss, hue, lightness, or light direction — while the other attributes stay fixed.","Appearance can be assembled from several exemplars by taking different dimensions from different images, such as hue from one object and gloss and lightness from another.","The image-space code transfers to real photographs even though training is synthetic, letting a real object's appearance be re-rendered on a new geometry.","Edits remain local to the material and leave geometry and background intact, which is the advantage the paper claims over text-instruction editing methods."],"supporting_citations":[{"why":"Defines the FactorVAE objective, including the total-correlation discriminator term, which the paper adapts for appearance disentanglement.","marker":"[KM18]"},{"why":"Supplies the measured BRDF database used to render a large portion of the training set.","marker":"[MPBM03]"},{"why":"Supplies an additional set of measured BRDFs that diversify the training materials.","marker":"[DJ18]"},{"why":"Provides the test-set images with ground-truth labels used for interpretability metrics and contributes material variants used in training and testing.","marker":"[SCW*21]"},{"why":"Demonstrates the normal-map-in-decoder conditioning strategy that the paper uses to keep geometry out of the latent code.","marker":"[DLC*22]"},{"why":"Defines the cross-attention adapter architecture used to condition the diffusion pipeline on the 6D appearance vector.","marker":"[YZL*23]"},{"why":"Provides the edge- and depth-conditioning branch used to inject target geometry into the diffusion pipeline.","marker":"[ZRA23]"},{"why":"Serves as the zero-shot material-transfer baseline whose geometry-appearance entanglement is compared against the proposed approach.","marker":"[CSM*24]"},{"why":"Baseline for instruction-based image editing, compared against the proposed fine-grained appearance edits.","marker":"[BHE23]"},{"why":"Supervised material-editing baseline used for comparison on gloss edits.","marker":"[SL23]"}],"fun_headline_variants":["Self-supervised 6D code splits material look, lighting for diffusion editing","No labels, six knobs: a compact code for material appearance transfer","Disentangled 6D material code: edit lightness, hue, gloss, light direction","Label-free autoencoder disentangles material appearance into six knobs","One 6D code, no labels: material look and lighting for diffusion transfer"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that feeding the decoder the target geometry as normal maps keeps geometry out of the six-dimensional code, a premise that is only measured on simple blob and sphere test geometries (Section 4.1) while the paper itself reports artifacts when decoding very different geometries (Section 4.2, supplement S3.3).","fun_headline_variants_meta":{"raw":{"variants":["Self-supervised 6D code splits material look, lighting for diffusion editing","No labels, six knobs: a compact code for material appearance transfer","Disentangled 6D material code: edit lightness, hue, gloss, light direction","Label-free autoencoder disentangles material appearance into six knobs","One 6D code, no labels: material look and lighting for diffusion transfer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000941,"raw_usage":{"total_tokens":3985,"prompt_tokens":874,"completion_tokens":3111,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":490,"completion_tokens_details":{"reasoning_tokens":3010}},"tokens_in":490,"tokens_out":3111,"duration_ms":19557,"temperature":1.0,"reasoning_tokens":3010,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:35:10.540178+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check is to render the same measured material and lighting on several complex unseen shapes, encode each image, and compare the six-dimensional codes: if codes drift with geometry, or if stepping along the gloss or hue dimension changes shadows or shape-dependent highlights, then geometry has leaked into the appearance code. The supplement already reports a template for this, with mean cosine similarity 0.879 between same-material pairs on two different geometries versus 0.321 for different materials; repeating that measurement on complex geometries with a full latent-traversal inspection would settle the claim.","supporting_citations":[],"review_version":1}