{"id":"6c33ad35-c38b-4967-a3cd-16cdd4013485","arxiv_id":"2412.15211","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Jointly relighting all input views with a multiview diffusion model, then fitting a radiance field with per-image shading embeddings, reconstructs specular 3D objects from inconsistently lit photos better than prior methods.","lead":"A team from Google and the University of Maryland trains a diffusion model that relights an entire set of photos of an object at once, so all photos appear lit by one chosen reference light, then fits a 3D radiance field to the relit images. The method keeps shiny, view-dependent reflections that earlier appearance-embedding and inverse-rendering pipelines flatten away.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The §3.2 assumption that relighting residuals are mostly warped specular highlights is only qualitatively supported by a light-probe visualization; if diffuse, shadow, or albedo errors occur, per-image normal perturbations cannot absorb them and the reconstruction would inherit blur.","rationale":"The reader's weakest-assumption analysis identifies exactly the load-bearing point: the shading embedding is only justified if relighting errors are equivalent to small surface-normal perturbations. I agree with that assessment. The paper gives some internal support for the rest of the pipeline: the N-frame ablation (Table 3) shows large gains from joint relighting, and the training-data ablation (Table 4) is consistent with the stated design. However, the evidence for the error-model assumption is thin: Fig. S2 is qualitative and uses a light probe, not the textured or shadowed objects that appear in the evaluation, and Table 5 shows only a 0.52 dB improvement from the shading embedding on shiny assets, which is not strong evidence that it absorbs all residual inconsistency. The paper's own Limitations section concedes the mask-and-pose requirement, but that is a standard and acknowledged constraint rather than the most fragile step. The diffuse-or-shadow failure mode is concrete and testable, and if it occurs it would weaken the central claim of high-fidelity view-dependent reconstruction. This does not warrant rejection, because the reported numbers may still stand; it does justify the conditional verdict already given by the reader, so I recommend no change.","tokens_in":14576,"tokens_out":7410,"duration_ms":76274,"concrete_test":"On the 20 held-out synthetic test scenes, compute the per-pixel error between each relit image and its ground truth under the reference illumination, then split the error into specular versus diffuse or shadow regions using known material and geometry from the renderer (for example, a specular mask or a threshold on the reference image's local contrast). Report the fraction of total relighting MSE that falls in non-specular regions. If this fraction is substantial (for example, above 20%), the §3.2 error model is violated and the shading embedding cannot be expected to absorb the residual inconsistency; if it is near zero and residuals are highlight-localized, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The method's robustness to relighting inconsistencies rests on the §3.2 premise that 'the subtle inter-image errors made by the diffusion model are usually due to the specular highlights being slightly warped,' so the per-image normal perturbation in Eq. 1 is sufficient. The only direct evidence is Fig. S2, which relights a spherical light probe and shows that extracted environment maps are distorted but content-preserving; that is qualitative and specific to a mirror-like probe. It does not establish the error distribution for textured objects with diffuse albedo, cast shadows, or complex geometry. If the diffusion model also produces diffuse color shifts, shadow-boundary errors, or invented or missing reflections, the shading embedding cannot absorb them: it only warps normals and cannot alter the environment content or per-image diffuse color, and the shared radiance field will blur or ghost those regions. The ablation in Table 5 shows only a 0.52 dB improvement over not using any embedding on shiny assets, which is modest evidence that the mechanism fully resolves the inconsistency problem. This is a correctness risk to the central claim of high-fidelity view-dependent appearance, not a demonstrated error.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-stage pipeline for 3D reconstruction from a set of posed images of an object captured under differing, potentially extreme illuminations. In the first stage, a multiview latent diffusion model jointly relights all N input images to match the illumination of a selected reference image, using camera poses encoded as raymaps and a reference mask to condition the denoising process. In the second stage, the relit images are fed into a NeRF-Casting-based radiance field that incorporates a per-image \"shading embedding\" (Eq. 1) to perturb surface normals, with the aim of absorbing small residual inconsistencies, particularly warped specular highlights, in the relit images. The method is evaluated on synthetic Objaverse scenes and real NA VI scenes, with quantitative results reported in Tables 1 and 2 and qualitative comparison in Figures 1, 4, and 5. The paper also presents ablations on the number of jointly relit frames (Table 3), on training with mirrored materials (Table 4), and on the choice of per-image embedding (Table 5), all of which support the proposed design choices.","tokens_in":14659,"tokens_out":4988,"duration_ms":47965,"significance":"If the claims hold, this is a practically useful advance: it avoids the ambiguities of inverse rendering, handles highly specular objects that prior methods render as mostly diffuse, and does not require a target environment map, unlike IllumiNeRF and Neural Gaffer. The strongest evidence is the ablations: joint relighting clearly beats single-frame relighting (Table 3), mirrored-material training data helps both shiny and standard assets (Table 4), and the shading embedding outperforms both no embedding and standard appearance embeddings (Table 5). The synthetic-to-real generalization demonstrated on NA VI is encouraging. The paper's claims are empirical rather than derivational, and no formal proofs are offered; the main value is in the system and the evaluation. The work is incremental relative to recent relighting-plus-reconstruction pipelines, but the joint multiview relighting formulation and the shading-embedding idea are plausible and well-motivated contributions.","major_comments":[{"comment":"The synthetic evaluation may not be an independent test of the relighting model. Section 4 states that the relighting model is trained on \"a dataset of ~300K high-quality objects, similar in appearance to Objaverse but with more diverse materials,\" while Section 5 states that the synthetic benchmark is \"objects from Objaverse.\" The relationship between these two datasets is not specified. If any of the 20 test objects in Table 1, or close derivatives thereof, appear in the training set, then the reported PSNR/SSIM/LPIPS numbers are not evidence of generalization to unseen objects. Please state explicitly whether the training asset dataset is disjoint from the Objaverse test split, and if not, report results on a held-out set of objects that are guaranteed absent from training. This is load-bearing because Table 1 is the central quantitative claim of the paper.","section":"Section 4 / Section 5"},{"comment":"The central assumption that relighting residuals are \"usually due to the specular highlights being slightly warped\" is supported only by the qualitative light-probe visualization in Figure S2. That evidence is specific to a spherical, perfectly mirror-like probe and does not characterize the error distribution for textured objects with diffuse albedo, cast shadows, or complex geometry. If the diffusion model also produces diffuse color shifts, shadow-boundary errors, or invented/missing reflections, the shading embedding in Eq. (1) cannot absorb them, because it only warps normals and cannot change the local diffuse color or the environment content. The 0.52 dB improvement over no embedding on shiny assets in Table 5 is modest support that this mechanism fully resolves the inconsistency problem. Please provide a quantitative analysis of relighting residuals, e.g., by decomposing per-pixel errors into diffuse-color, shadow-displacement, and specular-warp components on textured synthetic scenes, and show that the shading embedding specifically reduces the specular-warp component while not harming the others. This would substantiate the load-bearing premise of Section 3.2.","section":"Section 3.2 / Figure S2 / Table 5"}],"minor_comments":[{"comment":"In the discussion of Table 5, the sentence \"using standard embeddings as in NeRF-W [32] ('Shading embeddings')\" appears to contain a typo: the table row is labeled \"Appearance embeddings,\" so the parenthetical should read \"Appearance embeddings\" rather than \"Shading embeddings.\"","section":"Section 5.2"},{"comment":"The caption of Figure 2 says \"leftmost pane\" but the figure layout is described as multiple panes; please clarify the intended ordering, as the relighting diffusion model is shown before the radiance field in the current caption.","section":"Section 3.1 / Figure 2"},{"comment":"The classifier-free guidance scale is fixed at 3 and no sensitivity study is reported; since CFG is a user-set hyperparameter that can strongly affect the consistency and sharpness of diffusion outputs, please state whether the chosen value was tuned and, if so, on which validation set.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is technically solid and the ablations are internally consistent. The major revision is warranted by two load-bearing points that need additional evidence: (1) clarification and, if necessary, correction of the train/test overlap between the relighting model's training set and the Objaverse test set; (2) a quantitative characterization of the relighting model's residual error types to justify the shading-embedding mechanism. If these are addressed, the paper would likely be a strong contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely useful system paper, and the central claim—that you can reconstruct high-fidelity, view-dependent appearance from photos taken under drastically different lighting—holds up better than the baselines. The novelty is real: relighting many views jointly rather than one at a time, and a shading embedding that perturbs surface normals instead of using a NeRF-W-style appearance code.\n\nThe design is sensible and the ablations are consistent. Joint relighting clearly matters: one frame gives 28.39 PSNR, 64 frames give 31.34 on standard assets; the mirrored-material augmentation helps both shiny and standard objects; and the shading embedding beats both no embedding and appearance embeddings on shiny assets. The NA VI real-data results are stronger than the baselines, and the chrome figurine captures demonstrate something previous methods cannot do.\n\nBut I have two concerns, one substantive and one about reproducibility. The substantive one is the §3.2 assumption that diffusion relighting errors are mainly warped specular highlights. The support is Fig S2, a light-probe experiment showing environment maps extracted from relit sphere images are distorted but content-preserving. That is a mirror, so it proves the error mode exists for specular reflection, not that it dominates for textured objects with diffuse albedo, cast shadows, or complex geometry. The shading embedding can only tilt normals; it cannot fix a diffuse color shift or a shadow-boundary error. Table 5 shows the embedding yields a 0.52 dB gain over no embedding on shiny assets—real but modest. So there is a correctness risk, not a demonstrated failure. The reproducibility concern is more mundane: the model is trained on a private ~300K-object set 'similar to Objaverse', no code or data is released, and all quantitative tables report single runs without variance. These limit how much weight we can put on the exact numbers.\n\nDespite that, I would send this to peer review. The method is novel, the evaluation includes real data with external validity, and the ablations are honest—Table 5 shows a modest benefit, and the Limitations concedes masks and accurate poses are required. A serious referee could push the authors to test the error-type assumption more directly and to release the training pipeline. For my own work, I would cite the joint-relighting result and the shading embedding idea. Bring it to reading group.","headline":"A genuinely useful system paper—joint multiview relighting plus normal-warping shading embeddings—that clearly beats prior work, though the robustness mechanism is thinner than the results suggest.","tokens_in":15352,"tokens_out":4019,"would_cite":true,"duration_ms":22745,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that jointly relighting all input photos to a reference illumination with a multiview diffusion model, then fitting a radiance field with per-image normal perturbations, reconstructs shiny 3D objects from photos taken…","keywords":["3D reconstruction","multiview relighting","diffusion models","neural radiance fields","specular appearance","view-dependent effects","image harmonization","inverse rendering"],"falsifier":"Render a synthetic object under extreme lighting, run the relighting model, fit the radiance field, and inspect the residual images between the relit training views and the model's renderings: if large residuals concentrate in diffuse, shadow, or geometry-edge regions rather than near specular highlights, the normal-perturbation assumption is wrong. A direct control is to modify ground-truth relit images by intentionally warping their normals versus by additively changing diffuse albedo, and verify that the shading embedding absorbs the former but not the latter.","tokens_in":14233,"feed_emoji":"✨","tokens_out":9440,"duration_ms":73870,"temperature":0.7,"pith_summary":"Reconstructing a 3D object from photographs is hard when each photo was taken under different lighting, because the object's appearance changes from image to image. This paper argues that the right move is to remove that difficulty first: relight every input image to match one chosen reference image, using a diffusion model (a generative machine-learning model that rewrites an image under a new illumination) that processes all views together instead of one at a time, and only then reconstruct a radiance field from the harmonized images. The authors' key claim is that the residual errors left by even a good joint relighting model are mostly small warps in specular highlights, so they equip the radiance field with a per-image shading embedding that slightly re-aims the surface normals used for shading. With that mechanism, the method recovers sharp, view-dependent reflections from images captured under extreme illumination variation, where per-image appearance embeddings tend to flatten everything into diffuse color and inverse-rendering methods blur. Quantitative comparisons on synthetic and real benchmarks support the claim that this pipeline outperforms prior methods.","feed_headline":"Joint relighting recovers shiny 3D objects from varied-light photos","feed_subtitle":"Joint relighting of all views plus a normal-shifting embedding beats inverse rendering for shiny objects","key_machinery":"Two components carry the argument. The first is a multiview latent diffusion model: it denoises the latent code of each view simultaneously, attending across all views with 3D self-attention and conditioning on each camera pose (encoded as raymaps) and on a binary reference mask that marks which image supplies the target illumination. This joint denoising yields relit images that share one consistent interpretation of geometry and material. The second is the shading embedding: a per-image vector that, together with the radiance field's geometry feature, is fed through a three-layer MLP to produce the surface normal used for shading that image. Because only the normal is per-image, the model can shift a specular highlight by a few degrees from one view to the next, absorbing the diffusion model's small normal-estimation errors while keeping a single albedo and geometry. The reconstruction backbone is NeRF-Casting, which models view-dependent reflections by tracing secondary reflected rays into the same radiance field.","core_discovery":"The paper's central claim is that a 3D scene can be reconstructed from inconsistently lit photographs by first jointly relighting all views with a single multiview diffusion model to match a reference illumination, then fitting a NeRF-Casting-style radiance field whose only per-image adaptation is a shading embedding that perturbs the surface normals used for reflection. The discovery reported is that joint relighting makes the images consistent enough for this to work, and that the residual inconsistencies are dominated by specular highlights being slightly tilted, an error the shading embedding can absorb without corrupting diffuse appearance. Rendered novel views retain sharp, moving reflections, and the method outperforms prior diffusion-relighting, appearance-embedding, and inverse-rendering baselines on synthetic Objaverse scenes and on real NA VI photographs.","pith_inferences":["The relight-then-reconstruct decomposition might extend beyond objects to whole scenes: if a multiview diffusion model can harmonize entire scenes, the same shading embedding should prevent flicker in indoor or outdoor reconstructions where lights change between captures.","The shading embedding effectively converts the diffusion model's geometry estimate into per-image normal corrections; a testable extension is to read out these corrections to improve the diffusion model itself, for instance by fine-tuning on the radiance field's predicted normals.","Because the paper flags camera pose estimation for shiny objects as the hardest unsolved input, coupling the radiance field's inferred normals with learning-based pose estimators seems a natural next step, though the paper only suggests it implicitly.","Real internet photo collections of the same object rarely come with masks; a practical extension would be to estimate soft masks jointly, since the mask loss used on NA VI suggests the pipeline tolerates imprecise masks."],"forward_implications":["Specular and mirror-like objects, which prior per-illumination-embedding and inverse-rendering methods reduce to diffuse blobs, can be reconstructed at high fidelity simply from photos taken under very different lights.","The method needs no known environment map and no constant-lighting input; any input image can serve as the reference, so the user chooses the illumination under which the recovered object is viewed.","Increasing the number of views relit jointly, from 1 to 8 to 64, steadily improves reconstruction quality, suggesting the approach benefits from larger multi-view captures rather than saturating.","Training the relighting model on purely mirror-like objects improves performance even for mostly diffuse objects, so high-quality synthetic shiny data is a broadly useful training signal.","The hard problem of jointly estimating geometry and varying illumination decomposes into relighting then reconstruction, a division that future methods can build on."],"supporting_citations":[{"why":"Radiance-field backbone: supplies the traced reflected rays and geometry features that the reconstruction extends, and the baseline for the no-embedding ablation.","marker":"[45]"},{"why":"Provides the multiview diffusion architecture—3D self-attention, camera-pose raymaps, and set-based denoising—that the relighting model adapts from single- to multi-image relighting.","marker":"[15]"},{"why":"The main diffusion-relighting baseline; relights each image independently given a known environment map, which the paper compares against (with the true map) on Objaverse.","marker":"[55]"},{"why":"Defines the per-image appearance embedding approach that the paper ablates against; the shading embedding is designed to avoid its diffuse flattening.","marker":"[32]"},{"why":"Inverse-rendering baseline that estimates per-image illumination and materials; the strongest non-generative comparison on Objaverse and NA VI.","marker":"[25]"},{"why":"Source of synthetic 3D assets used for training the relighting model and for the standard-vs-shiny test splits.","marker":"[13]"},{"why":"Real captured dataset with pose and mask annotations used for the real-world evaluation on 18 scenes.","marker":"[22]"},{"why":"Latent diffusion architecture and weights used to initialize the relighting model.","marker":"[38]"},{"why":"Prior diffusion-based relighting method that also requires a reference environment map; cited as related work needing constant illumination and independent per-image relighting.","marker":"[24]"}],"fun_headline_variants":["Joint relighting tames extreme illumination for shiny 3D reconstruction","Diffusion relighting unlocks shiny 3D from variably lit photos","Relight first, then reconstruct: shiny 3D from extreme lighting","Multiview relighting beats inverse rendering for specular objects"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the residual errors of the joint relighting model are mostly small warps of specular highlights caused by slightly wrong surface normals, so a per-image normal perturbation is enough to make the relit images consistent, and that usable object masks and accurate camera poses are available, which are hardest to obtain for the shiny objects this method targets.","fun_headline_variants_meta":{"raw":{"variants":["Joint relighting tames extreme illumination for shiny 3D reconstruction","Diffusion relighting unlocks shiny 3D from variably lit photos","Relight first, then reconstruct: shiny 3D from extreme lighting","Multiview relighting beats inverse rendering for specular objects"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000112,"raw_usage":{"total_tokens":1037,"prompt_tokens":901,"completion_tokens":136,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":60}},"tokens_in":517,"tokens_out":136,"duration_ms":1822,"temperature":1.0,"reasoning_tokens":60,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:32:27.859053+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Render a synthetic object under extreme lighting, run the relighting model, fit the radiance field, and inspect the residual images between the relit training views and the model's renderings: if large residuals concentrate in diffuse, shadow, or geometry-edge regions rather than near specular highlights, the normal-perturbation assumption is wrong. A direct control is to modify ground-truth relit images by intentionally warping their normals versus by additively changing diffuse albedo, and verify that the shading embedding absorbs the former but not the latter.","supporting_citations":[{"cited_title":"Srinivasan, Dor Verbin, Keunhong Park, Ricardo Martin Brualla, and Philipp Henzler","cited_arxiv_id":null,"evidence_quote":"The main diffusion-relighting baseline; relights each image independently given a known environment map, which the paper compares against (with the true map) on Objaverse."},{"cited_title":"Srinivasan, Peter Hedman, Ben Milden- hall, Benjamin Attal, Richard Szeliski, and Jonathan T","cited_arxiv_id":null,"evidence_quote":"Radiance-field backbone: supplies the traced reflected rays and geometry features that the reconstruction extends, and the baseline for the no-embedding ablation."},{"cited_title":"Srinivasan, Jonathan T","cited_arxiv_id":null,"evidence_quote":"Provides the multiview diffusion architecture—3D self-attention, camera-pose raymaps, and set-based denoising—that the relighting model adapts from single- to multi-image relighting."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the per-image appearance embedding approach that the paper ablates against; the shading embedding is designed to avoid its diffuse flattening."},{"cited_title":"NeROIC: Neural Rendering of Objects from Online Image Collections","cited_arxiv_id":null,"evidence_quote":"Inverse-rendering baseline that estimates per-image illumination and materials; the strongest non-generative comparison on Objaverse and NA VI."},{"cited_title":"Objaverse: A universe of annotated 3d objects","cited_arxiv_id":null,"evidence_quote":"Source of synthetic 3D assets used for training the relighting model and for the standard-vs-shiny test splits."},{"cited_title":"NA VI: Category- agnostic image collections with high-quality 3D shape and pose annotations","cited_arxiv_id":null,"evidence_quote":"Real captured dataset with pose and mask annotations used for the real-world evaluation on 18 scenes."},{"cited_title":"High-resolution image syn- thesis with latent diffusion models","cited_arxiv_id":null,"evidence_quote":"Latent diffusion architecture and weights used to initialize the relighting model."},{"cited_title":"Neural gaffer: Relighting any object via diffusion","cited_arxiv_id":null,"evidence_quote":"Prior diffusion-based relighting method that also requires a reference environment map; cited as related work needing constant illumination and independent per-image relighting."}],"review_version":1}