Pith. sign in

REVIEW 4 major objections 6 minor 3 references

Relightable Full-Body Gaussian Codec Avatars

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Pith's one-line read This paper introduces the first relightable full-body avatar model that jointly handles the body, face, and hands, using 3D Gaussian splatting with light transport decomposed into local and non-local effects.

desk verdict Solid full-body relightable avatar system whose novel-illumination claim outruns the quantitative evidence; still worth a serious referee. read the letter →

arxiv 2501.14726 v1 pith:N4T2KROS submitted 2025-01-24 cs.CV cs.GR

classification cs.CVcs.GR
keywords relightableavatarsfull-body3DGaussiansplattingzonalharmonicslighttransportshadownetworkdeferredshadingneuralrendering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a full-body avatar—body, face, and hands together—can be made relightable and drivable by decomposing light transport into a local part and a non-local part. The local part is a diffuse radiance transfer learned in each Gaussian's own coordinate frame using zonal harmonics, which rotate cheaply when the body articulates. The non-local part is a learned shadow network that predicts occlusion shadows from a normalized irradiance map on a coarse tracked mesh. The specular part is rendered in screen space via deferred shading, capturing high-frequency reflections like eye glints without inflating the Gaussian count. If correct, this is the first method to jointly relight and animate the whole body from light-stage data at higher quality than a standard physically based rendering baseline.

What carries the argument

The central machinery is the three-way decomposition of light transport. Zonal harmonics (spherical harmonics with one axis of symmetry) can be rotated to world space far more cheaply than full SH coefficients, so the model learns diffuse transfer per Gaussian in its local tangent frame and rotates it under articulation. The shadow network is a UV-space convolutional network, adapted from earlier full-body avatar work, that takes a normalized irradiance map (visibility-weighted irradiance divided by unshadowed irradiance, Eq. 15) on a coarse tracked mesh and predicts a single shadow factor per texel, rendering non-local occlusion. Deferred shading rasterizes per-Gaussian specular normals, roughness, and visibility to screen space, then integrates the environment against a spherical Gaussian lobe centered at the reflected view direction, so specular detail is not limited by local Gaussian density.

What would settle it

Train the model on light-stage point lights whose directions cover only a limited angular range, then relight under an environment map containing a bright source from a direction outside that range and compare the rendered shadows to a path-traced reference. If shadow boundaries or darkness move to the wrong location, the irradiance normalization has not made the network illumination-invariant and the novel-illumination generalization claim fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that full-body relighting can be factored into three learned components that each handle what they are good at: zonal-harmonic diffuse radiance transfer in local coordinates for orientation-dependent local shading, an irradiance-conditioned shadow network for non-local self-occlusion between body parts, and deferred-shading-based spherical-Gaussian specular transfer for high-frequency highlights. The paper demonstrates on captured light-stage sequences that this combination outperforms a physically based rendering appearance model and its own ablations (SH instead of ZH, no shadow network, no deferred shading) in perceptual quality, and that it generalizes to unseen poses and to held-out point-light configurations.

Load-bearing premise

The shadow network, trained on normalized irradiance computed from a coarse tracked mesh, must generalize to lighting conditions it never saw during training; the quantitative evidence covers held-out point-light configurations from the same light stage, while environment-map relighting is shown only qualitatively.

Editorial extensions

If this is right

  • A full-body avatar can be relit and animated in near-real time without multi-bounce path tracing, since global illumination effects are captured by learned light transport rather than simulation.
  • The diffuse light transport parameter count drops from 113 to 51 per texel compared to the SH variant, while perceptual quality improves, making the representation more memory-efficient.
  • The shadow network generalizes to held-out point-light configurations and the method shows environment-map relighting qualitatively, which the authors take as evidence that the normalization scheme makes the network illumination-invariant.
  • Deferred shading recovers high-frequency specular effects such as eye glints that per-Gaussian shading misses when the Gaussian count is limited, closing a fidelity gap for full-body models.
  • The avatar can be animated to unseen poses and still render correctly under novel lighting, a combination that prior full-body methods did not jointly achieve.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A stress test the paper leaves implicit: feed the shadow network an environment map with a sharp directional source from an elevation not present in the light-stage training distribution; the normalization in Eq. (15) is only as illumination-invariant as the coverage of training light directions, so failure would show up as a misplaced shadow boundary rather than a brightness error.
  • The ZH local-coordinate design suggests a path toward avatars that are relightable from monocular or sparse-view video: if rotation is the only pose-dependent operation, the same transfer coefficients could be trained with far fewer lighting conditions.
  • The deferred-shading specular branch could be extended to fit a fuller BRDF (normal, roughness, plus a second specular lobe) without changing the splatting, which might push eye and jewelry highlights closer to path-traced quality.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents Relightable Full-Body Gaussian Codec Avatars, a relightable and animatable full-body avatar model built on 3D Gaussian splatting. The appearance pipeline combines three components: zonal-harmonic (ZH) diffuse radiance transfer learned in each Gaussian's local frame and evaluated in the world frame via the Gaussian rotation (Sec. 3.2.1); a UV-space shadow network that predicts a scalar shadow map from physically normalized incoming irradiance on a coarse tracked mesh, targeting non-local shadows from body-part occlusion (Sec. 3.3); and deferred shading for spherical-Gaussian specular radiance transfer, argued to preserve high-frequency reflections such as eye glints under a limited Gaussian budget (Sec. 3.2.2). The model is trained on multi-view light-stage data (512 cameras, 1024 controllable lights, roughly 5000-6000 frames per subject) with known illumination, using L1 plus LPIPS reconstruction and several regularizers. The authors claim this is the first relightable full-body avatar jointly modeling body, face, and hands, and they report quantitative results on held-out point-light frames for three subjects, ablating against PBR, unrotated SH, no shadow network, no deferred shading, and mesh normals, together with qualitative environment-map relighting.

Significance. If correct, this would be a meaningful advance over head-only relightable Gaussian avatars: the ZH formulation gives an efficient way to obtain pose-dependent diffuse transfer for articulated bodies without per-joint SH rotation, the irradiance-normalized shadow network is a principled alternative to per-iteration ray tracing for non-local shadows, and deferred shading decouples reflection frequency from Gaussian density for specular radiance transfer. The experimental methodology is generally solid: the ablations isolate each component, the held-out camera and pose splits constitute genuine generalization tests, and the Appendix A demonstration that a single Monte Carlo sample suffices for the irradiance input is a useful robustness check. The limitations section is candid, conceding that cloth dynamics may be physically implausible, that eye/face/hand fidelity is below specialized methods, and that the capture setup has limited scalability, which appropriately tempers the 'high-fidelity' and 'first' claims.

major comments (4)
  1. [Sec. 3.3, Eq. (15), Table 1, Fig. 9] The abstract's claim of 'superior generalization ability under novel illumination conditions' is supported quantitatively only for held-out point-light configurations (Table 1, 'Unseen Motion' column); environment-map relighting appears only qualitatively in Fig. 9. This distinction is load-bearing because the shadow network of Sec. 3.3 is trained on normalized irradiance maps computed from 10-20 sparse point lights, which are near-binary, whereas environment maps produce smooth, dense normalized-irradiance inputs; the physical normalization of Eq. (15) plausibly reduces but does not eliminate this distribution shift, and the network's invariance to it is asserted rather than demonstrated. The 1-sample robustness result of Appendix A (Table A.1) addresses noise tolerance, not illumination-class transfer. In addition, the shadow map is scalar and view-independent (Eq. 16) yet multiplies the specular term, so specular occlusion is modeled only approximately. I request either a quantitative environment-map evaluation (e.g., light-stage frames lit by representative environment maps, or comparisons against path-traced references) or a restriction of the generalization claim to the point-light setting.
  2. [Sec. 4.2, Table 1] No direct comparison is made to any published full-body relighting method, although Sec. 2.2 cites several as state of the art (Relighting4D [Chen and Liu 2022], IntrinsicAvatar [Wang et al. 2024], Animatable and Relightable Gaussians [Li et al. 2024b], MeshAvatar [Chen et al. 2024c]). The only external baseline is a PBR appearance model built on the authors' own learned geometry, so the 'first relightable full-body avatar' and 'superior quality' claims are not directly supported by comparison with prior systems. The dataset-incompatibility justification is understandable, but the paper would be materially stronger if one or two such methods were evaluated on a shared public benchmark (for example, ZJU-MoCap or monocular video data) with the proposed method adapted to that setting; otherwise the comparative claims should be explicitly scoped to the PBR baseline.
  3. [Sec. 4.3, Table 1 (SH row)] The ZH-versus-SH ablation conflates the representation choice with the rotation strategy, because the SH baseline is explicitly unrotated ('SH is not rotated as discussed in Sec. 3.2.1'); its degraded appearance on the hands and arms in Fig. 4 could therefore be due to the absence of pose-dependent transfer rather than to the SH basis itself. Since efficient rotation is precisely the motivation for ZH, the comparison supports the full design but does not isolate the basis choice. A rotated SH baseline at reduced order (e.g., third or fourth order, using Wigner-D matrices per Gaussian or per body part) would separate the two effects and substantiate the paper's second contribution.
  4. [Table 1, Sec. 4.3] The quantitative differences between ablations are small: Ours versus w.o. deferred differ by -0.07 dB PSNR and -0.0015 LPIPS on training motion, and Ours versus Mesh normal by 0.05 dB PSNR, while no error bars, per-subject breakdown, or repeated runs are reported. With only three quantitative subjects, the individual component claims (deferred shading, Gaussian normals) are supported at a level that might lie within run-to-run variation. The paper's explanation of the PSNR/SSIM reversal for deferred shading is plausible and the qualitative evidence in Fig. 7 is helpful, but per-subject metrics or multiple seeds would materially solidify the ablation conclusions.
minor comments (6)
  1. [Appendix B] The sentence 'Lalbedo and Lneg_color are the squared losses on negative diffuse color values and albedo values, respectively' contradicts the preceding sentence defining Lalbedo as an L1 loss on albedo values; please clarify which loss applies to which quantity.
  2. [Sec. 3.2.1] The notation z_k in R^{3x3l} for the learned ZH coefficient set is ambiguous; please state the dimensions explicitly (three colored lobes through third order plus three monochromatic lobes through eighth order).
  3. [Sec. 4.2 and Sec. 1] The typos 'PNSR/SSMI' and 'religthtable' should read 'PSNR/SSIM' and 'relightable'.
  4. [Sec. 2.2] The sentence 'Neural relighting with shadow conditioning has also been explored for relightable hands [Chen et al. 2024b; Iwase et al. 2023] exhibit more articulation compared to the human head' is grammatically incomplete; consider rewriting it.
  5. [Eq. (13)] The normalization constant 1/sqrt(2pi) * 2/(3 sigma) of the spherical Gaussian is unusual for a lobe parameterized by arccos(p . q)/sigma; please cite the standard spherical-Gaussian formulation or justify the constant.
  6. [Sec. 4.1] The paper does not state how the quantitative metrics are aggregated across the three subjects or how those subjects were selected; a brief statement would aid reproducibility.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: the learned components are fitted to light-stage data and evaluated on held-out conditions; self-citations are building blocks, not load-bearing evidence.

full rationale

This paper is a learning-based reconstruction system rather than a derivation, so the central equations do not reduce to their inputs by construction. The diffuse radiance transfer (Eqs. 7-9) learns zonal-harmonic coefficients from captured images; the specular model (Eqs. 12-14) is a spherical-Gaussian deferred-shading formulation; and the shadow network (Sec. 3.3) learns a mapping from normalized irradiance (Eq. 15) to a shadow multiplier (Eq. 16), supervised by captured images and tested on held-out poses and point-light configurations. Although normalized irradiance already contains the visibility function, the network output is not algebraically equal to the input, so this is a learned approximation rather than a self-fulfilling prediction. The paper relies on prior work by overlapping authors, especially Saito et al. 2024 and Bagautdinov et al. 2021, but only as architectural building blocks and baseline components; no load-bearing conclusion is justified solely by a self-citation. The abstract's claim of 'superior generalization ability under novel illumination conditions' is partly an evaluation-scope concern, since environment-map relighting is shown qualitatively only and the quantitative numbers come from held-out point lights; that is a correctness and generalization risk, not circularity. Overall, no step was found in which a prediction is identical by construction to a fitted input or to a self-cited uniqueness claim.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim depends on a handful of hand-set representation choices and modeling assumptions about the form of light transport, not on new physical entities.

free parameters (4)
  • Zonal harmonics configuration = n=8, colored to order 3, monochromatic orders 4-8 (51 coefficients per texel)
    Representational choice that determines the 51 vs 113 parameter efficiency claim; not derived from physics.
  • Regularization loss weights = LPIPS 0.1, offset 0.05, mask 0.1, normal_orient 0.1, alpha_sparsity 0.1, bound 0.01, albedo 0.01, neg_color 0.01…
    Hand-tuned hyperparameters controlling training; central to achieving stable reconstruction.
  • Roughness and scale bounds = scale lb=0.0001, ub=0.01; roughness lb=0.01, ub=0.25
    Manual clamping ranges in Lbound (Eq. B.2) used to keep Gaussians well-shaped.
  • Shadow network input samples = N=1 sample per pixel in the reported variant
    Monte Carlo sample count for normalized irradiance; paper shows 1spp has minimal accuracy drop (Table A.1).
assumptions (5)
  • ad hoc to paper Zonal harmonics can express the per-point diffuse radiance transfer as a sum of three ZH lobes aligned with tangent, bitangent, and normal directions (Eq. 9).
    This is a modeling assumption about the form of the light transport function; not derived from first principles. It limits the expressiveness of the diffuse transport to the chosen ZH basis.
  • domain assumption The normalized irradiance computed on a coarse tracked mesh (Eq. 15) contains enough information for a network to predict non-local shadows that generalize to new illuminations.
    The paper relies on this normalization to achieve the claimed generalization; it is tested qualitatively on environment maps but not quantitatively.
  • domain assumption 3D Gaussians associated with texels of a UV map on a tracked template mesh can represent the geometry and appearance of body, face, and hands with the allowed budget.
    The whole representation rests on this; the paper itself notes limited detail in eyes/face/hands compared to specialized models.
  • standard math Standard SH/ZH orthogonality and rotation properties (used in Sec 3.2.1) are correct.
    Background math relied on without proof.
  • domain assumption Light-stage captures provide calibrated known light sources and ground-truth illumination during training.
    Training uses known illumination; this is a strong setup assumption that limits scalability, noted in the paper's limitations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Relightable Full-Body Gaussian Codec Avatars." pith.science (2026). https://pith.science/paper/N4T2KROS

@misc{pith2026250114726,
  author       = {Pith},
  title        = {Pith review of: Relightable Full-Body Gaussian Codec Avatars},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N4T2KROS}},
  note         = {Machine review of arXiv:2501.14726}
}
read the original abstract

We propose Relightable Full-Body Gaussian Codec Avatars, a new approach for modeling relightable full-body avatars with fine-grained details including face and hands. The unique challenge for relighting full-body avatars lies in the large deformations caused by body articulation and the resulting impact on appearance caused by light transport. Changes in body pose can dramatically change the orientation of body surfaces with respect to lights, resulting in both local appearance changes due to changes in local light transport functions, as well as non-local changes due to occlusion between body parts. To address this, we decompose the light transport into local and non-local effects. Local appearance changes are modeled using learnable zonal harmonics for diffuse radiance transfer. Unlike spherical harmonics, zonal harmonics are highly efficient to rotate under articulation. This allows us to learn diffuse radiance transfer in a local coordinate frame, which disentangles the local radiance transfer from the articulation of the body. To account for non-local appearance changes, we introduce a shadow network that predicts shadows given precomputed incoming irradiance on a base mesh. This facilitates the learning of non-local shadowing between the body parts. Finally, we use a deferred shading approach to model specular radiance transfer and better capture reflections and highlights such as eye glints. We demonstrate that our approach successfully models both the local and non-local light transport required for relightable full-body avatars, with a superior generalization ability under novel illumination conditions and unseen poses.

Figures

Figures reproduced from arXiv: 2501.14726 by the authors.

Figure 1
Figure 1. Relightable Full Body Gaussian Codec Avatars. We present the first approach that enables reconstruction, relighting and expressive animation of full￾body avatars including body, face, and hands. Our approach combines learned, orientation-dependent diffuse radiance transport and deferred-shading-based specular radiance transport to enable complex light transport such as global illumination for fully articulated human… view at source ↗
Figure 2
Figure 2. Overview of our approach. Given a body latent code l𝑏 and a face latent code l𝑓 computed by a keypoint encoder and canonicalized viewing directions 𝜔ˆ𝑜 as input, we decode the geometry parameters of 3D Gaussians {R𝑘, s𝑘, t𝑘, 𝑜𝑘 } (Sec. 3.1), and appearance parameters consisting of light transport coefficients {z 𝑐 𝑘 , z𝑚 𝑘 }, normals {n𝑘 }, roughness {𝜎𝑘 }, and specular visibility {𝑣𝑘 } (Sec. 3.2). We integrate the … view at source ↗
Figure 6
Figure 6. We employ three subjects for qualitative and quantitative [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figures from the paper (4 more)
Figure 3
Figure 3. Figure 3: Our appearance model vs. PBR appearance model. The PBR appearance model fails to capture subsurface scattering effects for skins and translucent structures such as hairs. It also produces a darker appearance for concave structures such as ears by omitting global illumi…
Figure 4
Figure 4. Figure 4: ZH vs. SH for diffuse light transport. Note the incorrect shading on the right arm in the SH variant. (a) GT (b) Ours (w. shadow) (c) w.o. shadow [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Qualitative results shadow networks. The learned light transport is not sufficient to capture the shadowing effects caused by body articulation without the help of the shadow network [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 9
Figure 9. Figure 9: Relighting result on unseen motion. We show environment-map-based relighting on the left two columns and point-light-based relighting on the right two columns. ACM Trans. Graph., Vol. 1, No. 1, Article . Publication date: January 2025 [PITH_FULL_IMAGE:figures/full_fig…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

3 extracted references · 2 canonical work pages

  1. [2022]

    InConference on Computer Vision and Pattern Recognition (CVPR)

    High-Resolution Image Synthesis With Latent Diffusion Models. InConference on Computer Vision and Pattern Recognition (CVPR) . Boxiang Rong, Artur Grigorev, Wenbo Wang, Michael J. Black, Bernhard Thomaszewski, Christina Tsalicoglou, and Otmar Hilliges. 2024. Gaussian Garments: Reconstructing Simulation-Ready Clothing with Photorealistic Appearance from Mu...

  2. [2023]

    In Advances in Neural Information Processing Systems (NeurIPS)

    PrimDiffusion: Volumetric Primitives Diffusion for 3D Human Generation. In Advances in Neural Information Processing Systems (NeurIPS) . Zhaoxi Chen and Ziwei Liu. 2022. Relighting4D: Neural Relightable Human from Videos. In European Conference on Computer Vision (ECCV) . Zhaoxi Chen, Gyeongsik Moon, Kaiwen Guo, Chen Cao, Stanislav Pidhorskyi, Tomas Simon...

  3. [2024]

    SwitchLight: Co-design of Physics-driven Architecture and Pre-training Framework for Human Portrait Relighting

    SwitchLight: Co-design of Physics-driven Architecture and Pre-training Framework for Human Portrait Relighting. arXiv preprint arXiv:2402.18848 (2024). Gengyan Li, Abhimitra Meka, Franziska Mueller, Marcel C Buehler, Otmar Hilliges, and Thabo Beeler. 2022a. EyeNeRF: a hybrid representation for photorealistic synthesis, animation and relighting of human ey...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.