VI3NR: Variance Informed Initialization for Implicit Neural Representations

Chamin Hewa Koneputugodage; Sameera Ramasinghe; Stephen Gould; Yizhak Ben-Shabat

REVIEW 3 major objections 5 minor 33 references

VI3NR: Variance Informed Initialization for Implicit Neural Representations

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read One variance rule keeps implicit neural nets stable for any activation

desk verdict Useful generalization of SIREN/Kumar for arbitrary INR activations, but the backward-pass variance condition is an unstated mean-field approximation and the 'theory matches task performance' claim is softer than advertised. read the letter →

arxiv 2504.19270 v1 pith:A5F5RHIN submitted 2025-04-27 cs.CV

Chamin Hewa Koneputugodage , Yizhak Ben-Shabat , Sameera Ramasinghe , Stephen Gould This is my paper · ORCID

classification cs.CV

keywords implicitneuralrepresentationsweightinitializationvariancepropagationpreactivationactivationfunctionsGaussiansincMonteCarloestimation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Implicit neural representations (INRs) encode images, audio, and shapes as functions learned by small fully connected networks, and their performance depends heavily on how the weights are initialized. The paper claims that for any activation function there is a single free parameter—the preactivation variance—that can be tuned so that both the forward signal variance and the backward gradient variance stay constant across layers. It derives explicit formulas for the required weight variance and identifies the preactivation variance that satisfies both the forward and backward conditions. Because the activation statistics in those formulas are hard to compute in closed form, the paper shows they can be replaced by fast Monte Carlo estimates with little loss. If correct, the method turns a per-activation initialization design problem into a one-parameter choice, and experimentally it improves reconstruction for Gaussian and sinc activations in images, audio, and 3D surfaces.

What carries the argument

The load-bearing object is the variance-matching identity at Eq. (4): choosing $\sigma^2(W_i)=\sigma_p^2/(M_i(\mu^2(x_i)+\sigma^2(x_i)))$ makes the preactivation variance $\sigma_p^2$ survive every layer. The companion condition Eq. (13) imposes the same invariance on the backward pass, and because $\sigma_p$ appears in both, it acts as a free parameter that lets the two constraints be satisfied simultaneously rather than averaged. Activation statistics are computed either analytically or by Monte Carlo sampling of $N(0,\sigma_p^2)$; for Gaussian activations the needed statistic reduces to the ratio $\sigma_a/\sigma_p$.

What would settle it

On a 100-layer untrained MLP with weights set by Eq. (4) and $\sigma_p$ chosen from Eq. (13), measure the ratio $\mathrm{Var}(\partial L/\partial z_{i-1})/\mathrm{Var}(\partial L/\partial z_i)$ layer by layer; if the ratio drifts away from 1 with depth, or if the empirical correlation between $\partial L/\partial [z_i]_k$ and $f'([z_{i-1}]_j)$ is non-negligible, the backward condition does not deliver what the derivation claims.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes a variance-preserving initialization scheme that works for any activation. For layers with zero-mean i.i.d. weights, the preactivations converge to a Gaussian as width grows (Proposition 3.1), so preserving distributions reduces to preserving variances. The paper sets the weight variance at every layer to $\sigma^2(W_i)=\sigma_p^2/(M_i(\mu^2(x_i)+\sigma^2(x_i)))$, with expectations taken over $z\sim N(0,\sigma_p^2)$, so that each layer's preactivations keep variance $\sigma_p^2$. It then derives a backward-pass condition, $\sigma_p^2 (M_{i+1}/M_i)(\mu^2(f'(z_i))+\sigma^2(f'(z_i)))/(\mu^2(x_i)+\sigma^2(x_i))=1$, that keeps gradient variance constant as well, and solves for $\sigma_p$ by grid search since the condition depends on $\sigma_p$ itself. Previous initialization recipes emerge as special cases. For Gaussian and sinc activations the paper reports that this initialization reduces forward and backward variance drift in controlled deep-network tests and improves image, audio, and 3D surface reconstruction over random initialization.

Load-bearing premise

The backward-pass derivation assumes that at initialization the gradient arriving at a layer, the weight matrix, and the activation derivative are uncorrelated when variances multiply; in reality the gradient is built from those same weights and activation derivatives, so this independence is not guaranteed.

Editorial extensions

If this is right

Any new INR activation can be used with a principled initialization by estimating two scalar statistics of $f(z)$ and $f'(z)$ under a Gaussian; no per-activation mathematical derivation is required.
Deep coordinate networks with Gaussian, sinc, or other nonstandard activations should no longer suffer vanishing or exploding gradients at initialization when $\sigma_p$ is chosen from Eq. (13).
The initialization covers earlier schemes: the sine-activation recipe and the ReLU/tanh-style recipes correspond to particular choices of $\sigma_p$, so existing results are recovered rather than contradicted.
Because the best $\sigma_p$ also depends on the task, the method converts the initialization question into a small, cheap hyperparameter search that can be done once per task.
For Gaussian activations the theory predicts a linear relationship between the activation scale $\sigma_a$ and $\sigma_p$; the paper observes this trend in reconstruction performance, suggesting the same relationship can guide future activation-parameter choices.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Extension: the same ratio-based analysis should transfer to other radial basis activations, such as inverse multiquadrics, where the Gaussian derivation's dependence on $\sigma_a/\sigma_p$ is likely to recur.
Testable extension: if the backward independence assumption is violated, Eq. (13)'s predicted $\sigma_p$ should still suppress gradient drift for deep tanh and sigmoid networks; measuring layer-by-layer gradient variance on a 100-layer net would isolate where the mean-field approximation breaks.
Implicit consequence: the task-dependence of the optimal $\sigma_p$ suggests that an input-statistics-aware or loss-aware predictor of $\sigma_p$ could replace the line search in future work.
Unstated corollary: because the forward condition depends only on second-order statistics, the method may interact with normalization layers that alter those statistics, requiring $\sigma_p$ to be recomputed for the modified network.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know about this paper. First, it delivers a genuinely general forward-pass initialization for coordinate MLPs, extending SIREN and Kumar to arbitrary activations while adding a controllable preactivation variance sigma_p. Second, the backward-pass variance condition, which is the theoretical heart of the paper's 'any activation' claim, rests on an unstated independence assumption that is not exact. The paper deserves a serious referee, but it needs a major revision.

What is new and good: The forward-pass formula (Eq. 4) is clean: set weight variance to keep preactivation variance at a chosen sigma_p, and compute the activation moments analytically or with cheap Monte Carlo. The paper correctly recovers Xavier and Kaiming as special cases, and the practical recipe works. The experiments show large PSNR/IoU gains over random initialization for Gaussian and sinc activations across images, SDFs, and audio. The MC estimator for activation statistics is accurate with 10K samples. The demonstration that the theoretically optimal sigma_p becomes more predictive as network depth increases (Fig. 3) is an honest and informative empirical finding.

Soft spots: The backward variance derivation (Eq. 10, supplement Eq. 20) factorizes Var(∂L/∂z_{i-1}) as a product of variances of the gradient, weight, and activation derivative. That factorization requires these three to be independent. They are not: ∂L/∂z_{i,k} depends on W_{kj} through the forward pass, and the activation derivative f'(z_{i-1,j}) is coupled through the same weight. The paper never flags this as a mean-field approximation. In the large-width limit the dependence may be asymptotically negligible, but the paper presents it as an exact derivation. A referee should ask for either a rigorous asymptotic statement or an explicit approximation with supporting numerics. There is also a smaller issue in Proposition 3.1: the products p_{kl} are not independent because they share x_i; the CLT result survives under a conditional argument, but the proof as written has a gap. Finally, the 'theory matches task performance' claim is overstated. The theoretical slope from Eq. (13) and the task-optimal slope differ (Fig. 2), and the final numbers rely on a task-specific line search over sigma_p. The authors acknowledge this, and the depth-dependent trend is a nice nuance, but the headline should be 'theory guides' rather than 'theory matches.'

Who is this for? Researchers working on INR activations, especially Gaussian and sinc, or on initialization theory. It is a solid conditional contribution, not a breakthrough.

Recommendation: send to peer review. Ask for major revision to address the independence assumption, fix the CLT proof, and soften the 'theory matches' claim.

Referee Report

3 major / 5 minor

Summary. The paper proposes VI3NR, a weight initialization scheme for implicit neural representations (INRs). The method sets layer weight variances according to Eq. (4) so that preactivation variance is preserved across layers at a user-chosen value σ_p^2, with activation statistics computed either analytically or by Monte Carlo estimation. The authors further derive a backward-pass condition, Eq. (13), intended to keep gradient variance stable across layers, and argue that this generalizes Xavier and Kaiming initialization and applies to any activation function. Experiments on image, audio, and 3D shape reconstruction show that the initialization improves performance over random-normal baselines for Gaussian and sinc activations, with an additional claim that the theory of the initialization matches task performance, particularly for Gaussian INRs.

Significance. If fully correct, this would be a useful and practical contribution: a general initialization recipe for coordinate-MLPs that removes much of the activation-specific trial-and-error currently needed. The forward-pass variance analysis is clean, the Monte Carlo procedure for activation statistics is simple and efficient, and the controlled experiments (Tabs. 2–3) provide a useful comparison of initialization methods. The empirical gains on image, audio, and SDF reconstruction over random initialization are encouraging. However, the backward-pass derivation rests on an unstated independence/mean-field assumption that is not generally valid for nonlinear activations, and the claimed quantitative match between theory and task performance is only approximate, with the reported results relying on an additional task-specific line search. These issues undermine the strength of the 'any activation' and 'theory matches task' claims as currently stated.

major comments (3)

[Supplementary §6.1, Eqs. (17)–(21)] The derivation of the backward variance condition treats ∂L/∂z_i, [W_i]_kj, and f'([z_{i-1}]_j) as mutually independent when passing from Eq. (17) to Eq. (20). This independence does not hold for nonlinear activations: ∂L/∂z_i contains a factor f'(z_i) with z_i = W_i x_i, so ∂L/∂z_i depends on the same weight matrix W_i that appears explicitly in Eq. (8), and x_i = f(z_{i-1}) couples f'([z_{i-1}]_j) to the activations appearing in ∂L/∂z_i. The factorization in Eq. (20) therefore neglects higher-order moments of W_i and is at best a mean-field approximation. Because Eq. (13) is derived from this factorization, the claim that the initialization stabilizes gradient variance for 'any activation' is not established. The authors should state this assumption explicitly, justify its use in the mean-field regime, or provide a direct empirical validation of Eq. (10) for the activations used. Table 3, where Eb remains 20–36% even after selecting σ_p via Eq. (13), suggests the approximation is not highly accurate for Gaussian and sinc activations.
[Sec. 4.2, Fig. 2 and Sec. 3.5] The paper claims that the theory of the initialization matches task performance (abstract and Sec. 4.2). However, in Fig. 2 the theoretical slope of the backward condition is σ_r = σ_a/σ_p ≈ 0.66, while the image-reconstruction optimum follows σ_r ≈ 0.33 and the SDF optimum is much wider and offset. Since the main experiments (Tabs. 5–7) use 8-layer networks and perform an additional task-specific line search over σ_p (Sec. 3.5), the reported gains do not validate Eq. (13) as a predictor of the optimal σ_p for shallow INRs; they validate the combination of the initialization family plus task-level tuning. The qualitative trend in Fig. 3 for deeper networks is supportive, but the wording should be adjusted to state that Eq. (13) provides a good prior for deep networks and a useful starting point for shallow ones.
[Table 3 and Sec. 3.4] The text says that when σ_p is selected from Eq. (13) the method achieves low forward and backward errors for INR activations. The reported Eb values (sine 20.4, Gaussian 35.6, sinc 21.4, wavelet 22.5) are much lower than the σ_p = 1 case (100) but are not low in absolute terms; a 20–36% SMAPE means gradient variance still changes substantially over 100 layers. This weakens the 'stable gradient variance' claim even as an approximation and needs to be discussed, especially in light of the independence issue raised above.

minor comments (5)

[Sec. 3.3, Proposition 3.1] The statement that the elements [z_i]_k are independent is not correct unconditionally, since they share the same random vector x_i; they are conditionally independent given x_i. This does not affect the marginal variance computation, but the proof and wording should be corrected.
[Sec. 3.4, Eq. (10)] The notation switches from f'([z_{i-1}]_j) in Eq. (8) to f'(z_i) in Eq. (10). The two are equal only when the preactivation distribution is the same at every layer, which is the desired invariant; this should be stated explicitly before the switch.
[Sec. 3.5] The 'small local line search' is not specified in detail. It would strengthen reproducibility to state the range and number of σ_p values tested.
[Table 1, bottom row] For ReLU, the note says all σ_p satisfy Eq. (13), but the grid search then reports σ_p = 1. It would be clearer to explain how σ_p = 1 was selected in that case.
[Abstract] The abstract's phrase 'applies to any activation function' is stronger than the derivations support; the authors should qualify this to activations for which the mean-field assumptions hold and whose statistics can be estimated reliably.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Eq. (4) is a construction from a chosen target variance, not a fitted prediction, and the reported task gains rest on openly disclosed hyperparameter tuning rather than on a derivation that reduces to its own inputs.

full rationale

The central derivation is self-contained. Proposition 3.1 derives the CLT form of the preactivation variance, and Eq. (4) simply solves for the weight variance needed to realize a chosen target preactivation variance sigma_p^2; this is a design equation, not a prediction secretly fitted from the evaluation data. The backward-pass condition in Eqs. (10)-(13) is a separate derivation whose independence assumptions may be mathematically fragile, but that is a correctness/robustness concern, not circularity. Eq. (13) is solved by a numerical grid search over sigma_p using only activation statistics and the theoretical condition itself, not task performance. The paper is explicit that the final sigma_p used in the application experiments is further tuned by a 'small local line search' (Sec. 3.5) and reports this as hyperparameter tuning (Sec. 4), separately from the theoretical value from Eq. (13); for example, Table 8 reports sigma_p=0.08 from Eq. (13) versus sigma_p=0.15 by grid search. Thus the reported reconstruction improvements are empirical results after disclosed tuning, not deductions from the theory presented as predictions. Self-citations (e.g., DiGS [5]) appear only in related-work context and are not load-bearing. No equation in the paper reduces by definition to its own output, so no circular step is present.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The derivation rests on standard CLT and mean-field assumptions plus a poorly justified independence assumption in the backward pass. It also carries two fitted hyperparameters (sigma_p and the activation width or frequency). These are counted as free parameters because final performance depends on their task-specific values.

free parameters (3)

sigma_p (preactivation standard deviation) = varies; e.g., 0.08 theoretical vs 0.15 line-search optimum for Gaussian image INR (Table 8)
Free hyperparameter controlling preactivation variance. Chosen to satisfy Eq. (13) and then refined by task-specific line search, so it is partly fitted to data.
sigma_a (Gaussian activation width) = varies; grid-searched per task (e.g., 0.05 in Table 8)
Grid-searched per task; the initialization formula and task performance depend on this parameter.
a (sinc frequency parameter) = varies; grid-searched per task
Grid-searched per task for sinc activations; task performance depends on it.

assumptions (4)

standard math Lindeberg CLT applies to the per-layer weighted sums, making preactivations Gaussian with the stated variance (Prop. 3.1).
Requires independence and moment conditions on weight-input products; standard in initialization theory.
ad hoc to paper Backward-pass gradient products partial L / partial z_i, W_i, and f'(z_{i-1}) are mutually independent so their variance factorizes (supp Eq. 20).
Needed for Eq. (11) and Eq. (13); not proved and generally false because partial L / partial z_i depends on f'(z_i), hence on W_i.
domain assumption Input elements x0 are independent and uniformly distributed over [-1,1], and biases are approximately zero.
Used to set the first-layer weight variance in supp Sec. 6.2; holds for normalized coordinates but not for arbitrary regression inputs.
standard math Neurons in a layer are wide enough (Mi large) that finite-width corrections to the CLT are negligible.
Needed for the Gaussian preactivation assumption; experiments use width 128 or 256, where corrections are not quantified.

how reviews work

0 comments

Cite this review

Pith. "Pith review of VI3NR: Variance Informed Initialization for Implicit Neural Representations." pith.science (2026). https://pith.science/paper/A5F5RHIN

@misc{pith2026250419270,
  author       = {Pith},
  title        = {Pith review of: VI3NR: Variance Informed Initialization for Implicit Neural Representations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/A5F5RHIN}},
  note         = {Machine review of arXiv:2504.19270}
}

read the original abstract

Implicit Neural Representations (INRs) are a versatile and powerful tool for encoding various forms of data, including images, videos, sound, and 3D shapes. A critical factor in the success of INRs is the initialization of the network, which can significantly impact the convergence and accuracy of the learned model. Unfortunately, commonly used neural network initializations are not widely applicable for many activation functions, especially those used by INRs. In this paper, we improve upon previous initialization methods by deriving an initialization that has stable variance across layers, and applies to any activation function. We show that this generalizes many previous initialization methods, and has even better stability for well studied activations. We also show that our initialization leads to improved results with INR activation functions in multiple signal modalities. Our approach is particularly effective for Gaussian INRs, where we demonstrate that the theory of our initialization matches with task performance in multiple experiments, allowing us to achieve improvements in image, audio, and 3D surface reconstruction.

Figures

Figures reproduced from arXiv: 2504.19270 by the authors.

**Figure 1.** Notation diagram. For clarity, we illustrate a general MLP based INR architecture (layers 0, i and n shown). Layers are of the form ϕi(xi) = f(Wixi) where f is the activation function and Wi are the weights, the preactivations are z0, ..., zn−1, the postactivations are x1, ..., xn and the input and output of the network are x0 and zn. the activation function the choice of preactivation distribution matters (which w… view at source ↗

**Figure 2.** Heatmaps of the condition in Eq. (13) and task performance when using our initialization with Gaussian activations. In each heatmap the Gaussian activation function variance (σ 2 a) and preactivation variance (σ 2 p) are varied, revealing linear trends. 2 Layers 16 Layers 32 Layers 4 Layers [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

**Figure 3.** Parameter grid search for images with differing number of layers. We plot PSNR as a function of the Gaussian activation standard deviation (σa) and the weight distribution standard deviation (σp). As the number of layers increases, the slope converges towards the theoretical slope, and the linear trend sharpens. This matches our hypothesis that the backward condition Eq. (13) is more important as the number of la… view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Image comparison. Left to right: random normal init, our MC init, our init [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]

**Figure 5.** Figure 5: Performance gap vs. σa. As σa decreases, performance drops for both inits, but a significant gap remains. 10. Audio Reconstruction Implementation details The audio reconstruction results presented in the main paper differed from the image and SDF reconstruction setu…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 27 canonical work pages

[1]

Ash and C.A

R.B. Ash and C.A. Doleans-Dade. Probability and Measure Theory. Elsevier Science, 2000. 3

work page 2000
[2]

SAL: Sign Agnostic Learning of shapes from raw data

Matan Atzmon and Yaron Lipman. SAL: Sign Agnostic Learning of shapes from raw data. In CVPR, pages 2565– 2574, 2020. 1, 2

work page 2020
[3]

SALD: Sign Agnostic Learning with Derivatives

Matan Atzmon and Yaron Lipman. SALD: Sign Agnostic Learning with Derivatives. In ICLR, 2021. 1

work page 2021
[4]

Mip-nerf: A multiscale representation for anti-aliasing neu- ral radiance fields

Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neu- ral radiance fields. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 5855–5864,
[5]

Digs: Divergence guided shape implicit neu- ral representation for unoriented point clouds

Yizhak Ben-Shabat, Chamin Hewa Koneputugodage, and Stephen Gould. Digs: Divergence guided shape implicit neu- ral representation for unoriented point clouds. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19323–19332, 2022. 1, 2

work page 2022
[6]

Kodak Lossless True Color Image Suite

Eastman Kodak Company. Kodak Lossless True Color Image Suite. http://r0k.us/graphics/kodak/ ,

work page
[7]

Understanding the diffi- culty of training deep feedforward neural networks

Xavier Glorot and Yoshua Bengio. Understanding the diffi- culty of training deep feedforward neural networks. In Inter- national Conference on Artificial Intelligence and Statistics,

work page
[8]

Implicit Geometric Regularization for learn- ing shapes

Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit Geometric Regularization for learn- ing shapes. In ICML, pages 3789–3799. PMLR, 2020. 1, 2

work page 2020

Show all 33 references

[9]

Zhang, Shaoqing Ren, and Jian Sun

Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Delv- ing deep into rectifiers: Surpassing human-level perfor- mance on imagenet classification. 2015 IEEE International Conference on Computer Vision (ICCV), pages 1026–1034,

2015
[10]

On weight initialization in deep neural networks, 2017

Siddharth Krishna Kumar. On weight initialization in deep neural networks, 2017. 2, 3, 4, 5

2017
[11]

Efficient backprop

Yann LeCun, L ´eon Bottou, Genevieve B Orr, and Klaus- Robert M ¨uller. Efficient backprop. In Neural networks: Tricks of the trade, pages 9–50. Springer, 2002. 1, 2

2002
[12]

Bacon: Band-limited coordinate net- works for multiscale scene representation

David B Lindell, Dave Van Veen, Jeong Joon Park, and Gordon Wetzstein. Bacon: Band-limited coordinate net- works for multiscale scene representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16252–16262, 2022. 2, 7

2022
[13]

Finer: Flexi- ble spectral-bias tuning in implicit neural representation by variable-periodic activation functions

Zhen Liu, Hao Zhu, Qi Zhang, Jingde Fu, Weibing Deng, Zhan Ma, Yanwen Guo, and Xun Cao. Finer: Flexi- ble spectral-bias tuning in implicit neural representation by variable-periodic activation functions. In Proceedings of the IEEE/CVF Computer Vision and Pattern Recognition Co...

2024
[14]

Occupancy networks: Learning 3d reconstruction in function space

Lars Mescheder, Michael Oechsle, Michael Niemeyer, Se- bastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4460–4470, 2019. 1

2019
[15]

Srinivasan, Matthew Tancik, Jonathan T

Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view syn- thesis. In European Conference on Computer Vision, 2020. 2

2020
[16]

Nerf: Representing scenes as neural radiance fields for view syn- thesis

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view syn- thesis. Communications of the ACM , 65(1):99–106, 2021. 1

2021
[17]

Deepsdf: Learning con- tinuous signed distance functions for shape representation

Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning con- tinuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 165–174, 2019. 1, 2

2019
[18]

Pytorch: An imperative style, high-performance deep learning library

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zem- ing Lin, Natalia Gimelshein, Luca Antiga, Alban Desmai- son, Andreas Kopf, Edward Yang, Zachary DeVito, Mar- tin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steine...

2019
[19]

Beyond periodicity: Towards a unifying framework for activations in coordinate- mlps

Sameera Ramasinghe and Simon Lucey. Beyond periodicity: Towards a unifying framework for activations in coordinate- mlps. In European Conference on Computer Vision , pages 142–158. Springer, 2022. 1, 2, 6, 7, 8

2022
[20]

Pifu: Pixel-aligned implicit function for high-resolution clothed human digitiza- tion

Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Mor- ishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitiza- tion. In Proceedings of the IEEE/CVF international confer- ence on computer vision, pages 2304–2314, 2019. 1

2019
[21]

Wire: Wavelet implicit neural representations

Vishwanath Saragadam, Daniel LeJeune, Jasper Tan, Guha Balakrishnan, Ashok Veeraraghavan, and Richard G Bara- niuk. Wire: Wavelet implicit neural representations. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18507–18516, 2023. 1, 2

2023
[22]

A sam- pling theory perspective on activations for implicit neural representations

Hemanth Saratchandran, Sameera Ramasinghe, Violetta Shevchenko, Alexander Long, and Simon Lucey. A sam- pling theory perspective on activations for implicit neural representations. arXiv preprint arXiv:2402.05427, 2024. 2, 6

2024 arXiv
[23]

Implicit neural representa- tions with periodic activation functions

Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representa- tions with periodic activation functions. Advances in neural information processing systems, 33:7462–7473, 2020. 1, 2, 3, 6, 7

2020
[24]

Fourier features let networks learn high frequency functions in low dimen- sional domains

Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ra- mamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimen- sional domains. Advances in Neural Information ...

2020
[25]

State of the art on neural rendering

Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin- Brualla, Tomas Simon, Jason Saragih, Matthias Nießner, 9 et al. State of the art on neural rendering. In Computer Graphics Forum , pages 701–727. Wiley Online Library,
[26]

Geometry-consistent neural shape representation with im- plicit displacement fields

Wang Yifan, Lukas Rahmann, and Olga Sorkine-hornung. Geometry-consistent neural shape representation with im- plicit displacement fields. In International Conference on Learning Representations, 2022. 2 10 VI3NR: Variance Informed Initialization for Implicit Neural Representat...

2022
[27]

Derivations 6.1. Derivation for the backward pass Given ∂L ∂[zi−1]j = NiX k=1 ∂L ∂[zi]k [Wi]kjf′([zi−1]j), (17) we have that µ ∂L ∂zi−1 = NnX k=1 E [[Wi]kj] E ∂L ∂[zi]k f′([zi−1]j) (18) = 0 (19) and σ2 ∂L ∂zi−1 = NiX k=1 Var ∂L ∂[zi]k Var[[Wi]kj] E [f′([zi−1]j)]2 + Var[f′([zi−...
[28]

Thus by Eq. (4) we initialize our first layer weights with variance σ2(W0) = σ2 p M0 (µ2(x0) +σ2(x0)) (24) = 3σ2 p M0 (25) which is equivalent to byU([−c,c ]) where c =σp s 3 M0 (µ2(x0) +σ2(x0)) (26) =σp r 9 M0 . (27) 6.3. Analytical Expectations for Gaussians Let us assume th...
[29]

In their derivation they use sin π 2x in order to only consider the monotonic region of sine

SIREN’s initialization Note that SIREN [23] also uses sine activations and specif- ically uses σp = 1 . In their derivation they use sin π 2x in order to only consider the monotonic region of sine. Our method also givesσ2(Wi) = 2 Mi with that activation func- tion. In the code...
[30]

Comparison to Xavier and Kaiming init. Our initialization for Gaussians is σ2(Wi) = σ2 p Mi (µ2(xi) +σ2(xi)) (56) = σ2 p p σ2r + 2 Miσr (57) whereσr = σa σp , while Xavier initialization is of the form (for middle layers) σ2(Wi) = 1 Mi (58) and Kaiming initialization is of the...
[31]

We compare image reconstruction with Gaussian activation with the three different types of initializations in Fig

Image Comparison. We compare image reconstruction with Gaussian activation with the three different types of initializations in Fig. 4. 2 Figure 4. Image comparison. Left to right: random normal init, our MC init, our init. Figure 5. Performance gap vs.σa. Asσa decreases, perf...
[32]

Specifically, the network ar- chitecture used three hidden layers, each containing 256 el- ements, and the bias terms were initialized identically to the weights

Audio Reconstruction Implementation de- tails The audio reconstruction results presented in the main pa- per differed from the image and SDF reconstruction se- tups in several key aspects. Specifically, the network ar- chitecture used three hidden layers, each containing 256 e...
[33]

Improvement gap dependence on activa- tion function parameters We give results for smallσa in Fig. 5 Left. The results show that the proposed initialization outperforms random init for smallerσa values while a degradation in performance is ob- served for both. 3

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

Ash and C.A

R.B. Ash and C.A. Doleans-Dade. Probability and Measure Theory. Elsevier Science, 2000. 3

work page 2000

[2] [2]

SAL: Sign Agnostic Learning of shapes from raw data

Matan Atzmon and Yaron Lipman. SAL: Sign Agnostic Learning of shapes from raw data. In CVPR, pages 2565– 2574, 2020. 1, 2

work page 2020

[3] [3]

SALD: Sign Agnostic Learning with Derivatives

Matan Atzmon and Yaron Lipman. SALD: Sign Agnostic Learning with Derivatives. In ICLR, 2021. 1

work page 2021

[4] [4]

Mip-nerf: A multiscale representation for anti-aliasing neu- ral radiance fields

Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neu- ral radiance fields. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 5855–5864,

[5] [5]

Digs: Divergence guided shape implicit neu- ral representation for unoriented point clouds

Yizhak Ben-Shabat, Chamin Hewa Koneputugodage, and Stephen Gould. Digs: Divergence guided shape implicit neu- ral representation for unoriented point clouds. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19323–19332, 2022. 1, 2

work page 2022

[6] [6]

Kodak Lossless True Color Image Suite

Eastman Kodak Company. Kodak Lossless True Color Image Suite. http://r0k.us/graphics/kodak/ ,

work page

[7] [7]

Understanding the diffi- culty of training deep feedforward neural networks

Xavier Glorot and Yoshua Bengio. Understanding the diffi- culty of training deep feedforward neural networks. In Inter- national Conference on Artificial Intelligence and Statistics,

work page

[8] [8]

Implicit Geometric Regularization for learn- ing shapes

Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit Geometric Regularization for learn- ing shapes. In ICML, pages 3789–3799. PMLR, 2020. 1, 2

work page 2020