{"id":"f6ba3ff2-c47f-40c7-916e-f21912a656cd","arxiv_id":"2501.04387","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":1.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"A perspective and review arguing that activation function choice in RBMs controls interaction order, learning dynamics, and biological applicability.","lead":"This paper reviews how the type of activation function in Restricted Boltzmann Machines (RBMs) shapes their theoretical properties and biological applications. It synthesizes results on non-sigmoid activations, the RBM-Hopfield duality, and applications to neural, protein, and immune data, including a small illustrative toy simulation.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim is false: a non-linear activation RBM with weights (1,1,-2) shows zero third-order interactions, contradicting 'no RBM ... for any choice of connections w'.","rationale":"The reader's weakest_assumption focused on whether Eq. (2) is accurately reproduced from Bulso & Roudi (2021), noting that the paper does not derive it independently. I agree that Eq. (2) is central, but the more severe problem is that the paper draws an invalid inference from it: the statement that no non-linear activation can express only terms up to a given order for any choice of w is falsified by an explicit counterexample that uses the paper's own definitions. The counterexample is a valid RBM (hidden unit with a sigmoidal tanh activation, three binary visible units, weights summing to zero), and direct computation of the third-order interaction yields zero, so the marginal has degree at most two. This is not a niche pathology: any hidden distribution with an even cumulant generating function and weights summing to zero on a subset will cancel the top-order interaction for that subset. The original reference [2] may have intended a generic claim about random weights, but the paper as written makes an absolute claim. Because this theoretical result is used to motivate the role of activation functions in biological applications and to distinguish linear from non-linear activations, the error undermines a central pillar of the review. The toy simulation in Fig. 3 is not the main issue; the theoretical foundation needs correction. If the authors revise the claim to 'generically' or 'for almost all choices of w', the paper could be acceptable, but in its current form the central theoretical statement is false.","tokens_in":9844,"tokens_out":21228,"duration_ms":199889,"concrete_test":"Symbolically verify the counterexample: define a single hidden unit with ρ(z)=(δ(z-1)+δ(z+1))/2 so K(x)=log cosh x, take visible biases zero, and set weights w=(1,1,-2). Compute the third-order interaction for the visible subset {1,2,3} using the exact inclusion-exclusion formula from Eq. (2): I = Σ_{T⊆{1,2,3}} (-1)^{3-|T|} K(Σ_{i∈T} w_i). Check that the result is exactly 0. Additionally, scan the weight space for N=3, M=1 with K(x)=log cosh x and confirm that the equation I_{123}(w)=0 has non-zero solutions (e.g., any weights summing to zero). If the computation confirms cancellation, the paper's unconditional claim is false and must be revised to a generic-statement claim.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's central theoretical claim states that no RBM with a non-linear activation can express only terms up to a given order, for any choice of connections w (Section 'Non-Sigmoid activation function', after Eq. 2). This claim is contradicted by a concrete counterexample using only the paper's own framework. Take N=3 visible units, M=1 hidden unit with distribution ρ(z)=(δ(z-1)+δ(z+1))/2, so the cumulant generating function is K(x)=log cosh x, a non-linear (sigmoidal) activation. Choose weights w=(1,1,-2), i.e., w1=1, w2=1, w3=-2. The log-marginal is log p(v) = log cosh(v1+v2-2v3). Computing the third-order interaction for S={1,2,3} via the inclusion-exclusion formula (equivalent to Eq. 2) gives: I = log cosh(0) - log cosh(2) - log cosh(-1) - log cosh(-1) + log cosh(1) + log cosh(1) + log cosh(-2) - log cosh(0) = 0 - log cosh 2 - log cosh 1 - log cosh 1 + log cosh 1 + log cosh 1 + log cosh 2 - 0 = 0. Thus the third-order interaction vanishes exactly, and since N=3, no interactions of order ≥3 exist. The model therefore expresses only terms up to second order, despite having a non-linear activation, for a specific valid choice of weights. This directly falsifies the claim as stated. The issue is not merely that Eq. (2) is unverified; even accepting Eq. (2), the conclusion drawn from it is too strong. The correct statement is likely that generically (for random weights) non-linear activations induce all interaction orders, but there exist special weight configurations where high-order terms cancel. Because this claim underpins the paper's interpretation of activation functions and the biological applications (e.g., dReLU in proteins), the error is load-bearing for the review's theoretical message.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This perspective reviews how the choice of activation function in Restricted Boltzmann Machines shapes their representational properties and their use in biological applications. After introducing the RBM formalism, the paper presents a theoretical discussion centered on Eq. (2), the interaction expansion of the visible marginal, and uses it to contrast linear hidden activations, which map to pairwise Hopfield-like models and a retrieval/overfitting transition, with non-linear activations, which are claimed to generate all interaction orders. The rest of the review surveys applications to neural recordings, protein-family modeling and design, immunology, and the diffRBM transfer-learning strategy, and closes with open problems on activation-function choice, learning dynamics, and noisy overparameterized learning.","tokens_in":10168,"tokens_out":8044,"duration_ms":76674,"significance":"If the theoretical statements are corrected, this will be a useful and readable synthesis that connects an active theoretical literature on RBMs to concrete biological applications. Its strengths are its clear organization, its broad and mostly accurate citation of the literature, and its informative treatment of the protein and immunology applications, particularly the diffRBM construction. The paper also provides an instructive toy illustration in Fig. 3, although the simulation details are missing. The main weakness is that the central theoretical claim about non-linear activations is overstated to the point of being false as written, and this affects the downstream framing of the protein-modeling discussion. Since the false statement is local and can be repaired by a genericity caveat, I do not regard the error as fatal to the review's overall value.","major_comments":[{"comment":"The sentence 'there is no RBM, endowed with a non-linear activation, able to express only terms up to a given order, for any choice of connections w' is false as stated, and the counterexample comes from the paper's own framework. Take N=3 visible units, M=1 hidden unit, hidden prior ρ(z)=1/2(δ(z-1)+δ(z+1)), so that K(x)=log cosh(x), and take weights w1=1, w2=1, w3=-2. The visible marginal is proportional to exp[log cosh(v1+v2-2v3)], with vi in {0,1}. The third-order interaction for S={1,2,3}, obtained by the inclusion-exclusion expression equivalent to Eq. (2), is Σ_{T⊆S} (-1)^{3-|T|} log cosh(Σ_{i∈T} wi), which evaluates to 0 because the positive singleton terms log cosh(1)+log cosh(1)+log cosh(2) cancel against the negative pair terms log cosh(2)+log cosh(1)+log cosh(1), and the empty and full terms vanish. Since N=3, there are no interactions of order greater than 3, so this RBM expresses only terms up to second order despite having a non-linear activation and a valid weight choice. The correct statement is that non-linear activations generically generate all interaction orders, with possible exact cancellations at special weight configurations. Please revise the claim accordingly and adjust any text that relies on the universal version.","section":"Non-Sigmoid activation function, after Eq. (2)"},{"comment":"The discussion of the double-ReLU potential in the protein section presents the ability to learn high-order correlations as a consequence of the general theoretical claim about non-linear activations. Since that general claim is falsified by the explicit counterexample above, the connection needs to be reformulated: the high-order-correlation behavior of the dReLU RBM in [3] is an empirical/modeling result for trained weights, not a consequence of the universal theorem stated in the theory section. Without this reformulation, the theoretical motivation offered for Eq. (5) is not valid.","section":"Protein Families, around Eq. (5)"}],"minor_comments":[{"comment":"The notation in Eq. (2) is difficult to parse: the summation 'Σ_{j1<...<j_{s-p}=1}' should be written as 'Σ_{1≤j1<...<j_{s-p}≤s}', and the argument of K should be explicitly identified as Σ_{l=1}^{s-p} w_{k_{j_l} μ}. Please also state that Eq. (2) is the standard inclusion-exclusion expansion of K_μ(Σ_i w_{iμ} v_i) so that readers can verify it.","section":"Eq. (2)"},{"comment":"The toy simulation in Fig. 3 is not reproducible as reported: the learning rate, the number of gradient-descent steps, the initialization distribution, and the parameters defining p_LG(v) are not given. Since the figure is used to support a claim about convergence speed across activation functions, these details should be added to the caption or the main text.","section":"Fig. 3"},{"comment":"The subsection title 'Non-Sigmoid activation function' is confusing because the text includes both linear activations and the standard sigmoidal/log-cosh case as examples of non-linear activations. Please clarify whether 'non-sigmoid' refers to activation functions different from the standard binary-unit sigmoid or to all non-linear activations.","section":"Non-Sigmoid activation function (title and text)"},{"comment":"The sentence 'a similar dynamical variant of RBM, the Recurrent Temporal RBM (RTRBM) can be trained efficiently and exactly' is imprecise; standard RTRBM training uses approximate inference and contrastive-divergence-style updates. Please clarify in what sense the training is exact, or soften the claim.","section":"Neural Data, TRBM sentence"},{"comment":"The statement that the RBM-HM duality relates 'supervised learning in HM' to 'unsupervised learning in RBM' could be misread as a claim about learning rules. The formal equivalence is between the visible marginal of a linear-activation RBM and the equilibrium distribution of a Hopfield model; please state this explicitly.","section":"Phase diagram and compositionality"}],"recommendation":"major_revision","confidential_remarks":"The stress-test counterexample is decisive against the central theoretical claim as worded, and I agree with the reader that it is the main load-bearing issue. I recommend major revision rather than rejection because the fix is local: replacing the universal statement by a genericity statement and adjusting the downstream protein-modeling framing. Please also ask the authors to provide full simulation details for Fig. 3 and to clarify the terminology around 'non-sigmoid'."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the bottom line: this is a readable, well-organized review of RBMs and activation functions, but its headline theoretical claim is wrong as stated. The sentence \"there is no RBM, endowed with a non-linear activation, able to express only terms up to a given order, for any choice of connections w\" is falsified by a concrete example using only their own framework.\n\nTake N=3 binary visible units, M=1 hidden unit with ρ(z)=1/2(δ(z−1)+δ(z+1)), so K(x)=log cosh x, and weights w=(1,1,−2). Then log p(v)=log cosh(v1+v2−2v3). The third-order interaction computed by inclusion-exclusion is exactly zero: log cosh 0 − log cosh 2 − 2 log cosh 1 + 2 log cosh 1 + log cosh 2 − log cosh 0 = 0. So this non-linear RBM expresses only up to second-order terms for this w.\n\nThe correct statement, from Bulso and Roudi, is presumably that generically, for random weights, non-linear activations produce all interaction orders. That is interesting and worth saying. But \"for any choice of connections w\" is a different, stronger claim, and it is false. This matters because the paper later uses this claim to motivate the dReLU results in proteins.\n\nCredit where due: the survey of neural, protein, and immunology applications is accurate and useful; the RBM–Hopfield duality discussion is clear; the diffRBM description is helpful. The toy simulation in Fig. 3 is a nice illustration but lacks hyperparameters and code, so it's not reproducible as presented.\n\nSoft spots: the false claim is load-bearing, so the theory section needs a rewrite. The toy figure should either be fully specified or dropped. Also, the section title says \"non-sigmoid\" but the counterexample uses a sigmoid; the authors should be careful about what class of activation functions they address.\n\nWho is this for? People entering RBM research or wanting a map of biological applications. With the claim fixed, it would be a solid perspective.\n\nRecommendation: send to peer review, but require the authors to correct the \"any w\" claim and consider a generic-weights formulation, and to make the toy experiment reproducible.","headline":"A useful RBM review whose central theoretical claim is too strong—there is a concrete counterexample—so it needs a corrected statement before it can be trusted.","tokens_in":10700,"tokens_out":4827,"would_cite":false,"duration_ms":42472,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Activation functions, not just weights, decide which statistical interactions an RBM can express.","keywords":["Restricted Boltzmann Machines","activation functions","Hopfield model","interaction expansion","overfitting","protein sequence modelling","neural data analysis","unsupervised learning"],"falsifier":"Take a small RBM with a non-linear hidden activation, say a single ReLU hidden unit, and compute the coefficients of the visible interaction series exactly; if there is a choice of weights for which all coefficients of order three and higher vanish while the activation remains non-linear, the central impossibility claim is false. Equivalently, train a non-linear RBM on data generated by a pairwise model and check whether the fitted model shows strictly zero higher-order interactions beyond numerical noise; any non-zero higher-order coupling would support the paper's claim, while an exact zero would refute it.","tokens_in":9607,"feed_emoji":"🧠","tokens_out":6134,"duration_ms":53180,"temperature":0.7,"pith_summary":"This review makes a theoretical point about Restricted Boltzmann Machines (RBMs), two-layer stochastic networks used for learning structure in data. The point is that the choice of activation function for the hidden units controls which kinds of statistical interactions among visible units the model can represent: a linear activation confines the visible distribution to pairwise interactions, while any non-linear activation inevitably generates interactions of arbitrarily high order. Because of this, the same mathematical structure that makes linear RBMs equivalent to Hopfield attractor networks also links the Hopfield retrieval transition to overfitting in RBM learning. The authors then use this dichotomy to frame applications to neural, protein, and immunological data, where sigmoid or double-ReLU hidden units are chosen according to the interaction orders the data require. A careful reader should come away with the claim that the activation function is not an implementation detail but the main switch determining an RBM's expressive range.","feed_headline":"Non-linear RBM units always bring higher-order interactions","feed_subtitle":"A linear activation shrinks an RBM to a Hopfield network; anything else opens all interaction orders.","key_machinery":"The load-bearing object is the interaction expansion of the visible marginal, Eq. (2), in which the interaction of order $s$ is written as sums of $K_\\mu$ evaluated at weighted sums of visible units, where $K_\\mu(x)=\\log \\mathbb{E}_{z_\\mu} e^{z_\\mu x}$ is the cumulant generating function of the hidden unit distribution. The paper uses this expansion to derive the impossibility result for non-linear activations, and uses the linear-activation case to map the RBM onto a Hopfield network, whose storage-load transition becomes the overfitting transition in learning. It also functions as the language for comparing activation functions across applications.","core_discovery":"On the authors' account, the visible-unit marginal of an RBM is an exponential family whose interactions are set by the cumulant generating function $K_\\mu(x)=\\log \\mathbb{E}_{z_\\mu}\\exp(z_\\mu x)$ of the hidden units' distribution. Expanding the marginal in powers of the visible variables, every interaction order receives contributions from all hidden units, and the form of $K_\\mu$ decides whether the coefficients beyond a given order can vanish. The central theoretical claim, taken from earlier work, is that no RBM with a non-linear activation can express only terms up to a prescribed order for any choice of weights; the only activation that suppresses all interactions beyond pairwise is the linear one, in which case the RBM is exactly a Hopfield model with stored patterns given by its weights. The same duality lets the Hopfield critical storage load $\\alpha_c = 0.14$ be read as a bound beyond which RBM learning overfits. On the applied side, the paper claims that this is why protein and immunology applications need non-binary units with double ReLU potentials, while neural data applications often stay with binary sigmoid units.","pith_inferences":["Extension not in the paper: if the impossibility result is exact, then sparsity or regularization on weights cannot restore a purely pairwise RBM once a non-linear activation is chosen; the higher-order terms are structural, not an artifact of finite data.","Extension not in the paper: comparing fitted interaction coefficients across activation functions for a fixed dataset would quantify how much of the 'higher-order' signal is genuine data structure versus an artefact of the chosen activation, a testable diagnostic.","Extension not in the paper: the RBM-HM duality might transfer results on Hopfield storage of correlated or biased patterns directly to RBM learning on biased biological datasets, constraining when latent representations are expected to be trustworthy."],"forward_implications":["If the impossibility result holds, then any RBM trained with a ReLU, double-ReLU, or other non-linear hidden activation is, in principle, modelling arbitrarily high-order correlations, even when the data are pairwise; only linear hidden units guarantee a purely pairwise model.","The Hopfield-RBM correspondence implies that increasing the number of hidden units in a linear RBM beyond the analogue of $\\alpha_c$ will degrade generative performance through overfitting, giving a concrete failure mode rather than a mere heuristic.","For protein-sequence modelling, the double-ReLU potential of Eq. (5) is presented as conferring the expressive power needed to capture high-order constraints among amino-acid positions, which is why it outperforms sigmoid units on those data.","The compositional phase found analytically for RBMs with ReLU hidden units provides a mechanism by which visible configurations are represented by a small set of strongly active hidden units, relevant for interpreting latent representations of neural and protein data.","Differential learning (diffRBM) exploits the interpretability of a few added hidden units to isolate the statistical motifs that distinguish immunogenic antigens from generic ones, a corollary of the same latent-space structure."],"supporting_citations":[{"why":"Supplies the interaction expansion (Eq. 2) and the impossibility result for non-linear activations, the paper's central theoretical claim.","marker":"[2]"},{"why":"Establishes that a linear hidden activation maps the RBM to a Hopfield model, the basis of the RBM-HM duality.","marker":"[12]"},{"why":"Provides the Hopfield critical storage load $\\alpha_c = 0.14$ that the paper maps to the RBM overfitting transition.","marker":"[18]"},{"why":"Reviews and extends the RBM-HM duality to general priors, underpinning the phase-diagram discussion.","marker":"[15]"},{"why":"Derives the critical sample size for learning archetypes in the dual language, used in the overfitting discussion.","marker":"[16]"},{"why":"Shows that RBM weights trained on protein sequences recover structural contacts and that double-ReLU hidden units give high expressive power, grounding the protein application.","marker":"[3]"},{"why":"Demonstrates the compositional phase for RBMs with ReLU hidden units, used to interpret biologically relevant latent representations.","marker":"[30]"},{"why":"Introduces diffRBM, the differential learning strategy applied to immunology data in the final application section.","marker":"[41]"}],"fun_headline_variants":["Linear RBM units collapse to Hopfield; nonlinearity unlocks all orders","Only linear RBM units are Hopfield; others boost high-order terms","RBM activation choice: linear = Hopfield storage, nonlinear = full orders","Nonlinear RBM units generate all interaction orders; linear ones don't","In RBMs, linear activation = Hopfield limit; nonlinear = all orders"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole theoretical story about activation functions controlling interaction orders rests on the interaction expansion in Eq. (2), which the paper imports from an earlier work without re-deriving it; if that expansion is incorrect or mis-stated, the central claim about non-linear activations losing control of interaction order collapses.","fun_headline_variants_meta":{"raw":{"variants":["Linear RBM units collapse to Hopfield; nonlinearity unlocks all orders","Only linear RBM units are Hopfield; others boost high-order terms","RBM activation choice: linear = Hopfield storage, nonlinear = full orders","Nonlinear RBM units generate all interaction orders; linear ones don't","In RBMs, linear activation = Hopfield limit; nonlinear = all orders"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000727,"raw_usage":{"total_tokens":3253,"prompt_tokens":934,"completion_tokens":2319,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":2221}},"tokens_in":550,"tokens_out":2319,"duration_ms":16959,"temperature":1.0,"reasoning_tokens":2221,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:33:10.365556+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a small RBM with a non-linear hidden activation, say a single ReLU hidden unit, and compute the coefficients of the visible interaction series exactly; if there is a choice of weights for which all coefficients of order three and higher vanish while the activation remains non-linear, the central impossibility claim is false. Equivalently, train a non-linear RBM on data generated by a pairwise model and check whether the fitted model shows strictly zero higher-order interactions beyond numerical noise; any non-zero higher-order coupling would support the paper's claim, while an exact zero would refute it.","supporting_citations":[{"cited_title":"and Roudi Y","cited_arxiv_id":null,"evidence_quote":"Supplies the interaction expansion (Eq. 2) and the impossibility result for non-linear activations, the paper's central theoretical claim."},{"cited_title":"and Con- tucci P., Neural Networks, 34 (2012) 1","cited_arxiv_id":null,"evidence_quote":"Establishes that a linear hidden activation maps the RBM to a Hopfield model, the basis of the RBM-HM duality."},{"cited_title":"J., Modeling brain function: The world of attrac- tor neural networks (Cambridge university press) 1989","cited_arxiv_id":null,"evidence_quote":"Provides the Hopfield critical storage load $\\alpha_c = 0.14$ that the paper maps to the RBM overfitting transition."},{"cited_title":"and Agliari E., Entropy, 23 (2021)","cited_arxiv_id":null,"evidence_quote":"Reviews and extends the RBM-HM duality to general priors, underpinning the phase-diagram discussion."},{"cited_title":"and De Marzo G., Neural Networks, 148 (2022) 232","cited_arxiv_id":null,"evidence_quote":"Derives the critical sample size for learning archetypes in the dual language, used in the overfitting discussion."},{"cited_title":"and Monasson R","cited_arxiv_id":null,"evidence_quote":"Shows that RBM weights trained on protein sequences recover structural contacts and that double-ReLU hidden units give high expressive power, grounding the protein application."},{"cited_title":"and Monasson R","cited_arxiv_id":null,"evidence_quote":"Demonstrates the compositional phase for RBMs with ReLU hidden units, used to interpret biologically relevant latent representations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces diffRBM, the differential learning strategy applied to immunology data in the final application section."}],"review_version":1}