{"id":"d31799d5-3fc3-45cd-b254-751bc0ff338c","arxiv_id":"2505.23942","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SG-Blend, a learnable convex combination of a shifted Swish and GELU, reports modest accuracy and BLEU gains on CIFAR-10, IMDB, and a WMT14 subset, with no variance reporting.","lead":"SG-Blend is a new activation function that learns to mix a modified Swish (called SSwish) with GELU, and is tested on image and text classification plus a small translation task. The reported single runs show small gains over standard activations, but the paper lacks error bars and overstates its BLEU improvement.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-run comparisons without error bars cannot support the claim that SG-Blend consistently outperforms strong baselines; the reported margins are small enough to be run-to-run noise.","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing point: all comparisons are single runs without error bars, and SG-Blend's gains are small. My read does not change that. I considered whether a stronger internal issue exists, such as the extra learnable parameters (alpha, beta, gamma) giving SG-Blend an unfair capacity advantage over fixed baselines, or the WMT14 BLEU reporting inconsistency (abstract 5.63 vs table 0.0563). These are real but secondary; without repeated runs one cannot even establish whether the effect exists. The proposed test, multi-seed reruns with paired confidence intervals, directly settles the concern. If SG-Blend survives it, the paper still needs a corrected abstract and baseline tuning details; if it does not, the main claim should be withdrawn. Therefore the reader's CONDITIONAL verdict remains appropriate.","tokens_in":9613,"tokens_out":3422,"duration_ms":37546,"concrete_test":"Run a multi-seed evaluation of all configurations in Tables 3, 4, and 5 using the exact Section 3.1 protocol, with at least 10 independent seeds per activation. Report mean plus/minus standard deviation for the primary metric and compute paired bootstrap confidence intervals for SG-Blend versus the best baseline in each table; if intervals overlap or the effect reverses in any task, the 'consistent outperformance' claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that SG-Blend consistently outperforms ReLU, Swish, GELU, and Mish across vision and NLP benchmarks. This claim rests on Tables 3-5, but every cell in those tables is a single run with early stopping and no reported seeds, error bars, or significance tests (Section 3.1). The margins are small: +0.36% vs Swish on ResNet18 CIFAR-10, +1.13% vs ReLU on ResNet50, +0.08% vs Mish on IMDB, and +0.0563 BLEU on a 50,000-pair WMT14 subset. Given that no baseline hyperparameter search is reported and all activations share the same training schedule, these differences are within plausible run-to-run variation, especially for the small translation subset. The abstract's claim of a 5.63 BLEU gain versus the 0.0563 in Table 5 also indicates a reporting inconsistency that makes the empirical basis harder to trust. Without repeated runs and variance reporting, the paper cannot distinguish a real adaptive advantage from favorable initialization noise. This is the load-bearing assumption behind the 'consistent outperformance' claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes SSwish (Eq. 3), a vertically shifted Swish with learnable parameters beta and gamma, and SG-Blend (Eq. 5), a convex combination of SSwish and GELU with a learnable interpolation weight alpha. It reports image classification on CIFAR-10 with ResNet18/50, sentiment analysis on IMDB with a model described as BERT, and machine translation on a 50,000-pair WMT14 En-De subset with a small Transformer, claiming consistent gains over ReLU, Swish, GELU, and Mish with negligible overhead. An additional experimental section reports SSwish-only comparisons. The central claim is empirical: the adaptive interpolation is alleged to generalize better across vision and NLP tasks and architectures.","tokens_in":9854,"tokens_out":8597,"duration_ms":83022,"significance":"SG-Blend is a clearly specified and easy-to-implement activation function, and the anonymous code release plus multi-modality evaluation are positive features. The contribution is incremental, however: a learnable convex combination of two known activations with a shift parameter is not a large conceptual advance, and the reported gains are small. The decisive question is whether the single-run differences in Tables 3-5 represent true generalization improvement. Because the paper reports no seeds, error bars, or statistical comparisons, the significance currently rests on an unverified empirical claim. If multi-seed experiments confirm stable gains, the paper would be a useful practical addition to the activation-function literature.","major_comments":[{"comment":"Every configuration is evaluated once, with early stopping and no seeds, error bars, or significance tests (Section 3.1). The margins are small: +0.36 percentage points vs Swish on ResNet18 CIFAR-10, +0.68 vs ReLU on ResNet50, +0.08 vs Mish on IMDB, and +0.0563 BLEU on the WMT subset. These differences are within plausible run-to-run variation for a single T4 run, especially since no baseline hyperparameter search is reported and all activations share the same training schedule. This evidence cannot support the abstract's and Section 3.4's claim that SG-Blend 'consistently outperforms' strong baselines. The authors should rerun with multiple seeds per configuration, report means and standard deviations, and where appropriate apply paired or matched comparisons. The paper's own Limitations paragraph notes potential overfitting risk from the three learnable parameters, which reinforces the need for variance reporting.","section":"Section 3.1, Tables 3-5"},{"comment":"The abstract and Section 1 claim 'upto 5.63 in BLEU score,' but Table 5 reports SG-Blend's BLEU as 0.5735 versus GELU's 0.5172, a difference of only +0.0563. The factor-of-100 discrepancy must be resolved, and the BLEU scale (0-1 versus 0-100) must be stated explicitly. As written, this is a factual inconsistency in the headline empirical claim, and it makes the WMT result difficult to interpret.","section":"Abstract, Section 1, Table 5"},{"comment":"The main IMDB experiment is described as using 'a standard Keras implementation of a BERT model' and 'BERT base' in Sections 3.1 and 3.3.1, but Section 6.3 describes a 'lightweight 2-layer BERT-style transformer' with model dimension 64 and a training time of about 55 seconds. If these are the same model, the text substantially overstates the scale of the language model; if they are different models, the paper must state which one produced Table 4. The claim that the results demonstrate effectiveness in 'large-scale language models' is not supported without this clarification.","section":"Section 3.3.1 and Section 6.3"},{"comment":"SSwish is repeatedly called a 'symmetric' or 'first-order symmetric' variant of Swish, but Eq. (3) is simply a vertical shift: SSwish(x) = Swish(x) - gamma. A vertical shift does not make the function odd or symmetric about the origin, and the first derivative in Eq. (4) is neither even nor odd. The statement that 'by incorporating learnable slope and bias parameters, SSwish is designed to enforce symmetry in the activation's response' is therefore incorrect as stated. If the intended property is only that the learnable offset can center the mean activation, the paper should say so and drop the symmetry terminology; otherwise, the authors should provide a definition that actually enforces symmetry.","section":"Section 2.2, Eq. (3)"}],"minor_comments":[{"comment":"The text says GELU is 'unbounded below but approaches 0 slowly as x→−∞'; this is internally contradictory because the standard GELU is bounded below, with an infimum of about -0.17 near x = -1.","section":"Section 2.3"},{"comment":"The WMT14 evaluation uses a 50,000-pair subset with a 10% validation split from the same subset; the paper should explicitly state that these BLEU scores are not comparable to standard full-dataset WMT benchmarks and should report the exact evaluation script, since BLEU implementations can differ by an order of magnitude.","section":"Section 3.3.2"},{"comment":"The claim of 'negligible computational overhead' for SG-Blend is not directly measured; training time is reported only for SSwish versus Swish in Tables 6-7, not for SG-Blend versus the baselines in Tables 3-5.","section":"Section 5"},{"comment":"The additional experiments in Section 6 use different setups (a 2-layer Transformer and a custom CNN) from Section 3, and the relationship between these SSwish-only results and the main SG-Blend experiments is not clearly stated; consider labeling them explicitly as preliminary ablations.","section":"Tables 6-7"},{"comment":"The statement that ReLU, Swish, GELU, Mish, and SG-Blend all produce 0.00% dead neurons in the BERT model is surprising and needs a brief explanation of how dead neurons were counted, especially for ReLU.","section":"Section 3.3.1"}],"recommendation":"major_revision","confidential_remarks":"This is a standard empirical activation-function paper with a clearly specified method and a public anonymous code repository. The central problem is the complete absence of run-to-run variability information for a claim of consistent superiority, which is correctable but substantive. I also urge verifying the BLEU arithmetic, which currently differs by a factor of 100 between the abstract and Table 5, and clarifying whether the IMDB experiment uses a real BERT-base model or a tiny from-scratch transformer. I see no circularity or novelty-disclosure issue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Best to know upfront: SG-Blend is SSwish (Swish minus a learnable gamma) blended with GELU via a learnable alpha. That's it. The novelty is modest, but it's a legitimate new parameterization and the authors actually test it on vision and NLP with several baselines. The code is available, and they're honest about limitations (no generative models, overfitting risk).\n\nThe empirical claim is the problem. Every comparison is a single run with no seeds, no error bars, no significance testing. Margins are small: 0.36% over Swish on ResNet18 CIFAR-10, 0.08% over Mish on IMDB, 0.0563 BLEU on a 50k-pair WMT14 subset. Those are within run-to-run noise for these settings. The abstract says 'upto 5.63 in BLEU' when Table 5 reports +0.0563; that's a factor-of-100 error and it makes you wonder what else was checked quickly. Baselines share a single training schedule, and there's no evidence they were tuned comparably. So the paper's central claim — 'consistently outperforms strong baselines' — is not yet supported, even though the direction is uniform across tables.\n\nThe 'symmetry' framing is also off. SSwish subtracts a constant, which shifts the output down; it doesn't make the function symmetric. The derivative is unchanged by gamma. If the intended property is something like centered activations, say that instead. The related work is thin on learnable mixtures of activations; that's a minor omission.\n\nWhat's genuinely useful: the formulation is simple, the implementation is straightforward, and the experiments cover a reasonable spread of architectures. With multi-seed results, variance reporting, and a corrected abstract, this could be a solid small contribution. As is, it's a plausible activation-function paper that needs revision before the claims are evaluable.\n\nI'd send it to review, not desk-reject it, but I'd ask for seeds and a corrected abstract before it's citable.","headline":"A simple learnable blend of shifted Swish and GELU shows consistent small gains, but single-run results and a 100x BLEU typo undermine the 'consistently outperforms' claim.","tokens_in":10367,"tokens_out":2251,"would_cite":false,"duration_ms":23333,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes SG-Blend, a learnable blend of a symmetrized Swish and GELU, and claims it beats ReLU, Swish, GELU, and Mish on every benchmark tested.","keywords":["SG-Blend","SSwish","Swish","GELU","learnable activation","activation function","image classification","natural language processing"],"falsifier":"A multi-seed replication would settle the claim: run each reported configuration (ResNet18 and ResNet50 on CIFAR-10, BERT on IMDB, and the 50k-pair transformer on WMT14) at least five times with identical hyperparameters, and check whether the SG-Blend minus baseline margins consistently stay away from zero. The paper itself names the untested generative-model regime, so a second check would be the same comparison on a generation task.","tokens_in":9414,"feed_emoji":"🔀","tokens_out":12245,"duration_ms":101859,"temperature":0.7,"pith_summary":"The paper proposes SG-Blend, a smooth activation function that learns to interpolate between a symmetrized version of Swish (which the paper calls SSwish) and GELU using per-layer parameters ($\\alpha,\\beta,\\gamma$). Its central claim is that this adaptive blend consistently outperforms ReLU, Swish, GELU, and Mish on the reported benchmarks: CIFAR-10 with ResNet18 and ResNet50, IMDB sentiment with BERT, and a WMT14 English-German translation subset, with the largest reported gain in the translation metric. If the claim holds, SG-Blend offers a drop-in replacement that could let practitioners avoid choosing activations by task or architecture, at negligible computational cost. The paper also contributes SSwish itself, which in its additional experiments matches or slightly beats Swish.","feed_headline":"New activation blends Swish and GELU, beating both on tests","feed_subtitle":"SG-Blend learns per-layer interpolation and posts consistent gains on CIFAR-10, IMDB, and WMT14 translation.","key_machinery":"The load-bearing object is the parameterized convex combination in Eq. (5), whose two components are the new SSwish and the established GELU. SSwish, defined as $x\\sigma(\\beta x)-\\gamma$, augments Swish with a learnable sigmoid steepness $\\beta$ and a learnable vertical shift $\\gamma$; the shift allows the network to center its activation statistics, which the paper hypothesizes improves gradient symmetry. The blend weight $\\alpha$ is also learned, making the activation shape adaptive per layer. The smoothness of both components and the derivative identity in Eq. (6) — the SG-Blend gradient is the same convex combination of the component gradients — are what let the network tune its nonlinearity while preserving differentiable, stable gradient flow.","core_discovery":"The central claim, stated on the paper's own terms, is that SG-Blend — the learnable convex combination $\\mathrm{SG\\text{-}Blend}_{\\alpha,\\beta,\\gamma}(x) = \\alpha\\cdot\\mathrm{SSwish}_{\\beta,\\gamma}(x) + (1-\\alpha)\\cdot\\mathrm{GELU}(x)$ with $\\mathrm{SSwish}_{\\beta,\\gamma}(x) = x\\sigma(\\beta x)-\\gamma$ — is a more universally robust activation than its fixed-shape predecessors. The parameters $\\alpha\\in[0,1]$, $\\beta\\in[0.1,10]$, and $\\gamma$ are learned per layer by backpropagation, so each layer can specialize toward SSwish's controlled non-monotonicity or GELU's smooth probabilistic profile. Because both components are smooth, SG-Blend is $C^\\infty$ and its derivative is a smooth weighted sum, which the paper ties to stable gradient flow. Empirically, the paper reports best-in-table results in every configuration it tests, including 93.23% top-1 accuracy on CIFAR-10 with ResNet18 and 89.56% validation accuracy on IMDB with BERT.","pith_inferences":["Editorial extension: tracking the learned per-layer values of $\\alpha$ after training would show whether vision layers converge toward SSwish and language layers toward GELU, turning the paper's motivating hypothesis into a directly testable prediction.","Editorial extension: because each configuration was evaluated once, the same protocol with multiple random seeds would show whether margins like the 0.36% CIFAR-10 ResNet18 gap persist; this is the minimal experiment that would separate the blend's effect from training noise.","Editorial extension: the convex-combination recipe generalizes to other activation pairs, so SG-Blend can be read as an instance of a broader learnable-activation family rather than a single function."],"forward_implications":["SG-Blend can be inserted into existing ResNet and Transformer models as a drop-in replacement for ReLU, Swish, GELU, or Mish without changing the architecture and with negligible extra compute.","Per-layer learning of $\\alpha$ lets a single network settle on SSwish-like nonlinearity in some layers and GELU-like smoothness in others, which is what the paper identifies as the source of its adaptability.","The largest reported gain is in translation: on the WMT14 subset SG-Blend scores 0.5735 BLEU versus 0.5172 for GELU and 0.5001 for Swish.","SSwish alone already matches or exceeds Swish in the paper's additional experiments, so the symmetry modification contributes independently of the GELU blend."],"supporting_citations":[{"why":"Supplies Swish, the base activation that SSwish modifies.","marker":"Ramachandran et al. [2018]"},{"why":"Supplies GELU, the second component of SG-Blend.","marker":"Hendrycks and Gimpel [2016]"},{"why":"Defines the ReLU baseline used in the comparisons.","marker":"Nair and Hinton [2010]"},{"why":"Defines the Mish baseline used in the comparisons.","marker":"Misra [2019]"},{"why":"Provides the ResNet18 and ResNet50 architectures for the CIFAR-10 experiments.","marker":"He et al. [2016]"},{"why":"Provides the CIFAR-10 dataset.","marker":"Krizhevsky [2009]"},{"why":"Provides the IMDB sentiment dataset.","marker":"Maas et al. [2011a]"},{"why":"Provides the WMT14 English-German benchmark used for translation experiments.","marker":"Bojar et al. [2014]"},{"why":"Provides the BERT architecture used for IMDB experiments.","marker":"Devlin et al. [2019]"},{"why":"Provides the transformer architecture used for WMT14 experiments.","marker":"Vaswani et al. [2017]"}],"fun_headline_variants":["Learnable blend of SSwish and GELU beats both on NLP and CV","SG-Blend: per-layer interpolation of SSwish and GELU wins consistently","New activation adapts Swish and GELU blend per layer, tops benchmarks","Interpolating SSwish and GELU yields robust gains across tasks","Learnable mix of SSwish and GELU improves models with no overhead"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reported accuracy and BLEU differences between SG-Blend and the baselines are genuine generalization differences rather than run-to-run noise, because each configuration was evaluated once with early stopping and no random seeds or error bars.","fun_headline_variants_meta":{"raw":{"variants":["Learnable blend of SSwish and GELU beats both on NLP and CV","SG-Blend: per-layer interpolation of SSwish and GELU wins consistently","New activation adapts Swish and GELU blend per layer, tops benchmarks","Interpolating SSwish and GELU yields robust gains across tasks","Learnable mix of SSwish and GELU improves models with no overhead"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000581,"raw_usage":{"total_tokens":2758,"prompt_tokens":992,"completion_tokens":1766,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":608,"completion_tokens_details":{"reasoning_tokens":1659}},"tokens_in":608,"tokens_out":1766,"duration_ms":11583,"temperature":1.0,"reasoning_tokens":1659,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:37:32.221684+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A multi-seed replication would settle the claim: run each reported configuration (ResNet18 and ResNet50 on CIFAR-10, BERT on IMDB, and the 50k-pair transformer on WMT14) at least five times with identical hyperparameters, and check whether the SG-Blend minus baseline margins consistently stay away from zero. The paper itself names the untested generative-model regime, so a second check would be the same comparison on a generation task.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies Swish, the base activation that SSwish modifies."},{"cited_title":"Gaussian error linear units (gelus)","cited_arxiv_id":null,"evidence_quote":"Supplies GELU, the second component of SG-Blend."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the ReLU baseline used in the comparisons."},{"cited_title":"Learning Multiple Layers of Features from Tiny Images , 1 2009","cited_arxiv_id":null,"evidence_quote":"Provides the CIFAR-10 dataset."},{"cited_title":"Findings of the 2014 workshop on statistical machine translation","cited_arxiv_id":null,"evidence_quote":"Provides the WMT14 English-German benchmark used for translation experiments."}],"review_version":1}