PSILELU: A PARAMETRIC SIGMOID LINEAR UNIT FOR STOCHASTIC STABILITY IN DEEP NETWORKS

Abstract

We propose PSiLELU, a new parametric activation function, to overcome three limitations of the existing neural nonlinearities in terms of gradient stability, noise immunity and activation sparsity with deep networks. With tunable parameters α (leak strength) and β (sigmoid sharpness), PSiLELU interpolates input regimes with gradients that can flow through the neuron and no risk of dead (off) neurons. We provide theoretical assessment over variance and gradient boundedness under Gaussian perturbations, local and global Lipschitz continuity and multi-layer noise stability. Comparative analysis demonstrates enhanced gradient properties and out-of-sample diversity compared with state-of-the-art functions including ReLU and Swish. Empirical search over α–β configurations further demonstrates that PSiLELU automatically and effectively tunes its behavior to different depth and noise levels and provides a stable and flexible way of learning in noisy, high-dimensional spaces.