Scholarlink

<p>Activation functions are critical components of neural networks that introduce non-linearity and enable the learning of complex patterns in input data. However, traditional activation functions face challenges such as the dying ReLU problem and saturation, which hinder effective model training and constrain overall performance. This study presents the Adaptive Exponential Activation Function (AMX), a novel approach incorporating two learnable parameters to enhance network adaptability and optimize gradient flow. The AMX activation function is designed to address limitations existing activation functions (e.g., Rectified Linear Unit (ReLU), Exponential Linear Unit (ELU), Parametric Rectified Linear Unit (PReLU), Dual Line Activation Function (Dual Line), Dual Parametric Rectified Linear Unit (Dual PReLU), Flexible Rectified Linear Unit (FReLU), S-shaped Rectified Linear Unit (SReLU), Parametric Exponential Linear Unit (PELU), Continuously Differentiable Exponential Linear Unit (CELU), Fast Exponential Linear Unit (FELU), Elastic Exponential Linear Unit (EELU), Swish Activation Function (Swish), Mish Activation Function (Mish), Parametric Swish Activation Function (P-Swish) and Gaussian Error Linear Unit (GELU)) by dynamically modulating both positive and negative through learnable parameters (α&#xa0;and&#xa0;β). The performance of AMX was empirically evaluated on benchmark datasets (e.g., MNIST, Fashion-MNIST, CIFAR-100, CIFAR-10, CIFAR-10_noisy and Oxford-IIIT Pet) and show the state-of-the-art performance of AMX with reduced error rates compared to traditional activation functions (e.g., 76.85% test accuracy on CIFAR-100 (vs. 42.21% for ReLU) and reducing MSE to 0.0017 on CIFAR-100 (vs. 0.5779 for ReLU)). AMX’s characteristics were assessed across various architectures and shows its ability to effectively overcome neuron inactivity and maintain computational efficiency. Furthermore, the inclusion of a small constant (ϵ)&#xa0;ensures non-zero outputs for the negative inputs and improves learning stability. The comprehensive analysis shows AMX’s potential, and adaptability compared to the conventional activation functions in deep learning through its learnable parameters.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Adaptive exponential activation function (AMX)

  • 期刊论文

  • 发表日期：

    2026年3月15日

查看全文
  • Manoj Kumar Sharma

摘要

Activation functions are critical components of neural networks that introduce non-linearity and enable the learning of complex patterns in input data. However, traditional activation functions face challenges such as the dying ReLU problem and saturation, which hinder effective model training and constrain overall performance. This study presents the Adaptive Exponential Activation Function (AMX), a novel approach incorporating two learnable parameters to enhance network adaptability and optimize gradient flow. The AMX activation function is designed to address limitations existing activation functions (e.g., Rectified Linear Unit (ReLU), Exponential Linear Unit (ELU), Parametric Rectified Linear Unit (PReLU), Dual Line Activation Function (Dual Line), Dual Parametric Rectified Linear Unit (Dual PReLU), Flexible Rectified Linear Unit (FReLU), S-shaped Rectified Linear Unit (SReLU), Parametric Exponential Linear Unit (PELU), Continuously Differentiable Exponential Linear Unit (CELU), Fast Exponential Linear Unit (FELU), Elastic Exponential Linear Unit (EELU), Swish Activation Function (Swish), Mish Activation Function (Mish), Parametric Swish Activation Function (P-Swish) and Gaussian Error Linear Unit (GELU)) by dynamically modulating both positive and negative through learnable parameters (α and β). The performance of AMX was empirically evaluated on benchmark datasets (e.g., MNIST, Fashion-MNIST, CIFAR-100, CIFAR-10, CIFAR-10_noisy and Oxford-IIIT Pet) and show the state-of-the-art performance of AMX with reduced error rates compared to traditional activation functions (e.g., 76.85% test accuracy on CIFAR-100 (vs. 42.21% for ReLU) and reducing MSE to 0.0017 on CIFAR-100 (vs. 0.5779 for ReLU)). AMX’s characteristics were assessed across various architectures and shows its ability to effectively overcome neuron inactivity and maintain computational efficiency. Furthermore, the inclusion of a small constant (ϵ) ensures non-zero outputs for the negative inputs and improves learning stability. The comprehensive analysis shows AMX’s potential, and adaptability compared to the conventional activation functions in deep learning through its learnable parameters.