<p>Activation functions are a fundamental component of deep neural networks and enable the learning of complex data representations by introducing nonlinearity between layers. Existing activation functions such as ReLU, ELU, Swish and Mish have achieved successful results on various tasks. However, ReLU leads to the “dead neuron” problem, while functions such as Swish and Mish can cause numerical instabilities in deep architectures. In this paper, a new activation function called S2LU (Skew Student's Linear Unit) is introduced that aims to overcome the limitations of existing functions. The S2LU is based on the Skew Student's t-distribution and is designed to be robust against gradient vanishing, dead neurons and saturation problems. The function is differentiable, non-saturating in positive and negative input regions, zero-centered output, monotone and quasi-identity transforming. The effectiveness of S2LU is demonstrated through comprehensive experimental analysis. Importantly, S2LU maintains numerical stability in networks up to 600 layers deep, significantly outlasting standard functions like ReLU and GELU which fails respectively at 300 and 500 layers in the experiments. In classification benchmarks, S2LU achieved state-of-the-art results, reaching 94.168% accuracy on ResNet50/CIFAR-10 and 73.67% on ResNet50/CIFAR-100, outperforming both standard and recent competitors. Furthermore, when integrated into the YOLOv8 architecture, S2LU consistently improved object detection performance over the default baseline, yielding significant mAP increases including +0.3 mAP@50-95 on COCO128, +1.7 mAP@50-95 on the Package dataset, and a +2.9 mAP increase for the 'Positive' class in the Brain Tumor dataset. These findings show that S2LU is not only theoretically sound but also a practical alternative for modern deep learning applications.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

S2LU: An Activation Function Based on Skew Student's t-distribution for Improved Neural Network Performance

  • Özcan Küçükalİ,
  • M. Hakan Bozkurt,
  • Esma Ulutaş,
  • Işılay Bozkurt

摘要

Activation functions are a fundamental component of deep neural networks and enable the learning of complex data representations by introducing nonlinearity between layers. Existing activation functions such as ReLU, ELU, Swish and Mish have achieved successful results on various tasks. However, ReLU leads to the “dead neuron” problem, while functions such as Swish and Mish can cause numerical instabilities in deep architectures. In this paper, a new activation function called S2LU (Skew Student's Linear Unit) is introduced that aims to overcome the limitations of existing functions. The S2LU is based on the Skew Student's t-distribution and is designed to be robust against gradient vanishing, dead neurons and saturation problems. The function is differentiable, non-saturating in positive and negative input regions, zero-centered output, monotone and quasi-identity transforming. The effectiveness of S2LU is demonstrated through comprehensive experimental analysis. Importantly, S2LU maintains numerical stability in networks up to 600 layers deep, significantly outlasting standard functions like ReLU and GELU which fails respectively at 300 and 500 layers in the experiments. In classification benchmarks, S2LU achieved state-of-the-art results, reaching 94.168% accuracy on ResNet50/CIFAR-10 and 73.67% on ResNet50/CIFAR-100, outperforming both standard and recent competitors. Furthermore, when integrated into the YOLOv8 architecture, S2LU consistently improved object detection performance over the default baseline, yielding significant mAP increases including +0.3 mAP@50-95 on COCO128, +1.7 mAP@50-95 on the Package dataset, and a +2.9 mAP increase for the 'Positive' class in the Brain Tumor dataset. These findings show that S2LU is not only theoretically sound but also a practical alternative for modern deep learning applications.