<p>Activation functions play a crucial role in deep learning networks. However, the performance of learnable activation functions is significantly affected by parameter initialization as well as hyperparameters such as learning rates. Besides, fixed activation functions are insufficient for modern complex tasks. To address these limitations, we propose two new non-linear non-monotonic activation functions called Normalized Gating GELU (NGGELU) and Normalized Gating Swish (NGSwish), which introduce normalization into GELU and Swish functions. Our newly proposed activation functions can dynamically adjust the shape according to the distribution of each input sample for different models, datasets and layers without introducing new learnable parameters. Our experiments indicate that NGGELU, NGSwish and their variants all make full use of the information of each input sample. And they outperform state-of-the-art activation functions, including the classical models like ReLU, Swish and GELU, for various deep learning architectures and tasks.</p>