G2Dropout: gradient-guided dynamic dropout for efficient optimization of deep neural architectures through weights update analysis
摘要
One of the most effective methods for preventing overfitting in deep neural networks is dynamic dropout. In traditional random dropout, nodes are randomly deactivated during training, which improves generalizability but can reduce network accuracy. In contrast, dynamic dropout selectively deactivates specific nodes while retaining the most important ones. Dynamic dropout is effective in networks with a small number of outputs and relatively small datasets; however, as the number of outputs increases, the significance of individual nodes diminishes. Additionally, due to its high time complexity, dynamic dropout is not well suited for large datasets. In this paper, we propose a novel method that independently estimates the overfitting tendency of each node by leveraging the cost function, optimizer function, and node weights. Based on the cost and optimizer functions, we derive the relationship between each node’s weight (w) and its weight change (∆w), determined through gradient-based optimization. Nodes exhibiting higher ∆w are more prone to overfitting and are therefore the best candidates for dropout. We evaluate this technique against previous dropout methods across various deep neural network architectures using the MNIST, NORB, CIFAR-10, CIFAR-100, SVHN, and ImageNet datasets. The experiments demonstrate that, despite its low time complexity, this method is highly effective. Results indicate that the proposed method outperforms other dynamic dropout approaches in networks with a large number of outputs and on larger datasets. Furthermore, the findings suggest that this approach is particularly well suited for large-scale neural network training in high-performance computing (HPC) environments. Across distributed and parallel systems, G2Dropout enhances training efficiency and scalability by eliminating the need for recurrent node importance evaluations and reducing computational overhead. This makes it especially valuable for resource-intensive or real-time applications.