This paper aims to address the issue of the automatic selection of hyperparameters in the deep Q network (DQN) by investigating the influence of eight key hyperparameter indicators on the performance of three different improved DQNs. Our work examines the significance effect, main effect, and interaction effect of those hyperparameters. The goal is to explore how different hyperparameter settings affect the learning efficiency and performance of the improved DQN algorithms to find an effective automatic hyperparameter optimization strategy. The strategy simplifies the DQN models and enhances the efficiency and performance of the algorithms. We first select a set of hyperparameters with significant impacts on algorithm performance. Using an orthogonal experimental design, a multifactor experimental framework was constructed to explore optimal solutions under different combinations of hyperparameter levels. We then evaluated differences in hyperparameter sensitivity among different improved DQN algorithms. Significance analysis and sensitivity analysis were conducted for specific hyperparameters to reveal their main effects in different models and their interaction effects with each other. On the basis of these results, significant hyperparameters and combinations with interaction effects were retained in the model. Additionally, the levels of different hyperparameters were adjusted according to their sensitivity to enhance model stability. The proposed automatic hyperparameter optimization method provides empirical evidence for gaining insight into the relationships between the main effects and interaction effects of hyperparameters, as well as for developing effective hyperparameter optimization methods in reinforcement learning.