Anchor-based models use predefined anchor boxes to predict the locations and sizes of objects, but the effectiveness of the anchor box is highly dependent on the dataset and domain it is applied. In this paper, we address this problem by automatically setting the anchor boxes optimized for the dataset, employing a genetic algorithm. The proposed method was evaluated on two different datasets with similar distribution in their train and test sets. The approach was applied to various anchor-based models, all of which demonstrated performance enhancements in terms of mean average precision. Specifically, in the benchmark dataset, even a simple anchor-based model performed as well as state-of-the-art model. In addition, since the model learns more efficiently with the chosen anchor boxes, it is proven that our method contributes to the faster convergence of the models. Thus, this study introduces a novel approach for enhancing the performance of object detection models and significantly contributes to the field of anchor box optimization.