Screening X-ray baggage is crucial for border security, but image quality and available prohibited data are challenges. Lack of data in training AI models can affect the detection results. Here, we evaluate object detection based on YOLO-v8 with an imbalanced dataset from SIXray dataset. The proposed X-ray baggage security screening framework consists of image acquisition, annotation, classification, and threat detection phases. The YOLO-v8 architecture is used to train the model, allowing for quick and accurate identification of potential threats. Evaluation methods such as confusion matrix and precision-recall curve provide a comprehensive view of the classifier’s performance. The model was trained using 17,498 images and 1642 images for validation. The model performance achieves the largest recall value of 92%, and the largest precision of 93.80%. However, even with an F1 score is 63.81%, the experiment showing the YOLO-v8 algorithm can detect small objects and overlap objects in different orientations. This shows that the Yolo architecture can handle imbalanced datasets.