Nuclear power plants require regular maintenance of buildings, equipment, and components, which generates various types of High Dose Rate Radioactive Solid Maintenance Waste (HDRRSMW). It is essential to classify HDRRSMW based on material, type and other attributes. Currently, most nuclear power plants rely on manual sorting, which is inefficient and poses safety risks, so this is an important area to explore. The aim of this study is to improve the recognition efficiency of HDRRSMW by object detection. By improving safety and efficiency, the goal is to facilitate automated and accurate sorting of HDRRSMW. A solid maintenance waste image dataset containing 21,267 YOLO notes was collected. An improved model, YOLO-PBN, was developed on the basis of YOLO11. YOLO-PBN introduces the C2BRA module in the backbone to solve the problem of overlapping waste occlusion. In addition, the PMSFAN module is introduced in the trunk and neck to replace the C3k2 module to solve the problem of large size difference of garbage and insufficient extraction of scale features. Through experimental comparison, the mAp0.5 of YOLO-PBN increased by 2.2–97.5% compared with the benchmark model. The proposed YOLO-PBN model shows great potential for item identification of radioactive solid waste and mixed waste in nuclear power plants under complex operating conditions, providing considerable accuracy improvements over other popular target detection models.