Liver cancer is a common and often fatal disorder that is becoming more commonplace worldwide. An accurate and timely diagnosis is necessary for both effective treatment and patient survival. In machine learning techniques, particularly deep learning, obtaining a large and diverse dataset is still a challenge for deep neural network training, particularly in the medical industry. This paper presents a classification of circulating tumor cells based on the YOLOv8 algorithm. Tumor cell identification and classification can be achieved by utilizing the algorithm’s multi-layer high-level stacking, weight sharing, local connection, and pooling characteristics. The goal is to design a liver cancer classification system that makes it easier and increases the efficiency of doctors in analyzing the results of liver cancer. The models show the absolute the accuracy is 100%, 100%, 98%, 96% to Yolov8n, Yolov8s, Yolov8m, and Yolov8l respectively.