The optical flow method is able to detect targets completely in continuous frames, which in turn is used in environmental monitoring, early warning detection, and so on. However, due to the influence of the motion speed of the object, the algorithm has a ghosting problem. On this basis, an object detection algorithm based on background motion vector fusion is proposed. The algorithm first uses the HS optical flow method to calculate the optical flow matrix of the image, and then combines the image morphological processing to sieve the image background to obtain the initial motion region; the motion vector of the background is extracted as the threshold value of the region screening to carry out the second frame selection of the region after the morphological processing, so as to eliminate the ghosting. The experimental results show that the algorithm effectively eliminates the influence of the ghosting problem on target detection in the optical flow method and improves the accuracy of target detection.