In view of the problems of small key space and poor key sensitivity in classical image encryption algorithm, the characteristics of chaotic system and M set are analyzed, and a new image encryption algorithm is proposed in this paper. The parameter values generated by SHA-256 algorithm are substituted into the hyper-chaotic Lorenz system to form the initial values of the input plaintext image. Then pseudo-random sequences are generated by hyper-chaotic Lorenz system, and the sequences are scrambled and diffused. The image matrix is generated according to Mandelbrot set, and the image is scrambled again with the sequence generated by different iteration times, and the encrypted image is finally obtained by XOR operation. The experimental results show that the proposed image encryption algorithm has good performance in various evaluation criteria, the key space value of the algorithm is large enough, high key sensitivity and the correlation of adjacent pixels is significantly reduced, and the information entropy is close to the ideal value. Thus this paper proposes image encryption algorithm has good security, can satisfy the requirements of the digital image encryption technology.