Infrared small target detection is very important in many fields such as area surveillance, vehicle navigation, space-based detection, and infrared search and tracking. Aiming at the problems of low accuracy and high false alarm rate of traditional low-rank sparse decomposition model, this paper proposes an improved infrared patch-tensor model for Small Target Detection guided by a priori knowledge, which utilizes the second-order directional derivative filter to effectively extract Gaussian-like features and combines with the penalty factor to realize the target a priori information extraction. Then, the background prior of the complex background region is measured as the variance between the difference between the maximum gray value and the average gray value in each background window around the background target. Subsequently, leveraging the established prior knowledge of both target and background, the method utilizes the low-rank property of the background and the sparsity of infrared small targets to effectively distinguish between them. Experimental findings demonstrate the method's efficacy in accurately detecting infrared small targets amidst intricate backgrounds such as sky, ground, and sea.