The coal formation exhibits poor cementation, leading to the risk of wellbore instability during drilling. Ensuring the stability of the wellbore in coal formation is of great significance for the safe and efficient exploitation of coalbed methane. This paper established a geological model considering fault (weak plane of coal) and two types of formations based on the actual trajectory extension of horizontal wells using Rhinoceros. The model accurately reflects the penetration of horizontal well trajectories through undulating formations. The fluid–solid interaction simulation of the stress field and plastic zone distribution around the wellbore was conducted using FLAC3D software. The critical collapse pressure was calculated based on classic rock mechanics theories using MATLAB to verify the accuracy of the numerical simulation results. The results show that the difference between the maximum and minimum horizontal principal stresses around wellbore is 35–55 MPa at 0° and 180°, and 5–40 MPa at 90° and 270°. Therefore, the risk of wellbore instability is higher near 0° and 180°. As the drilling fluid density increases (from 1.00 g/cm3 to 1.75 g/cm3), the diameter of the plastic zone in the coal formation decreases from 9.2 times to 0.3 times, and in the fault, it decreases from 17 times to 0.7 times. According to the Mohr–Coulomb criterion, the lower limit of the safe density window for drilling fluid is calculated to be 1.67 g/cm3. At a drilling fluid density of 1.70 g/cm3, the diameters of the plastic zones in the fault and coal formation are 1.43 times and 1.14 times, respectively, that of the sandstone formation. At a drilling fluid density of 1.70 g/cm3, as the wellbore curvature increases, the diameter of the plastic zone around the wellbore increases from 0.52 times to 1.02 times. This paper provides a novel method for analyzing wellbore stability that reflects the real formation and wellbore trajectory, which is beneficial for the safe and efficient drilling of coal formation.