Internal gear drive has the advantages of compact structure, large load capacity, and high transmission efficiency, making it widely used in planetary transmission system. However, the current contact form of internal gear with involute tooth profile limits the load strength and transmission durability. So it is of great significance to improve the meshing performance of internal gear by finding a new tooth profile that conforms to the kinematic principle. A new type of internal gear transmission with an S-shaped tooth profile, combining involute and circular-arc curves, is proposed in this paper. Based on the meshing principle and tooth profile enveloping method, tooth surface design is achieved and a mathematical model of the internal gear with S-shaped tooth profile is established. Based on the first and second fundamental homogeneity, the curvature calculation principle for tooth surfaces of the internal gear pair with S-shaped tooth profile is summarized. Contact stress transformation law of the S-shaped internal gear under different working conditions is analyzed utilizing Hertz contact theory, and the contact stress comparisons between the S-shaped internal gear and the conventional involute internal gear are carried out by the established finite element models in ANSYS/Workbench. The virtual prototype of the S-shaped internal gear is proposed based on ADAMS software, and the meshing characteristics of internal gear drive under continuous rotating motion are examined. The change rules of normal contact force, circumferential force, axial force, and radial force are obtained after simulation, and also the contact force change between the S-shaped internal gear and the involute internal gear is analyzed under the same working conditions. The research results indicate that the contact and bending stress of tooth surfaces of the S-shaped internal gear are smaller than those of the conventional involute internal gear under the same basic parameters. The fluctuation amplitude and maximum value of the rotational speed and normal contact force of the S-shaped internal gear are significantly lower than those of the conventional involute internal gear. It has important theoretical significance and reference value for improving the mechanical properties of the current internal gear transmission.