Legged robots have gained increasing research interest in recent years due to their ability to traverse uneven terrains with higher mobility compared to wheeled robots. However, controlling the complex leg movements and gaits of multi-legged robots with many degrees of freedom poses a significant challenge. This paper presents a kinematics analysis and modeling approach for the gait of 18 degree-of-freedom (DOF) hexapod robots using reinforcement learning. The body trajectories of the hexapod are parameterized to reduce the dimensionality of the control problem. Reinforcement learning is applied to optimize the gait policy and trajectory generators, enabling the hexapod to learn dynamic gaits to walk forward, turn, and traverse obstacles. The parameterized body trajectories provide an efficient representation for training the gait policies. Results from simulations demonstrate the effectiveness of the proposed approach in producing stable walking patterns over unknown terrains. The methodology provides a framework for applying reinforcement learning to control complex high-DOF legged robots.