Low texture environment is a big challenge for localization and pose estimation for autonomous robots. Existing works are either not suitable for low texture environments or insufficiently fast and robust. In this paper, a tightly Error State Kalman Filter (ESFK) - based coupled Direct Depth-Inertial Odometry and Mapping (ESKF-DIO) framework is proposed to resolve these two issues simultaneously. ESKF-DIO first constructs a depth-residual function to optimize the pose estimation in low texture environment, which improves the efficiency through linearizing the depth-residual and bundle adjustment. A robust feature alignment mechanism is then introduced for accurate feature matching and tracking 2D positions of feature points. At last, linearizing bundle adjustment is used to get accurate camera pose estimation through optimizing the 2D position of feature points. Our proposed method is with an innovative visual-inertial tightly coupled structure that combines IMU and camera data through ESKF (Error-state Kalman Filter) and uses sliding window approach to refine the results. This operation provides accurate pose tracking and low computation cost. We test ESKF-DIO in three challenging environment datasets. The experimental results show that our work is highly robust and efficient in localization compared to state-of-art algorithms with lower estimation error.