Hands and objects often obstruct each other significantly, posing a considerable challenge for accurately estimating hand-object poses during human-robot interactions. To address this, we present a framework that simultaneously estimates the pose of 3D hand mesh and 6D object in real-time. The proposed framework features a two-stage network cascade. The first stage focuses on localizing regions containing hands and objects, while the second stage estimates both hand and object poses. The hand pose estimation uses a parametric model to infer the hand shape and pose parameters. A cross-model attention injection network is employed to enhance accuracy for both hand and object pose estimation, which regresses the hand parameters and object correspondences necessary for 6D pose estimation. Our method significantly improves joint hand-object pose estimation on two open-source datasets and operates in real time.