Accurately tracking the robotic arm and human joints is crucial to ensure safety during human-robot interaction. However, traditional pose tracking methods often exhibit insufficient performance and robustness in complex environments. The variations in the robotic arm’s environment and posture make it challenging for traditional methods to accurately capture the positions and posture of its joints. Specifically, when addressing challenges such as high similarity, occlusion, background complexity, and joint recognition failures, they often struggle to provide reliable and accurate results. To address these challenges, this paper proposes a human-robot pose tracking algorithm based on a new convolutional neural network model. To enhance detection accuracy, an improved color detection module is introduced to resolve joint misclassification. A geometric perception module is designed to accurately locate joints even under occlusion. Additionally, innovative iEMA and DBB modules are incorporated. The iEMA module employs edge detection technology to dynamically adjust thresholds for correct matching by improving the edge matching process. The DBB module refines the boundary box parameters for precise localization by introducing adaptive bounding box updates in real-time. This algorithm also integrates human pose recognition, enabling real-time pose recognition, thereby facilitating more intelligent and natural human-robot interaction. The algorithm has been rigorously evaluated on a custom-designed robotic arm platform. Experimental results validate the algorithm’s effectiveness and feasibility.