Angle of progression (AoP) is seen as a reliable and reproducible parameter for assessing fetal head descent. But fully automatic AoP measurement from transperineal ultrasound videos is a clinical challenge. This study will present a video-based AoP measurement technique and evaluate its accuracy and reliability. Our video-based AoP measurement method includes three steps: videos with a duration of more than \(26\pm 2\)  s were selected; for each eligible video, each frame was processed by our deep learning network. The frame’s class, segmented areas and target (pubic symphysis and/or fetal head) locations were obtained. Based on the frame’s class, frames with both segmented pubic symphysis and fetal head in each eligible video were selected for AoP measurement; and, for each eligible AoP measurement frame, first, determine the left and right endpoints of the segmented pubic symphysis through the left and right intersections of its segmented area and the detected frame. Second, determine the right tangent of the segmented fetal head connected to the right endpoint of the pubic symphysis. And thus, AoP was calculated as the angle between the right tangent of the segmented fetal head and the line connecting the left and right endpoints of the segmented pubic symphysis. Eventually, the AoPs of eligible frames in the same video were averaged to get the video’s AoP. The intra-class correlation coefficient and the limits of agreement were used to evaluate the agreement between our method and clinicians. 217 videos collected from Nanfang Hospital of Southern Medical University (Guangzhou, Guangdong, China) during 2019 - February 2020 were adopted to evaluate our method. There is a strong and statistically significant correlation between the AoP values measured by our algorithm and those by clinicians (R = 0.98, P < 0.001). An intra-class correlation coefficient is 0.977 (Confidence Interval: 0.965–0.985) and the limits of agreement are \(-6.43^{\circ }\) and 5.118 \(^{\circ }\) . An excellent agreement was found between our video-based AoP measurement method and clinicians in AoP measurement. It is a critical step towards real-time monitoring of fetal head descent.