<p>This paper is concerned with a backpropagation algorithm for training stochastic neural networks. The training process is formulated as a stochastic optimal control problem, which is solved using a generalized stochastic gradient descent algorithm. To improve computational efficiency, a sample-wise approximation scheme is applied to the backward stochastic differential equation. To the best of our knowledge, the most closely related existing convergence analysis was presented in [Archibald et al., SIAM Journal on Numerical Analysis, 62 (2024), pp. 593-621], which achieved at most half-order convergence. However, this half-order convergence rate is not optimal. To address this issue, we develop a high-order sample-wise backpropagation algorithm that mitigates the information loss inherent in sample-wise approximations and overcomes the structural limitations of forward networks. Under the convexity assumption, we prove that the proposed algorithm achieves optimal first-order convergence. Finally, we present several numerical experiments to validate our theoretical results and demonstrate the effectiveness of the proposed first-order sample-wise backpropagation algorithm.</p>