A lot of applied problems is often reduced to the Least Squares (LS) and the Weighted Least Squares (WLS) problems. In this paper, an algorithm for finding the solution of the LS and WLS problems with the minimum norm for symmetric positive semidefinite matrices is proposed. The three-stage regularization method for weighed least squares problem with a symmetric positively semi-definite matrix is proposed. The three-stage regularization method is an alternative to both the regularizing functional method and the discrete regularization method, which is based, for example, on the SVD algorithm, since in both cases the initial structure of the matrix is lost. Therefore, the proposed method is preferable in the case of a special matrix structure: sparse, bordered block-diagonal, skyscraper, etc. Moreover, the regularization parameter is selected in such a way as to guarantee the specified accuracy of the obtained pseudo-solution for linear systems with an error in the initial data. The estimations of error of the three-stage regularization method for finding of the weighed normal pseudosolution of WLS problem are got. An algorithm for obtaining a normal pseudo-solution of systems of linear equations with sparse symmetric positive semi-definite matrices on parallel architecture computers is proposed. A Parallel factorization algorithm for bordered block-diagonal matrix based on LLT factorization for a linear least squares problem with a sparse symmetric positive semidefinite matrix was developed and experimentally investigated. The algorithm was tested on a number of test problems. Its effectiveness is shown.