Distributed machine learning has recently become a critical paradigm for training large models on vast datasets. We examine the stochastic optimization problem for deep learning within synchronous parallel computing environments under communication constraints. Although averaging distributed gradients is the default method for gradient estimation, whether this is the optimal strategy remains an open question. In this work, we analyze the distributed gradient aggregation process through the lens of subspace optimization. By formulating the aggregation problem as an objective-aware subspace optimization problem, we derive an efficient gradient weighting scheme. Our method maintains statistical unbiasedness in the aggregation and is hyperparameter-free. We demonstrate improved performance of our method over ubiquitous gradient averaging on multiple MLPerf tasks while remaining efficient in both communicational and computational complexity. The code is provided as supplementary material.