The exponential growth of deep neural network (DNN) model size and data volume makes distributed training indispensable in cloud environments. Today’s cloud datacenters typically operate as heterogeneous clusters, where computing nodes are equipped with diverse GPU generations. In such environments, cloud providers face the critical challenge of optimally matching user-submitted training jobs with suitable GPU resources to accelerate the job training process. Existing works either improve computational efficiency by leveraging affinity between heterogeneous GPUs and jobs, or arrange jobs on the same type of GPUs as much as possible to reduce cross-rack communication overhead. However, none of these methods simultaneously considers both computation and communication factors, despite their combined importance in determining job completion time (JCT) for distributed training. In this paper, we propose HetSpeed, a Heterogeneous-aware resource allocation method to jointly cluster efficient computational resources and communication overheads to Speedup the model training process. HetSpeed formulates a binary integer programming problem that incorporates real-world scenario constraints and proves its NP-hardness. To solve this problem, HetSpeed presents an effective submodular-based greedy algorithm with a tight approximation ratio \((1-\frac{1}{e})\) . We evaluate HetSpeed with real-world job traces, and HetSpeed is able to minimize the cluster’s total cross-rack traffic, decreasing the average JCT by 27.8% compared to the state-of-the-art solution.