Human semantic segmentation is increasingly important in various human sensing scenarios such as motion analysis. However, traditional visual methods raise privacy concerns, while RF methods suffer from multi-path reflections or sparse signals. This paper introduces MIRaSeg, a novel multi-modal human semantic segmentation system that fuses millimeter-wave radar and the low-resolution infrared sensor. Specifically, we carefully design a unique feature extraction and fusion network to enhance human semantic segmentation of millimeter-wave point clouds using low-resolution infrared heatmap information. We construct a 400,000 frame activity dataset from 34 volunteers, the experiments show that MIRaSeg achieved an average accuracy of 83.21%, all of which outperformed the state-of-the-art methods.