Anomaly detection is crucial for ensuring the stability and reliability of software systems. With the increasing scale and complexity of software systems, traditional anomaly detection methods relying on single-type data, such as logs or metrics, have become inadequate. To detect system anomalies accurately and promptly, heterogeneous data fusion analysis has emerged as a popular and practical approach. However, existing heterogeneous data fusion methods often employ simplistic fusion techniques that fail to effectively utilize the joint information from different data types. In addition, these methods do not prioritize data integrity and exhibit poor robustness to imbalanced heterogeneous data. To address these challenges, we propose HeRF-AD, an end-to-end semi-supervised anomaly detection framework that enhances accuracy and robustness by separately modeling log and metric data and applying deep heterogeneous representation fusion. HeRF-AD abandons template parsing to maintain log integrity and introduces attention mechanisms for preliminary anomaly localization. We extensively evaluate HeRF-AD on two publicly available heterogeneous datasets and one synthetic dataset. Experimental results demonstrate that our model outperforms state-of-the-art methods in anomaly detection performance and robustness to data imbalance.