Current artificial intelligence based models for automated medical diagnosis lack human vision accuracy and efficiency for detecting cardiac structures and functional abnormalities from radiologic images. This study presents a human visual attention inspired deep learning model for the semantic segmentation of Cardiac Magnetic Resonance images to improve the accuracy and efficiency in detecting cardiac structures. The model integrates YOLO, Visual-saliency-map analysis of known features involving texture and pixel intensities, attention-based encoder-decoder model with minimal information loss. YOLO provides a fast region-of-interest detection, pruning redundant image-space and feature-space for efficient target localization. Visual-saliency-maps enhance the probability for pixels associated with target structures. Visual-saliency-map derivation uses a top-down and bottom-up analysis of computationally derived features. The top-down process applies the prior knowledge of known textural and pixel-intensity patterns associated with the cardiac structures. The bottom-up process derives semantically-rich local and global features from the images. Derived visual-saliency-maps are combined with feature-maps, derived by convolution layers, in the encoder-stage to boost the focus on relevant cardiac structures. Information loss in the encoder-decoder stage is minimized by integrating residual links, channel and spatial attention, visual-saliency-maps and edge-detection in the encoder-stage and passing this information as skip-connections to the corresponding decoder-stage. Algorithms are presented. The performance result shows that pruning the image-space and visual-saliency-map augmentation significantly improves the accuracy and inference-time of the semantic segmentation of cardiac structures.