The Cardiovascular Diseases (CVDs) is the frequent cause of death worldwide. Atherosclerosis development is a prime factor underlying CVD events such as strokes and heart attacks. Early detection of carotid plaques are crucial for preventing strokes. Visual characterization and classification of carotid plaque lesions is a time-consuming, error-prone and tedious process. There is a high interest in developing computer vision models that can be combined into original clinical practice. The classification of carotid plaque utilizing Magnetic Resonance Imaging (MRI) is important because it enables non-invasive and detailed validation of the plaques features and composition within the carotid arteries thus allowing the identification of vulnerable plaques with high rupture risk and future stroke that can assist the patient management and treatment decision procedures. Nevertheless, the MRI image’s noise, joined with the tiny plaque size and their composite appearance, makes it complex for differentiating among distinct plaque parts. The conventional approaches encounter problems in recognizing the unstable plague features accurately, especially when handling noisy MRI images. Hence, an intellectual carotid plaque classification approach is presented in this work for improving the early diagnosis and treatment procedures. This work initially collects the MRI images from the available resources. Subsequently, the gathered images are given to the multi-label segmentation stage, where the Transformer-based Unet++ (Trans-Unet++) model is employed. By employing the segmented images, the classification is performed, where the Pyramid Dilated DenseNet (PDDNet) is utilized for classifying the carotid plaque. In the end, the developed model’s performance is investigated and compared with the existing models for illustrating its efficacy.