We aim to predict the stability of carotid artery plaques from longitudinal ultrasound images. This is important since atherosclerosis is the primary cause of heart disease and stroke. Accurately predicting plaque stability would allow for more targeted follow-up and treatment, saving healthcare costs. We analyze data from over 400 patients followed for 3 years, exceeding the size of previous studies. We first localize the carotid artery and segment the plaque within the images. A self-supervised learning approach was used for plaque segmentation, leveraging the power of unlabeled data. The plaque stability predictor uses three image channels derived from the ultrasound image and its segmentation. As an auxiliary task, we predict the plaque width, which helps to prevent overfitting. The balance between the criteria is maintained automatically. Our estimate of the plaque width correlated well with expert measurements ( \(\rho =0.56\) ). We confirmed that there is a relationship between the plaque ultrasound appearance in longitudinal images and their stability. However, the future width correlation and the plaque stability prediction performance remained modest (AUC = 0.61), similar to previous studies.