Breast cancer poses a grave threat, being a primary cause of cancer-related fatalities for women globally. Early detection and precise diagnosis are pivotal in enhancing the likelihood of survival. Computer-aided detection systems can aid radiologists in detecting breast cancer on mammograms. This paper proposes a CAD system using Vector Field Convolution (VFC) and active model deformation for mass segmentation and Random Forest (RF) for breast cancer detection. The CBIS-DDSM database provides the mammogram images. Mammograms are processed with two focuses: the entire mammogram and the region having the mass. Preprocessing encompasses noise reduction, binarization, contour-based masking, erosion dilation and Hough transform for the mammogram, and Rough Set (RS) theory-based noise filtering and interference discernment for the region of interest. Segmentation is achieved using an active model deformation approach that minimizes internal and external energy to capture the desired features. Our research results reveal compelling evidence, suggesting that mammogram segmentation utilizing VFC exhibits superior accuracy compared to manual segmentation. Furthermore, comparing various machine learning algorithms, including Random Forest (RF), Support Vector Machine (SVM), and K-Nearest Neighbors (KNN), provides evidence supporting the superior performance of RF over SVM and KNN. Specifically, RF achieved, on its best performance, an accuracy of 83%, proving its effectiveness in mammogram classification tasks using 19 features from segmented mass and the full mammogram.