Breast cancer has emerged as a big reason of deaths in-between women. The multi-class classification of breast-masses using the SFM mass images is conducted by various computerized methods since last many years. Present work proposes a CAD design for multi-class classification of SFM mass images using deep feature-set and machine learning classifiers. The exhaustive experimentation is conducted by employing nine DL-based models and three ML based classifiers. These DL-based models used for extraction of deep feature-set are simple convolution series models / simple convolution DAG model/dilated convolution DAG models. The three ML-based classifiers i.e. ANFC-LH/ PCA-SVM/GA-SVM have been used extensively for classification task. Experimental work is carried on 518 SFM mass images chosen from DDSM dataset with 208 ϵ BIRAD-3, 150 ϵ BIRAD-4 and 160 ϵ BIRAD-5 classes, respectively. For segmenting masses from SFM mass images, ResNet50 semantic segmentation-model has been used. Segmented mass images are then used for extraction of deep feature-sets. The performance comparison of these DL-models, reports VGG19 model as the optimal model for deep feature-extractor. Deep feature-set is obtained using optimal feature extractor VGG19 model which may contain redundant values; therefore correlation based feature selection is employed to extract reduced deep feature-set. The performance of reduced deep feature-set is analyzed for multi-class classification using ML-based classifiers ANFC-LH, PCA-SVM and GA-SVM. The objective analysis of these CADs yields, VGG19 with ANFC-LH having highest estimated classification accuracy of 86% with individual class accuracy of 98, 80, 76% for BIRAD-3, BIRAD-4 and BIRAD-5 classes, respectively.