Breast cancer (BC) aggressiveness significantly impacts patient prognosis by influencing relapse, metastasis, and mortality rates. Currently, the assessment of tumor aggressiveness is primarily dependent on biopsies and pathological image analysis, which are challenging to obtain during follow-up stages after treatment, particularly if there is a relapse. Our study introduces a novel framework utilizing digital mammography, the standard for early breast cancer detection, to classify tumor aggressiveness non-invasively. This framework employs a Cascaded ResUNet with boundary-aware capabilities for precise tumor segmentation, ensuring accurate delineation of tumor boundaries. Key morphological and texture features are then extracted from the segmented regions to categorize tumor aggressiveness. Validated on an in-house dataset, our method achieved an accuracy of 90.6% and an Area Under the Curve (AUC) of 87.4%, with further tests on an external dataset confirming its generalizability with 88.7% accuracy and an AUC of 81.0%. These promising results underscore the potential of our approach for clinical applications, offering a robust method for the classification of breast tumor aggressiveness in scenarios where invasive procedures are impractical.