Mixture models are a prominent choice for unsupervised learning methods that comprise multiple factors, including feature representation/extraction and parameter estimation methods. In this paper, these two factors are improved. Concerning the first factor, a simultaneous hierarchical feature extraction and clustering framework is developed using the concept of feature saliency and Hierarchical Feature Learning (HFL) using the Spatial Pyramid Matching (SPM) method. The kernel distribution is a generalization over the well-known Dirichlet distribution, namely, the Multinomial Nested Dirichlet distribution (MNDD), which naturally realizes the hierarchy of the represented data. Concerning the parameter estimation factor, the above structure parameters are estimated through the Minorization-Maximization (MM) framework as an alternative to the well-known Expectation-Maximization (EM) method. As the EM method has an added complexity and requirements, the MM relaxes these limitations with a potential higher flexibility. This is shown in the results section through the three visual datasets used to verify the improvement of the novel framework. Moreover, the Minimum Message Length (MML) is used to determine the number of components.