In many text corpora, critical but low-frequency themes like mental health discussions are often overshadowed by more prevalent topics. Conventional methods often fail to effectively discover these minority patterns, unless strong prior signals are provided. Clustering methods aren’t much better; they tend to group documents by whatever is statistically prominent, hence they tend to favor dominant patterns. We propose COCNMF (Cluster-Oriented Constrained Non-negative Matrix Factorization), a joint model that integrates topic modeling and document clustering through a structured matrix factorization in a single optimization process. Unlike prior works, we do not enforce hard constraints or rely on strong seed word-based supervision. We instead softly guide one part of the model on the clustering structure and leave the other to freely learn from the data. We compare the ability of the model to find clusterings relevant to the minority themes, versus several state-of-the-art baselines, on a new benchmark where snippets of minority themes are injected into news articles. COCNMF significantly surpasses all baselines in clustering quality metrics, particularly in the identification of low-prevalence but semantically meaningful topics.