Low rank matrix factorization techniques, such as Non-negative Matrix Factorization (NMF) and Concept Factorization (CF), have attracted considerable attention in data analysis. However, both of them effectively only exploit the global Euclidean geometry, without fully considering the local manifold geometry. In this paper, we propose a novel sparse matrix factorization method for data clustering called Concept Factorization with Manifold and Orthogonal Constraint for Data Representation (CF-MO), which considers the geometric structures of the data manifold along with the sparse orthogonal constraint. CF-MO can discover the intrinsic geometric structure of the data, and lead to rigorous clustering interpretation by minimizing the redundancy between different orthogonal bases. Furthermore, we develop the iterative update rules for CF-MO, and prove the convergence of this algorithm. Experimental results demonstrate the effectiveness and performance of our proposed method compared to state-of-the-art algorithms on real-world images.