Hashing methods have captured increasing attention in image retrieval due to their computational efficiency and low storage cost. Most existing supervised hashing approaches focus on preserving the semantic similarity for discriminative hash codes learning. However, they may not discover intrinsic latent features embedded in high-dimensional feature space, and ignore the underlying affinity structures of data. In this paper, we proposed a novel Affinity based Semantic Collaborative Hashing (ASCH) method for image retrieval, which obtains the hash codes by the collaboration of latent discriminative features and semantic labels. Specifically, ASCH explores the data correlations by learning a self-representation based affinity matrix. To facilitate the semantic latent features learning, an asymmetric strategy is designed by factorizing an affinity matrix into the inner product of low-dimensional label matrix and latent feature matrix. Thus, the learned latent features not only explore the underlying low-rank data structure, but also encode the semantic class information. The hash codes are jointly learned in a unified framework by collaboratively projecting the latent features and labels. Based on the hash codes, a flexible regression model with adaptive marginalization is used to learn the hash functions for out-of-sample extension. Our proposed model can be efficiently solved with linear time complexity. We compare the ASCH with state-of-the-art methods across five benchmark datasets. The experimental results validate the feasibility and superiority of our proposed method.