Compared with matrix factorization based methods, metric learning based recommendation methods can better capture the relevance of user-item pairs and the similarities of item-item pairs. However, existing metric learning based recommendation methods ignore the influence of sample difficulty during model training, which results in a lack of flexibility in parameter learning. In addition, these methods focus on minimizing a pairwise loss and assume the preferences of users are independent while ignoring useful collaborative information for recommendation. To address the above problems, we propose a collaborative adaptive metric learning (CAML) based recommendation method. To better learn the model parameters, it adaptively updates the model parameters according to the difficulty of samples with consideration of their categories. Furthermore, it adaptively integrates the losses of multiple partial order relations to construct a listwise loss with consideration of the collaborative information for personalized recommendation, which can also solve the problem of inconsistent convergence rates of different losses. Extensive experiments on four real-world datasets demonstrate the effectiveness of the proposed method compared with state-of-the-art methods.