Classifying garbage in urban scene using multi-modal approach is a challenging task due to the complexity of the environment and diversity of collected data. Existing works usually extract features from multi-modalities, then combine them together to form final feature, which is used for classification later. Although these methods have achieved noticeable results, the correlation between different types of data might be lost, which can limit their performance. To address this problem, this paper introduces a multi-modal method for urban garbage classification with efficient feature integration. By fusing image data and each type of meta-data separately, then associating them to get final output, the proposed model can effectively classify several types of garbage. Extensive experiments show that the proposed method outperforms baseline models by a large margin.