Classifying similar inflammatory skin conditions poses significant challenges for dermatologists. We propose a deep learning approach using transfer learning with pre-trained CNNs to address this issue. By integrating anatomical information through feature fusion, our aim is to enhance diagnostic precision. Among various models evaluated for skin disease classification, MobileNetV3-Large performs excellently with the highest F1-score and minimal computational time. In particular, binary classifications for types of dermatitis, particularly atopic dermatitis and psoriasis lichen planus, proved to be the most challenging. Using feature fusion with anatomical information in a supervised learning setting, SVM emerged as the best performing model, increasing the F1-score from 0.82 to 0.88. This study highlights the importance of integrating limited and anatomical information to improve the diagnosis of complex skin conditions, paving the way for more accurate diagnostic tools for dermatology.