The exponential rise of internet memes, often humorous images with embedded text, across major social media platforms like Facebook, Instagram, and X (formerly Twitter) has sparked significant attention in recent years. This phenomenon, while entertaining for many, has brought to light a daunting challenge, the high occurrence of hate speech on these platforms. Addressing this challenge has become a communal responsibility, placing considerable pressure on social media companies to mitigate the spread of harmful content. In response to this escalating concern, our research introduces an innovative solution named “mCLIP,” which is a variation of CLIP. mCLIP employs a multimodal approach to effectively classify memes based on their levels of offensiveness and positivity. This paper discusses the advancement and execution of mCLIP designed for multilevel and binary classification of memes by focusing on evaluating its effectiveness in distinguishing between harmless humor and potentially harmful content. By utilizing advanced techniques in multimodal analysis, mCLIP contributes to the ongoing conversation about creating a safer and more responsible digital environment. We have performed our task on SemEval 2020 task 8 dataset [20] and our results outperformed SOTA models. Furthermore, the outcomes highlight the importance of taking proactive steps to deal with the issues raised by the changing nature of internet memes on social media.