Traditional Visual Question Answering (VQA) systems often struggle with new images or queries outside their training data, limiting their adaptability and practical use. This project addresses the zero-shot VQA challenge by developing a system that can intelligently interpret and respond to visual content without extensive pretraining on specific datasets. The system utilizes pre-trained models, such as SAM and BLIP 2, to generate captions for image patches. These captions are then used to extract relevant question-answer pairs, which are evaluated using a heuristic approach. Finally, a Large Language Model, such as facebook/opt-6.7b, predicts the final answer by leveraging the captions, user questions, and question-answer pairs. This multi-step approach aims to provide accurate responses without relying on extensive pretraining on specific datasets. By combining advanced techniques and novel methodologies, the project enhances the capabilities of VQA systems and paves the way for more robust and versatile AI applications in image understanding and interpretation. The system’s flexibility is further demonstrated through its two main implementations. The first implementation uses BLIP 2 to generate captions for images, where users can click on specific objects to create a mask for targeted captioning. The second implementation, utilizing an API from Mistral AI, allows users to specify preferences for factual, imaginative, positive, or negative captions, and supports visual question answering. This comprehensive system addresses the limitations of traditional VQA approaches and significantly improves the performance and versatility of visual question answering.