We present a novel approach to analyzing built environment variables (BEVs) using deep learning and Google Street View (GSV) images. By identifying and classifying BEVs, we aim to assist architecture professionals in understanding the relationship between heat-related health risks and BEVs. Traditional methods require extensive finetuning with human-labeled datasets, which is inefficient for analyzing diverse BEVs. Our approach integrates open-set object detection models with vision-language models to accurately identify buildings and classify wall materials without additional finetuning on our own human-labeled datasets. This versatile model can efficiently handle mixed materials, offering a cost-effective and scalable solution for analyzing the comprehensive built environment. The results will support architecture professionals in developing effective mitigation strategies for vulnerable populations living in less resilient housing, addressing public health risks associated with climate change.