More efficient methods for mitigating disaster and risk are constantly sought in earthquake-prone countries. In the Philippines, numerous frameworks for assessing building vulnerability have been integrated into common practice and continue to be developed until now. The process of visual assessment, however, may still be further expedited to avoid additional costs and effort in the assessment process. The current study capitalizes on the growing field of machine learning and seeks to find out if the visual assessment of buildings can be done using a trained convolutional neural network. The research classified Google Street View images of buildings in the Greater Metro Manila Area according to their height; identified out-of-plane setbacks, soft stories, split levels, and short columns; and entered this data into an optimized ResNet50 network. The hyperparameters were obtained through Bayesian optimization, and its performance was compared to a base network with training hyperparameters obtained from a past research. A total of 2100 images were obtained. The results showed that (1) there were significant imbalances in the overall image data set; and (2) the optimized networks were able to best identify three out of the five classifications, excluding soft stories and short columns. The main source of error was associated with the lack of statistical analysis as only the averages of accuracies and F1 scores of the networks were compared; the study may thus be unrepresentative of the accuracy of the trained networks. Hence, it is recommended that future studies apply techniques to mend the class imbalances and perform statistical analysis via student’s t-test or ANOVA (among others) for a more grounded conclusion. Nonetheless, there is promising potential for Bayesian optimization in automating building categorization, and it remains to be a systematic process for obtaining hyperparameters for classification tasks.