Pedestrian thermal comfort in urban outdoor spaces is heavily influenced by urban morphological characteristics. Drawing on a dataset of 100 measurement points from ten pedestrian streets in Shanghai from late spring to early summer, this study collected urban environmental data through “thermal walking” experiments. By integrating multi-source data—including 360° panoramic street-view semantic segmentation, on-site thermal environment measurements, and meteorological station data—within a machine-learning framework, we examined how urban morphology affects the mean radiant temperature (Tmr) and pedestrian thermal comfort votes (TCV). Using the DeepLab V3 model (trained on the open-source Cityscapes dataset), we extracted pixel proportions of buildings, vegetation, canopies, ground surfaces, and sky from panoramic images. Univariate analyses were then conducted to identify the influence of these morphological elements on Tmr and TCV. Building upon these insights, a random forest regression model, coupled with Shapley Additive explanations (SHAP) analysis, was employed to rank, and quantify key influencing factors. Results indicate that the urban meteorological air temperature and sky proportion in an image predominantly determine Tmr, while vegetation and buildings evidently lower Tmr and enhance pedestrian thermal comfort. These findings offer empirical evidence for climate-responsive urban design, underscoring the importance of prioritizing street greenery and shading amenities to improve outdoor thermal comfort and public health in hot weather.