We present Human Comfort Classifier (HCC): A framework for classifying human discomfort from video. Recognizing comfort and discomfort in social interactions is something that many of us do without having to think about it. However, identifying discomfort in others can be a challenge for individuals with social skills deficits, who often become socially isolated. Social isolation can lead to many negative outcomes for individuals and is recognized by the CDC and WHO as a priority public health problem. In this work, we propose HCC to detect discomfort in videos. This can be utilized for training for individuals with social skills deficits. HCC utilizes a multi-modal approach of pose estimation, facial landmarks, and natural language processing to determine comfort in real time. We utilize an explainable rule-based model to categorize behavior and achieve approximately 78% prediction accuracy on an interview dataset.