Although the idea of affective computing was proposed two decades ago, it is still not used to its full potential. This paper delves into its application in video games by introducing a gameplay personalization system based on machine learning solutions. Specifically, we perform perceived facial emotion recognition using Deep Convolutional Neural Network (DCNN) to infer the player’s emotional state from single-modality recordings and dynamically adjust the game difficulty. Our fine-tuned Xception model achieved F1-score equal to 70.46%. We focus on enhancing the robustness of current approaches against low-quality images motivated by the use of consumer-grade cameras and the limited network bandwidth, achieving a 3.08% improvement in F1-score. Additionally, we implement a simple affect-aware game based on Pong and conduct a study (N = 102) concerning the accuracy of emotion recognition and the overall experience of the game. Results indicate an increase in perceived difficulty (MOS 2.46 vs. 2.40) and enjoyment (MOS 3.22 vs. 3.17) for the affect-aware version.