Affective states are highly subjective and difficult to infer in a general setting. Making predictions without personal context is often infeasible due to the complexity introduced by individual traits and preferences. Therefore, personalization is vital for affective state recognition. We survey 41 papers and systemize the domain into four approaches to personalization: (1) implicit or (2) explicit using separated models, (3) explicit subject identification, and (4) models with general features. We also describe three machine learning components that can be utilized to improve fit to particular subjects: data processing, training procedures, and algorithm or model design, including future adapters. As validation is an essential component of the methodology, we discern three common approaches to data splitting: over samples, subjects, and time. Additionally, we identify some incorrect validation implementations that can lead to overfitting. Lastly, we discuss all identified approaches to personalization, data processing issues, training procedures, reasoning model design including deep architectures, validation, and ethical concerns.