Abstract <p>In various applications of optical communications, such as quantum key distribution (QKD) systems, the task of polarization control in optical fiber arises. A polarization controller (PC) on the receiver side is used to solve this problem. In our work, we investigate two machine learning approaches to polarization control: supervised learning (SL) and reinforcement learning (RL). We generalised analytical solution of the polarization control problem using SL-approach. Supervised learning algorithm was trained on simulations and then was validated in experimental setup. We compared supervised approach with analytical solution. The advantage of RL approach based on the capability of approximating nonlinear effects related to the imperfections of detection process. For the effective operation of polarization control algorithms, knowledge of the parameters of the PC is required. Reinforcement learning agent also eliminates control hardware calibration procedures, since the agent recognize necessary system parameters based on the experience of interacting with the environment. RL-agent was firstly pretrained on simulations and then was fine-tuned on the practical QKD system. We demonstrated that RL-agent’s tuning provides the 10% quantum bit error rate (QBER) on our setup, and converges in 10 steps.</p>