We investigate the problem of active learning of runtime monitors for cyber-physical systems (CPS) under uncertainty. In CPS, runtime monitors need to make decisions with only partial information about the system state and cannot always rely on having a precise environment model. As a result, the learning process and resulting monitors must be able to handle this type of uncertainty. We present a framework for the active learning of monitors and discuss the challenges in implementing oracles for membership and equivalence queries. We particularly apply the framework to a setting where uncertainty models are defined by Markov decision processes. We present initial results demonstrating the efficacy of our approach in learning accurate monitors using a set of benchmarks from the domain of autonomous systems.