Compute-in-memory (CIM)-enabled crossbar arrays made of various kinds of emerging non-volatile memory (NVM) devices have been widely considered for implementing neural network algorithms in edge artificial intelligence (AI) applications because they operate very fast, while consuming low energy. However, non-idealities of synaptic long-term potentiation (LTP) and long-term depression (LTD) characteristics of these devices are known to affect the on-chip inference and on-chip learning accuracy of this CIM hardware. Here, in this context, we first develop a novel, open-source system-level simulator, known as “CIM-Sim" to simulate CIM hardware while including various synaptic non-idealities (limited bit resolution, non-linearity and asymmetry of LTP and LTD, cycle-to-cycle variation, and device-to-device variation). Next, using “CIM-Sim”, we study the impact of all these aforementioned non-idealities on classification accuracy of a fully connected neural network (FCNN) and a convolutional neural network (CNN), with the hardware used both in inference-only and on-chip learning modes. We have used both conventional AI data sets like MNIST and CIFAR-10 as well as edge AI data sets like Roshambo (gesture recognition from electromyography (EMG)) for the purpose. Finally, we identify various emerging NVM technologies which satisfy the above synaptic requirements, based on recent experimental reports. Using “CIM-Sim”, we then compare the accuracy, total time, and total synaptic programming energy for on-chip inference and on-chip learning of CIM hardware made of these NVM synapse devices.