Time is a fundamental dimension in the evolution of real-world systems, yet it is not explicitly represented in standard neural networks, which rely on discrete layers and transformations. Neural Ordinary Differential Equations (NODEs) address this limitation by incorporating continuous-time modeling into deep learning. By defining feature dynamics with ordinary differential equations, NODEs provide a natural framework for representing processes that evolve over time. This makes them particularly effective for handling irregularly sampled data and modeling dynamical systems. However, NODEs face critical challenges, such as limited approximation capabilities, sensitivity to adversarial inputs, and high computational cost. This survey examines these challenges in detail and categorizes the methods proposed to overcome them. It also reviews a wide range of applications, including time-series forecasting, biological modeling, and graph learning. Finally, we outline promising directions for future research, aiming to advance the use of NODEs in continuous-time machine learning. This work offers a comprehensive overview of NODEs, bridging theoretical foundations with practical applications to support further developments in the field.