Dietary nutrition tracking is crucial for personalized healthcare. Traditional methods, including weighing, dietary review, and food frequency, have limitations of low operational efficiency, availability for small samples, and noise from the patient’s memory. Thus, it is essential to have an efficient and accurate method for tracking patients’ nutrition. First, we utilized the U-net-based semantic segmentation method to complete the deep learning process with dietary food images, encompassing feature extraction, category recognition, volume estimation, and weight data collection of patient foods. Second, a data-driven goal optimization model is built to achieve the multiple nutrient intake goals and balance the body’s energy requirements. Furthermore, considering the uncertainty set of dietary nutrition, a data-driven robust optimization algorithm is used to solve the optimization model under noisy data. Finally, we verified the effectiveness and robustness of the proposed method with real-world nutrition data, advancing the personalized services of nutrition tracking.