With the increasing adoption of edge computing, the necessity to offload tasks to edge servers is growing. Task dispatching becomes crucial in multi-server environments. Nevertheless, this process encounters difficulties due to the real-time acquisition complexities of communication and computation latency. Prior efforts have explored these latencies, but they presume easy acquisition of latency metrics, which is unrealistic in dynamic edge networks. To overcome these challenges, we introduce SmartTask, an innovative edge task dispatching system employing responsive network tomography to assess communication latency and deep neural networks to predict computation latency. To optimize the balance between dispatching speed and solution accuracy, SmartTask utilizes reinforcement learning for task scheduling. Additionally, SmartTask integrates curriculum learning to improve the performance and generalization of reinforcement learning. Evaluations indicate that SmartTask decreases end-to-end latency by an average of 26–86% compared to the baseline.