For drone-based Multi-Object Tracking (MOT), camera motion makes it challenging to maintain accurate object prediction and tracking on the image plane. In this study, we introduce an approach that leverages camera motion information to predict object motion in three-dimensional space and back-project it onto the two-dimensional image plane, resulting in more robust and stable data association. Furthermore, we integrate a single object tracker into our framework and use object motion prediction to guide its search region for more stable tracking. The evaluation through both simulation and real-world experiments demonstrates that our method outperforms other popular multi-object trackers. The source code and experimental data are released at https://github.com/npu-ius-lab/DbMOT .