This chapter presents a comprehensive review of datasets and evaluation protocols in visual object tracking (VOT). It begins by categorizing datasets based on sensory modality, covering single-modality datasets such as RGB, thermal, LiDAR, and event-based sources, and extending to multi-modal datasets like RGB-depth, RGB-thermal, RGB-LiDAR, and RGB-language. For each type, the chapter outlines representative benchmarks, data characteristics, and application scenarios. It also explores the fusion strategies employed in multi-modal datasets and their impact on performance in complex environments. Following the dataset overview, the chapter introduces standard evaluation protocols used in VOT research. Key performance metrics such as precision, success rate, expected average overlap (EAO), and area under the curve (AUC) are explained, along with evaluation procedures including One Pass Evaluation (OPE), Temporal Robustness Evaluation (TRE), and Spatial Robustness Evaluation (SRE). By summarizing both dataset benchmarks and evaluation methodologies, this chapter provides a structured perspective on the current landscape of VOT research and supports the development of robust and generalizable tracking algorithms.