Visual tracking in complex videos is challenging due to abrupt environmental variations. Handcrafted features are not able to address such variations efficiently. In order to provide robust tracking solutions, we propose to integrate handcrafted color features with the deep features extracted using convolutional neural networks. The features are fused using feature fusion which captures the sensitive relationship between them effectively. The classifier network classifies the positive scores from the negative scores and localizes the target accurately in the presence of heavy occlusion and dense background clutters. Exhaustive experimental analysis and comparison of the proposed tracker with five other recent trackers on video sequences from the OTB dataset proves it robustness and efficiency to address tracking challenges.