Object detection algorithms aim to classify and detect object class instances in images or videos. The effectiveness of these object detectors is due to substantial improvement in the deep convolutions neural networks. However, very few attempts have been made to explore the positive and negative object regions and to differentiate between similar objects with a distracting background in challenging scenarios. To achieve optimum training and completely leverage the capacity of model architectures, it is important to mitigate inter-class and intra-class variations during the training of the classifier to improve accuracy. To improve the accuracy, we proposed a new framework, using InceptionResNet-V2 and Resnet101 models as a backbone with a triplet loss function. The proposed framework learns the mapping from images to compact Euclidean distance, where the distance directly corresponds to a measure of object similarity. This new framework backbone trained with triplet loss function can be plugged into any detector. In our case, we selected SSD and replaced the original backbone network VGG-16 of SSD. The triplet loss function improves the classification performance and improves accuracy which leads to efficient object detection. Moreover, extensive experiments on the PASCAL VOC 2007 and PASCAL VOC 2012 datasets show the efficacy and enhancement of the proposed method, by comparing it with other states of the art methods.