Semantic segmentation is a well-known task in computer vision, with many applications in autonomous navigation. This task aims to partition the image (set of pixels) into multiple labeled subsets of pixels, often referred to as regions. Such a segmentation is key to the scene understanding. This also results in localization of the objects in the image. There exists many research papers that study semantic segmentation in outdoor (often road) scenes. Semantic segmentation of small objects is specially challenging, and critically useful. However, this did not get sufficient attention in the past. The problem is challenging due to (i) availability of only small number of pixels on the small objects, (ii) class imbalance in learning, etc. These make the today’s deep learning architectures less effective in performance when it comes to small objects. This can be directly observed from the fact that semantic segmentation performance on small objects in many popular datasets has low accuracy. In this paper, we investigate the challenges associated with and the directions for design of algorithms for segmenting small objects. It is well known that loss functions affect performance in semantic segmentation. The segmentation of small objects also depends heavily on the loss functions used in training the deep learning-based solutions. We also investigate this in this paper. This problem is more severe in Indian situations where many small objects are often seen on the road. The small objects (such as pedestrians) are also very important to segment out for autonomous navigation and driver-assistance systems. In the Indian driving scenario, the Indian Driving Dataset (IDD) provides a class of annotated small objects. These are captured and annotated in an unstructured environment. Thus, we focus primarily on segmentation of small objects in IDD.