<p>Crop weed segmentation is one of the most challenging tasks in the field of computer vision. This is because, unlike other object detection or segmentation tasks, crop and weed are similar in terms of spectral features, shape, dimensions, etc. For precision agriculture to flourish in terms of smart spraying of crops, efficient systems to distinguish between crop and weed are the need of the hour, which if precise, will take a huge step toward solving the issue of food scarcity. To tackle this issue, we propose new ensemble architecture of two models—a U-Net with a modified backbone and an encoder–decoder. These networks learn to distinguish between soil and crop and soil and weed, respectively, whose ensemble gives state-of-the-art results on pixel-wise annotations of combined crop and weed images. Moreover, it also learns that the model captures un-annotated features since each component of the architecture learnt either crop or weed features to high precision. Finally, the proposed architecture is compared with the U-Net and SegNet, which are popular segmentation networks, and consistently achieved better results.</p>