In this paper, a novel approach for color-based image segmentation is proposed, using a combination of sigmoid and softmax functions for probability estimation. This method is designed to address challenges in color segmentation such as unclear boundaries and image noise. An energy function is constructed comprising two main components: a data term based on color distance and a smoothness term to ensure continuity in segmentation. Through the application of the sigmoid function, color distances are transformed into class membership probabilities, thereby enabling the generation of soft segmentations. Concurrently, the softmax function is applied for multi-label classification, creating probability distributions for various object classes. The model’s performance is evaluated on the VOC2012 dataset, demonstrating its effectiveness in improving accuracy and quality of object segmentation. Experimental results indicate that the proposed method achieves superior performance compared to traditional threshold-based and clustering approaches, particularly for objects with distinct color characteristics. The integration of morphological operations for post-processing further enhances boundary delineation and noise reduction. This probabilistic approach offers a robust framework for color-based image segmentation across diverse imaging conditions.