Tracking by detection is an effective approach to addressing the multiple object tracking problem. Detections are extracted and matched across the different frames of a video. However, detection errors persist, leading to false negatives that degrade tracker performance. In this work, we propose an architecture to overcome detection failures. Instead of using bounding boxes, which lack precision in crowded situations, we propose obtaining and tracking segmentation masks for each object. Results on the MOT20 crowded dataset demonstrate our ability to improve the performance of state-of-the-art methods.