Scene understanding is a significant issue for maritime industries. Due to the large ranges involved, a model capable of localization at both short and long distances is particularly important. Currently, modern data sets have neglected the proper collection procedure and hardware to accommodate the scales of ranges at which small vessels appear. Due to other factors such as environmental challenges and data scarcity, current efficient deep learning techniques also fail to achieve high accuracies. We present High Resolution Collection Above Water (HR-CAW), a high-resolution dataset for realistic vessel segmentation, as well as addressing limitations with current data-driven approaches with a novel architecture, HC-PatchNet. Our results show that due to extreme data imbalance and diverse weather conditions, most methods for efficient high-resolution segmentation fail to converge, whereas the proposed method succeeds and achieves state-of-the-art performance.