Electron microscopy (EM) combined with energy dispersive x-ray (EDX) imaging (or ‘ColorEM’) of cells and tissues provides ultrastructural insight complemented with elemental context. The resulting hyperspectral datasets can be used to map the relative abundance of specific elements or subjected to more data-driven approaches such as spectral mixture analysis or clustering to highlight the ultrastructural components of interest. Despite the benefits of automatic segmentation over manual annotation, EDX imaging is two orders of magnitude slower than EM imaging precluding its routine use for segmentation. Large-scale ColorEM, however, does generate sufficient annotated labels, which we use as ground truth to train U-Net models, and thus enables the transfer of these labels to conventional EM data. Here, we present ColorEM-Net, a label-free segmentation technique based on features obtained from unsupervised clustering of ColorEM data. ColorEM-Net achieves label-free identification with over 95% accuracy for nuclei, lysosomes and exocrine granules. However, with an accuracy of 79%, the recognition of endocrine granules needs further effort in training for reliable segmentation. By reusing open-access ColorEM datasets, this approach facilitates automated segmentation of EM data, while eliminating the need for manual annotation and achieving scalability for tissue-scale segmentation.