Semantic communication (SC) has emerged as a novel paradigm aimed at revolutionizing the communication landscape through the efficient exchange of semantics. However, existing approaches are limited by redundant communication and the need for joint training at transceivers. To address these limitations, we propose a framework called Goal-oriented Invariant Representation-based SC (SC-GIR). The proposed method leverages contrastive learning to obtain an invariant and meaningful representation of source data that is task-agnostic. This latent representation facilitates efficient communication while retaining core features crucial for downstream task execution. Focusing on machine-to-machine tasks, we utilize a covariance-based contrastive learning approach to derive semantically dense latent representations. To evaluate the performance of the proposed approach, we consider different datasets for lossy compression. These compressed latent representations are subsequently employed in a goal-oriented artificial intelligence (AI) task. Experimental results demonstrate that the proposed SC-GIR framework significantly outperforms baseline schemes and conventional approaches.