We study categorical representation learning for abstract symbolic languages, where symbols are modeled as objects, relation types as morphisms, and tasks as functors that must preserve structure. From raw concurrence statistics we learn object embeddings and relation-conditioned morphism maps, yielding PMI-like, relation-aware representations rather than mere proximity. A task functor is constrained by a universal structure loss to commute with all learned morphisms. We instantiate the framework on a formal language rendered in two encodings. Despite very low supervision, the structure-preserving functor delivers high structural validity and improves token-level accuracy in the low-shot setting, outperforming a pointwise alignment baseline. A simple commutator residual diagnostic confirms that the learned functor preserves the algebra of relations, not just pointwise correspondences. The approach is compact and domain-agnostic, providing a lightweight inductive bias for structurally faithful translation whenever validity and compositionality matter but labeled parallel data are scarce.