<p>Crystalline constructions known as Cayley-Schreier lattices have been suggested as a platform for realizing arbitrary gauge fields in synthetic crystals with real hopping amplitudes. Here, we reveal that Cayley-Schreier lattices can naturally give rise to implementable lattice systems that incorporate non-Abelian gauge structures transforming into a space-group symmetry. We show that their symmetry sectors can be interpreted as blocks of pseudospin models–some of which correspond to true spinors–that can realize a wealth of different topological invariants in a single setup. We underpin these general results with concrete models and illustrate how they can be implemented in technologically available experimental platforms. Our work sets the stage for a systematic investigation of topological insulators and metals with non-Abelian gauge structures.</p>