One approach towards handling the large resource requirements of modern neural networks is to use structured weight matrices. In this paper, we analyze the approximation capabilities of such neural networks. In particular, we investigate sequentially semiseparable (SSS) matrices with one dimensional state variable. This class of matrices is quite limited in their expressiveness, but it facilitates an efficient matrix-vector multiplication algorithm. Our contribution is to prove that neural networks comprising SSS matrices with one dimensional state variable are universal approximators. With our proof, we show that the same approximation capabilities which have been shown for weight matrices of low displacement rank also apply for SSS weight matrices.