Convolution in the spectral domain is a linear-time computation, making it possible to train convolutional neural networks (CNNs) with significantly larger kernels than their spatial counterparts, which typically are limited to \(3 \times 3\) or \(5 \times 5\) kernels. To solve this, continuous kernel networks have been proposed. Continuous kernel networks replace fixed-size kernels with generating functions, enabling kernels with global context while maintaining a manageable parameter count. However, learning continuous kernels is prohibitively expensive, even for modest-sized datasets. In this work we propose an optimized method for efficiently training arbitrarily large MLP-backed kernels directly in the spectral domain. Our method addresses three key challenges in learning continuous kernel representations: computational efficiency during training, parameter efficiency in representation, and training speed. We evaluate spectral domain continuous kernel CNNs on medium sizes image classification datasets while achieving comparable accuracy to that of their spatial domain counterparts.