We propose a novel alternative to traditional randomly sampled mini-batches for gradient computation: using a fixed subset for complete pretraining of a neural network model. This approach enables deterministic convergence instead of a merely probabilistic one, as proven by the stochastic approximation theory, whose prerequisites are frequently violated by popular optimization algorithms. The approach is justified by the hypothesis that the loss minimum of the training set can be expected to be well-approximated by the minima of its subsets. Such subset minima can be computed in a fraction of the time necessary for optimizing with the whole training set. They are also compatible with efficient second-order optimization methods, such as the conjugate gradient optimizer. These methods are particularly efficient in the convex environment of the loss minimum. The image classification datasets MNIST, CIFAR-10, and CIFAR-100, (optionally extended by augmentation of training data) test this hypothesis. The experiments confirm that the models achieve performance equivalent to that when trained with the conventional training scheme. In conclusion, if the overdetermination ratio for the given model and dataset sufficiently exceed unity, even small subsets are representative. This results in a possible reduction of the computing expense to a tenth or less. This paper is an extended version of Spörer et al. [13].