Transformer models have established themselves as the state-of-the-art deep learning architecture in the field of natural language processing (NLP). Yet, despite huge quantities of data and compute, unsupervised sentence embeddings derived from raw pre-trained states are just little better than random embeddings. In this work, we try to reduce this gap even further by improving sentence vectors derived from random embeddings. In particular, we investigate the effects of different tokenization and vector sizes for semantic textual similarity, short text clustering, and classification tasks. The proposed methods show substantial improvements for classification due to the size of embeddings and also the effect on the tokenizer choice. Our work reinforces the random vectors method as a good and simple baseline. It also shows that for sentence vectors, the performance gap between pre-trained transformer features and simple random vectors is even smaller than previously thought.