The problem of similarity-based retrieval, in which a server retrieves a vector from a database that is most similar to a query of a client, is a fundamental problem for many applications. Fully Homomorphic Encryption (FHE) supports computations over encrypted data and thus can be used to preserve the privacy of the query and database during the similarity-based retrieval process. However, existing works that tackle the problem of similarity-based retrieval over FHE typically rely on sending an encrypted vector containing several computed similarity scores from the server to the client. This client-aided approach exposes too much information on the dataset, while also incurring high communication bandwidth that is linear in the size of the dataset. In this work, we present a similarity-based retrieval system in which the server sends the client only one ciphertext containing the retrieved entry, thus not exposing additional information on the dataset while improving the communication bandwidth to be constant. We conduct empirical experiments in which we perform similarity-based retrieval over a dataset of half a million encrypted vectors in less than 30 s with accuracy of 98.9%–99.9%, and improve the communication bandwidth by \(8 \times \) in the case of a single query vector and by \(512 \times \) in the case of a batch of 64 query vectors.