The widespread adoption of the Internet of Things (IoT) has led to a surge in image generation, with many being outsourced to the cloud to reduce storage pressures. To prevent from privacy leakage, it is wise to upload the encrypted form of images to the cloud server, however, encryption often leads to difficulty to image retrieval. Thus, the field of encrypted image retrieval has garnered significant interest for researchers. In this paper, a new encrypted JPEG image retrieval scheme using new adaptive images encryption algorithm and neural network is proposed. Specifically, the histograms of DCT coefficients are extracted from the cipher images as features vectors. These features vectors are then sent into a highly lightweight self-attention based neural network in retrieval process. Experiments results reveal that our scheme can attain enhanced retrieval precision with less computational cost compared to previous neural network based schemes. The lightweight self-attention networks (LSAN) we proposed can directly run with CPU, the GPU accelerated computing is not necessarily required. We upload our code at https://github.com/FrankZanyar/LSAN .