In the global market competition environment, fabric enterprises should enhance their rapid response capability to adapt to the current market situation of tiny batch, multiple varieties, and fast delivery time. The image retrieval technology can be applied to the management of yarn-dyed fabric to simplify design. Along with the evolution of computer vision technology, deep convolutional neural network model is used to extract the high-level abstract features of fabric images. However, during retrieval, the features of retrieved images need to be matched with high-dimensional features using distance calculation function, which cannot achieve the real-time requirements of retrieval in larger scale fabric image retrieval system based on fuzzy vector index search is proposed. Firstly, the high-level abstract feature vectors of fabric images are extracted with pre-trained resnet18 model, and then the codebook space and inverted index table are constructed by quantized and compressed feature vectors through PCA, K-means, Scalar Quantization and other techniques to improve the retrieval accuracy and retrieval speed of images. Eight groups of experiments are designed to obtain the parameters when applicable to a million-level fabric image library. The retrieval time for each image requires 1 ms, and the random mean average retrieval precision (RMAP) and random average recall rate (RAR) are 96.01% and 75.88%, respectively, outperforming existing methods while using only 2.74%–15% of the memory required by baseline models.