Intelligent robots have been widely used in many fields, which can help humans greatly improve work efficiency. However, mainstream methods tend to use imitation learning or reinforcement learning, assuming there are expert demonstrations or priors to train a generalist, which is costly and time-consuming. Benefiting from large models, we propose DenseRL for zero-shot robot manipulation. Through the combination of pretrained vision-language models, DenseRL can provide multivariate zero-shot visual rewards. More importantly, DenseRL only uses frozen pretrained models without any finetuning. Experiments on MetaWorld show that DenseRL can achieve the SOTA performance among zero-shot baselines, in which the average success rate is 16.7% higher than the best baseline.