Recommendation systems effectively filter massive amounts of information to recommend items that meet users’ needs. Recently, Large Language Models (LLMs) have demonstrated to have the potential as zero-shot learners and the strong task-solving capabilities in recommendation systems. Its key is that prompt learning can enhance the recommendation ranking ability of LLMs. Although there is a number of recommendation researches using LLM prompts, it is still not clear that the performance differences among various prompt learning techniques in terms of zero-shot ranking abilities. In this paper, we conduct a reproducibility study focusing on prompt-based zero-shot learning in LLMs for recommendation. Specifically, we reproduce six popular non-LLM recommendation models and four LLM methods using prompt learning. Extensive experiments on two widely used recommendation datasets shows that prompt learning can trigger LLMs to perceive interaction orders, and Chain of Thought (COT), a step-wise prompt strategy, can instruct LLMs to further improve recommendation performance by combining user preference and representative items. Moreover, our findings is LLMs have promising zero-shot ranking abilities while suffer from biases. This reproducibility study provide guidance and benefits for the development and application of prompt learning in recommendation research.