Reinforcement learning (RL)-based recommender systems have gained significant attention in recent years. However, the design of an effective reward function, which guides the optimization of the recommendation policy, is often challenging. Instead of relying solely on the reward function, exploring the causal factors underlying user behavior can be a promising approach to capturing dynamic user interests. Additionally, the limitations of simulation environments, such as data inefficiency, hinder the widespread application of existing methods in large-scale scenarios. Although some attempts have been made to convert offline datasets into simulators, the learning process becomes slower due to data inefficiency. Moreover, traditional RL algorithms lack the ability to learn directly from offline datasets, unlike supervised learning methods. In this paper, we propose a novel model called the Deep Distributional Offline Reinforcement Learning for Recommendation (DDRL4Rec). DDRL4Rec is an offline RL system that learns the distribution from datasets and does fine-turn during online interactions. In order to demonstrate the superiority of our model, we conducted extensive experiments on six real-world offline datasets and one online simulator.