In the context of multimedia content of huge scale, effective and efficient video summarization has become more and more important for improving web browsing experiences and information overload. In this paper, we introduce a new video summarization method by modeling it as a sequential decision-making problem. A Deep Summarization Network (DSN) is proposed for predicting the probability of considering each video frame in terms of its possible contribution to the final summary. For training the DSN, we suggest an end-to-end deep reinforcement learning system that works in a fully unsupervised way, without requiring annotated labels or human intervention. Our approach has its foundation in a well-crafted reward function that assesses generated summaries along two primary dimensions: diversity and representativeness. These elements are designed to ensure that the chosen frames not only convey the main content of the video but also encompass a good number of visual and contextual variations. The DSN is trained to maximize these rewards, resulting in brief, informative, and non-redundant summaries. Extensive experiments are performed on two standard datasets to confirm the efficacy of the new method. The experimental results show that our method is capable of rendering competitive performance compared with state-of-the-art methods, especially under unsupervised scenarios, thus revealing excellent potential to be applied in real-world systems for large-scale video analysis and multimedia retrieval.