This paper presents a method leveraging Large Language Model (LLM) and Vision Language Model (VLM) for zero-shot video summarization which identifies semantic-important shots in a video. Our key idea is to utilize the great ability of LLMs in text understanding, adopting a pipeline of video-to-text, text-summarization and finally text-summary-guided video summarization. We first convert a video into text by employing VLMs. Then, with the help of LLMs, we obtain text summary. Finally, the text summary is used to compute the saliency score, guiding the summarization of the video. Although the pipeline is straightforward, fully transferring the text understanding capability of LLMs to video summarization is challenging. First, text generated by VLMs are occasionally damaged, which impedes LLMs’ understanding of the video content. We propose a method to clean the damaged text as the basis corpus to be processed by our pipeline. Then, calculating the saliency score by computing the similarity between each frame and the summary often leads to score oscillations. We propose a multi-grained saliency scoring method using a bi-level text summarization to stabilize the scoring process. With the multi-grained saliency score, our zero-shot method with no training, achieves state-of-the-art performance on two benchmark datasets (SumMe and TVSum) compared to previous unsupervised methods.