Enhancing Question Answering in Lecture Videos with a Multimodal Retrieval-Augmented Generation Framework
摘要
This paper addresses the challenges of searching and extracting information from lecture videos as found on online platforms. With the increasing popularity of educational videos, learners are faced with the growing challenge of video search and finding relevant sequences within videos to answer questions. This research describes the development of a framework that relies on Retrieval-Augmented Generation (RAG) to address this challenge with a focus on the nature of lecture videos, which usually rely heavily on audio data (voice of the lecturer) and image data (lecture slides) that can partially be extracted as text. The resulting artifact allows a user to ask a question based on a corpus of videos and receive an answer based on the videos’ contents, along with information regarding the most relevant videos and sequences within the corpus. This research illustrates that a RAG system leveraging textual context from lecture videos is effective at improving question answering accuracy compared to relying solely on a standalone Large Language Model (LLM) without contextual enhancements. Further findings revealed that boolean answer correctness can aid in optimizing hyperparameters, that the independent retrieval of text from audio and visual modalities outperforms more complex strategies, and that retrieval followed by re-ranking outperforms other strategies.