Scene-Aware and Interpretable Video Captioning with TimeSformer-GPT2 for Real-Time Accessibility
摘要
The challenge here is to provide an interpretable, real-time video captioning pipeline that uncovers the accessibility and semantic indexing of multimedia content. Unlike existing methods that propose a single caption for the video, our method automatically divides the stream into visually homogeneous scenes using PySceneDetect and then captions every sequence from the common keyframe representatives. Finally, the final captioning of the generic story concludes with every appropriate semantic word filled in according to the spatio-temporal rhythm of the video. The model uses TimeS-former for spatio-temporal encoding and GPT-2 for autoregressive text generation and employs GPU acceleration when processing high-definition materials in an optimized way. The pipeline uses robust utilities at the preprocessing steps: FFmpeg for decoding and ImageMagick for rendering captioning, and it provides the result in interoperable codecs like SRT and JSON. The system ensures interoperability between the SRT and JSON formats. The system demonstrates robust performance on standard measures such as ROUGE-L, BLEU, METEOR, and CIDEr, as demonstrated in the VATEX corpus tests. With dynamically scene-ordered captions, the interpretability of deployment is facilitated by the system, and avenues are provided toward interpretable and low-resource video captioning approaches, deployable to real-world indexing and assistive systems.