Evaluating the perceptual quality of omnidirectional videos is crucial for optimizing virtual reality (VR) experiences, as these experiences rely on immersing viewers in a 360 \(^\circ \) environment. Unlike conventional planar videos, 360 \(^\circ \) VR videos require users to perceive viewport-based content through head-mounted displays (HMDs), where they can look around freely, creating unique quality assessment challenges. Traditional video quality assessment methods are often insufficient as they do not account for the interactive and immersive nature of VR. To address this, we develop a unified model that supports both full-reference (FR) and no-reference (NR) quality assessment methods for 360 \(^\circ \) videos. Our model includes three main modules: a feature extraction module, a quality regression module, and a temporal pooling module. The feature extraction module utilizes a two-stream structure that examines the spatial degradation and motion characteristics of the video, capturing nuances specific to 360 \(^\circ \) content. The quality regression and temporal pooling modules are designed to simulate the human visual system, providing accurate predicted quality scores. When tested on the VQA-ODV dataset, our approach demonstrates superior performance compared to other state-of-the-art FR and NR methods, highlighting its potential to improve VR content quality and enhance user experiences.