In this paper, we address the Visual-Audio Question Answering (VQA) task, which involves answering questions about objects, sounds, and their relationships in videos. Effective VQA requires capturing the presence and evolving relationships of subjects and objects over time, utilizing both video and audio modalities. We propose a novel framework, Audio-Visual Hyper-Graph VQA (AVHG-VQA), that constructs situation audio-visual hyper-graphs (AVHG) for answering video-related questions. AVHG offers a structured representation by detailing sub-graphs for individual frames and connecting them with hyper-edges, encapsulating relevant information compactly. Our framework trains a hyper-graph decoder to infer associations between people and objects from video snippets, using cross-attention between predicted AVHG and question embeddings to determine answers. The training process involves two stages: extracting relationships from video frames using a pre-trained scene graph model as labels and optimizing the model with cross-entropy and Hungarian matching loss functions. We extensively evaluate our framework on the challenging MUSIC-AVQA dataset, which focuses on video-audio modality information.