Comprehending social interactions involving multimodal cues is crucial for accurately interpreting complex social scenarios. Existing methods often rely on speaker-centric features, overlooking non-verbal behaviors from other participants, which limits their effectiveness in multi-party settings. To address this, we propose a multimodal framework that integrates gesture and gaze features from all participants. A cross-subject attention mechanism is introduced to dynamically assign semantic importance to individuals, enabling selective focus on informative cues. We evaluate our approach on the representative benchmarks across three tasks: speaking target identification, pronoun coreference resolution, and mentioned player prediction. Experimental results show improvements over baselines, which proves the effectiveness of the multi-participant feature integration and the cross-subject attention for social interaction understanding.