Accurate modeling and prediction of metro passenger flows are critical for enhancing operational efficiency and improving passenger experiences. Two key prediction tasks in this domain are Inflow/Outflow (IO) and Origin-Destination (OD) flow prediction. While existing studies primarily focus on these tasks independently, they overlook the interconnected nature of IO and OD flows, which can be leveraged for joint prediction to capture more comprehensive spatiotemporal dynamics. However, effective information sharing and the resolution of optimization conflicts among tasks remain challenging. To address these challenges, we propose a novel multi-task learning framework, the Multi-Task Feature Aggregation Module (MTFAM), for joint IO and OD flow prediction. MTFAM comprises spatiotemporal encoders and cross-task interaction modules. The encoders, built with Graph Convolutional Gated Recurrent Unit(GCGRU), extract spatiotemporal features from IO, OD, and Destination-Origin (DO) flows, with an additional matrix completion step for OD flows. Cross-task interaction is facilitated through multi-head cross-attention mechanisms and a fusion gate, enabling effective information exchange and synergy between tasks. Furthermore, a balanced loss function is introduced to mitigate optimization conflicts, enhancing prediction performance. Extensive experiments on two real-world metro datasets demonstrate that MTFAM outperforms state-of-the-art baselines across various accuracy metrics, validating its effectiveness and the contributions of its components.