Brain decoding is a pivotal topic in neuroscience, aiming to reconstruct stimuli (e.g., image) from brain activity (e.g., fMRI). However, existing methods rely on subject-specific modules and flatten 3D voxel grids, limiting generalization and discarding spatial information. To address these issues, we propose MindLink, a scalable cross-subject brain decoding framework designed to link multiple subjects into a single model by extracting subject-invariant features while preserving the spatial structure of 3D fMRI data. This is achieved by parcellating 3D fMRI into standardized cubic patches processed by a 3D Vision Transformer for informative representations. Domain adversarial training enhances cross-subject generalizability by extracting subject-agnostic features within a single model structure. We also introduce a two-level alignment strategy that effectively bridges fMRI and stimuli image embeddings through instance-level consistency and flexible token-level matching. MindLink achieves comparable or even better performance over state-of-the-art methods on the NSD dataset with a constant parameter size across subjects and demonstrates strong adaptability to new subject.