High content imaging (HCI) plays a pivotal role in target-directed drug discovery (TDD) by identifying compound activities across tests (or assays) designed for specific therapeutic targets. However, real-world assays often exhibit extreme label sparsity over large compound libraries, making accurate predictions challenging. Recent studies following multi-label learning (MLL) struggle in such scenarios when optimizing a single objective across multiple assays without assay-specific adaptations. To address this, we propose Mixture of Multi-Instance Learners (MoMIL), a multi-task learning (MTL) framework integrating hard-parameter sharing with assay-specific Multiple Instance Learners (MILs), enabling knowledge sharing and task-specific adaptations. Furthermore, we introduce complementary enhancements: HCI-specific foundation models (FMs), an assay selection algorithm, and a label imputation method to boost MoMIL’s learning capabilities. We benchmark MoMIL on two extensive HCI datasets, achieving up to \(\sim \) 6% and \(\sim \) 8% improvement over state-of-the-art MLL and MTL methods. Moreover, MoMIL shows strong generalization to unseen assays, outperforming assay-specific single-task learning (STL) methods in 11 out of 12 assays.