In this paper, we propose a multi-modal phishing website detection system based on federated learning, which can simultaneously utilise URL, HTML, and IMAGE data, and does not require the client to be bound to the data type during the inference stage: any client can invoke the corresponding modality head trained by other clients for detection. Methodologically, we implement a bucket aggregation strategy by role (expert level) on the FedProx paradigm, drawing on the multi-expert idea of Mixture-of-Experts (MoE) and the design concept of the FedMM algorithm. We remove the learnable routing and adopt hard gating (directly selecting the IMAGE/HTML/URL expert based on the modality label of the sample) to separately aggregate the parameters of each modality, to isolate the aggregation conflicts and convergence oscillations caused by different embeddings. We validate the effectiveness on two public datasets (TR-OP and WebPhish): the Fusion head achieves performance metrics of Accuracy (Acc) 97.5%, Precision (Prec) 98.1%, Recall (Rec) 97.3%, and 2.4% FPR on the two types of data on TR-OP. Meanwhile, in the ablation experiment, it achieves Acc 95.5%/Prec 94.3%/Rec 96.5%/FPR 5.9% on the image subset of TR-OP. On the text side, we applied two adaptations: for the simple-structured URL, we directly use GraphCodeBERT to generate embeddings; for the raw and uncleaned HTML, we adopt an early fusion of a three-way embedding design to reduce noise. On WebPhish’s HTML (relatively simple), we achieve Acc 96.5%/Prec 98.1%/Rec 95%/FPR 1.8%; on the raw HTML of TR-OP (relatively difficult), we achieve Acc 95.1%/Prec 95.4%/Rec 94.6%/FPR 4.6%. The results show that bucket aggregation with hard-gating experts can stably conduct federated training and enhance the usability of multi-modal under strict privacy boundaries, providing a more flexible and comprehensive federated solution for phishing website detection.