Manufacturing-as-a-Service (MaaS) leverages decentralised resources provided by a network of manufacturers, to deliver on-demand production services to consumers. The core challenge lies in optimally allocating resources to service requests, balancing resource utilisation with consumer satisfaction. This context introduces large-scale, multi-objective optimisation problems that traditional exact methods, such as Constraint Programming (CP), struggle to solve efficiently. Motivated by a MaaS setting of two competing producers of electronic boards for white appliances, we propose a hybrid Reinforcement Learning (RL)-CP approach for the distributed hybrid flexible flowshop scheduling problem. We aim at minimising total earliness and tardiness, while ensuring fairness among the providers by minimising load imbalance. An RL agent divides the set of requests into smaller subsets to minimise load imbalance, while yielding smaller job sets solvable through a CP model to minimise total earliness-tardiness. To evaluate our hybrid RL–CP framework, we benchmark it against a relaxed CP model and a multi-phase constructive metaheuristic. As we show, the hybrid RL-CP outperforms both the CP model and the metaheuristic baselines in most instances, within practical computation times.