<p>Market-making is an essential activity in every financial market. They provide liquidity to the system by placing buy and sell orders at multiple price levels. While performing this task, they aim to earn profit and manage inventory levels simultaneously. However, financial markets are not stationary environments; they constantly evolve, influenced by changes in participants, the occurrence of economic events, or the market trading hours, among others. This study introduces a novel approach to address the challenge of market-making in non-stationary financial markets with multi-objective Reinforcement Learning (RL). Traditional RL methods often struggle when applied to non-stationary environments, as the learned optimal policy may not be adapted to the new dynamics. We present Policy Weighting through Discounted Thompson Sampling (POW-dTS), a novel dynamic algorithm that adapts to changing market conditions by effectively weighting pre-trained policies across various contexts. Unlike some conventional methods, POW-dTS does not require additional artifacts such as change-point detection or models of transitions, making it robust against the unpredictability inherent in financial markets. Our approach focuses on optimizing trade profitability and managing inventory risk, the dual objectives of market makers. Through a detailed comparative analysis, we highlight the strengths and adaptability of POW-dTS against traditional techniques in non-stationary environments, demonstrating its potential to enhance market liquidity and efficiency.</p>