To address the high dynamicity of Reconfigurable Control Systems (RCS) at design-time, this paper proposes a novel Reinforcement Learning-based framework allowing for modelling and improving reconfiguration knowledge. The main goal of this proposal is to cope with design-time unplanned changes by giving the reconfiguration controller the ability to improve its knowledge through online learning feedback. In this research work, reconfiguration is defined as the ability to switch from one configuration to another, with the set of configurations referred to as Configuration Space (CS). Furthermore, Reinforcement Learning (RL) is adopted for CS modelling to handle early and effective exploration of various design choices. The proposed framework is derived from UML-based conceptual models of the CS using a set of generation rules that will be detailed. The resulting framework exhibits three key characteristics for CS modelling and exploration: the reconfiguration controller is designed as an RL agent (Reinforcement Learning Reconfiguration Agent or RLRA) that handles both offline (exploitation) and online (exploration) learning; the capability of automatically learning reconfiguration policies from predefined knowledge during the offline phase; and the ability of automatically evolving the CS through learning from run-time interactions with the controlled system at the online phase. Finally, the resulting framework is evaluated through a case study from the manufacturing domain which shows that this framework improves both the efficiency and effectiveness of RCS design.