Automated penetration testing (pentesting) is crucial to efficiently assess the cybersecurity of infrastructures. It allows to discover attack paths before they are actively exploited, hereby providing a chance to block them. Reinforcement Learning (RL) has demonstrated great promise in academia for cybersecurity, including for automated penetration testing. However, despite encouraging research, market-ready solutions are still limited, potentially due to the difficulty of the Simulation-to-Reality (Sim2Real) transition and the scarcity of appropriate RL environments, especially for industrial infrastructures. This paper addresses these challenges by first introducing essential features for RL cybersecurity environments. It then presents CRLF (Cyber Reinforcement Learning Framework), a novel environment designed to bridge the Sim2Real gap and support the training of autonomous pentesting agents on both IT and OT infrastructures. CRLF’s key features are high fidelity, OT support, user-friendliness via a graphical interface and enhanced agent generalization through a pseudo-random infrastructure generator. Experimental results demonstrate the potential of CRLF to facilitate sophisticated autonomous pentesting for critical ICSs.