Reinforcement learning (RL) often struggles to effectively balance exploration and exploitation, particularly under uncertainties in both rewards and policy parameters. Consequently, existing methods can suffer from premature convergence or require extensive hyperparameter tuning. To overcome these challenges, we introduce a novel framework based on the principle of negative free-energy maximization. This approach provides a unified treatment of uncertainty by modeling policies as distributions over their parameters and rewards using Bayesian estimation, yielding an objective function that integrates expected returns, their associated variance, and policy entropy. This objective, derived from a free-energy formulation approximated using cumulant-generating functions, inherently promotes exploration through entropy regularization and uncertainty-driven bonuses. We further present an efficient iterative algorithm employing Gaussian policy approximations and an adaptive inverse temperature, regulated by a Kullback-Leibler (KL) divergence constraint, which reduces the need for manual hyperparameter tuning. Our framework not only generalizes and extends prominent prior works but also offers a principled and scalable solution for robust policy optimization in complex, continuous environments.