<p>Handling probabilistic constraints in reliability-based design optimization (RBDO) presents significant challenges due to their complexity and computational burden. Additionally, non-monotonic limit-state functions and convergence issues related to initial design points exacerbate these difficulties, creating a critical gap in achieving efficient and robust optimization. This study introduces an advanced sequential optimization and reliability assessment (ASORA) approach that approximates the reliable design space (RDS) method with a new formulation based on the signed distance function (SDF) and the level set method (LSM). The LSM-SDF framework is specifically designed to mitigate convergence issues and computational inefficiencies by introducing an approximate quasi-reliable initial design point, effectively addressing the limitations of classical RBDO frameworks. The proposed method was applied in both mathematical and mechanical scenarios, including a new case study involving a reformulated gear optimization problem. In this case, the objective is set to minimize weight, while additional performance metrics—such as power loss, wear (expressed via specific sliding), and contact ratio—are incorporated as constraints. All results are validated using Monte Carlo simulation. The results indicate that ASORA achieves significant improvements in the number of iterations and function evaluations for all studied cases. These findings underscore its effectiveness in tackling complex RBDO problems, making it a promising approach for practical engineering applications.</p>