This paper introduces an approach to Bidirectional Evolutionary Structural Optimization (BESO) using the Scaled Boundary Finite Element Method (SBFEM). The design domain is discretized into an SBFEM mesh, and SBFEM analysis is performed to derive elemental sensitivity numbers. These sensitivity numbers are then transformed into nodal sensitivity numbers, which are smoothed across the design domain using a filter scheme. This filter scheme smoothes the sensitivity numbers across the entire design domain, automatically assigning sensitivity numbers to void elements. These void elements may have high values due to the high sensitivity numbers of nearby solid elements, potentially changing some void elements to solid elements in the next iteration. To enhance convergence, an averaging scheme is applied which incorporates historical information into the sensitivity numbers. Elements are sorted based on their sensitivity numbers from highest to lowest, and the target volume in each iteration adjusts incrementally through the evolutionary volume ratio (ER) until the constraint volume is achieved. The threshold sensitivity numbers for removing and adding elements are determined by the volume addition ratio (AR). The SBFEM analysis and the element removal/addition cycle continue until the objective volume is reached and the convergence criterion, defined by the change in the objective function, is satisfied. This approach is validated through several compliance minimization problems and benchmarked against the traditional Finite Element Method (FEM) using SBFEM.