Throughout this work, we examine an unreliable retrial queueing system in which clients may balk, the server recovers based on a threshold, and maintenance takes place during idle times. The server is in charge of client service in this framework, although it is unreliable due to unpredictable breakdowns. Once a certain number of clients are in the system, the server can be repaired. When a client arrives and discovers that the server is busy, they are redirected to the retrial orbit, a virtual waiting area, where they must wait for the server to make itself accessible for service. The mathematical framework of the Chapman–Kolmogorov (C–K) equations is based on the birth and death process. Then, using a matrix method, steady-state probabilities are computed. Different probabilities are used to create performance metrics and simulations are performed to see how changing the parameters affects the system’s performance. A cost function is developed for determining the optimal decision parameters and reducing the system cost by using particle swarm optimization (PSO). Moreover, we compare the outcomes of the numerical experiments with those obtained via a soft computing technique on the basis of an adaptive neuro-fuzzy inference system (ANFIS), providing valuable insights into the system’s dynamics.