Abstract <p>In prey-predator interactions, the introduction of the fear effect is crucial. It captures non-lethal predator impacts on prey behavior, making the interaction more realistic and accurate. The introduction of the fear effect enriches the understanding of population dynamics by providing a comprehensive insight into the interaction. In this study, the dynamics of an updated prey-predator model augmented by the fear effect are analyzed. We have identified the conditions under which the system’s fixed points switch from stable to unstable behavior, as verified by graphs. Through Neimark-Sacker and period-doubling bifurcations, we have highlighted the complex dynamics that fluctuate between stable, periodic, and chaotic regimes. We have used mathematical, graphical, and dynamical results to validate the complex dynamics. We have shown that chaos exists in the model using Marotto’s criterion. To control the model’s bifurcation, we employed a hybrid control technique. We have proved the local and global stability of the system in a stochastic environment. In particular, we have drawn two-parameter plots to unfold the rich dynamics. We have observed four distinct dynamic regimes in the plots: periodic, quasiperiodic, bifurcation, and chaotic. The Arnold tongues are observed as a sign of periodicity in the model. To identify different periods of Arnold’s tongues, we have drawn iso-periodic plots, which show that Arnold’s tongues produce period-adding sequences. From the numerical results, we observe that increased fear leads to greater vigilance, which in turn stabilizes the system. In contrast, reduced fear leads to reduced vigilance, resulting in a bifurcation in the system. Furthermore, increased noise causes instability in the population, while controlled noise has a less significant impact on the dynamics of the model.</p> Graphic abstract <p>Through this graphical abstract, the dynamics of a novel model of the environmental population are presented. The dynamics are divided into two environments: stochastic and deterministic. In the stochastic environment, the model is proven to be locally and globally stochastically stable. In a deterministic environment, the population model exhibits rich dynamics. Initially, the topological types of the three calculated fixed points are classified. Later, two types of bifurcations were identified: the period-doubling bifurcation and the Neimark-Sacker bifurcation. The periodicity of the deterministic model, arising from the Neimark-Sacker bifurcation, is demonstrated via Arnold tongues, and the associated periods are identified using isoperiodic plots. The existence of chaos in the model is mathematically proven, and a control technique is proposed to mitigate the chaos. Finally, it is concluded that the increased fear effect causes increased vigilance, which in turn causes stability in the system, while the decreased fear effect results in less vigilance, which in turn causes bifurcation in the system.</p>