Fractional calculus has gained prominence recently due to its ability to model complex systems with memory and non-local interactions. This study explores a nonlinear coupled fractional boundary value problem enhanced with novel integral boundary conditions. The problem involves two fractional differential equations of \(\Psi \) -Caputo type, describing the dynamics of variables \(\varkappa (t)\) and h(t) with orders \(\ell _1\) and \(\ell _2\) , respectively. These equations are interconnected through nonlinear functions \({f} \left( t, \varkappa (t), {h}(t) \right) \) and \({g} \left( t, \varkappa (t), {h}(t) \right) \) , with a novel integral boundary condition. Applications of such formulations span diverse fields like economics, energy management, population dynamics, and control systems, where fractional-order dynamics effectively capture intricate interactions and constraints. The integral boundary condition is crucial in enforcing specific balances or conservation laws within the system. To analyze these systems, analytical techniques and numerical methods are employed to assess the existence, uniqueness, and stability of solutions under Banach fixed point theory, notably the Leray–Schauder fixed point approach. This research contributes to advancing the analytical understanding and practical applications of fractional calculus in modeling and analyzing complex systems governed by nonlinear dynamics and integral constraints. Illustrative examples are provided to validate findings, highlighting the practical relevance of these results in various applications.