Consider the following higher order Hamiltonian system: \(\begin{aligned} {\left\{ \begin{array}{ll} (-\Delta )^{m} u = v^{p},\;\;\; & \hbox {in } \Omega ,\\ (-\Delta )^{m} v = u^{q_\epsilon },\;\;\; & \hbox {in } \Omega ,\\ u = (-\Delta ) u = \cdots = (-\Delta )^{m-1} u = 0, & \hbox {on } \partial \Omega ,\\ v = (-\Delta ) v = \cdots = (-\Delta )^{m-1} v = 0, & \hbox {on } \partial \Omega ,\\ u>0,v>0, & \hbox {in } \Omega , \end{array}\right. } \end{aligned}\) where \(m\ge 1\) is an integer, the exponents \(p, q>0\) satisfy the subcritical condition: \( \dfrac{1}{p+1} + \dfrac{1}{q_\varepsilon + 1} = \dfrac{N-2m}{N} + \varepsilon , \) and \(\Omega \subset \mathbb {R}^N\) is a smooth bounded convex domain. We first prove the existence of the least energy solution \((u_{\varepsilon } , v_{\varepsilon })\) for the above system. Then we use a Brezis-Kato type argument to study various asymptotic behaviors of \((u_{\varepsilon } , v_{\varepsilon })\) as \(\varepsilon \rightarrow 0\) , including a detailed description in terms of Green function, which generalizes the result of Han [20] and Guerra [18] to higher order system.
