We recently settled Duflo’s polynomial problem (Duflo, Open problems in representation theory of lie groups, edited by T. Oshima, Katata in Japan, pp 1–5, 1986) for monomial representations of nilpotent Lie groups (cf. Baklouti and Fujiwara, A solution to Duflo’s polynomial problem for nilpotent Lie groups monomial representations. Preprint). Here we study a similar problem for restricted representations. Let \(G = \exp \mathfrak g\) be a connected and simply connected real nilpotent Lie group with Lie algebra \(\mathfrak g\) , \(K = \exp \mathfrak k\) an analytic subgroup of G with Lie algebra \(\mathfrak k\) , \(\pi \) an irreducible unitary representation of G, and \(\pi |_K\) the restriction of \(\pi \) to K. Let \(D_{\pi }(G)^K\) be the algebra of the differential operators keeping invariant the space of \(C^{+\infty }\) -vectors of \(\pi \) and commuting with the action of K on that space. Let \(\varOmega \) be the coadjoint orbit of G corresponding to \(\pi \) and \(\mathbb C[\varOmega ]^K\) the algebra of K-invariant polynomial functions on \(\varOmega \) . Remark that \(\mathbb C[\varOmega ]^K\) is endowed with a Poisson product coming from the symplectic structure of \(\varOmega \) . We show in this chapter that the center of \(D_{\pi }(G)^K\) and the Poisson center of \(\mathbb C[\varOmega ]^K\) are isomorphic.