This paper investigates the adaptive and self-healing routing problem in heterogeneous ambient backscatter wireless sensor networks (AB-WSNs). In such networks, battery-less nodes harvest energy only from ambient RF signals generated by dedicated RF sources and communicate with each other by backscattering these signals, leading to nodes having different communication ranges, referred to as heterogeneous links. Consequently, routing in heterogeneous AB-WSNs suffers from intermittent and irregular connections, dynamic topology, heterogeneous links and ultra-low-power designs of battery-less nodes. To address these issues, we first introduce a multi-agent network model, and concurrently, we present a heterogeneous multi-hop channel competition mechanism to avoid collisions under heterogeneous links. We then model the routing problem as a Markov decision process, enabling battery-less nodes to employ reinforcement learning approaches. Finally, we propose a policy-based learning algorithm that allows each node to learn the optimal routing policy and perform adaptive and self-healing routing. Furthermore, to solve sparse reward issues arising from collision avoidance efforts, we simultaneously integrate the learning from demonstration and prioritized experience replay mechanisms into our proposed learning algorithms. We analyze the convergence of proposed learning algorithm and evaluate its convergence and efficiency through extensive experiments. The experiment results also validate the efficiency of the proposed mechanisms.