Anomalous Sound Detection (ASD) has been recently extended to a range of surveillance problems where sound can be ubiquitous, such as landscape sounds, biodiversity conservation and preservation of natural environments such as forests. Tree logging was therefore given special attention, as the sounds of chainsaws can be considered anomalous in a natural environment, especially in remote and inaccessible areas. In this context, this work aims to develop an ASD model based on Graph Neural Networks (GNNs) to improve the performance of detecting anomalous sounds for illegal tree cutting, mainly produced by chainsaws. To this end, two GNN-based methods are proposed, namely a first method using Graph Convolution Networks (GCN) and an improved method using Graph Attention Networks (GAT). Experiments confirm the suitability of GNNs, particularly those based on graph attention networks, improving significantly the anomaly detection scores, with an accuracy of 0.91, an AUC score of 0.89 and high and balanced precision and recall scores, thus outperforming standard audio classification methods such as those based on Convolutional Neural Networks (CNN). Furthermore, the GAT-based model is able to rely on a minimal set of acoustic features, which paves the way for a lightweight graph model for ASD.