The detection of artificially generated content on social media has become a critical challenge due to the increasing sophistication of large language models. While transformer-based classifiers achieve high detection accuracy, their black-box nature limits interpretability. Also, prominent interpretability tools (e.g., LIME, SHAP) focus on the importance of single features and neglect positions or context. Here, we take a step towards investigating the role of word order in classification decisions. Therefore we combine counterfactual generation using a 1+1 evolutionary algorithm, investigation of the evolutionary path, and LIME-inspired feature importance analysis. Applied to real-world social media posts, our approach offers first insights into text classification models beyond pure word importance.