AI code assistants, such as GitHub Copilot, are an increasingly popular coding aid, but they also present risks. Large language models (LLMs) upon which those assistants are built may generate insecure/incorrect code, either by accident or as a result of code poisoning attacks. In general, LLMs obfuscate the lineage of source code used for training. This is a problem, for example, in the context of supply chain security, where tracking provenance is of the utmost importance. While a number of recent approaches can flag AI-generated code based on a combination of lexical and syntactic features, such works have not been evaluated in realistic settings. First, we identify and operationalize a number of recently proposed AI code identification tools, measuring their baseline performance on datasets generated by state-of-the-art models. Then, we verify the robustness of such approaches to variations in training sets and prompting strategies. Results show that existing AI code detectors tend to be fragile and have limited accuracy in real-world scenarios.