Prompt engineering has proven effective across various tasks, minimizing reliance on extensive training data. However, its potential for complex word identification (CWI), a key step in lexical simplification, remains unexplored. This study evaluates the effectiveness of prompt engineering for CWI using open-source large language models (LLMs) and compared a new feature engineering-based that integrates diverse features into neural network classifiers. Experimental results show LLMs’ have strong language understanding and generation capabilities, yet feature engineering-based strategy has advantage for such specific classification tasks. Finally, we provide recommendations on how to improving designs of LLMs’ prompts in order for such classification tasks.