Instruction fine-tuning significantly enhances the task performance of large language models (LLMs) and their ability to generalize to unseen tasks. However, existing instruction fine-tuning techniques may overlook the training of critical thinking skills in models when responding to instructions. As a result, when presented with logically flawed instructions, models often comply, generating content inconsistent with objective facts or misaligned with user inputs, thus exhibiting sycophancy and the corresponding hallucination. In this paper, we introduce the NCA-MCQ (No-Correct-Answer Multiple-Choice Questions) dataset, derived from diverse tasks, and propose a three-step testing framework to evaluate models’ ability to challenge logically flawed instructions. Experimental results and sample analysis reveal that models that were fine-tuned by instruction, such as GPT-4o, Llama3.1, and the Qwen2.5 series, often exhibit excessive compliance with flawed instructions, leading to sycophantic hallucinations. Furthermore, the parameter scale of the model significantly influences its ability to question defective instructions.