Large language models (LLMs), because of their powerful multi-modal understanding and reasoning capabilities, provide a new technical pathway for automated speech fluency assessment. Inspired by LLM-based automatic speech recognition (ASR) models and text-related scoring models, we propose FluLLM, a LLM-based speech fluency classification framework for second-language learners, capable of accommodating both the open scenario (free expression without reference text) and the follow-up scenario (read-aloud with reference text). The framework employs a pre-trained Whisper model as the speech encoder and integrates its acoustic and semantic features at various hierarchical levels through a learnable dynamic-weighting fusion strategy. A lightweight modality adapter is designed to align the fused features with the LLM input space, and a linear classification head is attached to the final hidden states of the LLM to map them to fluency levels. Clear, structured prompts are devised for both the open scenario and the follow-up scenario to guide the LLM in generating classification outputs. On the Avalinguo Audio Dataset (AAD), which represents the open scenario, FluLLM improved by 2.83 and 2.93% points in accuracy and F1-score, respectively, compared to the baseline, and on SpeechOcean762 (SO762), which represents the follow-up scenario, it improved by 7.04 and 8.92% points in accuracy and F1-score, respectively. Both are significantly better than the baseline models.