Image captioning reliant on extensive training data and model parameters has achieved notable progress, yet at significantly elevated training costs. To improve caption quality without scaling the data or model parameters, recent advancements have focused on retrieving information from external repository to prompt large language models(LLM) for caption generation, which unavoidably introduces irrelevant noise. To address this noise interference effectively, we propose FreCap, a token-level prompt model derived from frequency-driven sentence breakdown. By decomposing the retrieved sentences and leveraging frequency-driven token-level prompts, our FreCap method enables convenient and efficient updates to external repositories, facilitates lightweight and rapid training, and mitigates the risk of generating suboptimal captions induced by erroneous prompts. Simultaneously, through the proposed feature similarity loss, FreCap captures accurate semantic information, producing captions that exhibit high fidelity to the corresponding image. FreCap seamlessly adapts to other domains without requiring additional fine-tuning or retraining. Extensive experiments demonstrate that FreCap, trained on the COCO benchmark, delivers competitive performance and remains comparable to state-of-the-art models trained on large-scale datasets when transferred to other domains, without necessitating retraining.