Bayesian Optimization (BO) is a powerful method for optimizing complex and expensive black-box functions. However, in real-world applications, the target scenarios are not entirely “black-box.” Historical optimization trajectories of similar tasks and meta-domain information can serve as priors and conditional assumptions to guide the direction of BO optimization. Although BO has made substantial progress in this area, such as kernel prior design, it is often limited by factors like data volume and complexity. In this work, we propose Bayesian Optimization by Prompting (BOPRO), which achieves efficient task optimization by automatically constructing meta prompts and leveraging the few-shot learning and in-context learning (ICL) capabilities of large language models (LLMs). All components of BOPRO are constructed using natural language, enabling modularity and plug-and-play functionality without the need for fine-tuning. Our experiments demonstrate the effectiveness of BOPRO in black-box optimization problems, highlighting its outstanding performance in tasks such as automated machine learning (AutoML) and prompt optimization. Our code is open-sourced at https://github.com/turnwind/BOPRO .