Optimization in High-Dimensional Search Space
摘要
This chapter addresses the challenge of optimizing high-dimensional functions, where traditional derivative-free optimization (DFO) methods struggle with scalability due to slow convergence and high computational costs. The focus is on problems with low optimal-effective dimensions, where only a small subspace significantly impacts the function value. The chapter introduces the Sequential Random Embeddings (SRE) technique, which sequentially applies random embeddings and employs DFO algorithms in each subspace to refine solutions. SRE reduces the embedding gap and improves optimization quality for a broad class of problems. The chapter is structured as follows: Sect. 7.1 defines functions with low effective dimensions, Sect. 7.2 discusses random embedding techniques, Sect. 7.3 introduces SRE, and Sect. 7.4 presents empirical studies on synthetic functions and classification tasks using the non-convex Ramp loss. Experimental results demonstrate that SRE significantly enhances the performance of state-of-the-art DFO methods in high-dimensional spaces, even for problems with up to 100,000 variables.