This chapter addresses the challenge of optimizing high-dimensional functions, where traditional derivative-free optimization (DFO) methods struggle with scalability due to slow convergence and high computational costs. The focus is on problems with low optimal-effective dimensions, where only a small subspace significantly impacts the function value. The chapter introduces the Sequential Random Embeddings (SRE) technique, which sequentially applies random embeddings and employs DFO algorithms in each subspace to refine solutions. SRE reduces the embedding gap and improves optimization quality for a broad class of problems. The chapter is structured as follows: Sect. 7.1 defines functions with low effective dimensions, Sect. 7.2 discusses random embedding techniques, Sect. 7.3 introduces SRE, and Sect. 7.4 presents empirical studies on synthetic functions and classification tasks using the non-convex Ramp loss. Experimental results demonstrate that SRE significantly enhances the performance of state-of-the-art DFO methods in high-dimensional spaces, even for problems with up to 100,000 variables.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Optimization in High-Dimensional Search Space

  • Yang Yu,
  • Hong Qian,
  • Yi-Qi Hu

摘要

This chapter addresses the challenge of optimizing high-dimensional functions, where traditional derivative-free optimization (DFO) methods struggle with scalability due to slow convergence and high computational costs. The focus is on problems with low optimal-effective dimensions, where only a small subspace significantly impacts the function value. The chapter introduces the Sequential Random Embeddings (SRE) technique, which sequentially applies random embeddings and employs DFO algorithms in each subspace to refine solutions. SRE reduces the embedding gap and improves optimization quality for a broad class of problems. The chapter is structured as follows: Sect. 7.1 defines functions with low effective dimensions, Sect. 7.2 discusses random embedding techniques, Sect. 7.3 introduces SRE, and Sect. 7.4 presents empirical studies on synthetic functions and classification tasks using the non-convex Ramp loss. Experimental results demonstrate that SRE significantly enhances the performance of state-of-the-art DFO methods in high-dimensional spaces, even for problems with up to 100,000 variables.