This chapter presents a self-contained view of optimization for deep network training without requiring prior knowledge of artificial neural networks. We describe how training reduces to optimization and what are the main algorithmic building blocks in this context. We also include specific algorithmic developments dedicated to, or mostly used for neural network training. Finally, we describe a few theoretical, essentially open, challenges in this context.