In the recent context, the multilayer neural network has found its edge on learning big data with good generalization. This has been evidenced by the superior prediction accuracy of deep networks on various applications. The main ground for realizing the high learning capacity with good predictivity comes from several major advancements in the field. These advancements include the processing platform, the learning regimen, and the availability of big data. However, despite the great success in applications, a good understanding of the learning representation and generalization properties has been far fetched. In this chapter, the mechanism of network learning is interpreted from the perspective of a linear system of equations in matrix form where the learning representation and generalization properties are investigated.