A systematic analysis of theoretical aspects of material instability is presented for a general class of timeindependent, incrementally nonlinear solids which admit a velocity-gradient potential. The second-order energy criteria of instability of uniform straining and of equilibrium are derived for a material element embedded in a quasi-statically deforming continuum. Several theorems concerning propagation of acceleration waves in an incrementally nonlinear material are proved, with particular reference to stationary discontinuities. Initiation of shear band formation either by a dynamic growth of an initial disturbance or by a quasi-static bifurcation is studied as a symptom of material instability. The derived condition for the instability of uniform quasi-static straining does not coincide with that for instability of equilibrium and both differ in general from the familiar condition of strong ellipticity loss formulated for the tangent moduli. This is illustrated on an example of a non-proportional deformation path of material obeying the J2 corner theory of plasticity.