It is critically important to minimize the risk of harmful decisions possibly made by neural networks in Safety-Critical Systems (SCS). In this paper, a relative safety-criticality metric is introduced to quantitatively characterize the importance of different classes for a multi-classifier, according to the probability of harm failures induced by its misclassifications in SCS. With this metric, Safety-Critical Classes (SCCs), which require a higher level of protection than other classes, can be identified for classification tasks. A fine-tuning optimization framework is proposed for multi-classifiers to minimize the possibility of losses potentially caused by their wrong operations on SCCs. By constraining the cross-entropy loss function with a tunable penalty term, our proposed training process can effectively improve the recalls of SCCs. Comparative experiments demonstrate the effectiveness of our work. And we believe our work can be potentially used as an optimization scheme to migrate existing multi-classifiers into SCS.