In a social media network, some users are highly influential and their posts get shared a lot across the platform. Concentrated changes in the discourse of just a few highly influential users are often enough to ripple throughout multiple communities. This analogy can be extended to fine-tuning a neural network. During fine-tuning, CNNs are typically adapted by appending a new linear layer with a classification head on top of a large pre-trained model. Instead, we argue that better adaptation can be achieved by first pruning the model to retain only the most important neurons and then fine-tuning. In this paper, we propose a novel graph-theory-based method for pruning neural networks, designed to enable better fine-tuning. In our method, each neuron is treated as a node with edges that represent similarity between them. Neurons are removed on the basis of their importance, which is computed using eigenvector centrality. The pruned model is then fine-tuned using only the most central neurons. We evaluated our method in the VGGNet, EfficientNet and ResNet models with the data sets TF-Flowers, Caltech101, and Oxford-Flowers102, and achieved better accuracy with significantly reduced model complexity. More specifically on Oxford-Flowers102 we get 48% accuracy compared to baseline VGGNet’s accuracy of 30%.