Self-Attention plays a crucial role in machine learning. At the heart of a transformer, the attention mechanism is used to build complex relationships between words in sentences. A transformer uses a sequence of layers with multiple attention heads. Studies have shown that not all attention heads contribute useful information. Some heads can be pruned with little to no effect on the accuracy of the model. However, pruning can have a huge impact on the computational performance such as reduction in inference time and memory usage. Many existing optimization techniques employ pruning with a focus on attention heads while little to no optimization in the Feed Forward Network (FFN) layers. The key challenge is identifying which heads and neurons should be pruned. In this paper, we employ a hybrid dual-step pruning algorithm that optimizes both attention heads and FFN neurons by pruning them to evaluate the impact of pruning. Through experimentation, we assess the performance of multiple transformers with multiple datasets. We further conclude if pruning is really beneficial and explain its real-life implications.