Multimodal Large Language Models (MLLMs) have achieved remarkable advances by integrating vision encoders with LLMs. The quadratic complexity of attention with respect to context length, however, imposes significant computational costs in these models, especially when processing high-resolution images or videos. To address this challenge, we propose a token pruning framework that is adaptive, language-guided, and end-to-end trainable. Positioned right after the vision encoder, it prunes irrelevant tokens before they reach LLM, which is the main factor of computational cost. Our framework adaptively prunes vision tokens by jointly considering the entire image and instruction, ensuring task-specific relevance while accounting for complexity. By pruning \(\sim \) 75% of the tokens, LLaVA-1.5 equipped with our framework reduces FLOPs by \(\sim \) 62% with just \(\sim \) 1% accuracy drop compared to original LLaVA-1.5 across 4 benchmarks. Notably, it outperforms original LLaVA-1.5 in 3 out of 7 benchmarks as enhanced focus on critical visual information outweighs the impact of information loss in some cases. Compared to state-of-the-art token pruning works, our work achieves remarkable performance across various benchmarks and models.