Medical decision support systems are essential for meeting the growing need for accessible, accurate, and timely healthcare information, especially in underserved or remote regions where healthcare professionals are scarce. These systems offer first-line guidance on symptoms, treatments, and preventive care, enabling faster triage and 24/7 support. They also assist healthcare providers by aiding decision-making and keeping them updated with the latest medical research, ultimately improving patient care and alleviating pressure on healthcare systems. This study focuses on three key medical decision support system paradigms: Large Language Models (LLMs), Knowledge Graphs, and Mixture-of-Experts (MoE). LLMs excel at understanding and generating human-like text, knowledge graphs structure medical knowledge by capturing relationships between entities, and MoE models optimize efficiency by dividing tasks among specialized experts. This paper aims to compare these approaches and evaluate the accuracy of the techniques across multiple benchmark medical datasets. It also examines their methodologies, contributions, and challenges to provide insights into their application in healthcare, with the goal of improving the accuracy, efficiency, and reliability of medical decision support systems.