Nowadays, Artificial Intelligence (AI) is transforming medical image analysis in different ways given the increasing ease of access to this technology and the relatively low cost. Yet, challenges remain and several difficulties are encountered when applying this technology in this field. This chapter aims to identify these key limitations by grouping them under different categories. First, we consider data issues which include quality, bias, and privacy. Then, a second limitation corresponds to algorithms that struggle with generalization and explainability. Integration into healthcare systems faces hurdles. Subsequently, we focus on the regulatory and human factors that complicate the adoption of AI technologies in the field of medical image analysis. Similarly, we identify the difficulties related to edge cases like rare diseases and global disparities, which limit access to such technologies. According to our study, the solutions needed to overcome these challenges require high levels of collaboration and innovation between different stakeholders in the field. Furthermore, we emphasize that it is very important that future work must address ethics and equity. Finally, our main conclusion is that AI’s full potential in this sector depends on overcoming these barriers in order to achieve the best performance and results.