It is now a near-given that AI systems will be used in a variety of healthcare scenarios. Yet, there is also an upswell of concern that such “black box medicine” may damage both patient needs as well as trust in care providers. Trust and trustworthiness have been stressed as important factors of justifiable deployment of AI when we consider these needs. Recently, there has been a tremendous amount of attention being paid to the notion of trust regarding technology, and in particular autonomous systems, with a focus on trustworthiness as a metric for justifiable deployment (Shneiderman, ACM Transactions on Interactive Intelligent Systems (TiiS) 10(4): 1–31, 2020; Kaur et al., ACM Computing Surveys (CSUR) 55(2): 1–38, 2022). In particular, the recent publication of the National Institute of Standards & Technology (NIST)’s Artificial Intelligence Risk Management Framework (AI RMF), as well as Newman’s Taxonomy of Trustworthiness (Newman, A Taxonomy of Trustworthiness for Artificial Intelligence. CLTC: North Charleston, 2023), underscores this concern. The AI RMF is to serve as a “living document” robust enough to be used for a variety of case instances, and intended to help AI experts assess trustworthiness of systems at different phases of an AI’s life cycle. This document establishes seven key characteristics of trustworthiness, without ever fully articulating what trust is. I hold we should take a step back from trustworthiness, and revisit trust itself. We ought to reflect on the trust involved when deploying Healthcare Artificial Intelligence systems.