The rapid advancement of artificial intelligence (AI)—especially the emergence of opaque ‘black-box’ model architectures—has raised concerns about moral responsibility. If the internal workings of AI are invisible even to engineering organisations, how do we determine who is responsible for the system’s results? Many have turned to explainable AI (XAI) paradigms to solve this ‘responsibility gap’, assuming that transparent models are necessary for accountability. We argue, however, that Aristotelian virtue ethics more adequately addresses the black-box challenge by locating responsibility in the character and dispositions of the engineering organisations that design, deploy, and use AI, independent of whether models are fully explainable. While the XAI paradigm tries to ensure that AI systems are sufficiently interpretable to identify causal contributors, virtue ethics cultivates phronesis (practical wisdom) and moral dispositions, allowing engineering organisations to respond flexibly and responsibly to novel or unpredictable scenarios even when causal processes remain opaque. To illustrate our case, we analyse how XAI approaches can falter when confronted with the opacity of large language models and their emergent behaviours. We then show how the ethics of Aristotelian virtue more adequately assigns responsibility by emphasising habituation, character development, and readiness to handle unforeseen consequences. We continue by exploring practical difficulties in integrating a virtue-ethical perspective in AI contexts, contrasting a more pessimistic view, highlighting institutional and cultural obstacles, with a more optimistic outlook that sees targeted education fostering virtuous engineering organisations. We close by concluding that even when practical difficulties are considered, virtue ethics remains our strongest tool to achieve responsible AI in a situation where algorithms are ‘black boxes’.