The proliferation of artificial intelligence across multiple sectors signifies a pivotal shift in how tasks are performed and decisions are made. This evolution needs a thorough examination beyond technical specifications to encompass its broader social and ethical implications. Critical infrastructures emerge as a particularly crucial point for integrating Artificial Neural Network systems, requiring careful scrutiny. In this paper, we explore the intersection of the “Human+Machine” paradigm and its implications for their deployment as critical infrastructures under EU AI Act. Relying solely on technical trustworthiness metrics will not be sufficient to foster genuine human trust and adoption. Hence, by defining deeper human factors, ‘satisfaction’ and ‘tranquility’, we present a redefined human-AI relationship, bridging this to responsible AI principles, e.g., Sustainable Development Goals. We propose a holistic understanding of human-oriented AI innovation, while introducing the concept of “human-on-the-loop” from a digital trust service perspective.