The EU AI Act is set to become a landmark regulatory framework for the ethical and responsible use of artificial intelligence across the European Union, presenting both opportunities and challenges for public sector institutions. This paper examines how public institutions in Germany perceive and respond to the Act’s requirements, focusing on their preparedness, operational adaptations, and the challenges they face. The paper uses qualitative data from semi-structured interviews with representatives of federal agencies and smaller municipalities. The findings reveal significant differences between larger and smaller institutions, with larger institutions demonstrating greater readiness due to established resources and expertise, while smaller municipalities struggle with regulatory uncertainty and resource constraints. The paper highlights that without targeted interventions and improved guidance, smaller entities may fall behind in AI innovation and adoption. These findings have implications for policymakers, suggesting the need for centralized support mechanisms, resource-sharing programs, and tiered compliance strategies to ensure equitable and effective implementation of the EU AI Act.