This article investigates the integration of automated decision-making within the Italian Public Administration, with a specific focus on Article 30 of the new Public Procurement Code (Legislative Decree No. 36/2023) and its alignment with the European Union’s Artificial Intelligence Act (AI Act). The analysis explores the growing application of algorithmic systems and machine learning technologies—including Large Language Models (LLMs)—in administrative procedures, highlighting their potential to enhance efficiency, impartiality, and consistency. At the same time, the paper critically examines the legal challenges posed by algorithmic opacity, the traceability of decision-making processes, and compliance with fundamental constitutional principles such as legality, transparency, non-discrimination, and accountability. The study addresses the safeguards introduced by Article 30, including the mandatory presence of human oversight, access to source code, and the implementation of technical and organizational measures aimed at minimizing errors and ensuring fairness. Particular attention is given to the classification of public procurement as a “high-risk” sector under the AI Act, emphasizing that risk derives not from the sector itself but from the specific modalities of technological deployment. Ultimately, the article underscores the need for a robust and coherent governance framework capable of ensuring the responsible adoption of artificial intelligence within the public sector. Such a framework must protect citizens’ fundamental rights—foremost among them the right to explanation—when faced with decisions generated by opaque or non-interpretable AI systems.