Beyond Correctness: Architecting Trustworthy Software for Autonomous Systems in the Age of AI
摘要
Traditional software for critical applications is typically built through a structured approach, with a clear set of predefined rules and logic imposing how the system will respond to different inputs, and prepared for safety certification accordingly. Recent innovations are pushing for conceptualizing, designing, implementing and validating autonomous systems that require the deployment of complex software functions which cannot be fully managed by traditional software development approaches. Those complex functions comprise a massive amount of inputs and virtually infinite input combinations, which are typically fed to a Machine Learning (ML) algorithm tasked to learn rules and logic that humans cannot derive due to excessive complexity or incomplete understanding of the phenomena. This paves the way for potential issues when architecting the software, and especially exposes it to facing inputs – referred to as unknowns - for which the behavior of the target software is untested. This has a detrimental impact on the overall software development, and it is the main obstacle to fulfilling certification processes in accordance to applicable standards. Domain experts should focus on trust rather than the traditional correctness, embracing software design patterns proven in use since decades, ultimately enabling the development of autonomous systems that will drive the upcoming fifth industrial revolution.