Toward a Science of Autonomous Systems: Reliability Engineering for AI

Nancy Leveson (MIT)

Abstract

We apply systems safety engineering principles to autonomous AI systems, arguing that AI reliability requires the same rigorous engineering discipline as safety-critical systems in aerospace and nuclear industries.

Eigenvector Insight — Zone III / PASF-PADE AnalysisNot part of the original paper

Eigenvector Research — Marco van Hurne

How this paper contributes to solving the Zone III problem (PASF-PADE)

Leveson's application of systems safety engineering to AI is the most important cross-disciplinary contribution to Zone III design. The aerospace industry spent decades learning how to build reliable autonomous systems — AI engineers should not repeat those lessons from scratch.

Why AI is not sufficient for Zone III without this

Zone III refers to high-complexity, high-risk, long-running agentic workflows — the class of enterprise AI deployments where a single failure can cascade across hundreds of steps. Standard AI models, trained to predict the next token, are not inherently designed for durable, governed, multi-step execution. This paper addresses one or more of the structural gaps that make Zone III deployments unsafe without explicit architectural intervention.

Key Contributions

→Systems safety for AI
→Reliability engineering methodology
→Safety-critical AI design principles

Topics

systems safetyreliability engineeringautonomous systemssafety-critical

Relevance Scores

Long-Horizon Score85

Enterprise Score92

Completeness80

Paper Info

Year2023

VenueMIT Press

Typetheoretical framework

ChapterCh. 6

Authors1

Zone III Analysis

Frameworks

AEGIS OCG