Safeguarding AI systems against harm

“

For me personally, the stakes of AI systems are high, especially where these are establishing unsafe conditions in social domains, thereby often reifying historical power asymmetries. Examples are the emergence of digital apartheid in South Africa through AI-based surveillance systems, or the horrendous treatment of workers through algorithmic gig work platforms. In many ways, the harms that emerge are externalized, i.e. not treated as part of the AI system design. A system safety perspective could help empower those subjected to either ill-willed or unprincipled faulty AI systems, by better understanding how such harms emerge across both the technical, social and institutional elements in the broader context, thereby bringing responsible actors and powerful players into view. We have done this for aviation and medicine too: when things go terribly wrong, there are procedures to carefully understand what happened to prevent more harm in the future. Often times, a full spectrum of actors are involved, from the users and developers of a technology to supervisory bodies, auditors, regulators and civil society. This tells us that it often takes much more to responsibly and sustainably safeguard complex systems subject to software-based automation. As such, the lessons and tools from system safety are useful to understand what is needed to prevent new forms of harm in contexts where AI is relatively new, and to inform standards that the design and governance of such systems should meet, based on decades of experience.

”