Historical failures and epistemic accidents: NASA’s Challenger disaster and the organizational risks of Generative AI

What is it about?

Companies are increasingly utilizing powerful AI, such as ChatGPT, for critical decisions. While this technology appears efficient, it poses a hidden danger: it can simplify complex information to the point where critical details and warnings are lost. This creates new risks that organizations may not see until it's too late. To understand this modern risk, this paper looks back at a famous historical failure: the 1986 NASA Challenger space shuttle disaster. In that case, crucial warnings from engineers about the shuttle's safety were lost in overly simplified slide-show briefings presented to managers. This study demonstrates that today’s AI systems can behave similarly, producing confident and well-written summaries while occasionally overlooking or omitting key data from the documents they process. This research reveals a recurring cycle: the demand for efficiency prompts organizations to adopt simplifying technologies, which then filter out essential knowledge, resulting in failures and a loss of human expertise. This loss of expertise then makes the organization more reliant on technology, repeating the cycle. The key takeaway is not to reject AI, but to use it wisely. Organizations must build in safeguards, such as requiring human expert review of AI outputs and fostering a culture that encourages questioning the machine. This is crucial for preventing future failures and ensuring technology serves us responsibly.

Featured Image