Faulty Tools or Disruptive Teammates? A New Theory of Human-AI Conflict and Compromise

What is it about?

As robots and virtual agents advance in their reasoning capabilities, they will increasingly fill humanlike roles in work teams in industry, transportation, healthcare, and the military. However, reasonable disagreements between humans and AI systems may arise, just as they do between human teammates. Resolving such conflicts is important for effective human-AI collaboration and human acceptance of AI coworkers. We present a new theory of human-AI conflict based on research supported by the US Air Force Office of Scientific Research (AFOSR) Trust and Influence program. It proposes that the mental model adopted by the human operator shapes the nature of conflicts with the AI, trust, and paths to compromise and conflict resolution. People understand conventional machines and automation as tools, so that trust mainly depends on machine performance and reliability. In some contexts for collaboration, the AI is intelligent but not humanlike. Human-AI disagreements are technical and impersonal in nature. Designing the AI to be transparent and comprehensible can promote conflict resolution. By contrast, in human-human teaming conflicts can become personalized and emotional which hinders rational resolution of the dispute. Similarly, when a humanlike AI dissents from the human, it may evoke anger or anxiety, suspicions about its intent, and perceptions that it is a poor team player. In this case, the person may use narrative rather than technical reasoning in making sense of the AI’s behavior. That is, they see the AI as executing a mindful sequence of actions guided by unhelpful or hostile motives. Conflict resolution requires naturalistic communication to understand the AI’s intent and choice of actions and find common ground. Design features such as AI inner speech can help the human understand its perspective and work towards an acceptable compromise.

Featured Image

Photo by Axel Richter on Unsplash

Photo by Axel Richter on Unsplash

Why is it important?

Human-AI teaming is becoming ubiquitous in professional settings. Like human experts, AI decision support systems will be able to justify and defend their judgments coherently even if the human disagrees. Such systems contrast with current Large Language Models, which tend to be sycophantic and easily swayed by prompting: i.e., they have no integrity. The mental models perspective contributes to understanding and mitigating human-AI conflicts, as well as calibrating trust in the AI to actual system capabilities. Use of a mental model that is compatible with system design, whether tool or teammate, will help the human operator to understand its strengths and limitations and adopt constructive conflict resolution strategies. The current conflict theory also suggests important new research questions, such as: • What mental models do people apply to advanced AI, beyond tool and teammate? Collaboration will be enhanced by a deeper understanding of how people perceive AI in different roles, both constructive (adviser, assistant) and malign (tyrant, incompetent). • How do people think about AI systems that change roles in a team autonomously or under instruction? Our mental models of human teammates are shaped by our history of working with them; can we adapt dynamically to working with an AI that may not display a consistent personality or skill set? • People differ markedly in their tendencies to anthropomorphize machines (even dumb ones). Individual differences can thus contribute to miscalibrating trust and poor handling of conflicts. How can we personalize system design to promote mental models compatible with both actual system capabilities and the individual’s preferences?

Perspectives