A Conversational Agent Framework with Multi-modal Personality Expression

What is it about?

Consistently exhibited personalities are crucial elements of realistic, engaging, and behavior-rich conversational virtual agents. Both nonverbal and verbal cues help convey these agents’ unseen psychological states, contributing to our effective communication with them. We introduce a comprehensive framework to design conversational agents that express personality through non-verbal behaviors like body movement and facial expressions, as well as verbal behaviors like dialogue selection and voice transformation. We use the OCEAN personality model, which defines personality as a combination of five orthogonal factors of openness, conscientiousness, extraversion, agreeableness, and neuroticism. The framework combines existing personality expression methods with novel ones such as new algorithms to convey Laban Shape and Effort qualities. We perform Amazon Mechanical Turk studies to analyze how different communication modalities influence our perception of virtual agent personalities and compare their individual and combined effects on each personality dimension. The results indicate that our personality-based modifications are perceived as natural, and each additional modality improves perception accuracy, with the best performance achieved when all the modalities are present. We also report some correlations for the perception of conscientiousness with neuroticism and openness with extraversion.

Featured Image

Photo by Phil Shaw on Unsplash

Photo by Phil Shaw on Unsplash

Why is it important?

We analyze the influence of different modalities such as body movement, facial expression, dialogue, and voice on the apparent personality of conversational agents. We analyze the correlation of OCEAN factors when different combinations of communication elements are used. Our results indicate that certain modalities are better in representing specific OCEAN personality factors.

Perspectives