What is it about?
LLMs can role-play different personas by simulating their values and behaviour, but can they stick to their role whatever the context? Is simulated Joan of Arc more tradition-driven than Elvis? Will it still be the case after playing chess? We argue that context-dependence should be studied as a dimension of LLM comparison. We focus on the expression of values by simulated individuals and adapt the methodology from psychology towards that end.
Featured Image
Photo by Ant Rozetsky on Unsplash
Why is it important?
The standard way to study Large Language Models (LLMs) is to provide many different queries from similar minimal contexts (e.g. multiple choice questions). However, due to LLM’s highly context-dependent nature, conclusions from such minimal-context evaluations may be little informative about the model’s behaviour in deployment (where it will be exposed to many new contexts). We address this issue by studying context-dependence as another dimension of LLM comparison.
Perspectives
Standard benchmarks present many questions from the same minimal contexts (e.g. multiple choice questions), we present the same questions from many different contexts. I hope this article motivates similar research.
Grgur Kovač
INRIA
Read the Original
This page is a summary of: Stick to your role! Stability of personal values expressed in large language models, PLOS One, August 2024, PLOS,
DOI: 10.1371/journal.pone.0309114.
You can read the full text:
Resources
Contributors
The following have contributed to this page