Robots can't automatically sense the significance of objects

What is it about?

To be effective in a human home, robots need to understand lots of "semantic" information: labels and uses for many household objects. Most of this semantic information is not immediately present in the environment, but must be learned by tracking local conventions of human behaviour. For example, a robot not only needs vision to sense that an object is a mug and is therefore capable of holding liquid, but also to know when a certain mug is purely ornamental and not to be drunk out of or put in the dishwasher. Deceptively simple problems like the ornamental mug illustrate a fundamental barrier to autonomously extracting semantic information: something that even humans cannot do. This in turn undermines many assumptions about the future of human–robot interaction and the nature of human perception.

Featured Image

Photo by Andy Kelly on Unsplash

Photo by Andy Kelly on Unsplash

Why is it important?

Tens of billions of dollars are being invested in household humanoid robots that will act as domestic helpers, butlers, nannies, or cooks. Is this possible? The household is an open-ended environment and its operation depends on highly local and often idiosyncratic human customs. Great advances have been made in computer vision, deep learning, grasping, and robot navigation. Robots can now avoid obstacles and deftly pick up objects. These advances conceal the more difficult problems of somehow learning and adapting to conventions of human behaviour that define how objects are to be used (also called object affordances). To know how particular objects are used in a particular household, a robot can either be told (which is a lot of work for the human users) or learn by lengthy observation (inefficient), or determine from an object's properties all its possible uses (which requires superintelligence). Investors and developers need to confront this problem which has, until now, never been explicitly stated.

Perspectives