What Makes a (Virtual) Face Human?

According to the United Nations, on November 15, 2022, the world population reached 8 billion people. Eight billion faces, of which each one of us can remember about 5,000, on average. And we recognize a face we remembered fast, within half a second. Quite an accomplishment!

We master face recognition skills already when we are around four months old. Right after birth we seem to be able to distinguish familiar from unfamiliar faces. And that again is quite an achievement given that our eyesight has not fully developed yet at that age, its range estimated to be limited to only 12 inches. Human face recognition is what humans are good at.

But what is it that makes those 8 billion faces human?

The question of what makes a human face human has become increasingly important as robots and virtual humans become more present around us. How do we develop virtual humans that behave but also look like human-intelligent tutors, chatbots, or elder-care assistants? And with every technology also having its downsides, how do we recognize a deepfake from a "deep real"?

Answering the question of what makes a face human-like is not that easy. In 1970, Japanese robotics professor Masahiro Mori proposed the so-called "uncanny valley hypothesis". Mori argued that when robot faces become more human, emotional response to the robot becomes increasingly positive. That may not be very surprising. The more human a face becomes, the more familiar the face becomes.

But here comes the important part: When a robot becomes almost entirely human-like, just right before it has become most humanlike, the rising positive feelings suddenly become extremely negative. The human-like robot does not look familiar any longer but looks eerie and almost repulsive. In other words, if developers make robots most human-like they run the risk of entering the uncanny valley of eeriness and repulsiveness.

Uncanny Valley effect (Mori, 1970)

Source: Max Louwerse

The uncanny valley is strange. After all, when we experience the picture of a stick figure that is hardly human-like and does not at all look familiar, we do not have any eerie feelings. We don’t care about the lack of eyebrows in the stick figure, the position of the mouth, or the lack of twinkle in the stick figure’s eyes. And yet when a robot is almost perfectly human-like, we start to get concerned exactly about these things: The eyebrows are too dark, the position of the mouth is a bit off, the eyes don’t quite look right, or the skin looks too smooth.

So what then what is it in the face that makes it most human-like? In a series of experiments we have tried to address this question using both human faces and artificial human-like faces.

In one experiment we asked participants whether they found a face human or not. We presented them with a series of artificial faces of virtual humans and tried to find out why participants found some of these faces more human-like than others. A computational model showed that the eyes and the skin provided the answer. When faces of virtual humans showed no reflections in the eye or when the face showed very smooth skin, the virtual humans were assessed less human-like than when the skin was less uniform and smooth and when the eyes showed no or few reflections.

Virtual human faces and the percentage of participants who rated them humanlike (Vaitonyte et al., 2021)

Source: Julija Vaitonytė, Pieter Blomsma, Maryam Alimardani, Max Louwerse

The computational task was followed up by an experiment in which we manipulated pictures of human faces. The pictures of these human faces had natural human skin, and we smoothed the skin; the eyes had their natural light reflections, and we removed those reflections. The faces still looked very human-like but were manipulated on these two aspects: the eyes and the skin.

In a face perception experiment, we used human faces and manipulated the eyes and skin in the pictures. The perfectly human faces now had a reflection in their eyes removed and/or had the skin texture be smoothed. It took participants significantly more time to evaluate when a face was human or not in the manipulated (smooth skin, eye reflections removed) conditions.

And it is not only a matter of careful evaluation of the human faces that shows eyes and skin matter in determining whether a face is human-like. In a more general cognitive task, we asked participants to remember whether they had seen a face with or without corneal reflections and with or without a smoothened skin. The findings also show up in memory. Both when participants saw virtual human faces and when they saw real human faces, increased corneal reflections and skin contrast yielded better memory for that face.

Memory Essential Reads

The Psychology of Memory

Can You Trust Your Memories?

These findings are relevant for cognitive and social psychologists alike. But they are also relevant for developers. On the one hand, it is better to stay on the safe side of the uncanny valley and not aim for human-like artificial faces. On the other hand, we need to understand what helps us cross the uncanny valley, so that intelligent tutoring systems, customer support systems, and clinical assistants become more human-like in behavior and looks.