Smile, you’re on camera: Virginia Tech research gauges public support for facial profiling technology

Should you judge a book by its cover? Or a Netflix show by its thumbnail?

We often judge other people simply by looking at them. We make quick inferences about their personality and character — whether they would be kind, funny, intelligent, or rude — after a few seconds.

“And now, artificial intelligence technologies claim to infer our personality and character just by looking at our pictures,” said Shilpa Madan, assistant professor of marketing at Virginia Tech’s Pamplin College of Business.

Companies are using facial profiling in their recruitment processes to judge applicants’ employability, customers’ trustworthiness to pay back their loans, a person's ability to take risks, and more. Of course, the technology also is being used for things that would be considered fun. Facial profiling-based smartphone apps claim to test your compatibility in love and friendship based on a selfie that you upload to the app.

As technology has advanced at an exponential rate over the last few decades, some businesses have taken the adage “the face is the window to the soul” quite literally.

However, apart from straightforward privacy and security-related concerns, facial profiling raises multiple issues about discrimination and bias. Given these concerns, why do people support — even actively participate in — facial profiling?

According to Madan’s research, published in the Journal of Personality and Social Psychology, “How you look is who you are: The appearance reveals character lay theory increases support for facial profiling,” one reason people support even extreme uses of facial profiling is that they believe that an individual’s appearance can indeed accurately reveal their character.

This means that the concern behind the use of facial profiling is not the technology being used or even what it is being used for. Rather, it is the belief that there is a reliable association exists between an individual’s appearance and character.

'Only bad witches are ugly'

The belief that there is a connection between how an individual looks and their character is not a new idea. “This has been part of human discourse since at least 300 B.C.,” Madan said. “As humans, we have always placed a lot of value on someone’s appearance.”

We can even see it in pop culture. In television and movies, heroes are often portrayed as handsome or beautiful, while villains are seen as ugly and abhorrent. In the classic film “The Wizard of Oz,” when Dorothy asks Glenda the Good Witch why she is so beautiful, Glenda retorts that “only bad witches are ugly.”

However, the science behind the assertion that people’s appearance can reveal their character is mixed at best. As Madan’s research shows, this does not stop people from believing that the connection exists.

Across nine studies conducted with nearly 3,000 participants, Madan’s research found that the more people believed that individuals' appearance reveals their character, the more they supported adopting facial profiling technologies. Included in the research were extreme examples of facial profiling, such as allowing the police to arrest someone because the facial profiling software thought they looked like a “criminal,” putting a student in a remedial class because they did not look “intelligent,” or allowing financial institutions to charge higher interest rates to certain customers because they didn’t look “trustworthy” enough.

“The more someone believed that appearance reveals character, the more confident they were that others could also judge character by appearance,” said Madan. “That then increased their support for facial profiling.

Interestingly, the target of facial profiling did not matter, according to Madan’s research. “You would assume that people would strongly object to being profiled themselves,” she said. However, participants were just as likely to support the use of facial profiling on themselves or their families if they believed that people’s appearance reveals their character.

“People generally believe they have good character,” she said. “That they have nothing to hide.”

Diving into the next research frontier

Madan’s research – with its strong societal implications around personal privacy and security – highlights the need for increased scientific discovery into facial profiling and the field of artificial intelligence at large.

Fortunately, Virginia Tech researchers are already answering this charge. In strong alignment with the university’s AI research frontier, Tech for Humanity initiative, Pamplin’s strategic pillar focused on advancing the human condition, and the marketing department’s focus on consumer well-being and technology, Hokies continue to explore the impact of technology innovation through a human-centered lens through research, academics, and experiential learning.

“Facial profiling is becoming more and more common, both in government and business, yet people do not understand its implications and ramifications. Scientists need to be clear about what can be determined – or not – by facial profiling, because the rest of the world will form their beliefs based on this consensus,” said Madan.

If not, the consequences could be dystopian.