Researchers develop infrared-based system to read body language

A joint research team from the University of Cambridge and Dartmouth College has developed a system for using infrared light tags to monitor face-to-face interactions. The technique could lead to a more precise understanding of how individuals interact in social settings and can increase the effectiveness of communications coaching.

The system, named Protractor by the Cambridge-Dartmouth team, uses invisible light to record how people employ body language by measuring body angles and distances between individuals. 

Prior studies have revealed that body language can influence many aspects of everyday life including job interviews, doctor-patient conversations and team projects. Each Protractor setting includes a specific set of interaction details such as eye contact and hand gestures for which an accurate monitoring of distance and relative orientation is crucial. 

“The ability to use invisible light to determine someone’s role and attitude in social settings has powerful implications for individuals and organisations that are concerned about how they communicate,” said Professor Cecilia Mascolo from Cambridge's Department of Computer Science and Technology, who led the research. 

Body language is already commonly studied through video sessions, audio recordings and paper questionnaires. Compared to the new, light-based system, these approaches can require invasive cameras, necessitate complex infrastructure support, and impose high burdens on users. 

“Our system is a key departure from existing approaches,” said co-author Xia Zhou from Dartmouth. “The ability to sense both body distance and relative angle with fine accuracy using only infrared light offers huge advantages and can deepen the understanding of how body language plays a role in social interactions.” 

Protractor is a lightweight, wearable tag resembling an access badge worn with a lanyard or clip. The device measures non-verbal behaviour with fine granularity by using near-infrared light from photodiodes. The light technology operates at a wavelength commonly used in television remote controls. 

Before settling on infrared light for the unit, the research team also considered ultrasound and radio frequency. In addition to the overall accuracy, infrared was favourable because light cannot penetrate human bodies, ensuring the accurate sensing of face-to-face interaction. Near-infrared light is also imperceptible to human eyes and keeps the sensing unobtrusive.

Although well-suited for measuring body language, the research team needed to correct for when a user’s hand or clothing could temporarily block the light channel. They did so by designing algorithms that exploit inertial sensors to work around the absence of light tracking results. 

In demonstrating the system, the researchers also had to devise a way for the sensors to accurately identify participants and to limit power consumption. 

“By modulating the light from each Protractor tag to encode the tag ID, each tag can then figure out which individuals are participating. To increase energy efficiency, we also adapt the frequency of emitting light signals based on the specific context,” said co-author Zhao Tian, a PhD candidate at Dartmouth.  

To study the technique’s effectiveness, the team used the Protractor tags to track non-verbal behaviours during a problem-solving group task known as “The Marshmallow Challenge.” In this task, teams of four members were given 18 minutes to build a structure that could support a marshmallow using tape, string and a handful of spaghetti. 

“Beyond simply observing body language with the tags, we identified the task role each group member was performing and delineated each stage in the building process through the recorded body angle and distance measurements,” said Alessandro Montanari, a researcher at the University of Cambridge. 

In the study of 64 participants, Protractor achieved 1- to 2-inch mean error in estimating interaction distance and less than 6 degrees error 95 percent of the time for measuring relative body orientation. The system also allowed researchers to assess an individual’s task role within the challenge with close to 85 percent accuracy while identifying stages in the building process with over 93 percent accuracy. 

According to the research team, the system will not only support social research, but it can also potentially provide real-time feedback during interviews and other interactions. Trainers, supervisors and team facilitators can use these findings to better understand team dynamics and intervene during intense problem-focused discussions to achieve higher creativity.

Protractor can also help study the impact of culture on body language in light of research that shows that cultural backgrounds can impact the way people think, feel, and act while working with others – an important feature in today’s highly-internationalized workplaces. 

Researchers at Maastricht University and the University of Nottingham also contributed to this study.

The research was published in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies and will be presented during UbiComp’18.

Adapted from a Dartmouth press release.