Feelings by phone

"EmotionSense" uses speech-recognition software and phone sensors in standard smart phones to assess how people's emotions are influenced by factors such as their surroundings, the time of day, or their relationships with others.

It was developed by a University of Cambridge-led team of academics, including both psychologists and computer scientists. They will report the first successful trial of the system today at the Association for Computing Machinery's conference on Ubiquitous Computing in Copenhagen.

Early results suggest that the technology could provide psychologists with a much deeper insight into how our emotional peaks - such as periods of happiness, anger or stress - are related to where we are, what we are doing or who we are with.

"Everyone has a mobile phone, so potentially they are a perfect tool if you want to track the behaviour or emotional condition of large numbers of people," Dr. Cecilia Mascolo, from the University of Cambridge's Computer Laboratory, who led the research, said.

"What we are trying to produce is a completely non-intrusive means of achieving that which also respects privacy. In time, it could have an enormous impact on the way in which we study human behaviour and give psychologists a deeper insight into what it is that makes different types of people tick."

EmotionSense uses the recording devices which already exist in many mobile phones to analyse audio samples of the user speaking. The samples are compared with an existing speech library (known as the "Emotional Prosody Speech and Transcripts Library") which is widely used in emotion and speech processing research. The library consists of actors reading a series of dates and numbers in tones representing 14 different emotional categories.

From here, the samples are grouped into five broader categories - "Happy" emotions (such as elation, or interest); "Sadness"; "Fear", "Anger" (which includes related emotions such as disgust) and "Neutral" emotions (such as boredom or passivity.

The data can then be compared with other information which is also picked up by the phone. Built-in GPS software enables researchers to cross-refer the audio samples with the user's location, Bluetooth technology can be used to identify who they were with and the phone also records data about who they were talking to and at what time the conversation took place.

The software is also set up so that the analysis is carried out on the phone itself. This means that data does not need to be transmitted elsewhere and can be discarded post-analysis with ease to maintain user privacy.

As reported in their conference paper, the research team tested the effectiveness of the system on a group of 18 volunteers at the University of Cambridge earlier this year.

Each subject was given a modified Nokia 6210 Navigator phone for a period of 10 days. They were also asked to keep a diary in which they recorded their emotional state according to a standard set of questions already used by social and behavioural psychologists.

The results showed that in about 70% of cases, the emotional analysis offered by the phone system agreed with the results of the survey, suggesting that with further modification this type of mobile phone technology could be a very accurate means of tracking the factors influencing people's emotions.

The pilot study also threw up some interesting suggestions about how circumstances may affect our emotional condition. Location appeared to have a pronounced effect on the users' state of mind. "Happy" emotions dominated the data when they were in residential locations (45% of all emotions recorded), but in workplaces "sad" emotions became the norm (54%).

The researchers also found that users exhibited more intense emotions in the evening than in the morning and that people tended to express their emotions far more in smaller groups than in larger crowds.

The research team is now working to refine the system further, by improving its emotion classification and its response to background noise.

Dr. Jason Rentfrow, a social psychologist at the University of Cambridge who also took part in the research, said: "This technology has the potential to transform the ways in which scientists study psychological states and social behaviour. The methods most often used rely on self-reports, which are subject to a number of limitations - people forget certain details and are sometimes inaccurate at reporting how often they engaged in particular tasks. Mobile sensing technology can overcome those limitations, providing unobtrusive and objective information about social behaviours and activities."