Methods of Emotion Recognition

In this section we shall examine some of the common methods of image recognition. Images and much of the information in this section are taken (ruthlessly without permission) from Hal's Legacy^[1].

Above we see a number of "color motion energy maps". The figure in the top row on the far left is a photo of a person with a "neutral" expression on his face. Successive photos to the right show a number of different emotions. By recording levels of movement in various parts of the image relative to the neutral image, and assigning colors to these levels, we generate the energy maps. As you can see, the energy maps are quite distinct from each other, and are thus quite easy for a compute to differentiate between.

These particular photos are from a test involving eight individuals selecting for the four emotions displayed. Recognition rates were as high as 98%.^[1]

A variety of other methods can be used for "optical" emotion recognition, such as observing the motion of individual muscles or features of the face. We can find examples of such experiments using neural networks^[4], or even more exotic systems using fuzzy logic^[5] to track movement of points on a face.

In the image above we see another method of emotion recognition; voice analysis. In each image, the red lines at the top represent frequency information (the pitch of the voice), whereas the blue fuzzy blobs beneath represent a spectral analysis and can be used for, amongst other things, determining the amplitude (the loudness) of the spoken words.

The top image would represent someone saying "I thought you really meant it." with a sad inflection. Note the fairly flat pitch throughout with a downward inflection at the end, the soft tone used, and the fact that the words are fairly spaced out. The bottom image would represent someone saying the same thing with an angry inflection; here the pitch is much more varied, the words are said much more loudly, and the words come closer together.

Voice data is useful in some circumstances, but in the majority of our common interactions with computers we don't speak to machines (yet, anyway).

There are also a wide variety of other sensors which can be used to sense emotion. To the left we see a girl wearing a number of "affective sensors". She is wearing a skin conductivity sensor on two of her fingers, a pulse volume sensor on one of her fingers, a respiration sensor around her torso, an electromyogram (measures muscle actuation) on her cheek, and a temperature sensor attached to her neck.

These sensors combine to provide a wide variety of data about her physical state from which we can read a great deal about her emotion; skin conductivity will increase with nervousness, anger, or fright, respiration will increase with anger or frustration, temperature will increase with anger and under duress, facial muscles will contract when happy or in the opposite direction when upset.

As you can see from the above photo, affective sensors are at present somewhat bulky and cumbersome. These sensors will, like other technology, continue to get smaller and easier to use. We already interact with our machines physically in a number of manners.

It would be trivial to add skin conductivity, temperature, and pulse sensors to most of today's keyboards and mice, as well perhaps other sensors such as velocity sensitive keys to measure the force with which one types. Cameras can be mounted on top of monitors. A lot of these sensors could also be added to a portable device, such as an electronic organizer.

Machines which Currently Recognize Emotion

There are a variety of machines which we use today which use a variety of methods to acquire affective information from us.

One such example is the TiVo^[6] from TiVo inc. This device is a kind of digital VCR. It records what you are watching continuously and allows you to rewind, or pause in a live broadcast. You can walk into your living room 15 minutes into "Star Trek", start at the beginning of the episode, and "fast forward" through the commercials, gradually bringing yourself up to "real-time". It will also allow you to pick shows you wish to record, and it will record them for you whenever they come on, without the need for you to explicitly tell it what channels the shows are on. Most of its "intelligent" behavior comes from program information it downloads from a central server every night through your phone line.

The interesting feature of this device, as far as affective computing is concerned, is that based on which shows you like and dislike, it will record other shows it thinks you might like automatically. How does this device determine what shows you like and dislike? Its affective sensors are quite simple, a huge green "thumbs up" button and a red "thumbs down" button on the remote. These buttons allow you to "rate" programs, giving them up to three thumbs up or down.

Another example of current-day emotion sensing technology is a video game system developed by NASA^[7] which uses affective sensors to reduce stress. NASA's system uses a technique known as biofeedback. A network of sensors connected to the head of an individual record brainwave activity. As the players brainwaves move toward alphawave-like activity (brainwaves associated with calm and low-stress) the game becomes easier to control. This encourages players to reproduce these brainwave patterns, and can be used to treat problems such as physical stress and attention deficit.

Machines which Will Recognize Emotion

The applications of this technology are widespread.

This technology could easily be used to provide qualitative feedback to designers of vehicles, buildings, and other spaces. A "buyer feedback form" provides these designers and engineers with some limited information about what people liked and disliked about their design, but imagine the kind of information they could get from someone carrying a portable device which recorded their emotional reactions as they moved about in a building, or drove around in a car; aspects of the environment which were confusing or frustrating could be much more easily found and eliminated, and the aspects of the environment which the user enjoyed could be expanded upon. This kind of affective information could also be gathered less invasively, using cameras in a building, or by adding affective sensors to the controls and seat of a vehicle.

Another very attractive area in which this technology could be employed is in education. Imagine a system designed to teach piano. Such a system could detect if a student was overly frustrated, and provide them with simpler pieces to learn. On the other end of the scale, such a system could detect if the student was bored or finding the pieces to easy, and give them more complicated pieces. The overall effect would be to moderate the material given to the student to play, making some challenging as this is when learning is optimal, and some inside the students' abilities as this provides material which gives a sense of reward^[1].

We have already talked about using affect sensing technology to help reduce stress; imagine though affect sensors connected to a portable device, such as an electronic organizer. We could record information such as stress levels throughout the day, giving doctors an indication of what activities we perform that we find overly stressful and that we should thus avoid.

The applications of this technology can be found in any domain, and are too numerous to list here. Suffice to say that much research is being done and will be done in this field.

This site copyright Jason Walton, 2000