ConductiveChat
A Galvactivator-Enhanced Instant Messenger Client
MAS 630: Affective Computing Final Project

Andrew T. Fiore <atf@media.mit.edu>
Vidya Lakshmipathy <vidya@media.mit.edu>
Joan M. DiMicco <joanie@media.mit.edu>

The Problem

While instant messaging clients are frequently and widely used for interpersonal communication, they lack the richness of face-to-face conversations. Specifically, the emotional content of a conversation is limited when the communication is restricted to text-only expression.

Our intention was to incorporate additional affect information into a text-based messaging interface with the goal of enriching the communication experience. By providing an additional channel of affective information, we hoped the application would enable for more emotionally intelligent conversations. On an expressive level, we strove to build an interface which communicated each user's affect in a consistent, intuitive manner, without needing explicit controls or explanations. On an experimental level, we wish to use this application to find out if adding new affect information to a text-based chat program can improve the users' abilities to communicate their intent and desires.

Background Research

Information richness is defined by Daft and Lengel as "the ability of information to change understanding within a time interval". Communications that can overcome different frames of reference and clarify ambiguous issues to promote understanding in a timely manner are considered more rich. Communications that take a longer time to convey understanding are less rich. According to Daft and Lengel's theory, media richness is a function of (1) the medium's capacity for immediate feedback, (2) the number of cues and channels available, (3) language variety; and (4) the degree to which intent is focused on the recipient. The greater social presence of a medium creates a greater immediacy and warmth of the communication, because of the greater number of channels. Using this criteria, instant messaging falls between the telephone and addressed written communication (for example, email) on a scale of information richness.

Studies have been done comparing communication methods with different numbers of communication channels, however few studies have looked at increasing the number of channels of one medium and studying it's effects on communication using that same medium. When comparing communication modes on a scale of decreasing richness, each successive mode has only a few channels inherent to face to face communication, but each has a different combination of these channels. Affective information conveyed through galvanic skin response (GSR) is a channel not present in face to face communication and as a result is a data channel has yet to be explored in the area of communication method studies.

The additional richness of providing affect information through GSR allows the recipient of the data to make a more informed assumption about the state of the user. based on some feedback about his or her affective state. Regardless of whether the affect data is perfectly timed with the text being written or if the change in affect is directly in resonse to the conversation at hand, this additional data has the ability to change the behavior of the recipient or at least provoke him or her to inquire about the affect state of his/her chat partner. In some ways, this channel might be more suited for computer mediated communication because it allows the data to be exchanged in a subtle, nonverbal, passive way. It does not require additional thought or work by the user and is reflected rather subtly and unobtrusively to the recipient.

The Application

The application, called ConductiveChat, incorporates affect arousal information into the text exchanged between two chatting instant messengers. The chat interface uses a user's galvanic skin response, as measured by the Galvactivator, to modulate the font color and size as it is typed. The color of the font (between black and red on the RGB scale) communicates the user's GSR level and the size of the text relates to the rate of change in GSR within the previous two seconds. The effect is an expression of higher levels of arousal by brighter written text and an expression of rapid increases in arousal through an increase in font size. Figure 1 below shows a screen shot of the application interface and an example of how the color of the text changes as the arousal level of the chatters increases, and the text size changes with instances of great increase in arousal.

Figure 1: screen shot of conversation

Technology

The instant messenger used for our application is a modification to the open-source client for AOL Instant Messenger, GAIM 0.55, which runs on Unix platforms with GTK. The client reads data from a Galvactivator connected to the serial port of a Linux computer via an IRX board with a PIC16C711 microcontroller performing analog-to-digital conversion. The modified GAIM client uses these data to adjust the size and color of the typed text on a per-letter basis.

Design considerations

We sought to incorporate the skin conductivity signal into our IM interface in an unobtrusive but readily useful way. The Galvactivator glove itself has a red LED whose brightness indicates the current level of skin conductivity on the wearer's palm. A single indicator, like a glowing circle on the screen, would provide equivalent information to a remote messaging partner. But conductivity varies over time, so the level when the user starts typing a message might not be the same when she finishes, nor will it be consistent from statement to statement. To reflect this, we wanted to express the temporal dimension of the data as well.

We considered a graph of the conductivity signal versus time, but then we realized that the text messages themselves also serve as a horizontal timeline. So we decided to color each letter of the text to convey the skin conductivity of the user as she types the letter. This encodes the temporal progression of the conductivity relative to the message as it is typed. Additionally, because the information lives in the text itself, the user need not look for or try to understand a separate display.

The absolute value of the skin conductivity signal provides an important conversational cue, but sharp rises and declines in its value also have meaning, such as when the remote user becomes flustered or embarrassed over a new topic. To punctuate such occurrences, we use the size of the characters in each message to encode the rate of change of the conductivity signal. Quickly increasing conductivity causes larger letters; quick decreases make smaller letters. Only large, rapid changes cause visible differences in the present version. Earlier versions were sensitive to much smaller fluctuations, but the jumble of different sizes of letters made the text hard to read. (Even if this weren't the case, these small fluctuations were likely artifacts of the Galvactivator and the analog-to-digital conversion rather than true variations in conductivity.)

Evaluation

Our evaluation of the interface involved both us using the interface for functional evaluations as well as a longer evaluation persiod with a naive user chatting with one of us. In our evaluation with the naive user, we explained the application and how the Galvactivator operated, but did not the algorithm for how the text would change in response t Galvactivator input.

Through this evaluation, we observed several things about the interface: