A Galvactivator-Enhanced Instant Messenger Client
MAS 630: Affective Computing Final Project

Andrew T. Fiore <>
Vidya Lakshmipathy <>
Joan M. DiMicco <>

The Problem

While instant messaging clients are frequently and widely used for interpersonal communication, they lack the richness of face-to-face conversations. Specifically, the emotional content of a conversation is limited when the communication is restricted to text-only expression.

Our intention was to incorporate additional affect information into a text-based messaging interface with the goal of enriching the communication experience. By providing an additional channel of affective information, we hoped the application would enable for more emotionally intelligent conversations. On an expressive level, we strove to build an interface which communicated each user's affect in a consistent, intuitive manner, without needing explicit controls or explanations. On an experimental level, we wish to use this application to find out if adding new affect information to a text-based chat program can improve the users' abilities to communicate their intent and desires.

Background Research

Information richness is defined by Daft and Lengel as "the ability of information to change understanding within a time interval". Communications that can overcome different frames of reference and clarify ambiguous issues to promote understanding in a timely manner are considered more rich. Communications that take a longer time to convey understanding are less rich. According to Daft and Lengel's theory, media richness is a function of (1) the medium's capacity for immediate feedback, (2) the number of cues and channels available, (3) language variety; and (4) the degree to which intent is focused on the recipient. The greater social presence of a medium creates a greater immediacy and warmth of the communication, because of the greater number of channels. Using this criteria, instant messaging falls between the telephone and addressed written communication (for example, email) on a scale of information richness.

Studies have been done comparing communication methods with different numbers of communication channels, however few studies have looked at increasing the number of channels of one medium and studying it's effects on communication using that same medium. When comparing communication modes on a scale of decreasing richness, each successive mode has only a few channels inherent to face to face communication, but each has a different combination of these channels. Affective information conveyed through galvanic skin response (GSR) is a channel not present in face to face communication and as a result is a data channel has yet to be explored in the area of communication method studies.

The additional richness of providing affect information through GSR allows the recipient of the data to make a more informed assumption about the state of the user. based on some feedback about his or her affective state. Regardless of whether the affect data is perfectly timed with the text being written or if the change in affect is directly in resonse to the conversation at hand, this additional data has the ability to change the behavior of the recipient or at least provoke him or her to inquire about the affect state of his/her chat partner. In some ways, this channel might be more suited for computer mediated communication because it allows the data to be exchanged in a subtle, nonverbal, passive way. It does not require additional thought or work by the user and is reflected rather subtly and unobtrusively to the recipient.

The Application

The application, called ConductiveChat, incorporates affect arousal information into the text exchanged between two chatting instant messengers. The chat interface uses a user's galvanic skin response, as measured by the Galvactivator, to modulate the font color and size as it is typed. The color of the font (between black and red on the RGB scale) communicates the user's GSR level and the size of the text relates to the rate of change in GSR within the previous two seconds. The effect is an expression of higher levels of arousal by brighter written text and an expression of rapid increases in arousal through an increase in font size. Figure 1 below shows a screen shot of the application interface and an example of how the color of the text changes as the arousal level of the chatters increases, and the text size changes with instances of great increase in arousal.

Figure 1: screen shot of conversation


The instant messenger used for our application is a modification to the open-source client for AOL Instant Messenger, GAIM 0.55, which runs on Unix platforms with GTK. The client reads data from a Galvactivator connected to the serial port of a Linux computer via an IRX board with a PIC16C711 microcontroller performing analog-to-digital conversion. The modified GAIM client uses these data to adjust the size and color of the typed text on a per-letter basis.

Design considerations

We sought to incorporate the skin conductivity signal into our IM interface in an unobtrusive but readily useful way. The Galvactivator glove itself has a red LED whose brightness indicates the current level of skin conductivity on the wearer's palm. A single indicator, like a glowing circle on the screen, would provide equivalent information to a remote messaging partner. But conductivity varies over time, so the level when the user starts typing a message might not be the same when she finishes, nor will it be consistent from statement to statement. To reflect this, we wanted to express the temporal dimension of the data as well.

We considered a graph of the conductivity signal versus time, but then we realized that the text messages themselves also serve as a horizontal timeline. So we decided to color each letter of the text to convey the skin conductivity of the user as she types the letter. This encodes the temporal progression of the conductivity relative to the message as it is typed. Additionally, because the information lives in the text itself, the user need not look for or try to understand a separate display.

The absolute value of the skin conductivity signal provides an important conversational cue, but sharp rises and declines in its value also have meaning, such as when the remote user becomes flustered or embarrassed over a new topic. To punctuate such occurrences, we use the size of the characters in each message to encode the rate of change of the conductivity signal. Quickly increasing conductivity causes larger letters; quick decreases make smaller letters. Only large, rapid changes cause visible differences in the present version. Earlier versions were sensitive to much smaller fluctuations, but the jumble of different sizes of letters made the text hard to read. (Even if this weren't the case, these small fluctuations were likely artifacts of the Galvactivator and the analog-to-digital conversion rather than true variations in conductivity.)


Our evaluation of the interface involved both us using the interface for functional evaluations as well as a longer evaluation persiod with a naive user chatting with one of us. In our evaluation with the naive user, we explained the application and how the Galvactivator operated, but did not the algorithm for how the text would change in response t Galvactivator input.

Through this evaluation, we observed several things about the interface:

  • The arousal level (shown by color) of the two users moves in sync, rising and falling as the excitement level of the conversation changes.
  • As previously observed with the Galvactivator, there is an adjustment period where the user's skin conductivity is low, and then eventually begins to rise and fall.
  • The derivative calculation at this point in our development was too crude and caused the font to change too greatly between two neighboring letters. (See Design Considerations for our enhancement)

    Our user provided some of the following insights into the interface:

  • He did not instinictively know there was a difference in the calculation of font color and size.
  • He learned how to manipulate his chat font by squeezing his palm to increase the Galvactivator's contact with his palm.
  • He learned how to ask the remote user emotional questions to illicit a change in the font color.


    Figure 2: User 1 chatting and transmitting GSR

    Figure 3: User 2 receiving Galvactivator-enhanced message

    Extenstive exerpts from the chat transcript are provided here. What is most interesting to observe is the gradual change in font color as the topic of the conversation changes. The font size changes are extreme in this version of the interface, but still serve to mark instances of great change in arousal levels.

    Future (more formal) Evaluation

    One of our initial reasons for building this application was to study whether or not arousal information can assist people in communicating more effectively through text-based interfaces. Towards this goal, we have outlined a user study we hope to conduct in the coming months (over Summer 2002). This study will attempt to demonstrate our hypothesis: that adding an additional affect channel to text-based chat enriches the users' communication ability to an extent that tasks and decisions shared between the users can be completed more easily.

    We will have three experimental groups in the study: 1) those using ConductiveChat, 2) those using a regular GAIM client with no GSR information collected or transmitted, and 3) those using an enhanced GAIM client which allows users to explicitly manipulate a control which will transmit arousal state in the same manner as ConductiveChat.

    To examine the effectiveness of a chat interface, we will construct a scenario in which users will be instructed to perform a specific task in which one user has the role of "information seeker" and the other has the role of "information source." An example of this could be where the "information seeker" has a list of movies playing this weekend and the "information source" has access to reviews, theatres, and showtimes. Their task would be to decide which movie they will see together that weekend. By dividing up the roles there is a necessity for the two users to communicate, and by asking the users to make a decision that will impact both users, we hope to encourage some affect arousal during the communication.

    During the experiment we will meausure the time spent sending messages, receiving messages, searching for information, taking notes, and waiting for messages. We hope to answer through these measurements and through follow-up surveys is whether or not the arousal information made the users to take more or less time to reach an answer. Do the users come up with a more satisfactory answer? Do they spend more time chatting or more time seeking information from external sources? Did the arousal information change the nature of their decision-making process.

    Our experimental design was inspired by the study run by Ochman and Chapanis on cooperative problem solving through different communication modes (listed below).


    Joanie Connell, Gerald Mendelsohn, Richard Robins, John Canny, "Don't Hang Up On the Telephone Yet!" In ACM GROUP (Conf. on Group support), Sept 2001.

    Maybury, Mark T., Human-Computer Interaction: State of the Art and Further Development in the International Context - North America. International Status Conference, Saarbruecken, 26-27 October 2001.

    Ochsman R.B., and Chapanis A. The Effects of 10 Communication Modes on the Behaviour of Teams During Co-operative Problem-Solving. International Journal of Man-Machine Studies, 6, 1974, 579-619.

    Scheirer, Jocelyn and Rosalind W. Picard, Affective Objects, MIT Media Laboratory Perceptual Computing Section Technical Report No. 524

    Thanks to....

    We received lots of invaluable advice and assistance from students around the lab. Among these generous people were:

    Raffi Krikorian
    Carson Reynolds
    Josh Lifton
    Stefan Marti
    Will Glesnes