By Yujie Hong | DDes’19,  Harvard GSD

The rail is too steep to climb (visual materials of this gif are from YouTube)

– Idea/Vision

My aspiration in this project is to propose a conceptual framework for augmenting designers’ intellect in design decision-making, by integrating VR/ AR technology and tangible inputs/outputs. I have been exploring the topic of applying AR/VR technology in architecture field for a long time. Taking this class really helped me narrow down my focus into some subareas. Partially inspired by the AfterMath project, the subarea I decided to pursue is assisting design decision-making. One proof-of-concept prototype is Structure.Ed, an AR+Lego toolkit that assists children to gain knowledge about structural engineering. After playing around with lego bricks, children can point phone camera at the lego structures they just made and the AR phone app will visualize potential consequences of assumed actions over them (eg. see if a bridge structure is stable or not when some heavy trucks run through it). I didn’t end up spending too much time in finalizing the implementation as I think developing the vision is more important and it provides a good guideline for future prototypes design.

– Motivation

In design process, architecture psychology is widely concentrated on “how will it look in actual?” Designers always think about their design and dream about how will it look after construction/ fabrication. This question usually makes a void for a very long period of time. But VR/ AR can give them all the answers they are looking for. It will connect a bridge between the present design and finished constructions, which might happen years after. In my master thesis project at Harvard, I did a user study to test architectural designers’ perceptions of scale in VR vs. in Rhino. The result shows that designers have much better estimations about scales of objects in virtual reality rather than in Rhino. As mentioned in Doug Engelbart’s conceptual framework, “advances in computing power, simulation, sensors, data analysis and machine learning make it highly plausible that computers will assist our decision-making process by providing a reliable estimate of the future-given situational and behavioral contexts of user interactions”. That is to say, as more information related to the problem has been collected, we can approach the problem more easily. By augmenting architects’ intellect in scale perceptions using VR/ AR, I believe they would feel more confident in making design decisions.

– Background Work (scientific or theoretical support)

Structural Engineering is a discipline that includes all the engineering knowledge related to the design of building. It is widely taught as a course in most architecture schools. Structural Engineering involves physics laws and empirical knowledge of the structural performance of different materials and geometries. “Structural engineers are trained to understand, predict, and calculate the stability, strength and rigidity of built structures for buildings and nonbuilding structures, to develop designs and integrate their design with that of other designers, and to supervise construction of projects on site. ”

“Structural engineers often specialize in particular fields, such as bridge engineering, building engineering, pipeline engineering, industrial structures, or special mechanical structures such as vehicles, ships or aircraft.” “Throughout ancient and medieval history most architectural design and construction was carried out by artisans, such as stone masons and carpenters, rising to the role of master builder.” “The physical sciences underlying structural engineering began to be understood in the Renaissance and have since developed into computer-based applications pioneered in the 1970s.”

Currently, there are many architecture firms and institutes investing a lot of resources and energy into the field of VR/AR. Most applications are designed as architectural space walk-throughs for facilitating architect-client communications. Interactive-wise, there are some interesting interior design tools for people to customize a virtual indoor space. However, AR/ VR education of structural engineering is a relatively under-explored research area.

From my perspective, the nature of structural engineering discipline involves physics knowledge (which can be calculated by mathematical equations like what have been explored in the AfterMath project) and empirical knowledge (involves predicting if certain actions like cracks will take place or not). It makes structural engineering education a perfect application area for AR-assisted design decision-making research.

– Related Work

Compared to virtual reality, augmented reality or mixed reality show greater potentials in construction projects. For example, employees at Gilbane Building Company, a commercial construction firm based in Rhode Island, now use HoloLens in construction sites. By overlaying the virtual models in the real site, you can look around and walk in the virtual design so you can check the pipelines and structures. It helps the company to notice real-world planning problems before building materials are wasted. ClayVision is an AR application that applied to urban scale. The system dynamically recognizes buildings in the city and transforms them into better-designed versions. It provides people with different experiences in the newly “elastic” city. Using a situated see-through display, HoloDesk allows users to manipulate virtual 3D models as if they are real objects. You can even use real object as a digital input to interact with the virtual models.

– Design and Implementation

Structure.Ed is an AR+Lego education toolkit for children to gain basic knowledge of structural engineering. First, children assemble a structural using lego blocks, for example, a bridge. Second, children can point the camera at the lego structure and the AR app will visualize potential consequences after simulating some trucks and cars running through this bridge. Third, designers may take advantage of the well-designed structure design as opposed to unstable structures in their design decision-making for real construction projects.

This proof of concept prototype displayed in this video is composed of GIFs I made by grabbing some visual materials from YouTube. But it demonstrates how the application can be used. For future implementation concerns, I plan to use Vuforia AR SDK and Unity 3D, which are all tools that I’m already familiar with.

– Usage Scenario

Assume an architect has designed a construction plan and he wants to make several design decisions to avoid problems. He can make a miniature model and points the AR camera at the model. He may changes the scale of the digital overlays to view his design in different sizes. And he can “modify” certain structures (like in the ClayVision project) to see if there is improvement. He may run simulations to see if the pipelines are well connected and circulated. And after he makes sure of everything, he can apply the modified version of design plan into real construction, to make a real bridge, road or building.

– Conclusion and Future Work

In this final project, I propose a conceptual framework called Augmenting Designers’ Intellect in Design Decision-Making by AR Technology. I made a proof-of-concept prototype that utilizes AR technology to visualize potential consequences of different structural design plans. It can also be treated as an educational tool for teaching this type of knowledge. I have shown the video demo to many friends and I also posted it on social media. I have received many positive feedbacks. Some people who work in the education business industry think it could be a very promising application if it is actually made and used in K12 education.

– Citations/Bibliography

Sang-won Leigh, and Pattie Maes. AfterMath: Visualizing Consequences of Actions through Augmented Reality. CHI 2015 Work-in-progress, 2015.

https://en.wikipedia.org/wiki/Structural_engineering

Takeuchi, Y., Perlin, K., ClayVision: the (elastic) image of the city, In Proc. CHI ‘12, 2012.

Zoran, A., Shilkrot, R., Paradiso, J., Humancomputer interaction for hybrid carving, In Proc. UIST’ 14, 2014.

Hilliges, O., Kim, D., Izadi, S., Weiss, M., Wilson, A., HoloDesk: direct 3d interactions with a situated seethrough display, In Proc. CHI ‘12, 2012.

Project Video

Idea/Vision

Although phones have become ubiquitous, and they are marginally more effective in keeping time, we still wear stylish watches and keep analog timepieces in our office to glance at every once in awhile.  Taking this into account, we have designed and produced EMIT.  EMIT is a clock that shows you your perception of time based on your emotional and cognitive state. We show the rate at which time flows for an individual by varying the speed of an extra clock hand through strobe lights. We also show the individual’s current perceived emotion by using different hues of RGB LED lights.
Time perception is complex as it is dependent on the different ways in which our brain experiences perceptions (Van Wassenhove 2008) and reconstructs those experiences in different parts of the brain (Rao, Mayer, Harrington 2001). Our goal in this project is not to find a holistic way of measuring one’s time perception, but to put together the different pieces that are known to influence time perception and to represent people’s distorted time perception to them in a meaningful and informative way. The hope is that increased awareness of time perception would trigger behavior change, e.g. take a break to reduce anxiety and slow down time perception.

Motivation

Humans are obsessed with the concept of time. We are always running after time, and there never seems to be enough of it to do everything we want. Our daily lives are scheduled metrically, governed by this unseen force that we always keep close to us.  We define this unseen force, time, as a duration/interval of one type or another. It is interesting to see that although our clocks tick at a fixed rate, our perception of time is not constant (Fraisse 1984; Joubert 1984; Gibbon 1986; Zakay 1990). Clocks show us objective time, but our subjective or mind time is what we interpret to be the duration of an interval between two events. Our subjective time is intimately tied to our emotional and cognitive (Droit-Volet, Meck  2007; Droit-Volet, Gil 2016) states.  For example, if we are happy or busy, we perceive time as going fast.  The opposite is also true if we are sad or free, time will seem to go by slow.  We wanted to build a device that could display people’s perception of time.  Through this representation, we hope to make people more conscious of their emotional and cognitive state.  We also hope that this device can serve as an indicator for other people in order to give a quick insight into a user’s state.

Background Research 

There is a lot of literature on the relationship between our emotional/cognitive state and our time perception. In the 80s researchers demonstrated that time seems to go by faster when you are expecting something positive to happen rather than when you are expecting something negative to happen (Edmonds, Cahoon, Bridges 1981).  This was one of the first clues that emotions were linked to the perception of time.  Later on in the early 2000s S. Droit-Volet, et al. (2007) went on to test this in more detail by evaluating the perceived duration of time intervals when one is staring at a smiling rather than a frowning person.  He found that positive emotions (smiling) make the intervals go faster and vice versa.  Around this time researchers also found that time appears to move faster during positive experiences and it slows down during negative emotional states, e.g. when we are sad, depressed (Gil, Droit-Volet 2009), afraid (Stetson, Fiesta, Eagleman 2007; Buetti, Lleras 2012), or lonely (Twenge, Catanese, Baumeister 2009).  These results show a clear correlation between emotions and the perception of time.   

 

Time perception is also affected by how engaged or attentive we are (Burnside 1971; Curton, Lordahl 1974; Tse, et al. 2004).  Researchers found that people who keep track of time perceive it as going slower rather than those who do not, they also found that people give shorter estimates of time when they are under a load rather than when they are not (Brown 1985).  These results also show that there is a strong correlation between the perceived and mind time and the cognitive state of the user.

 

Time perception is also tied to psychological conditions such as ADHD (Levy, Swanson 2001), schizophrenia (Franck, Pichon, Haggard 2005), impulsivity and borderline personality disorder (Berlin, Edmund 2004), and anxiety disorders (Bar-Haim, et al. 2010).  Time perception also affects our decision-making as impulsive decisions are highly correlated with an overestimation of time, i.e. a speedy perception of time (Wittmann 2008).    

 

Time perception is also a key consideration in areas of consciousness. The specious present is the time duration wherein one’s perceptions are considered to be in the present (James 1893).  Researchers have also found that mindfulness meditation lead to slower perception of time (Kramer, et al. 2013).   Chronesthesia (Tulving 2002), or the ability to go back and forth in time within your mind,  is also interpreted to be the navigation or manipulation of the subjective perception of time.
Because of all the research done on the field, we are able see that there are various factors that affect the perception of time amongst human beings.  Put together, there are 2 large areas that surface: emotion and cognition.  There is also a promising application area with regards to the modification of the perception of time to take more rational decisions.  We intend to take exploit these topics in our design of EMIT to try and conscientize users of their current perception of time and what it could be due to.  

Using the aforementioned relationships between emotional/cognitive states and time perception, we were able to create a mapping between our narrow set of emotional/cognitive state and time perception.

Related Work

In the following section we present related work regarding clocks that present new ways of visualizing time. To the best of our knowledge, there have been no other implementations of a an emotion-aware timepiece that includes the dimension of time perception embedded in a clock hand. Therefore, we focus on relevant projects that focus on representing or adjusting our perception of time.

In 1999, Mark Weiser wrote “The Computer for the 21st Century” and proposed devices that would “weave themselves in the fabric of everyday life until they are indistinguishable from it”.  There are several examples of such ‘calm’/ubiquitous computing devices. For example Breakaway – a desk sculpture that nudges users to take a break when they have been sitting too long (Jafarinaimi, et al. 2005), or PlantDisplay, which is an ambient display on plants (Kuribayashi, Wakita 2006). EMIT is a context- and emotion-aware ‘calm’/ubiquitous computing device that has an ambient display of user’s emotional and cognitive state. Nimio (Brewer, Williams, Dourish 2005) is an example of a tangible device that represents user activity using different light hues, and Tea Place (Lee, et al. 2007) is a an example which uses light to alter user’s moods.

The Durr watch is a device which tries to present the objective perception of time using a method different than clock hands: vibration.  The Durr watch vibrates every five minutes of objective time.  (Durr: http://skrekkogle.com/projects/durr/).   Durr sought to explore how a user’s perception of time varies when the only queue that they have is a consistent vibration.  However, the Durr watch was a prototype and the idea was put on hold.  Later on, a Media Lab Alumni called Che-Wei Wang made a vibration watch similar to Durr but with customizable intervals.  He explored how this would affect users.

Slow Dance is a Kickstarter project in which Jeff Lieberman tries to alter the perception of time using strobe lights.  At 80Hz, a strobing light is perceived by human eyes as constant, however, when a moving object is placed within the range of the light, the object will appear to move slower as if it were in slow motion.  We decided to exploit this behavior in our project to make objects seem slower or faster than they appear to be moving at.  (Slow Dance: https://www.kickstarter.com/projects/xercyn/slow-dance-a-frame-that-slows-down-time)

Hidden time is a clock which tries to conscientize a person of the passing hours.  The most recent hour is the clearest, and the oldest one is the least visible.  The effect that it creates is that a person should focus on the most recent hours rather than the oldest ones.  (https://www.indiegogo.com/projects/hidden-time-watch-design-fashion#/)  Similar to this is the Circadian clock, which presents a simpler interface to the regular clock.  As night time comes, the watch face becomes black and vice versa in daytime it becomes light (Circadian clock: http://abduzeedo.com/circadian-clock)

Design

 

In this section, we present the design process of the EMIT system. There were 3 main guidelines for our design:

1. Normal clock: It would ‘look’ as close to a normal clock as possible so that people may be aware of their subjective time as they look at the clock’s objective time. Most people are familiar with the concept of objective time on clocks, and clocks are a common object in people’s homes/offices. This way, we are augmenting an already acceptable and used product.
2. Extra hand and strobe light: We would add an extra hand to the clock and the speed of this extra hand would represent our subjective time. As time slows down for a person, the extra hand would go in slow motion, and vice versa. We decided to use the strobe light effect here as it allows us to change the perception of an object moving at a constant rate. Using strobe lights, we could show the hand completely frozen or moving in slow motion (forward and backward) or quickly jumping between positions (again, forward and backward).
3. Colors: People can perceive time as slowed down for different reasons, e.g. when they are afraid or sad or relaxed, etc. Similarly, time may speed up when we are happy or focused. To disambiguate the different states during which time slows down or speeds up, we decided to use colors to guide the users. For example, when a user is sad, the extra hand would slow down, but the clock would be purple colored. Whereas when the user is afraid, the extra hand would still slow down, but the clock would be blue colored.

The initial concept design accounted for the aforementioned guidelines. The EMIT timepiece would consist of a set of mechanical hands that present objective time in the innermost layer. When the clock is not enabled by a user in the near vicinity, the time perception and emotional cues are inactive. When a user desires to broadcast his/her emotional and time perceptual markers, he/she can use a smartphone to connect to the device and initiate a session. It is important to note that emotional and time perceptual displays are independent. Since the clock is designed to be a piece that might be on public display, the user can select which components he/she is comfortable sharing with the world.

Color Theory and Emotions

People’s emotional and cognitive states are intertwined with their time perception. For our clock, we decided to recognize five of Ekman’s six basic emotions (Ekman 2016) – Fear, Anger, Sadness, Joy, and Disgust. Ekman’s six emotions are commonly understood in the literature for affective computing, but the five emotions we chose out of the six basic emotions were depicted in the Disney movie Inside Out and are more likely to be understood by people outside academia. Our color theme is based on the colors associated with these emotions in Ekman’s Atlas of Emotions website (http://atlasofemotions.org/) as well as the colors used in Inside Out.

Inside Out Emotion Representations

Ekman’s Color Atlas

Strobe Lights

An important design decision that impacted the implementation of EMIT was whether to use mechanical or strobe light control for the time perception clock hand. Although mechanical control of a servo or stepper motor would allow use to implement a traditional clock hand effect of which we could adjust speed and direction, we include some advantages of choosing strobe lights over a more traditional, mechanical approach:

1. Ability to Disappear: A mechanical hand would always be there. Even if the hand was disabled, it would be constantly visible to the user. In our design, we strived to create a device for which the user could opt in or out for different markers. For instance, a user might not want to display time perception at some point during their day, in which case a mechanical display of time perception is unnecessary. With the strobe lights, the mechanical time perception hand is spinning so fast that it is practically invisible to the naked eye. Whenever the user wants to opt out of the time perception display, the strobe lights can be turned off, making the time perception hand invisible.
2. Multiple Hand Illusion: A mechanical hand would only allow use to display increases in speed and direction through the usage of the EMIT system. In the case of a strobe display, different frequencies can be used to achieve effects that would normally be impossible for a traditional mechanical hand. For instance, strobing at multiples of the rotating frequency of the motor can make it seems like there are multiple hands. Moreover, other effects can be implemented to create more complex interactions.
3. Multiple User Support: A mechanical hand would only support the display for the perception of a single user. In order to show the perception of more users, multiple hands would need to be included in the system to show differing perceptions of time.  Different sets of strobe lights can be programmed to strobe at different frequencies to create the illusion of multiple hands moving at different speeds, which would allow multiple users to concurrently interact with the system.
4. Subliminal Interface: Studies (Chebat, et al. 1993) have shown that visual cues can have an impact on the perception of time. Hanus, Kang, Ricker (N.d.), showed that using strobe lighting at different frequencies can slow down our speed up the time perception of a person. Although our project seeks to visualize and allow for a reflective experience, in the future we would like our platform to help align a user’s perception of time with their ideal mind time. For this, strobe light controls might prove a suitable alternative to subliminally alter a user who is looking at the clock.

 

CAD model

The Emit timepiece was modeled in the Fusion 360 CAD software. The model is available online in the following link:  http://a360.co/2paAYyA

Components

Based on our design concept, there are three main electronic components:

1. Rotation motor to spin the extra hand
   • This component will consist of a simple switch that will turn on and off the motor.  In a future iteration, the switch will be controlled by a microcontroller that detects the presence in order to conserve energy.
2. Strobe lights to create the strobe effect
   • This component consists of a simple transistor which controls whether a LED strip is powered on or off.    In future iterations this component will be improved to give more power to the LED strip.
3. Status lights to indicate change in user’s emotional state
   • This component consists of a Neopixel ring driven by an Arduino.

Implementation

Failure-First Prototype (V1)

The biggest concern we had about the clock was about the strobe effect. This was because we had never used strobe lights before and we were not sure how to create the effect.  We figured out that there were two big considerations in strobe effect:

1. Rotational motor
2. Consistent strobe light frequency and brightness

Initially, we used servo motors as a rotational motor because they allow control over the motors rotations. But servo motors have low rotations per minute (RPM), and even at the max speed, the rotating hand was visible to the naked eye and slow.  For the strobe effect to be useful, we wanted a very fast rotating hand so we switched to DC motors that have ~8000 RPM. The constraint with DC motor is that DC motors do not allow fine control over the motor speed.

In order to test the strobe effect, we used an iPhone app that created the strobe effect with the device’s  flash. This allowed us to test the motor and strobe light effect without having to make our own strobe effect and thus, made sure that our tests were independent of any errors in our strobe lights.

We ended up putting a motor with a Lego brick to simulate the rotating hand in a holder.  We placed this system inside a cardboard box.  We painted the brick with glow in the dark paint and used an Ultra Violet (UV) LED strip with strobing to make the hand appear.  We used UV light because we wanted the hand to only be visible/prominent when the strobing was on. With the UV-glow paint on the hand, the hand was only visible when the UV LEDs were strobing. However, the LEDs were not bright enough while strobing and we decided that our next step was to figure out a brighter light source to create the strobe effect.

Lessons learned from the first motor and iPhone strobe light test:

1. The motor and the hand needed to be relatively stable. Extra vibrations in either the motor or the hand prevented us from creating a smooth slow motion strobe effect.
2. The rotating hand had to be balanced about the center. If most of the weight was just on one side, the rotational frequency varied slightly, but that was enough to break the smooth frozen effect of the strobe light. To freeze motion using strobe light, we need the strobe frequency to be an integer multiple of the rotational frequency. If the hand was not balanced, the rotational frequency varied and thus, it was difficult to freeze motion using the strobe light.

Component-Complete Prototype (V2)

Strobe lights

We carried forward two issues/lessons from our failure first prototype:

1. Pick brighter strobe lights
2. Stabilize the motor and rotating hand

Our initial choice for a strobe light was this: https://www.amazon.com/gp/product/B013OIT17W/ref=oh_aui_detailpage_o01_s00?ie=UTF8&psc=1.

This was a very bright light (850 lumens) and we were able to control the frequency to strobe the light, and when we were testing it in front of the hand, it worked really well, i.e. it was bright enough to illuminate the hand as it was strobing. However, when we placed it in front of the hand inside the clock, it needed to be at least 7 inches away from the hand to illuminate completely the portion of the hand that was sticking out behind the light and henceforth be visible to the user.  Also, due to the internal LED driver of the light, we couldn’t strobe it accurately with a frequency larger than 30 Hz and that was a limiting factor for our rotational motor. Therefore, we decided to abandon this light because of its form factor, limited illumination, and small strobing frequency range.
Our next choice was these LED strips: https://www.amazon.com/gp/product/B00HSF65MC/ref=oh_aui_detailpage_o00_s00?ie=UTF8&psc=1. These LEDs turned out to be very bright and the flexibility of the strip allowed us to wrap the lights around the edges of the clock (facing the LEDs inwards so that they were focused on the rotating hand). We were able to strobe the lights at a very wide range of frequencies.

 

Status lights

We used Neopixel lights to show the emotion colors. We used a translucent white sticker to diffuse the light from the Neopixels.

Clock

We used real clock hands: https://www.amazon.com/gp/product/B01LXWJFPW/ref=oh_aui_detailpage_o01_s00?ie=UTF8&psc=1.

We stuck the clock mechanism behind the translucent screen so that only the clock hands were visible and the clock system itself was hidden.

Redbear Blend

Redbear Blend is an Arduino-based board with BLE communication. We use this board to control the status and strobe lights using commands from the Android phone.

Circuits

Pretty Final Prototype (V3)

We had two main goals for this prototype:

1. Replace the cardboard with wood
2. Make the inside circular rim of the clock slanted so that the strobe LEDs were more directed towards the extra hand. The sketch shows the sides are slanted instead of at an angle.

For this prototype, we incorporated the lighting, the clock, the strobing and a wood body.  We also incorporated the strobe control using a microcontroller with BLE.

Woodworking

We wanted to construct a solid base for the clock and opted to make it out of wood.  We designed three inner rings to be able to separate the large wooden base into makeable cuts.  We also opted to make the rings with a slanted inner wall to be able to shine the strobe lights into the clock.  After various troubled attempts, we were finally able to get a successful cut.  We then proceeded to sand these cuts and paste them together with wood glue.   

 

PHYSIOLOGICAL SENSORS

EMPATICA

Empatica provides heart rate, body temperature and galvanic skin response (GSR). We retrieved data from Empatica in an Android app using Bluetooth Low Energy (BLE). We use GSR to try to detect the arousal of a person, as changes in arousal are related to the perception of time.

MUSE

Muse provides electroencephalogram (EEG) data, i.e. alpha, beta, gamma, etc brain wave signals. We receive EEG data from Muse in our mobile app.  Based on this data, we then calculate the cognitive states of the user such as mellow, concentrated, etc.

AFFECTIVA

Affectiva API recognized basic emotions, such as happiness, fear, anger, sadness, etc..  We integrated the Affectiva API in our Android app and used the phone camera to take a picture of the user and pass it to the Affectiva API to recognize basic emotions

The complete system workflow diagram is as follows:

  

Usage Scenario

The EMIT system was designed with four use case scenarios in mind: time perception awareness, social indicator, reflection, and time bending. In this section, we describe how a user would make use of the prototype to achieve each of these goals, which are not necessarily independent of each other.

• Time Perception Awareness: The most basic usage scenario for EMIT is that of a tool to visualize your perception of time as a reflection of your inner emotional and cognitive states. In this case, the clock acts as a passive timepiece which can be positioned in a visible location where the user spends time. The EMIT clock system acts as an aesthetically pleasing piece of art and functions like any other normal clock. If and when the user wants it to, the user can connect his phone to the clock, which then processes all available biological data to add additional dimensions: time perception and emotion display.
  

• Public Indicator: Linked to the previous time perception awareness scenario, the EMIT also displays your cognitive and emotional states to those around you. As a result, the device acts as a public indicator, which allows others around you to make informed decisions about whether or not to approach you. For instance, the device can be placed in a visible spot in your office, where others who pass by can see whether or not you are busy or idle with a simple glance.

• Reflection: EMIT can also be used actively, as opposed to a passive timepiece that merely displays your perception of time. The EMIT application visualizes all incoming physiological data before aggregation. As a result, any user is able to sit in front of the clock and reflect on their inner state, both cognitive and emotional. Because the clock aggregates this information and visualizes it in an easily understandable format, the system affords dedicated sessions of reflection and meditation throughout the day.

• Time Bending: In this work, we refer to time bending as the ability to control, through practice, your cognitive and emotional state in order to change your perception of time. Moving a step beyond reflection, a user may train to modulate their biosignals using the EMIT system to slow down or speed up their perception of time. The biofeedback loop created by the system affords this kind of interaction which could come in handy in many cases. For instance, a user enduring a particularly boring presentation may use this honed skill to speed up his perception of time and feel better. Similarly, feeling like a week of vacation lasted really long can be a healthy experience for a weary mind.

Conclusion

In this work, we presented EMIT, a smart timepiece that portrays both objective and perceived time passing for its user. The implemented prototype used the MUSE sensing headband, the Empatica E4, and the Affectiva API to effectively become aware of its user’s inner cognitive and emotional state. This data is aggregated and portrayed in two ways. A Neopixel LED display portrays five emotional states(anger, fear, joy, sadness, and disgust) by changing color. Strobe Light frequency is subsequently adjusted to show a rotating outer hand moving at different speeds to represent the user’s perception of time. We discussed how such a system can be used as a tool for visualization, reflection, and training, as it allows a user to become aware of his current state, reflect upon it, and even train himself/herself to modulate his biosignals to achieve a conscious “bending” of time perception itself.

Future Work

We propose three different venues for future improvements for our EMIT timepiece. First, a more robust metric should be devised in order to aggregate the different biological data points. In order to achieve a more precise metric, it would be ideal to carry out data collection from MUSE and Empatica E4 and correlate it to time reproduction exercise. This could then be fed to a neural network that would learn the intricacies and patterns (if any) of how these biosignals alter time perception for a user, as opposed to naively aggregating these signals. A second, perhaps more interesting direction for this work, is to delve into the use case scenario of time bending. How can the clock enable users to train and change their perception of time? One possibility is to enable mechanisms to engage the user in active meditation over periods of time, vibrating or beeping to remind its user to engage in a training session. The second possibility is to capitalize on strobe light patterns and see if different frequencies can stimulate the brain and induce time distortions subliminally. Finally, we highlight that the EMIT timepiece currently works when the user is near the system. As a result, a lot of emotional and cognitive data that can be useful is lost when the user is on the move. The narrative clip, a device that takes pictures throughout the day at a specified interval to keep track of your life, can be used to further enhance the timepiece in a way that reflects more about your ongoing day.

Citations/Bibliography

Chebat, Jean-Charles, Claire Gelinas-Chebat, and Pierre Filiatrault. “Interactive effects of musical and visual cues on time perception: An application to waiting lines in banks.” Perceptual and Motor skills 77.3 (1993): 995-1020.

Hanus, Deborah, Christina Kang, and Elizabeth Ricker. “Time Perception: Audio Perception with Visual Cues.”

Lee, Kyudong, et al. “Ambient Lamp Display in the Active Home Ubiquitous Computing Environment for Relaxing and Mediation.” Future Generation Communication and Networking (FGCN 2007). Vol. 2. IEEE, 2007.

Brewer, Johanna, Amanda Williams, and Paul Dourish. “Nimio: an ambient awareness device.” Proc. of ECSCW 2005. 2005.

Rao SM, Mayer AR, Harrington DL (March 2001). “The evolution of brain activation during temporal processing”. Nature Neuroscience. 4 (3): 317–23. doi:10.1038/85191. PMID 11224550. Lay summary – Nature Neuroscience.

Kramer, Robin SS, Ulrich W. Weger, and Dinkar Sharma. “The effect of mindfulness meditation on time perception.” Consciousness and cognition 22.3 (2013): 846-852.

Wittmann, Marc, and Stefan Schmidt. “Mindfulness meditation and the experience of time.” Meditation–Neuroscientific Approaches and Philosophical Implications. Springer International Publishing, 2014. 199-209.

Bar-Haim, Yair, et al. “When time slows down: The influence of threat on time perception in anxiety.” Cognition and Emotion 24.2 (2010): 255-263.

Fraisse, Paul. “Perception and estimation of time.” Annual review of psychology 35.1 (1984): 1-37.

Joubert, C. E. (1984). Structured time and subjective acceleration of time. Perceptual and Motor Skills, 59, 335-336.

Zakay, D. (1990). The evasive art of subjective time measurement: Some methodological dilemmas. In R. A. Block (Ed.), Cognitive models of psychological time, (pp. 59-84). Hillsdale, NJ: Erlbaum.

Gibbon, John. “The structure of subjective time: How time flies.” Psychology of Learning and Motivation 20 (1986): 105-135.

Droit-Volet, Sylvie, and Warren H. Meck. “How emotions colour our perception of time.” Trends in cognitive sciences 11.12 (2007): 504-513.

Gil S, Droit-Volet S (February 2009). “Time perception, depression and sadness” (PDF). Behavioural Processes. 80 (2): 169–76. doi:10.1016/j.beproc.2008.11.012. PMID 19073237.

Droit-Volet, Sylvie, and Sandrine Gil. “The emotional body and time perception.” Cognition and Emotion 30.4 (2016): 687-699.

Edmonds, E. M., Cahoon, D., & Bridges, B. (1981). The estimation of time as a function of positive, neutral, or negative expectancies. Bulletin of the Psychonomic Society, 17, 259-260.

Stetson C, Fiesta MP, Eagleman DM (December 2007). “Does time really slow down during a frightening event?”.

Buetti, Simona, and Alejandro Lleras. “Perceiving control over aversive and fearful events can alter how we experience those events: an investigation of time perception in spider-fearful individuals.” (2012).

Twenge, Jean M., Kathleen R. Catanese, and Roy F. Baumeister. “Social exclusion and the deconstructed state: time perception, meaninglessness, lethargy, lack of emotion, and self-awareness.” Journal of personality and social psychology 85.3 (2003): 409.

Brown, Scott W. “Time perception and attention: The effects of prospective versus retrospective paradigms and task demands on perceived duration.” Perception & Psychophysics 38.2 (1985): 115-124.

Wittmann, Marc, and Martin P. Paulus. “Decision making, impulsivity and time perception.” Trends in cognitive sciences 12.1 (2008): 7-12.

Levy F, Swanson JM (August 2001). “Timing, space and ADHD: the dopamine theory revisited”. The Australian and New Zealand Journal of Psychiatry. 35 (4): 504–11. doi:10.1046/j.1440-1614.2001.00923.x. PMID 11531733.

Jafarinaimi, Nassim, et al. “Breakaway: an ambient display designed to change human behavior.” CHI’05 extended abstracts on Human factors in computing systems. ACM, 2005

Kuribayashi, Satoshi, and Akira Wakita. “PlantDisplay: turning houseplants into ambient display.” Proceedings of the 2006 ACM SIGCHI international conference on Advances in computer entertainment technology. ACM, 2006.

Van Wassenhove, Virginie, et al. “Distortions of subjective time perception within and across senses.” PloS one 3.1 (2008): e1437.

Berlin, Heather A., and Edmund T. Rolls. “Time perception, impulsivity, emotionality, and personality in self-harming borderline personality disorder patients.” Journal of personality disorders 18.4 (2004): 358-378.

Curton, Eric D., and Daniel S. Lordahl. “Effects of attentional focus and arousal on time estimation.” Journal of experimental psychology 103.5 (1974): 861.

Burnside, W. (1971). Judgment of short time intervals while performing mathematical tasks,Perception & Psychophysics,9, 404–406.

Franck N, Posada A, Pichon S, Haggard P (May 2005). “Altered subjective time of events in schizophrenia”. The Journal of Nervous and Mental Disease. 193 (5): 350–3. doi:10.1097/01.nmd.0000161699.76032.09. PMID 15870620.

Tse, Peter Ulric, et al. “Attention and the subjective expansion of time.” Attention, Perception, & Psychophysics 66.7 (2004): 1171-1189

Ekman, Paul “What Scientists Who Study Emotion Agree About“, Perspectives on Psychological Science (2016): 31-34

James, W. (1893). The principles of psychology. New York: H. Holt and Company. Page 609

Tulving, Endel. “Chronesthesia: Conscious awareness of subjective time.” (2002).

Sleepy Symbiosis

Adam Haar Horowitz, Ishaan Grover, Sophia Yang

Idea and Motivation

Edgar Allen Poe, August Kekulé, Henri Poincare, Thomas Edison, Salvador Dali: each of these thinkers regularly napped with a heavy steel ball in hand, jolted awake suddenly as they lost muscle control in Stage 2 sleep and dropped it onto the floor below. Why? To wake up and unearth creative inspiration found only in fragmented threshold consciousness, in the state of mind between drowsy Stage 1 Sleep and unconscious Stage 2. This second dream state, often overlooked, inspired everything from Surrealist painting to the discovery of Benzene’s molecular structure. Recent neuroscience has tied this semi-conscious state, called Hypnagogia, to creativity, learning, lessening of ego boundaries, fluid association of ideas, and shown it can be actively incepted with stimuli presented preceding sleep. Yet normally our hypnagogic images and ideas are entirely forgotten in the morning, when we remember snippets of different deep-sleep REM state dreams or remember nothing at all. And if we each go buy a steel ball, a simple sleep-intervention technology, we have only one option of wakeup depth, no method for inception, insight or data capture, and a system that wakes us up fully rather than maintain half-awake threshold states. We saw an opportunity with the spread of cheap consumer EEG to make a more flexible, inception programmable, data-driven system for users to direct and capture hypnagogic creativity at home. We worked with Brain Computer Interface EEG technology to create a neurofeedback-based system tracking sleep spindles—a biomarker of Stage 2 sleep—to provide real-time feedback on sleep state to users and both wake them up and record insights at adjustable thresholds of consciousness. Can integration with machines allow humans to access parts of our minds currently invisible to us? As consciousness begins to dissolve, our system kicks in to find out.

Salvador Dali’s The Persistence of Memory (1931)

 Neuroscience Background:

Creativity is an altered state of consciousness: in a moment of invention, “the creator breaks free of logic and deductive reasoning, of familiar pathways, of taken-for-granted approaches” (Khatami, 1978). The practiced pathways for cognition that structure our understanding of the world are abandoned and new, fluid associations arise. The theoretical framework for the functional neuroanatomy of altered states of consciousness generally, and creativity specifically, has often centered around deactivations in the prefrontal cortex (Dietrich, 2003; Dietrich, 2006). This center for executive, organizational cognitive function shuts down and “decrease in prefrontal activity creates less censorship from the mind, and more freely allows novel sounds and imagery to be imagined by the individual. This is flexibility,” says Dierdre Barrett of Harvard Medical School (Barrett, 1993; Barrett, 2001). These unpredictable bursts of novel insight and associations can be understood as a fragmentation of normal function, a passing breakdown of structured consciousness (Noreika, 2015). The question is, how can we cause these bursts and capture them to augment our creativity?

The same hypofrontality (frontal deactivation) underlying this flexibility is also common to the early stages of sleep that this project focuses on (Muzur, 2002). Humans go through Stage 1, 2, 3, 4, and Stage 5 (Rapid Eye Movement) sleep throughout the course of a night. Regularly remembered dreams are from REM state, but we have a second, more subtle dream-state earlier on: In the transition from wakefulness to sleep, the prefrontal cortex shuts down and cognitive alterations lead to the fragmentation of consciousness experienced as novel visual, auditory and bodily hallucinations or dreams, collectively called hypnagogia (Goupil and Bekinschtein, 2011). Hypnagogic hallucinations, linked to a decrease of theta, alpha, and beta power as measured by EEG preceding their report, can also include awareness of sleep onset (Kaplan et al., 2007), distorted perception of space (Bareham et al., 2014), and time (Minkwitz et al., 2012), as well as language alterations (Noreika, 2015). Hypofrontality thus provides an opportunity for creative idea generation in hypnagogia, if novel associations can be captured.

Image from Tagliazucchi (2013).

It comes as no surprise, then, that experiments have found evidence for a correlation between hypnagogia and enhancement of creative ability (Green, 1972; Green, et al., 1970, Green, Green, & Walters 1974; Parks, 1996; Stembridge, 1972; Whisenant & Murphy, 1977; Noreika, 2015). Hypnagogia includes illogical and fluid association of ideas, loosening of Ego Boundaries and anxiety reduction (Mavromatis 1983). Recent work has found that hypnagogia is, further, ‘inceptable’, reliably involving hypnagogic imagery related to repeated tasks and imagery in preceding awake stages. (Kussé, 2011; Stickgold, 2000) And the likelihood of thematic hypnagogia images is tied to potential learning range on a topic—suggesting hypnagogia is important for learning (Stickgold, 2000). Lastly, recent research has shown that speaking does not wake subjects out of drowsy hypnagogia, and accordingly they are able to record insights via audio (Noreika, 2015). So hypnagogia appears useful for creativity, controllable through inception, important for learning, and capturable through audio recording…this is getting interesting. How do we track it and find the right moments to intervene or record, when users are truly half asleep at threshold consciousness?

Sleep Spindles:

If we can successfully track the beginning of Stage 2, our system can wake subjects back into Stage 1 sleep using their previously recorded inception audio stimuli, containing whichever theme they want Stage 1 ideation to focus on. Sleep spindles are a reliable biomarker of the onset of Sleep Stage 2, and accordingly they give us an opportunity to track and target subject’s transition out of consciousness (Gennaro, 2002). In Sleep Stage 2, transient bursts of oscillatory activity begin in a specific range of brain frequencies, 9-15.5 hz. These rhythmic discharges are distributed across the cortex, and their sudden increase in amplitude makes them detectable by human eye or automated algorithm (Nicolas, 2001). These rhythmic bursts of synchronous excitatory post-synaptic potential reach the neocortex and are registered at the scalp on Electroencephalogram (EEG) as sleep spindles. As we are using an EEG system with electrodes concentrated over frontal lobes (the Muse EEG), we will focus on detecting the range of sleep spindles most common in frontal cortical areas, namely slow 9-13hz sleep spindles, specifically finding clearest classification in the 9-11hz range (Ujma, 2015). Since spindles orchestrate rhythmic synchrony across diffuse brain areas—synchrony being key to conscious perception binding color, space, smell and more into cohesive perceived meaningful stimuli—neuroscientists have gone so far as to posit a central role for sleep spindles in the generation and degeneration of consciousness; the thalamic dynamic core theory of consciousness (Ward, 2011). As our project focuses on explorations of threshold consciousness, the role of sleep spindles in degeneration of consciousness is key and exciting. Spindles as a tracked biomarker open up many doors for further exploration of consciousness and perception.

Opportunities to Contribute to the Field:

As far as we can tell, no system has been invented that offers neurofeedback on sleep stages. No system has been created that offers the option of inception in situ (during verified hypnagogia). No hypnagogic capture system has been invented that offers granular, data-driven, adjustable wakeup times with automatic audio capture of insights, allowing for exploration of threshold consciousness past loss of muscle control. And research into hypnagogia has been done, thus far, on astronomically expensive 32 channel EEG, effectively excluding the Maker, Hacker, and Hobbyist science and tech communities. Our system aims to fill each of these gaps using a $249.00 Muse EEG, with a fully available Developer Kit, Research Tools and Documentation free online. What follows is how we went about doing that: 

Digital Signal Processing (DSP):

Our system relies on accurate detection of sleep spindles. For the sleep spindle detection algorithm, we characterized a sleep spindle as activity in 9-11hz range that lasts for a duration of at least one second. Since we only care about detection of sleep stage 2, our loss function puts a huge penalty on false positives and a small penalty for false negatives. In other words, it is okay for the algorithm to miss sleep spindles but not okay to classify a sleep spindle and sound an alarm without the presence of one. We developed two algorithms to detect sleep stage 2, each with its own strengths and weaknesses. For the purpose of this text, we call them algorithm1 and algorithm2. Algorithm1 (Wallant, 2016) provides more predictions, has a lower precision (percentage of spindles correctly predicted) and has lesser hyperparameters. Algorithm2 provides lesser predictions, has a higher precision and more hyperparameters (Devuyst, 2011).  Since we receive data from the TP10 channel of the MUSE headband in real time as opposed to a 32 channel EEG device, our data is more noisy and requires more hyperparameters compared to existing algorithms. The DREAMS database focuses on central, fast spindles in the 12-14hz range. Because our Muse EEG has limited electrodes, we focused on frontal spindles in the 9-11hz range instead. We go into the details and preliminary results of each algorithm in the next subsection.

Algorithm 1

1)For every n seconds of data, pass the wave through a bandpass filter with 9hz as the minimum cutoff frequency and 11hz as the maximum cutoff frequency.

2)In a rolling window of 1.5 seconds, compute the rms value.

3)If rms value > threshold, return spindle detected.

Testing Algorithm1

On the dreams database, we tested the algorithm against an expert’s spindle notation with 12-14 hz as the cutoff frequency, as this series of expert notations focus on central fast spindles. Within an interval of +- 3 seconds, we got only 1 false positive for the excerpt of 108 spindles.

We further tested the algorithm on a human subject with a Muse EEG headband after changing the cutoff frequencies to 9-11 hz. The algorithm however, did not perform well owing to the noisy data from the headband. This suggested the use of another algorithm with more hyperparameters that can be tuned to each subject.

Algorithm 2

1)For every n seconds of data, pass the wave through a bandpass filter with 9hz as the minimum cutoff frequency and 11 hz as the maximum cutoff frequency. Additionally, pass the original wave through a bandpass filter with a minimum frequency of 0.5hz and maximum cutoff frequency of 40 hz.

2)Compute the rms value for each of the waves after passing the original wave through the two band-pass filters.

3)In a sliding window of 1 second, take the ratio of the rms values.

4)If ratio > threshold, add the timestamp to a list of possible spindle candidates

5)For every candidate timestamp, if there are < y candidate timestamps immediately following the candidate timestamp, the candidate timestamp is a true spindle, otherwise it is not.

The hyperparameters y and threshold are tuned for each person. After tuning the hyperparameters, the algorithm gave no false positive positives pre atonia and returned spindles post atonia. However, validation by a sleep neuroscience expert of each of our detected spindles is required to fully prove the performance of the algorithm.

Each detected sleep spindle will activate an alarm based on a previously recorded audio stimuli, to time inception to the moment of reentry into Stage 1 sleep from Stage 2.

But What If I’m Not Sleepy? Photic Entrainment and Muscle Control:

After designing our system for successful signal processing of sleep spindles, we realized that while the majority of users will make use of our system in the drowsy moments after waking up or in tired moments late at night, we should account for cases of non-drowsy users who nonetheless want to nap for generation of creative content in hypnagogia. To facilitate entry into drowsy sleep stage 1, we made use of photic and audio brainwave entrainment.

Brainwave entrainment, though not widely known, has been in use since the 1800’s. The protocol involves non-invasive stimulation of internal brain oscillations based on presentation of external oscillations users aim to engender in the brain, used both in humans and animals for research purposes (Boyden, Tsai, 2015). In short, a frequency outside the brain is meant to cause a frequency inside the brain. This effect relies on rhythmic neural responses to rhythmic stimuli, a Steady-State-Response (SSR), an effect that makes sense considering the brain responds to external stimuli with internal neural activity, and activity must be time-locked to stimuli for effective perceptual binding. Multiple studies have used rhythmic visual stimulation to drive neural oscillations in humans with a range of frequencies (Keitel, 2014). Many others have shown the presence of an auditory steady state response in humans, offering a method for exploring effective audio entrainment thought the be engendered by a Brainstem based response to environmental audio (Meltzer, 2015).

Evidence for entrainment still remains heterogeneous, with a mix of hobbyists and researchers exploring it, and difficulty separating informal and formal research. Our aim was to increase theta amplitudes (6-10hz) to ease subjects into Stage 1 Sleep, as significant increases in theta frequencies are tied to Stage 1 Sleep (Schtuze, 2015). We designed stimuli lasting 5 minutes, consisting of rhythmic alternating Black and White screen flashes and isochronic tones (audible sound emitted at regular intervals) each at 8hz. We tested this system on ourselves, presenting stimulus with eyes closed, and pulling EEG reading with Muse before and after stimulus, again with eyes closed. We saw a significant increase in theta activity (0.1169 Bels in channel AF7) as compared to time preceding stimuli (0.0341 Bels in channel AF7), and noted increases in drowsiness, thus deciding to use it as our stimulus for non-drowsy participants. The literature shows higher frequency ranges have been more thoroughly explored for entrainment than the theta we used, and shows further inquiry is needed into the effect of isochronic tones: future work will require us to test audio and visual entrainment at 8 hz separately with a far higher n of subjects. Please note: if you are going to use it below, please be aware of the risk of photosensitive epilepsy, which are seizures that occur at a rate of approximately 1.1/100,000 people when presented with flashing light stimuli (Quirk, 1995).

As often happens, in creating an effective solution to one challenge, we created another challenge for ourselves. Our entrainment stimulation created an effective significant increase in theta waves, but also created a significant increase in alpha waves. Our tuned digital signal processing for sleep spindles no longer worked, and failed even 10 minutes after stimuli, making it ineffective as a Sleep Stage tracking tool! We found no research on DSP post theta entrainment, and decided to deal with the challenge of processing in entrained states in future research, discussed below. In the meantime, for non-drowsy, entrained participants where sleep spindle detection is not possible, we created a more coarse system that sends an alarm, incepts and automatically records all based on loss of muscle control as measured by Force-Sensitive-Resistor.

Industrial Design of Mask for Photic Entrainment Delivery:

The material of the sleep mask core was chosen based on the level of comfort as well as its ability to hold phones with multiple sizes for delivering photic entrainment, alarms and audio recording. Main concerns were overall weight and cushioning ability between the human face and the rigid surface of phone. We decided on low density foam as the core material because it is light weight, it presents the right level of compression when force is applied, and it has enough structural integrity to hold the phone in place. We carved the structure of the foam to fit the human face and distribute the pressure between the face and phone. We lined the core with felt to make the surface of the mask soft to the touch. Finally, we used two elastic straps as the attachment mechanism to fit various head sizes.

 

Industrial Design of FSR Muscle Sensitive Glove:

We started the design process by looking at the most graceful and nondestructive gestures people might express when transitioning between wakefulness and sleep. Because people lose muscle control when entering stage 2 sleep, we designed a glove that sends a signal to the phone app when users are no longer able to hold a closed fist. We embedded a force sensitive resistor on the palm of the glove so users can comfortably touch the sensor during their wakeful state and send a signal to the upon sleep onset when force ceases.

 Interface Design: 

Current interface design is purely practical—buttons for recording inception, beginning photic entrainment, beginning spindle tracking, and playing past recordings. Future work will include an improved UI design

 User Experience:

Case Study:

After industrial design, we had time to test our system on one local painter, hoping to inspire surrealist hypnagogic artwork. As this was our first test, we waited for detection of multiple spindles in stage 2 to ensure avoiding false positives before Stage 2, and validated loss of muscle control in the hand (atonia 4 minutes after the subject lay down) before waking up the subject and recording insights.

Results were really exciting! We saw successfully incepted imagery (1); awareness of sleep onset (2); generation of useful, relevant ideas (3); distorted sense of space (4); illogic and fluid association of images and ideas (5). Reflections below:

“ I was conscious of the fact that I should observe my sleep, like aware of myself progressing into sleep (2). There was really fast images. One of them was a tiger since we talked about a tiger before (1). Some of them weren’t actual objects (5). The one I remember was a chessboard that had been compressed in the middle and stretched at the edges (4). Really quick images cycling through, and the warped, pulsating darkness pattern like when you close your eyes for a long time and you get that. And then I remember speaking to someone and the ideas were really relevant and I will use them in some essay and now I can’t remember any of it (3). It was so interesting. I remember lots of images. I really want to remember this stuff. I was convinced I was going to remember it all whilst I was sleeping because I was aware it would be useful in my life on some level, I thought I need to remember these images and this conversation they’re so interesting.“

It is interesting to note that visual imagery was more effectively remembered than new linguistic generation in hypnagogia. Future work will adjust the wakeup threshold past first detected sleep spindle to optimize for recall.

After the ending of the Case Study, the artist asked to be tested on again, remarking she found the process exciting from a personal exploration and idea generation standpoint. She later sent the image below as a representation of the distorted hypnagogic chessboard image described above.

Related Work:

 Neuro:

Stickgold (2000) work on Tetris Effect, where imagery before sleep enters dreams.

Gallant (2011) reconstructing visual imagery from brain activity, opening up the possibility of externalizing and recording internal imagery.

Horikawa (2013) present machine-learning models predicting the contents of visual imagery during the sleep-onset period with 60% accuracy, given fMRI activity and self-report.

Noreika (2015) work on using speaker self report and motor movement to capture hypnagogic hallucinations, like “putting a horse into a sort of violin case” (listed below).

Tech:

Relaxation Inducing Sleep Mask (US Patent US 8852073 B2) for a light-tight sleep mask presenting audible and/or LED photoluminescent visual stimuli to promote relaxation.

Chrona: Smart pillow that tracks sleep state based on motor movement and sleep by utilizing acoustic entrainment Low-frequency sounds boost deep sleep and high-frequency sounds ease you into your day. 

Dreem – EEG headband that tracks entry into REM and uses audio entrainment to enhance the quality of deep sleep.

Future Work:

Future work should (1) attempt to solve the issue of DSP post-entrainment (2) create an algorithm which automatically tunes parameters so adjustment to specific subjects is not necessary (3) test recall improvement as spindle detection alarm thresholds change (4) continue this process with a greater n of subjects and artists who work in non-visual fields, and investigate effects of hypnagogic training over time (5) test efficacy of multiple methods of inception into hypnagogia, visual or audio, and efficacy of different types of audio alarms, namely bone conduction vs air conduction sound (6) create an AI that converses with sleepers in hypnagogia and records, vs simple alarms and recording.

 

Video Link: http://sophiayang.net/video

 

References:

DREAMS Sleep Spindle Dataset: http://www.tcts.fpms.ac.be/~devuyst/Databases/DatabaseSpindle/

Bareham, C. A., Manly, T., Pustovaya, O. V., Scott, S. K., and Bekinschtein, T. A. (2014). Losing the left side of the world: rightward shift in human spatial attention with sleep onset. Sci. Rep. 4,

Barrett, Deirdre (1993). The “committee of sleep”: A study of dream incubation for problem solving. Dreaming, Vol 3(2), Jun 1993, 115-122

Boyden, Ed. Tsai, Li-Huei (2016). Nature. Gamma frequency entrainment attenuates amyloid load and modifies microglia. 540, 230–235

Devuyst, S. (2011). Automatic Sleep Spindles Detection. 33rd Annual International Conference of the IEEE

Dietrich, A. (2006). Transient hypofrontality as a mechanism for the psychological effects of exercise. Psychiatry Research, 145 (1) (2006), pp. 79–83

De Gennaro L, Ferrara M (2003) Sleep spindles: an overview. Sleep Med Rev 7:423–440.

Goupil, L., and Bekinschtein, T. A. (2011). Cognitive processing during the transition to sleep. Arch. Ital. Biol. 150, 140–154.

Huang, TL. A comprehensive review of the psychological effects of brainwave entrainment. [Altern Ther Health Med] 2008 Sep-Oct; Vol. 14 (5), pp. 38-50.

Jankel WR, Niedermeyer E. (1985). Sleep spindles. J Clin Neurophysiol 1985; 2:1–35.

Kaplan, K. A., Itoi, A., and Dement, W. C. (2007). Awareness of sleepiness and ability to predict sleep onset: can drivers avoid falling asleep at the wheel? Sleep Med. 9, 71–79.

Keitel, Christian, Cliodhna Quigley and Philipp Ruhnau (2014). Stimulus-Driven Brain Oscillations in the Alpha Range: Entrainment of Intrinsic Rhythms or Frequency-Following Response? Journal of Neuroscience 30 July 2014, 34 (31) .

Khatami, Manoochehr (1978). Creativity and Altered States of Consciousness. Psychiatric Annals. Volume 8: Issue 3 (57-64).

Meltzer, Benjamin (2015). The steady-state response of the cerebral cortex to the beat of music reflects both the comprehension of music and attention. Frontiers in Human Neuroscience (436).

Minkwitz, J., Trenner, M. U., Sander, C., Olbrich, S., Sheldrick, A. J., Hegerl, U.,et al. (2012). Time perception at different EEG-vigilance levels. Behav. Brain Funct. 8, 50. doi: 10.1186/1744-9081-8-50

Muzur, Amir, Edward F. Pace-Schott, J.Allan Hobson (2002). The prefrontal cortex in sleep

Trends in Cognitive Sciences, Volume 6, Issue 11, 1 November 2002, Pages 475-481

Nicolas, A., Petit, D., Rompré, S. & Montplaisir, J. (2001) Sleep spindle characteristics in healthy subjects of different age groups. Clin. Neurophysiol. 112, 521–527.

Nishimoto, Gallant (2011). Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies. Current Biology Volume 21, Issue 19, 11 October 2011, Pages 1641–1646

Ogilvie, R.D., R. T. Wilkinson. (1984). The detection of sleep onset: Behavioral and physiological convergence.Psychophysiology 21, 510.

Quirk, JA. (1995). Incidence of photosensitive epilepsy: a prospective national study. Electroencephalogr Clin Neurophysiol. 1995 Oct; 95(4):260-7.

Schimicek, P., Zeitlhofer, J., Anderer, P. & Saletu, B (1994). Automatic sleep-spindle detection procedure: aspects of reliability and validity. Clin. Electroencephalogr. 25, 26–29.

Schutze, MD. (2015). The difficulty of staying awake during alpha/theta neurofeedback training. Appl Psychophysiol Biofeedback. 2015 Jun;40(2):85-94.

Stickgold, A. Malia, D. Maguire, D. Roddenberry, M. O’Connor (2000)Replaying the game: Hypnagogic images in normals and amnesics. Science 290, 350.

Tagliazucchi, E. (2013). Sleep Neuroimaging and Models of Consciousness. Frontiers in Psychology. Review.

Ujma, Peter. (2015). A comparison of two sleep spindle detection methods. Front. Hum. Neurosci., 17 February 2015

Wallant, D. C. T., Maquet, P., & Phillips, C. (2016). Sleep spindles as an electrographic element: description and automatic detection methods. Neural Plasticity.

by Chrisoula Kapelonis, Lucas Cassiano, Anna Fuste, Laya Ansu, Nikhita Singh

SEA is a spatially aware wearable that uses temperature differentials to allow the user to feel the heat map of their interactions with others in space, real-time. By examining a user’s emotional responses to conversations with people and by providing feedback on a relationship using subtle changes in temperature, SEA allows for more aware interactions between people.

Emotional responses are analyzed and interpreted through the voice and the body. We used a peltier module as the base for the temperature change, and use BLE to get the location of users. An iPhone app was developed as well to allow users to get a more detailed view of how their relationships have changed throughout time (the app allows people to examine their emotional responses to people on a day-to-day level). Ultimately, SEA is a wearable that attempts to help people be more mindful of their own actions and responses in relationships and tries to foster and nurture healthier interactions.

______________________________________________________

Idea & Vision

Emotional awareness and relationship nurturing

Today, we communicate with others through interpersonal interactions and digital exchanges. And over time, these interactions develop into relationships that fall in the spectrum of positive, negative and everywhere in between. Relationships form after repeated interactions, and it is the reinforcement of qualities of these exchanges that determine the baseline state of that relationship; whether it is a positive, neutral, or negative one. Awareness of these nuances are important for the person to understand their dynamics with another person, and how they can and should further interact with them.  Knapp’s relationship development model [ is the reference point by which we have based our relationship metrics around. The model defines relationships in ten stages; five relating to relationship escalation, and the remaining five to relationship termination.

The Relationship Escalation Model consists of: Initiating, Experimenting, Intensifying, Integrating, and Bonding.

Initiating: This is the stage in which people first meet. It’s typically very short and a moment where the interactants use social conventions or greetings to elicit favorable first impressions. This is a neutral state, and one where there is no prior interaction to base a baseline off of.  

Experimenting: In this stage, the interactants try to engage in conversation by asking questions and talking to each other to understand if this is a relationship worth continuing and expanding. Often times, many relationships stay at this point if they are passive, or merely not worth continuing. This is also a neutral state, but it is a stage in which a relationship can either desire to go forward, stay neutral, or start to lean in the negative range.   

Intensifying: This is the stage where people start to get more comfortable with each other, and claims about the commitment of the relationship start to be conveyed. In this stage, formalities start to break down, and more informal interactions are favored. Here, people are starting to be perceived as individuals.

Integrating: At this point, the interactants become a pair. They begin to interact often, do things together, and share a relational identity. People at this point understand their friendship has reached a higher level of intimacy.

Bonding: This is the highest level of the coming together phase of a relationship. It is the moment where there is a formal declaration of the relationship such as “romantic partner,” “best friend,” or “business partner.” This is the apex of relationship intimacy so very few relationships in an individual’s life actually reach this level.

The Relationship Termination Model consists of: Differentiating, Circumscribing, Stagnating, Avoiding, and Terminating:

Differentiating: In this stage, individuals start to assert their independence. They develop different activities or hobbies and the two actively start to differentiate themselves instead of associate. In this stage it is not too difficult to turn the relationship around.

Circumscribing: Here, communication between the individuals starts to decay. There is an avoidance of certain topics of discussion, in order to prevent further aggression. Here the opportunities for revival are still plenty.  

Stagnating: This is the stage where the individuals involved avoid communication and especially topics revolving the relationship.

Avoiding: In this stage, the interactors physically avoid each other, and reduce the opportunities for communication.

Terminating: This is the final stage of the termination model. It is the moment when the individuals devide to end the relationship. It can happen either negatively or positively, depending on the context.

It has been shown that participation in positive social relationships is beneficial for health. Lack of social ties has been correlated with higher risk of death when compared to populations with many social relationships. Social relationships affect health in three major ways, behavioral, psychosocial, and psychological. Strong ties help influence healthy behaviors because they reinforce good habits, whereas negative ones reinforce bad habits, thus having a negative influence. In regards to psychosocial mechanisms, good relationships provide mental support, personal control and symbolic norms, all of which promote good mental health. Negative or toxic ties decay mental health because they degrade these pillars. And finally, physiologically, social processes have a direct relationships with physiological ones such as cardiovascular health, endocrine and immune functions. Thus, the positive or negative influence of our everyday interactions has a significant influence on our health and well-being.

Digital interactions

Through the adoption of social media, and digital communication devices, our relationships have evolved beyond merely face-to-face interactions and diversified to develop further without the person being present. This allows our digital interfaces to gather the data being fed about the specific dynamics of that relationship, and start to make inferences about its state on the spectrum. Our social platforms now have started ranking these relationships over time by bringing positive, more frequented ones to the forefront (such as “Favorites” in Facebook Messenger, or Instagram’s feed algorithm bringing your favorite accounts to the top), and showing ones that are less important to the user, less frequently, often never. This has led to users becoming passively aware of their relationship dynamics.

But for in-person interactions, we only have our perceived emotions as indicators of relationship state. Emotional response is subjective, and often very difficult to decode and interpret in the moment of interaction. Our feelings toward a relationship dictates how we act in the interaction, which greatly influences both how the other person reacts back to us and our perception of them. In real-time, because we are governed by our emotional filter on a situation, it is difficult to understand or control how we are perceiving that interaction. In the long-term, over repeated interactions this also holds true. Often, we feel that a relationship is developing in a positive or negative way, but have no concrete understanding of just how much, until we observe it in retrospect. There is no metric to understand these interaction baselines, deviations, or averages. We are exploring how to use wearables to bring a new sense of awareness to the user to have a more comprehensive understanding of their emotions.  

Wearables and self-awareness

Wearables can allow us to understand information about ourselves through the monitoring of the body and self in situ. They are used for a range of biometric tracking needs, and notification experiences. Often times though, wearables are meant to just have a singular, self-to-self relationship. They observe the body, or the communication to the body, and express information relating to that. But what if we could have social wearables that helped us be aware of our state with another person in face-to-face interactions?

Social wearables

“The next wave of wearables promises to usher in a Social Age, which is marked not necessarily by a movement away from information, but towards communication and self-expression.” -Amanda Cosco in wearable.com

This is where the intent of our project lies. The social age of wearables will allow the user to nurture relationships, grow empathy, and improve well-being and self-awareness. Our desire is to bridge the nuanced awareness of wearables, with the complications of face-to-face interactions, and design a wearable for social awareness. We hope to bridge the body and the mind, to link the internal self with the social self.

______________________________________________________

Related work: Interpreting the body and its signals

Our investigations in this project hinged deeply on the relationship between the body and the information it was telling us about the user’s emotional state. So we dove into understanding the current space of interpreting these signals, and understanding how emotions can be analyzed, understood, and conveyed on the body.

Bodily maps of emotions: relationship between temperature and emotions [9]

In this paper, researchers asked over 700 individuals from different countries where they felt different emotions on the body after watching clips, seeing images, and hearing snippets of audio, and mapped the answers to a human body silhouette. The researchers found that the bodily maps of these sensations were “topographically different” for different emotions such as anger, fear, happiness, disgust, sadness, and so on. Interestingly enough, the emotions were consistent across cultures. If we were to superimpose all of these sensations onto one body, we see that the chest is the location on the body where emotions are felt most. We kept this in mind when we were thinking about the form factor of our wearable.

Empathy amulet: connecting people through warmth [4]

The Empathy Amulet is a project designed by Sofia Brueckner at the Fluid Interfaces group at the MIT Media lab. It is a networked wearable device that allows anonymous users to share a moment of warmth with each other. Users hold the sides of the necklace and feel the warmth of someone else around the world who connected with them at the same time. This gives the wearer a momentary connection through temperature, joining them with another person in a moment of desired warmth or loneliness. The form factor of the project, the necklace, translated well as an interface for facilitating connection with warmth, and we were particularly inspired by this.  

Fan Jersey (X): amplifying experiences [11]

Wearable Experiments has made a smart jersey that allows sports fans to feel certain physical sensations related to what has happened in the game they are watching. Fans are able to feel the actions of another, the team they root for, translated into haptic vibrations. The experience of watching a game is amplified as all fans, as part of a shared experience, are able to feel these haptic vibrations when football plays such as the 4th Down, Field Goal, and touchdown happen. An interesting aspect of this shared experience is that a mutual emotion of excitement and happiness is translated into very simple haptic vibrations. The idea of simple output from a wearable translating a complex emotion like “excitement” is very interesting, and something we also explored with SEA.

Altruis X: stress and emotion tracking for well-being [12]

Altruis X is a wrist wearable that gives the user insights on stress triggers, sleep disruptions and meditative actions. It monitors the user over an extended period of time, and provides them with useful insights: for example, a notification that email usage after 7PM disrupts their sleep, or conversations with their dad have been lowering their stress levels. This project was particularly useful to us because of the emotional awareness it provided, and also because it allowed the user to only receive the notifications they wanted, and cut out the remaining noise.

OpenSMILE: valence and arousal prediction [15]

OpenSMILE is an open-source feature extraction tool that has the ability to analyze large audio files in real time. This tool allows for extraction of audio features such as signal energy, loudness, pitch and voice quality. This analysis gives us insight on the emotional state of the person speaking, which for SEA, is crucial in understanding how they are feeling about a particular interaction. In analyzing speech, we can link meaning and emotion.  

______________________________________________________

Related work: Affecting the body/mind through temperature

We perceive our environment through our senses and our vision and perception of the world depends on these channels that allow us to understand what is happening around us. The body receives external stimuli and can process different kinds of inputs: visual, tactile, olfactory, etc. Each of these inputs provides us with the information we need to understand and define our own reality. The smallest change of these perceptions can make us perceive the world slightly different and thus cause us to behave in different ways even without us being aware. Thalma Lobel has been exploring the effects of the environment through our senses and how can this transform our actions and interactions.

For this project, we explored different sensory modalities and decided to focus on temperature on the body as a means of enhancing and modulating emotions. Various research demonstrates how temperature has a high effect on emotions and our perception of the world. Temperature is a very strong stimuli that can let us enhance perception or use it as a medium for an interaction.

Cold and Lonely: Does Social Exclusion Literally Feel Cold? [2]

Zhong and Leonardelli demonstrate in this research paper how social exclusion literally feels cold. In their first experiment ‘participants who recalled a social exclusion experience gave lower estimates of room temperature than did participants who recalled an inclusion experience’. In their second experiment, subjects that were socially excluded in an online experience reported greater desire for warm food and drink than those who were included.

We can see with this work how there is a clear link between warmth and inclusion/friendliness and a link between cold and exclusion. In their own words:

‘These findings are consistent with the embodied view of cognition and support the notion that social perception involves physical and perceptual content. The psychological experience of coldness not only aids understanding of social interaction, but also is an integral part of the experience of social exclusion.’

Experiencing Physical Warmth Promotes Interpersonal Warmth.[5]

In these experiments, Williams et Al. explore the relationship between physical warmth and interpersonal warmth. They demonstrate how an experience of physical warmth has an effect on a perception of another person and the interaction that one can establish with others without the person’s awareness of such influence. They performed two studies to demonstrate such effects. In the first study, subjects that held a hot cup of coffee judge a target person as having a warmer personality (generous and caring). In the second study, participants that held a hot pad would choose a gift for another person instead of keeping it for themselves.

This research brings up the question: can adding a physical warmth factor to a relationship actually enhance friendliness? And does adding a cold factor to an interaction actually make it less friendly?

Hot Under the Collar: Mapping Thermal Feedback to Dimensional Models of Emotion [6]

Wilson et Al. explore in their research how can we map thermal feedback to dimensional models of emotions to be able to use it as a tool in human-computer interaction. Based on the association between temperature and emotion in language, cognition and subjective experience, they tried to map subjective ratings of a range of thermal stimuli to Russell’s circumplex model to understand the range of emotions that might be conveyed through thermal feedback. Based on this mapping, they questioned the effectiveness of thermal feedback to translate a range of emotions and posed the question: can thermal feedback convey the full range of emotions? They finally concluded that the mapping of thermal feedback better fits a vector model than Russell’s circumplex model.

AffectPhone: A Handset Device to Present User’s Emotional State with Warmth/Coolness [7]

Iwasaki et Al. present the AffectPhone which is a good example of how to convey emotional state with others on a remote interaction such as talking on the phone.

The AffectPhone uses a peltier module on the back of a mobile phone to present the user with an augmentation of their emotional state. The device tracks GSR (Galvanic Skin Response) from the user to process the level of arousal when having a conversation on the phone. The levels of arousal are sent to the mobile phone of the other person on the conversation and modulate the temperature on the peltier module. The system is designed to convey non-verbal information in an ambient manner. Bringing temperature to the forefront as awareness of emotional state can serve as a way of sensory augmentation and social interaction modulation.

Heat-Nav: Using Temperature Changes as Navigational Cues [8]

In Heat-Nav, Tewell et al. explore how temperature can be used as an interaction modality. They point out the fact that temperature changes are perceived over the course of seconds and that thermal cues are typically used to communicate single states, such as emotions.

They discuss how continuous thermal feedback may be used for real world navigational tasks and how to use temperature as a tool for spatial awareness.

Several research experiments have tried to integrate new sensory modalities in our perception system. However, as O. Deroy and M. Auvray mention in their article ‘Reading the world through the skin and ears: a new perspective on sensory substitution’, most of the systems developed have failed to live up to their goals as they assume that a sensory device can create a new sensory modality. In our case, we try to think about ways of enhancing current senses and augmenting their capabilities by expanding the input channel to the body and mind. Our aim for this project is not to create a new sensory system but rather to expand and enhance the interactions by using our existing sensory systems.

______________________________________________________

SEA – Spatial Emotional Awareness Wearable

SEA is a wearable for interpersonal emotional awareness and relationship nurturing. It analyzes the wearer’s voice during in-person interactions and provides them with thermal feedback based on the warmth or coldness of that interaction. It also establishes a baseline temperature for a specific relationship based on repeated interactions, so the user can feel their friendships while navigating through space.

The spatial experience

SEA’s capabilities allow the wearer to engage in two scales of awareness; the state of a relationship baseline, and real-time conversation. Each one of these scales plays a role in providing the user useful information about relationship dynamics. One is passive, and merely feeling the map of current states, and the other is active, listening and analyzing in real-time conversational data.

SEA affords the wearer the opportunity to experience a heatmap of their relationships while they are navigating through space. They can feel the warmth of a good relationship around them, and can also be aware of the presence of a negative influence in their space. In environments where relationships are undeveloped, and still in the early stages of their emergence, the user will feel no temperature difference from room temperature. The device will not provide additional heat or coldness. But in spaces where there are more fully formed relationships, the wearer will feel a more defined temperature gradient that deviates from the neutral room state. This lets the user become aware of the direction of the interactions they have in a particular space. SEA becomes a thermal navigator for physical space. 

In terms of instantaneous personal relationships, you can feel when a conversation is going really well, and developing past its previous state, and you can feel when a conversation is going back.

Temperature as translator

We were trying to understand if temperature could allow for both internal awareness (emotion) and external awareness (social and spatial). Instead of trying to load our wearable with all possible types of feedback (output), we attempted to create a wearable that was very subtle in the way it revealed information to a person. As discussed earlier in the related work section, the experience of temperature–heat and coldness– can impact how people behave in interactions with others. Something as subtle as holding a normal, iced-coffee as opposed to a warm cup of coffee can change how warmly someone behaves when talking with another person. We wondered if we could use temperature as a translator of positive feelings and emotions or negative feelings and emotions. We wanted to focus on subtle cues that could hint at something being different in a relationship, as opposed to outright letting someone know: “your relationship is suffering” or “your relationship is wonderful”. Relationships are complex: there will always be a mix of hot and cold feelings. Using a slight difference of temperature allows us to scale the perception of a relationship over time. This is while tuning the general feeling of the relationship throughout time. We wanted to emphasize the time element of the relationship and used temperature as a way to notice subtle deviations in the relationship that had been something over time.

______________________________________________________

Usage Scenarios

SEA can be used in many different scenarios that reveal how a person is feeling in the context of relationships and location. Below are some usage scenarios that the interface can afford the wearer.

Find warmth when you need it

There are many times when a user is in need of close relationships to improve an emotional state. In these moments, most people seek the warmth of people they care about and who you know care about them. A wearable like SEA allows a person to move throughout space and feel the warmth people around them based on locational proximity. As the user navigates in one direction, they can seek the warmth presented by the presence of a close friend, and use it to help navigate to them through space. SEA allows the user to specifically find people who will support them them in difficult moments, or even just on days where they are looking to find a good friend to chat with.

Avoid a toxic relationship

On rare occasions, there are relationships that sometimes head in the negative range for various reasons. Whether there was a conflict, a difference of opinions, or personality differences, a negative undertone is associated with the relationship. Sometimes, because of legal reasons, personal reasons, or for the integrity of a work environment, people tend to avoid these kinds of relationships. But sometimes, they are difficult to avoid because of proximity. SEA gives the user the chance to sense when a toxic relationship is nearby, and give them the ability to divert their path in order to avoid a bad interaction. This is mostly useful for relationships that have reached the termination phase, and are far from repairable/a dangerous presence.

Make bad relationship better

Because of the time element  SEA, a user is able to feel the progression of a relationship. If the relationship has always been very warm but is going through a rough patch and starts to get colder, SEA allows a person to notice this transition. Making people aware of fluctuations in relationships provides them with the opportunity to try and do something about a potential negative change. When a person starts to feel a mildly cold presence, it makes them aware of the change, and allows them to take action, and repair a decaying friendship.

Group scenario

When people interact with groups of friends, SEA considered the culmination of people in that setting as a group entity. Because different chemistries combine to create a different group chemistry, friends tend to act differently in specific group settings than with each person on an individual basis.

Understanding the temperature of an environment:

With SEA, the wearer can use temperature as a spatial navigator. Different environments we interact with on a daily basis hold different kinds of relationship dynamics. When a work setting is not the most positive environment, the user can understand the average of all interactions in a space and become aware that the environment there is toxic and could drive them to take action. And the opposite is possible as well. When a user feels continuous warmth, such as in a home setting or a dinner with friends, they have the ability to understand the supportiveness of that space, and perhaps could drive them to visit those spaces more.

Understanding the average temperature of interactions over a day:

Good and bad interactions influence your well-being and your perception about yourself. The companion app gives the user a snapshot of the average temperature of their interactions over the course of each day, so they can understand if their positive demeanor could be associated with multiple good interactions that day, or a depression was triggered by one really bad interaction that day, with very few others. This can help people keep track of their emotional well-being and take control of who they choose to interact with, and how they can help improve their states.

Feeling the relationships of another

Because SEA is customizable and detachable, users could switch their data sets by physically switching the bottom piece and inputting someone else’s data. This can have them either feel the relationships and interactions another person feels in space. It allows the user to develop empathy for the other person and their experiences.

Feeling the stresses of a person on your necklace

SEA can pick up the emotions/happiness/stresses of another person like a spouse to allow them to be empathic to the experiences of the other. When the user can feel the emotional state of a person they have switched their data with, they can start to be aware of why they are feeling as they are when they interact, for example when a partner is having a bad day, the user can be supportive. This is especially helpful, for example, for couples who want to understand how their partner is doing, or what they are feeling.

Awareness of a negative emotion like aggression

For a person that has issues with anger or depression, SEA can help them become aware of moments when they are acting in a way that is detrimental or negative and take action to change it. For example, when a user is becoming angry or starting to become negative, they can feel the necklace becoming cold, which could take them out of that state, and give them the chance to change the course of the interaction into a more positive one.

Awareness of topics of conversation and their emotional response

Because SEA analyzes the user’s voice in the interaction, it can also indicate when a certain topic of conversation is positive or negative as well. For example, when a user gets happy talking about a project they are working on, they can feel the happiness on their chest. The same goes for a difficult topic of conversation. This gives them the opportunity to avoid the topic or confront it with someone that’s supportive. They can start to become aware when a topic of conversation makes them happy often, or makes them sad.

______________________________________________________

Prototype Development

Hardware

Prototype Components

In order the keep the project complexity low and also make it possible to use the same circuit structure (data and electrical flow) to fit inside the SEA form factor presented here, we used one microcontroller (Arduino Micro Pro) receiving data from a Bluetooth Low Energy module (BLE), the last one is reading received signal strength information (RSSI) from other nearby BLE enabled devices, e.g. another SEA device. One of the main issues is power management and current flow. For this reason we added couple voltage regulators on the system, to supply 5V and 3.3V for the Microcontroller and BLE Module respectively. The Peltier Plate drains too much current (around 2A), which means it can’t be controlled directly by the Microcontroller I/O pins. Therefore we built a temperature controller system composed by MOSFET transistors driving a H-Bridge circuit (this will be better explored in the next topic).

Prototype 1 – components

System’s Control Diagram

Temperature controller

The device uses a thermoelectric cooler and heater component based on Peltier effect, creating a heat flux between the two sides of a plate. Using a H-bridge circuit driven by MOSFET transistors we can control the heat flux, therefore the device’s temperature.

Bluetooth Low Energy (BLE) Scanning System

As explained before, the distance tracking system is based on Bluetooth Low Energy Modules, i.e. inside each SEA device there is a BLE module that can work as a scanner or beacon (advertising BLE packages with informations like device name, type and manufacturer). Based on the exchange of BLE information the device can calculate the RSSI (received signal strength information – in dBm –power ratio in decibels (dB) of the measured power referenced to one milliwatt).        

BLE Modules communication / scanning test

Software

The Mobile Interface

SEA is controlled by a mobile phone application where the user is able to configure and visualize the interactions with others (a). The user can check his/her profile where a display of the current day SEA level will be displayed. The user is also able to see the interactions in time on a calendar (b). The calendar shows an evolution in time of the SEA interactions. A list of contacts is displayed on the Contacts section (c). All the users are listed and they will be displayed with a background colour according to the past interactions that the user had with each one of them. If the interactions with a user were not friendly, the user will be displayed in blue, if the interactions were warm interactions, the user will be displayed in red. It gives an average of the temperature felt with each one of the subjects for all the interactions had to that moment in time. Finally, a map of the spatial location of each user is displayed (d). The map also displays the colour of the average temperature per user and their influence in space around the main user.

Detecting emotions

The on-the-body form factor of SEA affords the ability to detect multiple inputs as a means to understand the emotions of the user. Both speech data and physical data (i.e. skin conductance and heart rate) are meaningful metrics for detecting the emotional state of an individual in an interaction. In addition to these modalities, semantic analysis can also be used to understand the digital footprint of a relationship through messages. Affective analysis using visual inputs from a camera could also be useful, but requires the user to be visible from a camera.

For the current prototype of SEA, we explored and implemented the ability to derive a subset of features that can be utilized to build a unified model for interpreting user emotions.

1. Valence & Arousal are powerful mechanisms for mapping emotions. We utilized the Python library pyAudioAnalysis to extract values for both valence and arousal using a SVM (support vector machine) classifier (13)
2. Emotional Prosody is characterized by fluctuations in pitch, loudness, speech rate and pauses in the user’s voice. We utilized OpenSmile to explore how to extract these features programmatically from audio files.
3. Spectrograms have recently been used with Deep Convolutional Neural Nets (14) in order to do speech emotion recognition. We utilized the Python script paura (13) to generate and play with spectrograms of sounds in real time.
4. Semantic Analysis can be used to process interactions over text including messages and Facebook. We also explored the ability to classify positive and negative values given a snippet of text. Enabling this at a larger scale to go through message blocks could be included in a future iteration of SEA.

______________________________________________________

Future Work & Conclusion

SEA’s technology and platform can be utilized in alternative ways relating to health, mindfulness, self-awareness, and empathic connections.

Future versions of SEA could allow a person to share their “feelings” with others simply by exchanging their “data” modules on their necklaces. This experience of being able to share “feelings” will potentially help establish more empathic connections between people by letting people understand how others might have interpreted the same situation or conversation. There is also the potential to experience the dynamic shifts of relationships.

Although SEA is currently in the form of a necklace, future versions can be located on other parts of the body: we can imagine a wrist form factor, for example.

The current version of SEA only provides one output on the body: temperature. Although this was a very conscious decision, there is the potential to add haptic vibration to the device. As the wearable is located on the chest, we could get heartbeat during conversations, events, physical activities, and playback the heartbeat as haptic vibrations while someone is going through a similar experience. This would allow for a connected experience between people, helping people understand what others might have gone through perhaps.

Conversely, future work could also include expanding on what modes of communication we are able to analyze as inputs on our platform. Currently, we have shown the ability to output features from speech and some basic text. It would be powerful to expand this to texts, social media messages, potentially even emails, and other forms of communication. In the digital age we are living in, interpersonal interactions extend beyond person to person conversations and enter the realm of social media and digital messages. These digital interactions can have just as much of an effect on people, and so we think it would be important if we can capture these interactions as well and integrate the information with SEA. Additionally, it would be meaningful to capture variabilities in heart beat and skin conductance of the user as a means to create a more robust model for emotional experience. In order for SEA to serve as a cohesive platform, work needs to be conducted to unify the various input and output modalities. A personalized machine learning model that is able to do this would be an important component of future development.

Alternative Usage Scenarios

Because SEA reveals how a person feels about an interaction, the possibilities are endless for it to be used as an interventional, awareness device in many different scenarios. Imagine a hospital setting with a caretaker/caregiver and a patient. Patients might not always be able to say what is on their mind or how they are feeling about how they are being treated by the caretaker or nurse or doctor. SEA would allow caretakers and caregivers to be more mindful of how they are treating their patients by giving them a sense of when words they say or actions they do influence their patients in a negative way.

We can also imagine SEA being used in the context of psychology. Psychologists and psychiatrists can train better using SEA and understand when their clients feel better or worse based on the types of words and advice they give. This could translate to relationship counselors as well.

In general, SEA could be used as a tool for people who are in professions that require them to speak and interact with many others: physical trainers, guidance counselors, teachers to name a few.

Ultimately SEA is a wearable that aims to provide people with a better understanding of how they feel in their interpersonal interactions.

Conclusion

SEA is a spatially aware wearable that provides people information about their interactions with others by using changes in temperature. A user’s voice during interactions with others is analyzed and translated into positive or negative emotional state using very subtle changes in temperature. Although currently a wearable + app that examines voice and vocal tone, SEA can become part of a larger platform that gathers emotional response and state from digital communication tools as well (Facebook, text, Skype, etc). The platform of emotional state and feeling awareness can be used in many different settings to help people be mindful of their actions and behaviors and the things they say. SEA can be used in a variety of ways, from a very personal, self-awareness way or in a very professional, caregiver/patient setting. SEA is the main component of a platform that will aim to help people be more mindful of how they are in relationships with others. Ultimately, SEA hopes to nurture healthier interactions between people.

______________________________________________________

References

[1] Avtgis, Theodore A., Daniel V. West, and Traci L. Anderson. “Relationship stages: An inductive analysis identifying cognitive, affective, and behavioral dimensions of Knapp’s relational stages model.” Communication Research Reports 15.3 (1998): 280-287.

[2] Chen-Bo Zhong and Geoffrey J. Leonardelli. Cold and lonely: does social exclusion literally feel cold? Psychol Sci. 2008 Sep;19(9):838-42. doi: 10.1111/j.1467-9280.2008.02165.x.

[3] Beebe, Steven A., Susan J. Beebe, Mark V. Redmond, and Lisa Salem-Wiseman. Interpersonal Communication: Relating to Others. Don Mills, Ontario: Pearson Canada, 2018. Print.

[4] Brueckner, Sofia. “Projects | Fluid Interfaces.” Fluid Interfaces Group, MIT Media Lab. N.p., n.d. Web. 20 May 2017.

[5] Williams LE, Bargh JA. Experiencing Physical Warmth Promotes Interpersonal Warmth. Science (New York, NY). 2008;322(5901):606-607. doi:10.1126/science.1162548.

[6] Graham Wilson, Dobromir Dobrev, and Stephen A. Brewster. 2016. Hot Under the Collar: Mapping Thermal Feedback to Dimensional Models of Emotion. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16). ACM, New York, NY, USA, 4838-4849. DOI: https://doi.org/10.1145/2858036.2858205

[7] Iwasaki, K., Miyaki, T., & Rekimoto, J. (2010). AffectPhone: A Handset Device to Present User’s Emotional State with Warmth/Coolness. B-Interface.

[8] Jordan Tewell, Jon Bird, and George R. Buchanan. 2017. Heat-Nav: Using Temperature Changes as Navigation Cues. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI ’17). ACM, New York, NY, USA, 1131-1135. DOI: https://doi.org/10.1145/3025453.3025965

[9] Nummenmaa, Lauri. “Bodily Maps of Emotions.” Proceedings of the National Academy of Sciences of the United States of America 111.2 (2014): 646-51. JSTOR. Web. 19 May 2017.

[10] Umberson, Debra, and Jennifer Karas Montez. “Social Relationships and Health: A Flashpoint for Health Policy.” Journal of health and social behavior 51.Suppl (2010): S54–S66. PMC. Web. 19 May 2017.

[11] WearableX. “Fan Jersey (X).” Let’s Xperiment. N.p., n.d. Web. 19 May 2017.

[12] Vinaya. “ALTRUIS X: FINDING STILLNESS IN THE CITY.” VINAYA. N.p., n.d. Web. 19 May 2017.

[13] Giannakopoulos, Theodoros. “pyaudioanalysis: An open-source python library for audio signal analysis.” PloS one 10.12 (2015): e0144610.

[14] Badshah, Abdul Malik, et al. “Speech Emotion Recognition from Spectrograms with Deep Convolutional Neural Network.” Platform Technology and Service (PlatCon), 2017 International Conference on. IEEE, 2017.

[15] AudEERING | Intelligent Audio Engineering – OpenSMILE.” AudEERING | Intelligent Audio Engineering. Audeering, n.d. Web. 19 May 2017.

Project by: Alexandra Rieger and Stefania Druga

Abstract:

Dementia and Alzheimer’s is currently the 6th most leading cause of death in the United States, more lethal than breast and prostate cancer combined (1). Currently, there are over 5.5 million Americans suffering from this disease. Employing theories of Human Machine Symbiosis we created MOTIF, a wearable device to support members of our society who need it most. MOTIF is a lightweight, auditory cueing system for individuals suffering from Alzheimer’s or Dementia. By playing songs in response to particular people, places and situations, MOTIF is able to trigger patient memories and provide context. Patients struggling with memory concerns can greatly benefit from this wearable musical intervention to improve their wellbeing and quality of life. Although MOTIF is targeted for individuals with Alzheimer’s, the multisensory cueing approach to cognitive wellbeing can be applicable in numerous conditions ranging from Autism Spectrum Disorder to Williams Syndrome.

Intro To Project and MOTIF

Although there are numerous services, medications, and care plans designated for patients with forms of Dementia such as Alzheimer’s, there are few interventions to improve the quality of life for individuals experiencing the symptoms accompanying these conditions. The highly researched factors known to improve cognitive wellbeing in patients with Alzheimer’s remains for the most part in research papers and does not become integrated into applicable models for clinical practice. Furthermore, hundreds of case studies, controlled experiments and qualitative reports reveal Music to be one of the most effective strategies to improve patient wellbeing. Although one would assume this to be both a simple and cost-effective solution, there are currently no symbiotic extensions incorporating music in patients for whom it would be most beneficial.

In light of this disparity, the MOTIF was developed. In the following sections, we will 1. first explore the background and condition of Alzheimer’s and Dementia, as well as studies exploring music interventions. Secondly, 2. we highlight the design of the Motif within this context. Thirdly, 3. we examine related work and design aspects which elevate the MOTIF’s function. Following this, 4. the design is analysed before solutions and evaluations are highlighted. Lastly, 5. we discuss challenges experienced and identify possible future scenarios, as well as strategic partnerships.

Alzheimer’s and Dementia Background

Scientist are still working to gain a fuller scope of the multifold factors which contribute to the progression and development of Alzheimer’s and Dementia. Factors related to genetics, health, environment and age are currently known to contribute to the abnormal buildup of plaques in the brain. These plaques are clusters of beta-amyloid, sticky proteins which accumulate between nerve cells. When beta-amyloid accumulations increase, cell-to-cell signaling at the synaptic level becomes obstructed. The plaque congestion triggers immune cells, leading to heightened inflammation as incapacitated cells are removed and absorbed. Disabled cells undergoing absorption often develop neurofibrillary tangles derived from the protein tau. Although tau generally acts as a regulator for the system’s transport circulation, in diseased regions, tau decays, essentially incapacitating the system of nutrient transport. Lacking essential supplies, these cells weaken and eventually dissolve. This causes regions of the brain tissue to shrink while ventricles containing cerebrospinal fluid, become engorged (2).

Symptoms Experienced in Alzheimer’s

As this process occurs, individuals begin to lose short-term and working memory. Stored knowledge of task-related skills decline and contextual knowledge such as place recognition, personal narrative and interpersonal connections decreases drastically. Patients experience these symptoms alongside elevated pain levels, disorientation and anxiety. At advanced phases they can often become completely unable to communicate: ‘locked in’ and therefore socially isolated (3). Eventually, patients may be unable to maintain self-care activities such as eating, speaking, hygiene practices and bodily functions. As the support of caregivers and clinicians increase throughout the progression of the disease, affected individuals can lose dignity in painful and dehumanizing circumstances.  

Although cures are still in progress through long-term studies, effective solutions are not currently near. There are however, specific interventions which target challenges faced by patients experiencing these remarkably challenging circumstances. One of the most effective strategies is in the form of music. Many studies reveal the benefit of music in Alzheimer’s Disease. Three selected studies are most relevant in this context as they specifically examine the benefits of music for: anxiety attenuation, memory recovery and social interaction – key symptoms faced in patients.

 

The Impact of Music in Anxiety Attenuation

A study conducted by H. B. Svansdottir Et Al. in 2006 at Landspitali University Hospital, Reykjavik, Iceland, revealed that music therapy has the capability to reduce anxiety in patients with moderate and severe Alzheimer’s disease and  dementia in  a case–control study. Patients (N=38) diagnosed with moderate or severe forms of Dementia were randomly assigned to participate in either the control or therapy group. The therapy group participated in interactive music singing and music listening sessions. Results indicate  “a significant reduction in activity disturbances in the music therapy group during a 6-week period – measured with the Behavior Pathology in Alzheimer’s Disease Rating Scale (BEHAVE-AD). There was also a significant reduction in the sum of scores of activity disturbances, aggressiveness and anxiety (4).” Although these results are encouraging, the longitudinal examination revealed the positive effects to remain for only four weeks following the end of music therapy. This aspect indicates that music interventions for these patients are effective but need to be implemented on an ongoing basis.

The Impact of Music Memory Recall

In a study performed by Nicholas R. Simmons-Sterna Et Al at Boston University in 2010, compared levels of memory recall in conditions including songs vs. non-music listening conditions. Participants diagnosed with Alzheimer’s Disease (AD) were encouraged to recall both personal memories and lyrical memories in conversation with researchers. Patients in the non-music condition experienced staggered and halted speech, struggling to recall basic wordings. However, in conditions where music was included in the interaction, participants experienced greater recall of lyric and personal memories. According to Simmons-Sterna, “the results confirmed our hypothesis that patients with AD performed better on a task of recognition memory for the lyrics of songs when those lyrics were accompanied at encoding by a sung recording than when they were accompanied by a spoken recording (5).”

The Impact of Music in Social Interaction

According to Ayelet Dassa, PhD and Dorit Amir’s  2014 study, “the role of singing familiar songs encouraged conversation among people with middle to late stage Alzheimer’s disease (13).” Researchers exposed a group of patients with late-stage Alzheimer’s to group-music therapy sessions over the course of a month. Patients participating in the sessions who listened to favorite songs from their younger years, experienced increased intersocial engagement with both fellow patients, family members and caregivers. Dr. Amir stated, “content analysis revealed that songs from the participants’ past, elicited memories – especially songs related to their social and national identity. Analyses also indicated that conversation related to the singing was extensive and the act of group singing encouraged spontaneous responses. After singing, group members expressed positive feelings, a sense of accomplishment, and belonging.” This study reveals how music integration  not only increases autobiographical conversation, but also present, spontaneous and improvisational discussion: key factors in social engagement.   

Cognitive Process Of Music Intervention in Alzheimer’s  

Music impacts the cognitive process in patients with Alzheimer’s on a multidimensional level. According to Simmons-Sterna Et Al, “Music processing encompasses a complex neural network that recruits from all areas of the brain, including subcortical regions such as the basal ganglia, nucleus accumbens, ventral tegmental area, hypothalamus, and cerebellum (Grahn, 2009; Levitin and Tirovolas, 2009 ; Limb, 2006) (6) and cortical areas such as the medial prefrontal cortex (Janata, 2009) (7) and orbitofrontal cortex (Limb, 2006) that are affected at a slower rate in AD compared to the areas of the brain typically associated with memory (Thompson et al., 2003). Thus, stimuli accompanied by music and a sung recording may create a more robust association at encoding than do stimuli accompanied by only a spoken recording in patients with AD.”  This reveals that music possess unique qualities capable of increasing cognitive recruitment throughout key brain regions.

 

Research Informed Design

The most difficult aspects of Alzheimer’s disease are the following factors: anxiety, lack of memory recall and the decline in social interaction. The studies cited in this paper  reveal that the implementation of hearing music, directly benefits the three named symptoms to a highly notable degree. This research informed the creation of the MOTIF, a device which is able to provide musical integration for patients with Alzheimer’s and Dementia in association with people, places and situations. The name is inspired by Wagner’s concept of “Leitmotif” meaning: a musical theme associated with a person, place, situation or idea. The word “motif” itself indicates: “a dominant recurring idea or pattern.” Knowing that individuals experiencing memory loss symptoms associated with AD are able to recall personal narrative and contextual clues through hearing particular songs, the MOTIF capitalises upon this.

The MOTIF is designed to allow patients to designate particular songs for people and locations. When loved ones or caregivers greet the patient by tapping their RFID card/keychain/ring/bracelet to the patient’s MOTIF necklace, their associated song will play, audibly cuing contextual memories for the individual with Alzheimer’s. As studies reveal that the impacts of music intervention in patients lasts only around 4 weeks post listening session (5), the MOTIF is contained in the form factor of a necklace, something which, unlike musical therapy sessions, will not “end” at a designated period but rather, act as a cognitive artifact: symbiotically connecting with the patient to extend their level of consciousness.

Related work

Ageing, Ingenuity and Design

In 2015, Yanki Lee & Patricia Moore (8) presented a series of case studies examining  how people age in different parts of the world making an important point that: by designing for elders we are designing for our future selves.

Researchers Lee and Moore spent several months with retired professors living on Tsinghua University Campus to explore how they approach ageing in a playful and creative way.

Their research not only showed how immensely creative elderly people are, but also how important it is for them to be part of a community and why they form this NORC (Naturally Occurring Retired Communities). In another paper in this domain, Professor Lee describes the roles of designers she envisions for a democratic innovation. From her experience (focused on design for social innovation in ageing) the primary task of the designer is to enable individuals to understand and interpret their own problems and situations –  subsequently innovating their own solutions.

New roles of designers in democratic innovation for elderly population

After their initial one-year ethnographic study to develop co-design relationships with elderly populations, they have defined three new roles of designers that are beyond facilitators of social innovation:

Make it public – Revealing What Is Possible

Yanki Lee and her team (8) have framed tactics employed by elderly individuals in their daily lives. Their aim is to examine how senior citizens bring innovation from their ageing processes into their everyday lives. The saying “continuing to contribute makes us live longer!” is a common feeling expressed by many of the retired scientists at Tsinghua Campus in Beijing, China. In fact, this echoes Khan’s (2013) finding of factors elderly individuals require to sustain their wellbeing: “1. To have a purpose 2. To have a sense of well–being 3. To feel at home and connected to others.” These sources reveal how connected communities are vital resources for retirees to search for continued purpose after retirement. Through observing the population at Tsinghua Campus, researchers identified three attitudes seniors expressed regarding their retirement: while some continue their lifelong professions; others expanded upon their work and developed new areas of research; further groups explored newer fields of interest or resumed unfinished pursuits

Driving Actions – Reimagining Everyday Life

Lee and fellow researchers focused primarily on methods devised by retired individuals to develop their own ways of tackling the ageing process. They also explored whether the elderly populations could be mobilised to organise participatory research projects. Lee’s team collected a variety of innovative ideas produced by retirees regarding the welfare of Tsinghua including: a healthy restaurant for elderly patrons, greener burial practices (e.g. tree burial), organ donation advice and a class on elderly care. The level of awareness regarding situations facing the community is an essential element to encourage self-actualisation which facilitate innovation.

A selected aspect is the ‘patterns of living’,  survey of housing usage with over 30 typologies of housing design on the campus.This informs our project by revealing how aspects of design can impact the daily lives of elderly individuals.

Create social innovation – Exploring Social Design Practice  

An important element for social innovation is be the investigation of the changing roles of designers when democratising innovation. Manzini and Rizzo (2012) from the DESIS network constructed a new typology of the role of designers: triggers, co-design members and design activists (9). Researcher’s Lee Et Al. believe designers should initiate social design experiments to provide novice practitioners with a concept of social-change agency (8). Through this, they encourage designers to create an informed relationship with the potential users of their object of design.

Related Projects

Simple Music Player for Dementia: Controllers That Are Easy To Use

E2L developed the ’Simple Music Player’ (10) supported by research from the Bath Institute of Medical Engineering (BIME). Their findings highlighted the therapeutic benefits of music on individuals with dementia for improving quality of life (especially when living alone) and for restorative effects on cognizance and lucidity. Unfortunately, many individuals with memory problems are unable to operate the modern music playing devices currently available on the market.

With the E2L player, family and friends can upload a personalized playlist and adjust the volume. Following these arrangements, the player is fully operational through the use of lifting and closing a lid to turn the music on and off, and one button to change the track. The music will always continue from where it was last stopped. The form factor is reminiscent of old radios and is therefore instantly recognisable as a music-playing device

Music Boxes for Memory : Treasure Box Metaphor

Video: https://vimeo.com/64630528

University of Brighton Design graduate, Chloe Meineck, created a music box for dementia sufferers filled with RFID-embedded objects that, when moved to certain spots, prompt related musical tracks.

Dr. Oliver Sparks, who had heard corroborating evidence from his patients and their relatives for many years, said: “Music of the right kind can serve to orient and anchor a patient when almost nothing else can” (11).  This is the premise that led Chloe to create the Music Memory Box for her subject, Barbara, an elderly resident with early-stage Alzheimers. The idea involves collecting a series of “treasures” — a life’s collection of memorable trinkets — and use these items to prompt related memories by matching them with evocative songs or compositions, played using RFID tags.

The project was entirely collaborative: Meineck Et Al., devised the idea, while Barbara suggested the objects, design and music. Real trinkets were used when possible, otherwise, Barbara would suggest an additional item and Meineck would draw it before making a model. Following this iterative process, Barbara and Chloe discussed the design and decided upon changes. Without Barbara’s participation, it would be impossible to achieve the final results — a design that can offer Barbara comfort in future years when her disease advances.This example, akin to research provided by Lee Et Al.,(8) emphasizes the importance of collaborative creativity when designing with elderly individuals. The theme of identifying evocative objects and music reveals that projects of this kind are especially successful when the user is included.

Proximity Button – Wearables That Don’t Restrict The Hands

The Proximity Button was invented by Natalie Price, the daughter of a Dementia Health-worker. It is designed to be an effective and affordable way to keep loved ones safe without being intrusive. When the button is clipped to the clothing of a patient, it connecting to the caregiver’s phone via Bluetooth. The button sends an alert when the patient leaves a designated vicinity. Price recognized the design was a particular challenge in testing: “It’s very common for people living with dementia to become upset with things that they aren’t familiar with. “Those wearing the earlier bracelet prototype kept asking me if I wanted my watch back, or fiddled with the strap in an agitated way.

Source: http://www.proximitycare.co.uk/

The pendant prototype didn’t cause any discomfort issues, but was too easily removed.“The badge prototype came out on top. Once attached, the person wearing the badge completely forgot it was there, and it stayed in place. The design has been tested and has proved totally successful; it’s discreet and doesn’t cause any discomfort.”

Our Solution

MOTIF, is a lightweight, auditory cueing system for individuals suffering from Alzheimer’s or Dementia. By playing songs in response to particular people, places and situations, MOTIF is able to trigger patient memories and provide context.

Different people and locations can be identified by using customized RFID tags either worn as necklaces or keychains. The MOTIF necklace is able to detect these different tags and play different songs for each individual.

Inside the necklace we used a microcontroller, Mp3 player shield, RFID reader/writer, speaker/recorder and a battery. Each necklace can be customized by using different designs and materials

Future scenarios

In the future, we envision exploring the use of MOTIF in the context of patients with other cognitive and physical pathologies including: Autism Spectrum, to reduce sensory overload episodes, Williams Syndrome for learning interventions, PTSD to increase grounding behaviors as well as general conditions surrounding aging and memory loss. Having recording and playback capabilities, we imagine the MOTIF could act as a reminder, encouraging patients to take medications at certain intervals of the day, notifying of doctors appointments or providing location information to help a patient’s orientation.

Moving forward, we seek to to minify the current prototype and enable BLE connection to a mobile app that could enable the users to associate new songs with their current location and write new tags on the spot.  Furthermore, despite extensive research on early stage detection bio-markers, we have yet to fully examine the therapeutic intervention of the MOTIF in the context of delaying the onset of Alzheimer’s. Through evaluating current research and findings, it is clear that music plays a vital role in the cognitive capacities of individuals with Alzheimer’s. Knowing the extensive benefit, music can provide to wellbeing (4), humanisation (5) and quality of life (13) for affected patients, we propose that it is inhumane to deprive the gift of music from these patients. MOTIF will help to establish a patient’s sonic memory palace and fill their lives with this powerful intervention on a consistent basis.

References:

(1) “Latest Alzheimer’s Facts and Figures.” Latest Facts & Figures Report | Alzheimer’s Association. N.p., 29 Mar. 2016. Web. 19 May 2017.

(2) Dawbarn, David, and Shelley J. Allen. Neurobiology of Alzheimer’s Disease. Oxford: Oxford UP, 2007. Print.

(3) Hof, Patrick R., and Charles V. Mobbs. Functional Neurobiology of Aging. San Diego, CA: Academic, 2001. Print.

(4) Svansdottir, H. B., and J. Snaedal. “Music Therapy in Moderate and Severe Dementia of Alzheimer’s Type: A Case-€“control Study.” International Psychogeriatrics 18.04 (2006): 613. Web.

(5) Simmons-Stern, Nicholas R., Rebecca G. Deason, Brian J. Brandler, Bruno S. Frustace, Maureen K. O’connor, Brandon A. Ally, and Andrew E. Budson. “Music-based Memory Enhancement in Alzheimer’s Disease: Promise and Limitations.” Neuropsychologia 50.14 (2012): 3295-303.

(6) Scheck, Anne. “Charles Limb, MD, on the Science of Music, and the Music of Science.” The Hearing Journal 66.11 (2013)

(7) Janata, Petr, and Lawrence M. Parsons. “Neural Mechanisms of Music, Singing, and Dancing.” Language, Music, and the Brain (2013): 307-28. Web.

(8) Lee, Yanki, and Patricia Moore. Ageing, Ingenuity and Design (2015): n. pag. Print.

(9) Manzini, Ezio, and Francesca Rizzo. “Small Projects/large Changes: Participatory Design as an Open Participated Process.” CoDesign 7.3-4 (2011): 199-215. Web.

(10) “Simple Music Player for People with Dementia and Similar Problems with Cognizance.”Dementiamusic.co.uk. N.p., n.d. Web. 19 May 2017.

(11) “MUSIC MEMORY BOX.” Studio Meineck. N.p., n.d. Web. 19 May 2017.

(12) “Home.” Proximity. N.p., n.d. Web. 19 May 2017.

(13) Dassa, A., and D. Amir. “The Role of Singing Familiar Songs in Encouraging Conversation Among People with Middle to Late Stage Alzheimer’s Disease.” Journal of Music Therapy 51.2 (2014): 131-53. Web.

Guillermo Bernal, Jiabao Li, John Stillman

Abstract

Focus Agent (FA) is an exploration into how an agent can improve its host’s focus and performance by changing their virtual environment. The FA learns how to visually and audibly augment or diminish the host’s world view for optimal performance through a neuro and bio feedback loop. As part of our research we built an FA prototype that is designed to provide optimal environmental states to optimize the host’s focus on personal tasks. In particular, our demonstration sought to improve meditation, creativity and logic tasks. To that end we built an AR system using a commercially-available EEG headset for input, an interface created in Unity along with a Microsoft Hololens for output. Through our experience building the FA we believe that this type of system can improve focus and task performance. There is, however, work to be done to improve the system. The user experience can be more intuitive and the agent interventions can be more subliminal. The hardware implementation must integrate the sensors with the display hardware. Finally, improved signal processing along with the implementation of a machine learning component are necessary to enable the agent to learn the needs of the hosts through an iterative feedback loop.

Motivation and Idea: Overwhelming digital intake

The balance between work and home is harder to manage than ever before. We are often so overcommitted and overbooked that the hectic pace of modern life takes a toll. This is also called cognitive overload and occurs when task difficulty exceeds resources available by a person – in this case task performance starts to decline.

 

If you’ve ever been trapped in a no-focus infinity loop, you know how hard it can be to get out.

The field of cognitive science has also shown us that there are ways to get our focus back as quickly and efficiently as possible.

Selective attention is a cognitive process in which a person attends to one or a few sensory inputs while ignoring the other ones. Selective attention can be likened to the manner by which a bottleneck restricts the flow rate of a fluid. The bottleneck doesn’t allow the fluid to enter into the body of the bottle all at once; rather, it lets the fluid to enter in certain amounts depending on the flow rate, until all of it has entered the bottle’s body. Selective attention is necessary for us to attend consciously to sensory stimuli in such a way that we will not experience sensory overload. To implement the focus agent we leverage how the brain chooses what to focus on Neuroscientists call this ‘selective attention’, and it comes in 2 different forms:

1. Top-Down

This is the holy grail of focus. Top-down focus is goal oriented. It is responsible for seeing the bigger picture and uses your past experiences to figure things out.

Happens when: You’re studying for an exam or trying to solve a difficult problem.

2. Bottom-Up

When a thought creeps up on you or something around you grabs your attention (like a ping, bing, or notification) you’re suffering from Bottom-Up focus. You can’t help but pay attention to what’s happening.

Happens when: You hear a loud noise, someone pops out of the bushes or your phone buzzes. [2]

We perceive the world indirectly by processing and interpreting the raw data from our senses and our thoughts and behavior are frequently biased by our senses. Our perception of people and our attitude towards tasks are influenced by stimulus that our environment present us like temperature, colors and sounds[13] 

Human- and computer-based interactions are ubiquitous in our everyday lives—everything from our smartphone, to a variety of computer systems creates serious cognitive demands when switching from one to the next.

We explored subliminal techniques to improve human-computer interaction. The main idea is to “inject” information into the human mind below active awareness, thus transferring supplementary information in a subliminal style without adding load on the cognitive channel. The main benefit of this approach would be the reception of additional, essential information even in the case where (almost) no capacity is left for information transmission in a traditional way.

Subliminal interfaces have long been explored in the HCI and Art community. Here we present two projects that explored subliminal interfaces from two different angles but very relevant to our explorations. Projects like Subliminal Wiretapping a subtly interactive artwork that utilizes random number generation modified through mind-matter effects to supply a continuous stream of words. Frequent, personal connections emerge from participants interpreting the stream as the words appear. The object of the artwork is to utilize the psyleron as a listening device [3x]. The psyleron taps into the conscious and subconscious streams of those participants aware of the device. The participants influence the letter choices that ultimately create words. Subliminal Wiretapping then becomes a mix of ouija board and automatic writing.[2]

Another project and closer to our topic is Incorporating Subliminal Perception in Synthetic Environments. An interactive visualization of a synthetic reality platform that, combines with psychophysiological recordings, enables the researchers to study in realtime the effects of various subliminal cues.

Here the display of subliminal information is assisting the user in the operation of the appliance upon detection of difficulties in interacting with movable internal components. Assuming that the user in an exploratory phase of explicit interaction with the various internal compartments of a refrigerator: upon detection of difficulties (which can take place through a combination of physiological signals and task performance analysis), the system may decide to provide subliminal assistance, inspired by an approach described by De Vaul et al. [4], contextualized to the type of parts on which the user attention has been concentrated and the current position and orientation of the user.

The intelligent interface designed to elicit and study various forms of implicit perception. This is a new kind of application which integrates the various contexts in which implicit processes are relevant, both in perception, information acquisition, and decision making. The intelligent part of the interface, which is based on AI planning techniques, makes it possible to experiment with various phenomena associated with subliminal perception, provided these can be expressed as a sequence of visualization operators. This should make it possible to explore more complex phenomena than transient display, including change blindness, flashing, crowding and masking. Meanwhile in our preliminary experiments we have shown that our system is capable of generating contextual responses on both EDA and fEMG, potentially closing the loop between explicit interaction, implicit presentation of information and implicit user response

How can we understand the user’s cognitive state and how can we intervene?

Detecting cognitive load from EEG signals

EEG signal is a non-stationary signals which have different frequency elements at different time intervals. In this study, EEG signals was analyzed using MATLAB Toolbox for EEG ANALYSIS AND CLASSIFICATION (EEGAAC) [5]. Matlab’s toolbox  is suitable method for multiresolution time frequency analysis. In this work, EEGAAC decomposed the EEG into four sub-frequency bands; Delta (1-4Hz), Theta (4-8 Hz), Alpha (8-12.5 Hz) and Beta (12.5-30 Hz). The wavelet decomposition level was set to 5-levels and one final approximation since we are interested in the frequency range of 0-30 Hz only. Table 1 gives a summary of the frequency distribution with wavelet decomposition levels.

Another method of measuring cognitive load is the NASA-TLX test [HKD∗99]. This test describes cognitive load in terms of subjective responses to a post-experiment survey. This methods is not ideal for the system architecture that we would like to explore but can be used to cross corroborate our EEG setup.  

Research on this topic shows that an increase of the rhythm for Beta waves are correlated to analytical type tasks[], suggesting that we can use this band to understand the user state during this type of tasks. Decrease in alpha rhythm power from level one to level two and failed to drop in level three of mental stress.

Neurofeedback Loop

Neurofeedback also called neurotherapy or neurobiofeedback, is a type of biofeedback that uses real-time displays of brain activity—most commonly electroencephalography (EEG), to teach self-regulation of brain function.  Research shows neurofeedback may be a potentially useful intervention for a range of brain-related conditions. It has been used for pain, addiction, aggression, anxiety, autism, depression, Schizophrenia, epilepsy, headaches, insomnia, Tourette syndrome, and brain damage from stroke, trauma, and other causes.[6]

Method

For our first step to creating the FA was to create a low fidelity prototype. After storyboarding an FA/host interaction scenario, we proceeded to create a non-interactive prototype video. We started with an environment rendered using the Unreal Engine and then layered various UI elements including the Focus Agent icon, the “To Do List” menu as well as environmental elements that would change according to the task at hand. We screened the video for the Media Lab MAS.S60 class as well as other solicited viewers. Feedback was incorporated into our design for the final prototype.

At this point we decided that AR would be the most appropriate format for our final prototype. In order to leverage available COTS hardware and software we chose to develop our system on the Microsoft Hololens platform. For our neurofeedback input we tested various EEG sensors including the Emotiv Epoc+ and Interaxon Muse headsets. While the Epoc system outputs a wider array of EEG signals than the Muse, it’s form factor did not allow it to be worn concurrently with the Hololens. For this reason we used the Muse for our testing.

We switched from using Unreal Engine to Unity to build user interface and virtual environment after deciding that the Unity SDK was better aligned to the Hololens. Vuforia was used to diminish and augment the host’s environment by masking distracting real-world elements as well as adding virtual elements to the host’s environment depending on the state of the Focus Agent. We prerecorded human dialogue to act as the voice of our FA.

To model how we will implement the machine learning layer of our system in the future we tested Dynamic Time Warping algorithm which will have to be explored further to process our EEG signals.

We envision the relation of the focus agent and the host to be mutually dependent.

1. Focus Agent – Host Relationship is a system where the Agent learns from its user in a mutual symbiotic relationship
2. The Focus Agent can subliminally change the environment visually and audibly (augment, diminish).
3. The Focus Agent can provide assistance to complete tasks based on user gestures.
4. The Focus Agent can communicate to other agents within the user’s network.

Conclusion and Future Work

We have learned that through neurosensing we can gauge a host’s attentional level.  We have also found that the host’s perceived environmental state can be altered by the FA to improve focus and performance across a variety of tasks ranging from creative to cognitive. Through Machine Learning the agent’s efficiency will improve over time as it acquires more data via the symbiotic feedback loop that we propose. With these enhancements and a more clinical study of the problem we believe that a system such as ours promises to improve human focus and enhance task performance.

Refinements we would like to implement include:

– higher fidelity, less intrusive AR hardware with a wider field of view
– improved object tracking
– ML implementation that learns to tailor FA intervention to the individual host and situation.
– better sensor input including a wider array of EEG sensors and the addition of biosensors such as EMG and ECG
– Advancements to the UI to make interaction more subliminal

With these improvements and a more clinical study of the problem we believe that a system such as ours promises to improve human focus and enhance task performance.

Reference

[1] Top-Down Versus Bottom-Up Control of Attention in the Prefrontal and Posterior Parietal Cortices
http://science.sciencemag.org/content/315/5820/1860
[2] Subliminal wiretapping http://dl.acm.org/citation.cfm?id=1640336&CFID=764659471&CFTOKEN=81109716 
[3]Incorporating subliminal perception in synthetic enviroments 
http://dl.acm.org/citation.cfm?doid=2370216.2370458
[4]The Memory Glasses: Subliminal vs Overt Memory Support with Imperfect Information
http://citeweb.info/20030106257
[5]EEG ANALYSIS AND CLASSIFICATION
https://www.mathworks.com/matlabcentral/fileexchange/55112-eeg-analysis-and-classification
[6] Neurofeedback
https://en.wikipedia.org/wiki/Neurofeedback
[7]NASA-TLX test
http://www.sciencedirect.com/science/article/pii/S1386505615000635
[8] Researchers find out why some stress is good for you
http://news.berkeley.edu/2013/04/16/researchers-find-out-why-some-stress-is-good-for-you/
[9] Syncopation, Body-Movement and Pleasure in Groove Music
http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0094446
[10] ‘Don’t Waste My Time’: Use of Time Information Improves Focus
https://people.ucsc.edu/~swhittak/papers/meTime_CHI16.pdf
[11] Subliminal Communication in Human-Computer Interaction
https://www.hindawi.com/journals/ahci/2011/156028/cta/
[12] Mental Stress Quantification Using EEG Signals.
https://link.springer.com/chapter/10.1007/978-981-10-0266-3_4

[13]Sensation, the new science of physical intelligence, Thalma Lobel