Now with more buttons!
First, you have a set of digital screens. A tabletop and some monitors.
You can add and manipulate any data you want on the screens: text, tables, images, audio, video, you name it. Manipulate these data with your hands, touch-screen style. Draw on them with a digital pen, white-board style. Add text with a keyboard, or write with the pen. Annotate, scribble, format, rotate, erase.
Technology already exists to merge physical pieces of paper with digital paper (e.g., ProofRite, PaperProof, various digital pen and dot patterned paper, and so on), so why not, we can add that technology to this workspace as well! All the things you can do with paper scattered across a desk, you can do here, with more media.
The key here is that you can lay out your data and ideas in a non-linear way. Take pages 3, 15, and 27 from a pdf and put them side by side. Set a clip from an article you read nearby. Draw a line connecting these to a set of audio recordings you have from a lecture. Easily move information between screens. Jot notes everywhere and anywhere.
But there’s a problem. Everything is digital, visual, and constrained to your screen or to pieces of paper.
How can we improve?
Just add robots!
Everything is cooler with robots.
So let’s add a social robot sitting at the table with you to facilitate the brainstorming process. Not as a replacement person, but to help you interact with your data and ideas. It can analyze all your content. It can ask questions. It can search for new or relevant information that you may not have considered. It can direct attention verbally and visually to previous ideas you’ve had to help you connect your ideas (e.g., Staudt, 2009).
The robot will be able to analyze the linguistic and semantic content of your screens. (Until image and video processing gets better, those may have to be tagged with text for now.) A lot of machine learning, AI, natural language processing, and data processing algorithms exist for analyzing large sets of data like these, generating semantic concept nets or category structures, finding similarity dimensions, feature sets, and so on. All this previous work can be leveraged here.
The robot can also be equipped with sensors: eyetracking in the tabletop surface and on the screens (eyetracking monitors already exist, e.g., from Tobii Technologies, and some people are trying to develop eyetracking webcams), cameras, a microphone. Data from these can be analyzed in order to monitor attention (e.g., Narayanan et al., 2012) affect/emotions (e.g., Calvo & D’Mello, 2010; D’Mello & Graesser, 2010), and other behaviors.
With all this information, the robot, as a social actor, can help you interact with your ideas. It can suggest relevant words or phrases, papers, articles, people who have expertise in areas you’re thinking about, news, video clips, and more. Drawing on the automated tutoring system literature, it can ask questions that will lead to deep thinking about ideas.
But…. robots can’t do everything!
Stormbot is a tool to help you stay on track. It can guide you away from destructive thinking and mental blocks by detecting your frustration. It can help you generate and connect ideas. But it’s still just talking about information represented visually on digital screens.
Buttons
One way of thinking about sorting, categorizing, and organizing ideas is based on their similarity and feature sets. Buttons — actual, physical buttons, like the kind on sweaters, jackets, and bags — are a good visualization of this. Buttons can be round, or square; they can be different colors; they can have different numbers of holes. You can group them based on their feature sets and similarities in a large number of ways: blue buttons with two holes. Red buttons with four holes. Cloth buttons. Metal buttons.
What if we could sort ideas the same way?
Recall that the robot, on the backend, is already analyzing the content of your screens. Why not extend and apply this: determine feature sets. Judge similarity. Group ideas into semantic categories or networks. Basically, assign “buttons” to your ideas.
Physical buttons!
Bring that into the physical world. Have actual, physical buttons linked to each of your ideas on the screen. This is the kind of thing that might be especially helpful as an ideation game for kids, with the robot character leading the game and facilitating the interaction.
The buttons could just be real buttons, or they could be specially 3D-printed, or they could even (and perhaps ideally) be reconfigurable on the fly, incorporating new ideas and sorting schemes as you change data, add ideas, and group ideas together. Whether you’re trying to design software, a new product, come up with a research question, or write a paper, the goal is to allow you to interact with your ideas on another dimension. The buttons, with the robot’s lead, allow you to see similarities and draw connections in a different way.
