Hapticolor: Interpolating Color as Haptic Feedback to Assist the Colorblind

Overview

“Do the colors of these socks match?” You often hear colorblind people asks that while shopping or at household. Hapticolor is a haptic wristband that not only helps them to differentiate confusing colors, but also with recognizing and comparing them.

Screenshot 2016-10-23 17.27.46.png

Hapticolor is a haptic wristband with 3 vibration motors that interpolates 12 colors (on a color wheel) with sequential long or short pulses.

Finally, they can see different shades of gray.

Problem Statement

Colorblind people encounters many difficulties in everyday life that people with normal color vision wouldn’t, such as matching socks. Based on our interviews, we learned that they not only want the ability to tell confusing colors apart, but they also want to be able to compare the distance of the colors they see. This is important in scenarios such as comparing and matching colors of apparels in the department store.

img_20120409_174722

Matching socks is difficult for colorblind people. Image courtesy of Kelly Carpenter.

screen696x696.jpegWhile there are mobile applications that allow users to take photos and shift the colors accordingly, doing so is often slow and awkward as the users need to pull out their phones, take a picture, and wait for the processing to finish. On top of that, these applications only allow them to distinguish colors, not recognize and compare them. We need a more convenient and versatile solution in these circumstances.

Enters Hapticolor.

 

Users

Hapticolor targets the colorblind population. That is approximately 200 million people all around the globe (8% of all male and 0.5% of female) who can benefit from our haptic wristband. Hapticolor enables colorblind people to distinguish, recognize, and compare the confusing colors in their surroundings without needing other people’s help. Hapticolor can be implemented in modern smartwatches and bracelets. With Hapticolor, colorblind users can finally go shopping, evaluate the graphic design, and conduct many other tasks without asking for help again.

General Approach

The Hapticolor project has five core members, with two engineers, two research leads, and one advisor. I’m one of the two research leads, and I was responsible for mentoring the other lead in this project due to my previous experiences with a colorblind project.

ChromaGlove (2).pngChromaGlove (1).png

In the beginning of this project, we explore gloves as the wearable to implement Hapticolor.

Hapticolor started off in a glove form factor. But it wasn’t very convenient in wearable scenario, where users need to interact with everyday objects with their fingers. We explored other form factors (such as rings) before settling on the wristband as it gives us enough surface area to work with, and is already commonplace in mobile wearable space with the proliferation of smartwatches and health bands.

After finding the suitable form factor,  we looked for a suitable color representation model to map different colors as haptic feedback. This is probably the most difficult problem in our research, as we need to find a representation that is familiar to the colorblind people and also integrates well with our haptic model. After extensive exploration, we came up with the color wheel model to realize this mapping.

rgb-Colourwheel.png

The RGB color wheel with 12 colors.

To find out if the model and the form factor were effective, I assisted the other lead to design user studies and talk to colorblind users. Last, as I was more experienced with managing projects, I was also tasked with managing the expectations of the project to ensure all members were productive and on schedule.

Constraints & Main Findings

Hapticolor uses the color wheel model, which has 12 different hues, to represent colors for the colour-blind users. Even though we can represent each hue with one haptic motor, recognizing 12 different motors on the wrist can be very challenging. This constraint becomes two of the three research problems we tackled in this project, namely: 1) how many motors can be placed on the wrist without sacrificing accuracy? and 2) Assuming we have fewer motors than the number of colors on the color wheel, how can we interpolate color information accurately and efficiently with different vibration dimensions (duration, pulse, intensity)? Which dimension is the best? We answered these questions by conducting controlled user experiments.

figure_1_2.png

How do you represent 12 colors with 3 vibration motors effectively and easily?

In short, we discovered that users can perceive 3 to 4 vibration motors on the wrist accurately (>90% accuracy), duration of the vibration (long or short) is the best dimension to interpolate information between motors, and that vibrations triggered sequentially are better than vibrations triggered simultaneously. In our ecological validity study with six colorblind users, we found that Hapticolor enables users to distinguish, recognize, and compare colors with accuracy from 94 to 100%, compared to 60 to 80% without any colorblind aids. For more data, please refer to our paper.

Design Process Stories

When we started this project, we were hoping to use haptic feedback to provide color information for colorblind people. We learned that this problem hasn’t been solved, as existing haptic solutions were not designed for humans. Instead, they were proof-of-concepts that did not take usability into considerations.

We took a different approach. We first took a step back to understand the context where a haptic-based solution is most needed. We achieved this by interviewing a few colorblind people on the kind of problems they encounter in everyday life. Our data suggested that colorblind people usually has three main categories of problems – differentiating, recognizing, and comparing colors in mobile scenarios like shopping.

After understanding the characteristics and resulting behaviors of the main problems, we started to brainstorm haptic modalities that can be treated as a continuum, and we reverse engineered the essential components (e.g., how many motors to use?) and dimensions (e.g., which vibration dimensions and temporalities?) to identify the best combination. We ran user studies to determine the number of motors that can be reliably perceived on fingers, palms, and wrists, and then we moved on to compare different vibrations dimensions (intensity, duration, pulses) and temporalities (simultaneous, sequential) to identify the most effective combinations. Finally, we used the data we collected to develop a working prototype and evaluated it with actual colorblind users, and with actual tasks. Throughout the design process, I found the process of brainstorming potential solutions for each individual component of the prototype to be especially exciting and invigorating, as it allows myself to flex my creative muscles in the process.

Lessons Learned 

In this project, the main lesson I learned was how to mentor team members with less research experience. Also, every person’s ability to handle stress while under pressure is different, and I learned that before understanding how they work, it is important not to impart our own (working) expectations on others. As a designer, we all have to learn how to develop empathy for the users, and I would argue that developing the same empathy for our team members is crucial to their ability to produce good work in the long run and in a sustainable fashion. To achieve that, mindful communication and empathy are important, and working on this project has helped me to realize that.

 

Advertisements