Korero: Facilitate Multiple and Specific References to Visual Materials in Asynchronous Discussion Interface

Overview and Problem Statement

In face-to-face learning discussion, we often refer to visual materials such as books, notes, videos, and in the classroom setting, even what the instructor is saying, while discussing the topics at hand. This makes discussion more effective and comprehensible, and in the era where students from very diverse backgrounds are learning together (e.g., MOOCs), the ability to make referencing becomes increasingly important.


In face-to-face discussion, affordances such as shared visual space and awareness are available to every discussant, which enables elaborate referencing.

Because of the shared awareness and visual space available in face-to-face, we can do pretty elaborate referencing, like refer to multiple materials at the same time while being very specific about what you’re referencing in the materials. However, at the moment, no asynchronous discussion interface (e.g., discussion forum) can accommodate this level of elaborateness in the same user interface. Recreating the richness of face-to-face interaction in the computer interface has been a longstanding challenge for user interface designers.

Screenshot 2016-11-05 14.56.20.png

In discussion forum, users can embed images and videos in their post, but it takes up a lot of visual space. While users can create a hyperlink to the material, it takes them to another browser window, which is inconvenient if they want to read the discussion at the same time, and that takes place pretty often during learning. The screenshot is taken from Coursera Machine Learning Course.

We explored the design space iteratively to come up with the right design requirements, and we develop Korero (means discussion in Maui language), a new asynchronous discussion interface that allows users to refer to multiple visual materials in a specific manner. We believe Korero will enhance the effectiveness and user experiences of learning discussion, and it represents a significant step in enabling engaging online peer learning in the future.

Figure 1.png

The Korero interface. It has a contextual activity window that allows discussants to view any materials alongside the discussion, and an on demand widget that enables them to get a glimpse at all the materials being referred, and select one to be expanded in the activity window.


While Korero can be used for any asynchronous discussion purposes, we researched and designed Korero specifically for learning-based discussion, where learners often refer to visual materials in the discourse. What is the size of the target audiences that might use this? In one of the more popular online learning models, MOOCs, more than 5 million people from all around the world have taken and enrolled in one¹. As these people come from diverse background, the potential language and knowledge gap in the discussion group becomes bigger, which makes the discussion harder and more inefficient to participate. Making referencing easier is one of the ways to reduce these gaps, since instead of speaking, learners can directly point to the learning content and elaborate their perspectives.


I led the five-person team that researched and developed Korero, with one software engineer and three research advisors. In the beginning, I conducted a brief literature review to ensure the research problem is valid, original, and significant (P.S: This is a framework I personally use to ensure the research problem is a good one).  Based on literature that reveals the limitations of existing interfaces, I designed a low-fidelity prototype using Apple Keynote that showcases the ability to specify a particular context in the material to refer, shown in the image below.

Screenshot 2016-11-05 15.11.38.png

The first UI prototype of Korero (v1). The figure showcases the referencing task where the user wants to select a particular material to refer in the activity window (left). 

After gathering feedback from my collaborators, we decided to explore a few other alternatives that put the discussion and the materials at different position within the UI. One of the alternatives we explored is shown in the image below. We conducted a pilot study with this prototype and compared it with two other existing interfaces. We designed a few referencing tasks that require the subjects to refer multiple materials specifically. After the pilot study, we also interviewed our subjects for more detailed feedback.

Korero Prototype for PS1 - Practice Session.020.jpeg

In the second iteration of the Korero prototype (v2), we decided to experiment with putting the discussion at the center while inserting snippets of materials on the side, with numbered links connecting them together. 

The pilot study (with 4 participants) was very important to our research and design process in a number of ways. First, it affirmed the referencing issues with the two existing interfaces. While previous literature had already pointed them out, we discovered that it is crucial to observe how users deal with those issues in order to come up with better design ideas for improving the overall user experiences of our prototype. Second, conducting preliminary usability test also allowed us to understand user’s mental model (of referencing) and the think-aloud protocol we implemented allowed us to pinpoint minor issues with the referencing workflow we devised.

Using the pilot study findings, I immediately sketched down a new iteration of the Korero prototype using paper and created an interaction flow to illustrate the interplay between different UI elements, shown in the image below.

Screenshot 2016-11-05 15.42.43.png

Paper sketch to illustrate the interaction flow of the third version of the Korero prototype (v3). We reverted back to the discussion panel UI paradigm (on the right side) used in v1, and put all the materials snippet on the bottom of the video activity window. The sketch here illustrates the 8th step in the interaction flow.

After getting advisors’ feedback on the sketches, I transferred the paper prototype into a high-fidelity prototype using Sketch App and InVision (see image below). Together, they bring the user interactions to life. In this process, I also adopted the Object-Oriented UX methodology by Sophia Voychehovski to delineate the components of each user interface objects (see image below).

General Summary of the Korero Interface.jpg

The fourth and final iteration of the Korero prototype (v4) before production. Instead of having a persistent snippet panel on the bottom of the activity window, we make it appear in an on-demand fashion (as a pop up) when the user clicks on the referred term. 

Screenshot 2016-11-05 15.53.44.png

Object Map of the Korero prototype.

With the object map and the hi-fi prototype, I worked with the software engineer to implement it into a web-based environment using Polymer and Firebase. The end product is shown in the last image of the Overview and Problem Statement section. We conducted two user studies to evaluate the referencing capabilities of our interface in different referencing conditions and found significant improvements in efficiency and effort required compared to discussion forum and anchored discussion interfaces.


One of the biggest constraints in this project is the visual spaces available in the desktop environment. As Korero aims to be an interface that facilitates context switching during learning discussion, there is a limited amount of space available to store and visualize these learning materials that are external to the main learning activity. We worked within this constraint by first understanding how learners discuss online asynchronously. After understanding their behaviors, we employed context-specific progressive disclosure in Korero’s interaction flow and system design to minimize cognitive load and visual spaces.

Lessons Learned

If I can do this project again, I’d have expedited the definition and framing process of the research problem. While it is known that finding the right problem to solve is the most important step in a project (and worth spending time on), in retrospect the amount of time I spent could have been reduced significantly if I had a framework to think through the important elements of a research problem from the beginning. Fortunately, halfway through the project I discovered one such framework from my advisor and applied it immediately to my research. In summary, the framework is followed: a good problem not only has to be novel (from related work) and valid (from pilot studies); it also has to be framed as significant and important. On the other hand, I also learned how to pick the right strategy to present a problem, depending on its nature. As the problem in Korero is related to the process of writing a discussion post, it is pivotal to visualize the complexity of the problem for the persons who are trying to understand the problem. Relying on written report is usually not effective and efficient, as it can be difficult and time-consuming to read. Therefore, it is important to help stakeholders visualize the kinks in the process through effective presentation and communication.

  1. Wikipedia page of MOOCs.