User experience Benchmarking

Arizona State University Mobile App

January 2019 - May 2019

 

What

This past spring I was a UX intern for Arizona State University (ASU )on the Digital innovation team. This team is an internal consulting group made up of designers, researchers, and project managers who work with different teams to craft a future vision for their product.

Contribution

As an intern, I researched, designed, and prototyped a more delightful and straightforward experience for ASU students and faculties on the multiple digital products, especially the focused milestone this year - myASU mobile app.

Leadership

During my time at ASU, I conducted the first usability testing as well as created its framework. I also assisted in conducting more than 10+ studies and workshops for the UX team.

 

Goals

The goal of the project was to formalize user interface design improvements to simplify the web application experience. Usability has been a focus across all of the ASU products and services, however, there is no source of truth on how to conduct it or measure it. Therefore, it is necessary to build a standard procedure from recruiting to conducting as well as a KPI matrix to consolidate the findings.

In this project, our high-level goals were to:

  • Creating a UX assessment tool and benchmark the user experience of the myASU mobile application

  • Conducting the first formal usability testing and compiling the findings from the data collection

  • Present findings and recommendations

  • Build a benchmark for the ASU mobile application

How

As an intern who lacks an in-depth understanding of usability, I started looking into articles and journals to get myself familiar with this subject. I summarized a few popular measures including both behavioral and self-report methods in the industry.

Getting to know more and more, I realized that usability means very different on a product and a task. That means a product can have bad overall usability but with a fantastic performance on some of the tasks (use cases). By starting from this point of view, if I want to benchmark the usability of the MyASU mobile app, a single usability testing is way not enough. Therefore, I included the questions related to the general satisfaction of the app.

The procedure of usability testing is equally important. From recruiting to testing to debriefing ASU didn’t have a comprehensive guideline to follow. With prior knowledge in psychological experimental methods, I built a standard process for future reference which become official.

Benchmarking

This was the first time ASU ever have usability testing. We introduced different quantitative measurements including both behavioral and self-report measures to measure the usability on the specific tasks. We also include the questions regarding overall usability and satisfaction in our measurement.

In sum, we have the following measurements:

Behavioral measures:

  1. Completion time

  2. Task success rate

  3. Error rate

  4. Back-button usage count

  5. Use of search

  6. Navigation process

 Self-report measures:

  1. Task ease 

  2. Confidence to complete the tasks

  3. System usability 

  4. Net promoter score

  5. General satisfaction

  6. SUPR-Qm

What we did

30 minutes one-on-one study session:

  1. Task analysis

  2. Questionnaire

  3. Short interview

Scenario & Tasks

There are three major use cases we wanted to include in this testing, which are:

  • Use the friend function to search and add “Chi-Ping Hsiung” as a friend

  • Explore the present ASU events and add one of the events to the calendar.

  • Confirm the event was added to the calendar and then do “check-in”

To do it, we developed the following scenario:

It’s your first day to be a part of the Sun Devil family. In a class, you met a new friend Chi-Ping and he invited you to hang out together sometime. However, he did not to give you any of his contact information. The only way to connect to him is through the ASU Mobile app. You add him as a friend in the app. Then, you want to know if there are any interesting events happening at ASU recently. You open the ASU app and explore the events, hoping to find one that attracts you. You get one and add it to the calendar on the ASU app. Finally, you double check if the event was added successfully on the ASU app and then go to “check-in” for the event.

PArticipants

11 participants in total:

  • 7 undergraduate & 4 graduate students

  • Our goal is to reach 20 participants to create a benchmark

Requirements:

  1. Need to meet at the least one of the following requirements:

    1. Current student at ASU or currently enrolling at ASU

    2. ASU Alumni

    3. Current employee of ASU

    4. Previous employee of ASU

  2. No recent experience of participating usability testing for ASU digital products

Insights

Task 1 (Add friends):

  • 4 out of 11 participants used the search function to look for the friend’s name

  • 2 participants used “Alumni Resources “ icon to access the friend function

Task 2 (Explore the existing event and add one of them on the calendar):

  • 5 participants click on “more info” to add the event on the calendar

Task 3 (Access to calendar/ schedule):

  • 7 out of 11 participants made errors: 

    • 5 participants skipped the “Schedule” at the first time

    • 3 participants entered into “academic calendar”

    • 3 participants checked Google calendar outside the ASU mobile app 

    • 2 participants clicked the arrow icon to access the calendar

Recommendations

Surfacing Friends

Task 1: Add Friends

  • Re-locate the “Friends” function

  • Prevent the usage of similar icons

Enhancing Calendar & Events

Task 2 (Explore the existing event and add one of them on the calendar):

  • Add a “add-it-to calendar” option function

  • Previous events vs Current & future events

Task 3 (Access to calendar/ schedule):

  • Add an “remove from calendar” function