Overview
This project was about gathering qualitative data about users experience opening and setting up a new audio product, and connecting to a Voice Assistant product to try new features. I moderated the sessions for this test, gaining valuable experience in running user tests and understanding users in person.
Role
Test moderator
Research synthesis
Research synthesis
Team
PIC: Allison Burgueno
Project Overview
We wanted to gain some insight into how users unbox, setup, and experience our soundbars and how they would connect it to Alexa to control it with voice.
What's wrong?
Can users setup a 2nd soundbar and connect it with Alexa? Do users know how to set up and use Alexa voice commands?
Solutions?
Advertise Alexa compatibility and functionality clearly on packaging, instructions, and software. Users will never know a certain feature or compatibility unless it is displayed clearly.
Goals & Outcomes
We would like to conduct an OoBE (Out of Box Experience) test to see if participants can set up a soundbar and connect it with Alexa, and understand how users control both the TV and soundbar with Alexa voice commands.
Users & Audience
The target users are those who are interested in quality, hi-res, and surround sound audio. The target users may have home theater solutions such as soundbars, stereo, surround, etc. The targets must also may or may not have smart home solutions, such as Alexa, Ring, Hue, etc.
Scope
Stakeholders would like to understand how users setup their soundbars and connect it to Alexa. We outsourced recruitment and facility to a local agency that specializes in marketing recruitment for test sessions. We were to assess the difficulties found in the setup and use of the soundbar. Additionally, we are limited in design by the engineers of the soundbar, designers of the instructions/box, and Amazon, due to allowing us certain compatibilities and functions that Alexa and our products operations.
Process
Define project scope and plan
Because this has been a design in progress for 2 iterations, the project scope did not change. The plan is to conduct multiple iterations of user testing to continue improving the app design and usability.
Define users
Because this is a "niche" audience (home theater owners with IOT), stakeholders and UX team defined that these types are the ones most likely to purchase our home audio solutions with voice assistant compatibility.
Recruitment and confirmation
Because of how specific the criteria was, we had to outsource to the external agency as we depleted our resources for internal employee and external family/friends database.
Research
We conducted 9 OoBE tests, 7 whom were male, and 2 females. All candidates must own a soundbar and have set up their soundbar themselves. They all had high interest in home theater sound and solutions, as well as own a voice assistant such as Alexa and Google Home.
OoBE Test
We begin by asking about their products they own; to gauge why they like and dislike it, how much they paid and why, and how they researched it and bought it. Additionally, we also gauged their use of their smart devices and voice assistants, such as common commands and their use cases.
They are given a scenario where they purchased the test product, but already currently own a soundbar. They also need to set it up with the voice assistant in the room. With the boxed item never opened, they are to open and setup the product as if they just bought it themselves in their own home.
I would time to time ask about their experiences and thoughts about certain aspects at certain "checkpoints" while the participants freely voice and interact with the product and app. Most of the test is primarily observing the participants actions and comments, to understand their mental and physical processes for setting up a soundbar, and see how they understand connecting a soundbar to a voice assistant means and works.
Most participants did not fully read the setup guide, thus having difficulty understanding how to set up the soundbar and get to the on screen display.
Because they did not get to the on screen display, the participants did not realize that the soundbar had to be connected to the internet to work with Alexa.
Most participants thought that you can connect devices (in this case a soundbar) to Alexa via bluetooth. Because instructions were unclear, participants were connecting the test phone to the soundbar with bluetooth, and tried to troubleshoot connection by using the soundbar app and Alexa app on the phone.
Some participants had trouble using voice commands to control the soundbar. Alexa uses very discrete commands for it to work, but users thought Alexa was smart enough to understand their commands.
We were surprised to find out that setting up the soundbar itself was already a challenge, and when not setup correctly, the rest of the steps were not followed, and correct setup was not achieved. Because of one failure, it threw off the whole flow and left some participants confused and stuck, had I not helped them. Because we did an expert review and went through the complete setup before the test, we already predicted where there may be pain points. Our predictions came through, although we did not anticipate for difficulty of physically setting up the soundbar.
As a moderator, the biggest thing I learned was preparation was key. I was too caught up in trying to remember how to talk to participants, that I did not anticipate how to troubleshoot technical difficulties and how to steer the test. Right off the bat, I wasted a lot of time letting the participant interact with the product freely, and did not know what to do when the test flow started to go awry. However, after seeing the worst, I was able to practice throughout the other sessions and began to anticipate and plan accordingly on my notes, listening to the participant, and observing their actions all at the same time.
Because this has been a design in progress for 2 iterations, the project scope did not change. The plan is to conduct multiple iterations of user testing to continue improving the app design and usability.
Define users
Because this is a "niche" audience (home theater owners with IOT), stakeholders and UX team defined that these types are the ones most likely to purchase our home audio solutions with voice assistant compatibility.
Recruitment and confirmation
Because of how specific the criteria was, we had to outsource to the external agency as we depleted our resources for internal employee and external family/friends database.
Research
We conducted 9 OoBE tests, 7 whom were male, and 2 females. All candidates must own a soundbar and have set up their soundbar themselves. They all had high interest in home theater sound and solutions, as well as own a voice assistant such as Alexa and Google Home.
OoBE Test
We begin by asking about their products they own; to gauge why they like and dislike it, how much they paid and why, and how they researched it and bought it. Additionally, we also gauged their use of their smart devices and voice assistants, such as common commands and their use cases.
They are given a scenario where they purchased the test product, but already currently own a soundbar. They also need to set it up with the voice assistant in the room. With the boxed item never opened, they are to open and setup the product as if they just bought it themselves in their own home.
I would time to time ask about their experiences and thoughts about certain aspects at certain "checkpoints" while the participants freely voice and interact with the product and app. Most of the test is primarily observing the participants actions and comments, to understand their mental and physical processes for setting up a soundbar, and see how they understand connecting a soundbar to a voice assistant means and works.
Findings
All participants could not find anything about Alexa on the packaging, product, or instructions. How is a user supposed to know that their product is Alexa compatible if it isn't advertised?Most participants did not fully read the setup guide, thus having difficulty understanding how to set up the soundbar and get to the on screen display.
Because they did not get to the on screen display, the participants did not realize that the soundbar had to be connected to the internet to work with Alexa.
Most participants thought that you can connect devices (in this case a soundbar) to Alexa via bluetooth. Because instructions were unclear, participants were connecting the test phone to the soundbar with bluetooth, and tried to troubleshoot connection by using the soundbar app and Alexa app on the phone.
Some participants had trouble using voice commands to control the soundbar. Alexa uses very discrete commands for it to work, but users thought Alexa was smart enough to understand their commands.
Conclusion
We were surprised to find out that setting up the soundbar itself was already a challenge, and when not setup correctly, the rest of the steps were not followed, and correct setup was not achieved. Because of one failure, it threw off the whole flow and left some participants confused and stuck, had I not helped them. Because we did an expert review and went through the complete setup before the test, we already predicted where there may be pain points. Our predictions came through, although we did not anticipate for difficulty of physically setting up the soundbar.
Takeaways
When one thing doesn't work, the rest doesn't work either. We as humans tend to go step by step, whether or not we read instructions. So if one step throws us off, it will affect the rest. In this case, almost no one could figure out how to set up Alexa because they truly just didn't know how and nothing on the package or instructions told them how. Being straightforward goes a long way. If the package or instructions clearly stated how or where to go to setup Alexa, then users would not have such a hard time.What I learned
As a moderator, the biggest thing I learned was preparation was key. I was too caught up in trying to remember how to talk to participants, that I did not anticipate how to troubleshoot technical difficulties and how to steer the test. Right off the bat, I wasted a lot of time letting the participant interact with the product freely, and did not know what to do when the test flow started to go awry. However, after seeing the worst, I was able to practice throughout the other sessions and began to anticipate and plan accordingly on my notes, listening to the participant, and observing their actions all at the same time.