Usability testing is one of the best ways to ensure your users are having a positive experience every time they use your product. Conducting these tests, however, can come with some serious questions. The first key question that needs to be answered is, should I conduct a moderated usability test or an unmoderated usability test? While each has its advantages, in this article we will focus on defining moderated testing and outline each step in conducting a successful test.
What is moderated usability testing?
Moderated usability testing definition
Moderated testing is a usability testing method that integrates an industry expert to moderate the usability test. Their presence usually provides a more in-depth analysis of the test and its outcomes.
The differentiating requirement for moderated usability testing is the presence of an expert. This expert moderator is responsible for facilitating the test for the user, including their interactions, questions, tasks, comments, and feedback. All of these interactions happen in real-time, which is why the moderator has to be a product expert rather than just a bystander. Their expertise directly affects how the user experiences the product as a whole, and can often determine the quality and efficacy of their experience.
Traditionally, moderated testing is done in an in-person environment, but thanks to online tools it can also be conducted remotely. When being done remotely, the moderator will require access to the user’s screen and facilities to conduct the test in real-time. Aside from engaging visually with the user, this also provides the moderator with an increased ability to understand how the user thinks and how they transition from point A to point B.
Seeing the user’s screen in real-time allows for moderators to directly observe the interactions beyond the buttons and end goals. So much of a user’s experience exists outside of the tangible actions they create. This could be the features that draw their eye, the places they hover, or the elements that dissuade them. These are the things that might get lost during an unmoderated test and should be kept in mind when deciding on a testing strategy.
Moderated testing advantages & abilities
There are many reasons to conduct usability tests within a product/service that will serve as improvement markers for that product. Beyond simply conducting tests, there are many advantages to using a moderated testing methodology, some of which we will detail here.
Remote tests are cheaper
It’s no secret that moderated testing can be quite expensive when implemented at scale, and we will go into this later. New technology, however, grants the possibility of remote testing.
Remote moderated testing is a great way to conduct moderated tests at a fraction of the price. These tools make moderated testing much easier to conduct and more accessible to a wide range of designers.
Remote moderated testing allows for the presence of a product expert to facilitate and comprehend the test while also skating beside the largest associated costs. If moderated testing wasn’t a feasible option for your product, consider looking into remote services.
Allows for direct visualization
Moderated testing is the best way to gain a first-hand experience of how users test the product. If you are interested in learning more about how people interact and view the product at hand, this is the best method for people to get as in-depth of an understanding as possible.
The visualization by having an expert act as a first-hand, real-time observer means that you can gather the user’s experience that would go unnoticed without their direct observation. This increased understanding and interactivity are some of the main reasons people opt for moderated testing.
Allows for increased interaction
One of the main reasons people choose to use moderated testing instead of unmoderated testing is because they need to have an expert interact with the user during the test.
Having a guide to walk users through their experience means they won’t get hung up on small flow details or areas that are not of interest to the designers and will be able to focus only on the elements that require feedback. These interactions between the moderator and the user make the usability test much more efficient and ensure you will receive directed positive feedback.
Moderated testing disadvantages
There is no right or wrong method to conduct usability testing, and whatever you choose is purely based on the types of testing that you are looking to conduct. That being said, some downsides come with moderated testing as well. Here are some of the downsides.
Price
As briefly mentioned before, conducting a moderated usability test can get extremely expensive. If it is a test with only a handful of participants it should be fine, but when this test is scaled up to tens, if not hundreds of users, the ability to utilize a moderator becomes much more expensive. While this downside doesn’t hold any bearing on the quality of the tests being conducted, it’s an element that must be discussed.
Not as scalable
It can be very hard to implement moderated testing on a large scale because of how in-depth it is. Each moderated test is fairly detailed and requires a dialogue between user and moderator, while also taking time to conduct specific tasks. This level of detail makes it much more difficult to scale tests to a large audience because each one needs a certain amount of attention and oversight.
Conducting unmoderated testing with large amounts of people is much easier because there is no live audience. When people can just log on, do their test, write their takeaways, and log off, they are the only ones committing time to the testing process. This makes it far easier to scale unmoderated usability tests.
This isn’t inherently bad — for some test cases you want fewer participants and more detail, but if you are wanting to conduct a test scenario with a large group of participants it can be quite difficult.
Testing speed
Similar to the difficulties that arise when not being able to scale a test to a large audience, doing a moderated test can sometimes hinder the ability of the user to provide swift and accurate feedback. There are two reasons for this.
The first reason isn’t universally applicable, but when in the presence of a moderator people can sometimes feel pressured to not be as critical as they would be on their own, and provide feedback that is only a partial truth. This can sometimes be circumvented by having an outside expert who is unrelated to the product, but still might hinder people’s ability to give honest feedback.
The second reason is that when you have someone walking you through a test, the test as a whole will usually take much longer than if someone was conducting it themselves. This doesn’t mean that they are cutting corners themselves, but that they prioritize taking time on the sections that count. This means that you can conduct unmoderated tests at a more rapid pace compared to a slower more deliberate system of moderated testing.
How to conduct moderated usability testing
After you gain a brief understanding of the pros and cons of moderated usability testing, it’s time to learn about how to implement it. Here are some steps and best practices to follow to conduct an effective moderated usability test.
Defining goals
The first step in facilitating a successful moderated usability test is defining the goals of the test, what the test will consist of, and how to gauge the feedback. Not only should your planning consist of the metrics of the test, but also the participants.
It’s important to target specific groups of participants before beginning the test to guarantee their ability to engage with the material properly. Selecting a targeted research group is a great way to narrow the testing process and focus the scope on the specific area you are looking to gain feedback in.
Aside from the participants, this step is all about organization and logistics. Figuring out where the test will take place, how it will be accessed, the steps the users will go through, etc. These standards are important for the design team to understand how the test will be conducted and also provide an assessment metric for the tests to be understood against.
(Fresco)
Find participants
This step is all about finding individuals that meet the requirements previously listed and having them participate in the testing process. Finding random people to participate in a usability test is very easy, but finding people that fit the exact mold of your target customer can often be a grueling and intensive process.
There are multiple different ways to recruit participants, and for each organization, they might go about recruitment differently. For some, you might already have a stream of testers that fit your mold and can be used repeatedly. For others, you might go out and conduct surveys of potential testers to see if they would want to participate. A third option would be to hire a recruiting company to find participants for you. This way you can simply hand off your user bio and allow people to come to you.
Whichever method you choose to find participants for your testing process, the most important factor is that they match the persona you created for the principal user.
Prepare questions
This step can be done at any point in the preparation process as long as your questions are prepared before the test begins. Typically, there are three phases of questioning during a moderated usability test.
- Pre-test
- Pre-test questions should center around the participants’ expectations for the product, their experience with similar products, and even questions about them more generally.
- Physical Tasks
- These questions will be all about the actionable tasks that the user takes while getting from point A to point B. These tasks and questions will vary greatly depending on the product that is being tested but should all focus on the actions taken by the user during the test.
- Reflection
- The reflection portion is where you begin to ask about how the user felt about their tasks. Have them describe their overall experience or their opinions on certain important features.
These questions will be important to understand every phase of the test from the user’s perspective and gain a clear vision of their interpretation of the product.
Conduct the test & analyze results
The last step is the most important, and it is where the actual test is conducted. During the test, two of the most important things to do, as a moderator, are to guide the user without influencing their opinions and take note of every small action and motivation. These will be key to gaining results that are as accurate as possible. Don’t act partial to any part of the product, simply allow them to experience it organically.
After completing the test, it will be time to analyze the results. This is the phase where your standards and metrics from the beginning will be used to interpret the actions, motivations, and opinions of the user. Try your best to keep your results straightforward and clear, and relate them to the initial goals you created for the testing period. If this is done, it should be easy to understand and implement the results.
Conclusion
Conducting usability tests is essential to optimize the experience your customers receive when interacting with your product/service, and moderated testing is one of the best ways to achieve this. If you are interested in learning more about how visual collaboration and online whiteboards can integrate with user experience, check out Fresco.
This post was contributed by Paul VanZandt at Fresco — Fresco is a startup focused on providing unique visual collaboration solutions and expanding the potential of virtual whiteboards.
Tim Rotolo 
Shruti Goli 
Ritvij Gautam