When running a user test there are many things to keep in mind to ensure that you walk away with the insights you set out for. We recommend tackling these five areas to design the best user test for your goals:
1. Have a goal in mind, what do you want to learn?
While a “Do you like this: yes or no” might be tempting to lead with, you want to have a clear learning objective with your test. This is not something you might directly ask your participant, either. If you are looking to learn how successful your onboarding flow is, you want to see where in the flow your participants are frustrated, confused or otherwise impeded from continuing. You won’t ask “Were you confused?” but you might ask how they would explain this to a colleague.
2. Identify (and recruit) the right test participants.
Sometimes you have a specific population you need to target, for example, maybe you need to test your product designs with coaches of elite athletes. In such a case it may be wise to source your own participants rather than find them from a user testing panel. Filtering your participants to a target segment, or a “look-alike” segment will get you results that are more authentic and actionable. Vempathy can help you with recruitment when you setup your UX test.
3. Limit the length of the test.
Just as important as having an objective you likely cannot ask a participant for 1 or 2 hours of testing. Limiting your test to five or less tasks can ensure that you have a clear focus and your test participants will complete the tasks you ask of them. IF there are too many, you risk disengaging your participants and this may affect the outcome of your test.
4. Set the stage and give clear instructions for each task, that aren’t leading.
You want to guide participants to navigate the test, but you also don’t want to be overly prescriptive on what to do. You are looking to learn how they use your product, you’re not telling them how to use it. Avoid “click here, then click here” type of task instructions, instead invite participants to accomplish something like “you’re looking to have a physical therapy appointment, find and make an appointment that fits in your schedule.” Instructive but not prescriptive.
5. Test with five to eight participants.
Research has shown that a minimum of five participants will give you sufficient feedback to take action. User experience researcher and professor Bob Virzi concluded that the first four or five participants find 80% of problems in a usability test and additional subjects are less likely to reveal new information. Additionally, he found that the most severe problems are more likely to be detected by the first few participants. The participants need to be of the right profile (see #2) and you won’t need more than seven or eight to start hearing similar points of feedback.
In our continued efforts to bring designers feedback from users within their existing workflows, we are excited to introduce the Vempathy plugin for Sketch. Much like with the Vempathy plugin for Adobe XD, Sketch users will now be able to leverage Vempathy’s advanced, end-to-end qualitative research platform.
With the new Vempathy plugin for Sketch, designers can build better digital experiences with rapid customer feedback and analysis powered by artificial intelligence. They will see exactly when test participants express different emotions while interacting with their designs.
“User research is such a crucial part of any product design process and the emotion reports provided by Vempathy’s tools bring a whole new dimension to your testing. This new Sketch plugin makes it easier than ever to incorporate Vempathy’s powerful insights into your existing workflow” said Daniel Duke, Head of Marketing at Sketch.
“We love empowering designers with research tools that can help them do their jobs better and, as Sketch users ourselves, we are thrilled to bring the power of Vempathy to other Sketch users around the world.” -Paul Cheek, CEO, Vempathy.
To get started with Vempathy and Sketch, please visit:
We’ve been there, everyone seems to have an opinion on your design. “Move the button to the left!” “Make it blue!” and our favorite cliché “Make the logo bigger!” While many designers might want to filter that feedback out, we say: embrace it!
Reactive, Directive, and Critique
There are three types of feedback and some will cause you to roll your eyes. This feedback is often from your colleagues. But what about from your users?
Reactive feedback is where you’d hear an expression such as “That’s awesome!” or “Ugh, that’s was designed by a small blind parakeet.” There’s an emotion here and likely you can identify it not only based on what they express, but also how it is expressed. From your colleagues, this may be a little frustrating, but from your users, this is gold. You need this reactive, expressive feedback from your users and you need to know how to take action based on it.
Directive feedback is often in the form of “If I was designing this, I would…” it offers a specific direction on what you as the designer should do. Sometimes these are in the form of a question “Why don’t you move this to the right?” Regardless of how it’s delivered, you can identify these as a form of direction.
Critique is wonderful feedback, when you can get it. Many people, including some designers, are not trained in giving true critique. Critique is when a the objectives of the design are clarified and for each objective, there’s a comment ho how the design has allowed the user to reach the objective. For example: “If your objective is for the user to begin the checkout process, the choice to place the checkout button in grey on the bottom might not be easily discoverable.” You’ll rarely see this type of feedback from your users as they’re not designers nor are they trained in identifying design objectives.
Capturing Reactive Feedback
When you run a test with Vempathy, you’ll capture and quantify the emotions from your users that can help pinpoint your users reactive feedback and turn it into something that’s actionable.
We know how your users feel. And you’re just a few steps away from knowing yourself! With Vempathy’s sentiment analysis engine, you’ll pinpoint which parts of your product designs create the best or worst reactions from your users.
Start with creating a test for your design.
After entering a name for the test (participants won’t see) and a welcome message (your test participants will see), you’ll add at least one task for your design.
You’ll be able to filter your test participants on a variety of demographic factors.
After a quick final summary check, launch your test! Within 48 hours, you’ll have great feedback on your designs.
Need more help?
Check out our guide for more details on how to get going!
Over the past few months we have prioritized working with industry players to form partnerships that will continue to elevate the design community. Today, we are excited to announce the first of these collaborations that will bring design research into designers’ existing workflows.
We have worked with Adobe to bring Vempathy to Adobe XD. With the new Vempathy plugin for Adobe XD, which is available now, designers get feedback on their designs where it counts. They will build better digital experiences with rapid customer feedback and analysis powered by artificial intelligence. They will see exactly when test participants express different emotions while interacting with their designs.
Once designers launch the Vempathy plugin for Adobe XD they can quickly choose any of their Adobe XD prototypes for their UX test. The Vempathy plugin for Adobe XD will then pre-populate a new UX test in Vempathy, which designers can customize by defining tasks for test participants to complete and filtering test participants to their target user persona. Vempathy then automatically sources test participants, records them, and analyzes their emotions to produce qualitative and quantitative UX test reports.
“Research and user testing is vital to the design process”, says Vijay Vachani, Director of Platform & Partner Ecosystem for Creative Cloud at Adobe, “We are delighted Vempathy is bringing it directly into the design workflow with the Vempathy plugin for Adobe XD. With the new plugin, UX/UI designers get seamless access to Vempathy’s research and user testing platform”
“We are very excited to work with Adobe to bring the power of Vempathy’s design research capabilities to designers who have chosen Adobe’s design software. We are confident that empowering designers with tools like Vempathy to improve their design through research is a win for the entire design community.” said Paul Cheek, CEO, Vempathy.
To get started with Vempathy and Adobe XD, please click here or visit:
At Vempathy we also conduct our own UX research and from our research and based on an extensive research study from Bentley University’s Human Factors in Information Design program we have learned that our customers find difficulty in understanding some of the analysis reports that our software produces. Since we practice what we preach, we have been actively exploring different methods of presenting the data generated by our artificial intelligence algorithms.
Knowing that UX designers and researchers rely on a common set of metrics, we have decided to change our reporting to reflect the simple and well understood nature of the Nielsen Severity Rating metric.
The Nielsen Severity Rating is a measure of the priority of attention that should be given to solve a specific problem. The rating takes into account the frequency, impact, and persistence of the problem to help designers and researchers better prioritize and allocate resources towards fixing it. The scale is very simple in that it spans from zero to four with zero being the lowest severity and four being the highest severity.
In Vempathy, the Nielsen score will be analyzed across test participants based on our emotion analysis and aggregated for each task in the UX test. With this newly surfaced scoring, designers and researchers will be able to quickly identify the task within their UX test that requires the most immediate attention. This functionality is coming soon! Stay tuned…
Getting started with Vempathy is simple and intuitive, but we understand that reviewing and understanding the analysis that Vempathy produces can be difficult for designers and researchers who are not already versed in common emotion metrics. For that reason, we have authored a new getting started guide for our customers. The new Vempathy Getting Started Guide walks you through each and every aspect of using Vempathy.
Whether you need help signing up, designing your first UX test, or analyzing results, the Getting Started Guide has the answers you need. The guide also features 20 screencasts that will show you exactly how to make the best use of Vempathy. Designed with best practices in mind, you are sure to pick up some tips and trick to get the most out of your UX research!
Interested in taking advantage of our Getting Started Guide? You can check it out right here >
Traditionally qualitative research has been just that: qualitative. Vempathy has changed the game for UX designers and researchers by introducing quantitative metrics for qualitative research. And now, with Vempathy’s latest release, designers and researchers can benchmark and track their progress.
With Vempathy’s UX test comparison feature, you can now compare Vempathy’s reports from one UX test to another. This allows you to run a test on your UX today, see the results, act upon them, and then run a follow up test to see if the change you made corrected the usability issues. In the new interface, you will find all of Vempathy’s reported metrics in one graph that show you the user journey and how it has changed from test to test.
This feature is live now and we welcome your feedback! Get started by making a copy of a UX test you have run in the past and it will automatically populate in the comparison dropdown on the reports page.
We have come to realize the importance of fine grained specifications when recruiting participants for your UX testing. While our free form text field to recruit participants works, it simply is not intuitive and that’s why we are happy to introduce improved recruiting today!
Now, when you design a new UX test with Vempathy you can easily specify demographics such as age, ethnicity, gender, income, employment status, and more! Further, you can create your own custom screener survey to more finely filter the participants completing your UX tests.
This new release is live now and we welcome you to give it a try and let us know what you think!
We know that getting started using a new UX research platform can be daunting, which is why we are so excited to announce the roll out of our new project setup. With our new project setup, Vempathy will guide you through all of the steps necessary to start your first project including study design, participant recruitment, and launch. Try it today and let us know if you have any questions!