TL;DR: Lean User Testing – How to Run User Tests Successfully
In a world where data-driven decision making is often prevalent, some people feel uncomfortable with agile methodologies as those provide only a few useful metrics. One of those few, however, is the cycle time from idea to shipping a valuable product increment to your customers.
If you want to optimize this metric for your organization, speeding up your product discovery process is essential. And this requires two things: a) rapid prototyping and b) people to test your prototypes with. That’s the main reason why running user tests continuously is so important.
Learn how to best organize and run user tests in this series of six blog posts. Today, we start with answering the “why” question and what huge benefits user tests will provide to your product discovery and delivery process.
The Purpose of User Tests
When testing software – for websites, mobile apps or desktop software – these tests are mainly divided into two types of tests: quantitative and qualitative tests.
Quantitative User Tests
Quantitative user tests are being used to determine the effectiveness of certain actions in a realistic environment with literally scientific methods. One example would be an A/B test to optimize a conversion rate with two different versions of a call-to-action element. (For example a button.) Google is known for excessively examining pretty much everything in the context of quantitative testing. It seems that from time to time the results are a little over the top, though, or as Douglas Bowman puts it: “Goodbye, Google”.
This quantitative method will optimize certain processes very well, such as the registration of new users or the sales funnel from the landing page of a marketing campaign all the way through to the confirmation page of an online sale. The method is based on the evaluation of visitor data and requires a certain amount of participants before the results can be considered statistically relevant.
Conclusion: Quantitative tests provide a picture of what users do, but not why they do it.
Qualitative User Tests
Qualitative user tests, however, are being used to fully understand the usability of an application. Unlike quantitative tests, this kind of test examines the question, why the users react in the way they do. This information cannot be determined in quantitative tests.
Qualitative user tests work very well in practice, as the most urgent problems of a website or application are usually easy to find. If for example four out of five testers ignore or use a feature differently than what it was originally intended for, then this indicates that there is cause for a possible change of this feature. Further testing with a larger quantity of participants is no longer required, as the result is clear – therefore, the marginal utility of each additional test will decrease.
Conclusion: The utilization of do-it-yourself user tests is a valuable and cost-effective means to identify heading in the wrong direction and/or redundant ideas early in the process. User tests enable developers to optimize functions and processes from the start and with that prevent irrelevant applications altogether. Nothing is more expensive in product development than a "we know what we need to build" attitude.
The benefits described making user testing an important tool of all agile methods of product development. During the Lean Startup method, user tests represent a part of the product discovery process and they support the product owner of the Scrum process during the development of the product roadmap – regardless of your company-specific dialect in those agile methods.
Conclusion: Do-it-yourself user tests are an elementary part of the product discovery process because they allow quick and cost-effective testing of hypotheses. Therefore, they are an important asset for any decision regarding any investments towards a valuable, usable and technically functional feature. To be able to push any product decision as far back as economically feasible, running user tests is non-negotiable. And pushing back those decisions is the essence of agile software development.
Even though the B2C e-commerce environment may be easier to handle than the B2B environment, in the following chapters described procedures can also be applied in an adapted form.
When Should You Start User Tests?
From my experience the answer is very simple: Very early in the life cycle of an idea. For successful product management, early validation of hypotheses is the most important step towards product discovery.
Conclusion: A good product organization excels at bringing down the cycle-time of testing hypotheses.
It is a misconception to think that user tests only make sense if during the interview the participants are able to work with a fully functional beta version of the new product. In fact, the opposite is true – even napkins with drawings on them, prototypes made out of paper and click-dummies are rather grateful test objects and are generally well perceived by all user testing participants. This often has something to do with the initial surprise of the more curious interviewees, as they feel that the user testing seems more sophisticated and that they are taken more seriously.
For the first feedback regarding an idea, you don’t even have to organize a user test. All you have to do is approach the person you would like to interview and ask them for 5 minutes of their time. This can happen in the Mecca of Berlin’s startup scene – the ‘St. Oberholz’ on the ‘Rosenthaler Platz’ – or in larger companies you ask for the first feedback from your colleagues. Highly recommended is here, for example, the canteen hacking: During the lunch hour, you join a table and ask the opinion of those sitting there. In my experience, most people love to help and they appreciate it, that they are seriously being asked for their opinion on an existing or new product.
Conclusion: The sooner the user testing begins, the better the results – even if this means, that an idea will be abandoned due to lack of interest. (You just saved money, brain, and time.)
Cannot see the subscription form?
Please click here
What Should Be Tested in User Tests?
The extent of user tests can cover a rather wide spectrum: From just one single feature up to complete applications. Therefore the question as to what should be tested can only be narrowed down. However, there are some general rules that apply to the test range:
- The more complex an application is, the less likely the application as a whole can be tested. Example: eBay – here, singled out processes or features can be tested – but certainly not „eBay“ as a whole.
- On the other hand, a mobile App can often be tested as a complete package, because of its most limited functional range.
- The more precise the test object is, the shorter will be the time for the interview. As an example: When an idea for a new function (feature) is supposed to be tested, this can be achieved with simple click-dummies within a time frame of around 30 minutes. Here the general perception of the user is sufficient whether the feature is useful or not. Only when the feedback is positive and actually leads to a prototype, the user tests in the second testing round should be more detailed in their approach towards improving the usability of the feature in question.
- Interviews over 60 minutes long should be avoided. Many participants have a concentration span of around 30 – 45 minutes. If an interview lasts longer, it’s not only featured by a diminishing return. Also the probability of possible “false positives” increases – where interviewees will say, what they think the interviewer would like to hear to finish the interview quicker.
- The smaller the scale of a test object, the more it makes sense to replace and/or support the testing of individual interviews with crowd testing. For this, the test object will have to be a little more advanced or at a later stage of the product, than it would be required for classic user testing. In a face-to-face user test, the interviewer may interfere with the test, where he/she may adjust the interview questions according to the feedback given by the interviewee. This does not work with crowd testing.
Recommendation: Before you test your own application, you should first test one or two of the main competitors. Why? Somebody has already built a fully functional “prototype”, which addresses a similar problem as your own application. It usually is a great learning experience to receive direct feedback on how your product could be improved with regard to the competition or what the interviewees like about your competitor.
How Often Should You Have User Tests?
User tests should be carried out on a regular basis, which means at least once a month – and that regardless of concrete tasks. It is important that user tests become a critical element of the product development process within your organization.
It is not important under what flag inside the company this will be established. For example, if the label ‘product discovery’ as a collaborative process helps within the Lean Startup methodology to keep the budget and attention of the stakeholders – then so be it. The main thing is that you can run user tests regularly, also referred to as “continuous user testing”.
With Whom Should User Tests Be Organized?
No user tests without users – and the choice of users will be determined by the application that is to be tested. If for example a general usability test is scheduled, pretty much all users – even strangers – are suitable interviewees. These kinds of tests do not require a certain target group.
However, if the test should be carried out for users, who only use their tablets and smartphones because a frontend of a web application shall become responsive – then obviously only these users need to be addressed, to keep the loss of divergence to a minimum.
Should you be looking for interview partners outside of your own user base? Absolutely. Especially as during the early stages of new products this is the only option – as you don’t have any users yet.
Conclusion: No interviewee is capable to identifying all of the usability problems in one user test. On average, a success quote of 50% is considered a good result. You will always need multiple attempts with different interviewees to collect a useful list of results. And there is nothing wrong with trying to identify these at first amongst your own users.
The Costs of User Tests
The initial costs for regular user tests are rather moderate and – based on my calculation – amount to around 10% of the costs of a usability agency in a similar set-up.
If the team is supposed to receive their own equipment exclusively for user tests, then the initial costs for the one-room method, which will be described later, including laptop, beamer, and screen recording software will amount to approximately $1,500. This is particularly useful, as the software set-up can be maintained easier, for example, if a VPN is needed to access staging servers.
The variable costs of the user tests described here are typically around $150 to $200 for room rent, plus $50 for beverages. Where it applicable there may be costs for advertising to acquire interviewees, if you want to use people outside of your own user circles.
The largest expense is usually the compensation of the interviewees for their time.
No external agency for the casting of interviewees was used. (All participants had answered to our own marketing campaigns.) If you cannot cast the interviewees yourself, hiring a casting service will set you back around $150 to $250 per interviewee. (Berlin rates.)
The overall effort for the organization of these user tests will usually take one whole man day to prepare invitations, create the itinerary, test arrangements as well as the evaluation and write the protocols.
On top of that, you need to add the staff costs for the participation of your colleagues–the observers–. (These will also apply, though, if you are using an external agency to run user tests.)
Next Post: Interviewees
See you in part 2 of this six parts series, when we will talk about interviewees: where to find them and how to get them interested in your user tests.
Get the Lean User Testing Manual
This blog-post series is available as a Kindle e-book or as a printed book via Amazon:
Related Posts
7 Best Practices On How To Build A Product Roadmap
No More LEGO® At Agile Workshops – I Am Tired Of Building Airports
Four Lessons Learned From Making Customer Value Your Priority
Part 2: Lean User Tests – How to Find the Right Candidates
Part 3: Lean User Tests – Inviting Interviewees
Four Lessons Learned From Making Customer Value Your Priority
📅 Scrum Training Classes, Workshops, and Events
You can secure your seat for Scrum training classes, workshops, and meetups directly by following the corresponding link in the table below:
See all upcoming classes here.
You can book your seat for the training directly by following the corresponding links to the ticket shop. If the procurement process of your organization requires a different purchasing process, please contact Berlin Product People GmbH directly.