TL; DR: Lean User Tests – Equipment and Location
This fourth part of the Lean User Tests series focuses on equipment and location: What hard- and software do you need to run your user tests, and where to run them? (Spoiler alert: It’s preferably not your office.)
How are your preparations progressing? If anything is impeding you from reaching this goal, please do not hesitate to contact me by commenting on this post, and I will gladly help you.
Where to Run Lean User Tests?
After all the preparations, you need to answer another important question: Where shall these interviews be held?
In an ideal world, you would hold these interviews at the typical location and in the typical situation, your product is normally used. If that sounds a bit like ethnography to you, it actually comes quite close to it. The reason for this approach is to rule out biases as far as possible.
For example, you don’t want to impress—or intimidate—interviewees with your posh corporate office. An invitation might flatter them, and the interviewees might unintentionally express their gratitude by answering your questions in a way they hope will meet your expectations. The problem is, it is not about you, but your product. And even worse, biases of this kind manifest themselves subconsciously.
However, testing your product in its natural habitat might be either not always possible, or it would be too complicated or too expensive. It, therefore, makes sense to hold the interviews at a neutral location that limits the unwanted influence on the candidates.
For this particular reason all user tests of XYZ.com were held at a co-working space in former East Berlin.
Choosing an offsite location for the user interviews also eliminated the chances of potential observers to randomly join an interview. The decision who is taking part in these user tests does require a certain seriousness. In this case, moving the user test location to the co-working space imposed a travel time of around 15 minutes one way. This proved to be an acceptable hurdle for the seriousness of our colleagues.
At the co-working space, the user tests were held in a large conference room during the time from 2 pm to 6 pm. The setup of the equipment started already at 1 pm, as the first interview partners often turned up early—sometimes at 1.30 pm. (As the host, you always need to be at the location first.)
The catering for all participants was organized by the bar of the co-working space. Basically, we provided coffee (in all variants) as well as cold beverages.
Conclusion: An external location should be preferred over your own office to avoid influencing the interviewees unnecessarily.
A note on hospitality: It is not advisable to offer interviewees muffins or any other food up front, as that will only complicate the interview and will draw it out. The interviewees won’t ask for it, as long as it isn’t mentioned in the invitation or the application form.
Cannot see the subscription form?
Please click here
Get the Lean User Tests Manual
/wp-content/uploads/AP-Lean-User-Testing-EN-2015-06-08-2500x4000-188x300.pngThis blog-post series is available as a Kindle e-book or as a printed book via Amazon.
Equipment with Soft- and Hardware
The requirements for the equipment are pretty straightforward. For the execution of a simple test for a normal website you only need a Windows laptop with a mouse, a web camera, and a microphone.
An application—or a release of an application—that hasn’t yet been published can easily be tested on internal staging systems. (Use a VPN, if security concerns are an issue.)
As browser software, I would recommend Firefox, Chrome, and IE. (Safari still tends to irritate Windows user.)
If a mobile application is supposed to be tested, then tablets and/or smartphones have to be provided. Don’t rely on the interviewees to bring their own mobile devices as you will need a standardized environment to compare test results.
Conclusion: Given the popularity of iPads and iPhones, using them in user tests is generally easy. When it comes to website tests, though, I would avoid providing Apple devices to interviewees. While most Apple users can use a browser on a Windows PC, it usually does not work easily the other way around. This effect will likely negatively influence the outcome of the user test.
If screen recording is required, for example, to analyze the click paths of users at a later stage, I would recommend the use of Camtasia.
Tool tip: Camtasia for Mac & Windows.
Note: After approximately ten interview sessions we stopped taping screen recordings. During that time we found out that:
- Nobody actually watched the videos afterward, because compared to reading the protocols the knowledge, that was gained additionally by watching the screen recordings, turned out to be minimal.
- Instead, it made much more sense to ask and prepare questions which were more detailed in advance or…
- …to limit the nature of the interview to fewer subjects, so that the questions won’t have to be that specific.
- Last, but not least, using a screen recording requires an explicit permission by the interviewee. Here, that permit was part of the user test agreement and led in several occasions to lengthy privacy discussions. (Which reduced the available time for the actual interview.)
On top of the devices you will need also a projector with a large resolution so that other participants can watch the screen of the laptop during the interview.
Note: I strongly suggest, that you have a replacement projector lamp with you.
Conclusion: All devices should be tested for their functionality prior to the test. This applies especially to Windows devices. Well-known problems with the equipment are:
- Not yet installed Windows updates. (Which—of course—need to start installing themselves at the beginning of the user test…)
- Expired licenses.
- Empty batteries.
- Defect projector lamps.
- Missing cables. (Such as an iPad or iPhone video cable for the projector.) Alternatively, you can use a screen recording software for the transmission of the video signal. On Mac OSX, this would be a software named “Reflector”, for example.)
User accounts
Be prepared: The interviewees should not be forced to create user accounts for the application to test during the interview unless this is the exact reason for the user test. I always make sure that these accounts have been created in advance, including all relevant email accounts.
It is also a good idea to provide various different payment methods for test purchases. All payment service providers offer dummy cards for this purpose.
Conclusion: If it is not possible to use test payment methods for technical reasons, customer service should be informed in advance that test purchases could be made during the user tests via the live application. Otherwise, it might be possible that the fraud protection mechanism may trigger alerts that will hamper the outcome of user tests. Also, ensure that those fake purchases get canceled afterward immediately, and will not create an additional workload in operations.
Test agreements, Compensation and Interview Itinerary
Before you start the interviews you will have to prepare certain paperwork:
- List of participants: These lists contain some of the application data of the interviewees and serve as background information for all participants, except for the interviewees themselves.
- Two copies of the test agreements for each participant: Before you begin the interviews you and the interviewee will have to set out a test agreement in writing. This agreement defines the lawful relationship between both parties. It will cover in particular the internal use and storage of data, recordings, and findings that will arise during the tests by the issuing company. At the same time, the duty of confidentiality of the interviewee according to the test agreement will be defined. For their efforts, the interviewees will receive compensation, for example, a voucher.
- Compensation: Compensation in form of a voucher brings various benefits and is, in my opinion, the better way, rather than paying out cash amounts. If it’s a voucher issued by the testing company the money will stay “in-house” and the interviewee looks forward to a positive buying experience. If this is not possible and a voucher by a different company—like for example Amazon—will be issued, it will at least satisfy the requirements of the accountants for legitimate documentation. Important is, that the participant will receive some form of monetary compensation. In the case of the XYZ.com, we printed an internal voucher code on a special gift certificate and handed it over like a gift to establish a sense of our appreciation on the side of the interviewee.
- Interview Itinerary: The interview Itinerary shows from the beginning the order of the questions for each interview and corresponds with the list of tasks. This helps with the preparation of the interviewer as well as with the briefing of the observers.
What Did You Learn Today?
This part of the “Lean User Testing” series covered location & equipment—the more organizational, hands-on part of running a user test.
Have you already announced your user test or probably invited the first interviewees?
See you in part 5, when we will talk about the best interview practices, the course of the interview, the follow-up, and the debriefing.
Get the Lean User Tests Manual
/wp-content/uploads/AP-Lean-User-Testing-EN-2015-06-08-2500x4000-188x300.pngThis blog-post series is available as a Kindle e-book or as a printed book via Amazon.
Related Posts
Part 2: Lean User Tests – How to Find the Right Candidates
Part 3: Lean User Tests – Inviting Interviewees
Four Lessons Learned From Making Customer Value Your Priority
📅 Scrum Training Classes, Workshops, and Events
You can secure your seat for Scrum training classes, workshops, and meetups directly by following the corresponding link in the table below:
See all upcoming classes here.
You can book your seat for the training directly by following the corresponding links to the ticket shop. If the procurement process of your organization requires a different purchasing process, please contact Berlin Product People GmbH directly.
