TL; DR: Lean User Tests – Inviting Interviewees
This part of the Lean User Tests series focuses on selecting and inviting interviewees: Who is a suitable candidate, how to invite them and why you will need replacement candidates.
You should now be around three to four weeks away from your next—or first—user test. If anything is impeding you from reaching that goal, please do not hesitate to contact me by commenting on this post, and I will gladly help you.
Thoughts on the Right Interviewees
In the B2C scenario mentioned above, we were mostly interested in user segments:
- Multiples buyers: These customers had purchased items on at least two occasions within six months prior to attending the user test.
- Prospective first time buyers: These users were either registered or had subscribed to one or more newsletters, but had not yet purchased anything.
In the case of the multiple buyers, the interview was supposed to determine what would make them buy more. And the second group was particularly interesting, as the company spent considerable amounts of money on acquiring new customers. These customer acquisition investments would only make sense if these new clients actually end up buying multiple times in the future.
Number of interviewees and observers during interviews:
Contrary to the usual set-up—separating the interview room from the observers—, we chose the one-room model:
In a large meeting/conference room the interviewers and interviewees sat on one side of the table, while the person recording the protocol and the observers sat on the other side at the table.
This process proved to be beneficial because on the one hand it requires a lot fewer logistics and is therefore much more cost-efficient. On the other hand, the observers have a greater chance to participate in the communication during the interviews.
The two-room model facilitates this only via indirect communication, which usually causes a time delay and is prone to misunderstandings. Direct participation of the observers, however, requires that they are qualified and especially trained, so they don’t highjack the interview with their interrupting questions.
During these interviews, the maximum number of individuals taking part additionally to the interviewee should not exceed four to five people. This seems to be the most successful combination and so far has not caused any problems during an interview.
The interview circle is therefore made up as follows:
- Interviewer: She leads the interviewee according to a script through the interview.
- The user or interviewee answers the questions of the interviewer and solves tasks that are given to her.
- Recording person: She supports the interviewer during the evaluation of the interview afterwards through the written recording of the interview and the most important answers of the interviewee.
- Observer 1 belongs to the circle of stakeholders, follows the interview and takes notes. She is allowed to ask questions for a better understanding.
- Observer 2: See above.
- Observer 3: See above. (Mostly, a third observer would be considered too much already.)
It has been established that it is pretty much impossible to lead the interview and at the same time record all the feedback. This is the reason why there should always be a recording person solely for this purpose.
It is also recommended that the recording person and the interviewer switch roles during the afternoon if the recording person has been trained accordingly. While the role of the interviewer requires a lot of experience and a good knowledge of the test subject, the role of the recording person–limited to taking notes—can easily be filled by one of the observers.
Identification of suitable interviewees:
The selection of the participants depends on the test subject. If you want to test a new frontend that is compatible with smartphones and tablets, you want to invite those, who use these devices actively on the website.
Otherwise, it is advised to reach as many different interview partners as possible. Relevant criteria are here, for example:
- Gender: Women are more constructive during interviews and communicate better than men. They have a sense of detail and are better at describing, why they do what they do—which is exactly the reason why qualitative tests are run.
- Age
- Previous experience with the product on offer. For example, the number of purchases or contacts with customer services prior to the interview.
- When the last purchase was made.
- Client group, this refers to a specific group of customers, also know as a cohort.
- Providing good reasons why they want to participate in the user test. Examples of this kind are (original answers of real applicants):
- “I want to take part to learn about the improvements and maybe contribute to these improvements. Thus, I may bring in my own wishes and remarks into these improvements.”
- “I find it exciting to test things and help companies, to bring out better products.”
- “The chance to better XYZ is very appealing to me. This may also provide me with the opportunity to actually get in contact with founders. Normally that is only possible via email. So, in short: I would be very excited to help with this new project!”
- “Some products on offer haven’t been particularly good in the past and with better information maybe these things would have turned out better.”
- “I am excited to be able to help with my participation and to experience such a process to better company structures – plus the compensation of course. I would like to play a part in the way the feedback system is being used. I am looking forward to a hopefully positive reply!”.
Cannot see the subscription form?
Please click hereInviting Interviewees
I used to organize the invitations to suitable interviewees with a second Google document. For this, the status, as well as additional information about them, is being aggregated, such as the mobile number.
For every interview slot—as long as enough applicants are available—two similar candidates should be invited. This invitation email will mention the actual location and also the compensation for interviewees’ time and effort for the first time:
Subject line:
Invitation to XYZ user test on Thursday, 27th February 2014, at 3pm (Action required: Confirmation needed by next Sunday)
Body:
Hello Erika,
We are excited that you would like to take part in our user test and would like to invite you for Thursday, 27th February 2014 at 3 pm. The user test will take place on [Street] near [train or bus station] (Please see the link: [Google Maps link]).
The entrance is inside the shop on the right-hand side of the building. There is no access via the courtyard. Please confirm your participation by coming Sunday. We hope you understand, that due to planning security we will invite two applicants to each interview, but we will only be able to accept one applicant for each slot. (Without your confirmation we will not be able to let you take part in the test.)
If you are able to attend, please provide your telephone number, where we can best reach you on the day of the user test from 11 am, should changes occur.
Coffee will be provided and the test will last around 40 minutes. Once the test has successfully been completed, you will be reimbursed for your time and receive a XYZ voucher of € 20.
The invitation text:
- Clearly sets a deadline for confirmation
- Mentions the date and time twice
- Provides a link to Google Maps for the exact location
- Creates urgency by explaining, that at least two applicants are invited for each interview slot
- Describes the itinerary of the day and the compensation for taking part.
The spreadsheet allows for a quick overview concerning the status of the user test: if there are any more open seats, how many participants were invited, how many more confirmations per available slot have been made etc.
Conclusion: I recommend that during the interviewee acquisition campaign all successfully assigned interview slots are immediately removed from the application form, so that new applicants can be steered directly to the remaining available interview slots.
Conclusion: Interviews between 3 and 5 pm are more difficult to assign than those between 2 and 3 pm, as well as from 5pm.
Although you don’t necessarily have to decline all of the unsuitable applicants, it is a good idea to create an open communication with those who were invited. You should therefore always confirm every invitation for participation (Status: “Confirmed”) and at the same time decline the applicant, who turns up late (Status: “Declined by us, 2nd winner”).
It is important at this point to ask for the mobile number of the participants, otherwise you would not be able to remind them the day before the test.
If no applicant confirms their invitation, you should obviously keep inviting other suitable applicants for the interview slot, until it is booked. It is not uncommon, that applicants–despite the fact that the confirmation was labeled “urgent” before–sometimes don’t reply at all or just days later.
An ongoing invitation effort is even more important for 60-minute slots by comparison to 30-minute slots. The risk of organizing an interview session that might fail just doubled.
Conclusion: Even though you send an applicant their invitation to take part shortly after she expressed her interest in participation that does not mean that they will confirm it. Only with a definite confirmation of an interviewee, an interview slot can be considered as booked. Around 10% of all participants will cancel their slot after confirming it, either by email or via telephone, or they simply don’t show up.
By the way, confirmations by phone, which end up on an answering machine, or confirmations to generic email addresses such as info@XYZ.com are also problematic. Probably, those won’t find their way back to you, the organizers of the user test. It is therefore always beneficial to inform the customer care department as well as the receptionists about scheduled user tests.
Invitation of Replacement Interviewees
The invitation of a replacement interviewee for an interview slot in the afternoon makes especially sense for 60-minute slots. While a no-show rate of two applicants for one afternoon of 30-minute slots seems acceptable, this would be a disaster for an afternoon comprising of 60-minute slots.
The replacement interviewees are effectively on standby, should a regular interviewee not show up, and they will receive the same compensation for their service.
Otherwise, you should—as long as you have enough applicants available—also consider inviting more suitable applicants for the replacement position via email. You may find those among the applicants, that clicked on “available anytime”. The invitation email could look as follows:
Subject line:
Invitation as replacement interviewee for the XYZ user test on Thursday, 27th February 2014 (Action required: Confirmation deadline is Sunday)
Body:
Hello Erika,
We are excited that you would like to take part in the XYZ user test and would like to invite you for Thursday, 27th February 2014 as a replacement interviewee.
What is a replacement interviewee?
As a replacement interviewee you will receive a voucher for €20 for the convenience that we may call you on short notice, should one of the other interviewee not show up. If all interviewees attend the test, you won’t have to do anything, and you will receive your €20 voucher later by mail.
We usually know by 12.30pm on the day of the user test, if one of the interviewees won’t be able to make it—which would mean that we would call you to come in.
The user test will take place in [Street] near [train or bus station] (Please see the link: [Google Maps link]). The entrance is inside the shop on the right-hand side of the building. There is no access via the courtyard.
Please confirm by next Sunday, if you can take part as a replacement interviewee. We hope you understand, that due to planning security we will invite two applicants for this position, but we will only be able to accept one applicant. (Without your confirmation, we will not be able to let you take part in the test.)
If you would like to be considered as replacement interviewee, please provide your telephone number, where we can best reach you on the day of the test from 11 am, as well as your postal address.
Conclusion: If your budget can afford it, hire two replacement interviewees.
What Did You Learn Today?
This part of the “Lean User Testing” series covers who are suitable interview candidates, how to invite them, and why you will need replacement candidates.
How are your preparations progressing?
See you in part 4, when we talk about what it is that you need to run your user tests, and where to run them. (Spoiler alert: Preferably not in your office.)
Get the Lean User Tests Manual
This blog-post series is available as a Kindle e-book or as a printed book via Amazon.
Related Posts
Part 1: Lean User Testing – How to Run User Tests Successfully: The ‘Why’ Question
Part 2: Lean User Tests – How to Find the Right Candidates
Four Lessons Learned From Making Customer Value Your Priority
📅 Scrum Training Classes, Workshops, and Events
You can secure your seat for Scrum training classes, workshops, and meetups directly by following the corresponding link in the table below:
See all upcoming classes here.
You can book your seat for the training directly by following the corresponding links to the ticket shop. If the procurement process of your organization requires a different purchasing process, please contact Berlin Product People GmbH directly.