Lean User Tests, Part 5 of 7: Best Interview Practices and the Course of the Interview

TL;DR—Lean User Tests Part 5

The fifth part of the Lean User Tests series deals with how to interview the participants and what the course of the conversation most likely will be.

How are your preparations progressing? If anything is impeding you from reaching this goal, please do not hesitate to contact me by commenting on this post, and I will gladly help you.

Successful Interview Practices for Lean User Tests

The interviewer should let the interviewee talk—meaning, he or she should entice the participant to articulate their thoughts. By all means, it should be avoided to suggest answers or ideas. Suggesting those would lead to false positives, as the interviewee might be tempted to say what he/she thinks the interviewer would like to hear. However, this is not the purpose of the interviews.

Best Practices on how to Run Interviews for Lean User Tests

Interviews: What to practice during lean user tests

• Don‘t interrogate: It’s supposed to be a conversation, and the interviewer is the listener. Encourage the interviewees to tell you stories, particularly about things you cannot observe.
• Encourage loud thinking: You need to get your interviewee to share their thoughts during them solving their tasks. A lot of people are not very good at this, as they are not used to speaking out loud what they are thinking.
• Ask naïve questions: It is therefore always good to ask the interviewees why they are doing things in a specific way while they solve a task. And don’t worry if your questions sound naïve to the observers – you can avoid making assumptions this way that the interviewees cannot know about. It will also make the respondents feel like the experts during the conversation.
• Stay neutral: You should not try to influence anything, but stay neutral. You should never tell the interviewee what to do. That they will have to figure out themselves.
• Show, don’t tell: While we like to listen to stories when they cover things we cannot observe, we also like to be shown things we can note, for example, have the interviewee walk you through all the steps of a process.

Interviews: What to avoid during lean user tests

• Leading questions: “If we provide this feature, would you use it?” – That’s a classic and an entirely useless question. Instead, always wait for the interviewee to come up with that. If they don’t do so, the feature apparently does not have priority for them.
• Asking for solutions: As interviewers, we are interested in the problems of the interviewees, so we can help solve them. Discussing solutions with them instead is most of the time not helpful as respondents lack the insight knowledge.
• Don’t solve the respondent’s problem: It happens regularly in interviews: The interviewee cannot solve a task. Observing the struggle, some interviewers feel the need to help the interviewee to accomplish the task. Wrong idea. By helping them, you probably miss important details why they are struggling in the first place, which might affect your ability to come up with a solution in a negative way.

Some questions for an easy start of lean user tests

A good way to ease into an interview would, for example, be the following questions:

• What are you doing professionally?
• How often do you use the Internet?
• Are you buying things on the Internet?
• Have you ever bought anything from XYZ.com?
• What are you (usually) buying from XYZ.com?

Conclusions

• Some interviewees like to show off a little and tend to drift off. These are mostly interview partners working in sales or field services. These should be reminded – softly but firm – to get back to the interview at hand.
• Any disturbances are—especially during 30 minutes interviews—rather annoying and disruptive because there simply is not enough time. Hence the strict policy on checking emails etc.
• Interviewees liked to talk about good and also bad experiences with XYZ.com. Some like to use the interview as some form of escalation to customer service. We didn’t usually check before the interview if there had been differences of opinions between the interviewee and XYZ.com. It is advised to keep that in mind when asking questions about bad experiences.

Cannot see the subscription form?
Please click here

Lean User Tests: The Course of Interviews

The Structure of the Interviews

The interviews are made up of repeating phases – regardless of the test subject:

• Greeting and an introduction of everyone present (5 minutes): In this phase, the participants will have time to take off their coats (takes longer in winter and when it is raining outside) and allows for a general greeting and settling in. After that everybody should just briefly introduce herself.
• Signing the test agreement (5 minutes): Here the test agreement should be explained and then be signed. Most interviewees will take this very relaxed, but some may read the entire document from start to finish. This is a good time to let someone in the group (the recording person or an observer) fetch the coffee and other beverages for the interviewee. So far, nobody has ever refused to sign the agreement. Should this occur, the interview needs to be canceled.
• Handing over the gift certificate with the voucher (1 minute): By handing over the compensation early, the situation is immediately more relaxed, and it creates trust, as practically all interviewees anticipate to receive the payment after the interview. Giving the voucher early may also please those of the interviewees, whose interest is mainly receiving the compensation. It helps to eliminate “false positives,” as it no longer matters if their answers are the ones the interviewer may want hear – because they received their compensation already.
• List with tasks (2-3 minutes): If certain processes have to be tested with a detailed set of tasks, it makes sense to present a list of these tasks to the interviewee. This summary should be explained before beginning the interview so that the respondent is fully informed and knows what to do.
• The actual interview: Lost time can hardly ever be recovered during tightly packed interview slots. Therefore the interviewer is also the timekeeper and has to make sure that no time is wasted or lost. The net interview time is as follows: For a 30 minutes interview it is 12–15 minutes, and for a 60 minutes interview it is 30–35 minutes.
• Summary and finish (5 minutes): At the end of the interview, the interviewer will summarize the most prominent findings. (Also as a feedback for the recording person.) You should all thank the participant for his or her time and effort and say goodbye. During the winter the part of getting dressed may take a little longer. It should be avoided to make the interviewee feel like she is ushered out, just because the next interviewee is already waiting outside.
• Preparation for the next interview (5-10 minutes): This part is also important. You will have to delete information from the previous interview session, e.g. cookies, stop the screen recorder and save the file, collect the recorded protocols, getting the paperwork for the next participant like the test agreement, voucher, and task list ready, as well as clean the table.

The interviewees of Lean User Tests

These are the most significant findings concerning interviewees from hundreds of interviews:

• Early-show: They like to turn up earlier to their appointment than planned. In this case, you might as well start with the interview earlier. An excellent team of interviewers might even be capable of holding two interviews at the same time.
• Late-show: They are late. For 30 minutes interviews, this is putting quite some stress on the schedule, unless the following interview can be pushed out a little or you can have two interviews parallel with a second interviewer and recording person.
• No-show: They just don’t show up at and don’t cancel. (Status: “No-show” in the Google doc.) This is very unfortunate. I usually don’t waste any more time in trying to find out the reasons, and apparently, this kind of behavior will not be rewarded with a voucher.
• Last minute cancellation: They cancel minutes before the interview so that the replacement interviewee cannot be activated. We usually inform the replacement interviewees around 12.30 pm, and that often works out well. If we, however, only learn about the cancellation by 3 pm, the chances of the replacement interviewee being able to jump in are rather slim—but it’s always worth a try.
• And you are? They show up despite having been declined via email. (Status: “Declined by us, 2nd winner” in the Google doc.) In this case, you either interview both or you send the second winner away. (In any case, we were generous and awarded the compensation for showing up.)
• Limbo: They are not available on their telephone during the morning hours, despite the prior agreement. This puts you in the position, where you don’t know, if this may be a no-show or if they simply forgot about the phone call and will turn up to the interview on time. In this case, I normally don’t activate the replacement interviewees.
• Kids & dogs: They do not come alone, but bring children (mostly babies) or friends, relatives, and partners (spouses). We even had dogs in the interviews. In the end, these companions didn’t have any influence on the success of the interviews.

Conclusions

• It is important to call every interviewee around 12 pm, to remind them of the interview in the afternoon. Some interviewees will forget their appointment. Any calls before 12 pm are not very successful. If someone is not available, a message should be left on the voicemail inbox, and I recommend sending a text message also.
• The way to the interview room should be signposted. These signs should be printed and put up. (Don’t forget to take scissors and tape if these are hard to come by at the venue.) This is particularly important in off-site venues that are not labeled with your company’s logo. Also, the reception of that off-site venue has to be informed so that they can assist the interviewees.
• You should not offer any food to the respondents. This will drag the interview out and makes it less successful. However, a post-interview-muffin is fine.
• Observers should be allowed to ask relevant questions, but the interviewer should always have the lead over the interview. If the observer would like to ask questions, they should be cleared before the interview if possible. If this weren't the case, the interviewer would have to intervene.
• Last, but not least: During the interviews, nobody should be allowed to use any laptops, smartphones or tablets. Everybody shall listen to the interview and take notes, but emails, messages or texts should be checked elsewhere. It would be considered disrespectful towards the interviewee.

Follow-up and Debriefing of Lean User Tests

The Follow-up

For the follow-up, the notes of the recording participants will be summarized into a form for every single interview. For this purpose, standard templates in the form of bullet-point lists have proven to be successful, because they enable a quick orientation over the core statements.

This structure contains, for example:

• The name of the interviewee
• Age
• Gender
• Motivation for taking part in the user test
• Client history
• A quote that summarizes this interviewee’s experience with the application
• A description of how the application was used.

Conclusion

Every recording person has their system as to how they record their notes. I recommend the use of handwriting on paper, as this doesn’t interfere as much as the typing on a computer might with the interview, and it also captures the information faster.

Also, it is useful to aggregate a series of interviews on a particular topic once more – think of it as an executive summary – and highlight the most significant findings, since recommendations on next steps can be derived directly from them. Use this executive summary as a teaser to distribute your findings to a larger audience within your organization.

Last, but not least will the follow-up also include the notes of the observers of other departments. Each of them should have identified the three most important issues for each interviewee, regardless of other interviewees reporting the same issues.

The Debriefing

The debriefing for the observers comes after the user test, and the interview protocols will be created in due time.

The meeting for the debriefing should not take longer than 30-45 minutes. The debriefing meeting will not only reflect the general findings of the tests, but it should also lead to an answer to the question: “What can be done or should be fixed for the next user test?”

The protocols should be made available to all prospective participants before the next user test and should become a part of the agenda of the upcoming briefing meeting as a requirement for participation.

What Did You Learn Today?

This part of the “Lean User Testing” series covered the “action” of a user test: How to interview your interviewees, what you can expect from them and the proper closing of the user session with a follow-up and debriefing.

How many interviewees have you signed-up? Are some interview slots still available?

See you in part 6, when we talk about what to do with the findings and whether crowd testing might be an alternative.

Get the Lean User Tests Manual

The “Lean User Testing” article series is also available as a Kindle e-book or as a printed book via Amazon.

Related Posts

Part 3: Lean User Tests – Inviting Interviewees

Lean User Tests – Equipment and Location (Part 4 of 7)

Four Lessons Learned From Making Customer Value Your Priority

📅 Scrum Training Classes, Workshops, and Events

You can secure your seat for Scrum training classes, workshops, and meetups directly by following the corresponding link in the table below:

See all upcoming classes here.

You can book your seat for the training directly by following the corresponding links to the ticket shop. If the procurement process of your organization requires a different purchasing process, please contact Berlin Product People GmbH directly.