Join more than 10,000 professionals who subscribe to Food for Agile Thought
and be better informed about what’s happening in agile development!
Our posts by Stefan Wolpers
Stefan — based in Berlin, Germany — has been working for 10+ years as product owner as well as agile coach/Scrum master and is a member of Scrum Alliance (CSPO, CSM). He is also a certified LeSS practitioner (CLP). During that time, he has developed B2C as well as B2B software, mainly for startups, including a former Google subsidiary.
Food for Thought’s issue #101—shared with 10,065 peers—focusses on how to get ‘agile’ to work in your organization. We learn that an old OSS manual on do-it-yourself organizational sabotage still sounds very familiar today. We then get advice on how C-level manager may become more agile, and why we should replace Taylorism’s mindset of command & control with trust.
We also listen to John Cutler explaining his feature factory concept and embrace the importance of mental models to improve the way we learn. Speaking of stepping up the game, you can do so, too, with your next set of A/B tests. Kevin Shanahan provides a step-by-step guide.
Finally, we follow Piyush Tantia when he describes a new way for engineering applications and human interactions: behavioral design.
TL;DR: Lean User Tests: Crowd Testing and What to Do with the Findings
This article of the lean user tests series covers what to do with the test findings and whether crowd testing might be an alternative to running user tests yourself.
What will you Do with the Findings of Your Lean User Tests?
At the end of each user test, you will end up with a list of findings. What will you do with these conclusions?
If you were testing hypotheses as part of your product discovery process, you would now be in a position to decide, which ones to bury and which ones to take to the next level, for example by building a higher fidelity prototype and retest the hypothesis in one of the upcoming user tests.
If you were testing existing applications, Lean User Testing also requires lean “fixing.” Therefore, you should solve those problems that have affected most users in a possibly simplest way. In my opinion, this could very well be an 80% solution. The main thing is that the situation will be improved. And this is important for two reasons:
With regards to the existing product roadmap and bug handling process, companies tend to ignore “newcomers” amongst their backlog of issues or push them on the back burner, as if for the next business value poker. This way those findings from user tests often lose momentum and run the risk getting lost in a big pool of all tasks.
Even if the company does work on the problems found during the user tests, you may find engineering be less cooperative to provide a quick & dirty fix and will insist on doing it right. That way, the solution to a problem might be embraced, thus crushing it.
However, most of the time small improvements are sufficient enough to improve the usability of a site significantly.
Conclusion: I recommend that you focus completely on the two or three most important issues and try to solve them with the least effort. This emphasis will improve the situation for a maximum of users and shows that you do not only make promises but deliver, too. Otherwise, there may be a risk that some stakeholders consider user tests as irrelevant and start either ignoring—or worse—opposing them.
Is Crowd Testing an Alternative to Do-It-Yourself Lean User Tests
Wikipedia explains crowd testing as follows:
“Crowdsourced testing is an emerging trend in software testing which exploits the benefits, effectiveness, and efficiency of crowdsourcing and the cloud platform. It differs from traditional testing methods in that the testing is carried out by a number of different testers from different places, and not by hired consultants and professionals. The software is put to test under diverse realistic platforms which makes it more reliable, cost-effective, fast, and bug-free.[citation needed] In addition, crowdsource testing allows for remote usability testing because specific target groups can be recruited through the crowd.
This method of testing is considered when the software is more user-centric: i.e., software whose success is determined by its user feedback and which has a diverse user space. It is frequently implemented with gaming, mobile applications, when experts who may be difficult to find in one place are required for specific testing, or when the company lacks the resources or time to carry out the testing internally.”
Source: Wikipedia (License: Creative Commons Attribution-ShareAlike 3.0 Unported.)
The service providers distinguish between two main areas of application:
Usability tests and bug hunting. The latter usually receive more attention, because of the search for “hard” errors, in general, is better suited as a business model compared to the “soft” analysis of user experience, usability, and user interface design.
The case where the product to be tested has already progressed quite far or is available online. (The test objective may be the continuous improvement of an existing application.)
The case where a test needs to be held with a very short lead-time, for example over a weekend, to get feedback on a new feature.
When crowd testing is used for bug hunting, it is also an alternative to your quality assurance department, if that doesn’t exist yet due to the size of the engineering department or due to the long release cycles. Some providers now specialize in outsourcing of quality assurance. Instead of exploratory tests, in these cases, it may make sense to focus your resources towards the creation of realistic test scenarios and to then run the actual test via a crowd testing provider.
The following description, however, does not focus bug hunting, but a usability analysis via crowd testing.
How Crowd Testing Works
I recommend everyone to have tried crowd testing at least once yourself before you utilize any crowd testing services.
For this purpose, I registered myself with Testcloud.de and did a usability test for the UK online shop of an outdoor equipment store.
Before the user test, which had to be done within a time frame of two days, I had to install a screen recording software first, because the offered web version was incompatible with the Java setup on my Mac.
To prepare myself, I studied the briefing for the UX test and the standard procedure for the creation of the screencast.
The actual test took around 50 minutes and was easy to follow with the test guide. There were smaller faults during the test like, for example, a provided English postal code from the test guide, which could not be validated on the website. But in general, the test was easy to accomplish. Most of the time required – roughly three hours – was taken up by the conversion of the Camtasia screencasts into an MPEG4 file, as well as uploading the video afterward.
For my time on a Sunday afternoon, I was compensated with €20. In addition to that, I was rewarded with another €10 for the best video in eight participants. When I calculated the compensation for this group of testers, total payments to the testers were less than 10% of the amount, the outdoor equipment store had to pay for the usability test.
Conclusion: A usability test in a crowd can be an alternative to running your own user tests. This does especially apply, when a test is to be held on short notice – like over a weekend. Or when there are no people available in your team to organize the user test by yourself. However, it is my opinion, that any user test you can organize and run yourself is better than running a crowd test. Because you are then in charge to apply any findings in a flexible manner and influence the actual user test in a positive way. And there is no way to do this in a crowd testing scenario: Once the briefing is filed and the test is on its way, all you can do is wait and hope for a good outcome.
Good Luck with Your First Self-Organized Lean User Test!
Practice makes you better, and this applies especially to agile methods of all kinds.
The greatest benefits of the Lean User Testing method described in this email course are that they can be quickly organized and won’t break the bank, to begin with. On top of that, they are useful from the start and can mean real progress for your product.
Or if you are a startup, it can mean reaching the next milestone with the existing funding—also known as extending your runway—, or discover product/market fit.
User tests are a great format for team building in the product and engineering team and to improve the communication with other stakeholders in the company.
User tests are in any case an excellent first attempt at the idea of product discovery and have great potential for the introduction of the agile transformation of a company. Why listen to your gut feeling, when you can ask your real customers on the way to the ideal product yourself?
I, therefore, recommend: Just do it. And if the company won’t allow it, then you should simply consider hacking them. With costs of maybe $200 – $300 it is no longer a question about the budget, but if you want to do it.
Conclusion
This article of the “Lean User Testing” series covers the aftermath of lean user tests and provides insight into the crowd testing option.
See you in part 7, where I will give you all the checklists to make your life of organizing and running user tests easier.
Get the Lean User Tests Manual
The “Lean User Testing” article series is also available as a Kindle e-book or as a printed book via Amazon.
Food for Thought’s issue #100—shared with 9,957 peers—focusses on how to figure out what to build. We learn how to apply continuous product discovery, how to achieve product-market fit, how to use empathy mapping, and why a hypotheses backlog helps to avoid cluttering the product backlog.
We also discover Scrum misconceptions by analyzing job ads for scrum masters, and what cross-functional means with regard qualifications of individual team members. Moreover, we understand ways to improve your Kanban—by Mr. Scrum Jeff Sutherland himself—, and how the principles of the agile manifesto can be matched to change management.
Finally, there is an entertaining interview with Atlassian’s head of R&D on messiness, anarchy, structure, and what all of this has to do with creativity.
Job ads for scrum master or agile coach positions reveal a great insight into an organization’s progress on becoming agile. To gain these, I analyzed more than 50 job ads for scrum master or agile coach positions. Learn more about what makes job ads such a treasure trove with the following 22 scrum master anti-patterns.
Food for Thought’s issue #99—shared with 9,744 peers—deals with hiring scrum masters, why applying the theory of constraints improves the agile mindset, and why a passive-aggressive behavior is bad for business, and not just for your culture. We also learn how to deal with difficult stakeholders, and what the ‘mental model’ fuzz is all about.
On the product side, we understand the power of proper agile roadmaps for collaboration and communication across an organization, and how a successful lean and agile journey looks like. (Courtesy of Telia TV in Sweden.)
Finally, Sebastian Deterding, a research fellow at the Digital Creativity Labs at the University of York, challenges us to consider the moral dimensions of our work as product people in a beautifully narrated presentation from MTP Engage.
The fifth part of the Lean User Tests series deals with how to interview the participants and what the course of the conversation most likely will be.
How are your preparations progressing? If anything is impeding you from reaching this goal, please do not hesitate to contact me by commenting this post, and I will gladly help you.
