Low costs, manageable effort, helpful results Guerrilla usability & user experience testing

"Of course, user testing would be good, but we don't have the money or time." You still hear this or similar sentences frequently, even in companies convinced of the advantages of human-centered designs. Still today, companies often shy away from the high costs of user tests. Thus, they make critical strategic decisions based on assumptions alone. However, this doesn't have to be the case: after all, there is guerrilla UX testing. Guerrilla UX tests are user tests with a manageable effort and low budget, but provide helpful insights into the user experience and usability. The usability expert Jakob Nielsen coined the term "Discount Usability Testing" in 1989. Today, however, the focus is not only on usability but on the holistic user experience. Hence, the term "guerrilla UX testing" has become established in recent years.

Discount usability often gives better results than deluxe usability because its methods drive an emphasis on early and rapid iteration with frequent user input.
Jakob Nielsen

The trick of guerrilla UX testing is that they reduce the effort involved in selecting test users, performing the tests, and analyzing the results. The ultimate goal is to repeat guerrilla UX tests frequently without having to forego valid results. Guerrilla UX testing is suitable for products or services that do not require in-depth specialist knowledge and address a broad target group. Typical examples are apps for public transport or navigation systems in the car.

Of course, standard user tests in laboratories and strict scientific procedures are sometimes needed. For example, if the target group is difficult to find in public places or if there should be hardly any external influences, guerrilla UX tests are probably not the best choice. In addition, guerrilla UX tests do not allow for generalizable statements or elaborate test procedures because the conditions are not controlled enough. However, especially in applied settings, many companies do not need these. Above all, they want to improve their products or services, and the only alternative is not to test at all. In these cases, guerrilla UX tests are exactly the right thing!

Understand your audience

Unlike in standard user tests, you do not recruit test users in advance when preparing a guerrilla UX test. Instead, you simply approach individual people or use conspicuous clothing, signs, accessories, or the like to make people curious and willing to participate. This approach reduces the recruitment effort enormously and enables a larger number of tests to be carried out within a short time.

The big challenge here is to identify people from the targeted user groups. While you can search for these people when recruiting, guerrilla UX testing relies on a few questions during short conversations before the tests start. The more specialized the target user group is (such as experts or managers), the more difficult they are to find for guerrilla UX tests. The problem can be mitigated by choosing the right location: a fitness studio, the pedestrian zone, an event, or a network meeting, depending on where you have the best chances of finding members of your target user groups. Places where many people spend time in a relaxed atmosphere are recommended.

At the beginning of the interview, you should first present the study and ask in writing for the users' consent to record it or to document their experience in some other way (more on documentation below). It is crucial that the participants know what they can expect and what the study is all about. Above all, it is essential that they understand that you are testing the product or service, not them as a person. Furthermore, you should maintain ethical standards in guerrilla UX testing. The participants must always have the option of not doing tasks or even aborting the test prematurely. In addition, the participants should also receive a small gift for time and effort.

15 minutes are enough

Guerrilla UX tests are usually designed to uncover weaknesses and show potential for improvement (so-called formative tests). To do this, you should first prepare the material for testing. Depending on your interest and level of development, you can use different materials, as the table below illustrates. Guerrilla UX testing is also particularly suitable for initial paper prototypes and drafts due to its loose character.

After that, the planning of the sessions begins. While usability studies in the lab tend to take 45 minutes or longer, guerrilla UX tests should not take more than 10 to 15 minutes. Thus, you should instead concentrate on a few areas and simply test again and again. In most cases, the participant will be required to solve a number of defined tasks. The tasks should, of course, reflect the product and match the aspects that you want to investigate. The tasks should be described briefly and simply. They should not be formulated suggestively, for example if they refer to a specific solution strategy. In the beginning, the participants are often given an open task through which they can move freely on the start page and get an overview. This first task shows how visitors orientate themselves and whether they understand what a product or service is all about. Finally, participants should be asked to explain what is going through their minds as they solve the tasks: The "thinking aloud" method is an essential and very informative part of user tests.

A guerrilla UX test also includes a final discussion, the so-called debriefing. During the test, you often notice things that you want to talk about again at the end. It is also worthwhile to react to the observed emotions and thoughts of the participants: It is not uncommon that they contain valuable information to interpret the events. Participants should also have the opportunity to get answers to their questions in the debriefing. For example, if a task has confused them, now would be the right time to resolve the confusion. You should then point out that the company will improve the product or service in this regard, thanking your participant for uncovering confusing aspects - according to the motto "users are always right".

Table: Material and objectives in guerrilla UX testing

Objective	Description	Materials
explore ideas	test first solution ideas, find basic problems	mockups, drafts
validate ideas	simulate specific, typical actions	functional prototypes
validate products & services	test final version, quality control, check achievement of objectives	finished or nearly finished version
compare products & services	compare several possible solutions (such as different ideas or competitors)	Mockups, drafts, functional prototypes or finished versions
enhance products & services	continuous improvements, adjustments to market changes	market-ready product

Possible objectives and materials for user experience tests with users (according to Carine Lallemand, "Méthodes de design UX", 2nd edition, 2018)

How many participants do we need?

Whenever you think about testing, an obvious question is: How many tests do I actually need? The tendency is to test until you get the impression that you have observed most aspects. Thus, a good indication that you have tested enough is when you hardly notice anything new several times in a row with identical tests. Of course, there are also some rules of thumb, such as Jakob Nielsen's famous recommendation that tests with five users are sufficient to find 85% of usability problems. However, the number depends very much on what kind of product or service you are actually testing. For example, some studies have found that websites, in particular, have a much lower detection rate of problems. If you want to know more about calculating a sample size based, you can find the necessary details in the book "Quantifying the User Experience" by Jeff Sauro and James Lewis.

Basically, however, the number of participants in guerrilla UX testing is not of central interest. First of all, a test with few or even only one user is better than no test at all. Experience shows that you always find a user experience problem that you didn't know about. Especially when testing business-critical areas, guerrilla UX testing pays off quite quickly.

In addition, it is particularly important to plan the number of participants as precisely as possible if there is only one study of user experience. However, guerrilla UX testing wants to reduce the effort of individual tests so much that you can test again and again. In the business environment, user tests are not intended to uncover as many weaknesses as possible, but to improve a product. Therefore, it is not the number of problems found in one go that is decisive, but rather that some, probably even the most common, are discovered. You can then work on solutions and test further testing. Over time, you will find more and more weak points. Therefore, it is more important to develop a test-driven culture close to the experiences of users. In other words, it is better to have several guerrilla UX test series with few users than a single test with many users, provided, of course, that you really work iteratively on solutions in between.

Observe and analyze

Whether guerrilla or not: user tests only make sense if a company works with the data. Thus, after the test, you should take up the most important aspects, create new solutions, and test them again. But how do you collect the results so that you can analyze them? In the laboratory, you would record the individual test sessions - usually with a screen recording and a video recording of the face and the statements made when "thinking aloud". If the participant agrees, you can, of course, do likewise when guerrilla UX testing. Tools such as the Mac application Silverback (39 US dollars) or screen-capturing and smartphone camera help with this.

However, observations during the tests are just as important. It is best to think about what is of particular interest in advance and create a corresponding observation sheet (see the example below). Such a template gathers fields for all interesting aspects on a single page: for example, where a participant clicks, which problems arise, which emotions a user shows or individual statements. The observation sheet should cover as many details as possible, but should not be confusing or difficult to fill in. An observation sheet has several advantages: It is suitable as a memory aid for the debriefing and helps to concentrate on the essentials during the analysis. With a bit of practice, you can even do a part of the analysis during the test. It is especially helpful when two observers are doing the test so that one of you can conduct the interview while the other fills in the sheet.

Making sense of guerrilla UX tests

The evaluation of a guerrilla UX test is intended to identify areas where improvements are particularly useful. Typically these are user experience problems. However, they could also be positive aspects that a company can further develop. In order to find these, you should take note of any potentially important events, for example expressions of "thinking aloud", tangible problems in interaction, or emotional reactions. These events, known as "critical incidents", are summarized by the observers into user experience problems after the sessions are over. A problem often manifests itself in various symptoms. It is also useful to record how often a problem has occurred and how serious it was.

In most cases, your list of observations will end up with more problems than one company can tackle at a time. It is, therefore, important to find out where the most promising potential for improvement lies. To do this, the team members come together and discuss their observations. Then they write all problems on sticky notes and arrange them in a goal matrix with the two axes "importance" and "feasibility". In doing so, the matrix quickly shows which problems are the most relevant, for example because they are easy to solve and particularly important.

Conclusion

Guerrilla UX testing can lead to valuable insights into the user experience. When used correctly, this cost-effective method enables companies to quickly and noticeably improve their products or services. But good user experience design involves much more than just testing. Therefore, guerrilla UX testing is not the only way to do something for good user experience. Read on in the next post on guerrilla UX design.

---

Note: This article is a translation of a version published in the German technology magazine t3n, issue 55, in April 2019, which is available as a German version online.