Dear Game Developers:

As a player, have you ever been so frustrated with a game's controls that you've quit it? Have you ever found a tutorial so unclear that it's more efficient to watch Twitch streamers than to go through the process yourself? Have you ever downloaded a game mod to increase font sizes or fundamentally alter the UI?

If so, you're not alone. And if not, you're lucky. Many products -- games, apps, and websites alike -- contain elements that make them hard to use or understand. Anything that makes a product harder to use is a "usability issue."

Usability issues lurk in many products, but while larger companies may have the resources to hire professional user researchers, smaller companies may not be able to do so. But that doesn't mean you need to go without usability testing -- there are multiple methods and tests that a single developer can conduct.

There are two types of user research to identify usability problems: qualitative research and quantitative research.

What Is Qualitative Research? To break it down to its simplest part, qualitative research answers "why." For example, what do players think of my tutorial, do they like it? Dislike it? Why?  It provides insights into where players are struggling and how one might address or fix those issues. 

What Is Quantitative Research? Very simply, quantitative research answers "how." A common approach to quantitative research is A/B testing, which pits designs against each other.  A research question for an A/B test could be:  "How much faster, if at all, do players complete a tutorial if I place the tooltips on the left versus the right?" I'll go over A/B testing, and other quantitative tests, in "Guerrilla Usability For Game Developers (Part 2)."

Guerrilla Qualitative Research 

Qualitative research allows you to listen and understand what players think about your game, and why they have that opinion. Professionally, there are many types of qualitative research tests that a researcher can use (1:1 lab studies, card sorting, diary studies, and more), but to keep your research cheap, you'll want to focus on 1:1 in-depth player interviews. (I'll go over this in a bit).

Now, when you've decided that you want to test your game qualitatively, there are four things you need to ask yourself. Blogger David Peter Simon explained it well in his post "The Art of Guerrilla Usability Testing":

1. What shall we test?
2. Where will we test?
3. With whom should we test?
4. How will we test?

What Shall We Test

Testing can be done at any phase in the design cycle. Only have your UI design sketched out on index cards? Fine. Have a fully functioning prototype? Awesome.

But "what shall we test" goes beyond "what type of resources do we need to run a test?" In terms of game development, it's "what game element do we want to focus on right now?"

Scenario: You've developed a lot of your game, and have started working on a tutorial. So far, you've designed a tutorial such that the tutorial text always sits on the right side of the screen, and updates when a user completes an action. You're confident that this will make sense to users, but you decide to test it just in case.

Where Will We Test

What you're testing will affect where (and with whom) you can test.

Want to test a mobile game? Consider bringing it to your local DMV and ask people if they'd be willing to play your game and give feedback on it while they wait. Want to test a computer game, or get feedback on a few designs sketched on paper? Bring your ideas to a coffee shop and ask people if they'd be willing to help you. Want to test a console game? Get feedback from players at conventions when they try their game.

Scenario: You bring a copy of your game to Starbucks and ask a few players to give feedback on the tutorial.

With Whom Should We Test?

If you're in a public space, find friendly-looking strangers who don't look to be rushing elsewhere. At the DMV and coffee shop, you're not necessarily going to find the types of people who would play your game after it comes out. But it's often people who aren't gamers who give unique feedback -- a regular gamer might know that WASD will move their character, but if a game is lacking a tutorial or feature that explains movement, a non-gamer might have no idea how to move their character.

If you're at a convention, you'll likely get people like your target population to test the game. When possible, it's good to test with these types of people -- but keep in mind the thoughts and concerns of non-gamers matter, too.

In addition, how many users you test with matters, too. Research suggests that running 5 participants will catch 75% of your usability issues -- after 5, you begin to have diminishing returns.

"Why You Only Need to Test With 5 Users" by Jacob Nielsen

There are times when you should run more than 5 participants -- so I suggest you read the article linked in the graph above.

If you're not comfortable approaching strangers, consider enlisting the help of acquaintances, or friends of friends -- people who aren't as concerned about your feelings when giving criticism about your game.

How Shall We Test?

There are a few ways to run a 1:1 player interview -- and it depends on the kind of feedback you're looking for. To get overall feedback about what the participant is thinking, use a "think-aloud" protocol to gauge what players are thinking. To get feedback about whether participants can find or do something, use a "task-based" test to identify problems with certain elements of your game. Of course, these two protocols can be combined into one.

Scenario: You test your tutorial with 5 people at Starbucks. Three of them do not see the tutorial immediately on the right side of the screen, instead inquiring aloud how they learn to play the game. Four of them says that the font on the tutorial box (once they see it) is too small and hard to read. 

Running a 1:1 In-Depth Interview

But running a qualitative study is more than just asking people what they think about your game -- a study is designed. Using the scenario listed above, I'll give you an example of what one could do to design a test.

1. Identify Research Question
   Question: What do people think about the current version of the tutorial? Is it easy to use?
2. Identify Testing Location
   There's a local coffee shop nearby that should work. I'll go on a weekend so that people aren't necessarily rushing to or from work. 
3. Design Research Study
   I want to know what people think of the tutorial, so I'll ask them to talk aloud about what they're thinking as they're going through it. If they ask about where tutorial text is, I'll ask, "Where would you expect it to be?" I'll also ask use some task based scenarios, so I'll ask: "Imagine you wanted to read the previous tooltip. How would you do that?" or "Imagine you want to disable the tooltips. How would you do that?" In addition, I'd also ask whether they thought the tutorial was complete, or if they thought it was helpful.
   Before bringing the study to Starbucks, I'll conduct my study with a friend/family member and time how long it takes. If it's more than 15 minutes, I'll cut down the study to the more essential parts.
4. Run Study
   I'll bring my computer to the coffee shop, ask people whether they'd be willing to test and give them an estimate of how long the test will take (< 15 minutes, I'm not paying people so they won't necessarily want to take too long). Offer to buy them coffee/food in exchange. Give them a consent form, let them know that they can stop at any time, and tell them that you want their honest feedback.I'll ask open ended questions, like: 
• What are your thoughts on this?
• Is there anything here that's confusing to you?
• Do you have a favorite thing about what you've seen here?
• Do you have a least favorite thing about what you've seen here?
5. Analyze Results & Develop Solutions
   After I've spoken to 5 participants, I'll go through my notes and look for patterns. What kinds of things did people like and dislike? How many individuals had trouble with a specific area? Did the participants have any particular feedback on how to improve the tutorial? After finding the patterns, I come up with some ways to fix what the participants mentioned. But it's important to note that not all feedback is helpful, and not all feedback should be included. When you're analyzing results, look for patterns and repetition -- just because one person said that she or he wants a feature doesn't mean it's worth including. 

Things To Keep In Mind

I must give words of caution: when you put your game in front of players, you will get criticism, and not all of it constructive. One of the most difficult parts of conducting user research as a designer is accepting criticism gracefully. When someone criticizes your game, they're not insulting you and they're not (necessarily) calling your baby ugly -- they're offering honest feedback. And honest feedback is what you need.

As you're running a 1:1 test, if someone gives negative feedback, don't  insult them. Don't say, "no, you're doing it wrong." Pretty much, in a usability test, the only things a designer should say are the open-ended research questions, any tasks you've prepared, and any probing questions like, "What specifically do you not like about that?" If they ask you a question about certain elements of the game, don't answer it. If they say something mean about it, don't say something mean back. It's difficult to do, but it's necessary. 

Summary

In summary, guerrilla usability testing can be used to get quick and dirty feedback on your game, at any point in the design cycle. Qualitative testing requires developers to put aside their feelings and ask for honest feedback, without getting defensive in return.


Finally, here are a few additional things to keep in mind:

1. When selecting your participants, try to get a balance of who you recruit. Don’t recruit all men, or all women -- even though you might not be talking with people who are big gamers, your target demographic will include all genders.
2. If you’re using a computer to test, consider installing a program that records audio and the screen. XSplit is a good choice. This way, you can look back through the videos and refresh what people said and did.
3. If/when a person asks you, a "how do I" question (how do I jump, which button should I press to run, etc), reflect the question back to them. "Which button do you think you would press to run? Why?"
4. If at any point in the study someone is no longer talking aloud, feel free to remind them to talk aloud. 

In Part 2, we’ll discuss how you can use tools like SurveyMonkey to quantitatively test the efficacy of specific game elements and compare different designs.


____

Want to learn more about usability testing? Check out these other resources:

• "Guerrilla Testing With Usability Cafe", YouTube Video.
• "Four Steps to Great Usability Testing (Without Breaking the Bank)" by Patrick Neeman
• "Talking With Users During A Usability Test" by Kara Pernice
• "Usability testing tips, helps, and guidelines" on UX For the Masses.

WOLCEN Studio's Wolcen: Lords of Mayhem is a hack and slash action RPG strongly reminiscent of Diablo II, Path of Exile, and Torchlight. Currently in early alpha, the game is rough-hewn stone -- it has a good shape, but little polish. 

Wolcen has gone the safe route, mostly, by not reinventing the wheel of the main UI. Compare the following images:

Wolcen: Lords of Mayhem Ability Bar

Diablo III Ability Bar

Torchlight II Ability Bar

Path of Exile Ability Bar

But where Wolcen does diverge, usability concerns follow.

Usability Concerns: Skills and How to Add Them

Where Wolcen struggles the most is how it displays and explains to users how to allocate and understand their skills. There are two types of skills: passive and active skills. The player gains active skills by finding and reading ability books -- and active skills gain 'experience' when used that then level those abilities up. Passive skills are acquired whenever the player levels up, and are not leveled up by use. 

There are a few issues with how Wolcen explains and displays its passive and active skills.

Adding To Shortcut Bar

Shortly after finding the first ability tome in the starting area, the player is presented with tutorial text that reads: "Lightning attacks are effective in water. Use the (S) key to assign Hands of Power in your Skills Bar." When I pressed (S), I expected the game to then allocate the lightning attack to (1) in my shortcut bar automatically. (It didn't.)

What happened was that the game opened an Active Skills pane, shown below.

Active Skills pane

Since the Tutorial box was still open when the Skills pane opened, I didn't see the skill bar behind it. So, when trying to assign the Hands of Power skill, I was clicking on the main skill bar, not the one hidden behind the tutorial pane (which is the correct way to assign). Thus it took me longer than I'd care to admit to correctly assign abilities to my skill bar.

Recommendations

• High Priority: Consider heavily redesigning the Active Skills page. Instead of having users click on any of their abilities, and then click on the skill bar, explain to them that they need to click and drag (or something) the icon into their skill bar.
• High Priority: Remove the copy of the skill bar from the Active Skills pane. The duplication of the feature is more confusing than it is helpful.
• High Priority: Make sure that your tutorial panes close after the player has completed that action! Part of the reason I didn't see the skill bar here at first was because it was hidden behind the box.

Unexplained Active Skill Elements

When an active ability is ready to level up, the player has the choice of leveling up that ability's Power Level or Utility Level.

... I have no idea what this means, and what in-game effect it has. Does upping its power level make it do more damage? What does utility even mean here?

In addition, the ability explanation pane, on the right of the "Active Skills" pane below, has other elements and icons that are not elaborated upon. For example, what are the three numbers above the ability description? Why is one of them red? Why is there an 'axe' icon on the screen? An hourglass icon?

These elements are not explained anywhere, so far as I can tell. And they should be.

Recommendations

• High Priority: You really need some kind of tutorial that explains or teaches all aspects of the leveling process (what is the difference between power level v. utility level; what do the icons mean; why are there many different numbers shown on the description pane, etc)
• Low Priority: Explain why the ability types ('Brutality,' 'Frost,' 'Lightning') matter; why do you have separate tabs for them?

Passive Skill Navigation

At the moment, there are five categories of passive skills: Protector, Rogue, Warrior, Wizard, and Utility. Since the game aims to allow players to have as much character and class customization as they want, players can choose from any of these categories. As such, if a player wants to compare two options, they need to tab back and forth in order to compare. 

What I'd recommend is doing something closer to Path of Exile.

POE shows all abilities on the screen simultaneously. The player can mouse over an icon to read its ability. The nice thing about Path of Exile's approach is that it allows for immense customization, and also enables the player to select and view an ability path before they confirm it completely (so if they decide they don't want to take two specific abilities, and haven't yet pressed "Apply" they can undo their choice.)

The other recommendation I have is to make your categories "Offense," "Defense," "Survivability," and "Utility" instead. The game can give a character a class based on their passive selections; but players can choose whichever abilities they want without choosing a class themselves.

Final Thoughts

The game is promising to be a good combination of Skyrim and the Diablo series. With the goal of ample character creation options (currently unavailable in the alpha) and the beginnings of an open-world layout, I'm excited for Wolcen to progress.

Recommendations

Health on top of character panel

• High Priority: Currently, the player's Health bar blocks out content on the Character information panel. Consider changing the screen such that the Character panel is always on top. 
• High Priority: Implement an openable "objective" pane that more fully explains what the player should be doing.
• High Priority: If implementing an objective pane isn't possible for the near future (too resource intensive), consider having the main character give some exposition as he's starting the dungeon (he got captured by ____ for _____ reason). Or something. The exposition should come before the objective is given.
• High Priority: Add the 'Shift' key's role into the control map. Like in other games, pressing the Shift key roots a character in place for attacking. This isn't on the control map; if you've played a similar game before you know it -- but it's not explained here.
  
  
• High Priority: Consider making the Ability Score (+) button more prominent on the character screen - it's currently hard to find.
• High Priority: Consider allowing players to press "Apply" or "Confirm" when adding new skill ability points after leveling up. They might want to see how increasing/decreasing a certain stat will affect their other stats. If they do not "Confirm" their selection, the points are not allocated.
• High Priority: Get someone to edit/proofread your English translation & grammar. There are a lot of errors & strange translations that need to be addressed.



Music settings bars.

• Medium Priority: Consider un-hiding the arrows next to these settings bars and having them always be visible next to the bars; the interaction will be far clearer and less error-prone
• Medium Priority: The font of the 'Mission Successful' pane is hard to read. Consider changing it.
  

  

  The shadowed font here, especially the white outline around grey, makes this pane difficult to read.

• Low Priority: As you have on other settings pages, give the active settings bar (whatever you've most recently selected) an orange/yellow/whatever you have treatment
• Low Priority: I'm not sure if this is a translation issue, but "Intelligence" is typically the word games use that influence the player's access to and efficacy with magic. "Power" is a synonym of Strength, and to a degree, Constitution, so in-game I wasn't sure which attribute would increase my mana pool. I'm consider renaming that to "Intelligence" just to be consistent with user expectations.
• Low Priority: Add quest/merchant symbols to minimap. I saw that there's a merchant with a blue * above his head. Make sure that emblems like that also appear on the minimap.

_____________

Apologies for missing my second post last month, folks! I was busy with my sister's wedding. It was an outdoor wedding in Florida, and it rained the entire weekend (Thursday, Friday, Saturday [wedding], Sunday). It was beautiful, though. Thanks again for reading!