PAX East was much louder and cooler than I expected. Last month, I got the opportunity to attend as both a general convention-goer (Saturday) and as a volunteer (Sunday) with Perception by The Deep End Games.
In my last post, Guerrilla Usability For Game Developers, I recommended that developers try to get usability feedback at conventions. Now, having done this myself, I have some tips and tricks on how to conduct quick-and-dirty qualitative tests at conventions.
- If you can, get a neutral observer.
In general, people are averse to hurting others' feelings. As such, there are times where an individual might have criticism about a product, but may be reluctant to divulge their thoughts in fear that they'll hurt someone's feelings.
If you want to get a broader spectrum of positive and negative feedback, consider recruiting a neutral observer - a friend who hasn't worked on your game but is willing to help you out at the booth. That person should not wear clothing branded with your game's name, to further emphasize that they're not involved in the development of the game.
Why? When interacting with players at your booth, that neutral observer can honestly tell the players that they didn't have anything to do with making the game, and that candid feedback, positive or negative, is welcome all the same. By not wearing branded clothing, the neutral observer reinforces that they're not directly connected with the game.
(In addition, allowing a neutral party to conduct the usability research allows you to focus on running the booth as a whole.)
Probably an hour into my volunteering, I was watching a player who had ample criticism about the demo and was very articulate in his frustrations. Afterwards, another volunteer remarked that that player was the first person to voice negative comments about the game. What's important to note is that unless you are directly asking for negative feedback, and express that it's not going to hurt your feelings, it's unlikely you'll hear a ton of criticism. I noted that just because he hadn't heard negative feedback before probably wasn't because no one else before that player had criticism. - Moderating limit of two.
In the very beginning of my shift, I wasn't sure how many players I could watch at once. Once I got into a rhythm, though, I found out that I could moderate two players simultaneously if they were sitting adjacent to one another. Trying to moderate three at once was very difficult and led to a drop-off in the quality of my observations.
Later, when players had different start times on the games, I found it to be much more difficult to moderate two players simultaneously (radically different problems in early and mid-game that I wanted to probe on). Once the start times were so out of sync, I started watching only one person at a time. - Similar start times.
If you can, try to get them to start the game at roughly the same time. Getting the players to start at the same time allows you to give your spiel about "I'm a neutral observer, tell me how you really feel" only once. This was far easier earlier in the day, when the line wasn't too long. - Scrap talk aloud and focus on body language.
While the talk aloud method works great in the lab, it's so loud at a convention that it's nigh impossible to hear what players are saying. Instead, be on call if they have a question about something and watch your players' body language. Some players slightly shake their heads when they're frustrated, or lean in slightly when they're having difficulty reading something. One of the players I watched had a clear tell for frustration -- his fingers would flare out and up off the mouse for just a second and he would tense up.
It was beautiful. When body language is clear, it's easier to probe on thoughts that the participant didn't initially feel like/think to share with the researcher.
Unlike in a lab, where you can more easily see a participant's face, you'll need to focus more on the head, shoulders, and hands of the player. - Consider exit interviews.
Later in the day, conducting orderly usability sessions will be naught but a dream. If you weren't able to watch a player go through the game, consider conducting exit interviews as they leave to get their candid feedback. One of the other volunteers redirected players done with the demo to me so that I could get their feedback & quotes. - Don't make usability testing a whole-day event.
I volunteered to help from 9am - 3pm, but by ~noon I had found all of the usability issues I was going to find. I lost track of how many players I spoke with or watched, but it was far, far over the 5 people one technically needs to get 75% of usability issues. If you're paying your neutral observer, consider only making her or his shift a few hours (or have their role shift a few hours into the event).
 |
| Photo of the Perception setup (credit: The Deep End Games) |
All in all, a great experience. Thank you Bill Gardner, and the rest of Deep End Games, for letting me tag along.
Posts
0 comments:
Post a Comment