Online Focus Groups: The Ups and Downs of Remote Usability Testing

Slo-Mo No-Shows and Other Considerations

Need to conduct focus groups with participants who live across a broad geographic area? We have some advice for you.

We recently completed a strategic plan for a client with a customer base across Canada. We learned in our stakeholder interviews that they had no real data on who was using their websites, or why. They had guesses. When we asked them what they’d like to ask their customers, the top question they came up with was “Why are you using our websites?”

Well, ethnographic research was out—although a three-month UM road trip would have been awesome—so we settled on three focus groups, conducted entirely online, with participants from across Canada. Each focus group had eight participants; each session lasted two days. Over the course of two days, we would post questions on the bulletin board and send participants reminder emails to answer them, or to follow up on earlier questions.

Software: Pick Your Poison

Our first issue: what software to use?

We ran through a few options:

  • Facebook Groups. But we’d need participants to be Facebook members. And use their personal accounts. We (or they) could set up dummy accounts, but even if we got past the privacy issues, we wouldn’t have been able to easily export the exchanges.
  • Google Groups: Very tempting. Using an email system would’ve put the discussion right in participants’ inboxes where it’s hard to miss, and the setup was dead easy. But our raw data, even sanitized with dummy email addresses issued to the participants, would have been in 24 inboxes. That didn’t feel right.
  • Free bulletin board software: We had conducted online focus groups before using PHPBB, a free package that gave us lots of control. Privacy locked down, data easily exported, emails alerts—the whole show.

We went for the bulletin board software.

But, while we got the control we wanted, we also got a lot of hassle. The setup and management weren’t difficult, but they were time-consuming. This route is worth it only if you’re doing many online focus groups. (By the way, we’ve found SurveyMonkey handy for consent forms.)

Tricky Questions…

We quickly ran into a few issues that don’t come up in a one-hour in-person focus group:

  • If a participant has two days to join the discussion, when can you safely call them a “no-show”? We had a few participants who didn’t answer our reminder emails, let alone join the discussion, until the second day—our “slow-mo no-shows.”
  • What’s stopping participants from writing the minimum to get their incentive? That prompt to quiet participants at the end of the table has a lot more overhead when you have to open an email client, write a note, send it, and see if they respond.
  • How do you generate the back-and-forth dynamic that comes about around a table? Is that even possible online?
  • Can your participants write well? Our recruiter screened participants by phone, and while the participants may have been articulate, they weren’t all good writers.

The Bottom Line

Our client got a better sense of why people were visiting their sites—in participants’ own words. Having a searchable database of comments was really valuable; just search for a keyword to get all comments, from across all three groups, one a particular topic.

But the logistics were time-consuming, and it was much harder to manage the conversation and get rich contributions.

If we do this again, we’ll likely use Google Hangout or Skype.

We’ll still get participants that we couldn’t gather in one room, but we’ll limit the discussion to one hour—and limit the overhead. And showing clients a recording—rather than sharing written comments—will be that much more powerful.

If you’re a visual person, we put together a poster about this project for the IA Summit 2012. Enjoy.

Slo-Mo No-Shows and Other Considerations