Evaluating Sketchy Ideas, Part 4: Focus Group Questionnaire Results

This post is the third to do with a recent focus group which was held on the topic of some very basic and rough concept ideas that I had come up with. It's also the last installment in a series of posts concerning the earliest concept development phase of my project.

--------

Questionnaire Items

The questionnaire items were meant to gauge each participant's reactions to various aspects of each of the four concepts.

All participants were presented with the same 10 questions -- and asked to rate their answers on a 5-point Likert scale -- for all four concepts (i.e., Ping, the Bulletin Board, an Easy Desktop Status Updater, and Hover Status). Using the same questions for each concept allowed for direct comparisons across concepts.

Questions were adapted from a questionnaire used during a prior Future Workspaces assessment.

A list of the questions used:

1. I would use this tool frequently and actively.
2. I think this tool looks easy to use.
   
3. This tool would help me to exchange information with my colleagues.
4. This tool would help others keep up-to-date with what's going on with me.*
5. This tool would help me keep up-to-date with what's going on with others.*
   
6. This tool would help me to get to know my colleagues more personally.**
7. This tool would take up too much of my valuable time.
8. Using this tool, I would be in control of what others can learn about me.
   
9. This tool could provide benefits to my work.
   
10. The benefits appear to outweigh the costs of using this tool.

* - These questions have to do with staying in sync.
** - This question has to do with staying in touch.

Responses were coded as follows:

• Strongly agree = +2 points
  
• Agree = +1
  
• Neither agree nor disagree = 0
• Disagree = -1
• Strongly disagree = -2

Results

Here is a graphic capturing the reactions of the four focus group participants to the four concepts (no doubt it's best viewed in its own web browser tab):



Discussion

Given the small number of data points statistical analyses would have been inappropriate in this case. However, as the graphic clearly shows, the questionnaire was a useful tool for assessing participant attitudes.

It is noteworthy to mention that the tools rated the highest overall (see the previous post for the complete results, or the graphic below) were associated with perceptions of having a higher benefit-to-cost ratio (i.e., Easy Status Updater and Hover Status). Not surprisingly, these were also the tools that people said they would use more frequently (in addition to Ping).

On the flip side, the Bulletin Board was considered to be the only tool that could potentially take up too much of the users' time.

Two factors seem to have driven people's overall rating of the concepts -- that is, time and the perceived presence of additional benefits. I'm going to take some time to justify and further explain this claim, so bear with me.

Here again is the graphic showing the participants' overall distribution of bonus and penalty points for the four concepts:


Overwhelmingly, the most poorly rated tool was the Bulletin Board.

But, if one examines the questionnaire results more closely, this rating has nothing to do with its potential for keeping colleagues up-to-date with one another. Pooling the results of the two up-to-date questions, the Bulletin Board actually out-rated the Hover Status tool or Easy StatusUpdater. Likewise, it also rated higher than any other tool when it came to potential for keeping colleagues connected.

On every other measure, the Bulletin Board drew mostly neutral responses.

In the end, the consensus of the business developers was that they did not find status updates useful to their work.

Perhaps, as I mentioned in the last couple of posts, this is because -- for the most part -- they work independently rather than in teams.

Although both the Easy Status Updater and the Hover Status tools focus on status updates, the effort expended in updating one's own or viewing an other person's status is very low -- one could argue, sufficiently low enough to make the use of such tools more acceptable.

----------

Given the findings from the expert session and focus group, I've begun a second design iteration. The next few posts will cover aspects of the redesign.

Evaluating Sketchy Ideas, Part 3: Focus Group Discussion and Rating Results

I recently held a focus group session with four in-house business developers. The goal was to get some feedback on the merits and weaknesses of a few rough concept ideas.

For more information on the concepts themselves and set-up of this focus group, please see the last post.

In this post I'll cover the outcome of the discussions which were held during the focus group, as well as the participants' overall impressions of the concepts. I'll wait until the next post to cover the questionnaire results, as those conclusions are sufficiently long enough to warrant their own post.

Discussion results (by concept)

I presented each of the four concept ideas (Ping, Bulletin Board, an Easy Desktop Status Updater, and Hover Status) in turn.

While presenting the concepts, I prodded the participants to engage in discussion by asking the following questions:

- Do you think this [concept] would benefit your ability to stay up-to-date with your distant colleagues? Why or why not?
- Do you think this [concept] would benefit your ability to stay connected with your distant colleagues? Why or why not?
- More generally, would it be useful to you? Why or why not?

This discussion was meant to supplement a questionnaire that each participant was asked to fill out following the presentation of each concept.

Some select comments made by the participants with regards to each of the four concepts are listed below, as well as some general trends.

Ping

- I would try it around noon, to see who is [around].
- It would be more useful for times when [one] is out-of-the-office.
- It seems to be more beneficial for people who work outside of the organization. Like for instance, when I am in the Hague. It would be useful to stay connected with main stakeholders when I am in that area of the country.
- As to staying connected or up-to-date, the application itself doesn't tell you much. These are things you would learn in person, after having used the application to meet up.
- By show of hands, all said they would use it.
- As to how frequently it would be used, this result was more mixed because they said it's use would be contingent upon the situations that happened to arise. [This point illustrates the opportunistic nature of the application.]
- Would like the ability to Ping groups of people, not just individuals.
- Would prefer a mobile web-based application (rather than have the operation of the application be contingent upon SMS*).

*It is important to note that one of my design constraints was explicitly not to build an application usable only by individuals who own smartphones.

Bulletin Board

- Only one of the group said that he uses an application in which he regularly updates his status (in this case, twitter).
- Everyone else found the act of status updating dubious (i.e., it is all about people talking about the sandwiches they are eating; there is no useful information being shared, &etc). Important note: These individuals did not actually have first-hand experience using such applications (e.g., twitter).

More specific quotes:

- I would update my status, but only if it was done automatically, from my calendar.
- As a sender, I am not sure that I would want to update my status all of the time.
- I am not interested in what other people are doing.
- It should be a mobile application, too -- but one that does not rely solely on SMS.
- There is too much information coming to me already -- I would need a filter.
- I do not want to know about the status of 100 different people, but for some specific people, yes. For instance, for a colleague that has been having personal difficulty lately, it is good to know if he is okay when I don't see him. But this would mean the application would need have some kind of a filter to get just his specific information only.
- For the more information-rich version of the Bulletin Board: If the idea could be adapted so I (and the people in my group) just send short messages for updates rather than spend three hours/week writing reports, I would use it because it would save me time. I don't like giving status updates, but if it helps reduce the work time by replacing all of this report writing, then I would do it.
- One of the participants said he would use the more information rich version (i.e., the one quoted directly above); for the basic version, it would need to include automatic updates as an input method to be more acceptable.

Easy Desktop Status Updater
- Questions regarding to where the status would be channeled (e.g., twitter, Bulletin Board, LinkedIn, &etc?)
- One person was very keen on using it only to have his status automatically updated.
- Three out of the four people said they would use it.
- Wants to be able to filter who gets what status information (by, for example, sending some status updates only to his closest work group and not the entire organization).
- Again would prefer a mobile web application (does not like the use of SMS to update one's status), because he said that it would be more convenient for when people are on-the-go.

Hover Status
- All said that such information, in the form presented, would be useful to them.
- Given feedback from a prior session, they were also asked whether the presentation of the information might prove "irritating"; they all said that they did not perceive that to be a problem.
- Also would like to get location-based information on their contacts which would be included in the hover stats.
- One said he likes the idea of "pulling information" to himself.

Overall impressions of the concepts

At the end of the focus group session, each participant was asked to give their overall impression of the concepts by distributing five bonus points (green stickers) and five penalty points (red stickers) among the four concepts. They could distribute the points however they wished.

Here are the results of that exercise:



As is evident, the Hover Status tool was the most enthusiastically endorsed. Ping and the Easy Desktop Status Updater received neutral-to-moderately positive ratings. Overwhelmingly, the Bulletin Board was given a poor rating by everyone.

It's important to keep in mind that these ratings aren't nuanced -- in other words they're gross-level endorsements or dismissals of the concepts as a whole.

Here are some of my thoughts on the best and worst ratings:

Bulletin Board

With respect to the Bulletin Board, two things should be kept in mind:

1. The business developers tend to coordinate in their work -- rather than collaborate with their colleagues. Moreover, much of their work (e.g., hunting down leads) is done independently. This might explain why there was a general lack of interest in knowing what their colleagues were up to.

However, all of the other mobile workers that I interviewed worked in project teams and engaged heavily in collaborative work. For this reason, there might be more of an interest in a tool that provides them with information about their distant colleagues. Also, aspects of the tool that were found to be disagreeable to the business developers were explicitly asked for by some Traditionalists that I had interviewed (e.g., a "News Flashes" feature).

2. I asked them to react to the information-rich version only; it is possible they might have been more positive about the basic version (which included only status updates). However, because of their group-level dismissal of the practice and use of status updates, it seems doubtful.

I can't help but wonder how others, who work in distributed project teams and in decentralized organizations might view the basic version.

Hover Status:
This tool was rated the highest, and for good reason. It is lightweight and useful -- especially in those moments when one must decide what strategy is best for contacting another person.

However, I think more needs to be done to rework this tool so that it helps colleagues in large, distributed organizations interact. It seems to me that this tool is best used primarily with respect to one's known colleagues. How this will help a person discover what is going on within the department or larger organization remains unclear.

--------

As mentioned previously, the results for each of the four 10-item questionnaires will be covered in the next post.