Evaluating Sketchy Ideas, Part 4: Focus Group Questionnaire Results

This post is the third to do with a recent focus group which was held on the topic of some very basic and rough concept ideas that I had come up with. It's also the last installment in a series of posts concerning the earliest concept development phase of my project.

--------

Questionnaire Items

The questionnaire items were meant to gauge each participant's reactions to various aspects of each of the four concepts.

All participants were presented with the same 10 questions -- and asked to rate their answers on a 5-point Likert scale -- for all four concepts (i.e., Ping, the Bulletin Board, an Easy Desktop Status Updater, and Hover Status). Using the same questions for each concept allowed for direct comparisons across concepts.

Questions were adapted from a questionnaire used during a prior Future Workspaces assessment.

A list of the questions used:

1. I would use this tool frequently and actively.
2. I think this tool looks easy to use.
   
3. This tool would help me to exchange information with my colleagues.
4. This tool would help others keep up-to-date with what's going on with me.*
5. This tool would help me keep up-to-date with what's going on with others.*
   
6. This tool would help me to get to know my colleagues more personally.**
7. This tool would take up too much of my valuable time.
8. Using this tool, I would be in control of what others can learn about me.
   
9. This tool could provide benefits to my work.
   
10. The benefits appear to outweigh the costs of using this tool.

* - These questions have to do with staying in sync.
** - This question has to do with staying in touch.

Responses were coded as follows:

• Strongly agree = +2 points
  
• Agree = +1
  
• Neither agree nor disagree = 0
• Disagree = -1
• Strongly disagree = -2

Results

Here is a graphic capturing the reactions of the four focus group participants to the four concepts (no doubt it's best viewed in its own web browser tab):



Discussion

Given the small number of data points statistical analyses would have been inappropriate in this case. However, as the graphic clearly shows, the questionnaire was a useful tool for assessing participant attitudes.

It is noteworthy to mention that the tools rated the highest overall (see the previous post for the complete results, or the graphic below) were associated with perceptions of having a higher benefit-to-cost ratio (i.e., Easy Status Updater and Hover Status). Not surprisingly, these were also the tools that people said they would use more frequently (in addition to Ping).

On the flip side, the Bulletin Board was considered to be the only tool that could potentially take up too much of the users' time.

Two factors seem to have driven people's overall rating of the concepts -- that is, time and the perceived presence of additional benefits. I'm going to take some time to justify and further explain this claim, so bear with me.

Here again is the graphic showing the participants' overall distribution of bonus and penalty points for the four concepts:


Overwhelmingly, the most poorly rated tool was the Bulletin Board.

But, if one examines the questionnaire results more closely, this rating has nothing to do with its potential for keeping colleagues up-to-date with one another. Pooling the results of the two up-to-date questions, the Bulletin Board actually out-rated the Hover Status tool or Easy StatusUpdater. Likewise, it also rated higher than any other tool when it came to potential for keeping colleagues connected.

On every other measure, the Bulletin Board drew mostly neutral responses.

In the end, the consensus of the business developers was that they did not find status updates useful to their work.

Perhaps, as I mentioned in the last couple of posts, this is because -- for the most part -- they work independently rather than in teams.

Although both the Easy Status Updater and the Hover Status tools focus on status updates, the effort expended in updating one's own or viewing an other person's status is very low -- one could argue, sufficiently low enough to make the use of such tools more acceptable.

----------

Given the findings from the expert session and focus group, I've begun a second design iteration. The next few posts will cover aspects of the redesign.