Evaluting Sketch Ideas, Part 2: Recipe for a Focus Group

Early in the week I held a focus group with some potential users. In total, four business developers showed up for the session (out of the six that replied, two were no-shows -- I won't dwell on what this means regarding my popularity). ;)

This post will cover the materials and set-up of that focus group. In the next post, I'll cover findings.

About the Participants

The business developers share some key characteristics with the Traditionalists that I had initially interviewed. Both groups find themsleves oftentimes out-of-office, spending a large portion of their working days criss-crossing the country to engage in meetings.

But there are also some important differences too. The Traditionalists devote a sizeable percentage of their jobs to project work. In focusing primarily on project work, it becomes critical for them to manage their communication channels in such a way as to support their ability to collaborate within their project teams effectively.

Business Developers, on the other hand, work more independently. Their main task is hunting down "leads". Certainly open communication with their intra-organizational peers is a benefit to this kind of work (e.g., it's important to know who is hunting down what lead and the outcome of that effort) -- but, it does not require a need to collaborate at the same level of intensity that project work does. Instead, it could be argued that it requires business developers to coordinate more often with their colleagues.

I mention this difference because it has some important implications for the results of the focus group session.

The Ingredients (aka, Materials & Set-up)

1. Presentation - You can view it here (however, it does look better if you follow the link to view it in full screen mode):Focus Group 27-7-09 - Concept Ideas x 4 Four very rough concept ideas were presented. An important note: If you view the slides, the following critical point might not be evident -- that is, the Bulletin Board concept is given two distinct visualizations. One is a basic version, that only shows status updates, short messages and mulitmedia, and is filterable by group (e.g., project team, department, organization, and all). Another, more information rich version of the Bulletin Board is also shown. This second version includes quick news updates, document sharing, and a bare-bones forum (in addition to everything available in the basic Bulletin Board -- it is basically the Daily Outlook concept from a previous post, however, the application need not be restricted to Outlook in this case). In addition, it supports threaded discussions. The criteria by which I wanted them to differentiate between these two concepts was whether or not they benefited from the added information -- or whether the additional information might pose too much of a cost (e.g., in the form of information overload) to them. Whether they received the information in the form of a desktop application, online, or in Outlook mattered less than whether or not they preferred the full version of the concept, or a scaled down version of it.
2. Cue cards - The explicit goal of this project is to: Support mobile workers in their ability to stay in sync and in touch. However, these terms -- in sync, in touch -- are easily confusable. So we went with "up-to-date" and "connected" for the focus group session because everyone has an intuitive sense of what these synonomous terms mean to them. The purpose of the cards was to remind the participants of the criteria that I wanted them to use in assessing the concepts. So, the basis of the discussion for each concept revolved around these basic questions: Would it help you stay up-to-date with your colleagues? And, would it help you stay connected to them?
3. Questionnaires - Following the presenation of each of the four concepts -- that is, Ping, the Bulletin Board, an Easy Desktop Status Updater, and Hover Status -- participants were asked to fill in a brief 10-item questionnaire. (A complete list of the questions will appear in a future "Results" post).
4. Bonus & Penalty Points - Each participant was given five green dots (or bonus points) and five red dots (penalty points). They were asked to distribute the points among the four concepts as they saw fit (e.g., if they wanted to assign all five bonus points to one concept, that was perfectly acceptable).
5. Food - What is a focus group session without food?

Results...

...Forthcoming in the next post.

Evaluating "Sketchy" Ideas, Part 1: Expert Feedback

Last week I met with a few in-house experts (i.e., members of the Future Workspaces team) here at Novay and asked them to give feedback on some "sketchy" concept ideas.

I went to them with six rough ideas for solutions to keeping mobile workers -- specifically the "Traditionalists", as I've come to refer to them -- in sync and in touch. I showed them a series of sketches illustrating each concept.

A very brief description of each of these follows:

1. Ping

Location-based awareness of colleagues that does not require the use of a GPS-enabled smartphone* to work. It helps mobile professionals check to see when their geographically-distributed colleagues are nearby so that they can then contact one another for an ad hoc, face-to-face meeting.

2. Bulletin Board

A means of sharing status updates, short messages, and multimedia that is filterable by group (e.g., "Project Team", "Department", or "Organization"). The application would act as a sort of dynamically updating desktop wallpaper.

3. Daily Outlook

An add-on for Outlook that configures the Outlook Today page and is meant to support the sharing of: 1. Status updates; (2) News Flashes; (3) Document sharing (via the use of links); and (4) a Q&A section (basically, a bare bones forum).

4. Outlook-based Blogging Tool

An add-on for Outlook that makes it easy to post to blogs as well as track, read and share blog posts from within Outlook (all without having to open a web browser).

5. Free Five

Personnel within a company are matched based on mutual professional interests. Individuals can make available "five minutes" (or more) of free time to meet with previously unknown colleagues, thus earning points. Earned points can be traded in a type of bartering system that allows participants to "buy" time slots from one another. The basic idea is to foster the making of new connections within large, decentralized organizations; help find experts; and, open new opportunities for collaboration.

6. Hover Status

By hovering over the name of a colleague -- for example, when composing a new email message -- a small window appears that displays that contact's latest status update (e.g., as fed from their calendar). This information can be used to help one decide which method is best for contacting an individual at that moment -- be it email, phone, or face-to-face communication.

_____________________________

While they were evaluating the concepts, I asked the participants to keep in mind the six design requirements (as outlined in a previous post). As a reminder of those requirements, I printed off some A4-sized cards listing the requirements and gave one to each participant. That way they could easily refer to the requirements -- and assess the extent to which each concept idea addressed them -- as they gave their feedback.

For each concept, participants were asked to give three pros, three cons, and three improvements for furthering the design. I used this feedback as the basis for a second design iteration (I plan to share some of this feedback in future posts). Then, I had them rank order each concept. Each first place ranking was given a score of six points, a second place ranking was given a score of five, and so on.

The outcome of the rank ordering is as follows:
1. Ping - 20 points
2. Hover Status - 18 points
3. Daily Outlook - 16 points
4. Bulletin Board - 14 points
5. Outlook Blogging Tool - 11 points
6. Free Five - 6 points

I used the feedback and rank orderings to modify the top four concepts. The following week I held a focus group with users. There I presented the Ping and Hover Status concepts, along with a new concept (which also came out of the expert assessment) -- an easy status updater that sits on the desktop. The Bulletin Board and Daily Outlook concepts were rolled into a single idea.

In the next post, I'll share more about this focus group.

-----

*I realize in the example here I've actually drawn a smartphone. That's simply a function of the fact that they are easier to draw (e.g., fewer buttons, paired with a nice big display!) As an aside: Although smartphone sales are on the rise -- according to a recent report, despite the fact that there has been a sharp increase in the number of smartphones sold -- they still only make up approximately 13.5% of the mobile phone market. And these numbers are reflective of the mobile professionals I talked to -- that is, the majority of them didn't own smartphones.

Design Philosophy

My coach suggested a great idea -- that is to come up with a detailed design philosophy. I'm sharing it here because -- although some of the content has already been published on my blog -- I think it nicely illustrates the logic underlying the user requirements I have chosen.

Curtis 2009 Design Philosophy

Study Conclusions

The above table highlights the major findings from two sets of interviews that I conducted with two distinct groups of individuals.

The "Traditionalists"

Findings from the first set of interviews have been covered in detail in a series of earlier posts. In short, the interview participants from this group possessed the following characteristics:

- Most (8 out of 10) worked for large companies (10,000+ employees).
- All worked on projects in geographically distributed teams.

The first three columns of the table specifically address issues related to the above individuals. I call them "traditionalists" because -- although they are mobile workers -- they are constrained in their use of tools and practices in such a way that many of their interactions with distant colleagues arguably mimic those of traditional, co-located office workers. That is to say that most co-located colleagues prefer face-to-face and email methods of interaction with their colleagues-- and, so do these individuals, as well.

The first column shows that these traditionalists have four basic needs. Their primary needs function at the project level, and have to do with staying in sync and in touch with their fellow team members. Secondary needs deal with the individual's interactions at the departmental (or group) level -- and beyond -- with regards to both staying in sync and in touch.

The second and third columns capture their ability to effectively deal with all of these needs. Not surprisingly, the tools and practices they use for handling their needs are satisfactory at the project level. However, breakdowns occur at the departmental level when they attempt to use the same tools to address their primary needs as their secondary. This makes sense, because using one-to-one communication platforms to target a large, diffuse and loosely connected group of individuals is an inefficient strategy. And, arguably, the tools the traditionalists most heavily rely on are best suited to one-to-one communication, including email. (As a tool for eliciting two-way communication, emails directed to individuals are much more successful than broadcast emails).

The Highly Connected Professionals

Given these findings, I next interviewed a group of highly connected (or networked) individuals. I also reviewed a set of interviews that had already been conducted for a previous study at Novay. My goal was to see if these individuals might incorporate some strategies or tools in their professional lives that might be of use to the traditionalists.

Identifying the gaps, and bridging them

One of the notable differences between the traditionalists and the highly connected professionals was that for the latter group, their needs were less distinctly segmented*. Instead, it would be more appropriate to talk about these individuals' interactions with their networks, as whole entities. Thus, a highly connected professional's principal need is to stay in sync and in touch with this diversely populated network. And, whereas it might be more appropriate to say that the majority of those in the traditionalist group largely associate their networks with people that they have interacted with (at some point) in person, with the highly connected professionals, this generalization does not necessarily hold. Highly connected professionals actively maintain networks that include any number of individuals they have never met face-to-face but who are still considered valuable.

Another striking difference is that in their interactions with their networks, the highly connected individuals employ diverse and experimental methods and tools. Oftentimes this means participation in online communities (e.g., blogging networks, Ning, or forums) and other forms of social media (e.g., twitter). To stay abreast of recent information -- from either individuals or other news sources -- RSS feeds are used to filter information.

Perhaps it seems a bit of a stretch to characterize an individual who actively uses twitter and RSS feeds as "experimental". But consider the following: According to a highly publicized report, there are approximatley 4.5 million twitter users worldwide. In addition, roughly half of the users on twitter have never even updated their status, not even once. However, given the ubiquity of twitter coverage in the media -- and RSS on websites -- it's easy to think of the use of such tools as much more commonplace than is probably justified.

So, it is probably not surprising that the use of such tools is not more widespread among the traditionalists I interviewed.

That said, there are some characteristics underlying the tool use of the the highly connected professionals that might be of use if applied to a solution for the communication breakdowns that occur for the traditionalists.

First, the highly connected professionals make generous use of filters. I categorize these filters in two ways. First are the emergent filters. Imporant information "emerges" from the buzz going on in an individual's network. An example is realizing that Google Wave is a highly anticipated application and perhaps worth investigating because many of one's connections (spanning across various channels) are commenting about it on the day that its preview is released.

The other type of filter comes from the use of information feeds. Using an RSS feed reader, for example, it is possible to process the latest pieces of information from only those sources that an individual feels are pertinent. So, rather than learning about Google Wave from a colleague, I might read about it in an article posted to the ReadWriteWeb feed.

By using both types of filters, there is some redundancy in the system. However, this could be construed as a good thing because it lessens the chance that important information will be overlooked.

Unfortunately for the traditionalists, when trying to stay up-to-date with departmental and organizational level information, comparatively they are at a loss. Not only do they lack the support for developing an adequate departmental-level network (which, in turn can be an excellent source of internal information), they also complained of difficulties directly accessing useful information within the organizations that they worked for.

A solution that provides an easy mechanism for information filtering would benefit the traditionalists because many complained that organizational information depositories were often fragmented amongst a number of databases and applications. Furthermore, they distrusted the relevancy of information from such sources because it was oftentimes outdated. If fresher, relevant information was funneled to them, they could avoid the laborious and frustrating task of having to seek it out.

Inadequate access to internal information is problematic for a number of reasons, not the least of which because the real value of reports and documents isn't in their completion but in the discussions, actions, and future collaborations that result from their dissemination. Here, too, the traditionalists can be seen as being at a loss.

Finally, the highly connected individuals embraced open communication -- by this I mean communication that is not necessarily one-to-one, but is instead oftentimes one-to-many. The interactions that occur when people participate in online forums and communities are typically one-to-many. A shift away from one-to-one communication could help the traditionalists nurture in sync and in touch relationships at the departmental level.

_____
* It's important to note that this difference might simply be an artifact of the way that they talked about their professional connections -- but of course, this is a risk inherent when relying on self-reports, in general.

Persuading people to change

Inspired in part by Mathemagenic and as recommended by my coach here at Novay, I've been publishing a few entries that have to do with preliminary results that will ultimately go into my final report.

However, truth be told, my current progress isn't well reflected in this blog. I've yet more results from a round of later interviews to publish, and plan to do so (and soon). The conclusions I've drawn from both sets of interviews are interesting, and will be shared here. And, I'm currently working on some usage scenarios for a couple of concepts...

But, first I'd going to take a moment to depart from progress reporting to do a bit of mental meandering on the topic of persuasion. Yesterday, during a Future Workspaces group meeting in which I presented some basic ideas for a couple of concepts, the question of how to effectively change people's attitudes came up.

This topic falls under the purview of persuasive technology. Perhaps the name most heavily associated with this field is that of BJ Fogg. According to Fogg, persuasive technology, generally defined, is the intentional use of technology to change human behavior and/or attitudes (Fogg, 1998). It is important to note that this does not imply the use of coersion -- as stated in the Golden Rule of Persuasion, it is unethical to use technology to induce people to do "something they themselves would not consent to be persuaded to do" (Berdichevsky & Neuenschwander, 1999). Socially-conscious examples of persuasive technology include an electronic doll for teens, meant to mimic the behaviors of a newborn infant (thus increasing awareness about the implications of teenage parenthood), or a website that is meant to promote and encourage healthy eating choices among children (King & Tester, 1999).

This topic came up in reference to some of the conclusions that I came to following both rounds of interviews. Namely, that staying in sync and in touch with colleagues at the departmental and organizational level can be a big challenge for mobile workers. There are some good reasons why this might occur (which I'll get into in more detail a future post), but suffice it to say for now that there are a few main ones.

• First, in doing primarily off-site project work, the number of opportunities to interact face-to-face with their departmental colleagues are few.
• Secondly, their primary needs deal principally with the activities that support their current project work and team -- thus making anything that falls outside of this narrow focus secondary.
• The favored tools for staying in sync and in touch at the project level include the phone, face-to-face meetings, and email.
• Finally, breakdowns occur when the preferred tools for staying in sync and in touch at the project level are also used for interactions at the more highly-distributed departmental level (and beyond).

In other words, the difficulty that a sizeable number of the mobile workers I spoke to face occurs when they try to use their favorite tools (the phone, face-to-face meetings, and email) to establish connections and stay up-to-date with a larger, more diffuse group of individuals -- that is, their departmental colleagues.

Certainly there are better tools for establishing and maintaining in sync and in touch relationships with a large group of individuals than the telephone. Twitter, for example, immediately comes to mind.

However, I also found new technology adoption among this group to be somewhat lagging. One commonly cited reason has to do with a lack of widespread following among their already established connections -- an argument that makes a lot of sense. After all, why use twitter if no one else you know is using it too?

But another big contributing factor had to do with the heavy emphasis placed on face-to-face communication.

All of this lead to a discussion about how to change people's attitudes. I also interviewed a group of highly connected (and networked) individuals, and their attitudes about the role of face-to-face communication tended to diverge from the first group of mobile workers. Generally speaking, in their view face-to-face communication was clearly beneficial -- but not always absolutely necessary, not even for collaborative work.

So, the question is, how to foster an attitude change in the more traditional group of mobile workers towards alternate tools or methods of communication?

The answer, I think, might lie in the use of persuasive technology.

To that end, I began by hunting down relevant examples. I looked into an effective system for building up an expertise localization database within organizations, ExpertFinder. Basically, the way it works is that a group of employees is sent an email asking them to assess their own expertise level in a given domain, as well as refer others who might also be experts in that specific area, too. In addition, users are able to add addtional domains, thus expanding the database with relevant topics.

This system is a nice example of the use of persuasive technology. In a pilot study, 500 individuals who worked in a R&D department of a large company submitted 2,500 emails covering close to 200 topics over the course of a single week. In other words, the system was incredibly effective at encouraging people to build up a database of expertise in a very short amount of time.

The evident question from all of this is how to draw on examples of persuasive technology in order to create a system that helps induce distant departmental-level colleagues to connect more effectively? Is it better to create a solution that works well with the tools mobile workers prefer (i.e., the phone, face-to-face meetings) or find a way to encourage them to expand their tool set?

I'm working on a couple of concepts that address both possibilites and will return to them in detail in a later post. Nevertheless, I'm interested in what others think.