Final Project Summary

Hooray! I can finally say this project has reached its conclusion. The project findings are summarized below. The complete project report can be found here, and corresponding presentation here.

The two main goals of the current project were to: (1) Identify the consequences and challenges of mobile work – particularly with respect one's ability to stay up-to-date (i.e., in sync) and connected (i.e., in touch) with his or her distant colleagues; and, (2) come up with potential design solutions to address these challenges.

Project Goal 1: Determine the role of in sync and in touch in mobile professionals' lives

Interviews were initially undertaken with a group of mobile workers in order to determine how much they valued staying in sync and in touch with their distant colleagues. Because all of those interviewed worked primarily in distributed project teams, they all spoke of expending a great deal of effort in order to nurture a sense of connectedness and stay up-to-date with their immediate, project-level colleagues. To that end, they generally traveled in order to hold regular face-to-face meetings with their team members, or were in daily contact with them by either phone or email (or both). So much effort was poured into activities that supported staying in sync and in touch with their project level colleagues that all those interviewed expressed satisfaction with the outcome of these efforts. In fact, staying in sync and in touch with one's project team was so important that it came to be considered a primary need of the mobile workers interviewed.

However, breakdowns occurred when they tried to use the same tools and practices for staying in sync and in touch with their project teams, as with the colleagues in their departments or wider organizations. The main problem arose when they used what one might consider traditional office tools – i.e., principally face-to-face meetings, phone and email – to infiltrate this second, much more dispersed and loosely connected group of individuals. Thus, these individuals came to be referred to as Traditionalists, namely because their choice of communication tools most closely resembled that of traditional, co-located office workers. They were also much more conservative in their approach to communication than a second group of highly connected networkers that was interviewed. To an extent, staying in sync and in touch with one's departmental or organizational level colleagues was considered to be of value to the Traditionalists (although not to the same degree as interactions with one's project level peers);communications at this level were thus designated secondary needs.

The problems for the Traditionalists were found to have two distinct levels. First, breakdowns in their ability nurture connections with their departmental/organizational level peers had dire consequences for the relationships that they had with those individuals. Some Traditionalists expressed a sense of confusion over what others in their department were currently working on; in extreme cases, the identities of some departmental peers were not even known. So, not only were there potential direct implications of not being able to stay in sync and in touch with one's departmental peers, but there were found to be indirect implications that had potentially detrimental repercussions for their project work, as well.

Project Goal 2: Come up with solutions to the problems faced by mobile professionals

As mentioned above, a second set of interviews was conducted with highly connected professionals. Findings from these interviews were used as inspiration for coming up with solutions to address the problems of the Traditionalists in keeping in sync and in touch with their departmental- and/or organizational-level colleagues. These were: (1) integrating the use of information filters (e.g., RSS feeds) into any solution; and (2) supporting one-to-many or many-to-many communication.

Next, a set of design six requirements (see the table below) was used to provide a foundation for some possible concept solutions.

After a series of analyses, and a couple of iterative design cycles, three concepts were developed into short illustrative movies; these were: (1) Ping, (2) Status Sharing tools, and (3) Bulletin Board. All three movies can be viewed here.

The next section covers the extent to which each of

the three concepts met the design requirements, according to findings from the

evaluations.

Assessment of the validity of the concepts…

1. With respect to the requirements
   

   

Comparison of the concepts to the requirements, including how those requirements were met (at least in part); * = This item is starred because it was the tool that was perceived to require the most effort by all those who took part in the final evaluation; however, results indicated that the degree of effort required was still perceived to be low, on average.

As shown in the table above (which is best viewed in its own browser window), it can be argued that – where applicable – all of the tools met all of the requirements. It is important to note that – because the solutions presented were early-stage concepts – it would be hard to concretely assess the effort involved in setting up each tool. Moreover, the evaluations were mainly meant to tap the extent to which the users valued the concepts themselves (e.g., did they mind sharing information regarding their activities and whereabouts, and similarly, did they also like to have access to this information from their contacts?) Therefore, a second requirement dealing with the need for a low-effort set-up of the system was left out of the table.

1. With respect to staying in sync
   

Both those interviewed and those who filled out the online evaluation indicated that the Bulletin Board and Status Sharing tools had the potential to help them stay up-to-date with their colleagues, to an equal degree. Ping was also thought to be a useful in sync tool, albeit to a slightly lesser degree than the other two concepts.

1. With respect to staying in touch
   

Although all of the concepts were considered to be adequate in touch tools, the Bulletin Board was considered to be the tool that was best suited to this goal. Both those interviewed and those who took the online evaluation noted its potential for sharing more personally-relevant information.

General reception of the concepts

According to the final evaluation, those interviewed appeared to prefer the Ping concept overall. With respect to the online evaluation, both Ping and the Status Sharing tool came out on top. Although these tools were not perceived to be the best tools – hands down – for staying either in sync or in touch, they were considered to be the easiest to use; were thought to have the lowest cost-to-benefit ratios; and, would require the least amount of time. Thus, it can be argued that requirement #3 (above) was given the most weight when assessing these concepts. Interestingly, the concept that was considered to have the most combined in sync and in touch potential – that is, the Bulletin Board – was the least appreciated overall, across all evaluations.

Future directions

Based on findings from the evaluations, it was suggested that either the Ping or Status Sharing tool be considered for further development. In the end, the problems faced by the Traditionalists can be thought of as large – but given their preference for simple, low-effort tools it is concluded that only very modest solutions will be adopted readily. The potential benefit in providing modest solutions, however, is that they might lower the threshold sufficiently enough so that the Traditionalists can meet both primary and secondary needs effectively.

Some thoughts on blogging...and an update

First, some reflections on this blog

I am currently in the writing phase, wrapping things up and writing a first draft of my final report. It's not possible to overstate the utility in maintaining a blog during a large project such as this one. Although the frequency with which I updated this blog was often spotty -- and certainly there are some posts that are more indicative of my own mental meanderings than anything else -- there remains much to draw on as I write up the project in final form. I'd recommend to anyone working on a lengthy project to do the same. As far as I can see, there's no reason not to. The cost of writing a blog entry while in the midst of the project pays off when it comes time to present one's findings to others -- at that point, it's a huge time saver. And, if issues related to privacy are a concern, it's possible to adapt the security controls accordingly.

An added benefit has been receiving feedback from my posts -- both online and in person. Although the volume of these comments -- especially those posted online -- has been fairly low, I still think blogging has been a worthwhile endeavor. Besides getting insightful feedback, I now have a repository of project milestones that I can easily refer to. This blog has ended up serving as a kind of memory for the project itself.

So, I'm definitely a convert to blogging. Whether I can be as open about my work in the future is questionable (and unlikely) -- I am in a unique position here at Future Workspaces because Novay is such an open organization, and because the nature of the project itself.

And now a brief update

During these past four weeks, I've made three short movies meant to illustrate the three concept ideas that were highlighted in the last series of posts (i.e., Ping, the Easy Desktop Updater + the Hover Status Tool, and Bulletin Board).

Here are those movies:

1. Ping

2. Status Sharing Tools (aka, Easy Desktop Updater + Hover Status Tool)

3. Bulletin Board

They're all quite simple in their presentation of the concept ideas, but I've been asked a few times how I made them. The first two (Ping and Desktop Updater/Hover Status) were made in Flash using hand-drawn sketches that I scanned and touched-up in Photoshop. The Bulletin Board movie is simply a PowerPoint presentation that I put together with a few more of my sketches and recorded using Camtasia.

I showed these movies during a series of one-on-one interviews with mobile workers. I spent a decent part of a couple of weeks traveling around to conduct the interviews with 5 mobile workers. I had them fill out the same questionnaires that I used during the focus group sesssion and used it as a basis for a in-depth discussion of the concepts. I also had them complete an exercise like the one from the focus group, in which they had to rate their overall impressions of the concepts using stickers.

At the moment, I'm in the process of writing up the results of the last round of interviews for the final report. In tandem, I've also posted an online evaluation of the three concepts. I'm trying to recruit -- through various means, such as twitter, Facebook, word-of-mouth, and of course through my colleagues at work -- mobile workers to take the evaluation. Thus far, many have watched the videos, and few have actually completed the questionnaires. I'm giving the evaluation a few more days, though -- and keeping my fingers crossed.

Evaluating Sketchy Ideas, Part 4: Focus Group Questionnaire Results

This post is the third to do with a recent focus group which was held on the topic of some very basic and rough concept ideas that I had come up with. It's also the last installment in a series of posts concerning the earliest concept development phase of my project.

--------

Questionnaire Items

The questionnaire items were meant to gauge each participant's reactions to various aspects of each of the four concepts.

All participants were presented with the same 10 questions -- and asked to rate their answers on a 5-point Likert scale -- for all four concepts (i.e., Ping, the Bulletin Board, an Easy Desktop Status Updater, and Hover Status). Using the same questions for each concept allowed for direct comparisons across concepts.

Questions were adapted from a questionnaire used during a prior Future Workspaces assessment.

A list of the questions used:

1. I would use this tool frequently and actively.
2. I think this tool looks easy to use.
   
3. This tool would help me to exchange information with my colleagues.
4. This tool would help others keep up-to-date with what's going on with me.*
5. This tool would help me keep up-to-date with what's going on with others.*
   
6. This tool would help me to get to know my colleagues more personally.**
7. This tool would take up too much of my valuable time.
8. Using this tool, I would be in control of what others can learn about me.
   
9. This tool could provide benefits to my work.
   
10. The benefits appear to outweigh the costs of using this tool.

* - These questions have to do with staying in sync.
** - This question has to do with staying in touch.

Responses were coded as follows:

• Strongly agree = +2 points
  
• Agree = +1
  
• Neither agree nor disagree = 0
• Disagree = -1
• Strongly disagree = -2

Results

Here is a graphic capturing the reactions of the four focus group participants to the four concepts (no doubt it's best viewed in its own web browser tab):



Discussion

Given the small number of data points statistical analyses would have been inappropriate in this case. However, as the graphic clearly shows, the questionnaire was a useful tool for assessing participant attitudes.

It is noteworthy to mention that the tools rated the highest overall (see the previous post for the complete results, or the graphic below) were associated with perceptions of having a higher benefit-to-cost ratio (i.e., Easy Status Updater and Hover Status). Not surprisingly, these were also the tools that people said they would use more frequently (in addition to Ping).

On the flip side, the Bulletin Board was considered to be the only tool that could potentially take up too much of the users' time.

Two factors seem to have driven people's overall rating of the concepts -- that is, time and the perceived presence of additional benefits. I'm going to take some time to justify and further explain this claim, so bear with me.

Here again is the graphic showing the participants' overall distribution of bonus and penalty points for the four concepts:


Overwhelmingly, the most poorly rated tool was the Bulletin Board.

But, if one examines the questionnaire results more closely, this rating has nothing to do with its potential for keeping colleagues up-to-date with one another. Pooling the results of the two up-to-date questions, the Bulletin Board actually out-rated the Hover Status tool or Easy StatusUpdater. Likewise, it also rated higher than any other tool when it came to potential for keeping colleagues connected.

On every other measure, the Bulletin Board drew mostly neutral responses.

In the end, the consensus of the business developers was that they did not find status updates useful to their work.

Perhaps, as I mentioned in the last couple of posts, this is because -- for the most part -- they work independently rather than in teams.

Although both the Easy Status Updater and the Hover Status tools focus on status updates, the effort expended in updating one's own or viewing an other person's status is very low -- one could argue, sufficiently low enough to make the use of such tools more acceptable.

----------

Given the findings from the expert session and focus group, I've begun a second design iteration. The next few posts will cover aspects of the redesign.

Evaluating Sketchy Ideas, Part 3: Focus Group Discussion and Rating Results

I recently held a focus group session with four in-house business developers. The goal was to get some feedback on the merits and weaknesses of a few rough concept ideas.

For more information on the concepts themselves and set-up of this focus group, please see the last post.

In this post I'll cover the outcome of the discussions which were held during the focus group, as well as the participants' overall impressions of the concepts. I'll wait until the next post to cover the questionnaire results, as those conclusions are sufficiently long enough to warrant their own post.

Discussion results (by concept)

I presented each of the four concept ideas (Ping, Bulletin Board, an Easy Desktop Status Updater, and Hover Status) in turn.

While presenting the concepts, I prodded the participants to engage in discussion by asking the following questions:

- Do you think this [concept] would benefit your ability to stay up-to-date with your distant colleagues? Why or why not?
- Do you think this [concept] would benefit your ability to stay connected with your distant colleagues? Why or why not?
- More generally, would it be useful to you? Why or why not?

This discussion was meant to supplement a questionnaire that each participant was asked to fill out following the presentation of each concept.

Some select comments made by the participants with regards to each of the four concepts are listed below, as well as some general trends.

Ping

- I would try it around noon, to see who is [around].
- It would be more useful for times when [one] is out-of-the-office.
- It seems to be more beneficial for people who work outside of the organization. Like for instance, when I am in the Hague. It would be useful to stay connected with main stakeholders when I am in that area of the country.
- As to staying connected or up-to-date, the application itself doesn't tell you much. These are things you would learn in person, after having used the application to meet up.
- By show of hands, all said they would use it.
- As to how frequently it would be used, this result was more mixed because they said it's use would be contingent upon the situations that happened to arise. [This point illustrates the opportunistic nature of the application.]
- Would like the ability to Ping groups of people, not just individuals.
- Would prefer a mobile web-based application (rather than have the operation of the application be contingent upon SMS*).

*It is important to note that one of my design constraints was explicitly not to build an application usable only by individuals who own smartphones.

Bulletin Board

- Only one of the group said that he uses an application in which he regularly updates his status (in this case, twitter).
- Everyone else found the act of status updating dubious (i.e., it is all about people talking about the sandwiches they are eating; there is no useful information being shared, &etc). Important note: These individuals did not actually have first-hand experience using such applications (e.g., twitter).

More specific quotes:

- I would update my status, but only if it was done automatically, from my calendar.
- As a sender, I am not sure that I would want to update my status all of the time.
- I am not interested in what other people are doing.
- It should be a mobile application, too -- but one that does not rely solely on SMS.
- There is too much information coming to me already -- I would need a filter.
- I do not want to know about the status of 100 different people, but for some specific people, yes. For instance, for a colleague that has been having personal difficulty lately, it is good to know if he is okay when I don't see him. But this would mean the application would need have some kind of a filter to get just his specific information only.
- For the more information-rich version of the Bulletin Board: If the idea could be adapted so I (and the people in my group) just send short messages for updates rather than spend three hours/week writing reports, I would use it because it would save me time. I don't like giving status updates, but if it helps reduce the work time by replacing all of this report writing, then I would do it.
- One of the participants said he would use the more information rich version (i.e., the one quoted directly above); for the basic version, it would need to include automatic updates as an input method to be more acceptable.

Easy Desktop Status Updater
- Questions regarding to where the status would be channeled (e.g., twitter, Bulletin Board, LinkedIn, &etc?)
- One person was very keen on using it only to have his status automatically updated.
- Three out of the four people said they would use it.
- Wants to be able to filter who gets what status information (by, for example, sending some status updates only to his closest work group and not the entire organization).
- Again would prefer a mobile web application (does not like the use of SMS to update one's status), because he said that it would be more convenient for when people are on-the-go.

Hover Status
- All said that such information, in the form presented, would be useful to them.
- Given feedback from a prior session, they were also asked whether the presentation of the information might prove "irritating"; they all said that they did not perceive that to be a problem.
- Also would like to get location-based information on their contacts which would be included in the hover stats.
- One said he likes the idea of "pulling information" to himself.

Overall impressions of the concepts

At the end of the focus group session, each participant was asked to give their overall impression of the concepts by distributing five bonus points (green stickers) and five penalty points (red stickers) among the four concepts. They could distribute the points however they wished.

Here are the results of that exercise:



As is evident, the Hover Status tool was the most enthusiastically endorsed. Ping and the Easy Desktop Status Updater received neutral-to-moderately positive ratings. Overwhelmingly, the Bulletin Board was given a poor rating by everyone.

It's important to keep in mind that these ratings aren't nuanced -- in other words they're gross-level endorsements or dismissals of the concepts as a whole.

Here are some of my thoughts on the best and worst ratings:

Bulletin Board

With respect to the Bulletin Board, two things should be kept in mind:

1. The business developers tend to coordinate in their work -- rather than collaborate with their colleagues. Moreover, much of their work (e.g., hunting down leads) is done independently. This might explain why there was a general lack of interest in knowing what their colleagues were up to.

However, all of the other mobile workers that I interviewed worked in project teams and engaged heavily in collaborative work. For this reason, there might be more of an interest in a tool that provides them with information about their distant colleagues. Also, aspects of the tool that were found to be disagreeable to the business developers were explicitly asked for by some Traditionalists that I had interviewed (e.g., a "News Flashes" feature).

2. I asked them to react to the information-rich version only; it is possible they might have been more positive about the basic version (which included only status updates). However, because of their group-level dismissal of the practice and use of status updates, it seems doubtful.

I can't help but wonder how others, who work in distributed project teams and in decentralized organizations might view the basic version.

Hover Status:
This tool was rated the highest, and for good reason. It is lightweight and useful -- especially in those moments when one must decide what strategy is best for contacting another person.

However, I think more needs to be done to rework this tool so that it helps colleagues in large, distributed organizations interact. It seems to me that this tool is best used primarily with respect to one's known colleagues. How this will help a person discover what is going on within the department or larger organization remains unclear.

--------

As mentioned previously, the results for each of the four 10-item questionnaires will be covered in the next post.

Evaluting Sketch Ideas, Part 2: Recipe for a Focus Group

Early in the week I held a focus group with some potential users. In total, four business developers showed up for the session (out of the six that replied, two were no-shows -- I won't dwell on what this means regarding my popularity). ;)

This post will cover the materials and set-up of that focus group. In the next post, I'll cover findings.

About the Participants

The business developers share some key characteristics with the Traditionalists that I had initially interviewed. Both groups find themsleves oftentimes out-of-office, spending a large portion of their working days criss-crossing the country to engage in meetings.

But there are also some important differences too. The Traditionalists devote a sizeable percentage of their jobs to project work. In focusing primarily on project work, it becomes critical for them to manage their communication channels in such a way as to support their ability to collaborate within their project teams effectively.

Business Developers, on the other hand, work more independently. Their main task is hunting down "leads". Certainly open communication with their intra-organizational peers is a benefit to this kind of work (e.g., it's important to know who is hunting down what lead and the outcome of that effort) -- but, it does not require a need to collaborate at the same level of intensity that project work does. Instead, it could be argued that it requires business developers to coordinate more often with their colleagues.

I mention this difference because it has some important implications for the results of the focus group session.

The Ingredients (aka, Materials & Set-up)

1. Presentation - You can view it here (however, it does look better if you follow the link to view it in full screen mode):Focus Group 27-7-09 - Concept Ideas x 4 Four very rough concept ideas were presented. An important note: If you view the slides, the following critical point might not be evident -- that is, the Bulletin Board concept is given two distinct visualizations. One is a basic version, that only shows status updates, short messages and mulitmedia, and is filterable by group (e.g., project team, department, organization, and all). Another, more information rich version of the Bulletin Board is also shown. This second version includes quick news updates, document sharing, and a bare-bones forum (in addition to everything available in the basic Bulletin Board -- it is basically the Daily Outlook concept from a previous post, however, the application need not be restricted to Outlook in this case). In addition, it supports threaded discussions. The criteria by which I wanted them to differentiate between these two concepts was whether or not they benefited from the added information -- or whether the additional information might pose too much of a cost (e.g., in the form of information overload) to them. Whether they received the information in the form of a desktop application, online, or in Outlook mattered less than whether or not they preferred the full version of the concept, or a scaled down version of it.
2. Cue cards - The explicit goal of this project is to: Support mobile workers in their ability to stay in sync and in touch. However, these terms -- in sync, in touch -- are easily confusable. So we went with "up-to-date" and "connected" for the focus group session because everyone has an intuitive sense of what these synonomous terms mean to them. The purpose of the cards was to remind the participants of the criteria that I wanted them to use in assessing the concepts. So, the basis of the discussion for each concept revolved around these basic questions: Would it help you stay up-to-date with your colleagues? And, would it help you stay connected to them?
3. Questionnaires - Following the presenation of each of the four concepts -- that is, Ping, the Bulletin Board, an Easy Desktop Status Updater, and Hover Status -- participants were asked to fill in a brief 10-item questionnaire. (A complete list of the questions will appear in a future "Results" post).
4. Bonus & Penalty Points - Each participant was given five green dots (or bonus points) and five red dots (penalty points). They were asked to distribute the points among the four concepts as they saw fit (e.g., if they wanted to assign all five bonus points to one concept, that was perfectly acceptable).
5. Food - What is a focus group session without food?

Results...

...Forthcoming in the next post.

Evaluating "Sketchy" Ideas, Part 1: Expert Feedback

Last week I met with a few in-house experts (i.e., members of the Future Workspaces team) here at Novay and asked them to give feedback on some "sketchy" concept ideas.

I went to them with six rough ideas for solutions to keeping mobile workers -- specifically the "Traditionalists", as I've come to refer to them -- in sync and in touch. I showed them a series of sketches illustrating each concept.

A very brief description of each of these follows:

1. Ping

Location-based awareness of colleagues that does not require the use of a GPS-enabled smartphone* to work. It helps mobile professionals check to see when their geographically-distributed colleagues are nearby so that they can then contact one another for an ad hoc, face-to-face meeting.

2. Bulletin Board

A means of sharing status updates, short messages, and multimedia that is filterable by group (e.g., "Project Team", "Department", or "Organization"). The application would act as a sort of dynamically updating desktop wallpaper.

3. Daily Outlook

An add-on for Outlook that configures the Outlook Today page and is meant to support the sharing of: 1. Status updates; (2) News Flashes; (3) Document sharing (via the use of links); and (4) a Q&A section (basically, a bare bones forum).

4. Outlook-based Blogging Tool

An add-on for Outlook that makes it easy to post to blogs as well as track, read and share blog posts from within Outlook (all without having to open a web browser).

5. Free Five

Personnel within a company are matched based on mutual professional interests. Individuals can make available "five minutes" (or more) of free time to meet with previously unknown colleagues, thus earning points. Earned points can be traded in a type of bartering system that allows participants to "buy" time slots from one another. The basic idea is to foster the making of new connections within large, decentralized organizations; help find experts; and, open new opportunities for collaboration.

6. Hover Status

By hovering over the name of a colleague -- for example, when composing a new email message -- a small window appears that displays that contact's latest status update (e.g., as fed from their calendar). This information can be used to help one decide which method is best for contacting an individual at that moment -- be it email, phone, or face-to-face communication.

_____________________________

While they were evaluating the concepts, I asked the participants to keep in mind the six design requirements (as outlined in a previous post). As a reminder of those requirements, I printed off some A4-sized cards listing the requirements and gave one to each participant. That way they could easily refer to the requirements -- and assess the extent to which each concept idea addressed them -- as they gave their feedback.

For each concept, participants were asked to give three pros, three cons, and three improvements for furthering the design. I used this feedback as the basis for a second design iteration (I plan to share some of this feedback in future posts). Then, I had them rank order each concept. Each first place ranking was given a score of six points, a second place ranking was given a score of five, and so on.

The outcome of the rank ordering is as follows:
1. Ping - 20 points
2. Hover Status - 18 points
3. Daily Outlook - 16 points
4. Bulletin Board - 14 points
5. Outlook Blogging Tool - 11 points
6. Free Five - 6 points

I used the feedback and rank orderings to modify the top four concepts. The following week I held a focus group with users. There I presented the Ping and Hover Status concepts, along with a new concept (which also came out of the expert assessment) -- an easy status updater that sits on the desktop. The Bulletin Board and Daily Outlook concepts were rolled into a single idea.

In the next post, I'll share more about this focus group.

-----

*I realize in the example here I've actually drawn a smartphone. That's simply a function of the fact that they are easier to draw (e.g., fewer buttons, paired with a nice big display!) As an aside: Although smartphone sales are on the rise -- according to a recent report, despite the fact that there has been a sharp increase in the number of smartphones sold -- they still only make up approximately 13.5% of the mobile phone market. And these numbers are reflective of the mobile professionals I talked to -- that is, the majority of them didn't own smartphones.

Design Philosophy

My coach suggested a great idea -- that is to come up with a detailed design philosophy. I'm sharing it here because -- although some of the content has already been published on my blog -- I think it nicely illustrates the logic underlying the user requirements I have chosen.

Curtis 2009 Design Philosophy