Usability Analysis of a Client Knowledge Portal (4 of 7)

CONTENTS
Introduction
Expert review
Task analysis
Usability testing results
Survey results
Appendix A
Appendix B

USABILITY TESTING
The complete list of tasks which users were asked to perform can be found in Appendix A. Since most of the tasks were successfully completed by all of the subjects, only the tasks on which subjects demonstrated difficulty are discussed in detail here.

Overall, subjects were able to successfully perform the following tasks without any difficulty:

1. Log in to the site.
   
2. Look up Dave’s email address.
   
3. Find out who is in the digital branding group in San Francisco.
   
4. Change your phone number in your profile.
   
5. Go back to home page.
   
6. Do a search on marketing.
   
7. Select file called “current status”. Open it. Close it and go back to site.
   
8. Find out where in the country John Quirk is located.
   
9. Do a search on a file called “testing.”
   
10. Find out the author’s email address.
    
11. Looking in the technology channel and look for white papers that look interesting to you.
    
12. If you find a listing of white papers that look interesting to you, add this page to your favorites.
    
13. Find another white paper in technology that looks relevant to you. View the white paper.
    
14. Performa search on another discussion group.
    
15. Remove the favorite that you added earlier.
    
16. Go to a discussion group within technology that looks interesting.
    
17. Add a message to this discussion group.
    
18. Look for information about deliverables associated with the Client process.
    
19. Imagine it was a month from now and you wanted to find a document you just published. How would you do this?
    
20. Leave the Knowledge Portal site.
    

Overall, subjects exhibited some degree of difficulty on the following tasks:

• Change your password.
  
• Perform a search on a discussion group that interests you.
  
• Find a description of the define stage in the Client process.
  
• Find information about the deliver stage.
  
• Submit a new project request.
  

Some users had difficulty changing their password because they thought that the change password tool should be located under profile. They had trouble performing a search on a discussion group because they had trouble locating discussion group search and because they had difficulty locating groups with content. They had trouble navigating through the Client process channel to locate specific content.

Overall, subjects exhibited a great amount of difficulty on the following tasks:

• Look for documents pertaining to Client’s work with Pepsi.
  
• Publish a document called “testing” to the documentation about Bath and Body Works.
  
• Go to home and find the document that you just published.
  
• Publish a file called “testing” using tools (or through categories, whichever method they did not use before).
  

Users experienced a large degree of difficulty finding content on Pepsi because they used search, which did not display any information about Pepsi. This was because no documents have been published on this client. Even though there is no content on Pepsi, search should let the user know that there is a section of the site devoted to this client. Users had trouble publishing documents to the site (see task analysis for detailed description of where users exhibit difficulty) and locating documents which were published as “offline.” They had trouble publishing by navigating to the relevant category through the site and through using tools.

A primary usability goal for the next phase of the Knowledge Portal is for users of the site to have no difficulty or a minor degree of difficulty on all of the tasks, which are challenging in the current design.

Employee Satisfaction Survey results
Subjects were asked to complete a short survey after the testing session (see Appendix B). It should be noted that while the highlights of the results are summarized below, a more comprehensive survey administered to a larger sample of employees would be necessary to obtain more conclusive information regarding employee satisfaction with the site. Such a survey is recommended after phase two of the site comes online because by that time a larger number of employees will have been using the site for an extended period of time.

The eight subjects who completed the survey ranged from managers to human resources staff to technical staff. Most spend more than 10 hours on the web per week, and only one had no experience using an intranet. Half use Internet Explorer and half Netscape Communicator, all recent versions (version 5 and versions 4.0 and above, respectively). Two subjects had never used the Knowledge Portal before and the remainder uses it at least 1-2 times per week.

Subjects were asked if they had used the Knowledge Portal to accomplish any of the following: to read the current stock price, read or obtain information about a document, publish documents, look up another employee’s contact information, read about current Client news, look up another employee’s area of expertise, download templates or presentations, read or post messages in a discussion group, locate specific project information, or acquire general knowledge about Client (example: learn about Client process). None stated that they had used the Knowledge Portal to read or post messages to a discussion group, while the most common uses were to read current Client news and to look up employee information.

When asked what information/functionality subjects would like to see on the site and what tasks they would like to be able to perform that would make their day-to-day activities easier, the following responses were collected:

• Org chart
  
• Job openings by geography
  
• Phone number lists for people in different locations
  
• Recent sales “wins”
  
• Expert cross-reference that would allow one to find an employee with expertise in a given area
  
• Sales channel where sales reports may be accessed and employees would have different levels of access to information based on position in company
  
• HR info including timesheets, PIMS, vacation hours, overtime hours, benefits information, 401K
  
• Logout
  
• Calendar function (like Netscape calendar)
  
• Ability to set up conferences/teleconferences
  
• Ability to search employee’s resumes by standardized keywords.

When asked what they like best about the site, the responses were the following:

• “clean” and simple look and feel
  
• the site organization
  
• the simple layout
  
• the ease with which information can be located on the site.
  

Suggestions for improvement are the following:

• A more comprehensive search function
  
• A site map
  
• The upload file page should be combined with the publish file page.
  

Subjects also answered a series of questions intended to address overall satisfaction with the site. The data is summarized below in Table 1 and has been collapsed across disagree/strongly disagree and agree/strongly agree. The results suggest that areas for improvement are making the site easier to learn to use, improving search, making the actions required to navigate through the site more consistent, reducing the number of clicks required to locate information, and making it more apparent where in the site documents can be published. The noteworthy results are in bold.

Table 1. Overall satisfaction ratings

Appendix A— Usability testing task list
I am going to ask you to go through a number of tasks. I would like to emphasize that you are not being tested; rather we are interested in testing the site. What I want you to do is to think out loud as you perform the tasks. This may seem unnatural, but I want you to verbalize your thought processes as you perform the tasks.

1. Log in to the site.
   
2. Look up Dave Stanton’s email address.
   
3. Find out who is in the digital branding group in San Francisco.
   
4. Change your phone number in your profile.
   
5. Go back to home page.
   
6. Do a search on marketing.
   
7. Select file called “current status”. Open it. Close it and go back to site.
   
8. Find out where in the country John Quirk is located.
   
9. Change your password.
   
10. Do a search on a file called “testing.”
    
11. Find out the author’s email address.
    
12. Perform a search on a discussion group that interests you.
    
13. Performa search on another discussion group.
    
14. Submit a new project request.
    
15. Looking in the technology channel and look for white papers that look interesting to you.
    
16. If you find a listing of white papers that look interesting to you, add this page to your favorites.
    
17. Find another white paper in technology that looks relevant to you. View the white paper.
    
18. Remove the favorite that you added earlier.
    
19. Go to a discussion group within technology that looks interesting.
    
20. Add a message to this discussion group.
    
21. Find a description of the define stage in the Client process.
    
22. Find information about the deliver stage.
    
23. Look for documents pertaining to Client’s work with Pepsi.
    
24. Look for information about deliverables associated with the Client process.
    
25. Publish a document called “testing” to the documentation about Bath and Body Works.
    
26. Go to home and find the document that you just published.
    
27. Imagine it is a month from now and you wanted to find the document you just published. How would you do this?
    
28. Publish a file called “testing” using tools (or through categories, whichever method they did not use before).
    
29. Leave the Knowledge Portal.

Appendix B—Customer satisfaction survey

1. What is your job title and responsibility?

2. For approximately how many hours do you spend on the web, per week?
_____0-2 hours
_____3-5 hours
_____6-10 hours
_____more than 10 hours

3. Indicate your degree of experience with other knowledge portal/knowledge management systems:
_____No experience
_____ I have some experience.
_____ I have a great deal of experience.

4. If you have experience with other systems, can you offer any feedback regarding that system in relation to our own Knowledge Portal?

5. What web browser do you use and what version?

6. Indicate your degree of experience with the site:
_____ I have never used the site.
_____ I use the site 1-2 times per week.
_____ I use the site 3-4 times per week.
_____ I use the site daily.
Other:___________________

7. I use the site in order to (place an X next to all that apply):
_____ Check the current stock price.
_____ Read or obtain information about a document.
_____ Publish documents.
_____ Look up another employee’s contact information.
_____ Read about current Client news.
_____ Look up another employee’s area of expertise.
_____ Download templates or presentations.
_____ Read or post messages in a discussion group.
_____ Locate specific project information.
_____ Acquire general knowledge about Client (example: learn about Client process)
Other: ___________________

8. What information/functionality would you like to see on the site that doesn’t already exist?

9. What tasks would you like to be able to perform on the site that you are currently unable to perform? Can you think of other capabilities that would make your day-to-day work activities easier?

KNOWLEDGE PORTAL#1
Name: Date: Cond:

10. Describe in detail any problems that you encounter when you use the site. How often does the problem occur? Is it an easy or difficult problem to overcome? Are you repeatedly bothered by the problem?

11. What do you like best about the site?

12. Do you have any suggestions for improvement?

Based on the Client Knowledge Portal web site, please rate each of the following statements on a scale from 1-5. Place a check in the appropriate column.

<< 1 | 2 | 3 | 4 | 5 | 6 | 7 | >>

Copyright 2002 Chromosome22. All Rights Reserved.

download the WORD .DOC (184k)