Usability Testing of Sandia's New Intranet Portal

1
Usability Testing of Sandia's New Intranet Portal

• Hilary Bienstock
• Usability Engineer
• Sandia National Laboratories

Sandia is a multiprogram laboratory operated by
Sandia Corporation, a Lockheed Martin
Company,for the United States Department of
Energys National Nuclear Security
Administration under contract DE-AC04-94AL85000.
2
Project Internal Homepage Redesign

• Go from static page with lots of links to portal
  interface
• Advantages
• Single sign-on
• Ability to push topical and personalized
  information
• Users can customize
• Goals
• Mechanism for keeping population informed
• Increased productivity and efficiency
• High user satisfaction

3
User Research

• Pre-Portal Development
• Ongoing Work
• During Development
• Post-Portal Deployment

4

• Pre-Portal Development

5
Techweb Before
6
In-House Research

• Time Trial Metrics
• June 2003
• Customer Satisfaction Surveys
• October 2002, January 2004
• Formal Analysis of Search at Sandia
• Summer 2003

7
Customer Satisfaction Survey

• Customer Satisfaction Survey results

8
Search Analysis Before

• 11 different types of search at Sandia

9
Search Analysis Recommendations

• Present state

• Desired state

One Place
techweb
advanced
directory
index
library
org finder
sand search
search only
techdirect
WFS
www
yellow pages
10
Literature Review

• Nielsen Norman Reports
• Adaptive Path
• Other references
• OReilly book on Information Architecture
• (Rosenfeld and Moreville)
• Edward Tufte
• Don Norman
• Jakob Nielsen
• Earl Babbie
• Byron Reeves and Cliff Nass
• Steve Krug

11

• Ongoing Work

12
Logging Site Data

• Customer feedback (mostly complaints)
• Anecdotal data
• Site metrics and clickthrough logs

13
Techweb Home Page June, 2004
Quick Weather 21,335
248,513 -52.3
Just Tell Us 503
Newscenter 11,649
TimeCard 91,910
Vac/Flex 12,969
Travel 11,354
Expense 13,356
Workbox 6,454
Conf Rm 5,782
Training 22,649
Benefits 21,969
Jobs 13,961
CPR/BRules 3,120
Security 2,953
Form 8,549
Old Teasers 31
Search 21
Person 134
NW 2,389
NPA 290
EIA 208
MTA 266
STE 257
4,975
IES 1,218
HS 348
OrgFinder 4,189
Web
Science Research frame-206 Biology-130,
Chemistry-222, Computer Science-223, Earth
Science-126, Math-167, Physics-234, Programs
Cap-111, Working Groups-75, more-156
Engineering Manufacturing frame-192
Catalogs-162, Chemicals-656, Databases-186,
Drawings-437, Guides-47, Manufacturing-213,
Procedures-71, Specialty-38, Standards-206, Test
Facilities-99, Calibration-66, more-68
2,441
1,650
Search Index 872
Index 37,220
Phone 6,950
Directory 27,039
Engineering Resource IMS-3,110, WFS-3,133,
Neevia-335, EaWeb-1,075, TBP-383, Metagroups-521,
COTS-88, Oracle-258, SciSearch-122, ProE-60,
EPs-77, SPUDS-85, NWSBU Policy-40, INSPEC-62,
E-source-, eCIS-174, CIS-95, MSDS-36,
ChiliSPICE-47, PRP-105, PCM-15, NM
Calibration-43, CenBase-45, DP-36, Material
Sel-, QC1-29, SSDVS-16, PRIME-13, CA
Calibration-43, Software Improve-18, Mfg
Process-26, Stockpile Dismantle-9, EGIS-11,
NearNet-9, LM EPIC-15, DMS-206 GRID-27, PTS-14,
PRS-90, LM21-54, Lean Six Mfg-68, NGDAS-14,
Reportville-248, XYCE-39, WebTMM-519, Trilinos-37
11,244
Yellow Pages 2,563
Gov Numbers 1,377
QwestDex 6,665
Advanced Search 561
About Me at Sandia frame-1,199 Awards-960,
Benefits-3,548, Credit Union-9,528, Employee
Con-295, Jobs-3,218, Health-820, HR-5,549,
Payroll-4,879, Perf Mgmt-313, Volunteer-131,
more-779
Information Sharing Publishing frame- 484
Classification-578, Collaboration-120, Events-78,
Image Serv-349, Lab Comm-435, Library-7,037,
Manuals/Guides-202, Outside Rel-67, Rec Mgmt-524,
more-272
Search Only 237
31,219
10,146
Search TechDirect 939
WWW 10,845
Maps NM 5,312
Infrastructure Utilities frame -611
Computing-3,307, COE-532, E-mail-2,112,
ESH-1,240, Facilities-881, Mail-272,
Phones-2,405, Vehicles-178, Security-1,032,
Web-132, more-403
Buy It Ship It Track It Move It frame-702
Computer-1,194, Eng Vendors-110, JIT-5,220,
Move-298, Procurement-2,960, Property-1,269,
Reapply-1,235, Receive-416, Ship-1,479, Store-50,
more-223
Maps CA 173
13,105
15,156
Maps Training 169
Maps Other 1,094
CCHD Alerts 2,268
Corporate Perspective frame-211 Corp
Identity-100, Corp Vision-53, CI-185,
Diversity-235, Ethics-154, Legal-79, LLT-48,
Lockheed-381, Ombuds61-, Planning-60, Prime-68,
Quality-86, SMU-83, more-144
Budget Financial frame-319 Budget-997,
Financial Report-6,153, Fin Process-278,
Oracle-8,183, SBS Job Aids-, Service Centers-114,
more-135
SNL External 2,306
1,948
16,179
NNSA SSO 434
QuickPages
6,795
Engineer 254
Manager 1,939
New Hire 2,466
Secretary 2,136
111,368 23.5
114,951 - 24.2
Request a change to TechWeb 61
14

• During Development

15
General Observations

• As an incidental part of another usability test
• Observation of co-workers, etc.
• Anecdotal evidence
• Gave us an overview of what was most and least
  important, etc.

16
General Usability Advising

• General usability advising
• I participated on 3 TWGs
• Gave usability perspective
• Team members learned to rely on me

17
Live Poll Search Boxes

• Procedure
• To solve a specific problem (1 vs. 2 search
  boxes)
• A survey wouldnt be detailed enough
• Needed high response rate
• Tested in cafeteria at lunchtime
• Brief (3 minute) interviews
• 2 interviewers, 1 recruiter
• Rewarded participants with cookies

18
Live Poll Search Boxes

• Procedure
• Showed sketches
• Asked how theyd use the sketch
• Compared with another sketch
• Feedback on preferences
• Naming of homepage
• Additional comments

19
Live Poll Search Boxes
20
Live Poll Search Boxes
21
Live Poll Search Boxes
22
Live Poll Search Boxes

• Results
• Users seemed to prefer 2 search boxes
• 1 for internal web search
• 1 for directory/people search
• We also got lots of feedback about other issues
• Advertising for the new homepage

23
Traditional Usability Test

• When site was working, but with no UI
• Investigate several potential problem spots
• Information architecture and navigation
• New placement of familiar elements
• Header and footer
• Search interface (again)
• Portlets
• Customization controls
• UI (sort of)

24
Traditional Usability Test

• Procedure
• Task list
• Observe quietly, take notes
• Both NM and CA sites
• Reward participants with time charge code and
  gifts

25
Traditional Usability Test

• Non-UI screen shot

26
Usability Test Findings

• Personalization
• Moving and minimizing is easy and intuitive
• Adding portlets is somewhat confusing
• Search Boxes
• Placement is fine
• Labels and labeling are confusing
• enter keywords
• phone
• info

27
Usability Test Findings

• Communities of Interest
• Users have no idea what this means
• Recommendation first page explaining the
  concept, could move away after users have read it
• Page setup
• Tabs across the top are not obvious enough (could
  be a UI issue)
• Commonly used links are still easy to find

28
Live Poll Engineering COI

• Brief questions (3 minutes) about content useful
  to engineers
• Both NM and CA sites
• Data was a starting point for COI content

29
Live Poll Advertising Poster
30
SearchPoint Design

• Team worked to solve problems highlighted by
  Search Report
• Results screen debate
• Results from all sources on one page
• One set of results on front page with tabs to go
  to other results set

31
(No Transcript)
32
(No Transcript)
33
(No Transcript)
34
SearchPoint Design

• Solution
• For technical reasons, chose results in tabs
• showing one set of results with tabs to go to
  other results set
• Incorporated Index results with web search
  results
• UI concept modeled after Google

35
Expert Review SearchPoint

• SearchPoint screen here

36
Heuristic Evaluation SearchPoint

• Evaluation of search from usability principles
• Ranked recommendations for functionality and UI
• Not all recommendations were taken

37
SearchPoint Heuristic Evaluation Results

• Providing multiple results sets is very useful
• Terminology may be confusing (SearchPoint,
  TechDirect)
• SearchPoint front page doesnt have tabs
• Cant get to all types of information
• Results are poorly formatted
• SearchPoint is poorly integrated into internal
  homepage

38

• Current Techweb

39

• Post-Release Usability

40
Monitoring Feedback

• Through official trouble tickets and comments
• Anecdotal
• Personal emails
• Observing users incidentally
• Making changes as necessary

41
(No Transcript)
42
Heuristic Evaluation Techweb Help

• Currently, 4 types of help
• Help Conversion from old to new homepage
• Job Aid
• Videos
• Project FAQ
• WebMentor
• Content is diverse, confusing, and poorly
  organized

43
Techweb Help Heuristic Evaluation Results

• Be more task-focused
• Reorganize content, separate out non-help
  information (in Techweb FAQ)
• Rename information (what is a job aid?)
• Get rid of poorly formatted WebMentor
• Delete obsolete information like What is a web
  browser?
• Consolidate authoring information into Internal
  Web Publishing Guidelines

44
Time Trial Metrics

• Duplicate previous test
• Same tasks, new homepage
• Gather quantifiable metrics
• After users are used to site
• Expect small loss in efficiency
• May prompt further usability testing

45
Usability Test SearchPoint

• Combination usability test and interview?
• After new SearchPoint UI is developed
• I will also work with the SearchPoint UI team

46
Revised Customer Satisfaction Survey

• Questions that help gauge feelings towards
  specific parts of the interface and the teams
  advertising effort
• Unlike old survey, more focused on product and
  less on teams customer service
• Will highlight areas for future research and
  development
• Will address team concerns that dont appear to
  merit their own study

47
Other Planned Usability

• Testing on COIs
• For benchmarking and future development
• Testing on Feedback Forms
• Do users understand the difference between
  reporting a problem and making a comment?
• Focus group of interested users
• New and useful content for development

48
Other Planned Usability

• Techweb Help
• Exploratory research on types of help needed
• Usability testing on existing help
• Other testing of new developments
• As needed
• Staying on team
• Keeping up with usability literature

49

• Current Techweb

50

• Hilary Bienstock
• hilaryb_at_stanfordalumni.org

Sandia is a multiprogram laboratory operated by
Sandia Corporation, a Lockheed Martin
Company,for the United States Department of
Energys National Nuclear Security
Administration under contract DE-AC04-94AL85000.