Up Down Left Right A Usability Postmortem

1
Up Down Left RightA Usability
Postmortem
Tony Rado Executive DirectorGlobal Quality
Assurance, SOEtrado_at_soe.sony.com
Paul Venuti DirectorDevelopment Services,
SOEpvenuti_at_soe.sony.com
www.soedevservices.com
2
Up, Down, Left, and Right. A Usability Postmortem

• Why usability?
• Usability statistics
• Organizational relationships
• Benefits of in-house lab
• Challenges of in-house lab
• Usability lifecycle
• 5 Phases
• 18 month trial and error
• Lessons learned
• Vivox (EQ, EQ2, SWG)
• Free Realms
• DCUO, Agency

3
A Usability Postmortem
Why Usability?

• Designers, Developers, Producers, and Executives
  watch target demographic play games
• Increase time to market
• Controlled environments
• We didnt have solution in place to extract
  critical information from target audience
• Issues with beta testing Misinformation Critics
  of game Not targeting audience but accepting
  audience

4
A Usability Postmortem
Usability Stats - May 08 to August 09

• 44 Usability Tests
• 1 Focus Test
• 2 Time-to-Task Data Collection
• Total of 473 Candidates
• 60 Candidates Competitive Analyses
• 47 Candidates Concurrent Vivox
• 23 Candidates Web Presence
• 18 Candidates The Agency
• 177 Candidates Free Realms
• 7 Candidates 3rd Party Contract

• 36 Candidates Free Realms First Impression
  Focus Test
• 71 Candidates Free Realms Combat and Racing
  Data Gathering/Balance
• 34 Candidates DCUO

5
A Usability Postmortem
Organizational Relationships

• Development Services
• Quality Assurance
• Development
• Marketing

6
A Usability Postmortem
Benefits of an In-House Usability Lab?

• Quality equates to success
• Quality buy-in from all vested parties
• Real time monitoring
• Ability to record A/V on site

• Educative
• No limit on testing iterations
• Immediate feedback / Immediate resolution

• Best practices call for between 8 and 13 of
  project budget be spent on Usability Jakob
  Nielsen, Alertbox, 7 January 03
• Not an ROI in classical sense
• Project cost measured in terms of money
• Usability measured in terms of increased usage,
  efficient use, high user satisfaction

7
A Usability Postmortem
Challenges of Building In-House Usability Lab?

• Changing Culture
• Shrinking Budgets -
• Infrastructure
• Technological Challenges
• Distance between lab and monitoring room
• Video
• Audio
• Real time synced camera and game play feed
  support for satellite studios

8
A Usability Postmortem
Embedded Video
9
A Usability Postmortem
Consultation Phase
Initial and Ongoing Communication

• Job request
• Planning meeting
• Technological challenges
• Define measurables

Lesson Learned

• ISSUE Requesting party disagreement
• Goals
• Data points
• Cannot please and appease everyone
• RESOLUTION Created one point of contact for
  which all communication funneled through

10
A Usability Postmortem
Recruitment Phase

• Generalized recruitment categories for both
  Internal and External candidates include
• Age
• Sex
• Hr/Wk playing games (Online/PC/Console)
• Favorite games
• Specialized 3rd party contract recruited an
  ethnic target for European/South American game

11
A Usability Postmortem
Internal External Candidates
Internal Candidates

• Cost Benefit Recruitment and incentive costs
• Initial use of internal candidates were employees
  then employee children
• Recruited internal candidates with same
  requirements as if recruiting externally

External Candidates

• Free Realms iterative Created database of
  employee children and friends
• Treated employees children as external
  candidates
• Recruited Boy Scout Troops, local kids
• Exhausted internal children pool and began
  external recruiting
• Time saved by utilization of external recruiting
  company

12
A Usability Postmortem
Candidate Bias
Internal Caroline Jarrett, in regards to internal
candidate bias, states that they may be your
target users, in which case they are perfect
Direct observation doesnt lie
Society of Technical Communications Usability
Special Interest Group March 28, 2009

• External
• Getting paid
• Desire to please Embedded Video
• On-site testing

13
A Usability Postmortem
Recruitment Phase
Lessons Learned - Internal

• ISSUE Internal bias would taint data due to
  desire to please or be less than honest
• RESOLUTION Ensured all candidates understood
• Setting of quality bar equated to job security!
• Pride in having input affect change
• Opinions would not affect job security!

Lessons Learned - External
ISSUE Candidate no-shows! Sometimes extended
testing cycle by one or two days RESOLUTION
External recruiting company double confirmation
including on testing day, Double book.
14
A Usability Postmortem
Test Plan Creation Phase
Foundation Plan to Field Measurable Data Points

• Contact Information
• Scenario Questions
• Deliverables
• Defined Roles

• User Profile Information (which ones were used)
• Pre-test Questionnaires
• Scenarios

Lesson Learned
ISSUE The Youre Doing It Wrong
syndrome RESOLUTION Test plan approval
15
A Usability Postmortem
Test Plan Example
16
A Usability Postmortem
Test Plan Example
17
A Usability Postmortem
Testing Phase

• Technicians Ensure environment stability and
  smooth technological sailing
• Moderators Conduct testing, exhibit patience
• Analysts Observe, gather data, ensure test
  follows plan

Lesson Learned
Dont give children sugar before a test
Embedded Video
18
A Usability Postmortem
Testing Phase
Lessons Learned
ISSUE Communication between monitoring room and
Ulab RESOLUTION Bluetooth headsets, IM, mIRC,
SMS ISSUE Not working as intended (Mechanics,
New Build) RESOLUTION Full test plan
walkthroughs before approval process and
walkthroughs on test day
Jakob Nielsens first rule of Usability Dont
listen to users!!!
Embedded Video
19
A Usability Postmortem
Post Test Reporting Phase
Largely Conducted by Analysts

• Top Ten Issues List Executive Summary
• Data gathering
• Video highlight generation
• Design suggestions
• Prioritized screenshots of Usability severity
  issues
• Report generation
• Deliverable
• Postmortem

20
A Usability Postmortem
Post Test Reporting Phase
Sample Free Realms Mini-Game Report
21
A Usability Postmortem
What We TestedWhat We Learned from It
22
A Usability Postmortem
Vivox Voice Chat Implementation
Embedded Video
Lessons Learned
ISSUE Spatial constraints became apparent.
Candidates were talking to one another RESOLUTION
Volume in the headphones was increased ISSUE
Multiple stakeholders driving different
agendas RESOLUTION Communication conduit
ensuring similar approach to static system across
multiple platforms
23
A Usability Postmortem
Free Realms
Embedded Video
24
A Usability Postmortem
Difference Between Testing Children and Testing
Adults

• Conducted research prior to beginning testing
• External focus groups observing and studying
  methods
• Reviewed past GDC presentations about testing
  with children
• Carla Engelbrecht, Teachers College, Columbia
  UniversityLittle Hands, Foul Moods, and Runny
  Noses The Ups and Downs of Researching Games
  with Kids GDC 2008

25
A Usability Postmortem
Difference Between Testing Children and Testing
Adults

• Cognitive and physical limitations in children
  have been long ago mastered by adults
• Example Free Realms Target Demographic 8-14
• Children Aged 7-12, Concrete Operational Stage
• Understand Time, Space, Another Persons
  Perspective
• Still rely on concrete objects and experiments to
  form ideas
• Take off kid gloves

Carla Engelbrecht, Teachers College, Columbia
UniversityLittle Hands, Foul Moods, and Runny
Noses The Ups and Downs of Researching Games
with Kids GDC 2008
26
A Usability Postmortem
Lessons Learned
ISSUE Confusion among parties as to whom we
were, and should, recruit, against target
demographics RESOLUTION Created User Profiles
based upon 3 candidate profiles created in
conjunction with Marketing and Development.
Embedded User Profile
27
A Usability Postmortem
Embedded Video
28
A Usability Postmortem
Lessons Learned
ISSUE Build validation prior to
testing RESOLUTION Point of contact between
studios, 3rd party client, and technicians ISSUE
Geographic challenges RESOLUTION Chat feature
for remote users to speak with San Diego office,
Remote viewing, Remote logging Result Active
participation from satellite studios
29
A Usability Postmortem
Conclusion
Greg Spence, Programmer EQ2 Obvious benefit was
ability to identify issues early on that we
wouldnt have caught until we went live. Helped
to see where players reacted and stumbled through
feature sets. Clint Worley, Producer EQ
Obvious that the way we were presenting it
(Vivox) wasnt compelling. We were looking at the
trees instead of the forest. Ryan Wells, Assoc.
Producer Free Realms It was great to be able to
make tweaks each week, make predictions about how
users would respond and see the results. Our
assumptions were often wrong and we learned just
how different our audience was from ourselves.