New York University

1
We survived! NYU upgrades from  Bb 6.2 to 8.0

• New York University

2
What will you learn in this session?

• What we learned
• What we would have changed
• Things to avoid
• Recommended upgrade outlines
• Questions welcome as we go along

3
New York University

• Largest private non-profit university in the US
• Population
• 3,100 full-time faculty 40,000 students
• 14 Schools and 2 separate IDs
• Students from all 50 states and 133 foreign
  countries.
• Campus/branch locations Manhattan, Westchester
  County, Rockland County 11 International
  locations
• Blackboard Learning System - Enterprise License
•  
• Presenters
• Project Manager - Lillian Moran
• Training Support - Shannon McPartland Kelvin
  Shivers
• Communication - Meredith Rendall
• Technical - Max Whitney

4
NYU Upgrade from Bb 6.2 to 8.0
5
Where we were where we ended up

• Then
• 6.2.3.19 also known as Version 6.2 App Pack 2,
  service pack 2, build 1
• Went as high as 9 front end app servers
• Solaris 9, mix of zone and non-zone servers
• Oracle 9
• Sun HA cluster
• F5 BigIP intelligent loadbalancing switch
• LDAP with custom authentication for
  B-school/portal - no SSL on Bb
• Now
• 8.0.422.0 also known as Version 8, Service Pack
  5
• 4 app servers 1 datafeed collaboration
  server
• Solaris 10, no zones
• Still on Sun HA cluster
• Still on F5 BigIP switch
• OpenSSO, SSL'd at the authentication layer.

6
How many months to upgrade ?

• Question 
• Given the information you have just learned about
  our University, how long did we need for a
  successful upgrade?
•  
•  
• Answer  
• Per group in number of months

7
This is what we thought........ (at the
beginning)

• October 2007                             August
  2008
•  
•  
•  
• 10 months - let's see how we broke that down

8
Months 1 through 5

• Fall 07 Semester
• October 07 announcement by CITO that we will
  upgrade
• November 07 thinking ...
• December 07 more thinking ...
• Note Bb Inc. TSM replies to our help tickets
  with "Please upgrade as
  soon as you can"  
• Spring 08 Semester
• January 08 product demo meeting w/ Bb Inc. NYU
  Schools
• February 08 
• upgrade Steering Committee is established 
• Bb Inc. contracted for 2.5 weeks to deliver an
  executable upgrade plan
• yes we all thought 2.5 weeks planning was enough!
  
•  
•  

9
Do you see a problem?

• Question

Do you see  a problem?
10
Months 1-5 Lessons Learned

• Have all the Technical facts before you estimate
  your go-live date
• what version, separate IDs, 2 competing SSO
  plans
• Have a firm grasp on the sequential character of
  project 
• dependencies of systems
• dependencies of resources  
• Identify people and resource, not doing so
  resulted in
• no finalized communication plan or team
• no training plans 
• no tech team identified

11
Months 6 7

• Spring 08 Semester Cont
• March 08 
• Team meeting w/ Bb Inc, 5 days
• Upgrade declared "Too Big to Fail!"
• Started evaluating version 7 vs 8
• Started communication
• Blackboard upgrade blog
• formal community-wide communication
•  
• April 08 
• Tech team established
• ICM issues version decision needed before work
  could begin
• Purchased Bb Training Materials (time crunch)
• Hardware debate is ongoing Solaris zones or not
• Version decided! Lemon Fresh Install selected!
• Dev training environments established
•  

12
Do you see a problem?

• Question

Do you see  a problem?
13
Months 6 7 Lessons Learned

• Communication for Fall 08 go-live this is way
  too late
• Communication for Spring 09 go-live this is
  good timing
• Version decision WAY TOO LATE for Fall 08 go-live
• unless you have a vanilla system
• even if we did, training constraints way too big
• Still didn't call it here! Lets keep going for
  Fall 08

14
Month 8

• Spring 08 Semester Cont
• May 08 
• Bb Inc. on-site for system training
• Contracted w/ Bb Inc. for more help on executable
  plan 
• who knew 2.5 weeks wasn't enough ?
• Training Support Team established 
• Communication
• Blackboard discussion forum created
• blog announces version highlights new features
  of 8
• upgrade date announced as 'sometime' AY 08-0927

15
Where we are to date
Aug 08 Go-Live Date!
Apr 08Version, Install option, Tech Team est.
Feb 08Steering Committee, Bb Upgrade Plan
June 08
Dec 07More Thinking
Oct 07Upgrade Announced
Text text
Nov 07 Thinking
Jan 08Product Demo
Mar 08Too Big Too Fail
May 08System training, Training Team est, Vague
comm.
July 08
16
Month 9

• Spring 08 Semester Cont
•  
• June 08 
• Upgrade date moved finally!
• Contracts w/Bb Inc. signed
• Training/Support
• sub-teams
• project plans
• established Bb Liaison Program
• fall 08 Beta-Test Group
• created pain-points doc
• Technical architecture debates continue
• SSL
• Database cluster location
• Solaris zone architecture

17
Month 10

• Spring 08 Semester Cont
•  
• July 08 
• NYU PM Bb Inc. Project Manager assigned (yes
  really!)
• Technical issues
• New requirement warm stand-by for Beta-Test
  Group
• Custom Auth code alternative considered - Sun
  Access Manager (aka OpenSSO) established as
  target for CustomAuth/SternID requirement
  satisfaction
• Hardware, software and network architecture first
  draft sent to Security for review 

18
NYU HW/Network Architecture

• Proprietary NYU Image

19
Months 6 - 10 Lessons Learned

• High Level
• high stakes decision making takes longer than you
  think
• implementation choices require a lot of buy-in
• several months waiting cost us too much time
• Communication
• still not having dedicated communication team
  not good
• NYU Schools were NOT ready to upgrade (lack of
  communication)
• efforts should start at least 6 months before the
  upgrade happens (plus month extra for planning
  upfront)
• Training Support
• 10 months is enough time to plan training, but
  not enough time to do training!
• limiting and organizing support paths allowed for
  support efficiencies
• Technical
• use of an external consultant can streamline
  decision-making by reducing the friction among
  departments

20
OH CRAP IT'S AUGUST!AAAAAAAND.... go?
21
Let's start this again! How many extra months
doesyour group think we need?
22
Let's start this again!

• NYUS ANSWER
• 4 months, new go-live date is start of Spring 09
  Semester
• (aka December 2008, ignore Thanksgiving this year
  yall)
• Preferable to upgrade mid-year because
• the community is available to train
• the community is more attuned to university-wide
  communication
• thank you

23
Month 10 continued

• Summer 08 Semester Cont 
•  
• August 08
• OpenSSO work began with Bb Inc. specialist
• New Bb Inc. Project Manager
• Other NYU projects turned into a roadblocks and
  extra work for our project
• university wide network topology project
• new security standards project
• upgrade of help-ticketing system
• Beta-Test Group Separate support hotline
  mandatory training
•  

24
Month 11

• Fall 08 Semester September 08
• Training Support
• 12 Beta-Testers start fall classes on version 8
• Information and Demonstration session schedule
  released
• NYU Portal re-designed to allow both 6.2 8
  courses
• Decided k-base top down approach to support
• Communication - team established whoot!
• Other
• PM road-show w/VIP Schools
• CUNY phone call
• Northwestern phone call
• Prioritized projects for Dec 08 and after
• Technical 
• Archive/Restore testing - round 1
• Course Health Check Tool
• Performance Testing end-date scheduled
• Bb 6.2 starts to fail due to start of semester
  load

25
Month 12

• Fall 08 Semester Cont 
• October 08
• Training Support
• Blackboard-specific subgroup added to main
  helpdesk
• Liaison kickoff meeting
• Communication
• All faculty email
• Connect article submitted
• Technical
• Hot fix installed (beta test prod. now
  different)
• UAT Testing
• Fresh archives taken
• Switch, network, application - all installed
• Vulnerability assessment
• Performance testing started (6 weeks)
• Ignore email, check the wall instead 

26
Month 13

• Fall 08 Semester Cont 
• November 08
• Training Support
• Course Request Form redesign
• Building Block Testing (various schools)
• Bb Inc. training (again) Bb Liaisons Helpdesk
  Trained
• New System Roles finalized and created
• Hands-on training for faculty begins
• Communication
• Outreach road-show
• First Emails RE Org Site Moving
• Technical
• 1st round of restores to 8.0
• Datafeeds began over Thanksgiving
• Other
• Remedy and MeetingMaker Upgrade

27
Month 14 Go-Live Date

• Fall 08 Semester Cont 
• December 08  
• Go-Live Date, 2.0
• reason to celebrate we hit this date new
  baby girl!
• Spring 09 course sites available via course
  request form
• No outages or downtime 
• Org Site priority restore request deadline
• Non-priority issues back at forefront

28
Months 11 - 14 Lessons Learned

• Communication
• email campaigns can work - with the right subject
  line
• blogs can be a great communication tool
• be careful of comments
• have ONE message - consistency is key!
• Training Support
• providing the helpdesk with the tools to assist
  makes support a lot easier
• avoid upgrading several major systems at the same
  time, unless you'd like to torture your helpdesk
  (and community)
• Beta-Test Group were helpful but not
  representative of community
• if you build it, they don't always come
• Technical
• investing time to develop a min by min deployment
  checklist makes for a quiet and calm launch

29
Where we are to date
Oct 08Bb Liaison, All Fac. Email, UAT, Perf.
Testing
Dec 08Go-Live Date, No down-time
Aug 08OpenSSO, Beta-Test Group
June 08Revise go-live date, Signed contracts
w/Bb Inc.
Apr 08Version, Install Option Tech Team est.
Feb 08Steering Comm. Bb Upgrade Plan
Dec 07More Thinking
Oct 07Upgrade Announced
Nov 07Thinking
Jan 08 Product Demo
Mar 08 Too Big To Fail
May 08 System training, Training Team est, Vague
comm.
July 08 PMs assigned, Cust Auth Code issues
Nov 08 Building blocks, New roles, 1st Restores,
Datafeeds
Sep 08 Comm, Archive/Restore, Course Health Check
30
Month 15

• Spring 09 Semester 
• January
• Jan. 8th Winter intensive courses start
• Jan 13th Outages due to LOG ROTATION
  PROBLEM (fixed Jan 20)
• Jan 20th OpenSSO starts failing, user
  community perceives the Bb
  upgrade is at fault
• Jan. 20th Regular Spring 09 courses start
• Maybe we should write some TOUs for the new
  system...
• Grade Center Clinics begin
• Known Issue Reporting implemented
• Train, train, and train...

31
Month 15 Lessons Learned

• Communication
• no matter how painstakingly you keep track of
  requests...something will go wrong.  
• be ready to apologize and make exceptions.
• communication type work kept increasing!
• communicate how specific changes will affect
  faculty
• Training Support
• there may never be enough training.
• there will be a significant increase in regular
  help requests
• some problems you can throw bodies at, like help
  request ticket volume
• Technical
• just because a client signs off on
  functionality, doesn't mean they've actually
  evaluated it

32
Month 16

• Spring 09 Semester Cont
• February
• Email support terminated - online kb and
  help-form only
• Terms of Use posted to service catalogue
• Returned to projects we couldn't complete during
  Upgrade
• Migration of courses (non-priority) was on-going
• Feb 17th, org sites off 6.2 (communicated same) 
• Is the upgrade over yet?

33
Month 16 Lessons Learned

• Yep, we're still training....
• Single point of contact for help, customized to
  get enoughinfo from users good
• Clearly define Priority vs. Non-Priority elements
  of upgrade and content migration
• What goal do we have to hit to say we are
  finished?
• multi aspect approach

34
Month 17

• Spring 09 Semester Cont
• March
• Transition from Bb upgrade to Bb Service
  Management Team
• what exactly is a Bb Service Management Team? 
• we should establish and define this first!
• NB continued Bb Steering Committee (monthly
  meetings) 
• Returned to one-on-one training 

35
Month 17 Cont

• Spring 09 Semester Cont
• March
• Bb Service Matrix  

36
Month 17 Lessons Learned

• After the upgrade there is no going back
  (Meredith)
• If you were on the project, you're now on the
  service team - except Max, welcome Nicola

37
Months 19 20

• Spring 09 Semester Cont
•  
• April
• Focus on getting off Bb 6.2 other minor
  projects
• May
• testing installing Service Pack 5
• end of semester faculty survey results
•  

38
Faculty Survey Results

• Most satisfying about using Blackboard?
• Easy to communicate with students. the
  convenience, document upload and submission, and
  ease of use
• Least satisfying about using Blackboard?
• Grade Center difficulties, removal of the Digital
  Dropbox, assignment tool, system downtime and the
  speed
• Most satisfying about support and training?
• convenience of training and support, helpful
  responses from Blackboard support, and satisfying
  speed of response
• Least satisfying about support and training?
• inconvenience of training and support,
  unsatisfying speed of response, unanswered
  questions or unsatisfactory answers to questions.

39
Months 17 - 20

•  Summer 09
• June
• Killed access to 6.2
• July
• Finally off Bb 6.2 - last week

40
Where we are to date
June 09Killed access to 6.2
Feb 09TOU, Non-priority projects
Apr 09Getting off of Bb 6.2
Dec 08Go-Live Date
Jan 09Log issues, OpenSSO issues, Training
Mar 09Bb Service Mgmt Team
May 09SP5, Faculty Survey
July 09Finally off Bb 6.2!
41
QUESTIONS ?
42
Contact Information Links

• PM Lillian Moran lillian_at_nyu.edu
• Technical
• Max Whitney max_at_nyu.edu
• Nicola Monat-Jacobs nicola_at_nyu.edu
• Training Support
• Kelvin Shivers kelvin.shivers_at_nyu.edu
• Shannon McPartland s.mcpartland_at_nyu.edu
• Communication Meredith Rendall
  mrendall_at_nyu.edu
• Use Links
• NYU Bb Help Form - http//askits.nyu.edu/contact
  us/
• NYU Bb Kbase - http//askits.nyu.edu/blackboard/
  

43
Training Support Summary of Lessons Learned

• 10 months is enough time to plan training, but
  not enough time to do training!
• limiting and organizing support paths allowed for
  support efficiencies
• providing the helpdesk with the tools to assist
  makes support a lot easier
• avoid upgrading several major systems at the same
  time, unless you'd like to torture your helpdesk
  (and community)
• Beta-Test Group were helpful but not
  representative of community
• if you build it, they don't always come
• there may never be enough training.
• there will be a significant increase in regular
  help requests
• some problems you can throw bodies at, like help
  request ticket volume
• Single point of contact for help, customized to
  get enough info from users good

44
CommunicationSummary of Lessons Learned

• Establish resources for huge communication effort
• Effort will only increase as upgrade continues
• Month 5 (of 14) start planning communication
• Month 6 or 7 start communication with community
• Media
• email campaigns can work - with the right subject
  line
• blogs can be a great communication tool
• Misc
• no matter how painstakingly you keep track of
  requests...something will go wrong.  
• be ready to apologize and make exceptions.
• communicate how specific changes will affect
  faculty
• have ONE message - consistency is key!

45
TechnicalSummary of Lessons Learned

• Have all the Technical facts before you estimate
  your go-live date
• Have a firm grasp on the sequential character of
  project (dependencies of systems resources)
• Month 6 of 14 is ok for version decision
• Use of an external consultant can streamline
  decision-making by reducing the friction among
  departments
• Investing time to develop a min by min deployment
  checklist makes for a quiet and calm launch
• just because a client signs off on functionality,
  doesn't mean they've actually evaluated it
• Clearly define Priority vs. Non-Priority elements
  of upgrade and content migration
• What goal do we have to hit to say we are
  finished (multi aspect approach)