Jan Warren

1
Performance Testing vs. Performance Monitoring

• Jan Warren
• Enbridge
• April 2009

2
What we shall talk about

• Some of the many terms..
• What the two disciplines mean
• Comparison and contrast
• Where they overlap (or do they?)
• Throw in a little ITIL
• Our tools
• Nirvana in each world
• In a joint ideal world..
• Conclusion
• Questions

3
Top of the food chain

• Performance Management
• System Measurement Management
• Measurement Management
• System Assurance
• Anything else?

4
Performance Testing

• AKA Quality Assurance (or Quality Control)
• Performance testing Testing conducted to
  evaluate the compliance of a system or component
  with specified performance requirements. IEEE
• Performance testing, unit testing, system
  testing, integration testing, functional testing,
  regression testing, load testing, stress testing,
  volume testing

5
Performance Monitoring

• AKA Quantity Assurance?
• Monitoring of the End User Experience (EUE),
  monitoring of system infrastructure (servers,
  network, switches, ..)

6
Compare and Contrast

• Testing
• Performance and accuracy
• Pre-production
• Controlled Load (forced/simulated)
• Agent based
• Agentless
• Network
• Server
• Capacity definition
• SLA baselining
• Recreate production issues
• Reproducible measurement under controlled
  conditions

• Monitoring
• Performance and availability
• Production
• Normal load
• Agent based
• Agentless
• Network
• Server
• Capacity planning
• SLA monitoring
• Diagnose production issues
• Live real time snapshot of in service
  performance

7
Where do they overlap (or do they)?

• NOT the environment
• The Medium (server, network, etc.)
• NOT the role of application folks
• NOT in terms of load
• NOT in their role i.e. Testing vs. Monitoring
• Possibly the tools

8
ITIL

• Performance Testing
• Release Management
• ICT Infrastructure Management
• Deployment
• Performance Monitoring
• Service Delivery
• Service Management
• Service Level Management

9
Our tools

• Testing
• HP
• Quality Center
• Quick Test Pro
• LoadRunner
• SiteScope
• Diagnostics (for SAP)

• Monitoring
• Compuware
• ServerVantage
• ClientVantage
• ClientVantage Agentless
• VantageView
• WhatsUp Gold
• PRTG
• Scrutinizer

10
Testing Nirvana

• True quality assurancenothing goes into
  production until completely tested and confirmed
  acceptable

11
Monitoring Nirvana

• Indicators warn us ahead of an issue and it is
  already resolved before it surfaces
• Failing that, the issue surfaces, the proper
  folks are alerted and the issue is being worked
  on before the calls to the helpdesk start

12
In an ideal world..

• Tests and infrastructure set up for performance
  testing could be ported to production for
  performance monitoring.

13
Conclusions

• Do I have any?
• ITIL keeps them separate
• Enbridge keeps them separate
• Currently, although the groups talk, nothing
  created in the pre-prod world can be ported to
  the production world
• Customer expectation management vs. Customer
  experience management
• Both groups often get forgotten during development

14
Questions?