Performance Load Testing Case Study

1
Performance Load TestingCase Study Agilent
Technologies
2
Agenda

• Introductions
• Background
• Testing Objectives
• Preparation Phase
• Execution Phase
• Analysis
• Lessons Learnt
• Contact Information

3
Introduction

• Ramesh Padmanabhan
• Entegration Software
• Consulting product company based in San Jose
• Proud to be service partners of
• Oracle Corporation
• Mercury Interactive
• Yash Technologies

4
Introduction

• Agilent Technologies
• 6 Billion Global Mfg Company
• Over 30,000 employees in more than 50 countries
• One of the largest global single instance
  installs of Oracle E-business suite
• Consolidated over 150 legacy systems
• Expect a maximum 5,000 concurrent users

5
Background

• Largest single instance install
• 3 HP Superdomes Production, Reporting, Planning
• Single US based data center
• Over 50 operating units
• Significant business volume in Asia Europe
• Consolidating over 125 different legacy systems
• Implemented all Financial MFG Modules

6
Testing Objectives
7
Testing Objectives

• Validate single instance strategy
• Validate network and hardware infrastructure
• Scalability to 5000 concurrent users
• Stress test for high water mark
• Set user response time expectations
• Identify and fix significant performance tuning
  issues within Oracle Applications
• Identify and drive solutions for hardware issues

8
Preparation Phase
9
Data Gathering

• Identified major transactions within each
  application module
• Questionnaires sent for legacy data volumes by
  geography (US, Asia, Europe)
• Short listed transactions with high volume or
  data intensive processing
• Identified user distribution by region and by
  application areas
• Determined estimation methodology for inquiry
  transactions

10
Hardware Preparation

• Ensure that the production configuration of
  back-end server and middle tier machines were
  set-up and configured
• Procure the Load generation agent boxes and have
  them installed and setup at the right locations
• Ensure that the Cisco load balancing router was
  correctly set up
• Set up network sniffing devices to get detailed
  metrics of network traffic

11
Software Preparation

• Procure and install LoadRunner on the agent and
  controller boxes
• Install LoadRunner and the Oracle Applications
  client on the machines of the scripters
• Install/Setup other database monitoring software
• Prepare scripts for detailed transaction analysis

12
Data Preparation

• Validated various application setups
• Initial cycles required all key master data to be
  fabricated
• Developed numerous scripts to extract key data
  elements like items, customers, vendors etc. to
  be used in transactions
• Ensured adequate breadth of data.
• Identified key data and parameters for background
  load

13
Develop LoadRunner Scripts

• Recorded scripts for all the critical and high
  volume transactions
• Adequate mix of inquiry and update txns.
• Parameterized all the critical pieces of data
  like item, customer, orders etc.
• Identified activities for which server response
  times were key and set up transaction timers
  around them e.g. commits, quick-picks etc.

14
Execution Phase
15
Build Test Scenarios

• Develop matrix for users by geography by
  transaction
• Manual scenarios
• Goal oriented scenarios
• Transactions split into three groups based on
  data dependency conditions

16
Run Tests

• 5 cycles of testing
• 1- validation cycle
• 2 complete cycle with converted data
• 3- Stress test cycle
• 4- Complete integrated test with key interfaces
  and customizations
• 5- Production simulation run
• Each cycle comprised of two major runs/day for
  two weeks. Each test run was about 4-7hrs long

17
Run Tests

• 5000 concurrent user load generated from 8
  LoadRunner agents 4 in US, 2 each in Europe
  Asia
• LoadRunner monitors set up for network, backend
  server middle-tier boxes
• Dedicated DBA and performance tuning experts
  monitored the HP Superdome server

18
Analysis

• Used LoadRunner Analysis tool
• Real time graphical interface to monitor the test
  progress
• Post run analysis includes numerous graphs and
  transaction timers
• More detailed analysis was done from the result
  data stored by LoadRunner in an Access database

19
Analysis

• Data from the analysis used to
• Set up realistic response time expectations from
  the end users
• Modify various database parameters in the
  init.ora to better performance
• Tweak settings of the Cisco load balancer for
  middle tier machines
• Identify and tune some of the application code
  that had bad performance

20
Limitations

• Some performance intensive processes could not be
  tested due to data dependency issues e.g.
  lock-box receipts
• Some dynamic and interactive processes could not
  be tested very well e.g. configured orders
• Some custom code not stable till the last cycle
• Some of the newer application modules not stable
  for a reasonable test
• Application version and patch set lags

21
Lessons Learnt

• Performance test will only be as good as the data
  collected in the analysis phase
• While performance test can significantly reduce
  risk of poor performance, it is not a guaranty
• Initial performance testing cycles should focus
  more on non-code related performance variables

22
Lessons Learnt

• Intensive code related performance testing
  tuning should take place after custom solutions
  have been put into testing and application patch
  sets are frozen
• Performance testing should be in the critical
  path of project plan and performance testing
  instances should be patched just like the BST
  instances
• Should plan on at least one marathon testing run
  that extends for 3 or 4 days

23
Contact Information
Ramesh Padmanabhan Entegration Software rpadmanabh
an_at_entegration.com 408-674-3701 www.entegration.co
m