Performance Assessment for Web-based Systems

1
Performance Assessment for Web-based Systems

• Min Song, Ph.D.
• IS 465

2
Topology of a Web Site
Internet
I S P
F i r e w a l l
Browser
Security Server
Web Server
Browser
Database Server
Browser
Application Server
3
General Considerations

• Application conception
• Whats to be done?
• Whats the whole purpose of this task?
• Design
• How is it going to be done?
• Flexibility, Dynamic
• Development
• What tools to use?
• What technologies to implement?
• TESTING
• What? its working well, isnt it?
• Deployment
• How soon?, Pronto!, Is fast isnt it?

4
The NEED to Test(Are we wondering)

• Is the site content meaningful?
• Is this application easy to use?
• How about browser compatible?
• Are we using the right tools?
• How reliable is our technology?
• Do the Servers have enough power?
• How many visitors are we expecting?
• Are the machines fast enough?
• How much activity can the site handle?

5
What and How to test?(Fundamental Considerations)

• Test Planning
• Browser Compatibility
• Design compliance
• Usability
• Efficiency
• Scalability
• Best suited OS platform
• Network bandwidth
• Hardware capacity
• Site availability
• Monitoring
• Security

6
Testing a Web Site

• Understanding what types of testing are
  needed for each application, is essential for
  several reasons
• Ensures you that the testing effort, will support
  the business and technical requirements.
• The activities required for each type of test and
  its associated phases, are included within the
  master test plan (and schedule).
• Helps identifying and plan for the environments
  and resources that are needed to prepare and
  execute each test type.
• Assures the achievement of any Testing Goals.

7
Testing a Web Site (continued)

• There are mainly two essential types of
  testing that should be accomplished.
• Functionality
• Performance

8
Functionality Testing

• What is it?
• Why is so important?
• What is
• Unit Test
• Integrated Test
• Functional Test
• System Test
• Regression Test

9
What is Functionality Testing?

• Is a set of skilled approaches and procedures
  which systematically verify that the
  applications components and process modules, are
  available and working as expected.
• Compliance with Business rules, Technical
  requirements, System Documentation, and User
  manuals is of course Mandatory!

10
Why is Functionality Testing important?
Missing Image
11
Functionality Testing -Unit Test

• Unit Test
• Unit tests perform basic tests at component level
  and test a specific business process,
  application, and/or system configuration.
• Unit tests ensure that each unique path of a
  business process performs accurately to the
  documented specifications and contains clearly
  defined inputs and expected results

12
Functionality Testing -Integration Test

• Integration Test
• Integration test is testing a specific feature
  together with other newly developed features
• Testing the interface of two components explores
  how components interact with each other
• Integration testing inspects the variables passed
  not only between two components, but also the
  global variables.
• Integration testing is specifically aimed at
  exposing the problems that arise from the
  combination of components
• Previously captured unit test scripts can be
  combined to create a variety of integration test
  cases
• This test phase assumes the components and the
  objects they manipulate have all passed their
  local unit tests

13
Functionality Testing - Functional Test

• Functional Test
• Functional testing is centered around the
  following items
• Valid Input - identified classes of valid input
  must be accepted.
• Invalid Input - identified classes of invalid
  input must be rejected
• Functions - identified functions must be
  exercised
• Output - identified classes of application
  outputs must be exercised.
• Systems/Procedures - interfacing systems or
  procedures must be invoked.

14
Functionality Testing System Regression Test

• System Test
• System testing studies issues and behaviors that
  can only be exposed by testing the entire
  integrated system
• Verifying these characteristics is very important
  for products intended for users on diverse OS and
  Hardware platforms.
• Identification of faulty Hardware and Software
  components, is feasible this way.

• Regression Test
• Regression tests are a series of tests to ensure
  that no adverse changes are introduced to the
  application during maintenance changes, upgrades,
  or other modifications.
• Regression testing is composed of a suite of
  components that are rerun when any
  one-application component has been modified
• This provides confidence that the system will
  function with newly implemented requirements.
• Regression testing is often an iterative process
  during a testing cycle

15
Performance Testing

• What is it?
• Why is so important?
• What are the types of Performance Tests?
• What is the difference between Load, Performance,
  and Stress Testing ?

16
What is Performance Testing ?

• Performance Tests determine the runtime
  behavior of the application and its supporting
  infrastructure, under certain conditions.
• Performance Testing is used to measure several
  system characteristics, such as processing speed,
  response time, resource consumption, throughput,
  and efficiency.

17
Why is Performance Testing important?

• The week of Feb 6, 2000 Hackers delivered over
  1-Billion transactions concurrently to each one
  of these sites.
• Yahoo
• eBay
• Buy.com
• Amazon
• eTrade

How did you think they performed ?
18
Performance Testing - Load Test

• Load Test
• Load Tests determine the applications behavior
  under load, up to and including its limits (not
  just at its limits).
• Load tests specifically refer to the load size
  (number of concurrent users) and related values.

19
Performance Testing - Stress Test

• Stress Test
• Stress testing determines the applications
  ability to handle large amounts of data.
• Stress Testing can be much more successful with a
  full load applied to the server.
• Running addition applications on the server in
  conjunction with the client side tests is an
  additional form of stress testing.

20
Performance, Load, and Stress Testing ?

• They all have different meanings, although
  they seem to be used almost interchangeably
  sometimes.
• To better understand these differences lets
  apply these terms while testing a car
• Performance Testing - a car does 0-60mph in a
  certain number of seconds under normal conditions
  
• Load Testing The car can pull a trailer
  weighing x - tons before it can no longer move
• Stress Testing - The car continues to work fine
  driving 60 mph for two hours, in 120 degree
  temperatures, with only 1 gallon of coolant in
  the radiator

21
Why Automate Testing?

• Is it necessary to repeat the sequence of actions
  many times?
• Do you need to run the same tests on multiple
  hardware configurations?
• Need to test with several concurrent users?
• Exercise multiple options, components, and
  configurations
• Save time and money

22
What are we testing for?

• Where should I start?
• How do we proceed?
• What tool do I select?

23
Simulating the Real World

• The MAIN purpose of running automated tests is to
  emulate the REAL world as close as possible while
  saving time and effort.

24
Tips for selecting an Automated Testing Tool

• Ease to use (and learn)
• Standards compliance
• Robust and Reliable (accurate)
• Flexible
• Affordable
• Maintenance and Support
• Important Is it the right tool for the job?

25
What are we Testing for?

• Everyone involved should be clear on what exactly
  we are testing for and the goals for the outcome
  of the test
• Testing For
• How many Total Concurrent Users ?
• At what point does the performance degrade ?
• Which server will max out first ? (Web, DB, App,
  etc.)
• How about Data integrity ?
• What is the Total Throughput ? (Bandwidth,
  pipeline)
• Goals
• Round time lt 15 sec
• Total Concurrent Users gt 300 / Min

26
Automation Web Testing Methodology
5-Phases
Modeling Paradigm
27
1.Define

• The Application
• Describe the application to test using a
  application model
• Describe the Hardware environment
• Create a Benchmark (Agenda) to be recorded in
  Phase 2.
• Define what tasks each user will perform
• Define (or estimate) the percentage of users per
  task.
• Benchmark - A benchmark procedure uses a
  predefined set of data and measures the results
  returned by the Application Being Tested (ABT).
• Benchmarks are intended to make comparisons
  easy!

28
1.Define

• Testing Types and Metrics needed
• Load
• Increasing Workload
• Start / End with how many users?
• Increment by how many users?
• How often to increment (Seconds)?
• Steady-State
• How many users?
• How long (Days/Hours/Minutes/Seconds)?
• Performance
• What are the reasonable response times in
  seconds? (each task/page)
• Stress
• What are the expected limits?
• How many Peak users ?

29
3.Modify
2.Record
1.Define

• Record
• Record the defined testing activities that will
  be used as a foundation for your load test
  scripts.
• One activity per task or multiple activities
  depending on user task definition

• Modify
• Modify load test scripts defined by recorder to
  reflect more realistic Load test simulations.
• Defining the project, users
• Randomize parameters (Data, times, environment)
• Randomize user activities that occur during the
  load test

30
Performance Test scenario
4. Execute

• Test Date 07/17/2001
• Virtual Users (VUs) Test Goals
• Start 5 Max Cycle Time lt 40 Sec
• Incremented by 5
• Maximum 200
  Initial Cycle Time 35 Sec
• 
  Sleep 32 Seconds
• Test Script(s)
• One typical user from logon through
  completion (continuously)
• This would be an average user

31
Performance Test scenario
5. Analyze
Cycle Time starts to oscillate, indicating a
bottleneck
32
Performance Test scenario
(Continued)
The CPU Started making large spikes right from
the start
33
Performance Test scenario
(Continued)
The TPS Throughput peaked out indicating
another Bottleneck
34
Performance Test scenario
Both Load Generator CPUs are spiking to 100
35
Conclusions

• Define the purpose of taking on a test
  automation effort
• Develop a test automation strategy (map out
  what's to be automated)
• Treat the automated process as any other
  software development task
• Do your homework (Investigate!)
• Test early and test often!
• Remember
• The effort of test automation is an investment
• Those performing the work should have the
  correct skill sets
• YOU SHOULD TEST UNDER LOAD!!!