SaaS Testing Overview - Foundation

1
SaaS Testing Overview
Ram Garg CEO Software Performance Assurance (SPA)
Software Performance Assurance Confidential
2
Outline
? SaaS Introduction ? What is it? ? Why
SaaS? ? SaaS Challenges ? Key Challenges ? SaaS
Testing ? How to approach SaaS testing? ? How
SaaS Testing is different? ? SaaS Test Tools
Software Performance Assurance Confidential
3
What is SaaS?
SaaS Introduction
4
SaaS What is it?
Software-as-a-Service (SaaS) is the application
layer of the cloud computing model.
Application
Software as a Service (SaaS)
Platform
Platform as a Service (PaaS)
Infrastructure
Infrastructure as a Service (IaaS)
5
SaaS Definition
Software as a service (SaaS), sometimes referred
to as "software on demand," is software that is
100 deployed over the internet . With SaaS, a
provider licenses an application to customers as
a service on demand, through a subscription or a
"pay-as-you-go" model. Wikipedia definition
6
Architecture Three Key Differentiators
Multi-tenant Efficient
Scalable
Configurable
Key Architecture Differentiators
7
Why SaaS?
SaaS Introduction
8
The Software Business Model Changes
non -SaaS
SaaS
Customer
Provider
Software Ownership
Infrastructure management Responsibility
Customer
Provider
Client - Server
Central
Installation
Larger, less frequent
Shorter, frequent
Upgrades
Multi-version
Single-version
Platform
9
Changing the Business Model cont
non -SaaS
SaaS
Configurability Requirement
Low
High, Critical
No
Yes
Multi-tenancy
No
Yes
Backward compatibility
Customer
Provider
Maintenance
10
Changing the Business Model cont
non -SaaS
SaaS
High, CapEx
Low, OpEx
Cost, Allocation
Close the deal, License Revenue
Provide value, Lack of Churn
Sales focus / Success
Initial sale maintenance
Ongoing Subscription
Profits
11
Ecosystem
SaaS (Software) PaaS (Platform) IaaS (Infrastru
cture)
? Starting point to every service within cloud
computing ecosystem ? Highest on the maturity
level and growth curve amongst XaaS ? Highest
revenue generating service within cloud model for
next several years
12
SaaS Catalysts
Intrigue of access to unlimited Computing and
storage resources, made available and paid for
only as needed (Pay as you go) Decreasing cost
of bandwidth Customer frustration with
traditional software buying cycle (license,
maintenance contract, upgrades) Perceived
increase in control over the vendor
relationship Early SaaS success stories
Salesforce.com, WebEx, etc. Demonstrate broader
value proposition You focus on your application
and not on infrastructure
13
SaaS Early Adoption
SaaS Introduction
14
SaaS Adoption Verticals
AMI-Partners report - "Software-as-Service Hot
Spots Education, Healthcare, and Financial"
Adoption rate of SaaS in SMB is higher than large
enterprises
All research points out that SaaS has been
adopted heavily in ? HR applications (ERP,
Salesforce automation, eRecruiting) ? Online
Collaboration (Web conferencing) ? CRM ?
Healthcare
In-Stat research on vertical market adoption
shows the healthcare market is currently most
satisfied with hosted applications, with
education identified as an important potential
target market also.
15
SaaS Adoption Factors
(source Forrester)
Availability Reliability (71)
Fast Deployment (45)
Reduced dependencies on internal IT (24)
Flexible consumption (35)
Simplified pricing model (32)
16
The SaaS Market is and will Continue to Grow
17
Today and tomorrow
Software Performance Assurance Confidential
18
Today and tomorrow
Software Performance Assurance Confidential
19
Today and tomorrow
Software Performance Assurance Confidential
20
Challenges
SaaS Challenges
21
Primary Concerns with the SaaS Model
Security concerns Concerns over compliance with
standards Concerns over integration with legacy
applications Concerns over reliability and
robustness of service Not comfortable with
putting mission-critical application on shared
infrastructure and have access over
internet SaaS can not be configured to meet my
corporation?s all specific needs
22
Numerous Challenges
Security
Performance
Availability
SLA
Compliance with Standards
Data Integrity
Privacy
Regulatory
Lack of confidence
Integration
Multi-tenancy
23
SaaS Testing
24
Why SaaS Testing?
SaaS Testing
25
Why SaaS Testing?
The average enterprise IT department devotes up
to 50 percent of its entire technology
infrastructure to development and test, with up
to 90 percent of the available test
infrastructure remaining idle. IBM Research
A delay in website page loading by even one
second can impact conversions by 7 and customer
satisfaction by 16. Aberdeen Group
Cloud Testing
Those using cloud facilities to run applications,
49.8 said they were doing so experimentally or
for prototyping 28.6 for non- critical business
applications and 21.6 for business critical
applications Evans Data Survey
The average online business had 8 days of
downtime a year due to performance issues leading
to lost revenues Aberdeen Group
26
Trend leads to trend
Overall growth in Cloud Computing (investment,
revenue, ROI, market cap, etc.) is leading to a
marked increase in adoption of SaaS Testing
27
What is SaaS Testing?
SaaS Testing
28
Classifying SaaS Testing
Testing The CSlaoauSd Testing applications
which are in cloud or migrating to cloud.
Testing the SaaS
Test the performance, reliability security of
applications
Cloud / SaaS Testing
TesTtiensgtiSngaaUSsiUngsinCgloCudloud
Testing applications using the cloud-based
hardware, software, infrastructure and computing
resources.
Testing SaaS Using Cloud
Tests performed like performance, load stress,
security and compatibility
29
How SaaS Testing is Different
Easier Challenges

• No client-server
• installation
• No multiplatform support
• No multiple version
• upgrades
• No backward compatibility

• Frequent live upgrades
• Too many pages
• Multiple privileges
• Immature SaaS-specific
• testing tools
• Access partitions critical

30
Approach
SaaS Testing
31
SaaS Testing Preparation
Prepare Test Plan and Test Suite
Assess Testing Needs
Understand Usage Patterns
Prepare Test Environment
Prepare Test Data
32
SaaS Testing Areas
Infrastructure Testing
Network testing
Application testing
Secured SaaS Application Ensure SaaS testing
does not damage the shared infrastructure
33
SaaS Functional Testing

• Perform Manual and Exploratory test (similar to
  traditional applications)
• Check Browser compatibility
• Perform regression
• SaaS regression must cover significant more
  functionality as single defect will impact all
  the customers
• Functional and regression test automation.
• This is important to comprehensive testing
  cost-effectively as above point indicates that,
  in case of SaaS, regression should cover more
  functionality.
• Test SaaS on cloud-based staging environment
• Testing on in-house environment and then directly
  deploying on cloud- based production environment
  may result in chaos
• Test in a multi-user environment where different
  users are performing different activities.

34
SaaS Security Testing

• Test application security
• Cross Site scripting, SQL Injection, HTTP header
  injection
• Test network security
• Secured communication, Backup and storage
  policies, Controlled access to sites and servers
• Test for possible attacks
• Test roles and privileges in multi-tenant
  environment
• Many roles and privilege levels
• Test all the upgrades to detect broken security,
• Ensure security patches are not missed
• Test data security / integrity among multiple
  tenants

35
SaaS Security Testing cont

• Identify scenarios where application might be
  vulnerable
• Test PCI compliance
• Test cookie setting, SSL configuration,
  information leak, etc.
• Log errors, warnings and suspicious requests
• Tests tampering with the licensing mechanism
  gaining unauthorized access to application data,
  editing registry entities or configuration files

36
SaaS Performance Testing

• Test performance extensively first time and for
  each upgrade
• SaaS is revenue generating and business impacting
  applications.
• Single performance issue will impact all your
  customers
• Test performance extensively as SaaS need to
  handle large number of users and we don?t have
  luxury of re-booting or going down every once in
  a while
• Test load characteristics, ramp up rate,
  protocols, load origination,
• Establish ability to test performance of your
  SaaS isolated fashion so that network related
  issues can be identified and passed to network
  infrastructure and internet providers
• Test key metrics and SLAs
• Test performance for mobile users

37
In-Service Upgrade Testing

• Ability to upgrade (full release, patches) system
  with zero- to-minimum down time is essential
• Global users make this complicated as somewhere
  in the
• world someone might be accessing your application
• Thoroughly test in staging environment before
  perform upgrade ass this is one of the most
  complex features
• Remote Accessibility and Usage
• Global access with acceptable response time
• 24 hours availability

38
Disaster Recovery

• Shutdown to bring up application quickly server
  configuration install and deploy software
  restore data
• Roll back to last known stable data
• Localization
• Make sure it supports international characters as
  SaaS application can be accessed from anywhere in
  the world
• Optimize by supporting target market language

39
Test Tools
SaaS Testing
40
Traditional and Cloud-based Test Tools
Cloud-based Test Tools
Traditional Test Tools
QTP, WinRunner Rational Robot
Rational Functional Tester SilkTest
Functional
Gomez PushToTest, SOASTA
Selenium, STAFF, Fitness, Watir
CloudTesting
LoadRunner, Rational Robot, Rational Performance
Tester Silk Performer
uTest Many others
Performance
Jmeter, Grinder, WebLoad, OpenSTA
41
Traditional Vs. Cloud-based Tools

• Testing SaaS Using Traditional Tools
• Many providers who provide SaaS testing using
  traditional testing tools
• Many open source matured tools
• Many options for professional services by
  independent service providers
• Professional service options allow cheaper cost

• Testing SaaS Using Cloud-based Tools
• A few providers who provide SaaS testing using
  their own cloud-based test tools
• Limited open source test tools options
• Captive professional service by test tools
  provider which is not even core for their
  business
• Captive profession service is sold at very high
  rate

42
Conclusion
43
Conclusion
Customers buy each month keep them happy SaaS
is only a delivering model, customers will not
compromise on functionality, stability, security,
and performance. They will not switch to SaaS it
if not better than traditional systems. Save
time on support, maintenance, and upgrade,
etc. You need to pay more attention to stuff
like accessibility, security, deployments. The
way you earn reputation Don?t buy cloud-based
test solution just because you are testing SaaS.
In most cases, traditional test solutions work
just fine and save your 30 50 cost.