Testing and your Y2K Project

1
Model Based Testing of Web Applications using
ASMLCase Study on SMARTCat and Action Words
Dr Kelvin Ross K. J. Ross Associates Pty.
Ltd. PO Box 131, West Burleigh, 4219 Ph 07 5522
5131 Fax [phone] kelvinr_at_kjross.com.au htt
p//www.kjross.com.au
2
SMARTCat

• Continuous test environment for monitoring live
  production applications
• Web application
• Tests can be edited online
• Runs scheduled tests 24x7
• Test results stored in database
• Alerting via email, sms and dashboard
• Historic service level reports produced on
  request or via batch for email subscription
• Implementation of SMARTCat
• Client HTML/Javascript
• Server PHP, Perl, Java, Postgres

3
SMARTCat Testing

• Currently have about 300 automated regression
  tests that are run for each build
• Action word test framework implemented in
  SMARTCat itself
• Adding regression tests require considerable
  thought and expertise
• Need to trace back to test case and requirements
  to understand coverage
• Would like to extend the test coverage achieved
  via automation

4
Action Word Testing

• Action word script for user actions
• Step driver for each action that performs
• Navigation
• Data entry
• Verification

\TC-01,gotoPage,welcome.php,SMART CAT -
Welcome \TC-02,login,admin,admin,OK \TC-03,logoutL
ink \TC-04,login,admin,dmin,Invalid
login \TC-05,login,admin,admina,Invalid
login \TC-06,login,admina,admin,Invalid
login \TC-07,login,,dmin,Invalid
login \TC-08,login,admin,,Invalid
login \TC-09,login,guest,guest,OK \TC-10,logoutLin
k \TC-11,login,tester,tester,OK \TC-12,logoutLink
\TC-13,aboutLink \TC-14,loginLink \TC-15,login,adm
in,admin,OK \TC-16,createClient,client1,12 Goline
Ct,Tallebudgera,4228,QLD \TC-17,openClient,client1
\TC-18,checkClientDetails,client1,12 Goline
Ct,Tallebudgera,4228,QLD \TC-19,deleteClient,clien
t1 \TC-20,logoutLink
5
Model Based Testing Goals

• Use a model of SMARTCat behaviour as a basis for
  test generation
• Test generation provides coverage over the model
• Undertake a variety of web based tests
• Functionality
• Scenarios
• Access control
• Web vulnerabilities, e.g. command injection, etc.

6
Statechart Based Approaches

• Limited due to expressiveness of model
• Collections difficult to support

7
Abstract State Machine Language (ASML)

• Microsoft Research Project - http//research.micro
  soft.com/fse/asml/
• Define specification asmodel
• Generate Finite StateMachine (FSM) from model
• Generate test sequencesfrom FSM

8
ASML Specification

• Rich specification/modelling language
• For describing our model we must provide
• Type definitions
• State definitions
• Method definitions
• A full specification is provided as a word
  document (SMARTCat.doc)

9
ASML Specification- Types

• Define the structures that will be referred to
  by
• state variables
• method input and output parameters
• E.g pages that can be visited in SMARTCat

10
ASML Specification- State

• Data stored throughout lifecycle
• Data updated by operations
• Defines variables of type
• Initial values defined

11
ASML Specification- Methods

• Defines operations on the state
• Precondition when it can occur
• Effect what changes are made
• E.g. login and logout
• Many other operations provided, e.g. AddUser,
  DeleteUser,
• See attached specification

12
ASML Test Generation

• Would like to generate possible sequences of
  methods
• ASML specification provided previously would
  provide infinite sequences
• States are infinite, e.g. set of users can
  continue to grow
• Parameters would produce infinite input
  possibilities
• When to stop????
• ASML provides capability to generate Finite State
  Machine (FSM) from spec
• BUT, we must limit state explosion and input
  parameter combinations

13
ASML Specification to Limit FSM Generation

• Additional specification properties are defined
  for the SMARTCat specification to provide
  effective FSM generation
• Type restriction
• State restriction
• Properties
• Filters

14
Type Restriction

• Limit possible values types can take on
• Limits input parameters

15
State Restriction- Properties

• An asml function used to compute equivalent
  states.
• A number of properties are defined, and the
  collection of which are used to abstract the
  state representation.
• If a concrete state is visited which is
  equivalent w.r.t. the state abstraction, we
  consider terminating the search at this point.

16
State Restriction- Filters

• A property returning a boolean which positively
  identifies those states we want to consider for
  exploration and drops all others
• E.g. PageFilter if client is SMARTLabs and
  only user defined is admin with password
  admin then consider all pages, otherwise we
  restrict navigation only between those pages
  designated important.

17
ASML FSM Generation
18
ASML Test Sequence Generation
19
Test Sequence Output

• Test Sequence output in XML
• Process with XSLT into required action word
  representation

lt?xml version"1.0" encoding"utf-8"?gt ltTestSuite
xmlnsxsd"http//www.w3.org/2001/XMLSchema"
xmlnsxsi"http//www.w3.org/2001/XMLSchema-instan
ce"gt ltSequencesgt ltSequence Id"0"gt
ltStepsgt ltStepgt ltAction
Name"internal SMARTCat.Login(System.String,System
.String) as System.Boolean" IsVoid"false"gt
ltArgsgt ltanyType
xsitype"xsdstring"gtsitemanagerlt/anyTypegt
ltanyType xsitype"xsdstring"gtadminlt/anyT
ypegt lt/Argsgt lt/Actiongt
ltOutput success"true"gt ltvalue
xsitype"xsdboolean"gtfalselt/valuegt
lt/Outputgt lt/Stepgt
20
Construction Process

• Define/extend specification
• Define initially quite restrictive Specification
  Restrictions
• Generate FSM
• Verify intended behaviour, return to 1 for
  changes
• Reduce parameter and state restriction to explore
  further on larger states
• Export test sequence for execution against
  Application Under Test (AUT)

21
Test Implementation

• Generation in Action Word framework
• SMARTCat
• JUnit/NUnitASP
• ASML Test Generator has capability to link to
  compliance tests and execute these form the ASML
  Test Generation tool
• Need to convert XML into the action word
  representation
• XSLT useful as a mechanism here

22
Future Work

• Extend specification into other areas
• Lifecycle of a site
• Lifecycle of a test
• Test Execution
• Other new features
• Partition of specs of modularisation
• Refine Specification FSM Properties/Filters to
  focus on
• Access Control tests
• Error handling
• Web Vulnerability tests

23
Improvement Wish List

• XSLT transformation on sequence exporting
• Ability to batch generate with different
  specifications
• Perhaps there is a make/Ant capability???
• Reporting on states in the test sequences
• Not in XML, but is in ASML Test tool
• Generate as comment, would help with debugging,
  as if step fails we can try to reproduce state
  and manually test
• Continually running spec as endurance test,
  perhaps with some likelihoods for routines
• Markov modelling
• Load/endurance/stress scenarios

24
Conclusions

• Pros
• Powerful mechanism for model based test
  generation
• Cons
• Learning curve
• Management of test generation for larger
  applications needs to be reviewed
• Different viewpoints regenerated
• Need to understand the implications of filter
  choices, etc. on coverage