Towards Further Automation of the Quality Assurance Cycle

1
Towards Further Automation of the Quality
Assurance Cycle
Alan Berg University of Amsterdam Central
Computer Services (IC)? Group Education and
Research Services
2
But first an advert
QA ing makes you sexy
3
QA or the Sakaiger gets it
4
QAing helps you get the toy
5
We have only 30 minutes, so only a brief biased
personal snapshot is possible
6
Agenda

• The potential for a QA event horizon
• What does the QA WG want?
• Current hints to the developer
• Where are we
• The Sash Tool
• Jameleon
• Conclusions

7
The QA event horizonRisk losing a deterministic
QA cycle
http//amazing-space.stsci.edu/resources/explorati
ons/blackholes/teacher/grabbag.html
8
To remain deterministic we will need more
automation
9
The QA event horizonRisk losing a deterministic
QA cycle

• The Obvious
• Code base increasing rapidly
• Not just Java, but also XML, Javascript,
  Python,CSS and many Java frameworks
• Iterative process that requires defined response
  times
• Requires manpower/resources on time
• Would love QA as part of Continuous Integration

10
What does QA WG want?

• A "stack" of open source tools for load testing
  and nightly integration testing
• A load-testing repository and "cookbook" for open
  source tools
• Regression testing work flow
• QA Process - no tool makes it into the core
  (stable) set of tests
• QA Process - no bug in JIRA is closed without an
  automated test that checks for that bug.

http//bugs.sakaiproject.org/confluence/display/QA
/QATestAutomation
11
Hints to the developer
12
Supporting developers

• Nightly
• builds Continuum Claret
• Javadocs - nightly
• Static code reviews UvA
• Internationalisation - LOI
• Unit testing developers
• Potentially through Continuum
• -Mock objects from Josha Holtzman
• Knowledge capture Confluence, community,
  conferences

13
Hinting nightly
http//qa1-nl.sakaiproject.org
14
Random Example
15
Duplicate Code
16
Do we wish to use the reports more aggressively?

• Are the developers taking note?
• Bugs reported are not always correct
• Bugs found not always important
• But duplicate code, unit test coverage, and
  failing to deal properly with exceptions hint
  strongly at quality.
• Answer probably not, but thankfully trends in the
  marketplace will improve the quality of results
  overtime.

17
The Sash Tool
18
The Sash Tool
AWSOME
http//qa1-nl.sakaiproject.org8080/
19
Realistic Proof of Concept

• Provision tool
• add/delete users,sites,members,tools,permissions
• upload files from file system to
  dropbox/resources
• Controlled by a property file in site resources
• liveTest
• Diagnose
• Report
• But have we enough APIs to do the job?

20
SASH TOOL (Steven Giffens)?

• Live
• Brilliant for learning
• Environment that targets sys_admin
• Easy to extend commands
• Consistent use of cover api's allow services to
  be easily guessed.
• Use to write provision, diagnose, live test
  tools. Just requires imagination and Sakai
  specific knowledge.

21
LIVE
import org.sakaiproject.tool.cover.ToolManager im
port org.sakaiproject.tool.api.Tool import
org.sakaiproject.component.cover.ComponentManager
import org.sakaiproject.content.api.ContentHostin
gService ident"...." all
ToolManager.findTools(null, null) counter1 for
( Iterator i all.iterator() i.hasNext() )
tool i.next() print(""counter"
"tool.getTitle()) print("ID
"tool.getId()) print(identtool.getDescription
()) print (identident) counter counter
1 print ("Interfaces") inters
ComponentManager.getRegisteredInterfaces() for (
Iterator i inters.iterator() i.hasNext() )
inter i.next() print("
"counter""inter) counter
Uses covers
Consistent use of java.utils
Trunk 2.4....Tools104,Interfaces568 live
22
Cover like ...
import org.sakaiproject.test.cover.TestManager im
port org.sakaiproject.testl.api.Test all
TestManager.findTests(TestManager.RUNALL) counter
1 for ( Iterator i all.iterator()
i.hasNext() ) test i.next() print(""co
unter" "test.getTitle()) print(test.getDescr
iption()) try print(test.doTest())
catch(FailedTestException e) print(Error
was e.getMessage()) finally counter

23
Remember

• QA Process - no tool makes it into the core
  (stable) set of tests
• BRAINSTORM ONLY
• Push tests back to the developer
• Runs against a live system
• Can be removed at compile time for production
  distributions
• Non standard Yeuch
• Consistent, and test driven
• Easy to script through Sash
• Only solves a certain significant class of tests

24
Jameleon
http//jameleon.sourceforge.net/
25
Regression testing workflow

• CRITERIA
• Run automatically
• Results consistent
• Many roads to Rome.
• Selenium RC
• Selenium RC wrapper in Maven, Python, Ant
• Cost benefit ratio favourable

26
Jameleon - Proof of Concept

• Wrapper around Selenium RC and other tools
• Junit,Jiffie (Internet Explorer,HttpUnit,generic)?
  
• Configured via XML
• Generates reports
• Can run headless via cron or continuum
• Data driven
• Almost self documenting

27
Data driven

• Makes tests server generic
• Property file per organization
• uva.properties
• nameadmin
• passadmin
• baseqa1-nl.sakaiproject.org8380
• csv with variables defined in first line
• eid
• newbiex
• newbie2

28
Self documenting
http//jameleon.sourceforge.net/
29
XML test cases
lt?xml version '1.0'?gt lttestcase
xmlns"jellyjameleon"gt lttest-case-summary gtAdd
user to SakaiQAlt/test-case-summarygt
lttest-case-author gtAlan Berglt/test-case-authorgt
lttest-case-level gtSMOKElt/test-case-levelgt
ltfunctional-point-tested gtAdd User to Sakai
baseUrllt/functional-point-testedgt
ltselenium-session startSeleniumProxy"true"
application"uva" beginSession"true"
seleniumMultiWindowMode"true"
seleniumStopProxyServerOnSessionClose"true"gt lt
csv name"add_user_sakai" countRow"true"gt
ltselenium-open functionId"Goto Portal"
url"http//base/portal/tool/!gateway-710?panel
Main" /gt ltselenium-type keys"eid_FIRST"
functionId"Type Firstname " locator"namefirst-n
ame" /gt ltselenium-type keys"eid"
functionId"Type username eid"
locator"nameeid" /gt ltselenium-type
keys"eid" functionId"Type password"
locator"namepw" /gt ltselenium-type
keys"eid" functionId"Type verify"
locator"namepw0" /gt ltselenium-type
keys"eid_at_dev.null" functionId"Type mail"
locator"nameemail" /gt ltselenium-type
keys"eid_LAST" functionId"Type Lastname"
locator"namelast-name" /gt ltselenium-click
functionId"Click submit" locator"eventSubmit_doS
ave" functionDelay"1000" /gt
ltselenium-select-window functionId"Select
relevant frame" windowName"null" /gt
ltselenium-click xmlns"jellyjameleon"
functionId"Click submit" locator"linkLogout"
/gt lt/csvgt lt/selenium-sessiongt lt/testcasegt

30
Workflow

• From Cron
• Download and build Sakai
• Copy different scripts to directory
• Run Jameleon
• (can run against IE, Firefox, Opera etc)?
• Generate reports
• Move reports to website
• Can be added as a script to run to continuum

31
Results

• Works
• Selenium plug-in still beta
• Tests are fragile and require maintenance
• Will require a significant effort for basic
  coverage
• Still very promising and should be considered.
• HATS OFF TO THE DEVELOPERS

32
My plans

• I have one day a week free to
• Concentrate on Sash scripts
• Wait until the next release of Jameleon.
• Beg the big brains to build a realistic test
  manager.

33
Questions
?
?
?
?
?