Concurrent Testing

1
Concurrent Testing

• Wai Hung Ng
• Jean Paul Nasab

2
Agenda

• What is Concurrent Testing?
• Sequential vs. Concurrent
• Problems in Concurrent Testing
• Attempt to Isolate Complexity
• Factors in Concurrent Tests
• From Unit Test to Concurrent Test
• Components to Monitor
• Load Testing
• Load Testing Metrics
• Concurrent Testing in SOA
• Model
• How to Test?
• Automation and Reusability?
• Current Products in Markets

3
What is Concurrent Testing?

• Multi-user testing used to determine the effects
  of accessing the same application code, module or
  database records at the same time.
• Concurrent Testing identifies and measures the
  level of locking, deadlocking and use of
  single-threaded code and restricting access to
  shared resources .

4
What is Concurrent Testing?(cont)

• Concurrent Testing is an appropriate way to
  determine many performance
• Bottlenecks capacity of an entire system is
  limited by a single component
• Resource contention issues Resource capability
• Buffer contention
• I/O contention
• Service interruptions Priority Management

5
What is Concurrent Testing? (cont)

• Concurrency is a property of systems in which
  several computational processes are executing at
  the same time, and potentially interacting with
  each other.
• Because the processes in a concurrent system can
  interact with each other while they are
  executing, the number of possible execution paths
  in the system can be extremely large, and the
  resulting behavior can be very complex.

6
Concurrent Testing Is Difficult

• Testing any software is difficult because of the
  large number of paths through the code. Testing
  concurrent software is more difficult because of
  non-determinism. Testing must account for
  differences in program execution that result from
  thread interleavings that change each time the
  program is run.
• Interleaving means services that interleave (run)
  at the same time.
• Multiple executions of a concurrent program with
  a fixed input may exercise different sequences of
  synchronization events and produce different
  results. This non-deterministic behavior makes
  testing concurrent programs difficult.

7
Problems Encountered

• Non-determinism is a serious problem in the
  testing and debugging of concurrent programs.
• Thread interleaving may be different on each run,
  changing the order of events.
• Testing a concurrent program requires many of
  these orders to be run, either all orders produce
  correct results or identify the presence of
  timing dependent errors called race conditions.
• We can not only produce a specific execution to
  locate the problem

8
Sequential vs. Concurrent

• Concurrent programs behave differently than
  sequential programs
• Testing criteria for sequential programs do not
  care the two characteristic of concurrent
  programs.
• One is nondeterministic execution
• Interactions between process (tasks)
  (Synchronizations, communications and wait)
• Synchronization between two statements means that
  one statement necessarily waits the other
  statement

9
Sequential vs. Concurrent

• Additional different not present in sequential
  programs such as
• race conditions and data races
• process where by the output of the process is
  unexpectedly and critically dependent on the
  sequence or timing of other events
• Deadlocks
• situation wherein two or more competing actions
  are waiting for the other to finish, and neither
  ever does
• Livelock
• The states of the processes change with regard to
  one another but none progressing
• The failure modes of concurrent programs are less
  predictable and repeatable than sequential
  programs
• Reproducing the failures is difficult

10
Problems in Concurrent Testing

• Minimum functions a test automation tool needs to
  support to expose concurrency issues
• Unit test of a specific function.
• Variations on the input data the unit test sends
  to a function.
• Business flows through a system function using
  sequences of functional unit tests.
• Operating the business flows in concurrently
  running threads.

11
Attempt to Isolate Complexity

• Encapsulation
• The value of encapsulation is that it makes it
  possible to analyze the behavior of a portion of
  a program without having to review the code for
  the entire program.
• Concurrent Interactions
• Once the concurrent interactions are
  encapsulated, the majority of testing efforts
  primarily will be on the concurrency mechanisms
  themselves.
• Concurrent interactions could be done in a few
  places, such as
• Workflow managers
• Resource pools
• Work queues
• Other concurrent objects
• it becomes simpler to analyze and test concurrent
  programs

12
Attempt to Isolate Complexity

• By limiting the scope of concurrent interactions
  to a few widely-used, well-tested components, It
  greatly reduce the amount of effort required to
  test an application.
• By reusing existing tested library components,
  such as the classes, package, services, it
  reduces the further testing burden.

13
Factors in Concurrent Tests

• Tests are probabilistic.
• Explore more of the state space.
• Explore more interleavings.
• Match thread count to the platform.
• Avoid introducing timing or synchronization
  artifacts.
• It is Non-deterministic!

14
From Unit Test to Concurrent Test

• many test automation tools are now using unit
  tests in ways that aid in surfacing concurrency
  problems
• How?

15
From Unit Test to Concurrent Test

• Stringing unit tests together into a sequence
  forms a functional test of a business flow.
• find a test automation framework wrapping unit
  tests into functional tests that may be operated
  concurrently to identify concurrency issues in an
  application

16
From Unit Test to Concurrent Test

• Functional Test
• operating a sequence of unit tests once
• Load Test
• operating sequences of the unit tests in
  concurrently running threads
• Service Monitor
• running the unit tests periodically
• As the TestScenario operates the test, a monitor
  watches the application host and logs usage and
  resource statistics. The test tools then
  correlate the monitored data to the unit test
  operation to identify performance bottlenecks and
  available optimizations.

17
Components to Monitor

• The key to using a test automation platform for
  concurrency testing is to identify what is
  happening on the server side
• Run-time statistics (heap, memory, thread, object
  instantiations, and garbage collection)
• Service call statistics (time to serve request,
  external service and database query times,
  properties settings, and service interface
  request values and schema)
• Subsystem statistics (if the system uses a
  database then I want to see the transaction logs,
  input parameters, and query operations)
• manual expense of the time to correlate the
  observed data to what the test automation tool is
  operating on the client-side

18
Load Testing

• most test automation tools produce a Scalability
  Index chart that shows Transaction Per Second
  (TPS) ratings for increasing levels of concurrent
  requests to a service
• Red 20 concurrent users
• Blue 40 concurrent users

19
Load Testing Metrics

• Respond time
• Number of transactions passed/ failed
• Throughput
• Load size
• CPU utilization
• Memory utilization
• Wait time

20
Examples of Load Testing

• Glassbox
• Automated test and troubleshooting tool for Java
• Do not have to have knowledge in codes
• Glassbox Inspector monitor traffics through JVM
• Virtual Application Model is constructed
• Analyze which part of the program is the
  bottleneck
• Isolate problematic parts of the application for
  analyze
• Load Runner
• Analyze which part of the program is the
  bottleneck
• Virtual User Generator to add concurrent users
• Virtual Application Model is constructed
• Analyze which part of the program is the
  bottleneck
• Isolate problematic parts of the application for
  analyze

21
Concurrent Testing in SOA

• Services in SOA
• are stand alone units
• are developed by different entities
• Have different coding styles, thus
  characteristics
• No specification or restriction guideline to
  follow
• But work together for one workflow controller
• SOA nowadays are web-based
• Very high chance for more than one instances or
  threads running at the same time
• Exceptionally important for concurrent testing!!

22
Concurrent Testing in SOA

• Identify concurrent services in the workflow
  controller, group interleaved services in small
  groups for isolated testing
• Identify services who are NOT controlled and pay
  more attention to test those services
• Since SOA services are plug and play, new
  services could be just grab and add on-demand,
  the workflow controller is either handling
  concurrencies in a generalized way, or there are
  no concurrency controls

23
How to Test?

• From Non-deterministic to Deterministic
• Analyze combinations of service runs
• Identify I/Os and traffics of each services
• Interleave those I/Os and traffics to form the
  combination set
• examining all interleavings for a given use of a
  shared variable
• Potentially a huge set of combinations
• In-order to facilitate concurrent testing,
  service publications should also include its
  I/Os specification such that test cases can be
  generated automatically

24
Model

• Testing Engine should be able to do the
  following
• Grab I/O specifications of the services
• Automatically generate deterministic test cases
  with the I/O specifications to simulate all
  interleaved situations
• Collect server, database and connector statistics
• Auto verify results for accuracy
• Visualize traffic and architecture reports
• Re-generate test cases upon service changes
• Generalized test engine, if services conform to
  I/O specifications

25
Automation and Reusability?

• Test case generation
• Data collection
• Result Analysis?
• A finite set of expected correct results could be
  generated
• Perform testing and get actual results
• Testing in deterministic
• Insert to test driver for comparison
• Correctness - Just like ordinary SOA testing
• Load Monitoring
• Performance Analysis
• Yes, it could, and it should, as it is SOA
  component

26
Three Aspects

• Functionality
• Should always get the correct expected results
• Interoperability
• Consistency of the results
• Memory allocations
• Deadlocks?
• Performance
• Services meet performance criteria
• Monitor any service degradation
• Database Loading
• Network and traffic capability
• SOAP bindings and message service performances

27
Current Products in Markets

• iTKO LISA SOA Testing Suite
• Test heterogeneous components
• Test workflow
• Concurrency testing ability for load and
  performance testing
• Borland SilkPerformer
• Visualized interfaces
• Automations
• Check server statistics and loads

28
Reference

• Frank Cohen Concurrency Testing in Java
  Applications http//www.theserverside.com/tt/knowl
  edgecenter/knowledgecenter.tss?lConcurrencyTestin
  gJavaApps
• Tetsuro Katayama, Eisuke Itoh, Zengo Furukawa
  Test-Case Generation for Concurrent Programs
• http//earth.cs.miyazaki-u.ac.jp/kat/papers/pdf/
  apsec99.pdf
• Deterministically Executing Concurrent Programs
  for Testing and Debugging http//plg.uwaterloo.ca/
  stevem/papers/PLC2006.pdf