Title: Complex Test Suites Organization
1Complex Test Suites Organization
- Victor Kuliamin
- kuliamin_at_ispras.ru
- ISP RAS, Moscow
2Modern Software Complexity
System Year Size(MLOC) Windows
3.1 1992 3 Windows NT 3.1
1993 6 Windows NT 3.5 1994
10 Windows 95 1995 15 Windows NT
4.0 1996 16.5 Solaris 7.0 1998
7.5 Red Hat Linux 5.2 1998 12 Debian 2.0
1998 25 Windows 2000 2000
29 Red Hat Linux 6.2 2000 17 Sun
StarOffice 5.2 2000 7.6 Debian 2.2
2000 59 Red Hat Linux 7.1 2001
30 Windows XP 2002 40 Red Hat Linux
8.0 2002 50 Debian 3.0 2002
105 Windows Vista Beta 1 2005 50 Fedora Core
4 2005 76 Debian 3.1 2005
216 Mac OS X 10.4 2006 86
- Modern software systems
- Are huge and have a lot of functions
- Have very complex interfaces
- Are developed by hundreds of people
System Year DevTeam Size Windows NT
3.1 1993 200 Windows NT 3.5 1994
300 Windows NT 4.0 1996 800 Debian 1.2
1996 120 Debian 2.0 1998 400
Windows 2000 1999 1400 Debian 2.2
2000 450 Debian 3.0 2002 1000
Debian 3.1 2005 1500
3Test Suites Complexity
System Test Cases MS Word XP
35000 Oracle 10i 100000 Windows XP gt2000000
System Year TestTeam Size Windows NT
3.1 1993 140 Windows NT 3.5 1994
230 Windows NT 4.0 1996 700 (0.9) Windows
2000 1999 1700 (1.2)
System Size(MLOC) Test suite size(MLOC) Java
1.5 5 13 (JCK)
- Scalability of testing techniques is strongly
related - with test suite organization
4Traditional Test Suite Organization
- Set of test cases
- Suitable for test suite size management
- Suitable for failure analysis
- Test suites for complex systems became huge and
poorly manageable - Many common parts and low reuse
- Hard to modify
Test suite
5Full Set of Problems
- Test execution
- Selection the necessary tests
- Test suite configuration
- Test result analysis
- Failure diagnostics
- Test adequacy reporting
- Test creation and maintenance
- Explicit relations with requirements, testing
goals, and components under test - Fight with complexity
- Reuse
6Approaches
- Attributing
- Configuration parameters
- Modularization
7Attributing
Test suite
- Adding a set of test case attributes
- Very simple
- Provides multi-aspect tests classification
- Provides links to requirements and components
under tests - Can help to measure test completeness
1 x 3F7 /tests/usr/t1
2 A B41 /tests/sys/t2
3 x 9E /tests/sys/t3
8Configuration Parameters
- A set of parameters that control test
- execution and test results assessment
- Provide additional flexibility, portability and
adaptability of test suite - Help to check optional requirements
- May be assigned dynamically
- May make test suite too complex
9Modularization
- Separation of modules
- performing specific tasks
- Facilitates reuse
- Enhances maintainability
- Facilitates requirements traceability
- May complicate test suite
- May complicate failure analysis
10Right Modules
- Test run-time support
- Scheduler
- Timer
- Event monitor
- Test adapter
- Test data generator
- Test oracle
11Test Adapters
- Make possible reusable and abstract tests
Test
SUT
Adapter
12Test Data Generators
- Provide data for tests
- Can contain filters or selectors
- Can be composed from smaller data generators
13Test Oracles
- Provide verdict of SUT behavior correctness
?
Test Data Generator
System under Test
Test Oracle
- Partial oracles
- Event or operation oracle
- Data integrity constraint
- Enhance reuse
- Simplify requirements traceability
14Oracle Composition
- Simple caseAny failure detected by partial
oracle is global - Complex data integrity constraints
- Behavior of distributed systemsInterleaving
semantics
?
?
?
?
?
?
?
?
15Conclusion
- Test suite organization techniques improving
testing scalability already exist - The most powerful one is modularization
- But usage of these techniques is restricted by
existing tools - All the techniques presented need proper support
in tools
16- Thank you for attention!
- Any questions?
- kuliamin_at_ispras.ru