TEST SUITE DEVELOPMENT FOR CONFORMANCE TESTING OF EMAIL PROTOCOLS

1
TEST SUITE DEVELOPMENT FOR CONFORMANCE TESTING OF
EMAIL PROTOCOLS

• Anastasia Tugaenko
• Scientific Adviser
• Nikolay Pakulin, PhD
• Institute for System Programming RAS, Moscow

SYRCoSE 2010. 1-2 June, 2010. Nizhny Novgorod,
Russia
2
Conformance Testing

• Many implementations of protocols from different
  developers are functioning in the contemporary
  Internet
• The reliability of data transfer substantially
  depends on implementations compatibility
• The basic method of attesting implementations
  compatibility is conformance testing (testing to
  conform to the standard)

3
Simple Mail Transfer Protocol
User agent (UA)
Relay mail transfer agent (Relay MTA)
Local mail transfer agent (Local MTA)
Internet
Local mail transfer agent (Local MTA)
Relay mail transfer agent (Relay MTA)
User agent (UA)
4
Simple Mail Transfer Protocol
S 220 foo.com Simple Mail Transfer Service Ready
C EHLO bar.com S 250-foo.com greets bar.com
S 250-8BITMIME S 250 HELP C MAIL
FROMltSmith_at_bar.comgt S 250 OK C RCPT
TOltJones_at_foo.comgt S 250 OK C RCPT
TOltGreen_at_foo.comgt S 550 No such user here C
RCPT TOltBrown_at_foo.comgt S 250 OK C DATA S
354 Start mail input end with ltCRLFgt.ltCRLFgt C
Blah blah blah... C ...etc. etc. etc. C . S
250 OK C QUIT S 221 foo.com Service closing
transmission channel
5
Post Office Protocol (version 3)
User Agent
POP server
commands
User
Mailbox
replies and data
6
Post Office Protocol (version 3)
S OK POP3 server ready C USER mrose S OK
mrose is a real hoopy frood C PASS secret S
OK mrose's maildrop has 2 messages (320 octets)
C LIST S OK 2 messages (320 octets) S 1
120 S 2 200 S . C RETR 1 S OK 120 octets
S ltthe POP3 server sends message 1gt S . C
DELE 3 S -ERR no such message C QUIT S OK
dewey POP3 server signing off (maildrop empty)
7
Mail Protocols Features

• Mail protocols are underspecified
• Mail protocols are nondeterministic
• Mail protocols requirements differ in the level
  of obligations (MUST, SHOULD, MAY, )
• Protocol architecture is extensible

8
Requirements for Test Suite

• Requirements traceability
• Availability of option defining the set of
  requirements supported by IUT
• Completeness in terms of requirements coverage

9
Traditional Methodology of Conformance Testing

• Test suite consists of formal given tests which
  are not connected to the implementation
• Connections between requirements and test
  purposes and between test purposes and their
  tests are informal
• Test purposes describe situations to be tested.
  Test purpose is realized in one or several tests
• Implementation is considered to conform to the
  standard if all test purposes have been passed

10
Testing with Mail Protocol Tester (James MPT)
Script
S 220. -- expected
response (pattern) C HELO example.org -- test
stimuli S 250. C QUIT S 221.
Result
Connecting to localhost25 lt-220 petr SMTP Server
(JAMES SMTP Server 2.3.1) ready ltdategt -gt HELO
example.org lt-250 petr Hello example.org
(localhost 127.0.0.1) -gt QUIT lt-221 2.0.0 petr
Service closing transmission channel closing
11
UniTESK Technology

• Specifications formalize requirements as pre- and
  postconditions
• From specifications test oracles are generated
• Test sequence is generated from test state
  machine traversal
• Observable behavior is automatically verified by
  test oracles

12
Method for Mail Protocols Conformance Testing
Developing of elementary tests
Creation of requirements catalogue
Designing of lite protocol model
Building test state machine with dedicated states
Requirements formalization
Enhancement of scenario and specification
Execution of test suites and analyzing the results
13
Method for Mail Protocols Conformance Testing
Developing of elementary tests
Creation of requirements catalogue
Designing of lite protocol model
Building test state machine with dedicated states
Requirements formalization
Enhancement of scenario and specification
Execution of test suites and analyzing the results
14
Method for Mail Protocols Conformance Testing
Developing of elementary tests
Creation of requirements catalogue
Designing of lite protocol model
Building test state machine with dedicated states
Requirements formalization
Enhancement of scenario and specification
Execution of test suites and analyzing the results
15
Method for Mail Protocols Conformance Testing
Developing of elementary tests
Creation of requirements catalogue
Designing of lite protocol model
Building test state machine with dedicated states
Requirements formalization
Enhancement of scenario and specification
Execution of test suites and analyzing the results
16
Method for Mail Protocols Conformance Testing
Developing of elementary tests
Creation of requirements catalogue
Designing of lite protocol model
Building test state machine with dedicated states
Requirements formalization
Enhancement of scenario and specification
Execution of test suites and analyzing the results
17
Method for Mail Protocols Conformance Testing
Developing of elementary tests
Creation of requirements catalogue
Designing of lite protocol model
Building test state machine with dedicated states
Requirements formalization
Enhancement of scenario and specification
Execution of test suites and analyzing the results
18
Method for Mail Protocols Conformance Testing
Developing of elementary tests
Creation of requirements catalogue
Designing of lite protocol model
Building test state machine with dedicated states
Requirements formalization
Enhancement of scenario and specification
Execution of test suites and analyzing the results
19
Example Generated Automaton for SMTP Protocol
20
Example Generated Report
21
Results

• Number of main requirements
• SMTP RFC 5321, 51 requirements
• POP3 RFC 1939, 58 requirements
• Size of the test suite
• Specifications SMTP 1400 lines, POP3 1600 lines
• Test scenarios 7 scenarios, 2500 lines total
• Mediators 3000 lines
• Implementations tested
• James 5 defects revealed
• HMail 2 defects revealed

22
Method Analysis the Lows

• The absence of quick test suites updating ability
  (but after all preparations have done one got not
  a single test but a set of tests responsible for
  certain requirements class)
• Formal specification description is very
  laborious process

23
Method Analysis the Advantages

• High order of reusing test suite components
• Test oracles
• Test data iterators
• Automated generation of test sequences
• Automated calculation of reached test coverage
• Automated verdict pronouncement

24
Conclusion

• The method for test sequence generation for email
  protocols implementation testing is presented
• Test suite for conformance testing of email
  protocols is constructed
• The following disagreements between email
  implementations and standards were detected
• Absence of required commands supporting
• Protocol rules violation
• Wrong reply codes to the protocol commands
• Cycling while redirecting mail

25
Thank You!Questions?
Anastasia Tugaenko tugaenko_at_ispras.ru