Designing User Interfaces Spring 1999

1
SE 746-NT Embedded Software Systems
Development Robert Oshana Lecture
35 For more information, please
contact NTU Tape Orders NTU Media
Services [phone]
oshana_at_airmail.net
tapeorders_at_ntu.edu
2
Embedded Systems Testing Part 2
From Beatty ESC 2002
3
Agenda

• More verification techniques
• Code inspection checklist
• Unit test standards

4
Testing

• Organized process of identifying variances
  between actual and specified results
• Goal zero significant defects

5
Testing axioms

• All software has bugs
• Programs cannot be exhaustively tested
• Cannot prove the absence of all errors
• Complex systems often behave counter-intuitively
• Software systems are often brittle

6
Finding spec/design problems

• Reviews / Inspections / Walkthroughs
• CASE tools
• Simulation
• Prototypes

Still need consistently effective methods !
7
Testing Spec/Design Reviews

• Can be formal or informal
• Completeness
• Consistency
• Feasibility
• Testability

8
Testing Evaluating methods

• Relative costs
• None
• Low
• Moderate
• High

• General effectiveness
• Low
• Moderate
• High
• Very high

9
Testing Code reviews

• Individual review
• Effectiveness high
• Cost Time low, material - none

• Group inspections
• Effectiveness very high
• Cost Time moderate, material - none

10
Testing Code reviews

• Strengths
• Early detection of errors
• Logic problems
• Math errors
• Non-testable requirement or paths
• Weaknesses
• Individual preparation and experience
• Focus on details, not big picture
• Timing and system issues

11
Step by step execution

• Exercise every line of code or every branch
  condition
• Look for errors
• Use simulator, ICE, logic analyzer
• Effectiveness moderate dependent on tester
• Cost time is high, material is low or moderate

12
Functional (Black Box)

• Exercise inputs and examine outputs
• Test procedures describe expected behavior
• Subsystems tested and integrated
• Effectiveness is moderate
• Cost time is moderate, material varies

Tip where functional testing finds problems look
deeper in that area !
13
Functional (Black Box)

• Strengths
• Requirements problems
• Interfaces
• Performance issues
• Most critical/most used features
• Weaknesses
• Poor coverage
• Timing and other problems masked
• Error conditions

14
Functional test process

• ID requirements to test
• Choose strategy
• 1 test per requirement
• Test small groups of requirements
• Scenario broad sweep of many requirements
• Write test cases
• Environment
• Inputs
• Expected outputs
• Traceability

15
Structural (White box)

• Looks at how code works
• Test procedures
• Exercise paths using many data values
• Consistency between design and implementation
• Effectiveness high
• Cost time is high, material low to moderate

16
Structural (White box)

• Strengths
• Coverage
• Effectiveness
• Logic and structure problems
• Math and data errors
• Weaknesses
• Interface and requirements
• Focused may miss big picture
• Interaction with system
• Timing problems

17
Structural (White box)

• Test rigor based on 3 levels of Risk (FAA)
• C reduced safety margins or functionality
• Statement coverage
• Invoke every statement at least once

18
Structural (White box)

• Test rigor based on 3 levels of Risk (FAA)
• B Hazardous Decision Coverage
• Invoke every statement at least once
• Invoke every entry and exit
• Every control statement takes all possible
  outcomes
• Every non-constant Boolean expression evaluated
  to both a True and a False result

19
Structural (White box)

• Test rigor based on 3 levels of Risk (FAA)
• A Catastrophic Modified Condition Decision
  Coverage
• Every statement has been invoked
• Every point of entry and exit has been invoked

20
Structural (White box)

• Every control statement has taken all possible
  outcomes
• Every Boolean expression has evaluated to both a
  True and a False result
• Every condition in a Boolean expression has
  evaluated to both True/False
• Every condition in a Boolean expression has been
  shown to independently affect that expressions
  outcome

21
Unit test standards

• What is the white box testing plan?
• What do you test?
• When do you test it?
• How do you test it?

22
Structural test process

• ID all inputs
• ID all outputs
• ID all paths
• Set up test cases
• Decision coverage
• Boundary value analysis
• Checklist
• Weaknesses

23
Structural test process

• Measure worst case execution time
• Determine worst case stack depth
• Bottom up

24
Integration

• Combines elements of white and block box
• Unexpected return codes or acknowledgements
• Parameters boundary values
• Assumed initial conditions/state
• Unobvious dependencies
• Aggregate functionality

25
Integration

• Should you do thiswhen?
• Depends on the complexity of the system
• Boundary values of parameters in functions
• Interaction between units
• interesting paths
• Errors
• Most common

26
Verification

• Verify the structural integrity of the code
• Find errors hidden at other levels of examination
• Outside of requirements
• Conformance to standards

27
Verification

• Detailed inspection, analysis, and measurement of
  code to find common errors
• Examples
• Stack depth analysis
• Singular use of flags/variables
• Adequate interrupt suppression
• Maximum interrupt latency
• Processor-specific constraints

28
Verification

• Strengths
• Finds problems that testing and inspection cant
• Stack depth
• Resource sharing
• Timing
• Weaknesses
• Tedious
• Focused on certain types of errors

29
Verification

• Customize for your process/application
• What should be checked
• When
• How
• By whom

30
Stress/performance

• Load the system to maximumand beyond!
• Helps determine factor of safety
• Performance to requirements

31
Stress/performance

• Examples
• Processor utilization
• Interrupt latency
• Worst time to complete a task
• Periodic interrupt frequency jitter
• Number of messages per unit time
• Failure recovery

32
Other techniques

• Fault injection
• Scenario testing
• Regression
• Critical functions
• Most functionality with the least tests
• Automation
• Risk of not re-testing is higher than the cost
• Boundary value testing

33
Tools
ICE Simulator Logic analyzer
Step through code X X X
Control execution X X
Modifying data X X
Coverage X X X
Timing analysis X X X
34
Code Inspection Checklist
35
Code Inspection Checklist

• Code correctly implements the document software
  design
• Code adheres to coding standards and guidelines
• Code is clear and understandable
• Code has been commented appropriately
• Code is within complexity guidelines
• Cyclomatic complexity lt 12

36
Code Inspection Checklist

• Macro formal parameters should not have side
  affects (lint message 665)
• Use parenthesis to enhance code robustness, use
  parenthesis around all macro parameters (665,
  773)
• Examine all typecasts for correct operation
• Examine affects of all implicit type conversions
  (910-919)

37
Code Inspection Checklist

• Look for off-by-one errors in loop counters,
  arrays, etc
• Assignment statements within condition
  expressions (use cchk)
• Guarantee that a pointer can never be Null when
  de-referencing it
• Cases within a switch should end in a break (616)

38
Code Inspection Checklist

• All switch statements should have a default case
  (744)
• Examine all arguments passed to functions for
  appropriate use of pass by value, pass by
  reference, and const
• Local variables must be initialized before use
• Equality test on floating point numbers may never
  be True (777)

39
Code Inspection Checklist

• Adding and subtracting floats of different
  magnitudes can result in lost precision
• Insure that division by zero cannot occur
• Sequential multiplications and divisions may
  produce round-off errors

40
Code Inspection Checklist

• Subtracting nearly equal values can produce
  cancellation errors
• C rounds towards zero is this appropriate here
  ?
• Mathematical underflow/overflow potential
• Non-deterministic timing constructs

41
Unit test standards
42
Unit test standards

• 1. Each test case must be capable of independent
  execution, i.e. the setup and results of a test
  case shall not be used by subsequent test cases
• 2. All input variables shall be initialized for
  each test case. All output variables shall be
  given an expected value, which will be validated
  against the actual result for each test case

43
Unit test standards

• 3. Initialize variables to valid values taking
  into account any relationships among inputs. In
  other words, if the value of a variable A affects
  the domain of variable B, select values for A and
  B which satisfy the relationship
• 4. Verify that the minimum and maximum values can
  be obtained for each output variable (i.e. select
  input values that produce output values as close
  to the max/min as possible)

44
Unit test standards

• 5. Initialize output variables according to the
  following
• If an input is expected to change, set its
  initial value to something other than the
  expected result
• If an output is not expected to change, set its
  initial value to its expected value
• 6. Verify loop entry and exit criteria

45
Unit test standards

• 7. Maximum loop iterations should be executed to
  provide worst case timing scenarios
• 8. Verify that the loss of precision due to
  multiplication or division is within acceptable
  tolerance

46
Unit test standards

• 9. The following apply to conditional expressions
• OR expressions are evaluated by setting all
  predicates FALSE and then setting each one
  TRUE individually
• AND expressions are evaluated by setting all
  predicates TRUE and then setting each one
  FALSE individually

47
Unit test standards

• 10. Do not stub any functions that are simple
  enough to include within the unit test
• 11. Non-trivial tests should include an
  explanation of what is being tested

48
Unit test standards

• 12. Unit test case coverage is complete when the
  following criteria are satisfied (where
  applicable)
• 100 function and exit coverage
• 100 call coverage
• 100 statement block coverage
• 100 decision coverage
• 100 loop coverage
• 100 basic condition coverage
• 100 modified condition coverage

49
Unit test checklist
Common coding error checks Status Note(s)
Mathematical expression underflow/overflow ltPass/Fail/NAgt
Off-by-one errors in loop counters ltPass/Fail/NAgt
Assignment statements within conditional expressions ltPass/Fail/NAgt May be detected by compiler, lint, cchk
Floats are not compared solely for equality ltPass/Fail/NAgt Lint message 777
Variables and calibrations use correct precision and ranges in calculations ltPass/Fail/NAgt
50
Unit test checklist
Common coding error checks Status Note(s)
Pointers initialized and de-referenced properly ltPass/Fail/NAgt
Intermediate calculations are not stored in global variables ltPass/Fail/NAgt
All declared local variables are used in the function ltPass/Fail/NAgt May be detected by compiler or lint
Typecasting has been done correctly ltPass/Fail/NAgt
Unreachable code has been removed ltPass/Fail/NAgt Lint message 527
51
Common coding error checks Status Note(s)
All denominators are guaranteed to be zero (no divide by 0) ltPass/Fail/NAgt
Switch statement handle every case of the control variable (have DEFAULT paths). Any cases that fall through to the next case are intended to do so ltPass/Fail/NAgt Lint message 744, 787 Fall through 616
Static variables are used for only one purpose ltPass/Fail/NAgt
All variables have been properly initialized before being used assume a value of 0 after power up ltPass/Fail/NAgt
52
SE 746-NT Embedded Software Systems
Development Robert Oshana 10 minute
break For more information, please
contact NTU Tape Orders NTU Media
Services [phone]
oshana_at_airmail.net
tapeorders_at_ntu.edu