Embedded System Testing

1
Embedded System Testing
2
Testing

• Consider how testing techniques may have made
  your state-machine project implementation go
  smoother
• Test button circuit separate
• Test software with debugger (instead of hardware)
• etc.

3
Testing

• You have all undoubtedly done software testing
• Much of what we talk about here will be similar
• Although the differences may be subtle, they can
  have huge effects on system implementation

4
Differences

• As much as we hate to admit it, more often than
  not embedded system software is ugly
• Its not that the programmers are sub-par, but
  due to the fact that there are more constraints
  on the system
• What are some of the constraints?

5
Differences

• Theres more to the system than just software
• In a pure software system (i.e. PC based
  application) we can safely assume that the
  hardware is working properly
• In an embedded system were never quite sure
• Thus, we must devise isolation techniques

6
Differences

• Pure software systems (PC based applications)
  are, for the most part, deterministic systems
• They pretty much know what to expect and when
  to expect it
• Embedded systems generally must react to events
  at arbitrary times
• Example?
• How does this affect testing?

7
Differences

• PC based applications spend a lot of time sitting
  around waiting for the user to do something
• Many embedded systems must operate in real-time

8
Differences

• PC based applications have an operating system to
  guide them along
• This can be good in that the OS does a lot of
  bookkeeping for us
• This can be bad in that the application is
  sharing the underlying OS and hardware with other
  applications
• Embedded systems may or may not have an OS

9
Difficulties

• Real-time and concurrency are often difficult to
  design and difficult to test
• Resource constraints (speed, size, ) are often
  difficult to design and test
• Testing tools are intrusive
• Reliability is absolutely critical
• All of these traits exist within an embedded
  system design

10
Bottom Line
11
PC Based Testing
12
Embedded Systems Testing
13
Why Test?

• Find bugs in both software and hardware
• Reduce product development and maintenance costs
• Improve system performance
• Ensure product acceptance/success in the market
  place

14
Finding Bugs

• It is theoretically impossible to prove in the
  general case that a program is correct
• Proved with The Halting Problem
• But, testing can prove that a program is
  incorrect
• The goal of testing is to detect all the possible
  ways that a system might do the wrong thing
• Dont fall into the trap of telling yourself the
  system is simple, theres no need to test it

15
Cost Reduction

• We saw this chart previously
• And you can now tack on after deployment to the
  end of the graph

16
Performance Improvement

• Embedded systems have numerous constraints of
  which two primary ones are
• Time
• Space
• Testing for performance addresses these
  constraints by
• Optimizing code
• Driving the partitioning decision

17
Market Acceptance

• Embedded systems dont sit in a sterile lab
  environment
• By their vary nature the exist in the harsh, real
  world
• The systems must work as specified
• Thus, many agencies demand thorough testing prior
  to introduction into the market
• DOD, FAA, FCC, QVC
• And, if the agencies dont get you, the consumer
  will

18
When To Test?

• Testing is done in phases
• Design phase
• Prototypes
• Development phase
• Unit tests
• Integration phase
• Regression testing
• More often than not, testing within the design
  and development phases is down played if it
  exists at all

19
Design Phase

• Software developers write snippets of code to
  test algorithms
• Hardware developers burn HDL designs to FPGAs
  to test algorithms/circuits
• These tend to be early proof of concept works
  and may or may not end up in the final design

20
Development Phase

• Unit tests are generally designed and implemented
  by the designer of the unit (module) under test
• Focus is on the operation of the module
• Tests may or may not have anything to do
  directly with the final integrated product
• The fact that a module works in isolation in no
  way guarantees that it will work within the
  system
• Example?
• But, unit tests are important
• When integration begins its nice to know that
  the pieces work individually

21
Integration Phase

• This is where testing to specification begins
• Tests relate directly to the customer
  specification and may involve environmental
  testing, temperature range testingas well as
  does it do what its supposed to do under all
  circumstances testing
• The goal is to find out if the system works the
  way the customer expects it to work
• Method of choice is regression testing
• This involves a suite of tests that are all run
  from beginning to end whenever a change is made
• Quite often these tests are run by an
  organization other than the design team
• A method of keeping everyone honest

22
Testing Techniques

• Black-Box testing (Functional Testing)
• The test designer does not get to see how the
  system is implemented when designing/running the
  tests
• These concentrate solely on the system
  (functional) specification
• White-Box testing (Coverage Testing)
• The test designer knows exactly how the system
  has been implemented and designs tests with
  breaking it in mind
• Gray-Box testing
• These are tests where you have suspicions about
  how a system may be implemented based on past
  experience
• You hammer away at what you know were weak spots
  in similar systems
• Competitors will do this as a means of reverse
  engineering your product

23
Testing Techniques

• Functional tests are the most difficult to design
  (this is where the halting problem comes to play)
• Unfortunately, theyre probably the most
  important
• Coverage tests are easy to implement as there are
  automated tools for doing them
• But if the functional tests are truly complete
  and the code is not fully covered, who cares?

24
When To Stop Testing?

• Coverage tests are easy (easier)when all the
  code has been executed through testing (and the
  tests pass), youre done.
• Functional tests get trickier
• For better or worse, your customer may tell you
  when youre done
• With each round of regression testing you may
  analyze the trends and determine when youre done
• Bottom line is youll never by 100 sure your
  system is correct in the general case so you
  play the statistics gameand hope you come out
  a winner
• It ultimately becomes a money issue

25
How To Test?

• Obviously, testing every possible input and every
  possible combination of paths through the system
  and under every possible environmental external
  condition will ensure success
• Equally obviously, you cant do this
• So, once again you play the statistics game

26
Functional Test Categories

• Stress tests
• Try to break the system by flooding input
  channels with signals, overloading memory,
  reducing power supply inputs, dropping the device
  on a hard surface
• Boundary value tests
• Check the extreme boundaries on input systems
• Especially useful for input power supplies

27
Functional Test Categories

• Exception tests
• Provide inputs that you know will cause failure
  conditions and see how your system handles them
• Remember, this is an embedded system and a BSOD
  may not be a suitable option
• Error guessing
• Figure out how to break a competitors system (or
  your previous version) then check to see how your
  system handles the same situation

28
Functional Test Categories

• Random tests
• Use a Monte Carlo simulation to generate input
  vectors then analyze the outputs/behaviors to
  check for consistency
• Performance tests
• The definition of a working system may have
  time/size/environmental constraints as well as
  getting the right answer constraints
• These should be spelled out in the specification
  and, therefore, included in the functional testing

29
Functional Test Categories

• And dont forget to test those cases that you
  know will never occur in the real world
• because they will!

30
Coverage Test Categories

• Statement coverage
• Design tests to execute every line of code at
  least once
• Decision or branch coverage
• Make sure every condition of/path through a
  conditional (if/then/else or switch) statement is
  executed
• Condition coverage
• Make sure every piece of a compound condition
  within a conditional statement is tested
• Consider this gem(if with no else)
• These clearly require white-box testing and may
  need external devices to force certain conditions

31
Failure Modes

• Always crashes (does the wrong thing) in the same
  place
• Relatively easy to detect and correct
• Standard techniques usually suffice
• Crashes (does the wrong thing) at seemingly
  arbitrary places
• Very difficult to detect and correct
• Problems in this category are generally due to
  timing (race conditions), sequencing (ordering)
  of events, or number (size) of events
• Recreating these conditions is often very
  difficult and time consuming, if not impossible
• H/W S/W co-verification discussed previously
  may be helpful here
• These often lead to redesigns in either the
  software, hardware, or both

32
Instrumentation For Functional Testing

• In general, you dont get System.out.println()
  functionality
• If you do via a monitor program like the Stamp
  IDE then youre not testing the real system
• You may resort to hardware devices (logic
  analyzers, oscilloscopes, ROM emulators) but
  these things may add loading i.e. capacitance
  to a system which either hides the error or
  causes a different error

33
Instrumentation For Coverage Testing

• Three categories (as specified previously)
• Statement Coverage instrumentation to verify
  that all statements are exercised
• Decision Coverage instrumentation to verify
  that all paths through all conditionals are
  exercised
• Modified Condition Decision Coverage
  instrumentation to verify that all terms of
  complex conditions are exercised
• Basically, the instrumentation consists of memory
  spies, logic analyzers, and PC side software to
  analyze the recorded results
• Its all statistics gathering
• The amount of data gathered can be staggering
  consider a recursive function call

34
Instrumentation For Performance Testing

• Again, its done with external devices
• Logic analyzers and oscilloscopes are set to
  catch triggering events and measure the time to
  respond, etc.

35
Testing Is Hard

• In all three cases (functional, coverage,
  performance) were dependant on external devices
  that may alter the functionality of the embedded
  system
• In a PC based application, we see a similar
  situation, but not as often, when we run a debug
  build vs. a release build

36
Remember

• Its safe to assume that the intentions of the
  designer were good
• Its equally safe to assume that the implementer
  fouled up the implementation
• Even if theyre the same person
• Therefore, test even the simplest of things!

37
An interesting example

• To save space, P14 was used for both the button
  and the servo
• It appeared to work (some/most of the time) and
  since the buttons are fraught with issues such as
  bounce and synchronization (with the software) it
  was easy to write off as an issue with the
  BasicStamp subsystem

38
An interesting example

• But, what was the real problem?
• Why did it appear to work sometimes and not
  others?