Chapter 9: Development tools

1
Chapter 9 Development tools

• What special support is required of development
  tools for embedded systems?
• We will begin by considering a native tool
  chain for desktop systems.
• Review from ECEn 324
• We will then consider differences for embedded
  tool set.

2
COSTELLO.C int whosonfirst(int x)
ABBOTT.C int idunno whosonfirst(idunno)
Native tool chain
Compiler translates C source files to machine
instructions, produces object files.
compiler
compiler
ABBOTT.OBJ MOVE R3,(idunno) CALL whosonfirst
COSTELLO.OBJ whosonfirst
Linker resolves addresses of different object
files in terms of offsets from base address of
program.
linker
memory
HAHAHA.EXE MOVE R1,2388 CALL 1547 MOVE
R1, R5 (value of idunno)
HAHAHA.EXE MOVE R1,22388 CALL 21547 MOVE
R1, R5 (value of idunno)
Loader loads program, modifies absolute
addresses in executable based on actual starting
location in physical memory
loader
1547
21547
2388
22388
3
Linker

• Determines addresses of labels that assembler
  could not resolve.
• Typically extern functions variables defined in
  other files.
• Assembler will have marked as needing to be
  fixed
• Instructions referencing these labels
• Data (pointers) initialized to address of other
  variables
• The linker puts together files, fixing up the
  address references between the files.
• If unresolved labels exist at this point, it is a
  fatal error.

4
Loader

• In a regular computer, a program can be loaded
  into a different region of memory every time it
  is executed.
• With no virtual memory, loader must fix-up
  absolute memory addresses in code and data based
  on load address.
• PC-relative addresses require no changes.
• Address modification not required in a system
  with virtual memory.
• Why? Lets review how a VM system works.

5
Virtual memory

• All addresses in program are virtual, not
  physical.
• A hardware address translation unit translates
  virtual addresses to physical addresses.
• Memory Management Unit (MMU)
• Operating system sets up the page tables that the
  hardware uses to do translation, and handles the
  exceptions.
• Each legitimate page is in memory or on disk.
• Allocation flexible need not be contiguous or in
  sequence.
• Bottom line VM simplifies actions of loader,
  allows application code to be larger than
  physical memory
• Downside OS complexity, overhead of page faults
  buffer misses

6
Tool chain for embedded systems
C and C source files
Assembly source files

• Note components
• cross-compiler
• cross-assembler
• linker/locator
• no loader
• Questions
• Why no assembler shown in native tool chain?
• How does this setup compare with our tools?
• How does executable file make it from host to
  target?

cross-assembler
cross-compiler
Object files
Object files
linker/locator
Executable file
operations on host
Executable file is copied to target somehow
target system
7
Cross-Compilers

• C code not perfectly portable what are
  consequences?
• Assume that your C code compiles with native
  compiler and runs correctly on host.
• What kinds of things go wrong in moving code to
  target?
• Using functions before declaring
• Using old style function declarations
• Word sizes (and hence int variables) may be
  different
• C structs may be packed differently
• Alignment restrictions may change data layout or
  limit accesses

8
Locator

• Quite different functionality than native linker
• Determines final memory image of program
• No loader will come along after and fix up
  addresses
• Locator can do this because
• No other program will be in memory at runtime no
  resource conflicts
• Locator can determine final address of
  everything, including kernel and library
  functions called in application code
• Locator includes a mechanism for programmer to
  determine placement in memory
• Some parts need to be in RAM, others in ROM

9
Intel hex file format
. . . 10106000FF908193E03400FAA9077B021227BE901B
101070008193E0FEA3E0FF128157124FD1125BDB98 10108
000EF700F1222EBEF70097B057A1279BB121C 0510900027B
E1212C48E 011095002238 011096002237 10109700E49
082AEF012278B7f197E007D807C0062 0410A7001281C622C
A 0110AB002222 1010AC007F807E0012808EEF60149082A
EE0700321 . . .
Example locator file format

• Other formats exist with similar information
  included.
• Essential information
• Data to write in memory
• Address to write data to

Checksum for the line
The data
Indication that this line contains data (and not
other kinds of information allowed in hex files)
First character always a colon
Address where these data bytes are to be written
in ROM
Count of data bytes on this line
10
More on memory placement

• Recall these constraints
• Code, initial values of data need to be in ROM.
• Writable data needs to be in RAM at runtime.
• How does the linker/locator know to put certain
  things in address space corresponding to RAM or
  ROM?
• Program and data are divided into segments, each
  of which is treated independently.
• Segments are contiguous portions of the runtime
  program.
• Contents are similar within a segment.

11
Why segments?

• Major benefit each segment can be placed
  separately at desired location in physical
  memory.
• Examples
• Special Start Code segment can automatically be
  placed where processor begins execution after
  reset. (main() in C code)
• Code segments can be placed in ROM.
• Constant data segments can be placed in ROM.
• Modifiable data segments can be placed in RAM.

12
Segment creation

• Created automatically by compiler.
• Takes place in desktop systems too, but usually
  transparent to user.
• Application programmer must make them explicit in
  assembly files.
• Assemblers not sophisticated enough to manage
  transparently.
• Naming should be consistent with
  compiler-generated segments.
• All systems have similar categories of segments.
• Actual names depend on tools and developer.

13
x.c
y.c
z.asm
Example segment use
Note segments startcodeudataidatas
tring How did each segment get created,
named? Why useful to separate segments in this
way? How does initialized data gets initialized?
cross-assembler
cross-compiler
cross-compiler
x.obj
y.obj
z.obj
x.c code
y.c code
z.asm code
x.c udata
y.c udata
z.asm udata
x.c string
y.c idata
z.asm start
linker/locator
z.asm start
x.c udata
x.c code
y.c udata
z.asm udata
y.c code
z.asm code
y.c idata
x.c string
RAM
idatshadw copied to idata at startup time
y.c idatshadw
ROM
14
Instructions to the locator -CSTART,IVECS,CODE0
-IDATA,UDATA,CSTACK8000
How linker/locator knows where to place segments
Memory

• Instructions can be
• given on command line
• (placed in makefile), or
• included in user-created
• assembly file.
• Starting address or ending address may be
  specified.
• Segments may be treated as a group or
  individually.

0
CSTART
IVECS
CODE
Resulting program in memory
(unused)
8000
IDATA
UDATA
CSTACK
(unused)
15
Handling initialized data

• How does normal tool chain handle initialization
  in code at right?
• How can it be handled in embedded tool chain?
• Initial value must be in ROM.
• Variable must be in RAM.
• Value must be copied at startup.
• No loader, so application responsible for
  copying.
• Many locators will automatically insert code to
  copy, but may require tinkering.
• Common technique copy shadow segments (with
  initial values) from ROM to RAM at startup.

define FREQ_DEFAULT 2410 ... static int iFreq
FREQ_DEFAULT ... void vSetFreq(int iFreqNew)
iFreq iFreqNew
16
Other initialization issues

• C standard specifies that any uninitialized
  non-stack variable starts with value of zero.
• May not be true of your embedded tools.
• Startup code may be inserted to do this, but
  dont count on it.
• Constant strings char sMSg Reactor is
  melting!
• Where does the system store the constant string?
• Few problems if only operation involving string
  is printing it.
• What if you modify string? strcpy (sMsg11,
  OK)
• Perfectly legal C
• Cross-compilers deal with this problem in
  different ways.

17
LINK MAP OF MODULE XYZ TYPE BASE LENGTH RELOC
ATION SEGMENT NAME -------------------------------
--------------------------------------------------
-------------------------------
X D A T A M E M O R Y
0000H 8100H
GAP XDATA 8101H
0001H UNIT ?XD?PROGFLSH XDATA 8101H
000CH UNIT ?XD?VPROG?PROGFLSH XDATA 810DH
0006H UNIT ?XD?CHKSM?PROGFLSH XDATA 8113H
0080H UNIT ?C_LIB_XDATA XDATA 8193H
0002H UNIT ?XD?MAIN?PAD XDATA 8195H
0002H UNIT ?XD?RXCALLBACK?PAD
C O D E M E M O R Y
0000H 0017H
GAP CODE 0080H 000FH UNIT PROGFLS
TSTA CODE 008FH 0055H UNIT PROGFLSA CODE 00E4H
01ADH UNIT ?PR?VPROG?PROGFLSH CODE 0291H
0073H UNIT ?PR?SEND?PROGFLSH CODE 0304H
001DH UNIT ?PR?RX?PROGFLSH CODE 0321H
0072H UNIT ?PR?CHKSM?PROGFLSH CODE 0393H 007EH I
NBLOCK SCC_INIT CODE 0411H 082EH UNIT ?C_LIB_CODE
SYMBOL TABLE OF MODULE XYZ VALUE TYPE N
AME ----------------------------------------------
--------------------------------------------------
---------------- - - - - - - PROC _FDECIMALASCII
TOBYTE X8301H SYMBOL p_b X8304H SYMBOL p_byA
scii X8307H SYMBOL sizeofAByAscii D0007H SYM
BOL fReturn D0006H SYMBOL bTemp - - - - -
- PROC _FDECIMALASCIITOWORD X8308H SYMBOL p_w
X830BH SYMBOL p_byAscii X830EH SYMBOL sizeof
AByAscii
Locator Maps
Maps provide a quick way of checking where the
locator actually placed segments.
Useful to know variable and function addresses
when debugging.
18
Executing out of RAM

• RAM is often faster than Flash and ROM.
• To exploit, startup code must copy program from
  ROM to RAM, then transfer control to it.
• Consider new challenge for locator
• Build a program that is stored at one address (in
  ROM), but will run correctly at a different
  address (in RAM).
• A bit tricky requires support from the RTOS
  development system.

19
9.3 How does program get to target?

• Several alternatives
• Write it to flash memory on target
• Put it in ROM or PROM, then insert chip into
  system
• Use a ROM emulator
• Use an in-circuit emulator
• Replaces microprocessor in target system
• Overlay memory in emulator can be used instead of
  memory on target
• Useful for debugging not in shipped products

20
PROM Programmer

• Used to program executable code into a PROM.
• The PROM should be socketed so it can be replaced
  easily.
• PROM approach good for production mode, but
  inconvenient for test and debug during
  development.
• Painful to pull, reinsert chip for every new
  test.
• Not surprisingly, other alternatives have been
  developed.

21
ROM emulator

• Plugs into PROM/ROM socket.
• Looks like ROM to target.
• Has connection to host to allow easy changes to
  memory.
• Much easier than burning a new PROM.
• Used during development and debugging only.
• Not shipped with working system!

22
Flash memory

• Flash is field programmable
• Host can connect to target, cause flash to be
  reprogrammed directly without replacing any
  chips.
• Software (bootstrap program) must be on target
  system to copy data from host to flash memory.
• Tricky cannot execute from flash while
  reprogramming it.
• Must copy itself from ROM to RAM and then run
  from RAM.

23
Field upgrades

• Product code fixes are very expensive.
• Product must be brought in to have memory
  upgraded or customer must install the memory
  upgrade.
• Some products lend themselves to automatic field
  upgrading.
• Satellite TV receivers set-top cable boxes
• Cell phones
• Software radios
• Tricky to do.
• Flash code could get corrupted (communication
  fails during update), then nothing works. How to
  prevent?

24
Chapter 10 Debugging

• From author
• If you write code with lots of bugs in it, you
  will ship code with lots of bugs in it.
• What does this say about testing and debugging?
• Why are these hard in general?
• Why are they even harder in embedded systems?
• How tolerant is the world of buggy software?
• Have we ever been exposed to anything else? ?
• How tolerant are we of buggy embedded systems?
• Phones, ATMs, medical equipment, digital cameras,
  iPods, etc.

25
Avoiding software bugs

• The best approach is to produce bug-free code,
  but no software is completely error-free.
• Although it cant be your principal technique for
  ensuring software quality, you are foolish to not
  do a lot of testing.
• Unfortunately, embedded systems pose special
  challenges for testing.

26
Problems testing on target system

• Target system may not be available or stable
  early on while code is being written and
  debugged.
• Difficult to generate pathological timing
  scenarios.
• Impossible to test all combinations, and
  difficult to know which combinations will cause a
  problem.
• Bugs are often not repeatable.
• Often show up with specific event sequence and
  timing.
• Tough to generate using standard software test
  suites.
• Embedded systems generally lack extensive logging
  capabilities to identify cause of failure.

27
Testing

• Conclusion dont rely on extensive testing on
  target system.
• Youll inevitably do some, but it cant be your
  main plan of attack.
• What else can you do?
• How about testing on the host?
• Debugging and testing is more convenient, but
  full code wont run.
• Timing will never be quite the same, so not
  useful for race-condition/ shared-data bugs, but
  useful for other classes of bugs.
• What can you do, how much work is required, and
  what does it buy you?

28
A method for testing on the host

• Separate application code into hardware dependent
  code (HD) and hardware independent code (HI).
• Essential creation of a clean interface between
  HD and HI.
• HI is just C code. Easy to compile and run on
  other computers.
• HD portion is more problematic.

Target system
Hardware-independent code
Hardware-dependent code
Hardware
29
Hardware dependent code

• It obviously is not portable, yet
• Actions of HI code essentially limited to
  responses to events/interrupts that come through
  the HD code which interfaces to all sensors,
  devices.
• All actions taken by application to change
  environment will use HD code which interfaces to
  all actuators.
• So your system cant really do anything
  meaningful without the HD code.
• How then could you test the HI code?

30
Test system
Target system
Hardware-independent code
Hardware-independent code
Hardware-dependent code
Test scaffold code
Hardware
Solution construct test scaffold code with same
entry points (API) as HD portion of
code. Scaffold code triggers actions in HI code
responds to actions by writing log files,
mimicking hardware actions, faking the behavior
of the hardware subsystems. Note that a detailed
hardware simulator is not necessary.
31
Testing the system

• ISRs and interrupt handlers must also be divided
  into HD, HI parts.
• Required organization HD code must call the HI
  part.
• Scaffold code can then mimic HD code, calling HI
  code to test response.
• This can be more effective, thorough than testing
  HI code on target.
• Example generating tick interrupts.
• Clearly responsibility of test scaffold code, but
  how to implement?
• Generate automatically at fixed time intervals?
• Generate directly and explicitly?
• The latter is preferred more likely to turn up
  bugs of strange combinations of events occurring
  within same tick interval.

32
Testing the system

• Important to create a scripting mechanism for
  convenience.
• Script files read by the scaffold code, specify
  events to generate.
• Since events are platform specific, youll want
  your own script language.
• General form Take this action (call this
  function) with these parameters at this time.
• Not difficult to create simple parser to read
  files, generate appropriate function calls, yet
  rewards are significant.
• Simple, powerful tools can make a big difference.
  
• Well worth the effort to consider test in design
  phase, build tools that make testing easier.
• Scaffold code can output results interleaved with
  script input easy to follow and confirm correct
  operation.

33
Sample script file

• For cordless bar-code scanner
• Each command causes scaffold code to call an
  interrupt routine
• kt0 call timer interrupt routine
• kn calls another timer routine specified number
  of times
• mr writes data into memory (as if received via
  radio) and calls radio interrupt routine

34
Sample output running script file
How useful would this mechanism be?
Output from scaffold code
35
Possible objections

• Too much of software is hardware dependent.
• Actually most software is hardware independent
  and can be tested in this way. (See next slide.)
• Scaffold software is too much work to create.
• Not that complicated it just reads a script and
  calls specified functions.
• Youd need a version of RTOS running on host.
• Most vendors happily supply this. Not a problem.

36
Fraction of Telegraph code that is HW dependent
37
Limitations of testing on host

• You cant test everything. Things you cant test
  include
• Software/hardware interaction
• Response time and throughput
• Shared data problems (and timing pathologies)
• Portability problems (endianness, packed
  data-structures, etc.)
• But it makes sense to test as much as you can
  before moving to the target for testing, which is
  more difficult.

38
Instruction set simulators

• Another way to test on host does not use
  scaffold code.
• Uses actual binary code (constructed by
  cross-compiler and linker/locator) that will run
  on target.
• Avoids portability problems with word-size,
  endianness, etc.
• Code is executed by simulator running on host.
• This approach tests both assembly and C code
  (unlike the scaffold approach).
• What are requirements for simulator?

39
Simulator requirements

• Must simulate all assembly instructions of target
  CPU.
• Must simulate all built in peripherals at some
  level.
• Timers, DMA, I/O devices, etc.
• Must simulate RAM and ROM at proper addresses.
• Should provide a debugger interface
• Set breakpoints
• Examine/change memory
• Single step execution
• Should track timing in terms of instructions or
  bus cycles.
• Can give accurate measurements of run time of
  various routines.
• Can help say something about throughput and
  response time.

How does our 425 simulator compare?
40
Simulator limitations

• Commercial system will know nothing about your
  custom hardware.
• Add yourself? Will vendor make source code
  available?
• Simulator is unlikely to discover obscure
  shared-data bugs.
• You are no more likely to do exhaustive testing.
• Tough to use scripting because simulators dont
  usually give access to hosts keyboard, screen,
  and file system.
• 425 simulator is unusual in this regard by
  design.
• Recommendation use simulator to test what cannot
  be tested using scaffold approach
• Startup code, ISRs, response time, etc.

41
Section 10.3 assert macros

• Low cost technique that catches lots of bugs.
• Sprinkle assert macro calls throughout code
• assert (pFrame ! NULL)
• assert (byMacAddrFrom lt ADDR_MAX)
• assert (pframe-gtbyMode MAX_MODE_USE_STATION)
  
• ...
• Makes your assumptions about machine state
  (global variable or parameter values) explicit.
• If condition is true, nothing happens. If false,
  output is generated, generally halting execution.
  Example
• Assertion failed ptr ! 0, file foo.c, line 27
  Abort (coredump)
• Implemented as macros so they can be turned off
  (undef debug) and thus generate no code in the
  final product.

42
Benefits of assertions

• Common use inspect parameters to functions.
• With assert( ), errors are caught much sooner,
  bringing the failure point closer to the error
  itself.
• Linux magazine June 2003
• Enthusiastic use of assert( ) can turn a
  three-day debug fest into a three minute bug fix.
  Practice the lazy developer mantra An assertion
  failed is an hour saved.
• Technique usually requires little more than
  include ltassert.hgt and calls to assert( ). (Not
  limited to embedded applications!)
• Moreover mere presence of assertions helps
  document operational details and assumptions.

43
Assertions in embedded systems

• Particularly useful in development and when
  testing on host.
• A bit harder to use on target system typically
  has no screen to write abort message to.
• Things you could do when assertion fails
• Make machine enter some easily detectable state.
  Examples
• Turn off interrupts, spin in loop.
• Turn on special pattern of LEDs.
• Write one or more special error codes to a
  specific memory location so you can determine
  what happened with a logic analyzer.
• Cause emulator or target debugger to stop
  execution somehow.
• Could execute an illegal instruction, for example.

44
Other tools

• Quote from text
• No book can do true justice to the experience of
  tracking down some subtle, inconsistent bug that
  only happens once every several hours and then
  only when your back is turned.
• You can probably relate, but worse on real
  projects due to limited I/O on target systems.
• What do you do? Bring in the heavy-duty tools.
• Volt meters, ohm meters, oscilloscopes, logic
  analyzers.
• Not part of typical programmers tool set!
• What can each of these do for you and what are
  limitations?

45
Volt meters, ohm meters, multi-meters

• Is the hardware working?
• Do all chips in the circuit have power?
• Is there a broken lead?
• Is the wiring possibly incorrect?
• Is a fuse blown?
• Is everything connected that should be?
• Is anything connected that shouldnt be?

46
Oscilloscopes

• Graphs voltage vs. time, potentially multiple
  signals
• Can select trigger to start its operation
• Typical questions that can be answered
• Is anything running?
• Is processor getting a decent clock input?
• Is memory getting chip-enable signals?
• Are output signals reasonable?
• Is there a loading problem or a bus fight?
• Storage oscilloscope
• Signals stored in memory, screen displays memory
  contents
• Can capture one-time events, but much more
  expensive

47
Logic analyzers

• Capture signals, store in memory, graph on
  screen.
• Can track many signals simultaneously.
• Up to several hundred if you are willing to pay
  and make all the connections!
• Typical operation trigger on symptom of problem,
  then look backward through captured data to see
  source of problem.
• Triggering mechanism can be very complex.
• Timing mode samples at fixed frequency once
  triggered.
• Captures data without reference to signals it
  records self-clocked.
• State mode capture based on events (clock) in
  system.
• Typical use see what instructions ran, what
  memory accessed.

48
In-circuit emulators

• Hardware emulator that plugs into CPU socket,
  appears to target system as regular
  microprocessor.
• Programmable or controlled by host.
• Functionality similar to desktop debugger
• Set breakpoints
• Single-step
• Dump register and memory contents
• Often includes overlay memory that can be used
  instead of actual memory in the target system.
• Overlays specify subset of memory, RAM or ROM.
• On memory accesses in specified ranges, emulator
  uses overlay.

49
Software-only monitor /debugging kernel

• Small debugging program in ROM on target system
  that knows how to
• receive software over serial line,
• copy to RAM, and
• run it.
• From interface on host, you can usually
• set breakpoints,
• examine memory and registers, etc.
• Typical methodology
• Compile code and download to target via monitor.
• Set breakpoints, run, and debug on target.
• Monitor is removed in final product.
• Requires no hardware modifications other than
  connection to host.
• Limitations timing changed, breakpoints
  problematic in real-time systems

50
Trends making testing more difficult

• Pins on chips are getting closer.
• Harder to attach instrumentation like logic
  analyzers, in-circuit emulators.
• ASICs, FPGAs are replacing many simpler parts.
• Much more internal state that cant be observed
  externally.
• Microprocessors with on-chip caches.
• You cant monitor accesses to internal cache.
• You can typically turn caches off, but this
  changes execution timing.
• Caches, pipelines complicate execution timing.

51
Chapter 11 A design example

• 75 pages of source code and discussion
• Simulates tank monitoring system
• Really works runs under DOS and µC/OS
• Hardware-independent part reasonably realistic
• Scaffold code replaces hardware-dependent code
• Simple user interface
• Look through it to get better feel for RTOS
  application code
• Is it organized as you would design it?

52
Authors afterword

• We never got a chance to design the tank
  monitoring system discussed at length in this
  book. It was brought to us at Probitas, the
  consulting firm where I work, already specified,
  designed, coded, and (supposedly) tested. The
  client brought it to us for some minor hardware
  and software enhancements.
• We made the hardware enhancements, fixing a
  few miscellaneous problems along the way, without
  too much difficulty. Then we delved into the
  software. It was written with a polling loop and
  some interrupt routines it did not use an RTOS.
  To get any kind of response, the software that
  calculated the levels in the tanks periodically
  saved its intermediate results and returned to
  the polling loop to check if the user had pressed
  any buttons. The software was written in
  interpreted BASIC. It was spaghetti.
• I leave it to your imagination to visualize
  the difficulties that we encountered trying to
  add features to this software without breaking it
  and without spoiling its response.
• This was a number of years ago now, and it
  would stretch the truth to say that I wrote this
  book in reaction to what I saw in that system.
  It gives me great satisfaction, however, to hope
  that this book will prevent at least a few
  similar horrors in the future.
• David E. Simon