A Look at DSP Benchmarks

1
A Look at DSP Benchmarks

• Presentation by Michael Seto

2
Agenda

• Why we need benchmarks
• What makes up a good benchmark
• What are some commonly used BAD benchmarks
• Choosing the right benchmark
• Benchmarks used by the industry

3
Why Benchmarks

• Want to see which processor is better quickly.
  But better in what aspect?
• Benchmarks can be blend of anything
• Raw speed
• Power consumption
• Memory usage
• Cost

4
What should a benchmark be?

• Repeatable
• Relevant
• Fair
• Have comparable results

5
MIPS/MOPS VERY BAD

• Millions of Instruction/Operations per second
• 120 MIPS can be slower than 100 MIPS
• Some instructions do more work than others.
• Ex. Loading a 32bit constant on the 68k takes 2
  instructions just one on the SHARC

6
MMACS Still bad

• Theres no precise definition of what a MAC is.
• Does it include the memory overhead involved in
  fetching arguments?
• Does it include moving the output?
• Doesnt really indicate how well the processor
  will perform What if your application relies
  infrequently on MAC?

7
Applications Not Great

• Commonly used for non-DSP oriented processors.
  Ex. SPEC, BYTEmark
• VERY compiler dependent. May or may not use
  available fancy DSP features
• Even if implemented in assembly, its still a test
  of the programmers skill

8
Applications Contd

• May measure the evaluation kit that came with the
  DSP processor not just the DSP processor.

9
Which benchmark to choose?

• Architecture independence
• Should perfectly reflect what the DSP chip will
  be used for
• Should blend in factors such as cost, power
  usage, in the proportions you care about
• It probably doesnt exist

10
Trust the professionals

• Famous last words Lets trust the industry.
• Whats available
• Vendor benchmarks
• EEMBC (Embedded Microprocessor Benchmark
  Consortium)
• BDTI
• TI and Analog Devices quote benchmarks

11
Why do they use BDTImark?
12
What does BDTImark test?
13
(No Transcript)
14
Conclusions

• Benchmarks should be repeatable, relevant, fair,
  and readily-comparable
• Benchmarks test an application that probably
  isnt the same as yours
• Take benchmarks with a grain of salt

15
References

• http//www.eembc.hotdesk.com/eembc20frequently20
  asked20questions.jhtml
• http//www.us.design-reuse.com/articles/article255
  5.html
• http//www.bdti.com/bdtimark/BDTImark2000.pdf
• http//www.bdti.com/articles/benchmark_icspat99.pd
  f

16
Questions?