Amen Anon et' al'

1
Amen Anon et. al.

• HPTS 99
• Are TP Benchmarks Still Relevant to the TP
  Community?

2
Summary

• Benchmarks drove much TP technology
• TPC benchmarks are not relevant to the rapid
  growth and rapid change of the Web environment
• Benchmarks are not challenging to the TP
  technical community

3
TPC Galvanized the TP Community

• Historically many of the base TP technology
  issues were developed to address high TP rates
  and win benchmarks
• Locking strategies
• Group commit
• Log striping
• Undo/redo strategies
• Etc. etc.
• See previous HPTS papers
• Along time ago 1000 (debit/credit) TPS was a
  dream and a motivator
• Now it is a common place vendor capability
• So what now?
• Have we lost our way?

4
TPC-C

• TPC-C is still the fairest benchmark for
  competition but it is
• Losing relevance
• Unrealistic configurations (RAID0 only, No
  failover)
• Mostly a database focus
• Measures price/performance and absolute
  performance but has no measure of
• Application Development costs
• Deployment costs
• Operations costs
• Application Maintenance
• Availability
• Cost and ease of scalability
• Yes there are H (1 result), R (1 result), and W
  is coming but
• Are they relevant to TP technologists?

5
TPC-C results
Top single system
1 64 CPUs 115396 (Sun/Oracle) 4 32 CPUs
102023 (HP/Sybase) 5 64 CPUs 93901
(Sqnt/Oracle/Dynix) 6 32 CPUs 92833
(HP/Oracle/HP-UX) 7 24 CPUs 53050
(Sun/Sybase) 8 16 CPUs 52117
(HP/Sybase/HP-UX)
Scale out

• 2 5 x 12 CPUs 110434 (IBM/Oracle/AIX)
• 3 8 x 12 CPUs 102542 (Alpha/Oracle/Unix)
• 9 2 x 22 CPUs 51871 (Sun/Oracle/Solaris)
  
• 10 4 x 8 CPUs 50208 (NT/NEC/Oracle)
• 21 4 x 4 CPUs 33936 (NT/Compaq/Oracle)
• 24 6 x 4 CPUs 27383 (NT/Tandem
  ServerNet/Oracle)
• 79 4 x 8 CPUs 14286 (IBM/AIX/PPC)

6
The Biggest TPC-C BenchmarkSunOracle115,395.73
tpmC _at_ 105.63 /tpmC 12.2 M available
8/22/99
7
TPM-C Relevance?
How much is 100 kTpmC?

• 100,000 users
• Each submitting 2.3 transactions/minute
• About 300 M transactions/day
• Greater than Yahoos page views during Christmas
  last year
• And remember
• Over 90 of the TPC-C transactions are buy vs.
  browse
• Average web site is less than 10 buy vs. browse

8
What happened to Scale Out

• Most results are not scale out
• Most Web sites start small and want to grow
  dramatically
• So far scale out hasnt won
• Is it what customers want?

9
New Benchmark

• Food for discussion during the next 2 days
• Benchmark
• Should focus on driving TP technology
• Should show incremental growth
• Closest to linear growth?

10
Summary

• Benchmarks drove much TP technology
• TPC benchmarks are not relevant to the rapid
  growth and rapid change of the Web environment
• Benchmarks are not challenging to the TP
  technical community