TPC Benchmarks: TPC-A and TPC-B

1
TPC Benchmarks TPC-A and TPC-B

• May 2002
• Prof. Sang Ho Lee
• Soongsil University
• shlee_at_computing.ssu.ac.kr

2
Who/What is the TPC

• Founded 1988 by 8 vendors (lead by Omri Serlin),
  now over 40 vendors/users
• Mission to define transaction processing and
  database benchmarks and to disseminate objective
  and verifiable TPC performance data to the
  industry
• De facto industry standards body for OLTP
  performance
• Administrated by
• Shenleey Public Relations
• 777 N. First St., Suite 600
• San Jose, CA 95112-6311
• Tel [phone]
• Most TPC specs, information, results are on the
  web page http//www.tpc.org/

3
TPC Current Benchmarks

• Benchmark C
• The current OLTP benchmark
• Two metrics Transactions-per-minute-C (tpmC) and
  price-per-tpm-C (/tpmC)
• Benchmark H
• Ad-hoc benchmark
• Two metrics Query-per-hour (QphH) and
  price-per-QphH (/QphH)
• Benchmark R
• Business reporting benchmark
• Two metrics Query-per-hour (QphR) and
  price-per-QphR (/QphR)
• Benchmark W
• Web commerce benchmark
• Two metrics web-interactions-per-second (WIPS)
  and price-per-WIPS (/WIPS)

4
TPC Obsolete Benchmarks

• Benchmark A
• Based on Debit/Credit benchmark
• Jun. 1995 obsolete
• Benchmark B
• Database half of Debit/Credit benchmark
• No network environment
• Jun. 1995 obsolete
• Benchmark D
• Complex decision support
• Multitable join, sorting, aggregation, ...
• Two metrics Power metric (QppD_at_size), Throughput
  (Qthd_at_size)
• Apr. 1999 obsolete

5
Aborted TPC benchmark efforts

• TPC Server benchmark (TPC-S)
• To create a server version of TPC-C
• Remove TPC-Cs front-end and remote terminal
  emulation (RTE) requirement
• Dec. 1995 resignation
• TPC-E, the Enterprise benchmark
• TPC-C is significantly more complex and robust
  than TPC-A, but still is not complex enough to
  stress very large, enterprise-class systems
• 1996 resignation
• TPC Client/Server
• Adding 3 transactions to TPC-C benchmark
• WWW changes the computing paradigm, so go to
  TPC-W
• mid 1995 resignation

6
TPC-A and TPC-B History

• 1985 DebitCredit benchmark
• Anon. et al,. A measure of transaction
  processing power, Datamation
• TP1 benchmark
• an implementation by vendors that typically
  departs from the DebitCredit specifications one
  way or another
• Aug. 1988 Transaction Processing Council formed
• Nov. 1989 TPC-A, OLTP with LAN or WAN
• Aug. 1990 TPC-B, OLTP with no network
• July 1992 TPC-C, On-Line Business Transaction
  Processing
• Jan. 1995 no new result on TPC-A/B (i.e. move to
  TPC-C)

7
TPC-A

• Published in November, 1989
• Councils version of the Debit/Credit test
• On-line transaction processing (OLTP) benchmark
• Use context of a bank application
• Metrics
• throughput tpsA-Local, tpsA-Wide
• price-per-performance dollars/tpsA-Local,
  dollars/tpsA-Wide
• About 300 TPC-A benchmark results were published
• Highest result was 3692 tpsA with a cost of
  4,873 per tpsA

8
Logical database design
9
TPC-A transaction profile

• Read 100 bytes including Aid, Tid, Bid, Delta
  from terminalBegin transaction Update Account
  where Account_ID Aid Read Account_Balance
  from Account Set Account_Balance
  Account_Balance Delta
• Write Account_Balance to Account
• Write to History Aid, Tid, Bid,
  Delta, Time_Stamp
• Update Teller where Teller_ID Tid
• Set Teller_Balance Teller_Balance
  Delta
• Write Teller_Balance to Teller
• Update Branch where Branch_ID Bid
• Set Branch_Balance Branch_Balance
  Delta
• Write Branch_Balance to Branch
• Commit transaction
• Write 200 bytes (Aid, Tid, Bid, Delta,
  Account_Balance) to terminal

10
Test configuration of TPC-A

• System under Test (SUT)
• one or more processing units (e.g., hosts,
  front-ends, workstations, etc.)
• hardware and software components of all networks
• data storage media, host system(s) supporting the
  database
• Driver System
• Remote Terminal Emulator (RTE) functionality
• Generates and sends 100 byte transactional
  message to the SUT
• Receives 200 byte responses
• Records message response times
• Performs conversion and/or multiplexing into the
  communications protocol
• Statistical accounting
• Driver/SUT communication interface

11
SUT, Driver, and Communication of TPC-A
12
Features of TPC-A

• Strong ACID with tests specified
• Response time constraints
• 90 of all transactions have a response time of
  less than 2 seconds
• Transaction arrival distribution random
• History file horizontal partitioning permitted
• Price
• all h/w and s/w over 5 year maintenance cost
  except physical communications media
• Detailed report required
• The test sponsor submits a full disclosure report
  (FDR)
• Response time is measured at driver

13
Scaling rules

• For each nominal tps configured, the test must
  use a minimum of
• Account records/rows 100,000
• Teller records/rows 10
• Branch records/rows 1
• History records/rows 2,592,000
• (90 eight-hour days 90
  8 60 60)
• Terminals 10

14
Pacing of transactions by emulated terminals

• Each emulated terminal, after sending a request
  to update the database to the SUT, must wait for
  a given Think Time after receiving that reply,
  before sending the next request

15
TPC-B

• Officially approved in August, 1990
• Councils version of the TP1 benchmark
• not OLTP benchmark
• No terminals with think time
• Batch transaction generator
• Metrics tpsB, dollars/tpsB
• About 130 TPC-B results were published
• Highest 2,025 tpsB with 254 per tpsB

16
TPC-A vs. TPC-B

• Similar with TPC-A
• same transaction profile, ACID requirements, and
  costing formula
• Difference from TPC-A
• Use of batch transaction generator
• No terminal emulation with think time
• No network configuration
• Concept of a user does not exist
• Response time is replaced by residence time (i.e.
  how long the transaction resides within the
  database server)