SPECweb2005

1
SPECweb2005 in the Real World Using Internet
Information Server (IIS) and PHP
01/27/2008 Sam Warner Steve Worley
2

Abstract

• Analysis of various systems
• Using IIS as web server
• Using PHP to server dynamic content
• SPECweb2005/PHP more representative to real
  world
• Results comparison
• PHP to published JSP results
• Dual-socket results and Multi-socket results

3
Introduction

• SPECweb benchmark has brought ability to show
  the relative performance of various systems
  manufactured in the Web Serving market
• SPECweb can be used for
• Research efforts at universities
• Within computer equipment manufactures to
  evaluate alternative technologies
• In the purchase process by Information Technology
  (IT) professionals
• SPECweb has two alternatives to enable
  performance evaluation
• Java Server Pages (JSP)
• All current published results
• Top results are unrealistic at approximately
  40000 users
• Hypertext Preprocessor (PHP)
• This paper provides results comparison not found
  in published results
• Help insure the benchmark remains useful in IT
  customer purchase process

4
Overview of the SPECweb2005 Benchmark

• Standard Performance Evaluation Corporation
  benchmark for evaluating the performance of World
  Wide Web Servers
• Supersedes SPECweb99 and SPECweb99_SSL
• Three workloads representing market segment that
  are part of the benchmark
• SPECweb2005_Banking
• SPECweb2005_Ecommerce
• SPECweb2005_Support
• Each workload variant enables measuring the
  maximum number of simultaneous user sessions
• Still achieving specific QOS metrics
• Still achieving error rate requirements

5
Overview of the SPECweb2005 Benchmark

• Benchmark score
• Individual sub-metric scores indicate the total
  number of simultaneous user sessions the server
  can support
• Overall score is the geometric mean of the three
  sub-metrics normalized to a reference platform
  score
• A score of 100 represents the same overall
  performance of the reference platform
• A score of 20,000 represents a score 200 times
  that of the reference platform
• The benchmark does not require a specific choice
  of web server software
• Only requirement is that the web server support
  HTTP 1.1 and SSL (HTTPS)
• Implementation of the web server will lead to
  differences in observed performance on the same
  system under test
• Current performance using the JSP kit achieves
  performance which is roughly 370x the reference
  platform

6
Top 15 SPECweb2005/JSP Scores
7
Overview of Top Fifteen Results

• All fifteen submitted with one of two web severs
• Sun Java System Web Server 7.0
• Accoria Rock Web Server V1.4
• 13 of the top 15 results
• Rock Web Server is undeniably fast
• No measurable market share from market analysis
  firms
• All results available at SPEC website
• http//www.spec.org

8
SPECweb2005 and the Real World

• IT customers use of performance benchmarks
• Seen as way to evaluate the complex combination
  of
• Number of users supported
• Content development language
• Web server software
• Web server hardware
• Dont have much time to understand the subtleties
  of a benchmark
• Forces OEMs to be constrained by competitive
  pressures to produce leading results
• Causes the publication of SPECweb2005 to diverge
  from real world IT performance

9
SPECweb2005 and the Real World

• Trend of web server connections supported
• Range of 20-2000 sustained customer connections
• As processing capacities have improved, so has
• Concerns and processing for security
• Complexity in formulating the responses for web
  server requests
• SPECweb99
• Results ranged from 400 to 25000 for the five
  years of publication
• Meanwhile, web server connection rates remained
  in the 1000s
• SPECweb2005/JSP
• Two and four socket systems have begun to achieve
  levels approaching 50,000 simultaneous users for
  a duration of ½ hour.
• In contrast to research papers where published
  connection loads tend to range from 2000 to 3000
  users
• SPECweb2005/PHP
• By comparing to SPECweb2005/JSP attempt is to
  bridge gap between real world loads and published
  results
• IT industry may be better served by comparisons
  to connection loads seen in their environments

10
SPECweb2005/JSP Scores since Inception
11
Analysis of Currently Published SPECweb2005
Results

• Request for Information and Request for Proposal
  processes have led many OEMs to develop
  publications for SPECweb2005.
• Forces OEMs to use software combinations that
  enable leading performance results
• Competition
• Great way to insure that the final purchased
  product meets the needs of the customer
• If benchmark closely matches the end-user
  environment, benchmark produces an informed
  purchase
• Top 15 results show connection rates that are
  10-20x higher than connection loads that are
  normally seen by web servers.

12
Comparison of Web Servers

• Of top fifteen SPECweb2005 results,
  thirteen use the Accoria Rock web Server v1.4
• Proprietary
• Non-open source
• No measurable market share
• Web Servers discussed in the paper

13
Web Servers Currently Used in Submissions

• Choice of Web Server up to OEM
• Results prepared as market collateral
• Market conditions dictate the decision of web
  server
• Best possible combination of hardware and
  software
• If less than ideal component used then consumer
  education required
• Results response to RFI
• Choice of highly relevant results to customer or
  industry leading results
• Lower cost to educate customer by simply
  achieving highest possible result
• Net result is that neither Apache or IIS,
  both recognized in their widespread use,
  have been used in any submissions in the last
  two years

14
Comparison of JSP to PHP performance
15
JSP use in submissions

• Presently all of the results submitted have used
  the JSP kit
• While JSP is widely used, it is not the primary
  content development language used
• To achieve leading industry results an OEM needs
  to use a software stack that has the lowest
  overhead to generate a response.
• The current implementations of SPECweb2005 in
  JSP has lower overhead than PHP

16
Analysis of Real World Workloads

• Industry results on SPECweb2005/JSP have
  improved by 50x since inception
• Publications from universities show that true
  connection loads trend at levels 1/10th of those
  observed in publications.
• SPECweb99 this same disconnect
• Using the SPECweb2005/PHP kit on equivalent
  systems we find that the change in connection
  loads observed is not as high.
• Intel Xeon X5355
• JSP shows 18,000-20,000 connections
• PHP shows 1000 connections

17
Web Server Statistics

• Aversa and Bestavros Paper
• TCP connection load of between 496 and 663
  requests served
• Response time between 0.92s and 0.26s
• Grottke paper
• Connection rates of 390 connections/second (c/s)
• Cao paper
• Connection rates of between 0.18 c/s and 34 c/s
• Connection rates of between 2.41 c/s and 230 c/s
• Seems reasonable to look at a scenario where the
  top connection rates are in the 2000-3000
  connections/second range

18
Discussion of use of IIS

• According to various web server market analysis
  companies approximately 35 of web servers on the
  internet use IIS
• Tends to be used in websites with higher traffic
  volumes
• No web server market analysis companies are able
  to measure the market share represented by Rock
  Web Server
• One is left to wonder how the IT industry
  correlates the performance reported from
  SPECweb2005 submissions to their own environment

19
Web Server Market Share
20
Discussion of use of PHP

• Variety of market analysis companies have
  analyzed the percentage of the market share for
  the various content development languages.
• These research firms show levels from 35 to 75
  for PHP
• Depends on whether the analysis includes all
  known websites or newer websites being actively
  developed
• The trend is that PHP development has increased
  from 25 to 50 of content development from
  2001 to 2007

21
SPECweb2005/PHP Results and Analysis

• The key value in a benchmark is its ability to
  accurately predict for the market segment
  intended the value of the new hardware or
  software component.
• With the SPECweb2005/PHP kit there appears to be
  more similarity
• Hardware requirements to achieve peak results are
  decreased dramatically.
• With the SPECweb2005/PHP results are closer to
  the default OEM configurations

22
Dual-Socket Results

• System chosen was Intel Xeon 5355 processor
  dual socket platform
• Support had the highest connection load
• Banking produced the lowest connection load
• Processor utilization
• 95 processor utilizations for Support and
  Ecommerce
• 80 processor utilization for Banking
• Banking workload using PHP has software
  serialization due in part to the session state
  having higher processing and IO requirements that
  limit the processor utilization achievable

23
Dual-Socket Response Times

• Response times for Ecommerce where nearly
  equivalent to SPECweb2005/JSP results
• Response times for Support were approximately
  0.83x that of JSP
• Response times for Banking were lower at 0.47x
  that of JSP
• Response times are nearly the same across the
  three workloads.

24
Multi-Socket Results

• System chosen was Intel Xeon 7140M processor
• Equivalent characteristics observed to two-socket
  system
• Connection rates improved slightly for Ecommerce
  and Banking
• Slight degradation on the Support workload
• Processor Utilization
• Support and Ecommerce higher than 90
• Banking slightly higher than 70 (Due to high
  software serialization)

25
Comparison of Configurations to Published Results

• Major determining factor for choice of JSP kit
  is the path length for JSP to server the dynamic
  content is much shorter than for PHP to perform
  the same task
• Significant contrast in hardware requirements
  required for a top performing JSP result and a
  top performing PHP result
• Network
• JSP requires 5-18 subnets of 1Gb
• PHP drops to two 1Gb subnets
• Disk
• JSP requires 10 times the storage

26
Conclusions

• By using the SPECweb2005/PHP kit a new system
  can be evaluated at a level of performance with
  respect to connection loads and response times
  that more closely matches a real world scenario.
• The amount of hardware required to analyze a new
  system with the SPECweb2005/PHP kit is much
  smaller than using the SPECweb2005/JSP kit

27

• www.cs.ualberta.ca/amaral/SPEC08